As some people will know, in the Barsoom series John Carter finds that Barsoom has two new colors, unknown on Earth. The people of Barsoom call them the Eighth and Ninth Rays. David Lindsay’s A Voyage to Arcturus (which I thought was earlier, but was actually published three years later, in 1920) has extra colors called ulfire and jale. HP Lovecraft’s The Color Out of Space has an unknown number of “shining bands unlike any known colours of the normal spectrum.”
Carl Sagan wrote that as a child he, inspired by Edgar Rice Burroughs, “spent many long minutes with my eyes tightly closed, fiercely concentrating on a new primary color. But it would always be a murky brown or a plum.”
There’s an article on this concept on the TVTropes wiki, but it doesn’t have any earlier fictional examples than Edgar Rice Burroughs (which doesn’t necessarily mean that he came up with the idea. TVTropes tends to be very focussed on more recent fiction).
Anyway, some scientists claim that they found a way to enable people to perceive ‘new’ colors. The link is here.
I remember when I was a child we had a View-Master, which allowed you to see 3-D images. Each 3-D image was a combination of two pictures. One of the ‘reels’ we had was based on the Transformers cartoon. They’d colored all the laser beams different colors in the two pictures. The effect was very strange – it wasn’t the same as what you’d get by mixing paint in the two original colors – and I wonder if that’s similar to the experiment described above.