Invert color in latent space?

[drawtheliner]

Inverting an RGB color is easy - just subtract each component from 1 (0.6 becomes 0.4, 0.1 becomes 0.9 etc.).

How can I do it in Latent space?

I have my colors in Latent, so I've been converting them to RGB to invert them, which leads to inconsistent results (due to Latent->RGB->Latent conversion). So if I could do it all in Latent would be better.