LUT for interactive demonstration of cone wavelengths and sensitivities

[domisjustanumber]

Hi everyone! I'm an artist working on a project can be a fun way to learn how human vision works. One element I'd like to include is an interactive tool that lets you play with different controls to experience how other people may see the world.

It seems like this library can probably do what I want, and I'd really love some help getting started as my use case doesn't seem to be a typical one.

The things I would like to do are:

1. Create a filter that extracts only the parts of an image that each type of cone would perceive. So an RGB image in, with 3x monochrome images out, each representing intensity of each pixel that the S, M and L cones would see.

2. Be able to change that filter so I could adjust the various cone peak sensitivites, so L cone from say 564nm to 580nm and see how it would affect the resulting image.

3. Reproduce various colour vision abnormalities
   This feels like it's a variation on the above, and somewhat covered in Why are the anomalous CVD LMS cone fundamentals colour matching functions from "Machado et al. (2009)" not normalized? #843

I'll be implementing this in TouchDesigner, so I think the way to do this is with a cube LUT that could take the input sRGB values (likely only 8 bit per channel, but I can convert it to linear and/or 32 bit float on the way in/out) and output the intensity each cone would perceive on a different channel of the RGB output.

i.e.

• S-cone intensity of each pixel output on the Blue channel
• M-cone intensity output on the Green channel
• L-cone intensity output on the Red channel

I can then split out the individual RGB channels in TouchDesigner and display and manipulate them from there. When controls are changed, I can use Python to re-generate a new LUT.

Could anyone help get me started with how to build this LUT?

[KelSolaar]

Hello,

I don't think you need to generate a LUT at all here, the colour.matrix_anomalous_trichromacy_Machado2009 definition generates a 3x3 matrix for a given RGB space that you could apply directly in GLSL within TouchDesigner. It should be really really fast!

Cheers,

Thomas

[domisjustanumber]

OH there are matrix_... functions too! Thank you so much. And also great idea to do this in GLSL. It looks like colour.matrix_cvd_Machado2009 and colour.matrix_anomalous_trichromacy_Machado2009 will do exactly what I want!

[domisjustanumber]

So I'm able to generate a 3x3 adjustment matrix based on the example code, and implemented it as a GLSL shader. It's super fast, so thank you for that suggestion!

However, it's still not quite doing what I'd expect, so I have a couple more questions.

I used the example code in the docs to generate the array:

from colour.characterisation import MSDS_DISPLAY_PRIMARIES
from colour.colorimetry import MSDS_CMFS_LMS
cmfs = MSDS_CMFS_LMS["Stockman & Sharpe 2 Degree Cone Fundamentals"]
d_LMS = np.array([15, 0, 0])
primaries = MSDS_DISPLAY_PRIMARIES["Apple Studio Display"]
matrix_anomalous_trichromacy_Machado2009(cmfs, primaries, d_LMS)

array([[-0.2777465...,  2.6515008..., -1.3737543...],
       [ 0.2718936...,  0.2004786...,  0.5276276...],
       [ 0.0064404...,  0.2592157...,  0.7343437...]])

1. In TouchDesigner, pixel values are 32bit float, 0-1 normalised, but the returned array seems to be in another range. Do you know what range the values are in? (e.g. the example above has 2.65 and -1.37 as output values)

2. If I use that example above, and pass [0, 0, 0] in as the adjustment vector, I still get a big change in the output array... yet I would expect no change at all (matrix of all 1s). Is that example the right way to be using the function?

[KelSolaar]

You might/will need to transpose your matrix in GLSL, I would run a few test cases against this:

import colour
import numpy as np
from colour.characterisation import MSDS_DISPLAY_PRIMARIES
from colour.colorimetry import MSDS_CMFS_LMS
cmfs = MSDS_CMFS_LMS["Stockman & Sharpe 2 Degree Cone Fundamentals"]
d_LMS = np.array([15, 0, 0])
primaries = MSDS_DISPLAY_PRIMARIES["Apple Studio Display"]
M = colour.matrix_anomalous_trichromacy_Machado2009(cmfs, primaries, d_LMS)

print(colour.algebra.vector_dot(M, [[0, 0, 0], [0.1, 0.2, 0.3], [1, 1, 1]]))

[[ 0.          0.          0.        ]
 [ 0.09039922  0.2255734   0.27279033]
 [ 1.          1.          1.        ]]

Note that the matrix itself is normalised as per-the-above.

[domisjustanumber]

I'm making progress - I've implemented colour.matrix_cvd_Machado2009 and my shader uses the matrix outputs to correctly adjust the picture based on the amount of Deuteranomaly, Protanomaly, and Tritanomaly. This is great!

However, I'm still getting some really weird results from matrix_anomalous_trichromacy_Machado2009

Setup with everything set to 0

If I now use the RGB transformation matrices from this:
colour.matrix_cvd_Machado2009("Protanomaly", 1)

I get this output matrix:

[[ 0.152286  1.052583 -0.204868]
 [ 0.114503  0.786281  0.099216]
 [-0.003882 -0.048116  1.051998]]

Which makes the image look as I'd expect:

(I'm using these images as reference)

However, if I then use colour.matrix_anomalous_trichromacy_Machado2009 the results seem to be very erratic. Looking a the values on a vectorscope it seems like maybe the calculations are rolling over the boundaries of colour spaces?

As this is best described with colours - here's an example video of me adjusting the L wavelength offset from 0 to 20 nm. You can see the image in the GLSL TOP goes wild and the colours on the scope sort of explode and reform again as I move through the scale:

I have confirmed that at 15nm I'm getting the same output as the example code in the docs, so I think I've set things up correctly there.

With the L offset at 20, I was expecting to get a matrix with similar values to the Protanopia one above, but instead I get:

[[ 0.18961355  1.4819917  -0.67160525]
 [ 0.07601187  0.60097371  0.32301442]
 [ 0.03109188  0.25260466  0.71630346]]

With an output image that looks like this:

And at 12 nm I'm getting extremely bizarre figures:

[[ -5.39006038  14.54919894  -8.15913856]
 [  1.95696406  -3.67029156   2.7133275 ]
 [ -0.20804356   0.71793259   0.49011098]]

With an output image that looks like this:

Similarly bizarre things happen when adjusting the M and S wavelengths - especially S that seems to bounce from one side of the extremes to the other every couple of nm in adjustment.

Is this something to do with the MSDS_DISPLAY_PRIMARIES or MSDS_CMFS_LMS values, should there be different values I pass in there?

[KelSolaar]

Hello,

I will have to check as this is certainly odd. I remember having issues matching the pre-computed matrices with the code that is meant to generate them. What you could do for the time being is using the precomputed matrices from that definition: colour.matrix_cvd_Machado2009.

Cheers,

Thomas

[KelSolaar]

Email I sent one of the authors in 2015, never got a reply:

Hi Gustavo,

I'm implementing your CVD model as part of our colour science API (https://github.com/colour-science/colour) and I had a few questions regarding the pre-computed matrices:

- Can you confirm that the matrices are meant to be applied on linear values. I think so but would like to triple check.
- I successfully managed to compute the Protanomaly and Deuteranomaly matrices, however I don't manage to find the same values for the Tritanomaly ones, is it possible to confirm the range you were using for them? From my code (if not wrong), the shift is somewhere between 5-59nm instead of 0-20nm:

Your pre-computed values for Tritanomaly (severity in domain [0, 1]):
        0.1: np.array(
            [[0.926670, 0.092514, -0.019184],
             [0.021191, 0.964503, 0.014306],
             [0.008437, 0.054813, 0.936750]]),
        1.0: np.array(
            [[1.255528, -0.076749, -0.178779],
             [-0.078411, 0.930809, 0.147602],
             [0.004733, 0.691367, 0.303900]])

Our computed values (shift in domain [0, 69]):

Shift: 0nm
[[ 1.  0. -0.]
 [ 0.  1. -0.]
 [-0.  0.  1.]]

Shift: 1nm
[[ 0.98061745  0.02458082 -0.00519827]
 [ 0.00561818  0.990856    0.00352582]
 [ 0.00210188  0.01241631  0.98548181]]

Shift: 2nm
[[ 0.96386322  0.04572412 -0.00958734]
 [ 0.01046594  0.98286764  0.00666642]
 [ 0.00398292  0.02401583  0.97200125]]

Shift: 3nm
[[ 0.9494437   0.06386329 -0.01330699]
 [ 0.01463071  0.97589022  0.00947906]
 [ 0.00565738  0.0348941   0.95944852]]

Shift: 4nm
[[ 0.9371156   0.07935433 -0.01646993]
 [ 0.01818481  0.96980396  0.01201123]
 [ 0.00713845  0.04513644  0.9477251 ]]

Shift: 5nm
[[ 0.92667567  0.09249219 -0.01916786]
 [ 0.02118844  0.96450879  0.01430277]
 [ 0.00843841  0.05481969  0.93674191]]

Shift: 6nm
[[ 0.91795292  0.1035231  -0.02147602]
 [ 0.02369211  0.95992044  0.01638745]
 [ 0.0095687   0.06401365  0.92641765]]

Shift: 7nm
[[ 0.91080258  0.1126541  -0.02345668]
 [ 0.02573846  0.9559673   0.01829424]
 [ 0.01054006  0.07278228  0.91667766]]

Shift: 8nm
[[ 0.90510154  0.12006027 -0.02516182]
 [ 0.0273636   0.95258822  0.02004818]
 [ 0.01136256  0.08118452  0.90745292]]

Shift: 9nm
[[ 0.90074479  0.12589035 -0.02663514]
 [ 0.02859825  0.94973059  0.02167116]
 [ 0.01204559  0.08927503  0.89867938]]

Shift: 10nm
[[ 0.89764263  0.13027111 -0.02791374]
 [ 0.02946853  0.947349    0.02318248]
 [ 0.01259793  0.09710479  0.89029727]]

Shift: 11nm
[[ 0.89571839  0.13331105 -0.02902944]
 [ 0.02999668  0.94540403  0.02459929]
 [ 0.01302773  0.10472158  0.88225068]]

Shift: 12nm
[[ 0.89490653  0.13510323 -0.03000976]
 [ 0.03020165  0.94386134  0.02593701]
 [ 0.01334252  0.11217041  0.87448707]]

Shift: 13nm
[[ 0.89515112  0.13572779 -0.03087891]
 [ 0.03009953  0.94269083  0.02720965]
 [ 0.01354924  0.11949388  0.86695688]]

Shift: 14nm
[[ 0.89640444  0.13525394 -0.03165838]
 [ 0.02970398  0.941866    0.02843002]
 [ 0.01365428  0.1267325   0.85961321]]

Shift: 15nm
[[ 0.89862595  0.13374168 -0.03236764]
 [ 0.02902658  0.94136341  0.02961001]
 [ 0.01366348  0.13392497  0.85241155]]

Shift: 16nm
[[ 0.9017813   0.13124327 -0.03302456]
 [ 0.02807706  0.94116221  0.03076073]
 [ 0.01358214  0.14110837  0.8453095 ]]

Shift: 17nm
[[ 0.90584144  0.12780448 -0.03364592]
 [ 0.02686362  0.94124366  0.03189271]
 [ 0.01341507  0.14831838  0.83826655]]

Shift: 18nm
[[ 0.91078159  0.12346613 -0.03424772]
 [ 0.02539321  0.94159074  0.03301606]
 [ 0.01316672  0.15558955  0.83124372]]

Shift: 19nm
[[ 0.91658017  0.11826547 -0.03484564]
 [ 0.02367183  0.94218759  0.03414058]
 [ 0.01284131  0.16295549  0.8242032 ]]

Shift: 20nm
[[ 0.92321764  0.11223764 -0.03545528]
 [ 0.02170487  0.94301913  0.035276  ]
 [ 0.012443    0.17044902  0.81710797]]

Shift: 21nm
[[ 0.93067543  0.10541708 -0.03609251]
 [ 0.01949741  0.94407056  0.03643203]
 [ 0.01197607  0.1781024   0.80992153]]

Shift: 22nm
[[ 0.93893483  0.09783882 -0.03677366]
 [ 0.01705454  0.94532698  0.03761848]
 [ 0.01144504  0.18594726  0.8026077 ]]

Shift: 23nm
[[ 0.94797606  0.08953965 -0.0375157 ]
 [ 0.01438165  0.94677303  0.03884532]
 [ 0.01085473  0.19401472  0.79513055]]

Shift: 24nm
[[ 0.95777731  0.08055911 -0.03833642]
 [ 0.01148467  0.9483926   0.04012274]
 [ 0.01021025  0.20233519  0.78745457]]

Shift: 25nm
[[ 0.96831389  0.07094053 -0.03925442]
 [ 0.00837041  0.95016853  0.04146107]
 [ 0.00951695  0.21093831  0.77954473]]

Shift: 26nm
[[ 0.97955723  0.06073199 -0.04028922]
 [ 0.00504681  0.95208235  0.04287084]
 [ 0.00878043  0.21985277  0.7713668 ]]

Shift: 27nm
[[ 0.99147395  0.04998723 -0.04146118]
 [ 0.00152327  0.95411403  0.0443627 ]
 [ 0.00800646  0.22910604  0.7628875 ]]

Shift: 28nm
[[ 1.00402482  0.03876661 -0.04279144]
 [-0.0021891   0.95624171  0.04594739]
 [ 0.0072012   0.23872409  0.7540747 ]]

Shift: 29nm
[[ 1.01716382  0.02713793 -0.04430175]
 [-0.00607719  0.95844153  0.04763566]
 [ 0.00637132  0.24873106  0.74489762]]

Shift: 30nm
[[ 1.03083717  0.01517713 -0.04601429]
 [-0.01012561  0.96068745  0.04943816]
 [ 0.00552418  0.25914881  0.73532701]]

Shift: 31nm
[[ 1.04498245  0.00296884 -0.04795129]
 [-0.01431647  0.96295114  0.05136533]
 [ 0.00466805  0.26999649  0.72533545]]

Shift: 32nm
[[ 1.05952807 -0.00939348 -0.05013459]
 [-0.0186292   0.96520201  0.05342719]
 [ 0.00381207  0.2812901   0.71489783]]

Shift: 33nm
[[ 1.07439315 -0.02180812 -0.05258504]
 [-0.02304051  0.96740747  0.05563304]
 [ 0.00296609  0.29304193  0.70399198]]

Shift: 34nm
[[ 1.08948801 -0.03416623 -0.05532178]
 [-0.02752446  0.96953329  0.05799117]
 [ 0.00214027  0.30526014  0.69259959]]

Shift: 35nm
[[ 1.10471498 -0.04635351 -0.05836147]
 [-0.03205266  0.97154419  0.06050848]
 [ 0.0013447   0.31794831  0.68070699]]

Shift: 36nm
[[ 1.1199696  -0.05825221 -0.0617174 ]
 [-0.03659463  0.97340452  0.06319011]
 [ 0.000589    0.33110511  0.6683059 ]]

Shift: 37nm
[[ 1.13514206 -0.06974332 -0.06539874]
 [-0.0411181   0.97507892  0.06603917]
 [-0.00011803  0.34472404  0.65539399]]

Shift: 38nm
[[ 1.15011832 -0.08070853 -0.06940979]
 [-0.04558941  0.97653296  0.06905645]
 [-0.00076856  0.35879333  0.64197523]]

Shift: 39nm
[[ 1.16478132 -0.09103206 -0.07374926]
 [-0.04997385  0.97773369  0.07224016]
 [-0.0013558   0.37329588  0.62805992]]

Shift: 40nm
[[ 1.17901237 -0.10060265 -0.07840972]
 [-0.05423605  0.97865024  0.07558581]
 [-0.00187403  0.38820931  0.61366473]]

Shift: 41nm
[[ 1.19269284 -0.10931571 -0.08337713]
 [-0.05834052  0.97925446  0.07908606]
 [-0.00231867  0.40350602  0.59881265]]

Shift: 42nm
[[ 1.20570591 -0.11707529 -0.08863062]
 [-0.06225218  0.97952152  0.08273066]
 [-0.00268618  0.41915347  0.58353271]]

Shift: 43nm
[[ 1.21793819 -0.12379579 -0.0941424 ]
 [-0.06593687  0.97943035  0.08650652]
 [-0.00297388  0.43511432  0.56785956]]

Shift: 44nm
[[ 1.22928123 -0.12940332 -0.09987791]
 [-0.06936186  0.97896405  0.09039781]
 [-0.00317965  0.45134679  0.55183285]]

Shift: 45nm
[[ 1.23963295 -0.13383673 -0.10579622]
 [-0.07249635  0.97811017  0.09438618]
 [-0.00330154  0.46780496  0.53549659]]

Shift: 46nm
[[ 1.24889903 -0.13704849 -0.11185054]
 [-0.0753119   0.97686089  0.09845101]
 [-0.0033375   0.4844391   0.5188984 ]]

Shift: 47nm
[[ 1.25699461 -0.13900559 -0.11798901]
 [-0.07778301  0.97521326  0.10256974]
 [-0.00328511  0.50119615  0.50208896]]

Shift: 48nm
[[ 1.2638468  -0.13969104 -0.12415576]
 [-0.07988784  0.97316961  0.10671824]
 [-0.00314171  0.51802026  0.48512145]]

Shift: 49nm
[[ 1.26939752 -0.13910534 -0.13029217]
 [-0.08160909  0.97073782  0.11087127]
 [-0.00290459  0.53485348  0.46805111]]

Shift: 50nm
[[ 1.27360616 -0.13726761 -0.13633855]
 [-0.08293472  0.96793165  0.11500306]
 [-0.00257128  0.55163661  0.45093467]]

Shift: 51nm
[[ 1.27645183 -0.13421602 -0.14223581]
 [-0.08385863  0.96477069  0.11908794]
 [-0.00213982  0.56831015  0.43382967]]

Shift: 52nm
[[ 1.27793494 -0.13000757 -0.14792737]
 [-0.08438111  0.96128016  0.12310096]
 [-0.001609    0.58481523  0.41679377]]

Shift: 53nm
[[ 1.27807814 -0.12471716 -0.15336097]
 [-0.08450909  0.95749051  0.12701858]
 [-0.00097859  0.60109465  0.39988394]]

Shift: 54nm
[[ 1.27692633 -0.11843592 -0.15849041]
 [-0.08425611  0.9534368   0.13081931]
 [-0.00024956  0.61709383  0.38315573]]

Shift: 55nm
[[ 1.27454618 -0.11126915 -0.16327703]
 [-0.08364214  0.9491579   0.13448424]
 [ 0.00057578  0.63276178  0.36666244]]

Shift: 56nm
[[ 1.27102494 -0.1033339  -0.16769104]
 [-0.08269323  0.94469567  0.13799756]
 [ 0.00149354  0.64805197  0.35045448]]

Shift: 57nm
[[ 1.26646832 -0.09475599 -0.17171232]
 [-0.08144081  0.94009392  0.14134689]
 [ 0.00249827  0.66292312  0.33457861]]

Shift: 58nm
[[ 1.26099741 -0.08566654 -0.17533087]
 [-0.07992082  0.93539732  0.1445235 ]
 [ 0.00358303  0.67733973  0.31907723]]

Shift: 59nm
[[ 1.25474504 -0.07619839 -0.17854666]
 [-0.0781726   0.93065023  0.14752237]
 [ 0.00473967  0.69127251  0.30398782]]

Shift: 60nm
[[ 1.24785188 -0.06648273 -0.18136915]
 [-0.07623774  0.92589565  0.15034209]
 [ 0.00595914  0.70469845  0.28934241]]

Shift: 61nm
[[ 1.24046263 -0.05664624 -0.1838164 ]
 [-0.07415902  0.92117439  0.15298463]
 [ 0.00723187  0.71760077  0.27516736]]

Shift: 62nm
[[ 1.23272286 -0.04680885 -0.18591401]
 [-0.07197943  0.91652441  0.15545502]
 [ 0.00854803  0.72996871  0.26148326]]

Shift: 63nm
[[ 1.2247766  -0.03708255 -0.18769404]
 [-0.06974146  0.91198045  0.15776101]
 [ 0.00989767  0.74179721  0.24830512]]

Shift: 64nm
[[ 1.21676463 -0.02757079 -0.18919384]
 [-0.06748663  0.90757395  0.15991267]
 [ 0.01127071  0.75308662  0.23564267]]

Shift: 65nm
[[ 1.20882334 -0.01836832 -0.19045502]
 [-0.06525508  0.90333307  0.16192202]
 [ 0.01265699  0.76384221  0.2235008 ]]

Shift: 66nm
[[ 1.20108384 -0.00956138 -0.19152246]
 [-0.06308536  0.89928273  0.16380264]
 [ 0.0140463   0.77407376  0.21187993]]

Shift: 67nm
[[ 1.19367091 -0.00122772 -0.1924432 ]
 [-0.0610141   0.89544476  0.16556933]
 [ 0.01542857  0.78379493  0.20077651]]

Shift: 68nm
[[ 1.18670162  0.00656369 -0.19326531]
 [-0.05907566  0.89183795  0.16723771]
 [ 0.01679438  0.79302234  0.19018328]]

Shift: 69nm
[[ 1.18028382  0.0137529  -0.19403671]
 [-0.0573018   0.88847803  0.16882377]
 [ 0.01813565  0.8017747   0.18008965]]

Any help would be highly appreciated :)

Cheers,

Thomas

[domisjustanumber]

Oh wow sifting through the archives there, thanks for looking!

I think you may be onto something in the log vs. linear colourspace idea, but I figured this is easier to show than describe so I made a video to show what's going on with the two different functions and their outputs: https://www.youtube.com/watch?v=ABs1xtE4Fg0

[KelSolaar]

Hello,

Thanks a lot for taking the time to make a YouTube video, this is awesome! :) If you look at the unit tests here (I added some more), you will find parameters for colour.matrix_anomalous_trichromacy_Machado2009 that should match the precomputed matrices, with the caveat that Tritanomaly is in range [5, 59] compared to [0, 20] for Protanomaly and Deuteranomaly:

colour/colour/blindness/tests/test_machado2009.py

Line 135 in 2c55196

cmfs = MSDS_CMFS_LMS.get("Smith & Pokorny 1975 Normal Trichromats")

Let me know how it goes.

[KelSolaar]

Actually, here is a notebook with some computations: https://colab.research.google.com/drive/1-Bx2w2k2YQnbek-umndNLRTMgmMFmFc-#scrollTo=X-XGaP69S59q

Try to use the "Typical CRT Brainard 1997" display as the "Apple Studio Display" might be causing what you see.

Protanomaly

Deuteranomaly

Tritanomaly

Here is for good measure how to apply the matrix in GLSL: https://www.shadertoy.com/view/dlKyWt

[domisjustanumber]

Ahh you're amazing - switching both the colour spaces to Smith & Pokorny 1975 Normal Trichromats and Typical CRT Brainard 1997 fixed everything and I'm now getting the same results as your sample images.

Thank you so much!

I also sat down with a cup of tea to read the paper and now understand which direction each of the cone wavelength changes are moving (higher is converged towards green/centre)... but now after cracking that joke about people seeing infra red I'm wondering if there are genetic mutations that shift the L cone sensitivity to longer wavelengths not just lower ones (or indeed S cones to shorter wavelengths and can see more UV), but we never test for it because we're looking for "less than normal" vision... hmm...

[KelSolaar]

The LMS CMFS should not change too much the outcome but the display primaries will for sure! Glad it is working for you now! I would not be surprised that there are people with offset sensitivities that are not in the three recognised ones, there are tetrachromats after all and many animals have different peaks. Worth noting that cameras have the red sensitivity offset toward infrared for better performance, i.e. less noise because of better discrimination due to less overlap with the green sensitivity.