How to perform colour correction with a "ColorChecker" reference image?

[bsdis]

Hi,
I have images of a colorchecker and I would like to perform color calibration of these images.
As I understand I have to do extract the 24 colour squares of the colorchecker, Convert from sRGB to RGB by applying inverse gamma, convert RGB to XYZ using 3x3 matrix, convert XYZ to Lab using standard formula and D6500 white point - perform a regression to find conversion coefficients - and then convert all pixels using the coefficients and the finally transform the converted image back into sRGB space.
Is that correctly understood, and is that possible to do using this library? (all the color conversions - not the extractions from images etc)
If yes - are there any examples available?

thanks

[KelSolaar]

Hi @bsdis,

It is absolutely possible and I do that quite regularly.

Assuming you have the data for the colour rendition chart, at the basic minimum, this is what you need to do:

colour/colour/characterisation/correction.py

Line 852 in c3245b1

>>> RGB = np.array([0.17224810, 0.09170660, 0.06416938])

We have a repository which does extraction and provides an example of colour correction here: https://github.com/colour-science/colour-checker-detection/blob/develop/colour_checker_detection/examples/examples_detection.ipynb

[bsdis]

Hi @KelSolaar
Ahh ok. Really nice :)
I hope its ok to ask some follow up questions:

1. In the notebook you preprocess your images with dcraw -T -g 2.4 12.92 *.CR2. This basically encodes images as sRGB png images right?
2. I ran your example on 4 test images i had of the same scene where i have changed the lighting in each scene.
   I get this result which is pretty nice actually

But it seems that especially if you look on the white background its still not the same on the "after calibration" images. Is this to be expected? Can that be fixed somehow?

[bsdis]

Here is actually a better image. I have multiplied with and capped at 255, which also shows that some channels are saturated after the calibration (First 4 are before calibration and last 4 are after).

Is this because i need to do a white balancing also?

[KelSolaar]

Hi @bsdis,

This is a great example of why relying on a colour rendition chart for colour correction is not the solution to all the imaging calibration problems as $COLOUR_RENDITION_CHART_MAKER claims it is.

What is happening here is that your scene illumination is changing while the ColorChecker Classic values are given for a particular illuminant, i.e. CIE D50. Ideally for things to really work, your scene should be lit with CIE D50 so that you can perform colour correction against the published values.

Ideally, you would need to measure the spectral irradiance of the scene and the spectral reflectance of the ColorChecker Classic (with a spectro-photometer) and recompute its values for the particular scene.

Alternatively, something cheap is to neutralise on the bottom row after the colour correction but it is obviously not ideal!

I hope that makes sense!

[bsdis]

Hi @KelSolaar

Aha - yes that makes perfect sense. I have to keep that in mind.
If I first wanted to try the cheap method, you say "neutralise on the bottom row after the colour correction". How exactly is that done? In your code, the final image is generated by doing:

    im = colour.cctf_encoding(
        colour.colour_correction(
            COLOUR_CHECKER_IMAGES[i], swatches, REFERENCE_SWATCHES))

So I guess im is in sRGB colorspace? Would i do the neutralise directly on this image, or would I convert it to another colorspace first?
Also - would you cut a square from each of the bottom patches to neutralize after, and how would you do it?

Sorry - I hope its not to many questions.

Thanks :)

[KelSolaar]

Yeah, so you would need to neutralise before that conversion, colour correction, white-balancing and, most of the operations must ideally be done on floating-point linear data. The idea is to compute the average RGB colour of one of the swatches of the bottom row, we commonly take the third from the bottom right in the VFX industry, divide your image by it so that this swatch is white (or close to it because you only took the average) and then multiply the image by the value of that swatch, it should be around [0.19~, 0.19~, 0.19~] for sRGB.

[bsdis]

Hi @KelSolaar

Hmm I think I might have misunderstood you.
I understood it like adding these 2 lines:

    im_ = COLOUR_CHECKER_IMAGES[i]/colour_checker.data['neutral 5 (.70 D)']
    im *= im_

So code becomes like this

for i, swatches in enumerate(SWATCHES):
    swatches_xyY = colour.XYZ_to_xyY(colour.RGB_to_XYZ(
        swatches, D65, D65, colour.RGB_COLOURSPACES['sRGB'].RGB_to_XYZ_matrix))

    colour_checker = colour.characterisation.ColourChecker(
        os.path.basename(COLOUR_CHECKER_IMAGE_PATHS[i]),
        OrderedDict(zip(REFERENCE_COLOUR_CHECKER.data.keys(), swatches_xyY)),
        D65)
    
    plot_multi_colour_checkers(
        [REFERENCE_COLOUR_CHECKER, colour_checker])
    
    swatches_f = colour.colour_correction(swatches, swatches, REFERENCE_SWATCHES)
    swatches_f_xyY = colour.XYZ_to_xyY(colour.RGB_to_XYZ(
        swatches_f, D65, D65, colour.RGB_COLOURSPACES['sRGB'].RGB_to_XYZ_matrix))
    colour_checker = colour.characterisation.ColourChecker(
        '{0} - CC'.format(os.path.basename(COLOUR_CHECKER_IMAGE_PATHS[i])),
        OrderedDict(zip(REFERENCE_COLOUR_CHECKER.data.keys(), swatches_f_xyY)),
        D65)
    
    plot_multi_colour_checkers(
        [REFERENCE_COLOUR_CHECKER, colour_checker])
    
    ##### ADDED WHITE BALANCING HERE:
    im_ = COLOUR_CHECKER_IMAGES[i]/colour_checker.data['neutral 5 (.70 D)'] # divide using broadcasting
    im *= im_
    ##########################
    im = colour.cctf_encoding(
        colour.colour_correction(
            im, swatches, REFERENCE_SWATCHES))
    b4after.append( [COLOUR_CHECKER_IMAGES[i],im] )
    plot_image(im);

But this gives me quite strange results.
What did i misunderstand here?

[KelSolaar]

You would need to divide by the average value of the colour checker in the image, i.e. COLOUR_CHECKER_IMAGES[I] and multiply by that of the reference one colour_checker.data['neutral 5 (.70 D)']

[bsdis]

As I understand the code, the following variables are:

COLOUR_CHECKER_IMAGES[i] <--- Sample image

colour_checker.data['neutral 5 (.70 D)']) <--- average value of neutral 5 swatch of sample image

REFERENCE_COLOUR_CHECKER.data['neutral 5 (.70 D)'] <--- reference value of colorchecker for neutral 5

So: divide by the average value of the colour checker in the image, i.e. COLOUR_CHECKER_IMAGES[I] and multiply by that of the reference one
would be:

im = (COLOUR_CHECKER_IMAGES[i]/colour_checker.data['neutral 5 (.70 D)'])*\
        REFERENCE_COLOUR_CHECKER.data['neutral 5 (.70 D)']

Is that correct?

However it still gives quite different backgrounds though.

So I guess thats due to my scene illumination changing while the ColorChecker Classic values are given for a particular illuminant like you mentioned, right?

What I actually wanted to achieve was to be able to take an arbitrary image under an arbitrary illumination and and arbitrary camera, and somehow calibrate those into same colorspace, so that I could directly compare the colors in the in images. In each picture there would always be a colorchecker of course.
So I guess thats actually impossible then?

[KelSolaar]

Is that correct?

Not really, you would need to divide each given image by its own colour checker Neutral 5 swatch average, because each image has its own illuminant, you need to neutralise for each one of them accordingly and individually.

So I guess thats due to my scene illumination changing while the ColorChecker Classic values are given for a particular illuminant like you mentioned, right?

Assuming you have neutralised each one of them, you will see that not only their whites are closer, but also that their exposure is roughly the same! You will still have discrepancies though because of the illuminant differences.

[bsdis]

Hi @KelSolaar
Hope its okay i keep this thread running.
In your notebook (https://github.com/colour-science/colour-checker-detection/blob/develop/colour_checker_detection/examples/examples_detection.ipynb)
you do:

    plot_image(colour.cctf_encoding(
        colour.colour_correction(
            COLOUR_CHECKER_IMAGES[i], swatches, REFERENCE_SWATCHES)));

the output of colour.cctf_encoding seem to be in the range -0.52 to 1.47. Should that not be in the range 0 to 1?

[KelSolaar]

Hi,

Hope its okay i keep this thread running.

Absolutely and good question! We never (pretty much never) clip data in Colour because this is what allows inversion of the transforms. Yes, you will often get negative values and values over one, this is by design. Whether it makes sense is something we are expecting the user to assess and is dependent on the usage. For your current case, because you are plotting an image, it would make sense to clip the values. They are effectively being clipped by plot_image just before being sent to plt.imshow because Matplotlib will raise an exception otherwise.

[bsdis]

I see. Instead of clipping, wouldn't it make more sense to keep the information as much as possible by shifting to min=0 and divide with max value to get 0<=x<=1 ?

[KelSolaar]

Good question again!

We cannot really decide for the user here, given we are often working with negative out-of-gamut colours or very large HDR values, e.g. [1.e7, 1e9], the image would be entirely black except for a small speckle, i.e. the Sun, in a lot of cases.

A single large value is needed to throw you off trajectory here and it happens all the time in the type of applications we use Colour for. We really do swim in an ocean of nans and infs :)

[bsdis]

haha yes ok. thats a really good point :D
Btw. I tried changing method to method='Finlayson 2015', but that seems to give exactly the same results as if that parameter is omitted....even down to the 12th decimal...isn't that a bit curious? Maybe the code always redirects to the same method nomatter what the user sets?

[KelSolaar]

depending on the number of terms/degree you use, you will get the same results, basically, all the methods are equivalent for degree 1 (linear).

[bsdis]

@KelSolaar I was hoping I could maybe get your opinion on the path I have chosen so far.

My usecase is that I need to compare colors in different images across many images.

The challenge here is that I have no idea about the lighting used on each image and I have no idea about the camera used for each image (could be some phone cam or some DSLR cam or whatever). The only thing I know is that an xrite colorchecker is present on each image taken. Input images are in sRGB jpeg which is default output from many devices..
As Initially stated i need to be able to compare colors across all images, which means i need to transform all images into same colorspace-coordinate system so to say. I know that it will be impossible to perfectly normalize the colors as long as i have no control of the lightsource and the device, but I am looking to do it as good as possible. For future images i can change the xrite colorchecker with something more appropriate if need be. Is the color-correction course the right way to go for a problem like that?

[KelSolaar]

I would maybe try to see if you can infer the illuminant from the scene, there is a great deal of literature on the topic, e.g. https://www.groundai.com/project/rehabilitating-the-color-checker-dataset-for-illuminant-estimation/. I never tried any of that unfortunately but it is certainly on my radar when time permits!

Otherwise, your approach is certainly correct!

[KelSolaar]

This publication is of particular interest: https://arxiv.org/abs/1504.04548

[bsdis]

AHa...that sounds really interesting... I will look into that. Thanks a lot for the pointers! Will be sure to feed back what I find out :)

[KelSolaar]

Closing this one for now! Feel free to add comments and I will re-open!

[alianoroozi]

Hi @KelSolaar,

I have some questions:

1. Should we apply white-balancing (neutralization) after color correction? If yes, should we use mean value of 'neutral 5 (.70 D)' swatch after color correction or before it? I am guessing after correction.
2. Should we do white-balancing in Lab color space or sRGB space?

Thanks

[KelSolaar]

Hi,

From a conceptual standpoint, there are only two places to white-balance:

• In the camera sensitivities space, e.g. Camera RGB.
• In the Standard Human Observer Cone Responses (LMS) space.

Whether to choose one space over the other depends on the usage of the imagery. As to why only those two spaces: those are the only places where there is theoretically no channel correlation or the least of it. Any other space might have channel cross-correlation and thus because the basis are not aligned with the camera or human observer responses basis will introduce some form of skewing in the colours.

Now to go back to your questions and starting with the last, i.e. 2, white balancing would ideally happen in cone space, i.e. using colour.chromatic_adaptation definition where you pick the source and target CIE XYZ tristimulus values. As for question 1, colour correction should white balance the source, within reason, if the target is white balanced. You could effectively neutralize after the fact, which would be mostly fine given that the colour correction matrix has introduced a lot of RGB channel correlation.

Here is something of interest we wrote a few months ago: https://www.colour-science.org/posts/the-colorchecker-considered-mostly-harmless/

[alianoroozi]

Thanks

[MMehdiMousavi]

Hi, Im trying the notebook to do a color correction of a custom image. However, I cant seem to get the thing to run correctly at all (even with no modifications). this is the error I encounter:

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-23-f039509c97de> in <module>()
      2 for image in COLOUR_CHECKER_IMAGES:
      3     for swatches, colour_checker, masks in detect_colour_checkers_segmentation(
----> 4         image, additional_data=True):
      5         SWATCHES.append(swatches)
      6 

C:\Users\Mehdi\AppData\Roaming\Python\Python36\site-packages\colour_checker_detection\detection\segmentation.py in detect_colour_checkers_segmentation(image, samples, additional_data)
    767     colour_checkers_colours = []
    768     colour_checkers_data = []
--> 769     for colour_checker in extract_colour_checkers_segmentation(image):
    770         colour_checker = cctf_decoding(
    771             as_float_array(colour_checker[..., ::-1]) / 255)

C:\Users\Mehdi\AppData\Roaming\Python\Python36\site-packages\colour_checker_detection\detection\segmentation.py in extract_colour_checkers_segmentation(image)
    691     for colour_checker in colour_checkers_coordinates_segmentation(image):
    692         colour_checker = crop_and_level_image_with_rectangle(
--> 693             image, cv2.minAreaRect(colour_checker))
    694         width, height = (colour_checker.shape[1], colour_checker.shape[0])
    695 

C:\Users\Mehdi\AppData\Roaming\Python\Python36\site-packages\colour_checker_detection\detection\segmentation.py in crop_and_level_image_with_rectangle(image, rectangle)
    459     width_r, height_r = as_int_array([width_r, height_r])
    460 
--> 461     M_r = cv2.getRotationMatrix2D(centroid, angle, 1)
    462 
    463     image_r = cv2.warpAffine(image, M_r, (width, height), cv2.INTER_CUBIC)

TypeError: Can't parse 'center'. Sequence item with index 0 has a wrong type

anyone know how to fix this?

[KelSolaar]

Hi @MMehdiMousavi,

The released version of "colour-checker-detection" has some know problems with latest versions of OpenCV. Please try the develop branch and let us know how it goes! We are working to release the new version soon.

Cheers,

Thomas

[MMehdiMousavi]

Hi and Thanks for the reply.
I was able to make it work using opencv 4.4.0,
Color correction isn't really my forte so I appreciate it if you can point me in the right direction.
I'm trying to color correct a synthetic image, where the scene illumination is affected by heavy scattering. Im including a virtual color chart as a reference so that I can find the correct color transformation filter (where the colors on the chart are exactly what they should be) and save it for reuse.

from what I understand reading the discussion above, that depends on the lighting profile the chart was lit with, is there any other way to achieve results without manually messing with the colors?

Thanks

[KelSolaar]

Hi,

This is a bit of a complex scenario here because you not only have to account for the illumination but also water absorption whose effect changes as a function of distance. I would recommend reading the Sea-Thru publication as it gives a good idea of the type of process involved.

Cheers,

Thomas

[rakeshwar07]

Hi @KelSolaar @bsdis @alianoroozi ,

I have been following your conversation with bsdis. It was very usefull. Instead of 24 colour average I am using a algorithm to find the dominant colour. It seems to work. But I am consistently facing an issue with calibrating some images that has more brigtness in it.

Also, I tried the white balancing as you mentioned earlier in the threat. But it woresens my problem after including it. I am attaching the code and the results. Please let me know where I am going wrong. And what I can do to solve that type of issue

?

[KelSolaar]

Your reference RGB colours here are expressed as 8-bit, most likely encoded values, they need to be represented as linear floating point values before attempting any correction.

[danoneata]

Hello and thank you for the very nice and comprehensive library!

I'm trying to perform color correction in a scenario in which I have a strong shading (vignetting) effect and I would need some advice on how is best to proceed.

Concretely, the image of the checker looks like this.

In order to remove the vignetting effect I divide the above image by an image of the white board, obtaining the following result:

I have a couple of questions.

1. When converting the reference colors to (linear) RGB (XYZ_to_RGB) I need to provide the illuminant of the scene where the photo is taken. What value should I use if I work on the vignette-corrected image? Currently, I'm using colour.RGB_COLOURSPACES["sRGB"].whitepoint (which corresponds to the D65 illuminant, as far as I understand), but I'm not sure if this is correct:

REFERENCE_SWATCHES = colour.XYZ_to_RGB(
    colour.xyY_to_XYZ(list(REFERENCE_COLOUR_CHECKER.data.values())),
    REFERENCE_COLOUR_CHECKER.illuminant,
    colour.RGB_COLOURSPACES["sRGB"].whitepoint,
    colour.RGB_COLOURSPACES["sRGB"].matrix_XYZ_to_RGB,
)

(I know that the light used for capturing the original image had a temperature of 5000K, but I don't know it's spectrum.)

2. It looks like the results are sensitive to the location of the checker in the original image the checker. In some cases the white background gets a yellowish tint:

• original image
  
• after vignette correction
  
• after color correction
  
  (For the above experiment, I've used a linear function—Finlayson 2015, with degree 1 and no root expansion.)

Thank you in advance and forgive my lack of knowledge; I was not familiar with intricacies of color science until very recently!

[KelSolaar]

Hello @danoneata,

1. It depends what on what you are trying to achieve. Typically, you would use your working space illuminant, e.g. D65 if sRGB.
2. This is expected as there is no colour preference in the fitting. Put another way, the error is minimised without favouring a colour over another thus achromatic colours can become chromatic. You can neutralise the output as a last step.

Cheers,

Thomas

[danoneata]

Thanks a lot for the reply, Thomas!

1. I'm a bit confused. Where does the illuminant used to light the scene come into play in the color correction process? From your very persuasive post I understand that illumination is an important aspect in getting sensible color correction results, but in the example I don't know where this information is used. Initially, I thought that one of parameters passed to the colour.XYZ_to_RGB correspond to the scene illuminant, but now I'm not sure about this. For example, if my scene was lit by a light with temperature 5000K, wouldn't using D50 be more appropriate than D65?
2. Thanks! I think balancing one of the neutral tones is a good idea. One thing that I'm observing, is that while I fix one of the values (e.g., white), the others may still preserve some chromacity. But I don't know if there is a fix for this; or maybe is pointing to some other flaw in my setup.

[KelSolaar]

The illuminant is critical when you are in a position where you can measure spectral data: At this point and with knowledge of the spectral reflectances of the chart and that of the scene illumination, you can generate precise reference values. Without the spectral data changing the whitepoint will be very much akin to white-balancing, so it is not important. You might as well pick the working space one because having white staying white is probably the next important aspect.

[danoneata]

Thanks a lot for all the explanations, Thomas! They were very instructive!

Just a quick update: the reason why I had trouble fitting the neutrals was simply because the original image was already in the linear space and I was unaware of this, so I'd applied a non-linear decoding function 😐