Add CICP color profile support

[VentuzTammoHinrichs]

Hey,

I'd like to make the case for adding CICP (Coding-Independent Code Points) color profiles to ImageSharp.

In a nutshell, CICP is a simpler way of specifying the color space of image/video data than embedding a full ICC profile. It basically boils down the color space to a bunch of enums with the most common sets of color primaries, transfer functions and matrix coefficients (for YUV, etc), so all you need to know the exact color space are four bytes.

It's ratified as standards ITU-T H.273 and ISO/IEC 23091-2 2019, full spec including all the math is here: https://www.itu.int/rec/T-REC-H.273-201612-S/en

This is a bit of an investment into the future - as it stands, the only image format included in ImageSharp that currently supports CICP is PNG as per the not yet fully done third edition ( https://www.w3.org/TR/png-3/ §11.3.2.6 ), but: Major applications such as Chromium are already supporting it for displaying HDR/WCG images, ffmpeg is using it for its API, and it has always been the way the H.264, HEVC and AV1 codecs specify the color space, so it's also how the HEIF and AVIF image formats are doing it, as well as JPEG XL and XS according to their spec.

Now here's the thing - I'm going to implement it anyway for our use case (in fact, I got the PNG part running already sans tests); the question is if you would be interested in including it into mainline. Just reading and writing the data is pretty trivial, and from what I've read about your ongoing work regarding ICC color space conversion it should be possible to do this with CICP profiles also without too much additional effort. We don't really need it for ourselves (it's all in the GPU) but I'd be willing to help out.

What I'm also probably going to do (but I guess this is rather optional for mainline) is a decoder flag for synthesizing a CICP profile from other available information if an image file doesn't support or include CICP data. A lot of formats either have their color space specced out, or include the locations of the primaries etc. in the header, or there are a couple of standard ICC profiles for commonly used HDR color spaces. That flag would make the decoders take a best guess, and if that should fail, all the enums in the standard include an "unspecified" value that pretty much forwards the problem to the client. :)

So... any feedback on this? Good idea? Or too obscure for now?

Thanks,
Tammo
Ventuz

[JimBobSquarePants]

Hi @VentuzTammoHinrichs ,

Thanks for your kind offer!

That would be fantastic if you could supply support to add this feature. We're adding animated support to png for v3.1 so have our eyes firmly on the upcoming png specification.

Reading, writing, and conversion would be amazing if you could provide that. I don't know if synthesis would be something we would actively use but I could be wrong! I can help guide you through the PR to make it as easy as possible to contribute.

On the note of full ICC profiles, if you or anyone you know has the expertise and time to help me figure out how to complete the conversion methods, I would really appreciate that. I'm completely stuck for now.

[VentuzTammoHinrichs]

Hi,

Happy to help. I've spent a bit more time on my implementation, and generally checking out the source, and here's what I think:

• As for reading and writing, I'll just send you a PR with my implementation so far, and wait for your feedback.
• Synthesis might be a bit out of scope, true, so I'll implement that client side. GetFormatMetadata() and a rather big switch block should do, and in the end it should be possible to refactor that into a helper inside ImageSharp in case you're interested.
• As for ICC profile based conversion, sorry, I'm at similar place as you and pretty much gave up on it (unless some of our clients really, REALLY require it) :)

Now, conversion. This actually is a topic where I don't feel qualified enough (yet) to make the required architectural decisions for you so here's a rough outline of what this topic might entail:

It makes the most sense to treat those CICP values in two separate blocks, primaries/transfer characteristics on one side, matrix/range on the other. Conversions are usually between RGB spaces (so, primaries/xfer), the matrix only comes into play if a format uses YCbCr or similar, such as JPEG, and can be treated completely independently.

So the good news is that on the RGB side, you already have the RgbWorkingSpace which represents primaries and transfer characteristics. Basically what needs to be done is:

• a function for creating a RgbWorkingSpace from a CICP primaries/xfer tuple
• store the CICP values as readonly metadata on the RgbWorkingSpace so we can extract and write them
• add all the primary locations and transfer functions missing so far (mostly PQ and HLG formats)
• for all of that, add an unbounded linear color RGB space with a reference white luminosity, we're going HDR and display-referred now
• finally make a function to convert from one RgbWorkingSpace to another one, and best one that applies optimization along the way (ie. skip transfer functions and/or primary conversions if they cancel each other out)

That should at least work. If you wanted to go even further, conversions should work automatically, but this means that every image (and if we're really pedantic, even every single color that's being passed around) should come with an RgbWorkingSpace attached, and every operation needs to take all the incoming color spaces as well as the outgoing one into account, and insert conversions accordingly. Just from experience: This is a deeper rabbit hole that you might initially think.

Anyway. On to the matrix. As said, this can be treated separately in the en- and decoders. Now the good part is that this isn't really necessary for the currently supported list of formats (PNG as the only supported one only does RGB anyway), but in case stuff like HEIF or the new JPEGs is upcoming, all you need to do is make YCbCrAndRgbConverter more general so it supports any matrix (not only the hardcoded Rec.601 one). Some of the spaces the matrix transforms to aren't strictly called "YCbCr" though, so you might want to introduce more color space classes instead, such as YCoCg or ICtCp.

There are a couple notable exceptions to the strict primaries => nonlinearity => matrix separation, mainly ICtCp (that uses a variant of LMS instead of RGB on the "RGB side" but signals Rec.2020 primaries, and also the matrix coeffs are dependent on the chosen transfer function), and the "constant luma" (YcCbcCrc) matrices that require linear (as opposed to gamma-space) luma and thus break the order of operations a bit. These are going to require extra care of sorts.

Ok, that was quite the wall of text for "I won't just do it", sorry :) . Thing is, our own use case would benefit from conversion inside ImageSharp, so I'm pretty ready to start coding here, but I'd like to get the scope and those details sorted out first before doing so and then having to rewrite my PR 10 times until it really fits.

Thanks again,
Tammo

[JimBobSquarePants]

Ooft!

That is a wall of text. Thanks for providing so much detail!!

I've had a look at your current PR and I think that's good to go (apart from the casing issue). I can add that to the upcoming v3.1 release.

Regarding conversion...

I plan on making significant changes #2531 to the current color space converter that will allow adding new color spaces significantly easier.

We can incorporate additional API changes there to accommodate your requirements so please comment if you can.

The only issue is that these changes cannot be made until I start working on v4 as I need some language features (C# 12/ .NET 8) to accommodate the API. I plan to start work on V4 as soon as v3.1 is shipped.

[Miloni2010]

@VentuzTammoHinrichs I am working on a similar thing, where I am trying to parse cicp information of a PNG. Currently, libpng does not support cicp, but we can locally make changes; that is how I plan to do this task. Can you send me your PR for a reference regarding the same or help out as to how you did your task?

[VentuzTammoHinrichs]

If you need a reference you can now just look into ImageSharp's code, my PR has made it in already (just find-in-files for CICP or something). :)

But if you only want to parse and write the info, it's actually pretty simple. Check the two links in my original question; the cICP chunk consists of four bytes that index into the tables in that ITU recommendation (same as the color chunk in MOV or generally BMFF). Easy to read and write.

... not that easy to actually process though - if you need to convert the actual pixels you'll have to implement all the math (or the relevant parts) contained in ITU H.273. This sadly wasn't part of my PR - I'm doing all this in the GPU and didn't need ImageSharp to do color space conversions.

[JimBobSquarePants]

Just to add. The new color space converter has been merged into the main branch. This should allow scenarios where custom input and output color spaces can be generated via the CICP profile for conversion.