artefacts in FCI RGBs using 3.8 µm

[gerritholl]

Describe the bug

The 3.8 µm channel and RGB that use it have artefacts during the night, such as black pixels. This includes the convection RGB and night microphysics.

To Reproduce

import hdf5plugin
import os
from sattools.io import plotdir
from satpy import Scene
from satpy.utils import debug_on; debug_on()
from glob import glob
fci_files_fdhsi = glob("/media/nas/x23352/MTG/FCI/L1c-cases/202412050600-north-sea/W*FDHSI*BODY*.nc")
fci_files_hrfi = glob("/media/nas/x23352/MTG/FCI/L1c-cases/202412050600-north-sea/W*HRFI*BODY*.nc")
sc = Scene(filenames=fci_files_fdhsi+fci_files_hrfi, reader="fci_l1c_nc")
sc.load(["convection"])
ls = sc.resample("scan")
ls.save_datasets(
        filename=os.fspath(plotdir(create=True) /
        "{start_time:%Y%m%d%H%M}-{platform_name}-{sensor}-{area.area_id}-{name}.tif"),
        fill_value=0)

Expected behavior

I expect an image without black pixels.

Actual results

I get an image with black pixels.

Screenshots

Environment Info:

• OS: openSUSE Leap 15.3
• Satpy Version: v0.53.0-43-g9b817a431

Additional context

I'm aware that the convection composite uses the difference between two solar channels, so it is not suitable to use during the night. But that still shouldn't make pixels black. The problem also occurs with night_microphysics and with an internal DWD RGB where the night part is similar to cloudtop.

[ameraner]

My first suspect is the ir_38 that has NaN brightness temperatures over cold clouds at night (due to extra low/negative radiances mostly generated by the ESL correction). This happens more often in the 1km version, while it gets mostly absorbed in the 2km version, but it can happen there as well... An example of this happening on the 2km version is today 05/12 00:00:

[gerritholl]

Indeed. Here is IR 3.8 for 2024-12-09 04:00 in the North Atlantic:

This is what happens to night microphysics:

Or with our internal RGB, similar to Satpys cloudtop:

Interestingly, this is IR 3.8 with the nir_emissive modifier applied:

[simonrp84]

Can something like this: #2806
Be implemented for FCI? That would "fix" the problem for qualitative use cases...

[gerritholl]

Thanks @simonrp84, that looks interesting. Negative radiances are clipped to the lowest non-negative radiance before calibrating to brightness temperature?

[simonrp84]

Yes. As long as it's an optional feature then it should solve the problem nicely without impacting other uses.

[gerritholl]

Another MCVE:

import hdf5plugin
from satpy import Scene
from pyresample import create_area_def
from glob import glob
fci_files = glob("/media/nas/x23352/MTG/FCI/L1c-cases/202412090100-atlantic/*BODY*O_0008_003[56].nc")
ar = create_area_def("ofz", 4087, description="oceanographer fracture zone", area_extent=[-4230000, 4675000, -3562000, 5232000], resolution=750)
sc = Scene(filenames={"fci_l1c_nc": fci_files})
sc.load(["cloudtop"])
ls = sc.resample(ar)
ls.save_datasets()

Looks like there are both white and transparent pixels.

As shown in irfanview:

[ameraner]

The root cause of both types of pixels is the same: already-low radiance values over cold clouds that get even lower due to the ESL correction and/or noise. Likely, the white pixels are very low radiance values that still have a valid BT (which is going to be extremely cold), and the black pixels are negative radiance pixels that cannot be converted to BTs and hence become NaN/transparent.

[gerritholl]

For pixels to become white, we'd need extremely cold radiances in all three channels, each of 3.8 µm, 10.5 µm, 12.3 µm. However, on closer inspection, I only see this for 3.8.

[gerritholl]

Clipping negative radiances does not solve the problem for composites that use a difference. By setting them to the lowest value, we get very small brightness temperatures:

In [44]: print((sc2["ir_38"][92, 6518:6523]).compute().data)
[217.14456 210.88675 139.34131 139.34131 214.36633]

So far so good; not obvious if we do a crude stretch between (say) 190 and 320. The cold pixels will still look cold and not stand out too much.

But in night_microphysics, we calculate IR 10.5 - IR 3.8. IR 10.5 does not have negative radiances or outlier brightness temperatures, so the very low IR 3.8 brightness temperatures end up as being very positive differences:

In [43]: print((sc2["ir_105"][92, 6518:6523] - sc2["ir_38"][92, 6518:6523]).compute().data)
[-6.282028    0.08493042 71.847885   71.73924    -3.3946533 ]

The enhancement stretches this difference between -4 and 10. This explains the green speckles. Wherever 3.8 is very cold, the delta becomes very high.

I cannot think of any easy solution to this one.

[gerritholl]

Alternatives for composites that take the difference between 3.8 µm and another channel:

0. Accept the status quo.
1. Fill the holes using some type of bucket resampler, that sets negative radiances not to the lowest non-negative radiance but to a value based on non-negative neighbours. Downside: computationally expensive, involved,
2. Use a specialised DifferenceCompositor that can set non-plausible differences to a configurable value, such as 0. The noise might still be visible, but should not be as obvious as now.
3. Abandon the affected composites.
4. Some other solution?

My preference would tend to alternative (2), a PlausibleDifferenceCompositor or so.

It affects:

• geo_color_low_clouds. Not much of a problem, as 1) clouds with extremely low BT at 3.8 µm will be shown as transparent, and 2) if they're so cold, there are very likely high clouds overlaid anyway.
• convection. When 3.8 µm is very cold, the enhancement shows it the same as when it's equal to 10.5 µm, so it's not an obvious outlier for cold clouds.
• night_microphysics. A serious problem, as BT 10.5 µm - BT 3.8 µm will be strongly positive and thus be shown as a 10K difference, whereas surrounding cold pixels have differences that are negative or near 0.
• night_fog has the same problem as night_microphysics.

[djhoese]

I will admit I have not read this entire thread, but did want to point out the issues we've had with night microphysics for ABI-style instruments: we get green spots in ABI's night_microphysics composite as well and they are not fixed by clipping the radiances/reflectances. The issue if I remember correctly is a saturation in one of the channels (C07 I think) that produces "valid" values as far as the input files are concerned, but is obviously incorrect when looking at imagery. We've just had to deal with it in my own projects.

[simonrp84]

It's not saturation but the opposite, the sensor has reached the dark signal level and doesn't give a meaningful response any more. This happens for C07 (or any other MIR channel) because the radiance at low temperatures in this part of the spectrum is much lower than the radiance for longer wavelengths like you'd see in C14 (or any other TIR channel).

Ideally, the L1 data would have a minimum count->BT value for the dark signal that is meaningful, but quite often - as seems to be the case here - it is set artificially low so to enable suitable offset of the digital count for the rest of the dynamic range. That's what gives the unreasonably low radiance/BT. They are technically valid, but the radiance is so low and the noise so high that they're meaningless numbers.

(edit) In terms of a solution, tbh there's not much you can do here. If you want to be really clever you could plot the curves of digital count and radiance against the expected dark noise + NEdT and use that to compute a more sensible minimum radiance/BT for a channel, but that would be very sensor specific and quite often the dark noise isn't published.

(edit again) Actually, looking at the documentation on sensor performance, BTmin for the 3.8 micron channel is 200K. So, for imagery we could clip all BTs below 200K and that would result in nice looking images. This approach isn't as good as the one in my edit above but would be much easier to implement. That said, this should definitely not be the default option within satpy, as there are valid, usable, values for this channel below 200K.

[gerritholl]

Not default for the reader, but a night_microphysics RGB could fill holes in 3.8 µm with 200 K or in (10.5 - 2.8) µm with 0 K. I suspect 200 K would still not solve the visual problem in the night_microphysics RGB, as the difference with 10.5 K would still be unreasonably large.