Try out performance optimizations for TiTiler for CloudFerro S3 Sentinel-2 L2A JP2

[j08lue]

Tiling the JP2 Sentinel-2 L2A files from CloudFerro S3 works [EDIT with limitations]. Perhaps performance can be improved with some GDAL tuning, as indicated here:

• Add more sample data to eoAPI #75 (comment)

Maybe this can avoid some of the OpenJPEG crashes we have seen?

• Debug OpenJPEG / JPEG2000 concurrent reading issues  #105

Acceptance criteria

• Tested whether there are performance gains with the mentioned GDAL tuning

[MathewNWSH]

check:
GDAL_DISABLE_READDIR_ON_OPEN=YES

and try to set chunk size to the one data is stored in.

CPL_VSIL_CURL_CHUNK_SIZE=98304
GDAL_INGESTED_BYTES_AT_OPEN=16384

the biggest optimization comes from using commercial JPEG2000 drivers like kakadu.

[lnicola]

GDAL_DISABLE_READDIR_ON_OPEN=YES

You want EMPTY_DIR. If you set it to TRUE, GDAL will stat each file individually instead of enumerating the files in the parent directory in one go. EMPTY_DIR makes it tell the driver there are no other files.

[j08lue]

Looks like there was a performance degradation in OpenJPEG, too, that may soon get addressed:

• Debug OpenJPEG / JPEG2000 concurrent reading issues  #105 (comment)