Optimize chunks splitting based on number of corresponding photons

[dachengx]

If we still do not allow re-chunking of simulation results,

fuse/fuse/plugin.py

Line 13 in ccbc24b

rechunk_on_save = False

a more optimized way is to set the number of tolerable quanta, especially electrons. Interaction with 1 electron and 1E6 electron takes the different sizes of chunks in higher data_type, e.g. propagated_s2_photons.

We have two ways to solve this:

1. For ChunkCsvInput, deduce roughly the size of the tolerable quanta; for ChunkInput, deduce roughly the size of the tolerable energy deposition in higher data_type.

   fuse/fuse/plugins/detector_physics/csv_input.py

   Line 177 in ccbc24b

   chunk_idx = dynamic_chunking(

2. Allow re-chunking in higher data_type.

[dachengx]

Or we should build a new base class like OverlapWindowPlugin, not based on time(endtime) but cluster_id.

[HenningSE]

Hi @dachengx I like the idea of revisiting the chunking settings for fuse. Initially rechunk_on_save was set to False in order to prevent memory issues in the later plugins (S2 photon propagation and raw records simulation). Now that these plugins will create smaller chunks on the fly, we might actually be able to allow rechunk_on_save for all non-Down chunking plugins.

[dachengx]

Hey @HenningSE . It will be a problem if the chunking makes photons from different cluster_id end up in different chunks. So we have to validate the potential change. That is not trivial. I will let you know once I have the PR ready.