Inclusion of EndlessCL simulator pre-rendered datasets

[MrtnMndt]

Hi all,

we are currently looking into bringing some rendered datasets of our recent paper "A Procedural World Generation Framework for Systematic Evaluation of Continual Learning" (https://arxiv.org/abs/2106.02585) into Avalanche. (As a first step before considering the more complicated inclusion of the simulator itself).

Before asking my question, the basic summary is that the simulator is a tool to generate data sequences and we have uploaded some pre-rendered example datasets that we have used for the paper's experiments here:
https://doi.org/10.5281/zenodo.4899267

Basically, there is 6 zip files of datasets there:

• a class incremental scenario (spawning objects like trees , humans etc.)
• a scenario with all classes present but with changing weather (i.e. task 1 clear, task 2 rain etc.)
• a scenario where the environment illumination changes over time in discrete steps (i.e. from day to night)

For all of the three above scenarios, there is a classification dataset (which is basically cropped images from the ground truth bboxes) and an original video (stored as consecutive images) with semantic pixel annotations. So 3 + 3.

Our present idea is to:
In benchmarks/datasets create a folder for our dataset, inside this folder crate one python file for the DownloadableData class and one file with _data for the data paths, checksum paths etc.
In benchmarks/classic we can create the “benchmark”, e.g. “c”ourdataset with the nc_benchmark return. Similarly, we can define versions using more generic generators for stuff like weather/lighting.
(Thanks @AndreaCossu for answering my earlier questions here!)

The main question is what do others think is the best way to generate the datasets/benchmarks in terms of separating/uniting them?
In principle they all come from the same data source (the simulator), so we could do 1 dataset class and benchmark with flags such as (lighting=true, class=true etc.), but at the same time, there would be a couple of flags and options that might overload it. I'm not sure this makes sense, given that each of the 6 scenarios is actually unique data. But I also wanted to ask before proceeding to generate 6 separate datasets.

So the options we could imagine are:
Option 1: 1 giant dataset with options for scenarios, classification vs. semseg. (The options would be necessary so that we don't want to download everything because it's a little tedious to download 10+Gb of data when you really only want to do an experiment with 1 specific scenario)
Option 2: 2 datasets, one for classification and one for video segmentation with options for the three scenarios
Option 3: 3 datasets, one for class incremental, one for illumination and one for weather, with options to get classification patches or videos for semseg
Option 4: 6 Unique datasets (as all the data is actually different.

And then of course hybrid versions like: 1 dataset with all flags + 6 benchmark classes under classic.

What do others think?

[vlomonaco]

Hi @MrtnMndt! Thanks for the detailed question :) I think it will help other contributors with similar needs.
I'd personally go for option 3 with 1 "classic" benchmarks with flags to select the "scenario" like we do in CORe50.

This would allow better flexibility if your use avalanche benchmark generators on top of the different datasets (you can use an option to decide if you want to do segmentation or classification) and still provide an unique access point to the simulator data with the "classic" wrapper.

But, it's really up to you... all the solutions you mentioned are still good to me!

[MrtnMndt]

Hey @vlomonaco .

Thanks for the feedback.

We've internally decided and have presently moved ahead with providing flags (like in Core50) in the datasets class, as you have just suggested, and have already started working on it. Like you said, this way, the avalanche benchmarks generators remain untouched and flexible, and we can just use the generic benchmark generator tool.

If we don't face any major difficulties (I don't think we will), it should be ready for a PR within a week or so.