Obtaining experience datasets with correct transforms

[prateeky2806]

Hi, I have been trying to create a specific dataclass using the avalanche benchmarks. I want to use the avalanche benchmarks to get obtain the list of pytorch datasets for all streams with correct split and transforms.

benchmark_original = RotatedMNIST(n_experiences=5)
benchmark = benchmark_with_validation_stream(benchmark_original, validation_size=0.2, input_stream='train')

train_datasets = get_datasets_from_aval_stream(benchmark.train_stream)
valid_datasets = get_datasets_from_aval_stream(benchmark.valid_stream)
test_datasets = get_datasets_from_aval_stream(benchmark.test_stream)

def get_datasets_from_aval_stream(stream):
    datasets = []
    for i in range(len(stream)):
        exp_dataset, exp_labels, exp_ids  = stream[i].dataset[:]
        exp_dataset = exp_dataset.squeeze(1)
        datasets.append(torch.utils.data.TensorDataset(exp_dataset, exp_labels))
    return datasets

Now I have two questions,

1. Given that the validation stream was created from train stream, which transformation is applied to the dataset in valid_datasets. I don't want random cropping, or horizontal flip in the valid_datasets. I want both valid_datasets and test_datasets to have the 'eval' transforms while for train_datasets I want the 'train' transforms.
2. Is there a direct way to obtain a list of pytorch datasets from a stream with using the function "get_datasets_from_aval_stream" which I created.

Thanks for your help!
Prateek

[vlomonaco]

Dear @prateeky2806 thanks for your question! :) For 1. you can take a look at the AvalancheDataset features, epsecially how we handle the transformations. For 2. no, we don't have a method that does that already, but in the basic case you can do just datasets = [exp.dataset for exp in stream], you'll get a list of AvalancheDatasets, one for each experience.

[lrzpellegrini]

Hi @prateeky2806

• By default RotatedMNIST will not contain transformations (apart from the random rotation, which is a frozen transformation not visible through the transform_groups field). You can add additional transformations by passing train_transform and eval_transform to RotatedMNIST.
• I think that the train transformations may be used by default in the valid stream. You can check the current_transform_group field of the datasets to get the name of the current group. However, keep in mind that, when using avalanche strategies, all datasets used in the evaluation phase are switched (a view is created, the dataset itself is not changed) to the eval transformations group, so the current group only matters when not using the strategies from Avalanche.

If you want to make sure that the correct group is used even when not using avalanche code, you can do something like (not tested):

benchmark_original = RotatedMNIST(n_experiences=5, train_transform=..., eval_transform=...)
benchmark = benchmark_with_validation_stream(benchmark_original, validation_size=0.2, input_stream='train')

train_datasets = get_datasets_from_aval_stream(benchmark.train_stream, 'train')
valid_datasets = get_datasets_from_aval_stream(benchmark.valid_stream, 'eval')
test_datasets = get_datasets_from_aval_stream(benchmark.test_stream, 'eval')

def get_datasets_from_aval_stream(stream, use_group):
    datasets = []
    for i in range(len(stream)):
        dataset_with_corrrect_transforms = stream[i].dataset.with_transforms(use_group)
        exp_dataset, exp_labels, exp_ids  = dataset_with_corrrect_transforms[:]
        exp_dataset = exp_dataset.squeeze(1)
        datasets.append(torch.utils.data.TensorDataset(exp_dataset, exp_labels))
    return datasets

Also be aware that, while the dataset[:] syntax is very simple to use to load the entire dataset, it may be very slow with big datasets because no parallelism is used.

[prateeky2806]

Hi @lrzpellegrini, thanks for your response! I checked and by default the validation split uses the train transform and I understand how this is not an issue if it is used with avalanche. I tried setting transform group manually for validation stream by doing valid_stream[i].dataset.with_transforms('eval') and then the current_transform_group shows "eval". But, I have another concern here, let's say I create a SplitCIFAR100 benchmark and then run benchmark.train_stream[0].dataset.transform_groups outputs {'train': (None, None), 'eval': (None, None)}. Hence, I am not sure if setting the transform group using with_transform() is doing any good here as the eval transforms are None. I digged into the code a bit and this is due to the fact that when AvalancheSubset is called inside of NC_Scenario these transforms are not passed hence the datasets objects with the stream have the transform_groups set to None. See the code here for what I am referring to. I might have completely missed other ways which handle this internally but I am still not sure that is the dataset shows the eval group to be None and its set to that then which transform it is using. Another question I have is that is there a way to explicitly print the transform that are used in a dataset as opposed to printing the group?

_default_cifar100_train_transform = transforms.Compose([
    transforms.RandomCrop(32, padding=4),
    transforms.RandomHorizontalFlip(),
    transforms.ToTensor(),
    transforms.Normalize((0.5071, 0.4865, 0.4409),
                         (0.2673, 0.2564, 0.2762))
])

_default_cifar100_eval_transform = transforms.Compose([
    transforms.ToTensor(),
    transforms.Normalize((0.5071, 0.4865, 0.4409),
                         (0.2673, 0.2564, 0.2762))
])
benchmark_original = SplitCIFAR100(n_experiences=10, return_task_id=True, fixed_class_order=np.arange(100),
                                                            train_transform=_default_cifar100_train_transform, 
                                                            eval_transform=_default_cifar100_eval_transform)
benchmark = benchmark_with_validation_stream(benchmark_original, validation_size=0.2, input_stream='train')

train_datasets = get_datasets_from_aval_stream(benchmark.train_stream, 'train')
valid_datasets = get_datasets_from_aval_stream(benchmark.valid_stream, 'eval')
test_datasets = get_datasets_from_aval_stream(benchmark.test_stream, 'eval')

def get_datasets_from_aval_stream(stream, use_group):
    datasets = []
    for i in range(len(stream)):
        dataset_with_corrrect_transforms = stream[i].dataset.with_transforms(use_group)
        exp_dataset, exp_labels, exp_ids  = dataset_with_corrrect_transforms[:]
        exp_dataset = exp_dataset.squeeze(1)
        datasets.append(torch.utils.data.TensorDataset(exp_dataset, exp_labels))
    return datasets

Even if I explicitly pass the transformations like above still benchmark.train_stream[0].dataset.transform_groups outputs {'train': (None, None), 'eval': (None, None)}.

[prateeky2806]

Hi @lrzpellegrini, any thoughts on this?

[lrzpellegrini]

I think there are two things to keep into consideration:

• The datasets created by SplitCIFAR100 (and other benchmark generators) are usually subsets of some original dataset (for instance, the CIFAR100 dataset from torchvision). Avalanche splits this original dataset by creating intermediate AvalancheDataset and AvalancheSubset datasets. The transformations are usually added to these intermediate datasets. In particular, for SplitCIFAR/MNIST/... they are added here. Chaning the transformation group of the final dataset will also switch (by creating a view) the transformation group of any intermediate dataset. There is no official way to check which transformations are currently used in the datasets hierarchy. However, I wrote this in a few minutes and it seems to be working:

from avalanche.benchmarks.utils import AvalancheDataset, \
    AvalancheConcatDataset, AvalancheTensorDataset, Compose, AvalancheSubset

def all_dataset_transformations(avl_dataset):
    got_transforms = None
    if isinstance(avl_dataset, AvalancheSubset):
        got_transforms = all_dataset_transformations(
            avl_dataset._original_dataset)
    elif isinstance(avl_dataset, AvalancheConcatDataset):
        if len(avl_dataset._dataset_list) > 0:
            got_transforms = all_dataset_transformations(
                avl_dataset._dataset_list[0])
    elif isinstance(avl_dataset, AvalancheTensorDataset):
        pass
    elif isinstance(avl_dataset, AvalancheDataset):
        got_transforms = all_dataset_transformations(
            avl_dataset._dataset)

    current_transforms = None
    if hasattr(avl_dataset, 'transform'):
        current_transforms = avl_dataset.transform

    if current_transforms is None:
        return got_transforms

    if got_transforms is None:
        return current_transforms

    return Compose([got_transforms, current_transforms])

It is quite rough, and for AvalancheConcatDataset will only check the transformations of the first concatenated dataset, but it should work.

If you print its result for all_dataset_transformations(dataset_with_corrrect_transforms) you will get:

Compose(
    RandomCrop(size=(32, 32), padding=4)
    RandomHorizontalFlip(p=0.5)
    ToTensor()
    Normalize(mean=(0.5071, 0.4865, 0.4409), std=(0.2673, 0.2564, 0.2762))
)

for the datasets of the training stream, while for the dataset from the test stream you will get:

Compose(
    ToTensor()
    Normalize(mean=(0.5071, 0.4865, 0.4409), std=(0.2673, 0.2564, 0.2762))
)

which means that the with_transforms is working as expected!

• Beware that, when you create a torch.utils.data.TensorDataset, the chain of transformations is lost. This is because tensor datasets will just return data from the provided tensors. The tensors you are using to create the TensorDataset are already properly transformed (because they are obtained from dataset_with_corrrect_transforms), but you will lose the probabilistic aspect of augmentations like RandomCrop or RandomHorizontalFlip.

[prateeky2806]

Thanks @lrzpellegrini, thanks for providing this nice function. Your last comments clears the last set of doubts I had. Thanks you for your time and explaining things to me.