Dividing data into tasks from dataset of type torch.utils.data.dataset.Subset based on label

[akashtadwai]

Hi. I'm currently thinking of testing CL methods in Network Intrusion Domain. I have tested it on CICIDS17 Dataset which contains 15 labels (1 Normal and 14 attack). I've implemented some of CL algorithms and their accuracies can be found here. To compare across all metrics such as memory and others. I wanted to use Avalanche API. I have seen the dataset generators from avalanche.benchmarks.generators, I have train and validation datasets of type as torch.utils.data.dataset.Subset , Now I wanted them to divide into 5 tasks, each task containing 3 labels using nc_scenario from avalanche.benchmarks.generators.scneario_generators when I do this, I'm facing an error saying that,

ValueError: Unsupported dataset: must have a valid targets fieldor has to be a Tensor Dataset with at least 2Tensors

This Error is from 1208 in avalanche_dataset.py. When I dug a little deeper what I understood was there is no Subset.tensors . What should I do to divide my data into tasks?
And if anyone has used Avalanche on a non-vision type dataset can you please give references to them.

[lrzpellegrini]

Hi Akash, that error basically means that the datasets you provide to nc_scenario must have a targets field. Alas, the Subset class found in PyTorch doesn't support it.

The targets field should be a list of int describing the class label of each pattern, which means that the size of that list must be equal to the number of patterns in the dataset.

You probably got your Subset from an original, bigger, Dataset: does the original dataset have a targets field? If it already has a targets field, then you can create the subset using AvalancheSubset instead of the Subset class from PyTorch:

subset = AvalancheSubset(original_dataset, indices=[...])

Let me know if this works for you. Also, if the original dataset doesn't have a targets field, we can consider adapting and integrating it in Avalanche (should be easy)!

[vlomonaco]

I think this question raises an important point about the need for relaxing the "targets" existence assumption. This is somehow related to #372 but if you want @lrzpellegrini I can create another issue to track this. What do you think?

[akashtadwai]

does the original dataset have a targets field?

@lrzpellegrini I am doing a general classification task, Yes, it has the targets field and that is an integer too!!. For splitting the whole dataset into train and validation, we just sampled the original torch.utils.data.dataset into 2 subsets by using torch.utils.data.Subset. The high-level implementation details can be found here. I kept a print statement in line 51 of the forementioned file which checks the datatypes of the Subset. I got a valid datatype of integer for the target. I am still confused why Avalanche gave that error.
The error and the print statement log is as follows,

Type check here: Type of train_dataset: <class 'torch.utils.data.dataset.Subset'>", 
Type of inputs: <class 'torch.Tensor'>, Type or Targets: <class 'int'> 
 One sample of TrainingData: 
 (tensor([8.1622e-07, 1.0000e+00, 4.5346e-08, 4.5346e-08, 1.8138e-06, 3.4009e-06,
        9.8249e-07, 8.4645e-07, 9.1447e-07, 1.1131e-07, 1.7080e-06, 1.7080e-06,
        1.7080e-06, 1.5115e-08, 9.3480e-08, 1.6029e-08, 3.3333e-01, 5.7545e-01,
        9.9780e-01, 9.0331e-04, 9.9870e-01, 9.9870e-01, 1.5115e-08, 9.9870e-01,
        9.9870e-01, 9.9910e-01, 9.9910e-01, 1.5115e-08, 9.9910e-01, 9.9910e-01,
        1.5115e-08, 1.5115e-08, 8.0110e-07, 6.1972e-07, 1.5572e-08, 1.5572e-08,
        8.4645e-07, 1.7080e-06, 1.2183e-06, 4.6561e-07, 1.3442e-05, 1.5115e-08,
        1.5115e-08, 1.5115e-08, 1.5115e-08, 1.5115e-08, 1.5115e-08, 1.5115e-08,
        1.5115e-08, 3.0230e-08, 1.5191e-06, 9.1447e-07, 1.7080e-06, 8.0110e-07,
        4.5346e-08, 1.8138e-06, 4.5346e-08, 3.4009e-06, 0.0000e+00, 0.0000e+00,
        3.0230e-08, 3.1742e-07, 9.0331e-04, 1.5115e-08, 9.0331e-04, 9.0331e-04,
        9.9780e-01, 1.5115e-08, 9.9780e-01, 9.9780e-01]), 1)
Traceback (most recent call last):
  File "backup.py", line 56, in <module>
    task_labels=False
  File "/home/suresh/Desktop/AnamolyDetection_MiniProject/Continual-Learning-Benchmark-master/avalanche/avalanche/benchmarks/generators/scenario_generators.py", line 174, in nc_scenario
    train_dataset = as_avalanche_dataset(train_dataset).train()
  File "/home/suresh/Desktop/AnamolyDetection_MiniProject/Continual-Learning-Benchmark-master/avalanche/avalanche/benchmarks/utils/avalanche_dataset.py", line 1176, in as_avalanche_dataset
    return AvalancheDataset(dataset)
  File "/home/suresh/Desktop/AnamolyDetection_MiniProject/Continual-Learning-Benchmark-master/avalanche/avalanche/benchmarks/utils/avalanche_dataset.py", line 143, in __init__
    self.targets: Sequence[int] = self._initialize_targets_sequence(dataset)
  File "/home/suresh/Desktop/AnamolyDetection_MiniProject/Continual-Learning-Benchmark-master/avalanche/avalanche/benchmarks/utils/avalanche_dataset.py", line 628, in _initialize_targets_sequence
    return _make_target_from_supported_dataset(dataset)
  File "/home/suresh/Desktop/AnamolyDetection_MiniProject/Continual-Learning-Benchmark-master/avalanche/avalanche/benchmarks/utils/avalanche_dataset.py", line 1208, in _make_target_from_supported_dataset
    raise ValueError('Unsupported dataset: must have a valid targets field'
ValueError: Unsupported dataset: must have a valid targets fieldor has to be a Tensor Dataset with at least 2Tensors

[lrzpellegrini]

I just gave a look at the code. As I expected the main source of problems is the fact that an instance of Subset is used when calling nc_scenario. I think I can fix this by explicitly handling the case in which Subset is used.

However, there is another problem: even if I add support to Subset, the original dataset must still have a targets field.

If I'm correct, this should be simple as changing the constructor of class IDSDataset as follows:

class IDSDataset(Dataset):
 
    def __init__(self,transform=None):
        self.path_ds = '../data/dataset.npy'
        self.path_lab = '../data/labels.npy'
        self.ds = np.load(self.path_ds)
 
        self.label = np.load(self.path_lab)
        self.targets = self.label[..., 0]  # <-- This line
        self.n_samples= len(self.ds)
        self.transform = transform
# rest of the code

also, changing train_val_dataset may be required before an explicit support for Subset is implemented. The following should work:

def train_val_dataset(dataset, val_split=0.33):
    train_idx, val_idx = train_test_split(list(range(len(dataset))), test_size=val_split,random_state=42)
    return AvalancheSubset(dataset, indices=train_idx), AvalancheSubset(dataset, indices=val_idx)

Let me know if it works!

[akashtadwai]

Let me know if it works!

@lrzpellegrini Thanks for the fix! It worked. Earlier when you meant

does the original dataset have a targets field?

I have misunderstood by thinking that you were describing it as something other than a classification problem 😅 . I realized now that is the variable of targets in the __init__.

Also, if the original dataset doesn't have a targets field, we can consider adopting and integrating it in Avalanche (should be easy)!

Would be helpful!! As default PyTorch implementations don't require targets field in __init__ itself.

And Just another small question @vlomonaco @lrzpellegrini, is there any way in nc_scenario or ni_scenario while training we could provide custom task order to train so that we could determine how the permutations of task affect the performance of the algorithm.

[lrzpellegrini]

I know that this has been solved, but in the meantime I added the support for Subset in Avalanche ;)

Of course the original dataset must have a valid targets field, but now you can create benchmarks by passing instances of Subset and even Subsets nested in another Subset!