Online Continual Learning

[HamedHemati]

Hi everyone!

For online continual models, the buffer may need to be updated with every batch of incoming data. By default, in the standard CL benchmarks, all batches from the same task are considered as one experience, and therefore the buffer will be updated only once after the task ends. As suggested by @AntonioCarta, I converted each batch of data to an individual experience. It can be done by creating a sub-set of the experience's dataset and replacing it with the created subset as below:

for task_id in range(len( benchmark.train_stream)):
    exp = benchmark.train_stream[task_id]
  
    # create a list of indices and shuffle them
    indices = list(range(len(exp.dataset)))
    random.shuffle(indices)

    # number of "sub-experiences"
    num_sub_exps = len(indices) // train_mb_size
  
    # make of copy of the original dataset
    main_ds = copy.copy(exp.dataset)
  
    # for each sub-experience, train create a subset dataset and replace the experience's dataset
    for subexp_id in range(num_sub_exps + 1):
        subexp_indices = indices[subexp_id*train_mb_size: (subexp_id+1)*train_mb_size]
        subexp_ds = AvalancheSubset(main_ds, indices=subexp_indices)
        exp.dataset = subexp_ds
        strategy.train(exp)
    
   # eval
    strategy.eval(benchmark.test_stream[:task_id+1])

Now, the strategy's buffer will be updated as expected in an online setup. The only possible issue is that by creating "sub-experiences" out of each batch, the optimizer is re-initialized. This can be problematic for cases where the optimizer uses momentum or any kind of information from previous updates in the same experiences.

Does anyone know a better or more general way to do it?

Thanks :)

[AntonioCarta]

A couple of things that we might want to help with OCL scenarios:

• a tutorial/example for OCL where we show all the gotchas. For example, about the dataloader's performance.
• Alternative: a training loop different from the BaseStrategy where we fix better parameters for OCL.
• a parameter for replay plugins to disable transformation and store raw tensors (large speedup for OCL).

[HamedHemati]

I think the alternative that you proposed can be very useful; having something like BaseOnlineStrategy inheriting from BaseStrategy. Steps like splitting the experience into individual sub-experiences can also be handled inside its train loop. For the optimizer, parameters to enable/disable optimizer re-initialization after each experience may be helpful and make things easier.