Online CL scenario using memory buffer only

[mhwasil]

Hi,
I am trying to implement an online CL scenario in avalanche with a replay strategy.

• The model sees the incoming data stream for each experience in a single pass or epoch
• The storage policy then selects the some samples to be kept in the memory buffer
• Then I want to train the model again with the samples in the buffer with multiple pass or epoch

This approach was also implemented in rainbow-memory here

In avalanche, there is an example of online replay here, but it seems that the implementation is a bit different because there is no way to re-train the model with multiple passes or epochs using the buffer only (please CMIIW).

is there an implementation of this particular setting in avalanche or is there a way where I can implement this in avalanche?

your help would be very much appreciated or a point to the existing discussion would also be helpful!

thanks

[niniack]

From what I see, the ClovaAI RM implementation seems to do this in the main loop. They train on the experience, update the dataset to just the buffer, and call train again.

I'm not familiar with a canonical Avalanche way, but I think you could recreate that logic pretty easily in your main training by doing a nested for loop. Something like

for i, exp in enumerate(benchmark.train_stream):
        # split experience into an online stream
        ocl_stream = split_online_stream([exp], experience_size=32)

        # Break down the ocl_stream even further
        for online_exp in ocl_stream:
        	cl_strategy.train(online_exp)

        	# add some logic to build 
        	# an exp from the buffer on the fly

        	buffer_exp = ...
        	buffer_train_epochs = ...

        	for epoch in buffer_train_epochs:
        		cl_strategy.train(buffer_exp)

Disclaimer: I haven't really worked with the online setting, so this is maybe not as thought out of an answer someone with more experience might be able to give!

I can also imagine ways you could achieve the same thing with a strategy plugin, but I personally think having this logic in the main training loop reads the clearest.

[AntonioCarta]

@niniack is correct here. You can check the ocl-survey repository and the related paper for implementations of OCL methods. You can adapt that to your needs by changing the sampling from the buffer if you need something different.

[mhwasil]

thanks a lot for the quick response.
After extensively digging into the avalanche code base, I found a way to re-train (or fine-tune) the model with the buffer.

I did not re-split the experience as suggested by @niniack or like this online replay. Instead, I want to pass the current stream to the model using one epoch and then finetune the model again with the buffer.

If I call cl_strategy.train again, it will trigger before_training_exp, after_training_exp, etc, which I do not want.

I only want to call _before_training_epoch, training_epoch and _after_training_epoch, since I am only interested in fine-tuning the model using the buffer with multiple epochs.

Instead, I created a new template inherited from SupervisedTemplate, where I finetune the model in the after_training_exp function.

class CustomOnlineReplay(SupervisedTemplate):

    def __init__(
        self,
        model: Module,
        optimizer: Optimizer,
        criterion=CrossEntropyLoss(),
        train_mb_size: int = 1,
        train_epochs: int = 1,
        eval_mb_size: Optional[int] = None,
        device: Union[str, torch.device] = "cpu",
        plugins: Optional[List[SupervisedPlugin]] = None,
        evaluator: Union[
            EvaluationPlugin, Callable[[], EvaluationPlugin]
        ] = default_evaluator,
        eval_every=-1,
        mem_size=500,
        train_buffer_epochs=10,
        train_buffer_mb_size=32,
        train_on_buffer=True,
        **base_kwargs
    ):
    
        super().__init__(
            model,
            optimizer,
            criterion,
            train_mb_size=train_mb_size,
            train_epochs=train_epochs,
            eval_mb_size=eval_mb_size,
            device=device,
            plugins=plugins,
            evaluator=evaluator,
            eval_every=eval_every,
            **base_kwargs
        )
        
        # set storage policy
        self.storage_policy = ""
        
    def training_epoch(self, **kwargs):
        for self.mbatch in self.dataloader:
            if self._stop_training:
                break

            self._unpack_minibatch()
            self._before_training_iteration(**kwargs)

            self.optimizer.zero_grad()
            self.loss = self._make_empty_loss()

            # Forward
            self._before_forward(**kwargs)
            self.mb_output = self.forward()

            self._after_forward(**kwargs)

            # Loss & Backward
            self.loss = self.criterion()
            
            self._before_backward(**kwargs)
            self.backward()
            self._after_backward(**kwargs)

            # Optimization step
            self._before_update(**kwargs)
            self.optimizer_step()
            self._after_update(**kwargs)

            self._after_training_iteration(**kwargs)
    
    def _train_exp_on_buffer(self, **kwargs):
        for _ in range(self.train_buffer_epochs):
            self._before_training_epoch(**kwargs)

            if self._stop_training:  # Early stopping
                self._stop_training = False
                break

            self.training_epoch(**kwargs)
            self._after_training_epoch(**kwargs)
    
    def _after_training_exp(self, **kwargs):
        # update buffer
        self.storage_policy.update(self, **kwargs)
       
        # update dataloader
        self.dataloader = ReplayDataLoader(
                data=None,
                memory=self.storage_policy.buffer,
                oversample_small_tasks=True,
                num_workers=4,
                batch_size=self.train_buffer_mb_size,
                batch_size_mem=self.train_buffer_mb_size,
                task_balanced_dataloader=False,
                shuffle=True)
        
        # fine-tune on buffer
        self._train_exp_on_buffer()

@AntonioCarta thanks a lot for sharing the ocl-survey link.

So far avalanche provides very nice flexibility for designing CL methods, many thanks to all maintainers and contributors.