Stream Accuracy approximation

[MattiaSangermano]

Hi, doing some experiments I noticed that in the logs the average of the experience accuracies is not the same as the stream accuracy. For example:

> Eval on experience 0 (Task 0) from test stream ended.
        Top1_Acc_Exp/eval_phase/test_stream/Task000/Exp000 = 0.9536
> Eval on experience 1 (Task 0) from test stream ended.
        Top1_Acc_Exp/eval_phase/test_stream/Task000/Exp001 = 0.9671
> Eval on experience 2 (Task 0) from test stream ended.
        Top1_Acc_Exp/eval_phase/test_stream/Task000/Exp002 = 0.0000
> Eval on experience 3 (Task 0) from test stream ended.
        Top1_Acc_Exp/eval_phase/test_stream/Task000/Exp003 = 0.0000
> Eval on experience 4 (Task 0) from test stream ended.
        Top1_Acc_Exp/eval_phase/test_stream/Task000/Exp004 = 0.0000
-- >> End of eval phase << --
        Top1_Acc_Stream/eval_phase/test_stream/Task000 = 0.3727

I expected a test_stream = (0.9536 + 0.9671) / 5 = 0.38684 and not 0.3727
Am I wrong in computing the test_stream metric? (This happens also at the end of the whole training, but in this case the difference is higher)

If you want to reproduce it I ran the replay.py example changing the parameters of the Naive strategy as follows:

cl_strategy = Naive(model, torch.optim.Adam(model.parameters(), lr=0.001),
                        CrossEntropyLoss(),
                        train_mb_size=10, train_epochs=1, eval_mb_size=100,
                        device=device,
                        plugins=[ReplayPlugin(mem_size=100)],
                        evaluator=eval_plugin
                        )

Thanks for your help :)

[AntonioCarta]

Which benchmark are you using? Top1_Acc_Stream should average over the number of patterns, not the number of experiences. If the experiences have the same number of samples, your computations are exact.

[MattiaSangermano]

Instead, it's correct to compute the stream forgetting as the average of the forgetting of each experience, right?

[AndreaCossu]

In my opinion, yes. To compute the stream forgetting as the average of forgetting for each pattern, the forgetting should be well-defined at the level of the pattern. I don't think this is the case because the forgetting is the difference between the training performance and the evaluation performance. To get a reliable estimate of the evaluation performance you need to average over multiple patterns, as done for the training performance. Otherwise the comparison might be unfair.
However, this is only my point of view, I don't know of any resource discussing this point.

[MattiaSangermano]

Thank you very much, now it's clearer to me

[AntonioCarta]

Oh yes, you are right! I'm using the MNIST dataset

Then it should be equal, Split MNIST has 2'000 images per experience.

should average over the number of mini-batches

Here I meant patterns, not mini-batches

[MattiaSangermano]

I think they are not equal. In the example replay.pythey are 0: 1874, 1: 2006 2: 1967 3: 2163 4: 1990. I saw this by looking at len(scenario.test_stream[i].dataset)

[AndreaCossu]

Hi @MattiaSangermano , thanks for reporting! I will look into this and let you know.