Early Stop

[Hua-rookie]

I am trying to use the EarlyStoppingPlugin in a ten tasks scenario, however, the 'best_state' and 'best_val' cannot be updated correctly. My code looks like this:
results = [strategy.eval(test_stream)]
for train_task, test_task in zip(train_stream, test_stream):
strategy.train(experiences=train_task, eval_streams=[test_task])
results.append(strategy.eval(test_stream))

After 'results = [strategy.eval(test_stream)]' being executed, the 'best_state' will remain at 0 while the 'best_val' will remain at the value of 'Top1_Acc_Stream/eval_phase/test_stream/Task009' until the end of the program.
As a result, the training phase will stop as soon as the epoch_num is equal to the patience.

So I wanna to know how to make the EarlyStoppingPlugin working correctly?

[Hua-rookie]

I found that there might be a bug in EarlyStoppingPlugin.
I have observed that this bug only occurs when a 'return_task_id' is True, not when it is False, and the reasons are as follows:

This part of code is in early_stopping.py, we can see that only the last metrics would be update to the 'self.best_val'.

This part of code is in evaluation.py, we can see that if we choose to 'return_task_id', the last metrics will remain to be the metric of the last task but not the current task it should be.

I hope my understanding is correct.

[AntonioCarta]

yes, there is a problem in the current early stopping implementation. The problem is that it only receives the name of the metric. In task-aware scenarios, the metric name depends on the task, so if you give a metric name for a specific task (.../Task000) then it's not going to work in the subsequent tasks.