Per Class Metrics

[VerwimpEli]

I believe right now metrics like accuracy and loss calculate the average loss across all samples within an experience. But, in some cases it's more interesting to look at the average class accuracy, or even at the accuracy on a per class basis. For instance, in datasets where the classes are very imbalanced across all experiences (e.g. COCO classification).

Implementing a separate Metric that keeps track of a metric per class wouldn't be hard. But maybe it's better to implement it on a lower level, in a similar fashion as how the Mean and Sum utilities work, such that other metrics can be used to get the average of all examples in an experience and the average per class in the experience, based on some parameter in the metric.

In theory, it should actually be already possible to get accuracy results on a per class basis through the confusion matrix metric, but that'd be kind of ugly in my opinion.

I'd be able to implement this, but wanted to discuss a bit first what the best way to do this would be.

[AndreaCossu]

Hi @VerwimpEli , thank you for reaching out! You are right, currently accuracy works at the level of experience and does not partition its value according to the different classes.
As you correctly said, one way is to retrieve the tensor of the confusion matrix to get the per-class values. With the latest PR (still to be merged) one other trick could be to assign as task label to each pattern its class. In this way, accuracy would compute result separated for each task, therefore for each class. However, this would still not be a clean solution since it would not apply to common benchmarks generators like nc_scenario, where you cannot specify task labels per pattern.

The main downside I see is that we risk to produce a large variety of metrics which are basically slight modifications of the existing ones. However, I think accuracy by class is quite used and it would be helpful for the user to have it as a first-level metric.

I think you can proceed with the implementation and open a PR as soon as you are ready so that we can better discuss. To avoid the accuracy.py file to become too large and hard to manage, I'd suggest you to produce a separate file, like class_accuracy.py or something inside metrics module. Also, you can follow the implementation of the accuracies plugin metrics so that the interface remains similar. The user should choose through a parameter which classes to monitor (None to monitor all of them, otherwise a list of classes).

Let me know what you think 😄

[VerwimpEli]

Hi @AndreaCossu, thanks for the reply. I'll open a PR a soon as I'm done, then we can discuss further 👍

[VerwimpEli]

Created PR #679 with an initial implementation.