Getting average precision class values from detectron2 model evaluation

[25benjaminli]

🚀 Feature

A clear and concise description of the feature proposal.

When using the inference_on_dataset function from detectron2.evaluation, it only provides the overall mAP50 value, and does not provide class AP50 values. However, it does provide individual AP values. It would also be helpful if a similar feature could be added for precision and recall. If there is currently a method to do this, please let me know.

I spent a long time trying to create my own AP, Precision, and Recall evaluators which turned out to be very slow, difficult to use, and may not be completely precise. I hope others will not have to go through the trouble of making it from scratch.

If it were to be implemented, the precision, recall, and AP values with their corresponding classes, would be included as keys in the ordered dictionary generated from inference_on_dataset.

I am currently testing something with fastrcnn, but I do not know if the type of model I am using makes a difference for the inference_on_dataset function. To be clear, I am using the COCO format.

[olivia632]

I wanted to contribute on this

[miyumiyumiyuki]

Please let me know if there is a script to calculate the individual AP50 values for AP50

[carandraug]

The values returned by inference_on_dataset are whatever evaluator.evaluate() return, so his is very specific to what evaluator you're using. If you're using COCOEvaluator, you can patch it to return per class "AP50" values this way:

diff --git a/detectron2/evaluation/coco_evaluation.py b/detectron2/evaluation/coco_evaluation.py
index fe8142c..5b6db0a 100644
--- a/detectron2/evaluation/coco_evaluation.py
+++ b/detectron2/evaluation/coco_evaluation.py
@@ -362,12 +362,13 @@ class COCOEvaluator(DatasetEvaluator):
         precisions = coco_eval.eval["precision"]
         # precision has dims (iou, recall, cls, area range, max dets)
         assert len(class_names) == precisions.shape[2]
+        assert coco_eval.params.iouThrs[0] == 0.5
 
         results_per_category = []
         for idx, name in enumerate(class_names):
             # area range index 0: all area ranges
             # max dets index -1: typically 100 per image
-            precision = precisions[:, :, idx, 0, -1]
+            precision = precisions[0, :, idx, 0, -1]
             precision = precision[precision > -1]
             ap = np.mean(precision) if precision.size else float("nan")
             results_per_category.append(("{}".format(name), float(ap * 100)))
@@ -380,12 +381,12 @@ class COCOEvaluator(DatasetEvaluator):
             results_2d,
             tablefmt="pipe",
             floatfmt=".3f",
-            headers=["category", "AP"] * (N_COLS // 2),
+            headers=["category", "AP50"] * (N_COLS // 2),
             numalign="left",
         )
-        self._logger.info("Per-category {} AP: \n".format(iou_type) + table)
+        self._logger.info("Per-category {} AP50: \n".format(iou_type) + table)
 
-        results.update({"AP-" + name: ap for name, ap in results_per_category})
+        results.update({"AP50-" + name: ap for name, ap in results_per_category})
         return results

[miyumiyumiyuki]

I was able to get data for AP50. I'm really thankful to you.

[bohui-lv]

Hello, thanks for your clear code, but for me, it seems didn't work, I think It may be like this because I installed the compiled detectron2 directly, If I install the compiled detectron2 directly, do I have to recompile after making changes?

The values returned by inference_on_dataset are whatever evaluator.evaluate() return, so his is very specific to what evaluator you're using. If you're using COCOEvaluator, you can patch it to return per class "AP50" values this way:

diff --git a/detectron2/evaluation/coco_evaluation.py b/detectron2/evaluation/coco_evaluation.py
index fe8142c..5b6db0a 100644
--- a/detectron2/evaluation/coco_evaluation.py
+++ b/detectron2/evaluation/coco_evaluation.py
@@ -362,12 +362,13 @@ class COCOEvaluator(DatasetEvaluator):
         precisions = coco_eval.eval["precision"]
         # precision has dims (iou, recall, cls, area range, max dets)
         assert len(class_names) == precisions.shape[2]
+        assert coco_eval.params.iouThrs[0] == 0.5
 
         results_per_category = []
         for idx, name in enumerate(class_names):
             # area range index 0: all area ranges
             # max dets index -1: typically 100 per image
-            precision = precisions[:, :, idx, 0, -1]
+            precision = precisions[0, :, idx, 0, -1]
             precision = precision[precision > -1]
             ap = np.mean(precision) if precision.size else float("nan")
             results_per_category.append(("{}".format(name), float(ap * 100)))
@@ -380,12 +381,12 @@ class COCOEvaluator(DatasetEvaluator):
             results_2d,
             tablefmt="pipe",
             floatfmt=".3f",
-            headers=["category", "AP"] * (N_COLS // 2),
+            headers=["category", "AP50"] * (N_COLS // 2),
             numalign="left",
         )
-        self._logger.info("Per-category {} AP: \n".format(iou_type) + table)
+        self._logger.info("Per-category {} AP50: \n".format(iou_type) + table)
 
-        results.update({"AP-" + name: ap for name, ap in results_per_category})
+        results.update({"AP50-" + name: ap for name, ap in results_per_category})
         return results

[carandraug]

Hello, thanks for your clear code, but for me, it seems didn't work,

How does it seems didn't work?

[...] If I install the compiled detectron2 directly, do I have to recompile after making changes?

No. But add a print somewhere to make sure that you made changes to the files being executed.

[bohui-lv]

How does it seems didn't work?
I add the AP50, but the output didn't change.

No. But add a print somewhere to make sure that you made changes to the files being executed.
Yes, I tried rebuilding detectron2 after that, and then it worked.
Thanks again

[cneupane]

The values returned by inference_on_dataset are whatever evaluator.evaluate() return, so his is very specific to what evaluator you're using. If you're using COCOEvaluator, you can patch it to return per class "AP50" values this way:

diff --git a/detectron2/evaluation/coco_evaluation.py b/detectron2/evaluation/coco_evaluation.py
index fe8142c..5b6db0a 100644
--- a/detectron2/evaluation/coco_evaluation.py
+++ b/detectron2/evaluation/coco_evaluation.py
@@ -362,12 +362,13 @@ class COCOEvaluator(DatasetEvaluator):
         precisions = coco_eval.eval["precision"]
         # precision has dims (iou, recall, cls, area range, max dets)
         assert len(class_names) == precisions.shape[2]
+        assert coco_eval.params.iouThrs[0] == 0.5
 
         results_per_category = []
         for idx, name in enumerate(class_names):
             # area range index 0: all area ranges
             # max dets index -1: typically 100 per image
-            precision = precisions[:, :, idx, 0, -1]
+            precision = precisions[0, :, idx, 0, -1]
             precision = precision[precision > -1]
             ap = np.mean(precision) if precision.size else float("nan")
             results_per_category.append(("{}".format(name), float(ap * 100)))
@@ -380,12 +381,12 @@ class COCOEvaluator(DatasetEvaluator):
             results_2d,
             tablefmt="pipe",
             floatfmt=".3f",
-            headers=["category", "AP"] * (N_COLS // 2),
+            headers=["category", "AP50"] * (N_COLS // 2),
             numalign="left",
         )
-        self._logger.info("Per-category {} AP: \n".format(iou_type) + table)
+        self._logger.info("Per-category {} AP50: \n".format(iou_type) + table)
 
-        results.update({"AP-" + name: ap for name, ap in results_per_category})
+        results.update({"AP50-" + name: ap for name, ap in results_per_category})
         return results

Thank you very much. This worked great.