Get Precision and Recall metrics

[jonas9983]

Hi!

I noticed that only the mAP metrics are saved. Even though this already provides good information, I wanted to know if it is possible to get the Precision and Recall metrics. I've seen that the Average Precision and Average Recall are logged after each epoch but I'm not entirely sure if or where they are saved. This could allow me to compute the F1-score and the Precision-Recall curves.

Another thing that could help is getting the predictions from the model so that a confusion matrix can be built.

Thanks!

[dnth]

Hey @jonas9983 I've added extensive logging into tensorboard. This includes Average Precision, Average Recall, F1 and even plots the PR curve for each validation epoch so you can monitor the progress.

The PR is here - #25

This is now available in the main branch. Try it out and let me know ;)

[jonas9983]

Hello! Wow this is awesome and what I've looking for exactly! Thank you very much.

---

Not sure if this is an entirely new issue, but I am getting an error which is related to your last changes.

To get the latest changes, I pulled from the main branch and created a new virtual environment and installed the requirements as mentioned on the README with

pip install git+https://github.com/dnth/DEIM.git@v0.2.1

Then, I ran the command

CUDA_VISIBLE_DEVICES=0 torchrun --master_port=7778 --nproc_per_node=1 scripts/train.py

After three epochs this error appeared:

Traceback (most recent call last): File "/home/joao/.pyenv/versions/3.11.9/lib/python3.11/tkinter/__init__.py", line 4105, in __del__ self.tk.call('image', 'delete', self.name) RuntimeError: main thread is not in main loop

It would continuosly log in the terminal until it ended the process with these errors:

Tcl_AsyncDelete: async handler deleted by the wrong thread?

PyTorch RuntimeError: DataLoader worker (pid(s) 15332) exited unexpectedly.

I looked up on Stack Overflow and found this solution: https://stackoverflow.com/a/29172195, which I added directly on the deimkit package on the .pyenv folder and the issue didn't happen anymore. Can you reproduce this error?

---

Hope this helps in some way!

Thank you!

[dnth]

Does this happen if you dont use torchrun? Eg just running python train.py

[jonas9983]

Yes, the errors also happen when you run with python train.py. It ran for 7 epochs and it crashed

[dnth]

Thanks! I think it could be from the plotting of the pr curve since it's using matplolib. I can't reproduce the error yet, but I will adopt your solution. Thanks for reporting!

[jonas9983]

Yes I agree. No worries! Happy to help!

Also, I think it would be interesting to have the system performance metrics, to monitor the GPU, CPU, RAM and so on. I know I can view this with nvidia-smi and nvtop but it would be more interactive if they were on tensorboard.

There are some references here:

https://stackoverflow.com/questions/67707828/how-to-get-every-seconds-gpu-usage-in-python
https://stackoverflow.com/questions/276052/how-to-get-current-cpu-and-ram-usage-in-python

With this, a new tab on tensorboard can be created (maybe system ?) which displays the system metrics.

I can't implement this right now, but can do it later