Recurrent Yolo v.8

[edoduc]

Hi everyone!

I'm currently working on mobile object detection. I benchmarked Yolo v.8 nano for this, and get around 40% of mAP. Now, I try to enhance my results by adding recurrence units in the framework (my data are ordered frames from videos). I am inspired from the paper A Recurrent YOLOv8-based framework for Event-Based Object Detection, but adding ConvGRU instead of ConvLSTM after the backbone C2f blocks.

In order to handle sequential frames in input, I modified the dataset and dataloader logics to accept (Batch, Seq, ch, w, h) shaped inputs with 'Seq' my sequence length (number of ordered frames in input).
Then, to process these multiple frames, I created a Deconcatenation block which outputs Seq*tensors([Batch, ch, w, h]), and a Framextract block which outputs the right frame from the deconcatenation output list. The idea is to pass each frame independently through the native Yolo v.8 backbone (with ConvGRU inserted after C2f). Sub-backbones are connected via the ConvGRU blocks and their hidden states. Only the last sub-backbone (for the frame in which I want to detect the object) ends with the SPPF block before continuing to the neck and the prediction head.

I adapted the yaml file consequently.

# YOLOv8.0n backbone
backbone:
  # [from, repeats, module, args]
  - [-1, 1, Deconcat, []] # 0  #Deconcatenate sequential frames

  # frame t-2
  - [0, 1, Framextract, [2]] # 1  # Output frame t-2
  - [-1, 1, Conv, [64, 3, 2]]
  - [-1, 1, Conv, [128, 3, 2]] 
  - [-1, 3, C2f, [128, True]]
  - [[-1], 1, ConvGRU, [128, [128], [3], 1]] # 5
  - [-1, 1, Conv, [256, 3, 2]]
  - [-1, 6, C2f, [256, True]]
  - [[-1], 1, ConvGRU, [256, [256], [3], 1]] # 8
  - [-1, 1, Conv, [512, 3, 2]]
  - [-1, 6, C2f, [512, True]]
  - [[-1], 1, ConvGRU, [512, [512], [3], 1]] # 11
  - [-1, 1, Conv, [1024, 3, 2]]
  - [-1, 3, C2f, [1024, True]]
  - [[-1], 1, ConvGRU, [1024, [1024], [3], 1]] # 14

  # frame t-1
  - [0, 1, Framextract, [1]] # 15  # Output frame t-1
  - [-1, 1, Conv, [64, 3, 2]]
  - [-1, 1, Conv, [128, 3, 2]] 
  - [-1, 3, C2f, [128, True]]
  - [[-1, 5], 1, ConvGRU, [128, [128], [3], 1]] # 19
  - [-1, 1, Conv, [256, 3, 2]]
  - [-1, 6, C2f, [256, True]]
  - [[-1, 8], 1, ConvGRU, [256, [256], [3], 1]] # 22
  - [-1, 1, Conv, [512, 3, 2]]
  - [-1, 6, C2f, [512, True]]
  - [[-1, 11], 1, ConvGRU, [512, [512], [3], 1]] # 25
  - [-1, 1, Conv, [1024, 3, 2]]
  - [-1, 3, C2f, [1024, True]]
  - [[-1, 14], 1, ConvGRU, [1024, [1024], [3], 1]] # 28

  # frame t
  - [0, 1, Framextract, [0]] # 29  # Output frame t
  - [-1, 1, Conv, [64, 3, 2]]
  - [-1, 1, Conv, [128, 3, 2]] 
  - [-1, 3, C2f, [128, True]]
  - [[-1, 19], 1, ConvGRU, [128, [128], [3], 1]] # 33
  - [-1, 1, Conv, [256, 3, 2]]
  - [-1, 6, C2f, [256, True]] # 35
  - [[-1, 22], 1, ConvGRU, [256, [256], [3], 1]] # 36
  - [-1, 1, Conv, [512, 3, 2]]
  - [-1, 6, C2f, [512, True]] # 38
  - [[-1, 25], 1, ConvGRU, [512, [512], [3], 1]] # 39
  - [-1, 1, Conv, [1024, 3, 2]]
  - [-1, 3, C2f, [1024, True]]
  - [[-1, 28], 1, ConvGRU, [1024, [1024], [3], 1]] # 42
  - [-1, 1, SPPF, [1024, 5]] # 43

# YOLOv8.0n head
head:
  - [-1, 1, nn.Upsample, [None, 2, "nearest"]] # 44
  - [[-1, 38], 1, Concat, [1]] # cat backbone P4
  - [-1, 3, C2f, [512]] # 46

  - [-1, 1, nn.Upsample, [None, 2, "nearest"]] # 47
  - [[-1, 35], 1, Concat, [1]] # cat backbone P3
  - [-1, 3, C2f, [256]] # 49 (P3/8-small)

  - [-1, 1, Conv, [256, 3, 2]] # 50
  - [[-1, 46], 1, Concat, [1]] # cat head P4
  - [-1, 3, C2f, [512]] # 52 (P4/16-medium)

  - [-1, 1, Conv, [512, 3, 2]] # 53
  - [[-1, 43], 1, Concat, [1]] # cat head P5
  - [-1, 3, C2f, [1024]] # 55 (P5/32-large)

  - [[49, 52, 55], 1, Detect, [nc]] # Detect(P3, P4, P5)

And the parse_model function in the task.py

elif m is Deconcat:  # Handle Deconcat
        c2 = ch[f] 
 elif m is Framextract:  # Handle Framextract
        c1 = ch[f]
        c2 = ch[f]
elif m is ConvGRU:  # Handle ConvGRU
        c1 = sum(ch[x] for x in f)
        hidden_sizes = args[1]
        c2 = hidden_sizes[-1]  # The output channels match the last hidden state size
        c2 = make_divisible(min(c2, max_channels) * width, 8)
        args = [*args]

But, when building the model and parsing the yaml file, I get an error at the second Framextract level because it seems that it receives the last layer output as input, instead of the deconcatenation output as specified with the argument f=0.
Could anyone help me on this please? I'm not sure if I modified well the parsing function, or if I missed some other necessary modifications somewhere else.

Thanks a lot.

[UltralyticsAssistant]

👋 Hello @edoduc, thank you for sharing your interesting project with us 🚀! Your approach to incorporating recurrence into YOLOv8 for sequential data processing sounds fascinating, and we appreciate the level of detail you’ve provided.

If this is a 🐛 Bug Report, please consider providing a minimum reproducible example (MRE) to help us better understand and debug the issue. This could include a minimal YAML configuration, the modified code snippet, and any specific commands or steps needed to reproduce the error you're encountering.

Here are some recommendations to ensure smooth experimentation:

Upgrade

First, ensure you are using the latest version of the ultralytics package, as updates often include bug fixes and enhancements:

pip install -U ultralytics

Ensure your setup also satisfies all additional requirements in a Python>=3.8 environment with PyTorch>=1.8. This ensures compatibility and avoids issues already resolved in newer releases.

Reference Documentation

You may find these resources valuable as you work through your customizations:

• YOLO Modes for an overview of task configurations
• Model Definitions to better understand how the YAML structure drives model parsing and execution
• Tips for Best Training Results for general guidance on achieving optimal results during training

Verified Environments

If you encounter difficulties running your custom implementation locally, you might consider testing in a verified environment. This ensures compatibility across frameworks and dependencies:

• Notebooks with free GPU:
• Docker Environments: See the Docker Quickstart Guide

Community Support

In addition to GitHub, you can explore and discuss ideas in the wider Ultralytics community:

• Real-time chat on our Discord 🎧
• In-depth discussions on Discourse
• Share findings and learn from others via our Subreddit

Status Check

Ensure to monitor the current CI status of the repository. If the CI badge below is green, this indicates that all Ultralytics CI tests are passing, verifying correct operation of YOLO on macOS, Windows, and Ubuntu:

🛠️ This is an automated response, but an Ultralytics engineer will review and assist you soon! Thank you for your patience and detailed contribution—it’s always inspiring to see innovative approaches like yours.

[glenn-jocher]

@edoduc thank you for sharing the detailed context! The issue appears to stem from an inconsistency in how Framextract is being parsed and the target layer's output is being assigned. Specifically, in the parse_model logic, the handling of the Deconcat and Framextract layers needs to ensure that the right outputs from the Deconcat block are being taken instead of the last layer output.

To address this:

1. Check the index assignment of Framextract layers in your parse_model function. When you call ch[f], ensure it properly corresponds to the Deconcat layer outputs rather than the default chain of previous layers.
2. Verify the sequence logic of index references in f as inputs to Framextract. If Deconcat provides multiple outputs, you may need to explicitly manage these outputs (e.g., via a list) rather than relying on single-layer outputs.

Lastly, check that the Deconcat effectively outputs the required sequential frames and confirm that the sequence index in Framextract maps correctly to individual frames. If the issue persists, modifying parse_model to better handle and trace multi-output scenarios may be necessary.

Feel free to follow up with additional code snippets for further debugging if required.

[ZadeAbhishek]

@edoduc I am also working on the same project for my college dissertation. I am encountering numerous errors and am relatively new to YOLO. Could you please help me with the coding part if possible? Thanks!

[edoduc]

Hello @ZadeAbhishek !
I am sorry, I did not manage to solve all the errors I encountered and I gave up then. Adding a fifth dimension in YOLO input requires a lot of modifications and I had no time for this.
Good luck in your project, I hope you will find the solution.

[silvada95]

Hi, I just made the code for Recurrent YOLOv8 available. You can check it here: https://github.com/silvada95/ReYOLOv8.

It was focused on event data, but if you do modifications on the EventVideoDataset file you may be able to adapt it to sequences of regular frames.

[ZadeAbhishek]

@silvada95 thanks a lot.

[edoduc]

@silvada95 , thanks for your repo and congrats for your paper.
Nice work!