Incorporating YOLOv8 Pose ONNX model

[pbanavara]

This is the PR for incorporating the YOLO v8 pose detection model natively using the ONNX model on iOS.

🛠️ PR Summary

_{Made with ❤️ by Ultralytics Actions}

🌟 Summary

This PR for the ultralytics/yolo-ios-app adds support for ONNX pose detection models and various improvements to the object detection features.

📊 Key Changes

• iOS Setup:

  • Added a Podfile for dependency management with ONNX Runtime and extensions.
  • Included bridging header YOLO-Bridging-Header.h for integrating ONNX libraries.

• UI and Config Updates:

  • Removed multiple sliders and segmented controls from the UI.
  • Updated Main.storyboard to remove obsolete settings and controls.

• New Utilities and Handlers:

  • Created Utils.swift for handling pose detection and visualization.
  • Added VNOnnxHandler.swift for processing images and vision requests using the ONNX model.
  • Added VNOnnxRequest.swift for managing ONNX model request setup.

• Enhanced Bounding Box Visualization:

  • Updated BoundingBoxView.swift to support pose keypoints visualization.
  • Introduced keypoint layers to display detected poses effectively.

• Capture and Model Handling:

  • Updated VideoCapture.swift to include front camera support.
  • Modified ViewController.swift to load and interpret ONNX models, removing Vision-related code.

🎯 Purpose & Impact

• Enhanced Object Detection Performance: The integration of ONNX models enhances the app's ability to detect and visualize poses, leading to more accurate detections.
• Refined User Experience: By simplifying the UI and eliminating unnecessary controls, user interaction becomes more intuitive and less cluttered.
• Developer Utility: New utilities and handlers make it easier for developers to work with ONNX models and customize object detection pipelines.
• Future-Proofing: Modernizing the codebase to use ONNX ensures the app stays compatible with cutting-edge machine learning technologies and tools.

[glenn-jocher]

@pbanavara hey there! 👋

Thanks for your effort and interest in integrating the YOLOv8 pose ONNX model with iOS. It's great to see such enthusiasm within our community. For this kind of integration, you'd typically need to ensure that the ONNX model is fully compatible with the iOS platform, which may require using ONNX Runtime or Core ML.

A simple outline to get started could be:

1. Convert the YOLOv8 ONNX model to Core ML using ONNX-CoreML tools (if not directly using ONNX Runtime).
2. Incorporate the converted model into your iOS app, ensuring you handle the input and output correctly.

Remember, performance and compatibility can vary, so testing is key. For more detailed guidance on working with ONNX models, you might want to check out the Ultralytics Docs. However, since specifics can get quite technical and platform-dependent, I recommend reaching out on specialized forums or platforms for iOS and ONNX integration for more tailored advice.

Keep up the great work, and we look forward to seeing what you achieve! 🚀

[pbanavara]

Hi @glenn-jocher This ONNX model is compatible with iOS and I have tested the same. Listing down the instructions for testing. Is there an issue with using the Cocoapods architecture for this app ? I will try to get the CoreML conversion working. As per the ONNX repo, there is no supported tool to convert from ONNX to CoreML, will have to do this directly from Apple CoreMLTools which has support for Pytorch models. So it's back to square one :)

• clone/fork this repo
• Download the yolov8_pose_e2e model from the official onnx link
• Run the yolov8_pose_e2e.py command with a test image to download the full onnx model.
• Include the model in the build settings of this app by copying the same.
• Build and run

[glenn-jocher]

Hey @pbanavara! 🌟

Great to hear the ONNX model works well with iOS, and thanks for sharing your testing steps!

Regarding the use of Cocoapods, it's generally a sound approach for managing library dependencies in iOS projects. If it suits your project structure and you're comfortable managing dependencies through it, there shouldn't be an issue.

For the CoreML conversion, indeed, direct conversion tools from ONNX to CoreML might be limited, but using Apple's CoreMLTools with a PyTorch model as an intermediary step is a smart workaround. It might add an extra step, but if it bridges the compatibility gap, it's worth it.

For clarity to anyone following this:

1. Convert your PyTorch model to ONNX.
2. Use Apple's CoreMLTools to convert this ONNX model to CoreML format.

This method gives you the flexibility of PyTorch's extensive model ecosystem and the performance optimizations CoreML offers on iOS devices.

Keep pushing forward, and don't hesitate to reach out if you run into any hurdles! Your journey contributes to the broader knowledge base of our community. 🚀

[glenn-jocher]

@pbanavara closing as we do not have a plan to support ONNX models in the app, we are instead focusing on expanding CoreML support for all tasks including Pose. Thank you for your contributions and effort!

@ambitious-octopus @john-rocky FYI