integrations/torchscript/

[giscus[bot]]

integrations/torchscript/

Learn to export your Ultralytics YOLOv8 models to TorchScript format for deployment through platforms like embedded systems, web browsers, and C++ applications.

https://docs.ultralytics.com/integrations/torchscript/

[tryprncp]

hi, I'm implementing yolov8 model in android gallery's search mechanism. the purpose of yolov8 model is to scan media files and return images that has a bounding box label that matches the search query. i can make it work with yolov5s.torchscript.ptl using org.pytorch:pytorch_android_lite:1.10.0 and org.pytorch:pytorch_android_torchvision_lite:1.10.0 but it wont work with yolov8s.torchscript. i really need help with this.

[glenn-jocher]

Hey there! 😊 It sounds like you've made great progress integrating YOLOv8 into your Android app's search mechanism! If you've successfully used YOLOv5 models but are facing issues with YOLOv8's TorchScript model, let's try to sort this out.

First, ensure you've exported the YOLOv8 model to TorchScript correctly using the latest YOLOv8 and PyTorch versions. Here's a quick snippet for exporting:

from ultralytics import YOLO

# Load YOLOv8 model
model = YOLO('yolov8s.pt')  # Use the appropriate YOLOv8 variant

# Export to TorchScript
model.export(format='torchscript')

This creates a file named yolov8s.torchscript. Ensure that this file is what you're trying to load in your Android app.

If you're still having trouble after confirming the export process, consider checking the PyTorch Android version compatibility. YOLOv8 may require a newer version of PyTorch Android than YOLOv5, so updating org.pytorch:pytorch_android_lite and org.pytorch:pytorch_android_torchvision_lite to the latest versions might help.

Lastly, since YOLOv8 is more recent, make sure you're using the latest release of the Ultralytics YOLO library for any compatibility fixes or improvements.

If these steps don't resolve the issue, feel free to share any error messages or more specifics about the problem you're encountering. I'm here to help!

[amarnathreddy0201]

Load the YOLOv8 model

model = YOLO("yolov8n.pt")

Export the model to TorchScript format

model.export(format="torchscript") # creates 'yolov8n.torchscript'

Load the exported TorchScript model

torchscript_model = YOLO("yolov8n.torchscript")

For loading "yolov8n.torchscript" here yolo is used.
is it possible to load the model using pytorch and prediction/Segmentation.

[amarnathreddy0201]

Model = torch.jit.load('yolov8n.torchscript')
results = Model(image)

It will predict the boxes, classes

[amarnathreddy0201]

@pderrenger I written small application for person detection but it's not detecting properly nd bounding boxes is some where else.

import torch
import cv2
import numpy as np

classes = [
"person", "bicycle", "car", "motorcycle", "airplane", "bus", "train", "truck",
"boat", "traffic light", "fire hydrant", "stop sign", "parking meter", "bench",
"bird", "cat", "dog", "horse", "sheep", "cow", "elephant", "bear", "zebra",
"giraffe", "backpack", "umbrella", "handbag", "tie", "suitcase", "frisbee",
"skis", "snowboard", "sports ball", "kite", "baseball bat", "baseball glove",
"skateboard", "surfboard", "tennis racket", "bottle", "wine glass", "cup", "fork",
"knife", "spoon", "bowl", "banana", "apple", "sandwich", "orange", "broccoli",
"carrot", "hot dog", "pizza", "donut", "cake", "chair", "couch", "potted plant",
"bed", "dining table", "toilet", "tv", "laptop", "mouse", "remote", "keyboard",
"cell phone", "microwave", "oven", "toaster", "sink", "refrigerator", "book",
"clock", "vase", "scissors", "teddy bear", "hair drier", "toothbrush"
]

model = torch.jit.load("yolov8n.torchscript")
image_ori = cv2.imread(r"C:\Users\amarn\OneDrive\Pictures\father mother.jpg")
image = cv2.resize(image_ori, (640, 640))
image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)
image_tensor = torch.from_numpy(image).float()
image_tensor = image_tensor.permute(2, 0, 1) # Change from HWC to CHW format
image_tensor = image_tensor / 255.0 # Normalize to [0, 1]
image_tensor = image_tensor.unsqueeze(0)

with torch.no_grad():
results = model(image_tensor)

results = results[0] # Remove the batch dimension if there's only one image

boxes = results[:, :4]
print(len(boxes))
object_confidences = results[:, 4] # Objectness score
class_confidences = results[:, 5:] # Class scores

conf_threshold = 0.5 # Example threshold; adjust as necessary
predictions = []
for i in range(len(object_confidences)):
if object_confidences[i] > conf_threshold:
class_id = torch.argmax(class_confidences[i]).item()
class_conf = class_confidences[i][class_id].item()

    if class_conf > conf_threshold:
        
        box_np = boxes[i].cpu().numpy() if boxes[i].is_cuda else boxes[i].numpy()
        x, y, w, h = box_np
        x1 = int((x - w / 2) )
        y1 = int((y - h / 2) )
        x2 = int((x + w / 2) )
        y2 = int((y + h / 2))
        cv2.rectangle(image_ori, (int(x1),int( y1)), (int(x1+ w), int(y1 + h)), (0, 255, 0), 2)

cv2.imshow('image', image_ori)
cv2.waitKey(0)

classes is from yolo only

[pderrenger]

It looks like there might be an issue with the bounding box coordinates. Ensure that the coordinates are correctly transformed from the model's output format to the image's coordinate system. Also, verify that the model is correctly loaded and that the input image is preprocessed as expected. If the issue persists, please try updating to the latest version of the Ultralytics package and check if the problem is reproducible. For more details on exporting and using TorchScript models, you can refer to the Ultralytics documentation.

[clausMeko]

@amarnathreddy0201 I could make the boxes work for my One-Item only use case. I guess the tensor structure was not really clear to you as it seemed to me you have profound knowledge. ty for heavy lifting 🥳

@glenn-jocher and @pderrenger if you care what use case I had and could not simply from ultralytics import YOLO: I chain YOLO with other models and have conflicts with imports, especially numpy v1 vs v2. So i figured export all as torchscript and chain them.

import torch
import cv2

# Knwoledge: https://github.com/ultralytics/ultralytics/issues/5863
# Knwoledge: https://github.com/orgs/ultralytics/discussions/9002
def preprocess_image(image_ori) -> torch.Tensor:
    image = cv2.resize(image_ori, (640, 640))
    image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)
    image_tensor = torch.from_numpy(image).float()
    image_tensor = image_tensor.permute(2, 0, 1)  # Change from HWC to CHW format
    image_tensor = image_tensor / 255.0  # Normalize to [0, 1]
    image_tensor = image_tensor.unsqueeze(0)

    return image_tensor

def main():
    # Load the TorchScript model
    map_location = torch.device('cuda') if torch.cuda.is_available() else torch.device('cpu')
    model = torch.jit.load('./best.torchscript', map_location=map_location)

    cv2_img = cv2.imread("../obj-det/datasets/test/images/00018_png.rf.d602082fdf686063a7d430aba6dfedde.jpg")
    image = preprocess_image(cv2_img)

    with torch.no_grad():
        results: torch.Tensor = model(image.to(map_location))

    results = results[0]  # Remove the batch dimension if there's only one image
    boxes = results[:4, :]
    object_confidences = results[4, :]
    conf_threshold = 0.5  # Example threshold; adjust as necessary

    dominant_item = torch.argmax(object_confidences)
    if object_confidences[dominant_item] < conf_threshold:
        cv2.imshow('nothing found', cv2_img)
        cv2.waitKey(0)
        return

    box_np = boxes[:,dominant_item].cpu().numpy() if boxes[:,dominant_item].is_cuda else boxes[:,dominant_item].numpy()
    x, y, w, h = box_np
    x1 = int((x - w / 2))
    y1 = int((y - h / 2))
    x2 = int(x1 + w)
    y2 = int(y1 + h)
    cv2.rectangle(cv2_img, (x1, y1), (x2, y2), (0, 255, 0), 2)
    cv2.imshow('image', cv2_img)
    cv2.waitKey(0)

if __name__ == '__main__':
    main()

[glenn-jocher]

Thank you for sharing your use case and detailed implementation! Exporting models as TorchScript for chaining is indeed a smart approach, especially when managing dependency conflicts. If adjustments are needed for bounding boxes or results, ensure the coordinate transformations align with the input image dimensions. For additional guidance on TorchScript usage, feel free to reference Ultralytics TorchScript Docs. Let us know if you encounter further issues!