Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[humble-devel] support multiple encoding #30

Merged
merged 4 commits into from
Sep 9, 2023
Merged

[humble-devel] support multiple encoding #30

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
2ffe64c
add support multiple encoding
Alpaca-zip Sep 9, 2023
9d25275
add launch file for kitti datasets
Alpaca-zip Sep 9, 2023
265c37b
change param name
Alpaca-zip Sep 9, 2023
a1c6167
Update README.md
Alpaca-zip Sep 9, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
50 changes: 30 additions & 20 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,16 +1,13 @@
# ultralytics_ros [![ROS2-humble Docker Build Check](https://github.com/Alpaca-zip/ultralytics_ros/actions/workflows/humble-docker-build-check.yml/badge.svg)](https://github.com/Alpaca-zip/ultralytics_ros/actions/workflows/humble-docker-build-check.yml)
ROS2 package for real-time object detection using the Ultralytics YOLO, enabling flexible integration with various robotics applications.

![yolo](https://github.com/Alpaca-zip/ultralytics_ros/assets/84959376/9da7dbbf-5cc0-41bc-be82-d481abbf552a)
| `tracker_node` | `tracker_with_cloud_node` |
| :------------: | :-----------------------: |
| <img src="https://github.com/Alpaca-zip/ultralytics_ros/assets/84959376/9da7dbbf-5cc0-41bc-be82-d481abbf552a" width="450px"> | **TODO** |

## Notice (For humble & WSL2)
`could not load library libcudnn_cnn_infer.so.8. Error: libcuda.so: cannot open shared object file: No such file or directory`
- The `tracker_node` provides real-time object detection on incoming ROS image messages using the Ultralytics YOLO model.

If you get an error like the above, add to the `.bashrc` file is:

`export LD_LIBRARY_PATH=/usr/lib/wsl/lib:$LD_LIBRARY_PATH`

## Setup
## Setup ⚙
```
$ cd ~/colcon_ws/src
$ git clone -b humble-devel https://github.com/Alpaca-zip/ultralytics_ros.git
Expand All @@ -19,32 +16,46 @@ $ cd ~/colcon_ws
$ rosdep install -r -y -i --from-paths .
$ colcon build
```
## Usage
### Run tracker_node
## Run 🚀
**`tracker_node`**
```
$ ros2 launch ultralytics_ros tracker.launch.xml debug:=true
```
## `tracker_node`
### Params
- `yolo_model`: Pre-trained Weights.
For yolov8, you can choose `yolov8n.pt`, `yolov8s.pt`, `yolov8m.pt`, `yolov8l.pt`, `yolov8x.pt`.
See also: https://docs.ultralytics.com/models/
- `publish_rate`: Publish rate for output topic.
- `input_topic`: Topic name for input.
- `output_topic`: Topic name for output.
For yolov8, you can choose `yolov8*.pt`, `yolov8*-seg.pt`, `yolov8*-pose.pt`.

| YOLOv8 | <img src="https://github.com/Alpaca-zip/ultralytics_ros/assets/84959376/08770080-bf20-470b-8269-eee7a7c41acc" width="350px"> |
| :-------------: | :-------------: |
| **YOLOv8-seg** | <img src="https://github.com/Alpaca-zip/ultralytics_ros/assets/84959376/7bb6650c-769d-41c1-86f7-39fcbf01bc7c" width="350px"> |
| **YOLOv8-pose** | <img src="https://github.com/Alpaca-zip/ultralytics_ros/assets/84959376/46d2a5ef-193b-4f83-a0b3-6cc0d5a3756c" width="350px"> |

See also: https://docs.ultralytics.com/models/
- `publish_rate`: Publish rate for `image_topic`.
- `image_topic`: Topic name for image.
- `detection_topic`: Topic name for 2D bounding box.
- `conf_thres`: Confidence threshold below which boxes will be filtered out.
- `iou_thres`: IoU threshold below which boxes will be filtered out during NMS.
- `max_det`: Maximum number of boxes to keep after NMS.
- `tracker`: Tracking algorithms.
- `classes`: List of class indices to consider.
See also: https://github.com/ultralytics/ultralytics/blob/main/ultralytics/datasets/coco128.yaml
- `debug`: If true, run simple viewer for output topic.
- `debug`: If true, run simple viewer.
- `debug_conf`: Whether to plot the detection confidence score.
- `debug_line_width`: Line width of the bounding boxes.
- `debug_font_size`: Font size of the text.
- `debug_labels`: Font to use for the text.
- `debug_font`: Whether to plot the label of bounding boxes.
- `debug_boxes`: Whether to plot the bounding boxes.

### Topics
- Subscribed Topics:
- Image data from `image_topic` parameter. ([sensor_msgs/Image](https://docs.ros.org/en/api/sensor_msgs/html/msg/Image.html))
- Published Topics:
- Debug images to `/debug_image` topic. ([sensor_msgs/Image](https://docs.ros.org/en/api/sensor_msgs/html/msg/Image.html))
- Detected objects(2D bounding box) to `detection_topic` parameter. (vision_msgs/Detection2DArray)
## `tracker_with_cloud_node`
**TODO**
## Docker with KITTI datasets 🐳
[![dockeri.co](https://dockerico.blankenship.io/image/alpacazip/ultralytics_ros)](https://hub.docker.com/r/alpacazip/ultralytics_ros)

Expand All @@ -53,9 +64,8 @@ See also: https://github.com/ultralytics/ultralytics/blob/main/ultralytics/datas
$ docker pull alpacazip/ultralytics_ros:humble
$ docker run -p 6080:80 --shm-size=512m alpacazip/ultralytics_ros:humble
```

### Run tracker_node
### Run tracker_node & tracker_with_cloud_node
```
$ ros2 launch ultralytics_ros tracker.launch.xml input_topic:=/kitti/camera_color_left/image_raw debug:=true
$ ros2 launch ultralytics_ros kitti_tracker.launch.xml
$ ros2 bag play kitti_2011_09_26_drive_0106_synced --clock --loop
```
41 changes: 41 additions & 0 deletions launch/kitti_tracker.launch.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
<?xml version="1.0" encoding="UTF-8"?>
<launch>
<arg name="yolo_model" default="yolov8n.pt"/>
<arg name="publish_rate" default="10"/>
<arg name="detection_topic" default="detection_result"/>
<arg name="image_topic" default="kitti/camera_color_left/image_raw"/>
<arg name="conf_thres" default="0.25"/>
<arg name="iou_thres" default="0.45"/>
<arg name="max_det" default="300"/>
<arg name="tracker" default="bytetrack.yaml"/>
<arg name="debug" default="true"/>
<arg name="debug_conf" default="true"/>
<arg name="debug_line_width" default="1"/>
<arg name="debug_font_size" default="1"/>
<arg name="debug_font" default="Arial.ttf"/>
<arg name="debug_labels" default="true"/>
<arg name="debug_boxes" default="true"/>
<!-- Object detection node -->
<node pkg="ultralytics_ros" exec="tracker_node" output="screen">
<param name="yolo_model" value="$(var yolo_model)"/>
<param name="publish_rate" value="$(var publish_rate)"/>
<param name="detection_topic" value="$(var detection_topic)"/>
<param name="image_topic" value="$(var image_topic)"/>
<param name="conf_thres" value="$(var conf_thres)"/>
<param name="iou_thres" value="$(var iou_thres)"/>
<param name="max_det" value="$(var max_det)"/>
<param name="tracker" value="$(var tracker)"/>
<param name="debug" value="$(var debug)"/>
<param name="debug_conf" value="$(var debug_conf)"/>
<param name="debug_line_width" value="$(var debug_line_width)"/>
<param name="debug_font_size" value="$(var debug_font_size)"/>
<param name="debug_font" value="$(var debug_font)"/>
<param name="debug_labels" value="$(var debug_labels)"/>
<param name="debug_boxes" value="$(var debug_boxes)"/>
<!--<param name="classes" value="0, 1, 2" value-sep=", "/>-->
</node>
<!-- Image view -->
<node if="$(var debug)" pkg="image_view" exec="image_view" output="screen">
<remap from="image" to="debug_image"/>
</node>
</launch>
4 changes: 2 additions & 2 deletions launch/tracker.launch.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
<arg name="yolo_model" default="yolov8n.pt"/>
<arg name="publish_rate" default="10"/>
<arg name="detection_topic" default="detection_result"/>
<arg name="input_topic" default="image_raw"/>
<arg name="image_topic" default="image_raw"/>
<arg name="conf_thres" default="0.25"/>
<arg name="iou_thres" default="0.45"/>
<arg name="max_det" default="300"/>
Expand All @@ -20,7 +20,7 @@
<param name="yolo_model" value="$(var yolo_model)"/>
<param name="publish_rate" value="$(var publish_rate)"/>
<param name="detection_topic" value="$(var detection_topic)"/>
<param name="input_topic" value="$(var input_topic)"/>
<param name="image_topic" value="$(var image_topic)"/>
<param name="conf_thres" value="$(var conf_thres)"/>
<param name="iou_thres" value="$(var iou_thres)"/>
<param name="max_det" value="$(var max_det)"/>
Expand Down
14 changes: 6 additions & 8 deletions ultralytics_ros/tracker_node.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ def __init__(self):
self.declare_parameter("yolo_model", "yolov8n.pt")
self.declare_parameter("publish_rate", 10)
self.declare_parameter("detection_topic", "detection_result")
self.declare_parameter("input_topic", "image_raw")
self.declare_parameter("image_topic", "image_raw")
self.declare_parameter("conf_thres", 0.25)
self.declare_parameter("iou_thres", 0.45)
self.declare_parameter("max_det", 300)
Expand All @@ -36,8 +36,8 @@ def __init__(self):
detection_topic = (
self.get_parameter("detection_topic").get_parameter_value().string_value
)
input_topic = (
self.get_parameter("input_topic").get_parameter_value().string_value
image_topic = (
self.get_parameter("image_topic").get_parameter_value().string_value
)
self.max_det = self.get_parameter("max_det").get_parameter_value().integer_value
self.conf_thres = (
Expand Down Expand Up @@ -77,9 +77,8 @@ def __init__(self):
1.0 / publish_rate, self.publish_detection
)
self.header = None
self.encoding = None
self.results = None
self.create_subscription(Image, input_topic, self.image_callback, 1)
self.create_subscription(Image, image_topic, self.image_callback, 1)
self.image_pub = self.create_publisher(Image, "debug_image", 1)
self.detection_pub = self.create_publisher(Detection2DArray, detection_topic, 1)

Expand All @@ -93,7 +92,7 @@ def publish_debug_image(self):
labels=self.debug_labels,
boxes=self.debug_boxes,
)
debug_image_msg = self.bridge.cv2_to_imgmsg(plotted_image, self.encoding)
debug_image_msg = self.bridge.cv2_to_imgmsg(plotted_image, encoding="bgr8")
self.image_pub.publish(debug_image_msg)

def publish_detection(self):
Expand All @@ -118,8 +117,7 @@ def publish_detection(self):

def image_callback(self, msg):
self.header = msg.header
self.encoding = msg.encoding
numpy_image = self.bridge.imgmsg_to_cv2(msg)
numpy_image = self.bridge.imgmsg_to_cv2(msg, desired_encoding="bgr8")
self.results = self.model.track(
source=numpy_image,
conf=self.conf_thres,
Expand Down