Camera_Lidar_Fusion mode problem in Autoware

[ExitedState]

I am currently working with Autoware and AWSIM-Labs to create test scenarios involving object detection. I've encountered an issue with the perception modes and would appreciate any help or advice from the community.

The car correctly stops and does not crash into other vehicles when I run Autoware with the perception_mode set to lidar. However, when I switch the perception_mode to camera_lidar_fusion, the car does not stop and ends up driving through NPC cars.

Here are the commands I am using:
For Lidar Mode:
ros2 launch autoware_launch e2e_simulator.launch.xml vehicle_model:=sample_vehicle sensor_model:=awsim_labs_sensor_kit map_path:=/home/nimda/Downloads/nishishinjuku_autoware_map perception_mode:=lidar
For Camera_Lidar_Fusion Mode:
ros2 launch autoware_launch e2e_simulator.launch.xml vehicle_model:=sample_vehicle sensor_model:=awsim_labs_sensor_kit map_path:=/home/nimda/Downloads/nishishinjuku_autoware_map perception_mode:=camera_lidar_fusion

What I've done

• Verified that all sensors are properly configured and calibrated.
  
• Look at Autoware documentation and forums but couldn't find a solution to this specific issue.

Additional Information:

• Autoware version:
  
• AWSIM-Labs repository: https://github.com/autowarefoundation/AWSIM-Labs.git
• ROS 2 version: Humble

I have attached a video demonstrating the issue. In the video, you can see how the car behaves differently in lidar mode compared to camera_lidar_fusion mode.
Lidar Mode : Lidar Video
Lidar Fusion Mode : Camera Lidar Fusion Video

Question:

1. Has anyone encountered a similar issue with the camera_lidar_fusion perception mode in Autoware? If so, how did you resolve it?
   

2. Are there specific configurations or additional steps required to make camera_lidar_fusion work correctly?
   

3. Any guidance or suggestions would be greatly appreciated.
   

Thank you in advance for your help!

[YoshiRi]

@ExitedState
Hi. I've seen your video but could not get sufficient information.
Since the autoware planner cosiders the predicted object by default, which name is /perception/object_recognition/objects, it's better to show your 3D rviz result for next time.

To get into this issue, could you follow following tree to clearfy what is happend to your simulation?

• prediction topic: /perception/object_recognition/objects
• tracking topic: /perception/object_recognition/tracking/objects
• detection topic: /perception/object_recognition/detection/objects
  (please check ros2 topic list in case of my typo)

In my sense, I guess clustering based pipeline (see node diagram for detail) have somehow failed.

It will be very helpful for us to know which topics are not coming or which node is not working. Thanks for your contribution!

[ExitedState]

Hello, First of all, I checked the prediction topic: /perception/object_recognition/objects and found that it is present in the ROS2 topic list, along with the tracking topic and detection topic.

• prediction topic: /perception/object_recognition/objects
  
• tracking topic: /perception/object_recognition/tracking/objects
  
• detection topic: /perception/object_recognition/detection/objects
  

Secondly, I'm not sure if this information is useful. When I used the ros2 topic echo command with all topics (e.g. prediction topic, tracking topic, detection topic), I found that the object list is empty for the prediction topic and tracking topic, while the detection topic shows no output.

Third, are these ROS Topic parameters in CameraSensorHolder correct?

And here is the video of the 3D RViz result in camera_lidar_fusion mode.
camera_lidar_fusion video

[YoshiRi]

Hi @ExitedState .

Secondly, I'm not sure if this information is useful. When I used the ros2 topic echo command with all topics (e.g. prediction topic, tracking topic, detection topic), I found that the object list is empty for the prediction topic and tracking topic, while the detection topic shows no output.

Good. Now we know something wrong with camera_lidar_fusion detection pipeline.

Third, are these ROS Topic parameters in CameraSensorHolder correct?

QoS settings sometimes causes issue but we should find out where the topic/node has trouble first.

Next task suggestion

Since the detection is not coming and lidar only process works, I think following camera-lidar-fusion pipeline has something to do with this issue.

Pipeline depends on autoware version. So, please check your node chain by rqt_graph or dear_ros_node_viewer.
Or you can use rtui to track pub/sub stream.

It may easier to use rtui to track topic/node chain and check its availability.

debug flow

You can check the node/topic from edge of output.

1. find the uncoming topic
2. check the node name that should publish the uncoming topic
3. search for the log or command line output by its node name (normally autoware node should publish some warning or error when unintended events happened)

Good luck.

[Rak-r]

Hey, I am trying to integrate my ros2 custom vehicle setup with autoware. My version is Humble. I couldn't find guide to install/setup autoware and AWSIM for ROS2 humble and Ubuntu 22.04? @YoshiRi @ExitedState

[YoshiRi]

please check this👍
https://autowarefoundation.github.io/autoware-documentation/main/installation/autoware/source-installation/