Inconsistent behaviour in scenarios with street barrier in leaderboard 1.0.

[olek-osikowicz]

Hi there,

I am currently working on flakiness (non-determinism) in evaluating ADSs in CARLA.

I have been evaluating TF++ and discovered that evaluating of same scenario "2" from evaluation set multiple times yields varying results.

The problem is the collision with the street barrier.
Both the ego car (TF++ agent) and CARLA autopilot vehicles cars just pass through the static object.
see the video recording

By manually investigating each replay car goes through the barrier every time.
However problem is that collisions are not reported consistently by collision sensor:

Details

repetition	N_collisions	DrivingScore
0	3	27.4625
1	2	42.25
2	0	100
3	1	65
4	1	65
5	0	100
6	5	11.6029
7	0	100
8	1	65
9	0	100

The number of collisions detected with that barrier ranging from 0 to 5 and the driving score ranging from 11.6 to 100.

To sum up the problem is two fold:

1. CARLA autopilot ignores street barriers as if they don't exist
2. Collision sensor reports diffrent behaviours in the same go-through case.

Software used

• OS: Ubuntu 22.04.4 LTS
• CARLA: 0.9.10.1
• Leaderboard 1.0 commit: 2165f5e4a53be8e96c501fa39965800489b1a0a5
• ScenarioRunner 1.0 commit: b9c342eee4ba03cfdec58b184f10f89b7053b65e