You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
UC-23-004-0001-With2Obstacles-ShalunRS : Ego looks like ignoring the traffic from its left.
UC-23-test-bikeway1 : Drivable area looks appropriately enhanced, but EGO stops for "Outside (of) Drivable Area" and Bicycle stops though "isBLIND:true" is set.
In addition to the above, 30 scenarios (including parameter variations) failed due to system errors (Simulation Health Check, Time Out etc.).
UC-v2-F-15-00003_001_case01 : Failed for (D1=8, D2=4) case. Ego was not able to back up due to "obstacle stop".
UC-v2-F-15-00006_001_case(01|02|03)_cmn_general : Failed -> StoryboardElementStateCondition has been modified to reflect the original use case intention of "reach near max speed within 200m" but the "act_ego_speed_check" was not met as EGO accelerates only up to 15kph though the max speed is set higher and the lane speed limit is 60kph (9 parameter cases).
UC-v2-F-15-00103_001_case01_cmn_general : "Obstacle stop" prevents EGO from proceeding. -> The scenario intention is to use the space outside of the lane, but current Autoware.universe main is not capable of it.
Test-SafetyPool-map1 : Has been successful for a month, but failed this time. (A simple lane change did not happen.)
Test-dense-urban_ITRI(2|3|4) : EGO cannot use the space in th opposite lane when it passes/overtakes NPC/Pedestrians.
Test-roundabout1_Shalun : Ego cannot avoid collision with the NPC in the round about. -> It looks like NPC is not detected (as the red bounding box does not appear in Rviz screen).
Test_MRM_Fault_Injection1 : MRM not conducted upon the fault injection.
UC-NTR-001-0003_case1 : Vn=5.55 (slowest) case failed for the excess deceleration (<-2.499m/s2).
UC-NTR-001-0005 : All 3 cases failed for "Ego moved before Npc takeover finished.
UC-NTR-001-0008 : Failed for Vslow= 3.472 case as Ego decelerated to < 80% of Vslow. -> Became successful after the map version-up and fixing Npc start timing, etc.)
UC-NTR-001-0009 : Failed for as Ego decelerated to < 80% of Vslow. (After explicitly setting Ego max speed to 4.2m/s, it then causes a crash against Npc.)
UC-NTR-002-0001_case(1|2), UC-NTR-002-0003_case(1|2) : EGO cannot overtake NPC when Npc is fast (=2.77m/s) and failed with the excess deceleration.
UC-NTR-002-0004 : Failed with the excess deceleration.
UC-NTR-003-0002 : Failed as Ego stops twice and one of the failure conditions was met.
UC-NTR-004-0001 : EGO did not avoid the NPCs coming in the wrong direction. ("max_negative_acc" was not modified.)
UC-VRU-001-0001_case1 : Failed with unnecessary stop, or "act_check_ego_stopped_in_overtaking".
UC-VRU-001-0001_case2 : Failed with the excess deceleration.
UC-VRU-002-0001_case(1|2) : Failed with the excess deceleration.
UC-VRU-002-0002_case(1|2) : Failed with the excess deceleration.
UC-VRU-002-0003_case(1|2) : Failed with the excess deceleration.
UC-VRU-002-0005 : Failed with the excess deceleration.
111/142 public road bus scenarios were successful. Identical to the result of recent weeks. (Strictly speaking this number was from a retry on September 30 with "Retry=1" option, though the number of success cases increased by two without any retries.)
UC-23-004-0001-With2Obstacles-ShalunRS : Ego looks like ignoring the traffic from its left.
UC-23-test-bikeway1 : Drivable area looks appropriately enhanced, but EGO stops for "Outside (of) Drivable Area" and Bicycle stops though "isBLIND:true" is set.
In addition to the above, 30 scenarios (including parameter variations) failed due to system errors (Simulation Health Check, Time Out etc.).
UC-v2-F-15-00002_001_case03 : 1/6 case failed as Ego stopped unnecessarily. -> Suceeded after explicitly setting Ego max speed = 4.2m/s.
UC-v2-F-15-00003_001_case01 : 3/4 cases failed as Ego was not able to back up as intended".
UC-v2-F-15-00006_001_case(01|02|03)_cmn_general : Failed -> StoryboardElementStateCondition has been modified to reflect the original use case intention of "reach near max speed within 200m" but the "act_ego_speed_check" was not met as EGO accelerates only up to 15kph though the max speed is set higher and the lane speed limit is 60kph (9 parameter cases).
UC-v2-F-15-00103_001_case01_cmn_general : "Obstacle stop" prevents EGO from proceeding. -> The scenario intention is to use the space outside of the lane, but current Autoware.universe main is not capable of it. -> Succeeded this week! (The scenario has not been modified.)
On September 29, 2024, Dense Urban scenarios failed with build error. @mitsudome-r and @brkay54 are kindly retrying them.
[recap of the discussions until last week]
Discussed the improvement requests for Evaluator.
Will continue discussion next week, Armağan's initial ides included embedding the use case definition (slides) and the scenario (NPC behavior) preview by the time-slider in the scenario editor screen, etc.
@Igata-ctrl explained that Evaluator is not an OSS but a product from TIER IV, so the development team has a long wish list form TIER IV and OEM projects.
May need to clarify the content of the wish list from AWF before discussing the priority with TIER IV CI/CD team. @Igata-ctrl to try to collect recent situation with the team.
Regarding the needs for calculating the accurate RSS safety distance from Vf(Npc) and Vr(Ego), TIER IV Mapping and Localization team will be able to add the functionality using UserDefinedValueCondition like UC-v2-F-15-01001_001_case01_cmn_general case. It may take more than two weeks depending on their resource margin/availability. -> @brkay54 to kindly submit an issue regarding this request mentioning Takeuchi-san of TIER IV (@kosuke55) who is working on UserDefinedValueCondition.
For the current scenario suite for weekly execution above, the RSS definition for minimum safety distance and 2.499m/s^2 for maximum comfortable deceleration have been applied.
@Igata-ctrl got the information in Simulation working group meeting that the resource allocated to Scenario Simulation v2 on the cloud may not be sufficient and it can make the simulation result not trustworthy enough. We may need to analyze the Simulation Log file to see if there are many waring messages in there when we get unexpected simulation result.
@Igata-ctrl to investigate if we can use real time speed of Ego and Npc to calculate RSS Safe Distance more accurately.
[Map] Successfully converted a sample OpenDRIVE map from Safety Pool database to Lanelet2, but we have following issues. Though Mohammad (Deepen.ai/AWF Safety Assurance WG lead) proposed in May TSC that Autoware should consider accepting OpenDRIVE maps in a direct fashion as it solves all these issues, given that we may need to map-less autonomy, it may not worth switching map format from now.
Speed limit, lane change flag and LHT/RHT are lost through the conversion.
The converted Lanelet2 map contains a lot of unnecessary points on line strings. (e.g. 16,000points on 1 km one way straight road)
[Scenario] Autoware requires EGO's destination in each scenario, while all of the Safety Pool scenarios do not have the destination. Though Safety Pool once suggested to run Autoware in the “roaming” mode which does not require the destination in the scenario, it sounds unrealistic for Autoware to operate without destination as the destination is almost always an important part of the use case/scenario intention. (e.g. turning left/right).
In the Safety Pool Webinar, they demonstrated Autoware running with Safety Pool scenario, but they say that it requires some special settings for it and it is not scalable.
[Scenario file format] Though both Safety Pool and Autoware (Scenario Simulator v2) use OpenSCENARIO v1.x (a.k.a. OpenSCENARIO XML) standard, Autoware (Scenario Simulator v2) uses ”.yaml” file to better handle the parameters, while Safety Pool uses “.xosc” file, which is the default of the OpenSCENRIO standard. If the number of scenarios is not too big, we might be able to convert them from .xosc to .yaml format manually. (Currently, no automated converter is available.) If the scenario simulator v2 is installed on-premise, it will accept the .xosc file in a direct fashion.
[Scenario availability] After signing up for the Safety Pool database, the number of the accessible/visible scenarios are limited. 32,588 scenarios are visible but most of the scenarios are marked with "access restricted" and only 1957 scenarios are accessible. The total number of the scenario which Safety Pool claims is one million. Still waiting for an answer from Safety Pool
Action Item
Documents
The bus ODD use case list which was assigned to the members is here.
The bus ODD use case list has been also uploaded here to make it visible to anyone on the net.
The ODD working group shared document folder is here. [Restored!]
The working group members who are interested in creating and testing scenarios are advised to create a free TIER IV account here.
Once you have created your account, please let @Igata-ctrl know the (long) User ID which appears on your login page. After @Igata-ctrl registered you to the AWF group in the Evaluator (CI/CD pipeline), you can go to AWF Autoware Evaluator page to see the simulation results, create/edit scenarios, etc.
If you already have a TIER IV account, your 4-digit User ID continues to work, so you do not need to register to TIER IV account again.
Administrative
The two meetings on December 25th (Christmas day) and January 1st (New Year day) have been cancelled.
Recurring weekly meetings have been scheduled. Please check Autoware Foundation events calendar and add this calendar to your own Google calendar by clicking the right bottom button and/or add your contact to ODD WG invitation group to receive invitations for future meetings.
ODD WG meetings are held weekly in the following single time slot.
7:00am, Monday (PST) / 6:00am, Monday (PDT) US Pacific Time
10:00am, Monday (EST) / 9:00am, Monday (EDT) US Eastern Time
4:00pm, Monday (CEST) / 3:00pm, Monday (CEDT) Poland time
reacted with thumbs up emoji reacted with thumbs down emoji reacted with laugh emoji reacted with hooray emoji reacted with confused emoji reacted with heart emoji reacted with rocket emoji reacted with eyes emoji
-
Attendee
Agenda/Minutes
On September 22, 2024, 298/340 "public road bus" and "pull out" scenarios succeeded on Autoware.universe main branch.
111/142 public road bus scenarios were successful. Identical to the result of recent weeks.
187/198 pull out scenarios were successful. Two scenarios became successful after explicitly setting Ego's max speed of 4.2m/s.
On September 22, 2024, 62/96 Dense Urban scenarios were successful on Autoware.universe scenario-dev branch.
On September 29, 2024, 294/340 "public road bus" and "pull out" scenarios succeeded on Autoware.universe main branch.
111/142 public road bus scenarios were successful. Identical to the result of recent weeks. (Strictly speaking this number was from a retry on September 30 with "Retry=1" option, though the number of success cases increased by two without any retries.)
185/198 pull out scenarios were successful. Two scenarios became successful after explicitly setting Ego's max speed of 4.2m/s.
On September 29, 2024, Dense Urban scenarios failed with build error. @mitsudome-r and @brkay54 are kindly retrying them.
[recap of the discussions until last week]
Discussed the improvement requests for Evaluator.
Regarding the needs for calculating the accurate RSS safety distance from Vf(Npc) and Vr(Ego), TIER IV Mapping and Localization team will be able to add the functionality using
UserDefinedValueCondition
like UC-v2-F-15-01001_001_case01_cmn_general case. It may take more than two weeks depending on their resource margin/availability. -> @brkay54 to kindly submit an issue regarding this request mentioning Takeuchi-san of TIER IV (@kosuke55) who is working onUserDefinedValueCondition
.For the current scenario suite for weekly execution above, the RSS definition for minimum safety distance and 2.499m/s^2 for maximum comfortable deceleration have been applied.
@Igata-ctrl got the information in Simulation working group meeting that the resource allocated to Scenario Simulation v2 on the cloud may not be sufficient and it can make the simulation result not trustworthy enough. We may need to analyze the Simulation Log file to see if there are many waring messages in there when we get unexpected simulation result.
@Igata-ctrl to investigate if we can use real time speed of Ego and Npc to calculate RSS Safe Distance more accurately.
Discussion (RSS distance and comfortable deceleration) pages have been added to the Use cases for Dense Urban Area 2024 presentation.
Reached a consensus to use RSS definition for minimum safety distance and 2.499m/s^2 for maximum comfortable deceleration for low speed shuttle.
Discussion of the condition/threshold setting so that we can better track the Autoware maturity through scenario simulation.
After the discussion, it turned out "min_acc" parameter should better be reverted to original value of -1.5 (this week).
Current discussion with Safety Pool
Action Item
Documents
The bus ODD use case list which was assigned to the members is here.
The bus ODD use case list has been also uploaded here to make it visible to anyone on the net.
The ODD working group shared document folder is here. [Restored!]
Discussions and Q&As in AWF Discord ODD WG channel are also encouraged.
Tools
[Autoware Evaluator (CI/CD pipeline)]
[Scenario Simulator V2 (Scenario testing framework)]
The GUI Scenario editor
[TIER IV account]
Administrative
The two meetings on December 25th (Christmas day) and January 1st (New Year day) have been cancelled.
Please check the ODD WG wiki page.
Recurring weekly meetings have been scheduled. Please check Autoware Foundation events calendar and add this calendar to your own Google calendar by clicking the right bottom button and/or add your contact to ODD WG invitation group to receive invitations for future meetings.
ODD WG meetings are held weekly in the following single time slot.
Beta Was this translation helpful? Give feedback.
All reactions