2024-12-16 ODD working group weekly meeting (218)

[Igata-ctrl]

Attendee

• Daniel Shih (TIER IV/Reference Design WG/National Taiwan University)
• Tran Huu Nhat Huy (Tokyo Institute of Technology, student)
• David Walmroth (PIX Moving)
• @mitsudome-r Ryohsuke Mitsudome (AWF TSC/TIER IV)
• @Igata-ctrl Hiroshi IGATA (TIER IV)

Agenda/Minutes

• On November 17, 2024, 295/340 "public road bus" and "pull out" scenarios succeeded on Autoware.universe main branch.

  • 110/142 public road bus scenarios were successful.

    • UC-LD-001-0002-Gebze : 1/3 case failed with time out. -> Succeeded with a simple retry.
    • UC-23-test-bikeway1 : Drivable area looks appropriately enhanced, but EGO stops for "Outside (of) Drivable Area" and Bicycle stops against Ego vehicle though "isBLIND:true" is set.
    • 30 other scenarios failed. All of these are on Shalun map and were succeeding on and before August 18, 2024. Currently, these scenarios are failing only by putting the Ego destination in or beyond the intersection ahead. @mitsudome-r is kindly investigating if Autoware.universe was modified in regards to handling of the intersections or the traffic signals after August 18.
      • @mitsudome-r commented that this issue is probably associated with the intersection module in planning of Autoware.universe. Software working group will kindly look into it.

  • 183/198 pull out scenarios were successful. 15 scenarios failed.

    • UC-v2-F-15-00002_001_case01 : 1/6 case failed as the expected back-up did not happen.
    • UC-v2-F-15-00002_001_case04 : 1/6 case failed with unnecessary stop of Ego. -> Looks OK after explicitly setting Ego max speed to 4.2m/s (15km/h).
    • UC-v2-F-15-00003_001_case01 : 1/4 case failed as Ego was not able to back up as intended. (It failed even after "act_ego nostop" condition was released.)
    • UC-v2-F-15-00006_001_case(01|02|03)_cmn_general : 3/4 cases failed for each of the scenario cases -> StoryboardElementStateCondition has been modified to reflect the original use case intention of "reach near max speed within 200m" but the "act_ego_speed_check" was not met as EGO accelerates only up to 15kph though the max speed is set higher and the lane speed limit is 60kph (9 parameter cases).
    • UC-v2-F-15-00007_001_case04_cmn_general : 1/36 failed with unnecessary stop of Ego. -> Ego max speed was already to 4.2m/s (15km/h) and succeeded in a retry.
    • UC-v2-F-15-00102_003_case01_cmn_general : 1/2 failed with unnecessary stop of Ego. -> Ego max speed was already to 4.2m/s (15km/h) and succeeded in a retry.
    • UC-v2-F-15-01001_001_case01_cmn_general : Failed. The playback looked successful but some of the success conditions "(Is the /planning/planning_evaluator/metrics_modified_goal_{longitudinal | lateral | yaw}_deviation_mean (= ) is lessThan 1.0?)" were not met.

• On November 17, 77/96 Dense Urban scenarios were successful on Autoware.universe scenario-dev branch.

  • Test-SafetyPool-map1 : Has been successful for a month, but failed this time. (A simple lane change did not happen.)
  • Test-roundabout1_Shalun was ported onto ITRI map as Test-roundabout2_ITRI and succeeded.
  • Test-dense-urban_ITRI(2|3|4) : EGO cannot use the space in the opposite lane when it passes/overtakes NPC/Pedestrians.
  • Test_MRM_Fault_Injection1 : MRM not conducted upon the fault injection.
  • UC-NTR-001-0003_case2 : 1/3 case failed but succeeded in a retry.
  • UC-NTR-001-0005 : All 3 cases failed for "Ego moved before Npc takeover finished.
  • UC-NTR-002-0003_case1 : 1/3 case failed with the excess deceleration.
  • UC-NTR-004-0001 : Both 2 cases failed for EGO not avoiding the NPCs coming in the wrong direction.
  • UC-VRU-002-0001_case(1|2) : All failed with the excess deceleration.
  • UC-VRU-002-0005 : Failed with the excess deceleration.

• On November 24, 2024, 297/340 "public road bus" and "pull out" scenarios succeeded on Autoware.universe main branch.

  • 112/142 public road bus scenarios were successful.

    • UC-23-test-bikeway1 : Drivable area looks appropriately enhanced, but EGO stops for "Outside (of) Drivable Area" and Bicycle stops against Ego vehicle though "isBLIND:true" is set.
    • UC-LD-001-0010-1-Gebze : Failed but succeeded with a simple retry.
    • 30 other scenarios failed. All of these were on Shalun map and succeeding on and before August 18, 2024. Currently, these scenarios are failing only by putting the Ego destination in or beyond the intersection ahead. @mitsudome-r is kindly investigating if Autoware.universe was modified in regards to handling of the intersections or the traffic signals after August 18.
      • @mitsudome-r commented that this issue is probably associated with the intersection module in planning of Autoware.universe. Software working group will kindly look into it. (Same as last week.)

  • 185/198 pull out scenarios were successful.

    • UC-v2-F-15-00002_003_case01 : 1/6 case failed, but it succeeded after explicitly setting Ego max speed to 4.2m/s (15km/h).
    • UC-v2-F-15-00003_001_case01 : 2/4 cases failed as Ego was not able to back up as intended. It looks like there's not sufficient space behind Ego to back up.
    • UC-v2-F-15-00006_001_case(01|02|03)_cmn_general : 3/4 cases failed for each of the scenario cases -> StoryboardElementStateCondition has been modified to reflect the original use case intention of "reach near max speed within 200m" but the "act_ego_speed_check" was not met as EGO accelerates only up to 15kph though the max speed is set higher and the lane speed limit is 60kph (9 parameter cases).
    • UC-v2-F-15-01001_001_case01_cmn_general : Failed. The playback looked successful but some of the success conditions "(Is the /planning/planning_evaluator/metrics_modified_goal_{longitudinal | lateral | yaw}_deviation_mean (= ) is lessThan 1.0?)" were not met.

• On November 24, 75/96 Dense Urban scenarios were successful on Autoware.universe scenario-dev branch.

  • Test-SafetyPool-map1 : Has been successful for a month, but failed this time. (A simple lane change did not happen.)
  • Test-roundabout1_Shalun was ported onto ITRI map as Test-roundabout2_ITRI and succeeded.
  • Test-dense-urban_ITRI(2|3|4) : EGO cannot use the space in the opposite lane when it passes/overtakes NPC/Pedestrians.
  • Test_MRM_Fault_Injection1 : MRM not conducted upon the fault injection.
  • UC-NTR-001-0001_case2 : 2/3 cases failed but succeeded after changing min_acc to -2.499.
  • UC-NTR-001-0003_case2 : 1/3 case failed but succeeded in a retry.
  • UC-NTR-001-0005 : All 3 cases failed for "Ego moved before Npc takeover finished.
  • UC-NTR-002-0003_case1 : 1/3 case failed with the excess deceleration.
  • UC-NTR-004-0001 : Both 2 cases failed for EGO not avoiding the NPCs coming in the wrong direction.
  • UC-VRU-002-0001_case(1|2) : All failed with the excess deceleration.
  • UC-VRU-002-0005 : Failed with the excess deceleration.

• On December 1, 2024, 299/340 "public road bus" and "pull out" scenarios succeeded on Autoware.universe main branch.

  • 113/142 public road bus scenarios were successful.

    • UC-23-test-bikeway1 : Drivable area looks appropriately enhanced, but EGO stops for "Outside (of) Drivable Area" and Bicycle stops against Ego vehicle though "isBLIND:true" is set.
    • 30 other scenarios failed. All of these were on Shalun map and succeeding on and before August 18, 2024. Currently, these scenarios are failing only by putting the Ego destination in or beyond the intersection ahead. @mitsudome-r is kindly investigating if Autoware.universe was modified in regards to handling of the intersections or the traffic signals after August 18.
      • @mitsudome-r commented that this issue is probably associated with the intersection module in planning of Autoware.universe. Software working group will kindly look into it. (Same as last week.)

  • 186/198 pull out scenarios were successful.

    • UC-v2-F-15-00002_002_case02 : 1/6 case failed, but it succeeded after explicitly setting Ego max speed to 4.2m/s (15km/h).
    • UC-v2-F-15-00003_001_case01 : 1/4 case failed as Ego was not able to back up as intended. It looks like there's not sufficient space behind Ego to back up.
    • UC-v2-F-15-00006_001_case(01|02|03)_cmn_general : 3/4 cases failed for each of the scenario cases -> StoryboardElementStateCondition has been modified to reflect the original use case intention of "reach near max speed within 200m" but the "act_ego_speed_check" was not met as EGO accelerates only up to 15kph though the max speed is set higher and the lane speed limit is 60kph (9 parameter cases).
    • UC-v2-F-15-01001_001_case01_cmn_general : Failed. The playback looked successful but some of the success conditions "(Is the /planning/planning_evaluator/metrics_modified_goal_{longitudinal | lateral | yaw}_deviation_mean (= ) is lessThan 1.0?)" were not met.

• On December 1, 76/96 Dense Urban scenarios were successful on Autoware.universe scenario-dev branch.

  • Test-SafetyPool-map1 : Has been successful for a month, but failed this time. (A simple lane change did not happen.)
  • Test-roundabout1_Shalun was ported onto ITRI map as Test-roundabout2_ITRI and succeeded.
  • Test-dense-urban_ITRI(2|3|4) : EGO cannot use the space in the opposite lane when it passes/overtakes NPC/Pedestrians.
  • Test_MRM_Fault_Injection1 : MRM not conducted upon the fault injection.
  • UC-NTR-001-0001_case1 : 1/3 case failed with the excess deceleration < -2.499.
  • UC-NTR-001-0003_case2 : 2/3 cases failed with the excess deceleration < -2.499.
  • UC-NTR-001-0005 : All 3 cases failed for "Ego moved before Npc takeover finished.
  • UC-NTR-002-0003_case1 : Succeeded. (was: "1/3 case failed with the excess deceleration.")
  • UC-NTR-004-0001 : Both 2 cases failed for EGO not avoiding the NPCs coming in the wrong direction.
  • UC-VRU-002-0001_case(1|2) : All failed with the excess deceleration.
  • UC-VRU-002-0005 : Failed with the excess deceleration < -2.499.

• Summary of currently filing scenarios.

  • Unnecessary Ego stop (It was not OK to explicitly set "Ego max speed".)
    • Following scenarios have been reverted to the original condition, "Ego max speed" was deleted. Therefore, some these currently succeeding scenarios may fail with unnecessary Ego Stop from next week.
      • UC-v2-F-15-00001_001_case01
      • UC-v2-F-15-00002_001_case01
      • UC-v2-F-15-00002_001_case03
      • UC-v2-F-15-00002_001_case04
      • UC-v2-F-15-00002_002_case02
      • UC-v2-F-15-00002_002_case03
      • UC-v2-F-15-00002_003_case01
      • UC-v2-F-15-00003_001_case01
      • UC-v2-F-15-00003_001_case03
      • UC-v2-F-15-00003_001_case04
      • UC-v2-F-15-00003_001_case05
      • UC-v2-F-15-00003_002_case02
      • UC-v2-F-15-00003_002_case03
      • UC-v2-F-15-00004_001_case03
      • UC-v2-F-15-00007_001_case04_cmn_general
      • UC-v2-F-15-00102_003_case01_cmn_general
      • UC-v2-F-15-00007_004_case02_cmn_general
      • UC-v2-F-15-00007_002_case03_cmn_general
  • Scenarios that are failing by placing the Ego destination in or beyond the intersection ahead (planning module issue?)
    • Example scenario:
      • UC-PB-008-0001_20_20-Shalun (recent result)
  • Missing topic "metrics_modified_goal_{longitudinal | lateral | yaw}_deviation_mean"
    • Example scenario:
      • UC-v2-F-15-01001_001_case01_cmn_general (recent result)
  • Scenarios that are failing by exceeding the "min_acc" parameter (maximum comfortable deceleration), etc.
    • Example scenario:
      • UC-NTR-001-0001_case1 (recent result)

• On December 8, 2024, Autoware build failed and no new result was obtained.

• On December 15, 2024, 294/340 "public road bus" and "pull out" scenarios succeeded on Autoware.universe main branch.

  • 112/142 public road bus scenarios were successful.

    • UC-23-test-bikeway1 : Drivable area looks appropriately enhanced, but EGO stops for "Outside (of) Drivable Area" and Bicycle stops against Ego vehicle though "isBLIND:true" is set.
    • UC-PB-009-0001-Db20-Hsinchu : 1/2 failed this week, but it succeeded in a retry without any changes.
    • 30 other scenarios failed. All of these were on Shalun map and succeeding on and before August 18, 2024. Currently, these scenarios are failing only by putting the Ego destination in or beyond the intersection ahead. @mitsudome-r is kindly investigating if Autoware.universe was modified in regards to handling of the intersections or the traffic signals after August 18.
      • @mitsudome-r commented that this issue is probably associated with the intersection module in planning of Autoware.universe. Software working group will kindly look into it. (Same as recent weeks.)

  • 182/198 pull out scenarios were successful.

    • UC-v2-F-15-00002_001_case01 : 1/6 case failed, but it succeeded in a retry.
    • UC-v2-F-15-00003_001_case01 : 1/4 case failed as Ego was not able to back up enough to make space ahed to pull out.
    • UC-v2-F-15-00006_001_case(01|02|03)_cmn_general : 3/4 cases failed for each of these scenarios. -> StoryboardElementStateCondition has been modified to reflect the original use case intention of "reach near max speed within 200m" but the "act_ego_speed_check" was not met as EGO accelerates only up to 15kph though the max speed is set higher and the lane speed limit is 60kph (9 parameter cases). The cases in which ego-speed = 2.78 m/s[10 kph] only succeeded.
    • UC-v2-F-15-00007_001_case04_cmn_general : 1/36 case failed with unnecessary stop of Ego, but succeeded in a retry.
    • UC-v2-F-15-00007_004_case01_cmn_general : 1/18 case failed with unnecessary stop of Ego, but succeeded in a retry.
    • UC-v2-F-15-00101_003_case01_cmn_general : 1/2 case failed with unnecessary stop of Ego, but succeeded in a retry.
    • UC-v2-F-15-01001_001_case01_cmn_general : Failed. The playback looked successful but some of the success conditions "(Is the /planning/planning_evaluator/metrics_modified_goal_{longitudinal | lateral | yaw}_deviation_mean (= ) is lessThan 1.0?)" were not met.

• On December 15, a build error occurred with "scenario-dev" branch and no result obtained for Dense Urban scenarios (and LSV for Reference Design WG).

• Troubleshoot of Huy's on-premise simulator setup. The long path to the map files may have been the cause.

• [recap of the discussions until last week]

• As for the following scenarios in the Low Speed Vehicle suite (Reference Design WG), both worked fine after deleting Npc1's "Max speed = 0 m/s" by clicking the trash can icon to the right. This may apply to other failing scenarios in the LSV suite.

  • RefLSV-UC-A-03-0003
  • RefLSV-UC-A-03-0004
  • It was an OpenSCENARIO specification that Npc does not move even if you set a certain speed in the Actions, if you had defined "Max speed = 0 m/s" for Npc.

• Continued discussion of the Jay walkers use case including the vehicle-pedestrian interface, the (pedestrian) intention estimation, etc.

• As for the new scenario suite (catalog) for LSV, it seems that almost all the scenario files are just changed with names. May need to ask Daniel-san to revert all the titles to the original one and take the following time-consuming process to duplicate and modify the copied scenarios for LSV/Reference Design use.

  • Open the original scenario with the scenario editor in Evaluator.
  • [Export yaml]
  • [+ New Scenario]
  • [Import yaml]
  • Give the new name like "RefLSV-UC-A-01-0001" in the [Title] column in the scenario editor
  • [Save] the scenario
  • Deselect the obsolete scenarios from the [Catalogs]Urban-Dense - [SUITES] LSV for Ref Design #63636 and add new ones.

• Started discussion of the Jay walkers use case.

• Had a Safety Pool/AWF meeting regarding the existing issues.

• Discussed with Daniel whether the scenarios are appropriately copied to LSV catalog instead of modifying the original. -> @Igata-ctrl to double check it by next week.

• Preliminary discussion with TIER IV Evaluator team revealed that the import of OpenSCENARIO 1.x files will be supported hopefully by the end of 2024. Currently, Scenario Simulator v2 uses the proprietary t4v2 format in yaml to better handle the parameter variations. This nee import function enables the use of out side scenario editor like AVL SCENIUS™️ Scenario Designer and help importing scenarios from Safety Pool database, though the issue of map format and the lack of destinations may remain.

• Discussion to create Evaluator improvement wish list.

  • Embed illustrations/animation in the Scenario Editor description pane or enumeration. (Armağan)
  • Integration with Vector Map Editor. (Igata) -> Clarify if some of this function is already available.
  • Create a discussion to collect needs/wants for Evaluator from larger audience.

• Discussed the improvement requests for Evaluator.

  • Will continue discussion next week, Armağan's initial ides included embedding the use case definition (slides) and the scenario (NPC behavior) preview by the time-slider in the scenario editor screen, etc.
  • @Igata-ctrl explained that Evaluator is not an OSS but a product from TIER IV, so the development team has a long wish list form TIER IV and OEM projects.
  • May need to clarify the content of the wish list from AWF before discussing the priority with TIER IV CI/CD team. @Igata-ctrl to try to collect recent situation with the team.

• Regarding the needs for calculating the accurate RSS safety distance from Vf(Npc) and Vr(Ego), TIER IV Mapping and Localization team will be able to add the functionality using UserDefinedValueCondition like UC-v2-F-15-01001_001_case01_cmn_general case. It may take more than two weeks depending on their resource margin/availability. -> @brkay54 to kindly submit an issue regarding this request mentioning Takeuchi-san of TIER IV (@kosuke55) who is working on UserDefinedValueCondition.

• For the current scenario suite for weekly execution above, the RSS definition for minimum safety distance and 2.499m/s^2 for maximum comfortable deceleration have been applied.

• @Igata-ctrl got the information in Simulation working group meeting that the resource allocated to Scenario Simulation v2 on the cloud may not be sufficient and it can make the simulation result not trustworthy enough. We may need to analyze the Simulation Log file to see if there are many waring messages in there when we get unexpected simulation result.

• @Igata-ctrl to investigate if we can use real time speed of Ego and Npc to calculate RSS Safe Distance more accurately.

• Discussion (RSS distance and comfortable deceleration) pages have been added to the Use cases for Dense Urban Area 2024 presentation.

• Reached a consensus to use RSS definition for minimum safety distance and 2.499m/s^2 for maximum comfortable deceleration for low speed shuttle.

• Discussion of the condition/threshold setting so that we can better track the Autoware maturity through scenario simulation.

• After the discussion, it turned out "min_acc" parameter should better be reverted to original value of -1.5 (this week).

• Current discussion with Safety Pool

  • [Map] Successfully converted a sample OpenDRIVE map from Safety Pool database to Lanelet2, but we have following issues. Though Mohammad (Deepen.ai/AWF Safety Assurance WG lead) proposed in May TSC that Autoware should consider accepting OpenDRIVE maps in a direct fashion as it solves all these issues, given that we may need to map-less autonomy, it may not worth switching map format from now.
    • Speed limit, lane change flag and LHT/RHT are lost through the conversion.
    • The converted Lanelet2 map contains a lot of unnecessary points on line strings. (e.g. 16,000points on 1 km one way straight road)
  • [Scenario] Autoware requires EGO's destination in each scenario, while all of the Safety Pool scenarios do not have the destination. Though Safety Pool once suggested to run Autoware in the “roaming” mode which does not require the destination in the scenario, it sounds unrealistic for Autoware to operate without destination as the destination is almost always an important part of the use case/scenario intention. (e.g. turning left/right).
    • In the Safety Pool Webinar, they demonstrated Autoware running with Safety Pool scenario, but they say that it requires some special settings for it and it is not scalable.
  • [Scenario file format] Though both Safety Pool and Autoware (Scenario Simulator v2) use OpenSCENARIO v1.x (a.k.a. OpenSCENARIO XML) standard, Autoware (Scenario Simulator v2) uses ”.yaml” file to better handle the parameters, while Safety Pool uses “.xosc” file, which is the default of the OpenSCENRIO standard. If the number of scenarios is not too big, we might be able to convert them from .xosc to .yaml format manually. (Currently, no automated converter is available.) If the scenario simulator v2 is installed on-premise, it will accept the .xosc file in a direct fashion.
  • [Scenario availability] After signing up for the Safety Pool database, the number of the accessible/visible scenarios are limited. 32,588 scenarios are visible but most of the scenarios are marked with "access restricted" and only 1957 scenarios are accessible. The total number of the scenario which Safety Pool claims is one million. Still waiting for an answer from Safety Pool

Action Item

Documents

• The bus ODD use case list which was assigned to the members is here.

• The bus ODD use case list has been also uploaded here to make it visible to anyone on the net.

• The ODD working group shared document folder is here. [Restored!]

• Discussions and Q&As in AWF Discord ODD WG channel are also encouraged.

Tools

• [Autoware Evaluator (CI/CD pipeline)]

  • Cloud based DevOps (integration of the development tools including the scenario editor and the scenario simulator below)
  • A product from TIER IV and offered for the official Autoware Foundation projects like Cargo Delivery and Public Road Bus, etc.
  • The user guide is available TIER IV document site
  • As it consumes AWS resource, the (batch) weekly execution of the scenario simulation is managed by the Software/ODD WG leads

• [Scenario Simulator V2 (Scenario testing framework)]

  • Stand alone scenario simulation tool
  • An OSS from TIER IV freely available for any Autoware developer/researcher
  • The documentation of the Scenario Testing Framework (open sourced from Tier IV) is on GitHub

• The GUI Scenario editor

  • Web based GUI scenario editor freely available from TIER IV
  • You can create and export scenarios with this web interface
  • The user guide is available TIER IV document site

• [TIER IV account]

  • The working group members who are interested in creating and testing scenarios are advised to create a free TIER IV account here.
  • Once you have created your account, please let @Igata-ctrl know the (long) User ID which appears on your login page. After @Igata-ctrl registered you to the AWF group in the Evaluator (CI/CD pipeline), you can go to AWF Autoware Evaluator page to see the simulation results, create/edit scenarios, etc.
  • If you already have a TIER IV account, your 4-digit User ID continues to work, so you do not need to register to TIER IV account again.

Administrative

• The two meetings on December 25th (Christmas day) and January 1st (New Year day) have been cancelled.

• Please check the ODD WG wiki page.

• Recurring weekly meetings have been scheduled. Please check Autoware Foundation events calendar and add this calendar to your own Google calendar by clicking the right bottom button and/or add your contact to ODD WG invitation group to receive invitations for future meetings.

• ODD WG meetings are held weekly in the following single time slot.

  • 7:00am, Monday (PST) / 6:00am, Monday (PDT) US Pacific Time
  • 10:00am, Monday (EST) / 9:00am, Monday (EDT) US Eastern Time
  • 4:00pm, Monday (CEST) / 3:00pm, Monday (CEDT) Poland time
  • 5:00pm, Monday (TRT) Turkey time
  • 10:00pm, Monday (CST) Taiwan time
  • 11:00pm, Monday (JST) Japan Time