Latency experimenting and analysis library for ROS2

Introduction

This library uses CARET (repository here, documentation here, paper here), a library for measuring end-to-end latency in ROS2 systems. The details of the installation can be found here, while a script that should do this automatically for you is provided here.

The ROS2 workspace that you are working in must be built with CARET. **Every time you modify the ROS2 workspace, you must build the project with CARET. This can be done with the following command:

colcon build --symlink-install --cmake-args -DBUILD_TESTING=OFF

CARET uses the Linux traces under the hood, which are also used by Linux to record events (irespective of ROS2). Since CARET is built with your ROS2 workspace, CARET will start to record traces every time you start a ROS2 process. When you kill the ROS2 process, CARET stops recording traces and saves them in a folder, by default as a sub-folder of ~/.ros/tracing. The created folder with traces contains information about each generated trace. CARET uses many samples to create a distribution of the latencies. After recording, CARET requires a special .yaml file to be made for a trace folder which is called "architecture file". This architecture file contains details about the nodes, their publishing/subscription topics and which executors they belong to. After the architecture file is created, the user needs to manually write in the architecture file the path of the ROS2 graph for which they want to find the end-to-end latency. Then, CARET offers functions to calculate the end-to-end latency, generate a .csv with all samples and also plot them. All these steps are required for only one set of initial values in ROS2. Thus, if one wants to test more initial values, then these steps must be automated, and this is what this library does. The flow of the library is illustrated in the figure below:

Steps

1. Record experiments with the record_experiments.py script from the record folder:

   python3 record_experiments.py /path/to/yaml/file/with/executor/config <name_of_working_ros2_workspace>
   

   You may modify the record_experiments.py file to run experiments for whatever executor configuration you want. More about how to create an executor configuration below, in the [Defining an executor configuration] section. The record_experiments.py script does two things:

   • Store the desired executor configurations in a data structure
   • Calls the run_experiment() function for each executor configuration

2. Transfer trace folders from RPi to laptop with the retrieve_traces.bash script from the scripts folder:

   ./retrieve_traces.bash -i <IP_address_of_the_Raspberry> -u <raspberry_username> -p <raspberry_password> -f <which_trace_folder_to_process: latest|all>
   

3. Create CARET architecture files for all trace folders with the retrieve_traces.bash script from the scripts folder:

   ./make-archi-files.bash <folder_with_trace_folders>
   

4. Add the path from the ROS2 graph that you want to analyze (i.e. for which you want to find the end-to-end latency), to the architecture file. The path must be given a name, and it is defined by providing its start and end nodes. Add this path to all trace folders with the add-path-archi.py script from the process folder:

   python3 add-path-archi.py /path/to/folder/with/trace/folders <start_node_of_ros2_path> <end_node_of_ros2_path> <desired_name_for_ros2_path> --subfolders True
   

5. Generate end-to-end latencies and .csv with all the latency data from the trace as byproducts of generating plots for each trace folder. Do this with the get-plots.py script from the process folder:

   python3 get-plots.py /path/to/folder/with/trace/folders <given_name_of_ros2_path> /path/to/folder/where/to/store/the/plots --subfolders True
   

6. Use a script to iterate over all trace folders, read the data for each trace folder, and plot them in the same plot. For instance, you can use the t2-10-all-plots.py script from plot folder:

   python3 t2-10-all-plots.py
   

   One has to modify the t2-10-all-plots.py script to use the correct path to the folder containing the trace folders.

Defining an executor configuration

The run_experiment() from the record_experiments.py script launches the ROS2 package that creates the executor configuration. The executor configuration is created at run-time from this executor_architecture_template.yaml file. The record_experiments.py uses its array of configurations to overwrite the executor_architecture_template.yaml file for each different configuration. Thus, if you want to make your own executor configurations you just need to create YAML objects that follow this predefined structure:

executors:
- name: executor1
  type: single_threaded
  cores: [1, 2, 3, 4]
  nodes:
  - name: node1
    payload: 35.00KB
    publish: topic1
    subscribe: NONE
  - name: node2
    payload: 77.00KB
    publish: NONE
    subscribe: topic1
- name: executor2
  type: multi_threaded
  cores: [1, 2, 3, 4]
  nodes:
  - name: node3
    payload: 128.00KB
    publish: topic3
    subscribe: NONE
  - name: node4
    payload: 56.00KB
    publish: NONE
    subscribe: topic4

The structure contains a list of objects called executors.

• Each executor object has 4 fields: name, type, cores, nodes.
  • The name is any string.
  • The type is either single_threaded or multi_threaded.
  • The cores is either a single integer or a list of integers e.g. [1, 2, 3, 4].
  • The nodes is a list of objects.
    • Each node object has 4 fields: name, payload, publish, subscribe.
      • The name is any string
      • The payload is a string and can be any real value followed by KB.
      • The publish and subscribe are both strings and represent the publish and subscribe topics of the node respectively. If a node doesn't publish/subscribe to a topic then that topic must be the string "NONE".

Examples of YAML objects representing executor configurations can be found here.