ROS 2 common issues and mistakes

But what would be the best ways to make sure that all the Nodes have the sim-time parameter set correctly? In the very beginning, we used to declare the use_sim_time parameter in our parameter files for each Node individually. And since we use mostly the same launch files and parameter files for both the simulation and the real robot, we had to override the parameter by substituting it with a value we get as a launch argument. This required declaring use_sim_time in every launch file we had and passing it through multiple layers of them, making things quite verbose. Also, it was easy to forget to substitute the parameter for each individual Node declared in launch files.

Later on, we started to use the “SetParameter”-method which sets the sim time globally for the same launch context. This means that any Node or Launch include which is declared after this parameter set, will run under this sim time. This allowed us to get rid of the verbosity of launch argument declarations, and ensure that all the Nodes run with the same use_sim_time value.

import launch_ros

def generate_launch_description():
  return LaunchDescription([
    launch_ros.actions.SetParameter(name='use_sim_time', value=True)
    …
])

However, this method isn’t foolproof either. SetParameter needs to be declared in every entrypoint launch file, so if your system has multiple entry points, it needs to be re-declared multiple times. During development and debugging, if single Nodes or Launch files are launched independently, different from normal conditions, it might be very easy to forget to set the parameter correctly. Also, you’ll need to remember to do it for any external Node you might launch, such as Rviz. It is good to note that explicitly declaring use_sim_time parameter for a Node in the parameter file or in the launch file will override the effect of SetParameter.

I’m hoping to someday be able to set use_sim_time as an environment variable for ROS 2, which would further use it for all the Nodes that are run on that device.

Synchronous calls in callbacks

One common way to architecture Node is to implement it as a class and to make it spin with a blocking spin() – function. Currently, with default Node executor settings, it is not possible to call a service synchronously from another callback in this way as it will lead into a deadlock. In my experience, this is one of the most common issues developers bump into, especially when coming from ROS 1 to ROS 2.

A common pattern in our code is that we call multiple other services from another service callback, and we’ll need to wait for their results before returning the main service result. In this case, it doesn’t really matter if you call the service with synchronous call() or asynchronous call_async(), the service response will never arrive anyway due to the executor not spinning while it is executing the callback. And this cannot be even fixed by manually spinning as spinning in a callback would lead into recursive spinning, which isn’t allowed. The problem exists for both rclpy and rclcpp.

The solution is to use MultiThreadedExecutor with either MutuallyExclusiveCallbackGroup or ReentrantCallbackGroup, so that the Node is still spinning while the callback is executing. To find examples and to get a deeper understanding into the topic, you can check out our blog post “Deadlocks in rclpy and how to prevent them with use of callback groups”.

There has been discussion around this topic in many places, and there exists an open issue in rclcpp for it. Hopefully there will be some kind of solution to this eventually.

Invalid odometry data

Over the years we have learned how important valid odometry data is. It plays a key role in default navigation and SLAM solutions in ROS 2, so any weirdness in odometry data will directly affect autonomous driving capabilities.

Some most common reasons for odometry issues are:

• Inaccuracy (we recommend using motor encoders!)
• Relying on one odometry source – use sensor fusion, since almost all of the odometry sources have their weaknesses; wheels slip, IMU drifts, etc.
• Timestamps – try to stamp the odometry messages as close to measurement as possible
• Delays in the system – Is the data still fresh when it is received by the Node that uses it?
• Twist velocities in the odometry message being calculated incorrectly or reported in a wrong frame

We had an example some time ago related to the last point, when we integrated Boston Dynamics Spot to use nav2. It walked autonomously very well into one direction, but to others it walked as if it had taken a couple glasses of something stronger. It walked backwards, sideways, very slowly or doing criss cross movements with its legs, without following the given path. After multiple long debugging sessions, we ended up finding out that the odometry velocities in the ROS 2 spot_driver are reported in the wrong child frame. Since then, this has been already fixed for the ROS 1 version of the driver.