High packet drop when recording Ouster data in ROS 2 — single-core bottleneck?

Software
1

Hi all,

I’m trying to record sensor data in ROS 2 using the Ouster driver package. The package builds and runs without errors, but I am experiencing a large amount of packet drops during recording.

I already followed the recommended network and DDS tuning described here:
:backhand_index_pointing_right: Autoware DDS settings & network configuration

In short, I am using CycloneDDS as the middleware and applied the suggested sysctl network settings to increase UDP buffer sizes and reduce fragmentation time. This helped somewhat, but the drops are still significant.

System specs

  • CPU: Intel® Core™ i7-14650H (24 cores)

  • Network: not saturated (verified during recording)

  • OS driver: appears to be the main bottleneck

What I observed

  • os_driver seems to be saturating a single CPU core, while other cores are mostly idle.

  • My assumption is that deserialization of packets (or message processing in general) is happening on a single thread.

  • This might explain why packet loss persists even though network bandwidth and hardware capacity are sufficient.

Questions

  • Is there a way to parallelize or multithread the deserialization process in the Ouster driver or ROS 2 RMW layer?

  • Has anyone tried optimizing this, or are there configuration flags I may have overlooked?

  • Could this be a limitation of the current Ouster ROS 2 driver implementation?

Thanks in advance! Any suggestions or pointers (patches, forks, or configuration tips) would be very helpful.

2

Hi @joaquin-distance,

This is not a common issue that is being reported by our user base, are you on the last version of the driver? Which distro are you using?

With the machine spec that you provided you shouldn’t have issue with packet drop, it should be possible to carry over all the operations without using multi-threading. We have on thread that performs the packet read which passes them down the pipeline with minimal processing (IMU message generation + LaserScan accumlation). The PointCloud message generation is done on a separate thread since it does has more overhead, than other message types. This keeps the connection loop lightweight and responsive.

That being one could try several things to isolate the issue:

  • Try enabling RAW packet publish alone and save them into a bag and then replay it and check if there is still a packet drop, if there is this eliminates the theory that it is rather packet deserialization overhead.
  • I would also try enabling only the PointCloud topic and check if that helps (this would reduce the overhead incurred when publishing the IMU and Image topics.
  • If your pointcloud processing doesn’t require all the fields then you might try to use a different point_type such as the xyz, xyzi or xyzir which have much less fields to publish and thus less is need to be copied during deserialization (conversion) and require less bandwidth to publish.

Another thing to note here is that since version 0.12.3, the driver is designed to drop or skip whole scans rather than dropping packets when it can keep up due to pipeline processing speed limitation.

If you still think this packet drop is driven by the overhead of packet desearialization I would suggest profiling the driver and finding where the bottleneck seems to be on your setup (different machines can expose different behavior).

One area which I think may benefit from multi-threading is PointCloud composition part: ouster-ros/ouster-ros/src/point_cloud_compose.h at 06e7e29acea8046f61bc7703a93feb71d15bda8d · ouster-lidar/ouster-ros · GitHub but you need to conscious about the type of PointCloud being composed whether it is organized or unorganized (unorganized is a little harder to do in parallel).

Hope this helps!