Multi-Sensor Viz Crash Issue (4+ Sensors) with ouster-sdk

Hi,

I’m currently testing multi-sensor visualization using ouster-sdk 0.15.1 on a Windows 10 Pro laptop. My goal is to achieve real-time alignment with the following command:

ouster-cli -e extri.json 192.168.x.x,192.168.x.x,192.168.x.x,192.168.x.x viz

Here’s what I’ve tried so far:

  • Using 3 sensors connected via Ethernet-to-USB adapters and a simplified extri.json file (only modifying Y values), I was able to run the visualization successfully for about 5 minutes using:

    ouster-cli -e extri.json 192.168.x.x,192.168.x.x,192.168.x.x viz

  • When I moved to 4 sensors connected via a hub (D-LINK DGS-1008A) to the PC, launching viz would cause the app to immediately close.

  • However, I could still discover all 4 sensors using ouster-cli discover.

I tried the following sequence:

  1. Visualizing each sensor individually – worked fine.

  2. Visualizing 2 sensors at once – worked fine

  3. Visualizing 3 sensors at once – worked fine.(I think it was fine. but..)

  4. Visualizing 4 sensors at once – crashed within a minute.

  5. Reconfigured as sensor 3ea → hub → PC ← sensor 1ea, and it ran for about 5 minutes before shutting down.

  6. After rebooting the PC and connecting only 3 sensors via the hub, the visualization ran for 17 minutes before closing on its own.

Additional info:

  • My laptop (HP ZBook) has two GPUs: Intel UHD 630 and NVIDIA Quadro 600. By default, it uses Intel UHD, so I forced Python to use the NVIDIA GPU.

  • When using ouster-cli source <ip> viz with a single sensor, it streams fine for 30+ minutes.

  • However, using ouster-cli source <ip1>,<ip2> viz for multi-streaming, it crashes almost immediately.

  • When visualizing just one sensor, I was able to stream all day without any issues.

My questions:

  • Has anyone successfully visualized more than 4 sensors simultaneously using ouster-sdk for longer periods (1+ hours)?

  • Is it normal for viz to crash after a few minutes with multiple sensors?

  • Are there any recommended specs for stable multi-sensor visualization?

  • Could the hub (DGS-1008A) be the limiting factor?

  • I’m using OS0 and OS1-128 sensors.

Any insight would be greatly appreciated. Thank you!

Has anyone successfully visualized more than 4 sensors simultaneously using ouster-sdk for longer periods (1+ hours)

We definitely used the ouster-sdk/cli to visualize more than 4 sensors (actually I’ve had recording of 20+ sensors) but I don’t remember running them for an hour to confirm that we did.

Is it normal for viz to crash after a few minutes with multiple sensors?

No, that shouldn’t be the case.. which sdk version are you using and can you provide us with console output of the crash while using ouster-cli –tracback source ... viz

Are there any recommended specs for stable multi-sensor visualization?

32GB of RAM and a good graphics card.

Could the hub (DGS-1008A) be the limiting factor

The only way this could contribute to the failure or bad outcome is if there are many dropped packets or packets are way out of order that somehow ouster-sdk is not able to handle them properly. You can eliminate that this is a viz problem rather than a packet handling problem from the source by replacing the viz command with the save command.

Hi, samahu

Thank you for your suggestion.

I would like to share the results from using the --traceback option and the save command as you recommended.
As mentioned previously, I am currently using Ouster SDK version 0.15.1. For this test, I changed the PC’s graphics card to an NVIDIA Quadro 600.

The following setup was used:

  • Devices: OS1-128 Rev6 and OSdome-128

  • All sensors were connected directly to the PC.

  1. --traceback Result:
    (Please refer to the attached image)

In particular, whenever I attempt to visualize data from multiple sensors simultaneously (especially 3 or more), I consistently encounter similar error messages. This seems to be a recurring issue, and I have attached a screenshot for your reference.

  1. save Command Result:
    I successfully recorded and visualized data from 2 and 3 sensors respectively, each for under 5 minutes using the save command and playing back the resulting PCAP files.

However, since our main goal is real-time streaming, I would like to know if there are any recommendations or methods to improve the performance of the viz command for multi-sensor visualization.

Any insights you can provide on resolving or mitigating this issue would be greatly appreciated.

Thank you again for your support.

Best regards,

The log indicates that the ouster-cli application crashed after it stopped receiving packets from one of the sensors, is it always this one particular sensor that always end up not interacting with the app? If the answer is yes I would check if the sensor generated any alams during that period on the diagnostics page of the sensor. If not then I would examine the network.

You can also try increasing the the timeout period to 5 seconds just for testing using the the -t option (default is 1 second).

ouster-cli source -t 5 … viz

I would also make sure that you increase the network buffer size per our network recommendation:

sudo sysctl -w net.core.rmem_max=2147483647

also I would check if the proper graphics driver is installed

@Samahu
Hi Samahu,

Thank you for your assistance.
I have identified the cause of the issue—it was related to my graphics settings. I tested with three sensors, and the connection now works well. Although occasional data drops still occur, I consider it meaningful that continuous streaming is now possible.

Once again, I appreciate your support.

Best regards,

1 Like