Bench test suite (Tools/bench_test): serial port changes after reboot/bootloader entry on Linux, breaking flashing and subsequent tests

Hi all,

I’m running the bench test suite (run_bench_suite.py, sih/flight_mission.py) on a CubeOrange+ under Ubuntu and ran into several issues related to serial port handling. I’d like to check whether these are known and what the recommended workflow is.

Issue 1: Port changes when a test reboots the board
Running ./run_bench_suite.py /dev/ttyACM0: one of the tests reboots the FC, the device re-enumerates as /dev/ttyACM1, and all remaining tests fail because the suite keeps using the old path.

Issue 2: by-id path works for tests but not for flashing
Using /dev/serial/by-id/usb-CubePilot_CubeOrange+_0-if00 avoids Issue 1. However, when the suite is asked to build the current tree and flash before testing, the board enters the bootloader and the by-id path changes to [bootloader by-id name, e.g. usb-CubePilot_CubeOrange+.x_BL_0-if00]. The flashing step then cannot find the board, so I had to build and flash manually via make ... upload.

Issue 3: Inconsistent results between runs
Results are not reproducible between runs; a test that runs in one attempt may not run (or fails) in the next. So far I got one good run on 1.18 and one mostly-working run on 1.17, but could not reproduce them.
Heartbeat is received, but the parameter read times out while 1.17 testing.

Hi kmert,

These look like three related but distinct issues in the bench-test workflow.

1. Serial device re-enumeration

Using /dev/ttyACM0 as a persistent identifier is fragile because Linux may assign a different device number after a reboot.

A /dev/serial/by-id/ symlink is generally a better option for the normal application connection, provided the device exposes a stable identity.

The test runner should still handle disconnection and reconnection explicitly, rather than assuming that the original serial connection remains valid.

2. Application versus bootloader identity

The bootloader and the running PX4 application can enumerate with different USB identities.

A single by-id path may therefore not work for both flashing and testing.

I would keep device discovery for the flashing phase separate from device discovery for the test phase, and verify the device identity after each transition.

A reasonable workaround is to flash separately, then run the bench suite against the application-mode by-id path.

3. Intermittent parameter timeouts

Receiving a MAVLink heartbeat does not necessarily mean the parameter service is ready.

I would check whether the suite:

  • Waits for the correct target system/component.

  • Reopens the serial connection after a reboot.

  • Clears stale MAVLink messages and test state.

  • Uses bounded retries for parameter requests.

  • Waits for the application to become fully ready before starting the next test.

I would also log USB disconnect/reconnect events, heartbeat timing, parameter request/response timing and the exact test that triggered each reboot.

Suggested regression tests

A useful upstream improvement would be a reconnect-aware test runner with explicit application/bootloader discovery, bounded readiness checks and clear timeout diagnostics.

Could you share the repository/branch containing run_bench_suite.py, the exact CubeOrange+ PX4 build target and one failing run log?

That would allow us to identify the relevant code path before proposing a patch.

Best regards,
Yiğitcan Öztürk
PAMILANGA LIMITED

For Issues 1 and 2, a udev rule helped me with a similar setup: match on the USB vendor ID (and the board serial if you have more than one) instead of the product name, so the same symlink is created both when the app is running and when the bootloader enumerates. Something like:

SUBSYSTEM==“tty”, ATTRS{idVendor}==“2dae”, SYMLINK+=“fc0”

in /etc/udev/rules.d/99-fc.rules, then sudo udevadm control --reload-rules && sudo udevadm trigger. Check the real IDs first with udevadm info -a -n /dev/ttyACM0 | grep -E "idVendor|idProduct|serial". After that you can point both the suite and the upload step at /dev/fc0. It doesn’t fix the reconnect logic in the runner itself, but it removes the path changing under you.