Three things that cost me flights running offboard control with external vision on PX4 SITL (lockstep, uXRCE-DDS, Gazebo Harmonic)

PX4 Autopilot
1

Hello,

I fly an autonomous navigation stack on PX4 SITL v1.17 in Gazebo Harmonic over
uXRCE-DDS: ROS 2 Jazzy plans and publishes trajectory setpoints in offboard
mode, and the vehicle’s position comes from my own estimator through
/fmu/in/vehicle_visual_odometry, with GNSS and the magnetometer off. Over
the last weeks three PX4-side findings each cost me a flight before I
understood them, and none of them was in my planner. Posting them in case they
save someone else the same weeks, and because I would like to be corrected
where I am wrong.

The stack is open source and simulation-only:

11_room_crossing

(GIF: the setup in flight, in an industrial room 58 m short of the goal. Left
is Gazebo, right is RViz: the pink cloud is the obstacle memory the planner
searches, the cyan fan is the stereo depth of this frame, orange is the
trajectory PX4 is flying in offboard mode and the magenta sphere is the goal.
The position PX4 fuses comes from a visual-inertial filter on those same
images; there is no GNSS and no lidar on this airframe.)

The whole flight: https://www.youtube.com/watch?v=OUuAj2WNKzs

1. uXRCE-DDS time synchronisation resets when SITL runs below real time

SITL is lockstep, so the autopilot’s clock is the simulation clock: in my
recordings PX4’s boot-relative stamps sit 12 to 20 ms from the ROS time of the
setpoint received beside them. But UXRCE_DDS_SYNCT is on by default and
synchronises that clock against the agent’s wall clock. As soon as the
simulation runs slower than the wall clock (mine sits at a real-time factor of
0.8 to 1.0 while rendering a stereo pair), the two drift apart, and PX4 resets
the synchronisation roughly every 15 s. Each reset emits one boot-relative
stamp and then steps the estimated offset by 0.6 to 1.1 s.

A consumer that checks stamp monotonicity, as mine does, reads that as an
epoch reset and refuses the samples until the new epoch is established: about
1.1 s without an authoritative vehicle state, seventeen times in one flight.
One of those seconds fell on a route turn while the vehicle was flying a
horizon it had already committed to, and it hit a wall.

Lidar flights at a real-time factor of 1.00 never reset, which is why this was
invisible for weeks.

What I do now: param set UXRCE_DDS_SYNCT 0, and every node maps the
autopilot’s stamps to ROS time with the identity instead of the timesync
offset. Reacquisitions per flight went from 17 to 0. The real-vehicle path
keeps the synchronisation; this is a simulation-only setting.

Question: is there a reason to keep the synchronisation on in lockstep SITL at
all? From here it looks like it can only introduce error, since the two clocks
are the same clock.

2. EKF2_EV_NOISE_MD and what an odometry’s noise actually is

With EKF2_EV_CTRL 11 I first passed my estimator’s own variances
(EKF2_EV_NOISE_MD 0). That is wrong for an odometry, and not for the reason
I expected: a visual-inertial filter with first-estimate Jacobians never lets
features shrink the directions no camera observes, so its honest variance of
its position in the world grows without bound. An autopilot that has no
other source of position would gradually stop believing the only position it
is given.

So I moved to a fixed noise (EKF2_EV_NOISE_MD 1) and set EKF2_EVP_NOISE to
0.1 m, which is roughly how accurate the estimate is. That is also wrong. The
parameter has to cover what one update moves the pose by, because that is
what arrives at the filter as an innovation. In a fast turn my filter gated 13
of 19 features, its pose moved 0.4 m and came back within two frames, and PX4’s
gate (five sigma, so 0.5 m) rejected the odometry. The vehicle dead-reckoned
1.8 s of a 5 m/s dash on its IMU and then reset its position by 1.19 m when
fusion resumed. My controller treats a position reset beyond 0.33 m as a lost
frame and closes the navigation, so the mission ended there.

Measured on five flights, consecutive frames move my pose by up to 0.22 m.
EKF2_EVP_NOISE is 0.3 m now (a 1.5 m gate) and EKF2_EVA_NOISE 0.05 rad. No
rejection since.

Question: is there a recommended way to feed a filter whose innovations are
bursty? A fixed noise wide enough for the bursts is also wide enough to accept
a genuinely bad pose, and I have not found a better answer than “make the gate
wider and validate the estimator elsewhere”.

3. Publish the odometry faster than the sensor that produces it

The camera pair gives a pose 7.5 times a second, and the autopilot ends its
external-vision fusion after 200 ms without one. One late frame is enough to
open that gap, and restarting the fusion is expensive: in an earlier milestone
with a lidar-inertial estimator, three restarts reset the vehicle’s height by
0.80, 1.17 and 2.25 m.

My estimator therefore carries its last state forward through the IMU samples
received since and publishes every 40 ms, each pose dated at its own moment in
the synchronised clock rather than at the moment of publication. The filter
itself is not advanced by this; it is the same estimate extrapolated.

Two related settings from the earlier milestone, in case they help: I fuse
position and yaw but not the estimator’s velocity (EKF2_EV_CTRL 11), because
a visual-inertial velocity fused with a tight variance drove the autopilot’s
height to 4.8 m for a 1.1 m climb at takeoff; and EKF2_HGT_REF 3, because on
a barometer reference the autopilot’s altitude drifted a metre from the frame
my map and planner live in, and the vehicle flew into a structure a metre
below where it held itself.

One more, for anyone comparing an estimate against Gazebo’s truth

EKF2_GPS_DELAY defaults to 110 ms, which is right for a real receiver. The
Gazebo bridge stamps the simulated navsat sample when it receives it, so the
filter subtracts a delay that is not there and places the vehicle where it
would have moved to since: my position estimate ran 0.11 s ahead of the true
pose along the motion, which is 0.35 m at 3 m/s, and my lateral tracking error
at p99 fell from 0.13 to 0.29 m down to 0.03 to 0.18 m after setting it to
zero in SITL. If you fly offboard in Gazebo and the vehicle seems to cut
corners, this is worth checking.

Happy to share logs or parameter files for any of these.

2

Very useful write-up. The time-sync observation is particularly interesting because it suggests that some failures which appear downstream as “external vision instability” may actually originate earlier in the timing chain.

One experiment I think would be valuable is to correlate the complete sequence:

uXRCE-DDS time-sync event → EV sample timestamp/age → EKF2 innovation/test ratio → EV fusion state → estimator reset → offboard control response.

Rather than looking only at the point where EV fusion stops, I would try to identify the first observable divergence between a healthy run and a failing run.

For example, around each time-sync reset:

  • Does the effective age of the EV measurement change discontinuously?

  • Is there an innovation/test-ratio excursion before fusion is rejected?

  • Does estimator_status_flags change before the position reset?

  • Is the reset counter increment the first estimator-visible consequence, or is there an earlier detectable event?

  • How much time passes between the original timing disturbance and the downstream offboard-control effect?

If the same sequence repeats around the ~15 s events you observed, that would make it possible to distinguish the initiating timing fault from the later EKF2 and controller symptoms.

It could also be interesting to preserve those events as a small reproducible ULog/ROS 2 trace pair: one healthy interval and one failing interval. That would make the behaviour much easier to reproduce across PX4 / uXRCE-DDS / simulator versions.

Have you captured a ULog together with the corresponding ROS 2 timestamps from one of the runs where the reset repeats?

3

Hi @formiat

Thanks for the report. Regarding your first issue, that topic is actually discussed in ROS, Gazebo and PX4 time synchronization and disabling timesync is advised when ROS 2 is following GZ clock.
It is quite buried in the docs though so it can be missed.

Sync is not disabled by default because it depends on the ROS 2 settings. If you don’t use use_sim_time:=true on ROS 2 side then it makes sense to keep it on.

4

Thanks, that distinction is important.

So the practical rule seems to be:

ROS 2 on Gazebo sim time → disable UXRCE_DDS_SYNCT
ROS 2 on wall clock → keep timesync enabled

What may be worth documenting more explicitly is that this is not just a timestamp preference — a wrong clock-domain choice can propagate downstream into monotonicity checks, EV fusion gaps, estimator resets and eventually offboard-control behaviour.

For reproducibility, I’d record the clock configuration alongside the flight parameters in any SITL experiment:

  • use_sim_time
  • UXRCE_DDS_SYNCT
  • Gazebo real-time factor
  • PX4 / ROS 2 versions

That would make these failures much easier to reproduce and compare across setups.