ULog crash analysis: how do you identify the first causal failure instead of the final failsafe?

PX4 Autopilot
1

I’ve been looking at several PX4 incident logs recently, and one pattern keeps coming up: the most visible failsafe or termination event is often not the event that actually initiated the failure.

For example, in one recent analysis the vehicle eventually exceeded ~80° attitude and entered failsafe. Looking only at the final events could easily make the failsafe or tilt threshold look like the cause.

But reconstructing the ULog chronologically showed a different sequence:

  • commanded attitude remained small,
  • measured attitude began diverging from the setpoint,
  • actuator outputs subsequently reached extremes,
  • the vehicle continued losing attitude control,
  • failure detection asserted later,
  • and flight termination/failsafe occurred after that.

That made me think about a more general question for PX4 log analysis:

What is the most reliable way to identify the first causal failure in a ULog, rather than simply identifying the final detected failure?

The workflow I’ve been using is roughly:

command/setpoint → measured response → estimator state → controller/rate response → actuator outputs → vehicle state transitions → failure detection → failsafe

Then I work backwards only after finding the earliest meaningful divergence between expected and observed behaviour.

For attitude-control incidents, for example, I find the distinction between these two cases particularly useful:

A. Setpoint diverges first
Investigate where that command originated and which control path generated it.

B. Setpoint remains reasonable but measured attitude diverges
Move downstream toward rate tracking, control authority, actuator saturation and the physical motor/ESC/power path.

I’m interested in how PX4 developers and experienced log analysts approach this.

Do you generally have a preferred set of ULog topics/events for establishing the first divergence, especially when estimator, controller, actuator and failsafe events occur within only a few seconds of each other?

Also, are there cases where working from the first observable divergence can be misleading because the actual initiating condition is only detectable later in the logging chain?

I’d be interested in comparing methodologies.

2

Did estimator_status_flags show any innovation gating right before the divergence or was the EKF clean the whole way through?

3

Good question. That’s exactly one of the possibilities I’m trying to distinguish.

I haven’t established whether innovation gating preceded the divergence in this particular log, so I wouldn’t characterize the EKF as clean yet.

My next step would be to align estimator_status_flags with the relevant innovation test ratios, estimator reset events, attitude setpoints, measured attitude, rate tracking and actuator outputs.

The important distinction is whether an estimator inconsistency preceded the first observable control divergence or whether the estimator remained consistent while the vehicle failed to track its commands.

It also raises a broader methodological question: how do we distinguish the first observable divergence from the actual initiating failure when the available telemetry cannot directly establish causality?

Have you encountered cases where innovation gating was the earliest useful indicator, even before attitude or rate tracking visibly deteriorated?

4

Yeah, seen this a few times. Pattern’s usually the same: gating only gives you lead time when there’s a redundant sensor covering for the bad one.

Couple examples off the top of my head:

  • mag interference from power leads - innovation ratio spikes on throttle-up, way before yaw actually drifts, because gyro integration is holding yaw fine in the meantime. only shows up in the innovation test ratio, fused att looks normal
  • baro blocked by tape/foam - baro innovation climbs for a while but alt hold looks perfect because GPS alt is carrying it. doesn’t show downstream until GPS alt also gets sketchy
  • accel clipping near a vibration node - innovation gets noisy/elevated a sec or two before attitude estimate itself starts moving, since EKF just down-weights it and trusts gyro prediction

if there’s no redundant source though (e.g. gyro fault, nothing to fall back on for rate), gating buys you basically nothing - flag and visible divergence show up at the same time.

so tbh at this point I just check innov ratios before I even look at att/rate plots. if there’s lead time, redundant sensor was covering. if there isn’t, that tells you what was load-bearing when it broke.

what’s your innovation test ratio look like on this one, did it move before the 80° event or same time as everything else?

5

I checked the innovation test ratios in the ULog, and your point about looking upstream of the visible attitude divergence is particularly relevant here.

There is an interesting finding: the magnetometer innovation test ratios were already elevated well before the attitude divergence at 462.123 s.

For the primary EKF instance (1), the recorded values include:

  • mag_field[0]: approximately 3.24 at 456.740 s.

  • mag_field[2]: approximately 12.41 at 455.242 s.

However, looking further back shows that these elevated magnetometer ratios were already present at least 20 seconds before the attitude divergence.

Both EKF instances also exhibit similar behaviour.

So there is an earlier observable sensor-consistency anomaly, but I would not classify it as the initiating cause of this crash without further evidence.

The important distinction here is between the earliest detectable anomaly and the earliest anomaly that can actually be connected to the subsequent loss of control.

The confirmed control sequence remains: small commanded attitude, measured attitude divergence, actuator outputs reaching extremes, and later failsafe activation.

Your suggestion highlights an important limitation of first-divergence analysis: an earlier anomaly is not automatically a causal precursor.

Thanks for raising this. It’s a useful distinction for developing a more rigorous reconstruction methodology.

6

Yeah, agreed, earlier anomaly isn’t automatically causal. Was that mag ratio already high while disarmed, or did it jump at arm/throttle-up? If it’s the second one, that’s probably current interference or bad cal, a standing condition and not a fault onset

Also, a mag problem mostly hits yaw. It shouldn’t saturate the motors with a small attitude setpoint and an 80° tilt I’d check raw gyro against the attitude estimate to make sure the rotation is real, then compare esc_status vs actuator_motors and see if a motor stopped doing what it was told

7

Good point. I haven’t yet established whether the elevated mag ratios were already present while disarmed or whether they appeared around arming/throttle-up, so I don’t want to interpret them as a fault onset yet.

I’ll check that boundary explicitly.

I also agree that the motor saturation with a small attitude setpoint makes the gyro/actuator path more interesting than the magnetometer alone.

My next pass will therefore be:

  1. disarmed → armed → throttle-up mag innovation history,
  2. raw gyro vs attitude estimate around the first divergence,
  3. actuator_motors vs esc_status, if the required ESC telemetry is present in the log.

I’ll report the timestamps of the earliest disagreement rather than infer the cause from the first abnormal signal.

Thanks this gives me a much cleaner way to separate a standing condition from an actual fault onset.

8

sounds like a plan if esc_status isn’t in the log, control_allocator_status saturation plus battery_status current/voltage will still tell you a lot. post the timestamps when you’ve got them curious which one lands first.

9

Absolutely. If esc_status is unavailable, I’ll use control_allocator_status together with battery_status as the next evidence boundary.

I’ll keep the comparison timestamp-based and report which observable divergence occurs first, without treating temporal order alone as proof of causality.

I’ll post the results once I’ve completed that pass. Thanks this gives us a good fallback path.