Disarm and crash in offboard mode

After several days of successful flights we had an incident on our px4 based system. We switched into offboard mode and let our onboard controller take over, everything started out fine (as it had been for many flight hours prior) then about 20 seconds later px4 disarmed and the system fell flat to the ground. Flight log is here:

https://review.px4.io/plot_app?log=e8782a60-87f5-42d1-895d-fa4560a8befc

Some important points to make:

This was NOT a power failure. The system was still powered up at the crash site. We had telemetry all the way to the ground. Our radios have about a 30 seconds reboot time and so even a quick interruption of power would have meant lost telemetry. All onboard logging was correctly terminated upon disarm.

The log does not show the system falling, logging was shut down before this happened.

The telemetry log from qGroundcontrol confirms the system disarmed while in the air, and shows clean data all the way until impact.

The vehicle_command file shows some expected commands from our controller (comp 172) but no disarm command from our controller or qGroundControl. This is confirmed by the telemetry log.

Shortly prior to this incident there was a slight loss of GPS quality and the “velocity check bit” was set briefly.

Everything we have checked seems to indicate the px4 unilaterally decided to disarm while still flying. Why might this have happened?

We had exactly the same, but with ardupilot, not PX4. The reason for that was an I2C “runaway”, caused by a bug in the software. This caused the flight controller to disarm during flight. It has been fixed since. Maybe the PX4 flight stack have something similar?

You can see details here: