Crash due to incorrect attitude estimation

PX4 Autopilot
1

Hi,
We were testing a sligtly modified version of PX4 v1.8.2 on one of our drones when we suddenly got an incorrect estimation of pitch, roll and yaw. We are using the EKF2 estimator.
The log is here: https://logs.px4.io/plot_app?log=2847fa53-b844-4cb0-8224-0eb023d80b4c
The error came at 9:38.553

Can somone help me identify why this happened?

2

Wow, that’s an interesting one. And sorry about the crash!

I don’t have a conclusion but a couple of things that caught my eye:

  1. Gyro sensor measurements seem correct, no jumps.
  2. Accel sensor measurements seem correct, no jumps.
  3. Mag sensor measurements change considerable at 9:38.5, but don’t jump.
  4. EKF2 raises mag flags, presumably triggered by the mag measurements.
  5. Roll and pitch attitude estimate jump like crazy at 9:38.6, that’s scary, and I don’t think the EKF2 should react that way even if the mag goes completely nuts.
  6. Two ESCs complain about HAL sensor errors at 9:39. I wonder if this is a cause or symptom.

@Paul_Riseborough I would appreciate your input.

3
  • The mag sensor measurmenst are unreliable on this platform because of very large motors.
  • ESCs complain because of the proppellers are blocked after the crash. So this is a symptom.

I quite agree with you that the EKF2 shouldn’t react that way if any sensor goes nuts.
I would have expected the estimator raise an error over mavlink and ignore the crazy sensor. In this flight we got no warning before the aircraft suddenly leaned over.

1 Like
4

Krevne,

Can you please tell me approximately what motor RPM your setup is using to hover (or motor KV) and the number of blades on the props.

Also what autopilot hardware are you using and how is it mounted to the frame.

Looking at the data there is evidence the IMU experiences some internal aliasing or clipping starting at 578.5 seconds.

Screen Shot 2019-03-20 at 6.37.36 am.png839Ă—755 118 KB

Screen Shot 2019-03-20 at 6.38.23 am.png840Ă—760 66.1 KB

The EKF uses an inertial navigation calculation that requires IMU data to an assumed level of accuracy, and then applies progressive corrections tot he data based on other sensor readings, eg Magnetomer, Baro, GPS, etc. The amount of correction is based on assumed accuracy of each sensor. If IMU data goes bad outside the assumed errors then it cannot be corrected by other sensors, however I do not think this is the main issue.

The EKF builds up a large magnetic heading innovation (inconsistency between magnetometer and inertial nav) immediately after takeoff

Screen Shot 2019-03-20 at 6.55.01 am.png836Ă—757 60 KB

This is due to extremely high levels of magnetic interference, making the magnetometer completely unusable, eg:

Screen Shot 2019-03-20 at 7.08.12 am.png697Ă—755 88.2 KB

This resulted in the magnetometer data being rejected by the EKF shortly after takeoff, however that meant it was having to inertially dead-reckon yaw for the remainder of the flight. After 10 minutes the accumulated error was enough to destabilise the EKF. If yaw error build sufficiently, then GPS velocity and position which are in earth frame cannot be used to correctly correct for Roll/Pitch error that accumulates in the inertial nav.

In summary the loss of attitude reference was caused by incorrect integration of the autopilot. However, given the EKF was broadcasting the problem via the estimator_status message, which shows the mag_test_ratio above 1 (failed magnetometer), for over 9 minutes prior to the crash, the SW deficiency in this instance is a failure to notify the operator.

2 Likes