Hello! We’ve been battling some EKF errors on our fleet of aircraft, and are trying to understand what is causing them. We have aircraft that will periodically throw prefilight fail: EKF Internal Checks, as well as EKF Accel high bias. A log of this can be found here, which starts at boot to capture the phenomenon. https://logs.px4.io/plot_app?log=b2e02342-da60-46e8-8e7f-df959aab8c55
Our aircraft is an octo 8, flying pixhawk cube, Here2 GPS. Our firmware is based on px4 1.7.3 release.
We’re looking to better understand the underlying events that can be throwing these errors, and I’m trying to track backwards in the EKF code from the point the error was thrown. Any guidance on debugging this or comments based on looking at the logs would be helpful! I have looked through all the flags that are set and using the EKF Python tool I can see where the innovations and tests exceed the thresholds. However I’m not clear on what is actually CAUSING that to be the case.
In most cases, the EKF preflight errors are periodic, and we can typically arm and fly with nothing unusual happening.
Any Guidance on how to track these down would be helpful.
Assuming the aircraft is tuned and balanced, I would guess it has something to do with the cube’s multiple IMUs being summed instead of being processed independently like other more advanced autopilots do. Much better EKF results that way. Summing multiple sensors that are sampled at different rates and then looking at those harmonics all combined can’t be pretty. Doing them independently also allows for redundant EKF instances. Can you disable some of the IMUs? Compare a single hard-mount to a single internal-soft-mounted.
I’ve dug into the logs even further, and we’re seeing two different problems:
-
Getting IMU high accel bias. I’ve monitored the bias, and it stabilizes quickly and doesn’t change during the flight, and is consistent through 6+ flights. It just happens that the bias is slightly above the threshold for throwing the high bias warning. ~3e-3 vs 2.4e-3. I’m thinking either cal issue, or the accel is just like that and I need to turn up the detection threshold a little.
-
We get preflight failed: EKF internal checks messages quite a bit. I think I’ve correlated to random single entry spikes in magnetometer we’ve been seeing. I’ve also discovered that I don’t think our HERE2 GPS magnetometer is working with our 1.7.3-based build of PX4, so it switched to one of the cube’s internal mags. I’m still not sure why I’d get single sample spikes of such high magnitude. Compatibility with Here+ V2
@Jeremy_Zanzig Could you please enable logging from boot (SDLOG_MODE parameter) and send the log when you encounter this issue. That will tell us what is the problem.
@magicrub The signals from the different IMUs are not summed, or where did you see this?
The log file in my first post has logging enabled from boot and contains the errors.
https://logs.px4.io/plot_app?log=b2e02342-da60-46e8-8e7f-df959aab8c55
If useful, I have several others too.
I was under the impression that PX4 handles the multiple IMUs by selecting a prime one, and if it fails some set of checks it picks a new one to use. There is a preflight check that compares the output of all the sensors for consistency however.
Thanks!
Sorry, I missed that. Will have a look.
@tumbili did you happen to see anything in the logs? Thanks!
@Jeremy_Zanzig Yes, I had some time to look at it. The ekf reports a pre-flight fail at the beginning of your log (pre_flt_fail flag in estimator_status message)
The conditions for that flag to trigger can be found here
You can basically see that this can happen if the estimator innovations are too large. We check innovations for velocity, position, height and heading. In your case it looks like it might the heading innovation that triggers these failure.
I can see some spikes in the heading innovation when the vehicle is on the ground, probably you should check out your compass data.
this recent improvement might help some of your issues - might be worth testing if you are able: https://github.com/PX4/Firmware/pull/11350
