Hi,
Our aircraft was losing global position and going into Failsafe mode for several seconds, while heading seemed to be healthy/correct.
The global position losses happened in the turns of a survey grid. Due to not-optimal wind conditions, the aircraft’s movement in the turns was not very sharp.
The comparison of raw gps path and the global position path is here. It’s clearly visible that the estimated global position deviates from the raw gps path, then resets.
The positional errors eph for global and local position increase at the time of the problem; however gps seems to be accurate enough.
No heading measurement error, only horizontal estimated position had problems, and only in the turns but (almost) in each of them. And each time the aircraft recovered from failsafe and continued to fly.
Could it be the aircraft’s motion causing problems with the ekf2? If so, with what settings can it be more robust? Or what other things could cause this frequent and regular failures in turns?
Firmware version: PX4 v1.13.3 firmware with custom controller modules using EKF2
GPS: CUAV RTK dual gps system
Flightcontroller board: Pixhawk 6X
Thanks for any help
Looking at the aircraft’s movement more closely:
global position (red dots and line on the map) seem to #follow# the heading (blue arrows), and assuming the aircraft is moving in the direction of its heading, i.e. assumes that the course on grand is same/similar as the heading angle. Even though the gps position (brown dots) are moving the opposite way.
Which is not necessarily true, for our aircraft. (The direction of movement is shown by the arrows on the map.)
A related question about the ekf parameter settings, does the EKF2_AID_MASK parameter’s ‘GPS yaw fusion’ setting mean that the heading value from dual GPS is fused into the position estimate, or into the heading estimate?
I was thinking, fixed wings generally follow their heading, or heading changes - if the fixed wing heading is turning left, the trajectory will probably follow it. This is not the case for multicopters of course. Our aircraft is an airship, it is somewhere between the two in that manner - its heading does determine the trajectory but it is affected much more by the wind than a fixed wing /heavier than air/ that it cannot and should not be used for position estimation imo.
Could the estimator library have some parts that differentiates between vehicle types?
As you mentioned, your system does not have a valid global position. This is caused because the local position estimate is not valid anymore. So I’m pretty sure your GPS data is not getting fused for some time and the vehicle is dead-reckoning. Maybe its first using wind-dead-reckoning for some time but then the airspeed measurements are not valid anymore within the turn (if you have one), which then causes the reset to GPS despite failing checks. In general it would be really useful if you attach a log, so we can check all the simple failures quickly. Otherwise I can only speculate without spending a lot of time investigating.
I would also strongly recommend to upgrade to 1.15, since 1.13 was first released in 2022.
1 Like
In general it would be really useful if you attach a log, so we can check all the simple failures quickly. Otherwise I can only speculate without spending a lot of time investigating.
Thanks @haumarco for the answer. Due to the proprietary/commerciality of the flight and our controller, I cannot share full logfile unfortunately. I know it’s hard to debug anything without it, I appreciate any tips and observations. If a specific plot/topic can help, I can probably share snippets of it.
So I’m pretty sure your GPS data is not getting fused for some time and the vehicle is dead-reckoning.
I can confirm that the horizontal position observation was rejected by the estimator before the global position was determined lost. From the ekf’s sourcecode I suppose that must be the gps, as we don’t use external vision.
However the gps did not seem to have problem in-flight, always seeing 30+ satellites, and having under 20cm position accuracy.
Maybe its first using wind-dead-reckoning for some time…
Wind-deadreckoning was not used during the flight at all, according to the ekf logs.
This is caused because the local position estimate is not valid anymore.
I have also noticed the local_position’s velocity estimate to diverge from the gps-s velocity estimate (N and E components). And that is estimated from integrating the IMU. We have yet to testfly it, but could there be that inaccurate IMU and GPS offset parameters cause this kind of divergence?
I would also strongly recommend to upgrade to 1.15, since 1.13 was first released in 2022.
Yes, upgrading for a newer version of px4 is among the plans as well
The main issue is that I dont have knowledge/intuition about the behavior within 1.13. So I dont even exactly know what you have available in your logs. You can check the test_ratios of each sensor to see if it gets accepted by the EKF or not, I dont think there was already a “fused” flag.
Yes it is possible that you introduce a velocity offset through the transformation between IMU and GPS. Maybe your GPS system uses 2 antennas and also provides heading? Maybe the magnetometer messes up the heading?
If you can’t figure things out, try to update to 1.15 and or create a isolated log file. You can also join the Q&A call which takes place every Wednesday on Discord.
We are looking into ways of how and what to share. It’ll take some time, but I’ll get back as soon as I know more.
Yes we are using 2 gps antennas, and switching between them might worsen the effects of the preset gps antenna offsets, especially given that EKF2 only has a parameter set for 1 antenna. It’s the same in newer versions, too (1.14 and 1.15) as I see the parameter list, which seems like a big flaw/incompatibility in EKF. Unless I have missed some parameters on how to set up the location of the second antenna.
