Frequent local position loss despite good GPS

Hello,

I am having a strange issue on PX4 1.15.4 where the ‘local position invalid’ failsafe is frequently getting triggered, despite consistently good GPS data.

I did some digging into the EKF and saw that this seems to be caused by the GPS position aid src not getting fused, so I added estimator_aid_src_gnss_pos to the logged topics at full rate. I see that the “fused” flag is getting set to false, however, the innovation_rejected flag is never set true:

What’s very confusing to me is that when I look at the source code (Line 79 of position_fusion.cpp), the fused flag only gets set to false if innovation_rejected is true. The only other way for this flag to get set to false is through the resetEstimatorAidStatus, but when that happens, innovation_rejected also gets set to true. So I don’t understand how this is logically possible.

Can someone help troubleshoot? Thanks!

Here is the log: https://review.px4.io/plot_app?log=af60260d-30aa-4693-a393-7829ad0bd36f

What sort of GPS is this? It looks like one connected via DroneCAN? Here3+ or HerePro?

I’m confused because all the GPS data seems very static except the velocity accuracy is changing sometimes.

Have you done any updates or manual configuration of the uBlox inside the GPS? Something definitely seems off.

I am using the ARK RTK GPS over the CAN1 bus. Any thoughts @AlexKlimaj ? But yeah, I don’t think the GPS is the issue because like you say the GPS data is very static.

Here’s some data between the GPS sensor and the estimator. We can see that the fusion stops when there are “bumps” in the GPS sensor eph, (and since the gps is no longer getting fused, the estimator eph starts increasing), but I wouldn’t expect these relatively small increases in sensor eph to cause lack of fusion. Plus, like I said, the innovation_rejected flag is not getting triggered, so I don’t logically understand how it’s possible that the fused flag is getting set to false.

@dagar it looks like you wrote the part of the code that handles aid_src fused and innovation_rejected, do you have any ideas?

It seems related to this issue

If it is repeatable can you try flying main?

Yes, looks like the same issue. I did some deeper debugging and it looks like the GPS speed accuracy check is failing - though it wasn’t showing up like that because the estimator_gps_status topic is only logged at 1hz.

So runGnssChecks is failing because the speed accuracy is briefly increasing above 0.5 m/s (default threshold), and then the GPS is not getting fused because isTimedOut does not getting triggered for 5 seconds.

So the question is, why is the speed accuracy jumping this much? It does look like this is coming straight from the GPS sensor. And should the default threshold be this low?

Not sure about the default threshold. Maybe open a PR in PX4 to change the default value and hopefully that will stimulate a discussion.

Yes the speed accuracy comes directly from the GPS module. If satellite count is high but you’re seeing increased eph/epv/speed_accuracy I would say that’s an indication of interference or jamming.