EKF issues after some time on the ground, no movement

Hi,

I’m working on a custom board based on PX4 flight stack. At the moment I’m working on 1.8.0 version, with some modules removed. I took whole EKF module to see how it will behave on it and I bumped into an issue with EKF estimates. The setup is:
-uC: STM32F407VGT6 (from stm32f4discovery)
-sensors: MPU9250(accel, gyro, mag), MS5611(baro), ublox NEO-M8P (base) in moving baseline configuration (sending RTCM data + standard with position, velocities, height etc, without satellite info), so the gps driver is slightly modified
-sensor drivers are not modified
-SD card with logs and parameters.

Few days ago I went to check how it will work to see estimates of velocities, position, attitude, height etc. The sensors are temperature compensated according px4 guide.

I was playing with it before with some walking for example and everything looked reasonable. At the day of another test I have few logs that look quite ok(002 for reference) with the exception of last one(003). In the last one it looks like the ekf went crazy. The board was all the time laying on the ground untouched and after some time estimated vel_n was showing sometimes even 40m/s movement and after analysing ekf logs I didn’t see anything like it before, and sensor’s readings(imu, baro, gps are look normal)

Can anyone help me with pointing what could be the reason?

Logs ecl_ekf analysis with “good” behaviour attached as files. And “bad” behaviour screenshots below, since the pdf is too big for uploading.

  1. log002 is first and it looks quite ok.

link to flight review:
https://logs.px4.io/plot_app?log=07bffdc9-13ed-4316-8993-29a81f7ef416

ecl_ekf analysis:
log002.ulg.pdf (2.3 MB)

  1. log003 is where it went crazy (it’s unfortunately to big to upload either the pdf here (it has 4MB) or ulog file to logs.px4.io, website crashes), but config, location, parameters can be taken from 002:


Thank You in advance!
Mateusz

Interesting, thanks for sharing the plots. I hope @Paul_Riseborough can comment on it.