Strange magnetometer surges. MAG #0 fail: STALE!

Hello all,

there are strange magnetometer surges (1-3 seconds long) happen occasionally:

Log file link: click log_mag_jumps.ulg — Yandex Disk

Firmware version: PX4 v1.10. Hardware: Pixracer; I use one external magnetometer (HERE2) and turn off the internal one.
These jumps are followed by the message
[sensors] MAG #0 fail: STALE!. During the “jumps”, mag indeed outputs a fixed value by all 3 axis:

I know that I haven’t provided much data nor steps to reproduce, and also this “bug” is very subtle. I am creating this post in hope that maybe someone has encountered the same issue. Any advice/questions are welcome.

Thanks for sharing. This does not look good. How did you configure the Here2? Is it used over I2C or UAVCAN?

Here2 connected over I2C. UAVCAN was disabled.

I have seen spikes with Here 2 over I2C but never timeouts like this. Have you checked that the I2C lines are not too long and not close to other wiring? Does it happen when you move the cables in a way?

In general they recommend to use the Here 2 over UAVCAN but I think this issue should tracked down nevertheless.

Some more data points that would be interesting:

  • Does it happen on other hardware, e.g. not with a Pixracer but something else?
  • Does anything else in the log file have these pauses?
  • Does it happen with other Here 2 hardware, or do you only have one?
  • Does it still happen with a fast (more expensive) SD card? (There could be some other effects be happening.)

Could you upload that log to http://logs.px4.io please, so I can have a look? Thx.

Log file: https://logs.px4.io/plot_app?log=f0e45222-df3a-44b3-9fd5-a11c9047c9f6

This will certainly require more time to investigate and answer all of your questions. I will keep this updated.

Thx. Alternatively, you could try 1.11 beta and see if it is fixed. There have been quite some changes to these drivers, so maybe you’ll be lucky.

I continue to see the these glitches. Some of them are П-shape like on the plots above, followed by MAG #0 fail: STALE! message, and some look like pikes:

Comparing the x-axis data to the corresponding INTERNAL mag:

@CarlOlsson can you please specify the model of magnetometer you use (is it here2?), do you connect it over I2C or UAVCAN and what is the voltage on your board?

This was HMC5883 over i2c

I have the same behavior.
My configuration is Pixracer R15, PX4-1.8.2, LIS3MDL internal magnetometer is used for priority and no external magnetometer is used.

While flying: Strange magnetometer surges happen! and errrormessage “MAG #0 fail: STALE!”
Then switch to another mag:
2:14:834 WARNING [sensors] Remaining sensors after failover event 0: Mag #0 priority: 1
2:14:834 WARNING [sensors] Remaining sensors after failover event 0: Mag #1 priority: 50

Did anyone handle this, If there are some bugs in the firmware driver please let me know.

PX4 v1.8 is very old. You should update to v1.13. Likely this problem has been resolved in the meantime.

I’m running PX4 (1.13.0 and 1.13.2) with PixHawk Cube 2.1 Black w/ Here 2 GPS and I’m still getting magnetomter failed message. Its hit and miss. I have to do several reboots to get good measurement from magnetometer. But if it starts with good readings, it stays good as long a power remains on.

Are you using CAN and if so, is it connected to CAN1? In that case, try using CAN2 instead.

There is a problem with the board auto-detection when something is connected to CAN1.

I currently have nothing connected to CAN1. I have ESC and UACAN adapter (Babel) connected throguh CAN2.
I’m using Here2 GPS (ICM20948 and MS5611) connected with standard 8-pin cable (Serial3 and I2C1) through a custom interface board. Is MAG #0 data from ICM20948? When I execute the "icm20948 status at MAVLink Console, I get “INFO [SPI_I2C] Not running” message. I do seem to have xmag, ymag, and zmag reading throught SCALED_IMU and SCALED_IMU2 (both FC and GPS installed in their default orientation with no rotation) but they don’t match. xacc,yacc, zacc do for both IMUs match.
Is there an easy way to deteremine which sensors and mapped to the various data in PX4 and through which interface?

Thanks,
Ben


You would have to select the correct bus with this command.

I’m not certain how to exactly get a nice mapping. That’s indeed a bit tricky.

What’s the output of sensors status that could sometimes tell you which device on which bus is published where.

Another thing to try is dmesg which is enabled from some boards that gives you the startup output.