PX4 crash when switching to HOLD mode

We had a crash in HOLD mode with the drone going slowly down while flying around 2 meters from ground. From my analysis, height estimation is good (no GPS lost, no height estimation failed). The crash happent at 12:26:606. The offboard mode before gave a consign to the drone to go up all the time but as soon as we switch to HOLD mode, velocity setpoint in Z request to go down and the crash happen…

Any helps on that behaviour would be greatly appreciated !

The PX4 version is v1.10.1.

Here is the log:
https://logs.px4.io/plot_app?log=1aebd732-aee6-4a5a-827b-d5d76c899a25

Could it be related to this 1 second where nothing is logged just before the crash ?

I’m confused because thrust and the actuator_outputs are clearly going up at 12:26.

It looks like you get some kind of bad resonance in your frame at 12:26 moment. Vibrations became very high.
I saw this in our prototypes: when it vibrates high, it starts to go down.
Check your drone that way: fly carefully in stabilize mode right in front of you and rize throttle slowly. Look at the vibrations and drone’s behaivour in them.

This is the mystery which I can’t solve… :sweat_smile:

Velocity setpoint in z request to go up, manual control input request to go up, thrust and actuator output are going up (there is a small peak down for the actuator just before to switch to loiter mode which I don’t explain), but the drone is going down…

you’re talking about the bad resonance in the vibration metrics graph ?

IMO this vibration comes from the drone touching the vineyard at 12:26:606.

@JulianOes @kpetrykin Another thing I saw from the 3D view. There is a small glitch in the animation just before the drone starts the descent (at 12:25:200). Is it something you already notice in other flights ?

Thank you for taking the time to have a look to my issue !

When thrust goes up? Did it already crash/touch the ground? We should be able to see that from when the accel spikes.

Before the ground, there is maybe 1.5m of vineyard pole. The first peak of the accelerometer is at 12:26:500 where one of the propeller touches the vineyard. So there is 1 second where the thrust goes up before the crash but the drone did not respond to that command. When the thrust are at 100% at least one of the propeller is already in the vineyard.

I’m working with MAVSDK. A possibility could be that when I leave the offboard mode, the last consign before to leave is to go down, the drone might keep that consign until it properly switches to the HOLD mode. I’ll try to catch this behavior in simulation to see what’s happening

1 Like

There is Z position setpoint goes down right before the crash (this graph is upside-down)

@kpetrykin right that confused me too. It basically follows the estimation, until then it suddenly recovers.

The local position in Z is reversed… I did not pay attention to that before :sweat_smile:

So if I understand correctly, when it switches from offboard to hold mode, the cpu freezes (or something is blocked) and estimation of the Z is not updated until it restarts the estimation again ?

If so, I saw this behavior in multiple of my logs. In this one nothing happen during 1 second from 4:43 to 4:44 and I’m not switching between modes at this moment

@Katawan what MicroSD card are you using?

We are working with the Pixhawk 4 with the default SD card Sandisk edge 8gb micro sd

Ok, I suggest to use a faster/more expensive card, and not the default cheap/garbage card. They show big worst case delays and currently PX4 doesn’t cope well with that.

I’m not saying that was necessarily the reason but it might be.

Ok I will. Even if it’s not the reason, the goal is to have a pro version so I’ll go with the best

Probably anything decent from a brand will do just fine :confused:.

I just thought that it could be just a barometer fluctuations (if you don’t use any other height source) :thinking:

I have GPS as primary source for height and we have a RTK (9P) which did not failed during all the flight. Possibility would have been that something goes wrong because we lost multiple GPS before the crash…

I guess barometer is still used by EKF and we were close from trees and the ground so it could be a sudden change in the pressure :thinking:

Reviving this thread as I’m helping @Katawan fix this and other issues and also because it has happened again.

https://logs.px4.io/plot_app?log=fe20676a-5e55-4332-9f4d-c214b291dd03

It maintains the same behaviour of the drone going down because it indeed wants to go down, then corrects it eventually. both the z and z vel match the setpoints. This also happens while in Hold mode, not in the last timestamps of Offboard so it seems that something is wrong here. Even if the last message sent by offboard was not a good one, Loiter should not have this behaviour.

Any tips on where to look into or tests to do?

For the reference here is a similar log where this problem did NOT happen:

https://logs.px4.io/plot_app?log=9717a42a-50b1-4200-a940-f58fa123da0d

Thanks in advance @kpetrykin and @JulianOes