Unintended mode change caused a crash

🧪 Flight Testing & Log Analysis
1

Hi everyone!

I was flying with a private label S500 variant equipped with PX4. I was only using mission flight mode to execute brief speed test tasks. However, on my last flight, the drone suddenly switched to manual mode, although I wasn’t even holding the RC, so it was not a mistake a misbehaved finger made. As you can guess, there was no thrust after that moment, and I encountered a freefall. So the question I could not solve is, why the heck is this device entered into manual mode in the middle of a mission?

I attached the logs here: https://review.px4.io/plot_app?log=336d0a61-ba15-4fd2-8c1b-abf4ccf8af24

Appreciate any bit of help. And wish y’all a great holiday.

2

Sorry to hear about the crash!

So far I see two things:

  1. In the Logged Messages (at the bottom in flight review): you see multiple RC regained and lost messages. So clearly some sort of manual control was arriving, at least sometimes. This might be a hint.

  2. Then, it seems like throttle goes to 0.5 randomly, and then back to 0. Not sure if that triggered it, or if it has nothing to do.

bokeh_plot
bokeh_plot1260Ă—504 33.7 KB

Do you have any idea what causes the manual control regained or lost? Can you try to reproduce that on the bench?

2 Likes
3

Hi Julian, thanks for the quick response.

The RC was on and connected all the time for safety purposes, i.e., to have access to RTL and kill switches in case of an emergency. But it was not used during this flight at all. I have no idea why there is a thrust in that chart.

I have experienced occasional temporal RC losses in my previous flights as well, but in those cases, the drone followed whatever the default safety settings were, like switching to hold mode or return mode. Yet changing the RC signal loss timeout from 0.5 s to 4 s in QGC allowed the drone to continue its ongoing mission even if it experienced such signal losses. So, switching to manual mode was a total surprise.

What caused those temporal RC signal losses in such a short distance (~250 m) is another mystery. My best guess is the radio interference between the DSMX AR8000 transceivers and the WiFi that I used to track the mission’s progress in QGC on my laptop.

Best.

4

Oh sorry, I misunderstood you then. I thought you had no RC.

Have you tested what sort of signals are sent (or rather received) when there is a signal loss? It’s important to do that on the bench to know whether a signal loss is correctly detected. What you don’t want is that the receiver sends some sort of failsafe values that you dont expect while PX4 does not know it’s in RC failsafe.

Also see: Safety Configuration (Failsafes) | PX4 User Guide

I had a quick look in PlotJuggler:

RC status
RC status1200Ă—900 22.4 KB

Something is not right.

RC input is actually always “not lost”, so presumably ok, and the manual_control_setpoint.data_source is 1 which is RC (see PX4-Autopilot/ManualControlSetpoint.msg at main · PX4/PX4-Autopilot · GitHub).

However, it suddenly switches to 3 which is MAVLink instance 1. This means that some MAVLink instance is sending manual control input.

Could it be that you had a joystick or gamepad configured and connected to the computer? It seems to be switching to that for a bit, and then back to RC.

2022-12-24_18-00
2022-12-24_18-001329Ă—1585 256 KB

Once it switches back to RC input, it detects that the sticks have been moved, as it says “Pilot took over position control using sticks”.

Strangely not all events show up in logs.px4.io but they do in PlotJuggler, not sure why. (@bkueng could you check why some events don’t show up in Flight Review?)

The two things that seem odd to me are:

  1. Why did it register this as “pilot took over control”? It shouldn’t do so because the RC controls have not moved as such.
  2. Why did it switch to manual mode? It should go to position or altitude first, as that’s safer (and doesn’t drop from the sky with throttle at 0).

One more thing: you have changed the COM_RC_IN_MODE param, I believe. This param controls how it switches between RC and MAVLink input.
If you use option: RC only, MAVLink only, or keep first, you are much safer against these sorts of problems because it won’t switch mid flight.

1 Like
5

Dear Julian, thank you very much for such a detailed breakdown.

I still do not exactly know why the drone switched to manual mode without any prompt. But, after reading your comments about another MAVLink instance messing with the drone and a potential connection with a gamepad, I suddenly recall that I have previously enabled the virtual on-screen controller on QGC (my laptop was touchscreen) for simulation purposes. I do not use it anymore, but as it was enabled, I may have touched that part of the screen by mistake. Does this make any sense? May this be the cause?

Does this also explain why the logs (and QGC during the mission) state that the RC connection was lost for a while during such a short-distance flight? – Because the virtual controller (on the laptop) was connected to the drone through WiFi, not a long-range radio. If this was the case, then no mischievous fingers might have ever been involved in the situation, but instead, losing the virtual controller’s connection to the drone, which is actually WiFi, and regaining it later may have caused such an odd behavior of changing the flight mode to manual.

Best.

6

Yes, that sounds indeed likely to have been the culprit!

1 Like
7

I didn’t check specifically, but generally these are messages changed in main. Meanwhile the metadata is also in the log.

8

Wouldn’t the ID get bumped if they are changed? Or only on release?

9

What do you mean exactly? The event ID’s are derived from the event names.

10

Oh I guess I’m confused. So where do the strings come from? Are they baked into the logfile?

11

Yes now they are. But not in the log above. Flight review previously used the metadata from main, pushed by CI.

1 Like