Unexpected and dangerous behaviours after going in & out of Offboard mode

Overview

In our tests recently, model Sky Fury, a VTOL with PX4, performed unexpected and dangerous behaviours after getting out of Offboard mode. A Sky Fury crashed and another Sky Fury turned almost upside down (but finally landed safely by manual controller).

The most weird thing is that, in both flights, weird behaviours of Sky Fury only happened outside of Offboard mode when no scripts were running.

I hope the community could share some thoughts on this. Thanks!

Test Specifications

Test Procedure

Both of the two flights were supposed to test Switching between Mission Mode and Offboard Mode. The procedure is as the figure below depicts.

Before takeoff, a plan of several waypoints is uploaded through QGroundControl.

The script running on the companion computer:

  1. starts before takeoff;
  2. waits until the drone reaches a certain waypoint (#1 for example in the figure);
  3. sends a signal of switching into Offboard mode;
  4. controls the drone in body rate / thrust command (0 in roll & pitch & yaw rate, 0.3 in thrust) for 1 second;
  5. sends a signal of switching back to Mission mode;
  6. sends a signal of switching into multi-copter (MC) configuration;
  7. waits for transition into multi-copter configuration to be completed;
  8. sends a signal of switching into fixed-wing (FW) configuration;
  9. waits for transition into fixed-wing configuration to be completed;
  10. exits.

Flight #1

The first attempt of this test caused a crash. After successfully switched into the offboard mode, the script encountered an error and exited. All attempts of guiding the drone to land by QGC failed.

  • The drone wouldn’t head for waypoint #5 as it was supposed to after switching back to Mission mode by QGC.
  • The drone wouldn’t loiter at the point of QGC’s command.
  • The drone was constantly decreasing altitude (but didn’t seem like stall).

Mission Plan

Complete Trajectory

QGC Screen Recording with Onboard Computer

  • Detailed Record

    ID for Reference Operation / Behaviour Step of Script Process Timestamp In Video
    1-1 The script started Step 1 0:01
    1-2 The drone takeoff-ed and followed the mission plan Step 2 0:11
    1-3 When the drone reached waypoint #4, the script successfully sent the signal to switch into Offboard mode Step 3 1:56
    1-4 The script encountered an error and exited Step 4 1:56
    1-5 The drone switched into Position mode immediately 1:57
    1-6 QGC controller switched the drone into Mission mode to let it continue the mission 2:10
    1-7 The drone didn’t follow the direction of waypoint #5 2:20
    1-8 QGC controller selected a point on the map and attempted to guide the drone to loiter around this point 2:30
    1-9 The drone switched into Hold mode, and didn’t go for the correct direction; meanwhile, the drone was decreasing altitude and gaining airspeed to more than 30m/s 2:39
    1-10 The drone kept flying in a circle with ~300m radius while constantly decreasing altitude 2:55
    1-11 The drone crashed 3:28

Confusion

  • [1-7] Why the drone didn’t head for waypoint #5?
  • [1-8, 1-9] Why the drone didn’t head for the selected point and began to lose altitude?

Flight #2

The second attempt was successful when:

  • The script sent the command into Offboard mode.
  • Kept sending 0 ypr rate & 0.3 thrust command for 1 second.
  • Sent the command back into Mission mode.
  • Sent the command of transition to MC.
  • Wait for the drone to slow down and finish transition.
  • Sent the command of transition to FW.
  • Wait for the drone to accelerate and finish transition

Then the pilot took control of the drone just before the transition was completed. ****The operations afterwards:

  • The pilot switched into Mission mode again.
  • The drone headed for waypoint #6 in MC.
  • QGC guided the drone to a selected point then switched into Mission mode.
  • The pilot switched into FW in Hold mode.
  • The drone reached out of manual comm range and triggered RTL.
  • QGC guided the drone to a point near home.
  • The drone reached the orbit.

Then the drone had some unexpected and dangerous behaviours.

  • The drone kept losing altitude and speed, seemed like stalling.
  • QGC reset altitude setpoint, attempting to increase altitude.
  • The drone kept losing altitude, then after a serval seconds, almost rolled upside down.
  • Quad-chute was triggered.
  • The pilot took over and landed.

Mission Plan

Complete Trajectory

QGC Screen Recording with Onboard Computer

Detailed Record

ID for Reference Operation / Behaviour Step of Script Process Timestamp In Video
2-1 The script started Step 1 0:00
2-2 The drone takeoff-ed and followed the mission plan Step 2 0:21
2-3 When the drone reached waypoint #5, the script successfully sent the signal and the drone switched into Offboard mode Step 3 3:04
2-4 The script controlled the drone for 1 second Step 4 3:04
2-5 The script sent the signal to switch into Mission mode Step 5 3:05
2-6 The drone re-entered Mission mode successfully 3:05
2-7 The script sent the signal to switch into multi-copter (MC) configuration Step 6 3:06
2-8 The drone took ~8 seconds to decrease speed and finished transition into multi-copter configuration Step 7 3:13
2-9 The script sent the signal and attempted to switch the drone into fixed-wing (FW) configuration Step 8 3:14
2-10 The drone accelerated, almost finished transition Step 9 3:17
2-11 Manual controller took control and switched into Stabilised mode, then Position mode 3:19
Manual controller switched into Mission mode 3:23
2-11 The drone decelerated while approaching waypoint #6 3:27
2-12 After reaching waypoint #6, the drone switched into Position mode 3:43
2-13 QGC controller let the drone re-entered Mission mode and headed for waypoint #7 3:52
2-14 QGC controller guided the drone to a selected point on the map, the drone entered Hold mode 4:28
2-15 Manual controller switched into fixed-wing configuration 4:45
2-16 The drone accelerated and transitioned successfully into fixed-wing configuration in Position mode 4:51
2-17 The drone exceeded manual control range and triggered RTL 4:56
2-18 QGC controller guided the drone to a point closer to takeoff point 5:10
2-19 The drone switched into Hold mode 5:10
2-20 After reached the orbit, QGC controller noticed that the drone was losing airspeed and altitude, seems like stalling 5:31
2-21 QGC controller re-selected altitude setpoint to be higher 5:44
2-22 The drone failed to re-gain altitude, and after a few seconds, the drone suddenly rolled over 5:53
2-23 Quad-chute emergency was triggered and the drone switched to multi-copter configuration 5:54
2-24 Manual controller took control and landed the drone 6:03

Confusion

  • [2-20, 2-21, 2-22] Why the drone lost height & speed and rolled almost over?

Investigation and Reproduction of the Issue

We investigate whether the behaviours was caused by the PX4 and QGC versions. Additionally, after the VTOL enters POSCTL mode, does using different guidance methods lead to a crash?

We tested various PX4 version(1.14.2, 1.15.2) and QGC(4.4.4, 4.4.0) versions. In each version, multiple guidance methods were also tested(MISSION, HOLD-MC-FW-MISSION, HOLD-MISSION, MC-FW-MISSION).

Ultimately, we found that crashes only occurred in PX4 version 1.15.2 and QGC version 4.4.4 and the operation is switching VTOL in MISSION directly.

Number Result QGC Version PX4 Firmware Version QGC Operation after POSCTL mode
1 Crash 4.4.4 1.15.2 MISSION
2 Continue Mission 4.4.4 1.15.2 HOLD-MC-FW-MISSION
3 Continue Mission 4.4.4 1.15.2 HOLD-MISSION
4 Continue Mission 4.4.4 1.15.2 MC-FW-MISSION
5 Continue Mission 4.4.4 1.14.2 MISSION
6 Continue Mission 4.4.4 1.14.2 HOLD-MC-FW-MISSION
7 Continue Mission 4.4.4 1.14.2 HOLD-MISSION
8 Continue Mission 4.4.4 1.14.2 MC-FW-MISSION
9 Continue Mission 4.4.0 1.14.2 MISSION
10 Continue Mission 4.4.0 1.14.2 HOLD-MC-FW-MISSION
11 Continue Mission 4.4.0 1.14.2 HOLD-MISSION
12 Continue Mission 4.4.0 1.14.2 MC-FW-MISSION
13 Continue Mission 4.4.0 1.15.2 Wait for a while-MISSION
14 Continue Mission 4.4.0 1.15.2 Immediately-MISSION
15 Continue Mission 4.4.0 1.15.2 MISSION
16 Continue Mission 4.4.0 1.15.2 HOLD-MC-FW-MISSION
17 Continue Mission 4.4.0 1.15.2 HOLD-MISSION
18 Continue Mission 4.4.0 1.15.2 MC-FW-MISSION

We frequently encountered errors when uploading flight paths to the QGC. We wanted to determine whether the crashes were caused by this issue.

Number Result QGC Version PX4 Firmware Version Operation of Plan Uploading in QGC
1 Crash 4.4.4 1.15.2 "QGC already has a path
2 Crash 4.4.4 1.15.2 "QGC already has a path. After uploading a new path
3 Crash 4.4.4 1.15.2 QGC already has a path. Without uploading a new path, slide the control directly to take off.
4 Continue Mission 4.4.4 1.15.2 "QGC already has a path. After uploading a new path
5 Continue Mission 4.4.4 1.15.2 Upload the path, then start the program when NOT READY is displayed in red.
6 Continue Mission 4.4.4 1.15.2 QGC already has a path. After uploading a new path, slide the control to take off.
7 Continue Mission 4.4.4 1.15.2 QGC has no path. After uploading a new path, slide the control to take off.
8 Continue Mission 4.4.4 1.15.2 "Start the program first. Halfway through

After testing various scenes, we found that crashes mainly occurred when the plan was not re-uploaded.

Consistent QGC ground station operations to avoid bug:

Try sliding to switch back to Mission. Maintain the same program breakpoint positions. Use PX4 version 1.15.2, QGC version 4.4.4. Ensure QGC remains Disconnected before starting simulation and script.

Number Result QGC Version PX4 Firmware Version
1 Continue Mission 4.4.4 1.15.2
2 Continue Mission 4.4.4 1.15.2
3 Continue Mission 4.4.4 1.15.2

No crashes occurred in the simulation. However, we cannot guarantee that crashes will not happen in real-world experiments.

Hello,

Thank you for posting a detailed response. However, it would be best if you post your log files so we can see what happened

Hi, @AWilkins_Ascend, thanks for reply! For log files:

  • Flight #1: We failed to collect any log files from SD card after crash. The list of log files was just empty.
  • Flight #2: 09_15_53.ulg - Google Drive

Hello all!

We have summarised the things happened in simulation in VTOL Descent and Crash in Mission Mode after Attempt to Switching into Offboard Mode, where we tried to reproduce the issue in Gazebo Classic to investigate. This is a much shorter post, and I think the issues are pretty much the same.