Cube O+/PX4 v1.15.2 erroneous angular rates setpoints & motor output after switching from offboard to mission

[ZiyiYY]

Overview

We conduct some tests with the model “Sky Fury,” a VTOL using PX4, in fixed-wing Offboard mode.

Our experimental process is as follows, going through Mission mode - Offboard mode - Mission mode:

• Upload a mission plan to QGC to enable the VTOL to complete vertical takeoff (quadrotor configuration)- configuration conversion - waypoints tracking (fixed-wing configuration) in Mission mode.
• After reaching waypoint #7, our testing program controls the VTOL to enter the Offboard mode, and the VTOL completes the test according to the program commands.
• After the test is completed, the program controlled the VTOL to re-enter the Mission mode. In the mission mode, the VTOL continues the unfinished waypoints tracking and landing process.

We completed two experiments according to the above process. One of the flights was successful. In another flight, after the VTOL completed the test in Offboard mode and switched to Mission mode, the VTOL suddenly lowered its head and dived, crashing. We attempted to analyze the cause of the crash based on the log data, but discovered strange phenomena. We hope someone can help us take a look together.

Test Specifications

• Model: Sky Fury

• Configuration: Standard VTOL

• Controller: CubePilot Cube Orange+

• Firmware: PX4 v1.15.2 (self-compiled with adding vehicle_command and mission_result topics in dds_topics.yaml)

• Computer OS: Ubuntu 20.04

• ROS Version: ROS2 Foxy

Crash phenomenon

We noticed irregular control commands after mode switch (Offboard → Mission):

• Constant motor 5 output.

• Disappeared roll rate setpoint.

Crash analysis

Ulog analysis

• Flight Review about the Successful Flight- VTOL Standard
• Flight Review about the Crash - VTOL Standard

Limitations of plotting with FlightReview & PlotJuggler

• Lines disappeared when value becomes NaN → hard to tell the value stopped updating or became NaN

• Separated plots → hard to align timestamps to track event order

Considering the above limitations, we have processed the flight log as follows

• Using ulog2csv tool to convert .ulg flight log to .csv files.

• Each uORB message inside PX4 is related to (at least) one .csv file.

Examine NaN values in raw data around mode switch (offboard → mission)

For the crash, we noticed irregular attitude & rates setpoint after mode switch in Flight Review. We attempted to look for NaN values in .csv files.

• Noticed NaN pitch setpoint in vehicle_attitude_setpoint_0.csv.

• Search for all NaN values around the time in .csv files.

• Sort by timestamps.

• First NaN occured in tecs_status_0.csv.

Combination of first three frame of tecs_status_0.csv after mode switch (Offboard → Mission)

In the table below, each row shows the first three frames of tecs_status_0.csv data immediately after the mode switch event. Some data fields show NaN all the time, but some data fields turn into NaN only in crashed flights.

A	B	C	D	E	F	G	H	I	J	K	L	M	N	O	P	Q	R	S	T	U	V	W	X
Flight	timestamp (ms)	pitch_integ	pitch_sp_rad	altitude_sp	altitude_reference	height_rate_reference	height_rate_direct	height_rate_setpoint	height_rate	equivalent_airspeed_sp	true_airspeed_sp	true_airspeed_filtered	true_airspeed_derivative_sp	true_airspeed_derivative	true_airspeed_derivative_raw	total_energy_rate_sp	total_energy_rate	total_energy_balance_rate_sp	total_energy_balance_rate	throttle_integ	throttle_sp	throttle_trim	underspeed_ratio
1(Normal)	816907408【Exit offboard】	0	0.041738346	nan	161.29286	4.4085965	nan	1.322579	4.4085965	nan	nan	24.293663	0	0	0	12.972863	43.233562	12.97007	43.233562	0	0.69807917	0.61916125	0
	816927409	-0.00023351247	0.047500014	114.96299	161.38004	4.2931247	nan	1.2873667	4.4348574	23.579952	23.66699	24.293663	-0.12533455	0	0	9.582713	43.245823	15.66959	43.491096	-0.00019434083	0.6780832	0.6191608	0
	817107399	-0.0026750497	0.02566637	114.96299	162.03934	3.0331945	nan	0.87527823	4.5128117	23.759949	23.84919	24.21053	-0.07226791	-0.08757169	0	6.8600817	43.337265	10.333192	46.37572	-0.0020307153	0.64412755	0.6192113	0
2(Crash)	634419561【Exit offboard】	nan	nan	nan	175.30899	0	nan	-0.19228211	0.24917519	nan	nan	20.50453	nan	0.1495693	0	nan	13.670247	nan	-0.6232743	0	nan	0.61651033	0
	634619600	nan	nan	nan	175.30899	0	nan	-0.17947999	-0.5922527	nan	nan	20.721893	nan	0.32302478	0	nan	9.40862	nan	-12.501699	0	nan	0.6165206	0
	635419517	nan	nan	nan	175.30899	0	nan	0.08907776	-3.866304	nan	nan	22.462873	nan	1.2745875	0	nan	-3.0895734	nan	-66.54639	0	nan	0.6165052	0

Observation of Post-Mode-Switch Data

For these 2 flight logs, we looked into every uORB message log after mode switch, trying to find irregular NaN values.

vehicle_attitude_setpoint after mode switch

Flight	q_d[0] NaN Rate	q_d[1] NaN Rate	q_d[2] NaN Rate	q_d[3] NaN Rate	pitch_body NaN Rate
1(Normal)	<1%	<1%	<1%	<1%	<1%
2(Crash)	100%	100%	100%	100%	100%

tecs_status after mode switch

Flight	height_rate_direct NaN Rate	pitch_integ NaN Rate	pitch_sp_rad NaN Rate
1(Normal)	100%	0%	0%
2(Crash)	100%	100%	100%

position_controller_status after mode switch

Flight	nav_pitch NaN Rate	yaw_acceptance NaN Rate
1(Normal)	0%	100%
2(Crash)	100%	100%

Observed Crash-Flight Anomalies

• vehicle_attitude_setpoint: Crash flights show 84.6-100% NaN in q_d[0-3] and pitch_body

• tecs_status.pitch_integ: Crash flights show 98.4-100% NaN, normal flights show 0%

• tecs_status.pitch_sp_rad: Crash flights show 98.4-100% NaN, normal flights show 0%

• position_controller_status.nav_pitch: Crash flights show 96.3-100% NaN, normal flights show 0%

Observation by far

• Possible direct cause of crashes: pitch_integ becomes NaN eventually.

• In normal flights:

  • at the first frame after leaving offboard:

    • NaN occurred

    • pitch_integ and throttle_integ are 0

  • normal values after

• Crash happened when pitch_integ or throttle_integ not 0 at the first frame after leaving offboard.

Confusion

• There was no difference between these two flight tests, but only one of them led to a crash.

• For now we only observed irregular values (NaN) in some data when crash happened, but could not figure out the reason.

• The phenomenon seems to imply an internal bug of PX4 when switched back from Offboard to Mission, however there is no clear evidence to show that what triggered the bug.

Another crash caused by mode switch(Offboard → Mission)

In our previous experiment, there was also a crash due to mode switch. We also analyzed the flight logs and found the same problem as described in this post.

During Mission Mode the VTOL Disintegrated after a Large Roll maneuver

We have not yet identified the essential cause of the crashes. We welcome anyone to further discuss this issue with us and offer some assistance.