First, thank you all for your time. Second, I hope that someone help me with an issue that I’m seeing that has me stumped. I have now recreated this issue on 3 airframes and two Cube Orange+ flight computers.
The issue I’m seeing is that when the vehicle is switch from a “manual mode” (e.g. stabilized) to an autonomous mode (e.g. loiter), the flight controller is sending very large and erroneous pitch and roll setpoints. Below are the events that have happened.
After the Opterra flight we re-accomplished all of the calibrations and tried to recreated the issue again, but weren’t able to. Separately we also used a different Cube O+ on a Rascal fixed wing aircraft to try to recreate the issue, but weren’t able to. We assumed the original issue must have been bad calibrations, and moved on. Then on Friday (24Nov23) we flew the Rascal again and the issue returned, this time also resulting in a crash:
I’m happy to provide any additional information you need.
I sincerely hope you can help with the issues we are seeing. PX4 has been a great resource for us, but these issues with the Cube Orange+ and v1.14.0 are hurting our confidence for continuing to conduct research.
I had a look at the logs and tried to make sense of what I see:
28Oct23
You switch to mission but then immediately after to Manual, presumably because the sticks are moved which triggers Manual. Any idea whether and why this taking over happened?
I’ll try to help clear up some confusion. I was generally referring to “manual” modes as modes where the pilot still has control (e.g. manual, stabilized, acro, etc.) and “autonomous” modes as those where the pilot is out of the loop (loiter, mission, etc.)
28Oct23 flight: I flew in Stabilized mode and then switched to Mission mode via the RC controller. As soon as I switched to Mission mode (~6:20) the aircraft abruptly pitched/rolled toward the ground. At this point I quickly switched to Manual mode to recover, but did not have enough altitude.
03Nov23 flight: The switch to Hold/Loiter mode (via the GCS) was very brief at ~2:25 to see if we could recreate the issue, which we did. The rest of the flight was done in Stabilized/Manual mode with no issues.
24Nov23 flight: The Geofence Failsafe Trigger was set to Return Mode. The Return to Launch Settings were “Loiter and Land after 120s”. The pilot did not initiate the mode switch to Loiter…it was triggered by the geofence breach. In the logs it looks like the geofence breach preceded the switch to Loiter, but they happened simultaneously and automatically.
Our suspicion is that the issue lies in the TECS or the FixedWingPositionController. We have not seen this issue in v1.13.
I’m interested in helping you address the root cause because I’m about to test the Cube Orange+ on my VTOL. I had no idea what happened or why, but I was aware that some setup could accelerate the situation into a catastrophic failure.
Regarding the Flying Wing log file, I found that the vibration is quite high, it might have come from the airframe or inappropriate tuning of PID controller gain and cutoff frequency.
Regarding the PID controller gain, is the UAV well-tuned?
No airspeed sensor installed onboard, why? I experienced flying UAV in high wind and the overall performance is not quite good if no airspeed sensor.
Better reduce the limit of these parameters? after you prove that it all works, try to increase it again as needed.
FW_P_LIM_MAX, 30.0 → 20
FW_P_LIM_MIN, -30.0 → -15
FW_R_LIM, 50.0 → 40
FW_R_RMAX, 70.0 → 55
Have you tried to fly the UAV in Altitude and Position before? What about its behaviors?
The attached image shows the pitch angle and pitch rate, it looks so weird that the pitch angle setpoint is a positive sign (FC tried to pitch up) but the pitch rate setpoint is a negative sign (tried to pitch down).
Thank you for helping to solve this issue. Can you tell me which specific log you pulled that screenshot from? There are a lot of logs on that link and I couldn’t find that particular one.
Lol…didn’t realize that was from my own log, sorry. Yes, the conflict between the pitch setpoint sign and pitch rate setpoint sign doesn’t make sense.
To address your other points:
Yes, the vibrations were high on the home-built flying wing (likely due to the Cube being mounted fairly close to a large prop). I thought this might have been the cause, but when the issue was seen on the Opterra flying wing and the Rascal conventional fixed wing, I don’t think vibes are the root cause.
We don’t have an airspeed sensor installed because, using standard Cube Oranges and V1.13, we never had any issues flying in manual or autonomous without one. We planned to incorporate airspeed later, but then encountered this issue.
We felt that the UAVs were well-tuned. I probably should have made this more clear in the initial post, but after the 03Nov23 incident when we re-accomplished the calibration we flew the Opterra and Rascal in several modes (Auto-Takeoff, Stabilized, Mission, Hold, Auto-land) without problem. The logs all looked normal and both aircraft flew fine.
Then we flew the Rascal again on 24Nov23 (hadn’t changed anything since the successful flights) and the issue popped back up resulting in a crash. I’m reluctant to try to “solve” the problem by limiting the pitch and roll parameters because a) we didn’t have to do this with v1.13 and b) from the flight logs the FC isn’t honoring the limits as they currently are anyway.
I sincerely thank you for your help. I am personally at a dead end trying to figure it out, especially since it seems unpredictable (like I said, we had several “checkout” flights where the issue didn’t come back up before this most recent incident).
I’m happy to provide any information that could help (log files, param files, other equipment information, etc.)
Yes I was made aware of this issue yesterday and am currently debugging it. What I see so far points towards an issue in/interfacing with the altitude/airspeed controller (TECS).
Symptoms:
pitch setpoint is at -5 rad (which is the default min pitch of TECS) → it seems that it was not updated with the correct min pitch as set on this airframe
airspeed_filtered is updated even though we don’t have any airspeed sensor input into the filter
TECS goes into “underspeed mode”, even though that should only be possible when an airspeed sensor is present
As usual it was a chain of bugs that lead to the crash, in your case the combination of flying airspeed-less (which is not as well tested as when flying with airspeed), and not switching into any altitude controlled flight mode briefly while still on the ground.
Absolutely…I would be happy to contribute to the community. Thank you again for working on this fix so quickly.
If you don’t mind, I’ll follow up to see what steps I should try exactly that would re-create this issue if it were still there (i.e. do I need to NOT switch to an altitude controlled mode while on the ground).
@C_G We’ve merged the fix into the release 1.14. In case you want to test that version. I can also send you a built binary if you can’t build PX4 yourself.
