I am using a Pixhawk 2.4.8 flight controller running PX4 v1.8.2 on an agriculture drone custom-built. Position and stabilize mode both work great (have been tuning both of these modes over the past couple weeks). This drone was built for a collegiate student group participating in a competition, so we have multiple people on our team looking at this issue.
Unfortunately, our team experienced a crash today when testing out a mission. I uploaded a mission to my Pixhawk via USB, restarted the system, and took off in stabilized flight mode. After getting into the air, I switched to position mode to hold position. Everything looked fine, so I switched into Mission mode to execute my preloaded mission. The flight controller then immediate went into flight termination, causing the drone to crash. Our pilot attempted to switch into stabilize mode after the drone began falling, but unfortunately flight_termination had already been triggered and there was no way to recover by switching states.
Some things to note:
A datalink was not used. This definitely was a mistake on our part, but I did not realize that flying without a datalink in mission mode can lead to flight termination, especially when the RC controller is still available (which should, at the minimum, be able to recover from this sort of error, instead of having the controller enter flight termination)
“DL and GPS lost: flight termination” shows up at 02:39 in the log. However, it does look like we had at least a GPS fix throughout the duration of this flight, with under 5m horizontal/vertical uncertainty and more than 10 satellites visible.
Pilot takeover occurred at 2:40, which did not cause the motors to spin up since the drone went into flight termination
The jamming indicator is higher than we’d like, although it is below 80. The help guide for Flight Review states that a value around or above 80 is dangerous, although I do not know if the PX4 would necessarily interpret this as a loss of GPS.
Our platform does have high vibrations, which we are working to mitigate. However, this seems unrelated to the crash since the drone does fly fine in stabilize/POSHOLD mode.
I’ve done some digging around in the Firmware code and found the following block that caused this crash to occur. From what I understand, this block checks to see if an AUTO mode like mission is being used, and flight terminates if the datalink is lost. This is supposed to happen when the GPS is also lost, though I do not see where the check for GPS is made.
Did the PX4 Firmware act as intended in this situation, or did Commander not check for a GPS fix when also checking for a datalink loss? Would it be possible to disable this specific flight termination case without disabling flight termination via RC switch or Mavlink?
This is my first post on this forum, so please also let me know if any more information is needed or whether this should be posted elsewhere for better support. Our team is working on fixing up the drone and adding vibration isolation and gel pads to our Pixhawk controller and motor mounts to better mitigate the vibration issue. This is also our first crash using PX4, so any tips on how to work with the SITL simulator or documentation online we may have missed that would be relevant to avoiding an issue with AUTO mode in the future.
Sorry to hear about your crash! And your first post is great, plenty of information, a log, and even a video! This is helpful.
By default flight termination is disabled with a circuit breaker, so I would not expect this to actually lead to termination, unless you specifically configure it that way.
When you switch to mission, trust goes to 1. So it tries to save it but the outputs are zero, so it can’t.
GPS seems absolutely fine, I don’t understand what should be wrong about it.
Ok, so I see that from your parameters, you have flight termination enabled (so the circuit breaker disabled). Was that actually what you wanted?
Additionally, you have disabled the GPS failure detection, so in theory it should not react on GPS lost.
On a first look, you seem to be correct. If it’s true, this is absolutely terrifying ! I will look into this in more detail and try to reproduce in simulation!
Edit:
I believe the reason this was not uncovered before was that no one was using your combination of termination enabled and datalink lost.
Thanks for looking into this, Julian. Looks like a pull request has already been opened on this issue since my original posting. Surprisingly, the motor arms and some stand-offs were the only things that broke after that crash, so our team is working this whole week on getting the drone back flying by this coming weekend.
The root of the issue seems to be the functionality of CBRK_FLIGHTTERM, which the PX4 enables by default to avoid any kind of flight termination. Our team disabled this parameter since we wanted to add flight termination functionality in the event of RC loss after a certain period of time, which is required by our competition. Originally, we planned on detecting/triggering this with a companion computer running MAVROS and custom code. However, it appears that we would need to change CBRK_FLIGHTTERM from its default value to make this possible. To get our drone back flying safely by this weekend, would it be a good idea to cherry-pick the pull request here onto our v1.8.2 based PX4 firmware fork? This would effectively just eliminate the buggy flight termination code in the event of RC/Datalink loss while still allowing our companion computer to send flight termination commands based off our specific competition rules. We’ll also make sure to test this with SITL extensively before hardware testing.
Also, thanks for the tips on tuning. We weren’t able to find much info about PX4 tuning for agricultural-size drones, so a lot of the tuning has been based off a combination of the PX4 tuning guide, Flight Review step responses, and just general feel of the drone in stabilize mode. We’ve also collected a running list of all our logs and PID tunings here to make it a more systematic process. Specifically, this is a recent log with more aggressive manual inputs in an enclosed environment with no wind for collecting more information to tune with. Most notably, we recently increased P gains to around 0.1 and I gains to 0.04 for both pitch/roll axes, which has led to better control in position mode but a harsher flight in stabilize mode. Unfortunately, our pilot is not very comfortable with acro mode, so we have only been able to do tuning in stabilized.
For the vibrations issue: Since we’re working on rebuilding part of the drone frame this weekend, we are also looking to reduce vibrations using this vibration gel on the flight controller, along with high flexibility silicon wires for all connectors to the flight controller to further increase isolation. Additionally, we’re planning on reducing IMU_GYRO_CUTOFF to 20Hz and disabling MC_DTERM_CUTOFF, as recommended by the PR here for larger frames. I’ll consult the PX4 documentation more on vibration dampening, but any tips for improving flight for agricultural-sized frames are highly appreciated.
Thanks for looking into this issue, we’ll be sure to report back about results from our testing once the drone is back up and running.
Sounds good, we will just go off of what is on master right now until another 1.9.0 release is made.
For current PX4 mount, we’re using a vibration mount like this, which does use double sided tape to attach the pixhawk to the vibration mount and the vibration mount to the avionics tray. The Pixhawk and Raspberry Pi companion computer are both installed in a 3D printed sliding avionics tray, which we use to quickly access those components for servicing. The box looks like the following:
We have done a bit of work to neaten the mess of wires leading to the Pixhawk on the left, although it’s still not the best. Current plan is to add the gel linked in my last post to attach the Pixhawk to the vibration mount, use high-flex silicon wire for all cables leading to the Pixhawk, and add some sort of foam material between the motor tube clamps and the tube arms that the clamps attach to. We could also dampen the entire avionics box (shown above), although it would be difficult to isolate noise coming through the cable that we use to feed signals in/out of that box (a DB-25 cable attached to a breakout board).
Ok, this shouldn’t be too bad from a vibration point of view.
I can think of 3 things that you could try:
Reduce MC_PITCHRATE_P and MC_ROLLRATE_P by 10-20% and check the logs if the situation improved.
Check motors and props for imbalances (e.g. chipped off prop or something nasty like that).
Try a harder mount than what you currently have (e.g. double sided tape right under the Pixhawk). I’m not sure about this, it’s just a guess but something must be exciting these angular vibrations.
@comranmorsh I did an impulse response test of an autopilot mounted on those blue vibration isolators some time ago.
Note that the test was done to estimate the natural frequency of the isolation mount. This is a “qualitative plot”, which means that the value of the amplitude isn’t really useful. The phase is not computed here and it is certainly not zero
You can see that two resonance modes are located at around 45Hz and 70Hz (one of those was maybe not directly caused by the isolation mount, but by the drone itself) and can be pretty bad if the blade-pass frequency of your drone is in that region. Having a super soft isolation mount with a low natural frequency and small damping ratio can be dangerous for large drones and you have to make sure that nothing on your drone can generate a vibration at that specific frequency.
Also keep in mind that trying to attenuate the resonance peak when the natural frequency is low by increasing the damping ratio of the isolator can lead to stability problems of the rate loops due to an increased phase lag.
! I will look into this in more detail and try to reproduce in simulation!
