Help diagnosing a crash?

🧪 Flight Testing & Log Analysis
1

I uploaded this log as an example of a problem I’m seeing. I’m hoping someone can give me a hint of what to do (or what I’m doing wrong?)

I’ve been building my own flight controllers since the mid-2000’s but I’m really new to px4 and quadcopters (so please be gentle) :slight_smile:

I calibrated the mags, gyros, and accelerometers. I can arm and take off in stabilize mode and fly around all day (drifting in the wind, but manually correcting.) Everything looks good on qgc, the little artificial horizon moves correctly if I pick up the quadcopter (preflight) and bank or pitch it. The compass points in approximately the right direction.

However, once in flight, if I flip over into position hold or mission modes, the EKF immediately resets, craps out, and the quad goes scooting across the field in a random direction. I can flip back into stabilize mode and mostly land safely (although it bounced and flipped on the flight I am uploading here.)

I’m confused why the ekf would be super great until the moment I flip into a position hold mode, and wait until then to completely barf? (I don’t know a lot about px4 internals, but I would have guess the ekf is just always running and doesn’t care what flight mode it is in?)

Anyway, I appreciate any help or insight anyone can offer (quad is an older f450 kit build with genuine dji motors and props.)

Is there any other info I can provide that would help understand what’s going on?

Thanks in advance,

Curt.

2

Hi Curtis!

Just a couple of clues that could help you, altought I’m not 1.11 and loiter user.

  1. What auto-pilot are you using?
  2. What GPS are you using?

The only magnetometer recognized (MAG0 ID:327690) seems to be affected by the electromagnetic enviroment.

imagen
imagen910Ă—408 40.5 KB

You can list al the magnetometers in the console of QGC using:
listener sensor_mag -i 0
listener sensor_mag -i 1
listener sensor_mag -i 2
…

3

Thanks for writing back with some ideas. Sorry for taking so long to respond to your help, but finally circling around to this project again.

I inherited this quad and the controller is in a plastic case (white) between the f450 frame layers and would require substantial disassembly to figure out the exact board (unless there is a way I can query from QGC?) I’m trying to reach the original builder to see if they recall the specific hardware details. Again, I’m a super newbie with px4.

I can verify the gps is a 3dr gps/compass – a bit dated based on the older style (pre JST-GH) connector, however the original builder used a cable with only 4 wires so the i2c lines are not there. It sounds like I need to source the correct cable: 6-pin JST-GH on one end and picoblade (or whatever the old style was called) on the other end. I think we may have these back in our lab, but I’ve been working from my basement since last March … ugh.

I found the mavlink console in QGC – wow, that’s a cool feature to have available. I verified I only have the onboard/internal mags — reporting zero error counts at the moment.

Can I ask a naive question? I don’t understand why the mag issue only peaks when I activate a position hold or mission mode. Prior to that it bumps around, but there are no specific warnings or errors. I walked around for 5 minutes holding the copter in different attitudes to see if that would help the EKF converge. It was ok, it reported reasonable heading (I mean +/- 20-30 degrees on the ground before flying … dji doesn’t do much better than that on a lot of days.)

I can take off and fly manually in stabilize mode all day long without any warnings or issues. But when I flip the switch into mission mode, I see these mag error spikes, the ekf2 blows up, and the copter scampers across the field in a random direction. So I’m already in flight (presumably with representative mag interference and flying just fine … but it’s not until I flip into mission mode when we see the big interference spikes in the data.)

This is me guessing: it seems like the EKF runs in some sort of reduced (AHRS?) mode when I’m flyiing in stabilize or walking around on the ground, and it doesn’t really kick into serious GPS/INS mode until I activate a mission or position hold mode? But at that point (when I’m already in flight) it immediate barfs and the copter zips across the field in a random direction until I take back manual control. Am I guessing about right as to what is going on with internal ekf mode switching based on flight modes???

For whatever it’s worth, where I’m going with all of this … our lab bought a big pre-built hexacopter with px4. I was really really hoping to get some good px4 experience with this smaller f450 before putting a couple grand in the air and doing something stupid with it. That was my thought process.

So ideally I could find a way to get this f450 flying practice missions and routes, and ideally I’d learn something and accumulate a bit of practical px4 experience. and maybe come to a better understanding of what’s actually happening under the hood … that is what I’m hoping to accomplish here … don’t know if it’s possible though. I’ve been building my own flight controlles for too long, so probably overly biases towards my own way of thinking about things … but trying to learn!

Best regards,

Curt.

4

Hi Curtis,

I don’t thinks that you can verfy the model of auto-pilot from QGround.

I usually use the HERE family GPS by I2C, so I’m not familiar with 3dr, but doesn’t sounds good to have only one magnetometer (the internal) detected.

In the Stabilized mode you don’t usually have GPS or magnetometers issues because in this mode the auto-pilot is only working to have the UAS “stabilized”; this means roll and pitch angle=0 using for that the gyroscope and acelerometer, nevermind the position or yaw. You can learn more about the flight modes following this links:
https://dev.px4.io/master/en/concept/flight_modes.html

Here you can find more information about the Emergency yaw reset - magnetometer use stopped message:
https://docs.px4.io/master/en/advanced_config/tuning_the_ecl_ekf.html#yaw-from-gps-velocity

This two lines must be always equal to zero. Other value could be a sensor problem (calibration, interferences,…).

imagen
imagen898Ă—396 32.3 KB

I assume that you have tried to recalibrate the sensors.
Check the GPS or instal a new one.
Check the power wires, they should be as far as possible from the pixhawk or signal wires.
If the quadcopter is inherited, take in mind also the possibility of hardware damage because of a crash.

In the PX4 and pixhawk world you are always learning and sometimes you feel disappointed because of the lack of informacion for your own issue, but is very heratwarning going step by step overcoming the problems, because is a great enviroment. We need to help each other as we can :slightly_smiling_face:

5

Thanks again for your reply. I figured out I have a Pixracer R15 installed. It’s extremely likely we purchased this direct from mRo. (It was sandwhiched between f450 frame layers, so not trivial to just look at.)

The original gps was an older 3dr branded model with the 1mm spacing (old style) connector, but the matching port on the pixracer was JST-GH. We had the right cable 6-pos cable, but it only had 4 of the wires/pins installed so the i2c lines weren’t there. I was thinking about a choice between annoying cable surgery vs. replacing the whole gps with something newer, so I just did that.

So now I have installed a much newer mRo gps module (ublox8 with compass) and a full 6-pin JST-GH to JST-GH cable.

From the mavlink console I can run the listener command (thanks for the heads up on that trick!) and I can see the 2nd magnetometer message so it appears to be detected and working.

Question #1: how do I tell the system to use the external compass? Is this somehow automatic, or do I need to find a setting somewhere. I hunted around in qgc, but didn’t see anything immediately obvious.

Question #2: If I want to avoid surprises in the air, is it legit to launch in some sort of position hold mode? I think I tried to do that once with the old gps + internal compass and it let me take off but the moment I got airborne the ekf puked and I zipped off in a random direction about 1’ above the ground and hit a pole and flipped and broke all my props before I could find the switch to flip back into stabilize. Basically wondering if there are best practices to avoid ekf suprises on launch or mode switches.

Thanks again for the help and sorry if some of these questions are super dumb.

Curt.

6

Hi Curtis,

Answer #1: Usually this is automatic. However, you ca set the ID of the external magnetometer in the CAL_MAG_PRIME parameter. Furthermore, if yo want you can disable the internal magnetometer using the CAL_MAGx_EN parameter (replace de x with the correct instance of magnetometer 0, 1, 2…). Remenber to calibrate again the compass after any change just in case.

Answer #2: I’am not sure about what to answer or which is the most correct. You have the Hold/Loiter mode, but in case of GPS attitude problem the drone is going to move. I can say you what I do when I’am testing the drones or set-ups.

  1. Before take-off, arm in stabilized mode and test that the drone tries to move in all the directions as you move slightly your RC stick.
  2. Take-off slowly in stabilized mode (you need to use roll, pitch and yaw stick in order to mantein de position and heading.
  3. Bried test of the drone performance in the air (PID set-up).
  4. In a safe altitude, switch to position mode. The drone must maintain literally the position. If the drone starts driffting at its own to some direction, switch fast to stabilized mode again.
    If the drone doesn’t mantein correctly the position, usually is a GPS or compass issue.

Good luck!

1 Like