Was test flying today, and the plane completely locked up. I was in hold mode trying to upload a new mission when the board completely locked all control from the plane I had no manual, nor mission control. Motor continued running. What could have caused this?
Sorry to hear that - do you have a log file? What software version did you flash?
@dagar no log file. Plane is gone.
It’s the custom firmware version dagar sent me with the modified wheel controller.
Good news is I have a second plane, I’ll try to replicate it on the ground.
edit: @LorenzMeier thought you were dagar
For reference, this is what I sent.
Questions -
How many previous flights do you have with that specific flight controller?
How many with that firmware?
Do you save the mavlink log from QGroundControl?
How long was it in hold before it locked up?
Did it seem to correspond with the actual mission sync (upload start)?
50+ flights with this flight controller
Around 30 with this specific firmware
yes, where should I upload it?
I had the plane cruising around for about 6 minutes, moving it around the map using hold mode.
And appeared to. I was looking at the computer screen, hit upload and then the telemetry disconnected and when I looked up the plane appeared to be locked up. I believe all servos froze, Including the motor at whatever percentage it was at.
Do you have dropbox or google drive? You might also be able to post it in gitter and link. https://gitter.im/PX4/Firmware
Something public so others can review.
It’ll be interesting to see if mavlink stops abruptly.
Is syncing during hold in air something you routinely do? How many waypoints were in the mission? What type?
I hope you don’t mind all the questions. I’m looking for anything at all unusual that might give a hint or a specific case to test. Any chance you’ll find the vehicle? The onboard log or a reproducible failure are our best chance at getting enough information to find the problem.
I have a drive I can share it with you.
Mavlink does stop immediatley at the freeze. Otherwise I would have a better Idea of where the plane ended up.
No it is not, I was just testing a bunch of stuff in air today, and figured that be something good to try out.
Just two waypoints within the mission, the first one was a loiter to altitude set at 50m. The second one was a land waypoint.
Keep asking questions, it is interesting to me too.
Tommorrow morning I’m going up in a real airplane, it was a bright white plane, hopefully I’ll be able to spot it from the air if it didn’t land in a lake. Otherwise I’ll keep attempting to reproduce the issue.
@dagar here is a telemetry link
https://drive.google.com/file/d/0ByGQI8gP4hzqYjZQWGg2VHZ1aHc/view?usp=drivesdk
Let me know if this works
Here’s the end of the telemetry from the last HEARTBEAT on. You can see the message “Landing waypoint above last nav waypoint” in the final second of the telemetry from the autopilot, which comes from the mission feasibility checker looking at the new mission you’ve just uploaded.
After this you can still see status messages from the remote radio for another 85 seconds until it disappears.
System load seems unusually high for fixedwing (~75%), but I should compare with similar logs.
2017-01-20 15:33:57.17: HEARTBEAT {type : 1, autopilot : 12, base_mode : 157, custom_mode : 50593792, system_status : 4, mavlink_version : 3}
2017-01-20 15:33:57.17: MISSION_CURRENT {seq : 0}
2017-01-20 15:33:57.23: MISSION_ITEM_REACHED {seq : 2}
2017-01-20 15:33:57.23: ATTITUDE {time_boot_ms : 346035, roll : 0.617643654346, pitch : 0.0309686493129, yaw : -0.475825130939, rollspeed : -0.0753746181726, pitchspeed : 0.478176355362, yawspeed : 0.44404733181}
2017-01-20 15:33:57.23: RC_CHANNELS {time_boot_ms : 346037, chancount : 7, chan1_raw : 1503, chan2_raw : 1501, chan3_raw : 1099, chan4_raw : 1497, chan5_raw : 1901, chan6_raw : 1099, chan7_raw : 1160, chan8_raw : 65535, chan9_raw : 65535, chan10_raw : 65535, chan11_raw : 65535, chan12_raw : 65535, chan13_raw : 65535, chan14_raw : 65535, chan15_raw : 65535, chan16_raw : 65535, chan17_raw : 65535, chan18_raw : 65535, rssi : 255}
2017-01-20 15:33:57.23: RC_CHANNELS_OVERRIDE {target_system : 1, target_component : 0, chan1_raw : 1503, chan2_raw : 1501, chan3_raw : 1099, chan4_raw : 1497, chan5_raw : 1901, chan6_raw : 1099, chan7_raw : 1160, chan8_raw : 65535}
2017-01-20 15:33:57.25: GLOBAL_POSITION_INT {time_boot_ms : 346051, lat : 383051659, lon : -898552993, alt : 227748, relative_alt : 96693, vx : 1648, vy : -977, vz : -30, hdg : 33324}
2017-01-20 15:33:57.39: ATTITUDE {time_boot_ms : 346152, roll : 0.668213009834, pitch : 0.016943288967, yaw : -0.426212280989, rollspeed : 0.776129841805, pitchspeed : 0.0380697436631, yawspeed : 0.369183152914}
2017-01-20 15:33:57.39: MISSION_REQUEST {target_system : 255, target_component : 0, seq : 1}
2017-01-20 15:33:57.39: MISSION_CURRENT {seq : 0}
2017-01-20 15:33:57.41: MISSION_ITEM_REACHED {seq : 2}
2017-01-20 15:33:57.41: VFR_HUD {airspeed : 15.0004911423, groundspeed : 19.3240776062, heading : 337, throttle : 56, alt : 227.822113037, climb : 0.232041910291}
2017-01-20 15:33:57.43: ATTITUDE {time_boot_ms : 346273, roll : 0.695339143276, pitch : 0.00491905910894, yaw : -0.381325811148, rollspeed : -0.271936476231, pitchspeed : 0.345104604959, yawspeed : 0.333914875984}
2017-01-20 15:33:57.61: RADIO {rssi : 90, remrssi : 80, txbuf : 100, noise : 36, remnoise : 18, rxerrors : 225, fixed : 362}
2017-01-20 15:33:57.61: RADIO_STATUS {rssi : 90, remrssi : 80, txbuf : 100, noise : 36, remnoise : 18, rxerrors : 225, fixed : 362}
2017-01-20 15:33:57.65: ATTITUDE {time_boot_ms : 346398, roll : 0.647665917873, pitch : 0.0196558050811, yaw : -0.305596232414, rollspeed : -0.518543958664, pitchspeed : 0.436673015356, yawspeed : 0.434209436178}
2017-01-20 15:33:57.65: MISSION_ACK {target_system : 255, target_component : 0, type : 0}
2017-01-20 15:33:57.65: MISSION_CURRENT {seq : 0}
2017-01-20 15:33:57.65: MISSION_ITEM_REACHED {seq : 2}
2017-01-20 15:33:57.65: RC_CHANNELS {time_boot_ms : 346501, chancount : 7, chan1_raw : 1503, chan2_raw : 1501, chan3_raw : 1099, chan4_raw : 1497, chan5_raw : 1901, chan6_raw : 1099, chan7_raw : 1160, chan8_raw : 65535, chan9_raw : 65535, chan10_raw : 65535, chan11_raw : 65535, chan12_raw : 65535, chan13_raw : 65535, chan14_raw : 65535, chan15_raw : 65535, chan16_raw : 65535, chan17_raw : 65535, chan18_raw : 65535, rssi : 255}
2017-01-20 15:33:57.68: RC_CHANNELS_OVERRIDE {target_system : 1, target_component : 0, chan1_raw : 1503, chan2_raw : 1501, chan3_raw : 1099, chan4_raw : 1497, chan5_raw : 1901, chan6_raw : 1099, chan7_raw : 1160, chan8_raw : 65535}
2017-01-20 15:33:57.68: GLOBAL_POSITION_INT {time_boot_ms : 346510, lat : 383052388, lon : -898553415, alt : 227919, relative_alt : 96864, vx : 1856, vy : -628, vz : -57, hdg : 34577}
2017-01-20 15:33:57.71: ATTITUDE {time_boot_ms : 346519, roll : 0.634312689304, pitch : -0.00396460620686, yaw : -0.245132014155, rollspeed : 0.23974776268, pitchspeed : -0.0910935327411, yawspeed : 0.556037425995}
2017-01-20 15:33:57.85: HIGHRES_IMU {time_usec : 346575284, xacc : 1.63658154011, yacc : 2.38166499138, zacc : -9.06373119354, xgyro : 0.346379518509, ygyro : 0.229975834489, zgyro : 0.564336240292, xmag : 0.313365250826, ymag : 0.308061867952, zmag : 0.435744076967, abs_pressure : 0.0, diff_pressure : 142.342193604, pressure_alt : 311.01940918, temperature : 20.7199993134, fields_updated : 7167}
2017-01-20 15:33:57.96: NAV_CONTROLLER_OUTPUT {nav_roll : 37.325466156, nav_pitch : -12.9666662216, nav_bearing : 72, target_bearing : 72, wp_dist : 46, alt_error : -3.0887298584, aspd_error : 0.044261932373, xtrack_error : -3.12340164185}
2017-01-20 15:33:57.96: POSITION_TARGET_GLOBAL_INT {time_boot_ms : 346577, coordinate_frame : 0, type_mask : 0, lat_int : 383053784, lon_int : -898548357, alt : 231.054000854, vx : 0.0, vy : 0.0, vz : 0.0, afx : 0.0, afy : 0.0, afz : 0.0, yaw : 0.0, yaw_rate : 0.0}
2017-01-20 15:33:58.00: SYS_STATUS {onboard_control_sensors_present : 0, onboard_control_sensors_enabled : 0, onboard_control_sensors_health : 0, load : 746, voltage_battery : 14244, current_battery : 2470, battery_remaining : 58, drop_rate_comm : 0, errors_comm : 0, errors_count1 : 0, errors_count2 : 0, errors_count3 : 0, errors_count4 : 0}
2017-01-20 15:33:58.00: BATTERY_STATUS {id : 0, battery_function : 1, type : 1, temperature : 32767, voltages : [4323, 4323, 4323, 65535, 65535, 65535, 65535, 65535, 65535, 65535], current_battery : 2470, current_consumed : 1682, energy_consumed : -1, battery_remaining : 58}
2017-01-20 15:33:58.00: EXTENDED_SYS_STATE {vtol_state : 0, landed_state : 2}
2017-01-20 15:33:58.03: SERVO_OUTPUT_RAW {time_usec : 346577898, port : 0, servo1_raw : 1534, servo2_raw : 1396, servo3_raw : 1363, servo4_raw : 1549, servo5_raw : 0, servo6_raw : 0, servo7_raw : 0, servo8_raw : 0}
2017-01-20 15:33:58.03: ALTITUDE {time_usec : 346590040, altitude_monotonic : 311.01940918, altitude_amsl : 227.970489502, altitude_local : 93.0325393677, altitude_relative : 96.9154968262, altitude_terrain : 133.355010986, bottom_clearance : 94.6154785156}
2017-01-20 15:33:58.03: GPS_RAW_INT {time_usec : 346562424, fix_type : 4, lat : 383052181, lon : -898553286, alt : 221656, eph : 59, epv : 95, vel : 1951, cog : 33782, satellites_visible : 20}
2017-01-20 15:33:58.07: LOCAL_POSITION_NED {time_boot_ms : 346587, x : -30.1235256195, y : 75.424118042, z : -93.0325393677, vx : 18.7994098663, vy : -5.79671144485, vz : -0.430879384279}
2017-01-20 15:33:58.18: WIND_COV {time_usec : 346588286, wind_x : 0.0, wind_y : 0.0, wind_z : 0.0, var_horiz : 0.0, var_vert : 0.0, wind_alt : 227.970489502, horiz_accuracy : 0.0, vert_accuracy : 0.0}
2017-01-20 15:33:58.29: STATUSTEXT {severity : 2, text : Landing waypoint above last nav waypoint}
2017-01-20 15:33:58.43: ATTITUDE {time_boot_ms : 346668, roll : 0.666078746319, pitch : -0.01043592114, yaw : -0.144781649113, rollspeed : 0.103385701776, pitchspeed : 0.776057720184, yawspeed : 0.528709411621}
2017-01-20 15:33:58.43: MISSION_ITEM_REACHED {seq : 2}
2017-01-20 15:33:58.43: MISSION_CURRENT {seq : 0}
2017-01-20 15:33:58.43: ATTITUDE {time_boot_ms : 346817, roll : 0.686068296432, pitch : 0.022175328806, yaw : -0.028983682394, rollspeed : 0.0565736405551, pitchspeed : 0.534198284149, yawspeed : 0.454563111067}What do you think is happening in the lockup? Is there such thing as a system overload?
How do we know the flight controller locked up and it was not the telemetry link that was lost?
Unless the airplane is very naturally stable, I doubt that it would stay aloft for a long time without any stabilization inputs from the flight controller.
I’ve had it happen on the ground before where all the servos completely lock up with power going to them.I have never had it happen when the plane was armed, or when the plane was receiving input. And also because It did not let me switch the plane back to manual from my radio, along with the plane flying in a “fixed” manner in sky as if all the control servos were locked. I could hear the throttle and it sounded locked in one position also.
I don’t know if it is specifically a board lockup, but that’s what I called it originally. It really could be any number of things.
@dagar just caused a lockup. Do you want me to upload the file to logmuncher?
It’s a very large file as it took me a long time to cause it to happen.
@Kjkinney there are a few more files on the sdcard that might be valuable. Can you dump them somewhere and share?
The next thing would be connecting the system console (https://dev.px4.io/advanced-system-console.html) and reproducing the lockup while connected.
It might be easier to just dump the entire sdcard contents into a shared google drive folder (or dropbox).
https://drive.google.com/open?id=0ByGQI8gP4hzqVzdkN21qZ2hVWjA
Here’s a google drive link to the SD card contents. Let me know if it works
Nothing directly helpful, but it looks like you were able to reproduce it via the same sequence of events. How many tries did it take?
Would you be able to do it again with the console connected? When it locks up I’m expecting the console will print a stack dump, and that should help me pinpoint exactly where the problem is in the code.
In the current development version of PX4 there’s automatic hard fault logging to the sdcard, but I wouldn’t recommend you start flying it quite yet.
The lockup occurs at the Mag #0 failover line. It is very easy to repeat, I can be in any mode and all that I have to do is reupload a mission that either has too steep of a glideslope or one where the land waypoint is higher than the previous waypoint. It is fairly random when it occurs, sometimes it happens on the second upload sometimes on the sixth or seventh but it is repeatable.
I was hoping for a stack dump, but any reproduce is good. I’ll open a github issue to continue this and try to make it happen on my own pixracer.