angri
July 18, 2017, 7:07pm
1
I notice that setting SYS_COMPANION to something different from default 57600 leads to extremely unstable flight in HIL in mission mode. Apparently there are not many options to choose from with baudrate higher than 57600, so I use 921600. With that setting vehicle starts shaking very rapidly immediately after takeoff especially in mission mode (it is noticeable in other modes, but doesn’t appear to be so severe). When on the route the vehicle even flips around multiple times until it is unable to stabilize anymore in which case it just falls. I tried with several versions of the firmware, this behavior is consistent in 1.5.4, 1.6.1, 1.6.3, 1.6.5 and the current master.
The crash is easy to reproduce: set SYS_COMPANION to 921600, set up mission, run HIL@jmavsim (I use pixhack, fwiw). Here is an example of such crash log: http://logs.px4.io/plot_app?log=645cc0dc-ab83-4887-abfb-afa439756b1b
So what do I do to fix this? What is the actual reason of this problem? What more testing can be done? I got scared about this 100% crash chance up to the point when I can not test setting SYS_COMPANION on the real hardware.
dagar
July 30, 2017, 9:26pm
2
It’s cpu load.
These are way higher than normal.
Can you run uorb top
on autopilot? I’m wondering if the simulated sensor data rate is way too high.
angri
July 31, 2017, 2:15pm
3
Thanks for your reply. Here’s what I get after crash in HIL simulation:
update: 1s, num topics: 77
TOPIC NAME INST #SUB #MSG #LOST #QSIZE
led_control 0 1 90 54 4
vehicle_control_mode 0 11 4 1 1
sensor_combined 0 7 18 35 1
vehicle_status 0 9 4 6 1
actuator_armed 0 8 4 0 1
safety 0 1 45 0 1
vehicle_local_position 0 8 18 44 1
vehicle_attitude 0 6 18 16 1
vehicle_status_flags 0 0 90 0 1
commander_state 0 1 90 77 1
servorail_status 0 0 45 0 1
actuator_outputs 1 2 45 59 1
multirotor_motor_limits 0 1 45 0 1
control_state 0 4 18 9 1
ekf2_innovations 0 1 18 0 1
estimator_status 0 4 18 48 1
task_stack_info 0 1 1 0 2
vehicle_attitude_setpoint 0 5 57 128 1
vehicle_local_position_setpoint 0 3 57 92 1
wind_estimate 0 4 18 28 1
ekf2_timestamps 0 1 18 0 1
dagar
July 31, 2017, 2:28pm
4
After crash? If you run top what’s the cpu utilization of ekf2, mc_att_control, mc_pos_control at that point?
I’ll try to find some time to test jmavsim HIL later today. We just need to figure out what’s driving the crazy cpu utilization.
angri
July 31, 2017, 3:00pm
5
Yes, I ran simulation until (simulated) vehicle crashes, then I stopped jMAVsim, connected using mavlink shell and ran uorb top. Here’s one more crash result:
nsh> top
PID COMMAND CPU(ms) CPU(%) USED/STACK PRIO(BASE) STATE
0 Idle Task 26585 73.306 228/ 748 0 ( 0) READY
1 hpwork 1659 1.095 816/ 1780 192 (192) w:sig
2 lpwork 207 0.099 640/ 1780 50 ( 50) w:sig
3 init 5186 0.000 1760/ 2484 100 (100) w:sem
268 mavlink_shell 0 0.000 880/ 2020 100 (100) w:sem
100 dataman 153 0.000 704/ 1180 90 ( 90) w:sem
102 sensors 12396 0.199 1456/ 1980 249 (249) w:sem
104 commander 2367 1.593 2656/ 3644 140 (140) w:sig
105 commander_low_prio 5 0.000 592/ 2996 50 ( 50) w:sem
112 pwm_out_sim 2841 0.000 672/ 1172 100 (100) w:sem
114 px4io 5333 0.796 952/ 1484 240 (240) w:sem
122 mavlink_if0 1444 1.294 1640/ 2436 100 (100) w:sig
123 mavlink_rcv_if0 562 0.398 1320/ 2140 175 (175) w:sem
135 mavlink_if1 9457 7.270 1736/ 2372 100 (100) w:sig
138 mavlink_rcv_if1 581 0.398 1616/ 2140 175 (175) w:sem
185 log_writer_file 260 0.298 560/ 1060 60 ( 60) w:sem
173 mavlink_if2 8663 6.972 1736/ 2388 100 (100) w:sig
174 mavlink_rcv_if2 14579 0.398 1816/ 2140 175 (175) w:sem
184 logger 4953 3.784 3104/ 3532 245 (245) w:sem
271 top 1 0.000 1168/ 1684 100 (100) RUN
220 ekf2 32392 0.597 5064/ 5884 250 (250) w:sem
223 mc_att_control 9452 0.000 1176/ 1676 250 (250) w:sem
235 mc_pos_control 13454 0.498 1152/ 1876 250 (250) w:sem
241 navigator 381 0.000 1064/ 1772 105 (105) w:sem
Processes: 24 total, 2 running, 22 sleeping
CPU usage: 25.70% tasks, 1.00% sched, 73.31% idle
DMA Memory: 5120 total, 1536 used 1536 peak
Uptime: 149.619s total, 26.585s idle
nsh> uorb top
update: 1s, num topics: 77
TOPIC NAME INST #SUB #MSG #LOST #QSIZE
led_control 0 1 90 52 4
vehicle_control_mode 0 11 3 2 1
sensor_combined 0 7 18 17 1
vehicle_status 0 9 3 3 1
actuator_armed 0 8 3 19 1
safety 0 1 45 0 1
vehicle_local_position 0 8 18 27 1
vehicle_attitude 0 6 18 33 1
vehicle_status_flags 0 0 90 0 1
commander_state 0 1 90 81 1
servorail_status 0 0 45 0 1
actuator_outputs 1 2 45 61 1
multirotor_motor_limits 0 1 45 0 1
control_state 0 4 18 9 1
ekf2_innovations 0 1 18 0 1
estimator_status 0 4 18 46 1
task_stack_info 0 1 1 0 2
vehicle_attitude_setpoint 0 5 58 126 1
vehicle_local_position_setpoint 0 3 58 97 1
wind_estimate 0 4 18 29 1
ekf2_timestamps 0 1 18 0 1
dagar
August 1, 2017, 3:01pm
6
Could you try manually specifying the rate for jmavsim? My guess is it’s defaulting to something way too high.
First try -r 250
.