Unstable flight in HIL with SYS_COMPANION set to 921600

I notice that setting SYS_COMPANION to something different from default 57600 leads to extremely unstable flight in HIL in mission mode. Apparently there are not many options to choose from with baudrate higher than 57600, so I use 921600. With that setting vehicle starts shaking very rapidly immediately after takeoff especially in mission mode (it is noticeable in other modes, but doesn’t appear to be so severe). When on the route the vehicle even flips around multiple times until it is unable to stabilize anymore in which case it just falls. I tried with several versions of the firmware, this behavior is consistent in 1.5.4, 1.6.1, 1.6.3, 1.6.5 and the current master.

The crash is easy to reproduce: set SYS_COMPANION to 921600, set up mission, run HIL@jmavsim (I use pixhack, fwiw). Here is an example of such crash log: http://logs.px4.io/plot_app?log=645cc0dc-ab83-4887-abfb-afa439756b1b

So what do I do to fix this? What is the actual reason of this problem? What more testing can be done? I got scared about this 100% crash chance up to the point when I can not test setting SYS_COMPANION on the real hardware.

It’s cpu load.

These are way higher than normal.

Can you run uorb top on autopilot? I’m wondering if the simulated sensor data rate is way too high.

Thanks for your reply. Here’s what I get after crash in HIL simulation:

update: 1s, num topics: 77
TOPIC NAME                    INST #SUB #MSG #LOST #QSIZE
led_control                      0    1   90    54 4
vehicle_control_mode             0   11    4     1 1
sensor_combined                  0    7   18    35 1
vehicle_status                   0    9    4     6 1
actuator_armed                   0    8    4     0 1
safety                           0    1   45     0 1
vehicle_local_position           0    8   18    44 1
vehicle_attitude                 0    6   18    16 1
vehicle_status_flags             0    0   90     0 1
commander_state                  0    1   90    77 1
servorail_status                 0    0   45     0 1
actuator_outputs                 1    2   45    59 1
multirotor_motor_limits          0    1   45     0 1
control_state                    0    4   18     9 1
ekf2_innovations                 0    1   18     0 1
estimator_status                 0    4   18    48 1
task_stack_info                  0    1    1     0 2
vehicle_attitude_setpoint        0    5   57   128 1
vehicle_local_position_setpoint  0    3   57    92 1
wind_estimate                    0    4   18    28 1
ekf2_timestamps                  0    1   18     0 1

After crash? If you run top what’s the cpu utilization of ekf2, mc_att_control, mc_pos_control at that point?

I’ll try to find some time to test jmavsim HIL later today. We just need to figure out what’s driving the crazy cpu utilization.

Yes, I ran simulation until (simulated) vehicle crashes, then I stopped jMAVsim, connected using mavlink shell and ran uorb top. Here’s one more crash result:

nsh> top
 PID COMMAND                   CPU(ms) CPU(%)  USED/STACK PRIO(BASE) STATE 
   0 Idle Task                   26585 73.306   228/  748   0 (  0)  READY 
   1 hpwork                       1659  1.095   816/ 1780 192 (192)  w:sig 
   2 lpwork                        207  0.099   640/ 1780  50 ( 50)  w:sig 
   3 init                         5186  0.000  1760/ 2484 100 (100)  w:sem 
 268 mavlink_shell                   0  0.000   880/ 2020 100 (100)  w:sem 
 100 dataman                       153  0.000   704/ 1180  90 ( 90)  w:sem 
 102 sensors                     12396  0.199  1456/ 1980 249 (249)  w:sem 
 104 commander                    2367  1.593  2656/ 3644 140 (140)  w:sig 
 105 commander_low_prio              5  0.000   592/ 2996  50 ( 50)  w:sem 
 112 pwm_out_sim                  2841  0.000   672/ 1172 100 (100)  w:sem 
 114 px4io                        5333  0.796   952/ 1484 240 (240)  w:sem 
 122 mavlink_if0                  1444  1.294  1640/ 2436 100 (100)  w:sig 
 123 mavlink_rcv_if0               562  0.398  1320/ 2140 175 (175)  w:sem 
 135 mavlink_if1                  9457  7.270  1736/ 2372 100 (100)  w:sig 
 138 mavlink_rcv_if1               581  0.398  1616/ 2140 175 (175)  w:sem 
 185 log_writer_file               260  0.298   560/ 1060  60 ( 60)  w:sem 
 173 mavlink_if2                  8663  6.972  1736/ 2388 100 (100)  w:sig 
 174 mavlink_rcv_if2             14579  0.398  1816/ 2140 175 (175)  w:sem 
 184 logger                       4953  3.784  3104/ 3532 245 (245)  w:sem 
 271 top                             1  0.000  1168/ 1684 100 (100)  RUN   
 220 ekf2                        32392  0.597  5064/ 5884 250 (250)  w:sem 
 223 mc_att_control               9452  0.000  1176/ 1676 250 (250)  w:sem 
 235 mc_pos_control              13454  0.498  1152/ 1876 250 (250)  w:sem 
 241 navigator                     381  0.000  1064/ 1772 105 (105)  w:sem 

Processes: 24 total, 2 running, 22 sleeping
CPU usage: 25.70% tasks, 1.00% sched, 73.31% idle
DMA Memory: 5120 total, 1536 used 1536 peak
Uptime: 149.619s total, 26.585s idle

nsh> uorb top













update: 1s, num topics: 77
TOPIC NAME                    INST #SUB #MSG #LOST #QSIZE
led_control                      0    1   90    52 4
vehicle_control_mode             0   11    3     2 1
sensor_combined                  0    7   18    17 1
vehicle_status                   0    9    3     3 1
actuator_armed                   0    8    3    19 1
safety                           0    1   45     0 1
vehicle_local_position           0    8   18    27 1
vehicle_attitude                 0    6   18    33 1
vehicle_status_flags             0    0   90     0 1
commander_state                  0    1   90    81 1
servorail_status                 0    0   45     0 1
actuator_outputs                 1    2   45    61 1
multirotor_motor_limits          0    1   45     0 1
control_state                    0    4   18     9 1
ekf2_innovations                 0    1   18     0 1
estimator_status                 0    4   18    46 1
task_stack_info                  0    1    1     0 2
vehicle_attitude_setpoint        0    5   58   126 1
vehicle_local_position_setpoint  0    3   58    97 1
wind_estimate                    0    4   18    29 1
ekf2_timestamps                  0    1   18     0 1

Could you try manually specifying the rate for jmavsim? My guess is it’s defaulting to something way too high.

First try -r 250.