Sitl segmentation fault

PX4 Autopilot
1

After a successful compile of the firmware, I went on (without writing it to my pixracer) with http://dev.px4.io/simulation-sitl.html

make posix_sitl_default jmavsim

ends in:

BUILD SUCCESSFUL
Total time: 3 seconds
SITL COMMAND: /usr/src/debian/px4-firmware/px4-firmware.git/build_posix_sitl_default/src/firmware/posix/px4 /usr/src/debian/px4-firmware/px4-firmware.git /usr/src/debian/px4-firmware/px4-firmware.git/posix-configs/SITL/init/lpe/iris
data path: /usr/src/debian/px4-firmware/px4-firmware.git
commands file: /usr/src/debian/px4-firmware/px4-firmware.git/posix-configs/SITL/init/lpe/iris
creating new parameters file
creating new dataman file

| ___ \ \ \ / / / |
| |/ / \ V / / /| |
| __/ / \ / /| |
| | / /^\ \ ___ |
_| / / |_/

px4 starting.

WARN [param] error importing from ‘rootfs/eeprom/parameters’
Command ‘param’ failed, returned 1
SYS_AUTOSTART: curr: 0 → new: 4010
INFO [dataman] Unkown restart, data manager file ‘rootfs/fs/microsd/dataman’ size is 103090 bytes
BAT_N_CELLS: curr: 0 → new: 3
CAL_GYRO0_ID: curr: 0 → new: 2293768
CAL_ACC0_ID: curr: 0 → new: 1376264
CAL_ACC1_ID: curr: 0 → new: 1310728
CAL_MAG0_ID: curr: 0 → new: 196616
CAL_GYRO0_XOFF: curr: 0.0000 → new: 0.0100
CAL_ACC0_XOFF: curr: 0.0000 → new: 0.0100
CAL_ACC0_YOFF: curr: 0.0000 → new: -0.0100
CAL_ACC0_ZOFF: curr: 0.0000 → new: 0.0100
CAL_ACC0_XSCALE: curr: 1.0000 → new: 1.0100
CAL_ACC0_YSCALE: curr: 1.0000 → new: 1.0100
CAL_ACC0_ZSCALE: curr: 1.0000 → new: 1.0100
CAL_ACC1_XOFF: curr: 0.0000 → new: 0.0100
CAL_MAG0_XOFF: curr: 0.0000 → new: 0.0100
SENS_BOARD_ROT: curr: 0 → new: 8
SENS_BOARD_X_OFF: curr: 0.0000 → new: 0.0000
COM_RC_IN_MODE: curr: 0 → new: 1
NAV_DLL_ACT: curr: 0 → new: 2
COM_DISARM_LAND: curr: 0 → new: 3
NAV_ACC_RAD: curr: 10.0000 → new: 2.0000
COM_OF_LOSS_T: curr: 0.0000 → new: 5.0000
COM_OBL_ACT: curr: 0 → new: 2
RTL_RETURN_ALT: curr: 60.0000 → new: 30.0000
RTL_DESCEND_ALT: curr: 30.0000 → new: 5.0000
RTL_LAND_DELAY: curr: -1.0000 → new: 5.0000
MIS_TAKEOFF_ALT: curr: 10.0000 → new: 2.5000
MC_ROLLRATE_P: curr: 0.1500 → new: 0.2000
MC_PITCHRATE_P: curr: 0.1500 → new: 0.2000
MC_PITCH_P: curr: 6.5000 → new: 6.0000
MC_ROLL_P: curr: 6.5000 → new: 6.0000
MPC_HOLD_MAX_Z: curr: 0.6000 → new: 2.0000
MPC_Z_VEL_P: curr: 0.2000 → new: 0.6000
MPC_Z_VEL_I: curr: 0.0200 → new: 0.1500
EKF2_GBIAS_INIT: curr: 0.1000 → new: 0.0100
EKF2_ANGERR_INIT: curr: 0.1000 → new: 0.0100
INFO [platforms__posix__drivers__ledsim] LED::init
INFO [platforms__posix__drivers__ledsim] LED::init
INFO [simulator] Not using /dev/ttyACM0 for radio control input. Assuming joystick input via MAVLink.
INFO [simulator] Waiting for initial data on UDP port 14560. Please start the flight simulator to proceed…
Options parsed, starting Sim.
Starting GUI…
3D [dev] 1.6.0-pre12-daily-experimental daily

libEGL warning: DRI2: failed to authenticate
Init MAVLink
INFO [simulator] Got initial simuation data, running sim…
INFO [pwm_out_sim] MODE_8PWM
INFO [tone_alarm] startup
/usr/src/debian/px4-firmware/px4-firmware.git/Tools/sitl_run.sh: line 134: 15144 Segmentation fault (core dumped) $sitl_command
[100%] Built target jmavsim

It seems that build_posix_sitl_default/src/firmware/posix/px4 crashes with SIGSEGV in uORB::Manager::node_open().

Is there anything obvious that I’m missing?
Thanks

2

I could not repro this on my end. Does updating from master and running a clean build fix this?

3

Already on master, and after a ‘git pull’, ‘make clean’, ‘make px4fmu-v2_default’
the final ‘make posix_sitl_default jmavsim’ still crashes in the same way.

How can I get this px4 to compile without optimization?
How can I get this px4 to run in gdb?

Thanks,
Carlo

4

For the live of me I can’t figure out how to compile a Debug type (instead of RelWithDebInfo)…
Please help?

5

Hi @CarloWood

you can use make posix_sitl_default jmavsim___gdb to run px4 in gdb.

hope that helps

6

Thanks. I had figured that out in the meantime, but it still compiles px4 with optimization, which makes debugging nearly impossible. How to disable optimization for px4?

7

I never had problems debugging this way. And we don’t have an option for this, but you can change the flag -Os in
https://github.com/PX4/Firmware/blob/master/cmake/common/px4_base.cmake#L675

to -O0.

1 Like
8

Hi all, wanted to do a follow up on this with what happened afterwards.

I wrote patches that allowed me compile specific modules without optimization by setting an environment variable that is a list of reg.exp. matching cmake targets for which compiler optimization has to be turned off (for the posix_sitl simulation case). You can find those patches here: https://github.com/CarloWood/Firmware/commits/sitl_debugging
and the pull request for that is https://github.com/PX4/Firmware/pull/5443
It has not been merged yet for some reason.
I wrote some documentation for this, which you can find here: https://github.com/CarloWood/Devguide/blob/d8c8bd418e6fcef71432a5511a5416e264f5c1f9/simulation-debugging.md

This allowed me to find out that my crash was caused by a stack overflow, most likely caused by the fact that my pointers are 64bit instead of 32bit as expected for the hard coded stacksizes. So, eventually the crash was fixed with this commit: https://github.com/PX4/Firmware/commit/4a5eae23a29d452596ee0c8d0961f661ae635214
which thus has already been merged into master.

9

Thanks Carlo! Have you seen Beats review comment from a week ago? That’s what is blocking the merge: https://github.com/PX4/Firmware/pull/5443#issuecomment-245030249

10

Oh, I totally missed that there was a new comment. Just now working from a tablet didn’t help with that. Back at my PC now and will fix this, thanks!