I am having an issue when spawning multiple PX4 SITL with gazebo. In fact, the problem initially showed when spawning multiple vehicles, but the same problem persists when spawning a single vehicle.
I am following the documentation and when running make px4_sitl gz_x500 or just make px4_sitl and then in separate terminals PX4_SYS_AUTOSTART=4001 PX4_GZ_MODEL=x500 ./build/px4_sitl_default/bin/px4 -i <instance>, the vehicle will build and then throw the following error indefinitely:
INFO [init] gazebo already running world: default
WARN [init] PX4_GZ_MODEL_NAME or PX4_GZ_MODEL not set using PX4_SIM_MODEL.
INFO [gz_bridge] world: default, model name: x500_0, simulation model: x500
INFO [lockstep_scheduler] setting initial absolute time to 168228548000 us
ERROR [vehicle_imu] 0 - gyro 1310988 timestamp error timestamp_sample: 168228652000, previous timestamp_samERROR [vehicle_imu] 0 - accel 1310988 timestamp error timestamp_sample: 168228652000, previous timestamp_saERROR[vehicle_imu] 0 - gyro 1310988 timestamp error timestamp_sample: 160175804000, previous timestamp_samERROR [vehicle_imu] 0 - accel 1310988 timestamp error timestamp_sample: 160175804000, previous timestamp_saERROR [vehicle_imu] 0 - gyro 1310988 timestamp error timestamp_sample: 168228656000, previous timestamp_samERROR [vehicle_imu] 0 - accel 1310988 timestamp error timestamp_sample: 168228656000, previous timestamp_saERROR [vehicle_imu] 0 - gyro 1310988 timestamp error timestamp_sample: 160175808000, previous timestamp_samERROR [vehicle_imu] 0 - accel 1310988 timestamp error timestamp_sample: 160175808000, previous timestamp_saINFO [commander] LED: open /dev/led0 failed (22)
WARN [health_and_arming_checks] Preflight Fail: No valid data from Accel 0
WARN [health_and_arming_checks] Preflight Fail: ekf2 missing data
WARN [health_and_arming_checks] Preflight Fail: No valid data from Gyro 0
WARN [health_and_arming_checks] Preflight Fail: Compass Sensor 0 missing
ERROR [vehicle_imu] 0 - gyro 1310988 timestamp error timestamp_sample: 168228660000, previous timestamp_samERROR [vehicle_imu] 0 - accel 1310988 timestamp error timestamp_sample: 168228660000, previous timestamp_saERROR `
The vehicle_imu error will just repeat indefinitely. This was previously working for both the single instance and multiple vehicle instances of the SITL. Any help here would be much appreciated. Thanks!!
Yes I replace <instance> with 0,1,2,… although I have also tried 1,2,3… with the same issue. I am working on the main branch.
The issue seems to be somewhat sporadic. Usually, I get the above error. Occasionally, I will get the following error on 1 or 2 out of maybe 5 vehicles that I have spawned (for example):
[px4] Startup script returned successfully
pxh> ERROR [vehicle_angular_velocity] unable to find or subscribe to selected sensor (1310988)
ERROR [vehicle_angular_velocity] unable to find or subscribe to selected sensor (1310988)
This will repeat indefinitely. And then even more sporadically, 1 or even 2 of the vehicles (generally out of 3-15) will work successfully, while the others error out with one of those 2 errors.
I have tried removing gazebo and re-installing it.
Since posting I have wiped the hard drive and installed Ubuntu 22 to start fresh. I have done the following:
git clone https://github.com/PX4/PX4-Autopilot.git --recursive
bash ./PX4-Autopilot/Tools/setup/ubuntu.sh
cd PX4-Autopilot
make px4_sitl
In a new terminal:
cd PX4-Autopilot
PX4_SYS_AUTOSTART=4001 PX4_GZ_MODEL=x500 ./build/px4_sitl_default/bin/px4 -i <instance>
Replacing <instance> with 0,1,2,3… and still get the following errors:
INFO [gz_bridge] world: default, model name: x500_0, simulation model: x500 ERROR [gz_bridge] timed out waiting for clock message ERROR [gz_bridge] Task start failed (-1) ERROR [init] gz_bridge failed to start ERROR [px4] Startup script returned with return value: 256
[px4] Startup script returned successfully pxh> ERROR [vehicle_angular_velocity] unable to find or subscribe to selected sensor (1310988) ERROR [vehicle_angular_velocity] unable to find or subscribe to selected sensor (1310988)
The very occasional missing BAROMETER error or ACCELEROMETER leading to EKF 0 errors.
I can’t seem to figure out how to reproduce any of these specifically, but 1 of the 4 happens every time. When spawning a swarm, some vehicles will have different errors than others. Any ideas?
Yes it should be. Now that I am on Ubuntu 22 it automatically installs with bash ./PX4-Autopilot/Tools/setup/ubuntu.sh
With that being said, when I initially ran my script after wiping and installing Ubuntu 22, I was able to spawn 5 gz_x500s successfully. Following that, the errors started to occur. I believe it may be an issue with gz server not killing correctly or something along those lines. At the present moment, every vehicle fails with
INFO [gz_bridge] world: default, model name: x500_0, simulation model: x500 ERROR [gz_bridge] timed out waiting for clock message ERROR [gz_bridge] Task start failed (-1) ERROR [init] gz_bridge failed to start ERROR [px4] Startup script returned with return value: 256
and kill the appropriate processes.
To avoid incurring in this issue in the future:
Start the first instance -i 0 and the wait a bit before the second, in this way you make sure that the gz server is fully running (or you will end up with two servers / 2 gui…)
When you kill the instances, start from -i 1 and leave -i 0 for last.
Understood. This still leaves the following errors/warnings for each vehicle.
Quads
INFO [px4] Startup script returned successfully
pxh> INFO [tone_alarm] notify negative
WARN [health_and_arming_checks] Preflight Fail: No valid data from Accel 0
WARN [health_and_arming_checks] Preflight Fail: No valid data from Gyro 0
ERROR [vehicle_air_data] BARO #0 failed: TIMEOUT!
ERROR [vehicle_air_data] BARO #0 failed: TIMEOUT!
WARN [health_and_arming_checks] Preflight Fail: No valid data from Accel 0
WARN [health_and_arming_checks] Preflight Fail: No valid data from Gyro 0
ERROR [vehicle_air_data] BARO #0 failed: TIMEOUT!
ERROR [vehicle_air_data] BARO #0 failed: TIMEOUT!
WARN [health_and_arming_checks] Preflight Fail: No valid data from Accel 0
WARN [health_and_arming_checks] Preflight: GPS Vertical Pos Drift too high
WARN [health_and_arming_checks] Preflight Fail: No valid data from Gyro 0
WARN [health_and_arming_checks] Preflight Fail: No valid data from Accel 0
WARN [health_and_arming_checks] Preflight Fail: No valid data from Gyro 0
WARN [health_and_arming_checks] Preflight Fail: No valid data from Accel 0
WARN [health_and_arming_checks] Preflight Fail: No valid data from Gyro 0
WARN [health_and_arming_checks] Preflight Fail: No valid data from Accel 0
Planes
[px4] Startup script returned successfully
pxh> INFO [airspeed_selector] No airspeed sensor detected. Switch to non-airspeed mode.
WARN [health_and_arming_checks] Preflight Fail: No valid data from Accel 0
WARN [health_and_arming_checks] Preflight Fail: Airspeed invalid
WARN [health_and_arming_checks] Preflight Fail: No valid data from Gyro 0
INFO [airspeed_selector] Airspeed sensor healthy, start using again (-1, 1)
WARN [health_and_arming_checks] Preflight Fail: No valid data from Accel 0
WARN [health_and_arming_checks] Preflight Fail: No valid data from Gyro 0
VTOLs
INFO [gz_bridge] Requested Model Position: 0,0,0,0,0,0
Error [Utils.cc:174] Missing element description for [persistent]
INFO [lockstep_scheduler] setting initial absolute time to 8000 us
INFO [commander] LED: open /dev/led0 failed (22)
INFO [tone_alarm] home set
INFO [mavlink] mode: Normal, data rate: 4000000 B/s on udp port 18572 remote port 14550
[Err] [MulticopterMotorModel.cc:528] You tried to access index 1 of the Actuator velocity array which is of size 1
[Err] [MulticopterMotorModel.cc:528] You tried to access index 2 of the Actuator velocity array which is of size 1
[Err] [MulticopterMotorModel.cc:528] You tried to access index 3 of the Actuator velocity array which is of size 1
[Err] [MulticopterMotorModel.cc:528] You tried to access index 4 of the Actuator velocity array which is of size 1
[Err] [MulticopterMotorModel.cc:528] You tried to access index 1 of the Actuator velocity array which is of size 1
OR
INFO [px4] Startup script returned successfully
pxh> INFO [tone_alarm] notify negative
WARN [health_and_arming_checks] Preflight Fail: No connection to the ground control station
ERROR [vehicle_air_data] BARO #0 failed: TIMEOUT!
ERROR [vehicle_magnetometer] MAG #0 failed: TIMEOUT!
WARN [health_and_arming_checks] Preflight Fail: No valid data from Accel 0
WARN [health_and_arming_checks] Preflight Fail: No valid data from Gyro 0
WARN [health_and_arming_checks] Preflight Fail: No connection to the ground control station
ERROR [health_and_arming_checks] angular velocity no longer valid (timeout)
WARN [airspeed_selector] Airspeed sensor failure detected. Check connection and reboot.
WARN [health_and_arming_checks] Preflight Fail: No valid data from Accel 0
WARN [health_and_arming_checks] Preflight Fail: Airspeed invalid
WARN [health_and_arming_checks] Preflight Fail: No valid data from Baro 0
WARN [health_and_arming_checks] Preflight Fail: No valid data from Gyro 0
WARN [health_and_arming_checks] Preflight Fail: No valid data from Compass 0
WARN [health_and_arming_checks] Preflight Fail: No connection to the ground control station
ERROR [vehicle_air_data] BARO #0 failed: TIMEOUT!
ERROR [vehicle_magnetometer] MAG #0 failed: TIMEOUT!```
I can't tell if these are bugs and if I should go back in the commits or if I am doing something incorrectly. Appreciate the help!
