Hi. I am trying to control a large number (6-10) of gz_x500 quadcopters using python and a PX4 + Gazebo SITL. With 10 drones usually 7-8 take off and 2-3 stay on the ground. The show as flying in QGC.
I am using commit 56327284672bb0bc721edad39d67d6f51900bf17 of PX4-Autonomy which was committed 12/16/2026 along with Gazebo Harmonic 8.10 on Ubuntu 22.04.
I’m using a bash script called start_drones.sh to create 10 SITL instances and attach them all to the same Gazebo instance. A python script circle_flight.py then commands them all to fly in a circle formation using offboard mode. Usually 7 or 8 drones takeoff and fly the mission successfully. Most of the time the 9th and 10th drones fail to take off but in QGC it says they are flying.
I am running this on a laptop with 32 CPU cores, a NVIDIA 4080 GPU which I confirmed is being used by Gazebo, and 64GB of RAM. My load average is between 6 and 12 during the test.
I have tried:
Enabling GPU, confirm its working with nvidia-smi
Put Gazebo into headless mode
Setting PX4_SIM_SPEED_FACTOR to 0.5
Checking for failed MAVLINK messages using mavlink_stats.keys()
Allowing 60 seconds for the EKFs to converge initially
Is this a known issue when running large numbers of drones? Is it likely this is a system resource issue?
So how exactly do they fail? Do they not take off at all? Or do they take off but the visualization is wrong? Do they take off looking at the logs, or printfs?
Thanks for the quick response. Here is a screenshot of a 6 drone test. You can see in QGC that vehicle 4 is still at the launch site but it says its flying and in offboard mode. In gazebo the drone you can see partially on the sidewalk is vehicle 4. All the other drones successfully took off and flew the route properly and then landed. For some reason vehicle 4 is almost always the one that doesn’t take off with my 6 drone test. For a 10 drone test its almost always drones 9 and 10 but it varies.
Here is the output from a recent trial I did with 6 total drones. This output is from drone 2 which did not take off
=== Drone 2 Log Started at 2026-01-29 16:52:53 ===
[ 6.05] INFO [px4] instance: 2
[ 6.05] INFO [px4] working directory /home/nick/PX4-Autopilot/build/px4_sitl_default/rootfs/2
[ 6.05]
[ 6.05] ___ __ __ __
[ 6.06] | __ \ \ \ / / / |
[ 6.06] | |/ / \ V / / /| |
[ 6.06] | __/ / \ / /| |
[ 6.06] | | / /^\ \ __ |
[ 6.06] _| / / |/
[ 6.06]
[ 6.06] px4 starting.
[ 6.07]
[ 6.07] INFO [px4] startup script: /bin/sh etc/init.d-posix/rcS 2
[ 6.07] env SYS_AUTOSTART: 4001
[ 6.07] INFO [param] selected parameter default file parameters.bson
[ 6.07] INFO [param] importing from ‘parameters.bson’
[ 6.08] INFO [parameters] BSON document size 717 bytes, decoded 717 bytes (INT32:23, FLOAT:12)
[ 6.08] INFO [param] selected parameter backup file parameters_backup.bson
[ 6.27] COM_DL_LOSS_T set to 10.0
[ 6.28] COM_RC_LOSS_T set to 1.0
[ 6.29] COM_RC_LOSS_T: curr: 0.5000 → new: 1.0000
[ 6.29] COM_OF_LOSS_T set to 1.0
[ 6.30] + COM_OF_LOSS_T: curr: 5.0000 → new: 1.0000
[ 6.31] COM_OBC_LOSS_T set to 5.0
[ 6.64] INFO [dataman] data manager file ‘./dataman’ size is 1208528 bytes
[ 6.76] INFO [init] Gazebo simulator 8.10.0
[ 6.76] INFO [init] Standalone PX4 launch, waiting for Gazebo
[ 8.82] INFO [init] Waiting for Gazebo world…
[ 11.88] INFO [init] Waiting for Gazebo world…
[ 14.95] INFO [init] Gazebo world is ready
[ 14.96] INFO [init] Gazebo model pose: 10 0 0 0 0 0
[ 14.96] INFO [init] Spawning Gazebo model
[ 16.38] INFO [lockstep_scheduler] setting initial absolute time to 2500000 us
[ 16.44] INFO [gz_bridge] world: baylands, model: x500_2
[ 16.44] INFO [init] Setting simulation speed factor: 1.0
[ 16.72] INFO [init] Setting camera to follow x500_2
[ 17.59] INFO [init] Camera follow offset set to -2.0, -2.0, 2.0
[ 17.70] INFO [commander] LED: open /dev/led0 failed (22)
[ 17.70] WARN [health_and_arming_checks] Preflight Fail: barometer 0 missing
[ 17.70] WARN [health_and_arming_checks] Preflight Fail: ekf2 missing data
[ 17.71] WARN [health_and_arming_checks] Preflight Fail: system power unavailable
[ 17.71] WARN [health_and_arming_checks] Preflight Fail: No connection to the GCS
[ 17.85] INFO [uxrce_dds_client] init UDP agent IP:127.0.0.1, port:8888
[ 17.95] INFO [mavlink] mode: Normal, data rate: 4000000 B/s on udp port 18572 remote port 14550
[ 18.06] INFO [mavlink] partner IP: 127.0.0.1
[ 18.07] INFO [mavlink] Ignore command 401 from 255/190 to 2/1
[ 18.07] INFO [mavlink] Ignore command 401 from 255/190 to 5/1
[ 18.13] INFO [mavlink] mode: Onboard, data rate: 4000000 B/s on udp port 14582 remote port 14542
[ 18.14] INFO [mavlink] mode: Onboard, data rate: 4000 B/s on udp port 14282 remote port 14032
[ 18.16] INFO [mavlink] mode: Gimbal, data rate: 400000 B/s on udp port 13032 remote port 13282
[ 18.25] INFO [logger] logger started (mode=all)
[ 18.25] INFO [logger] Start file log (type: full)
[ 18.25] INFO [logger] [logger] ./log/2026-01-29/22_53_05.ulg
[ 18.25] INFO [logger] Opened full log file: ./log/2026-01-29/22_53_05.ulg
[ 18.26] INFO [mavlink] MAVLink only on localhost (set param MAV{i}BROADCAST = 1 to enable network)
[ 18.26] INFO [mavlink] MAVLink only on localhost (set param MAV{i}_BROADCAST = 1 to enable network)
[ 18.26] INFO [px4] Startup script returned successfully
[ 18.30] INFO [tone_alarm] home set
[ 18.40] INFO [mavlink] Ignore command 512 from 255/190 to 1/1
[ 18.44] INFO [mavlink] Ignore command 521 from 255/190 to 1/1
[ 18.46] INFO [mavlink] Ignore command 512 from 255/190 to 1/1
[ 18.46] INFO [mavlink] Ignore command 512 from 255/190 to 1/1
[ 18.46] INFO [mavlink] Ignore command 512 from 255/190 to 1/1
[ 18.46] INFO [mavlink] Ignore command 512 from 255/190 to 1/1
[ 18.46] INFO [mavlink] Ignore command 512 from 255/190 to 1/1
[ 18.46] INFO [mavlink] Ignore command 512 from 255/190 to 1/1
[ 18.47] INFO [mavlink] Ignore command 512 from 255/190 to 1/1
[ 18.47] INFO [mavlink] Ignore command 512 from 255/190 to 1/1
[ 18.47] INFO [mavlink] Ignore command 512 from 255/190 to 1/1
[ 18.47] INFO [mavlink] Ignore command 512 from 255/190 to 1/1
[ 18.47] INFO [mavlink] Ignore command 512 from 255/190 to 1/1
[ 18.47] INFO [mavlink] Ignore command 512 from 255/190 to 1/1
[ 18.48] INFO [mavlink] Ignore command 512 from 255/190 to 1/1
[ 18.48] INFO [mavlink] Ignore command 512 from 255/190 to 1/1
[ 18.48] INFO [mavlink] Ignore command 512 from 255/190 to 1/1
[ 18.48] INFO [mavlink] Ignore command 512 from 255/190 to 1/1
[ 18.48] INFO [mavlink] Ignore command 512 from 255/190 to 1/1
[ 18.48] INFO [mavlink] Ignore command 512 from 255/190 to 1/1
[ 18.48] INFO [mavlink] Ignore command 512 from 255/190 to 1/1
[ 18.49] INFO [mavlink] Ignore command 512 from 255/190 to 1/1
[ 18.49] INFO [mavlink] Ignore command 512 from 255/190 to 1/1
[ 18.49] INFO [mavlink] Ignore command 512 from 255/190 to 1/1
[ 18.49] INFO [mavlink] Ignore command 512 from 255/190 to 1/1
[ 18.49] INFO [mavlink] Ignore command 512 from 255/190 to 1/1
[ 18.49] INFO [mavlink] Ignore command 512 from 255/190 to 1/1
[ 18.49] INFO [mavlink] Ignore command 512 from 255/190 to 1/1
[ 18.50] INFO [mavlink] Ignore command 512 from 255/190 to 1/1
[ 18.51] INFO [mavlink] Ignore command 401 from 255/190 to 1/1
[ 18.87] INFO [mavlink] Ignore command 512 from 255/190 to 6/1
[ 18.91] INFO [mavlink] Ignore command 521 from 255/190 to 6/1
[ 18.94] INFO [mavlink] Ignore command 512 from 255/190 to 5/1
[ 18.94] INFO [mavlink] Ignore command 512 from 255/190 to 6/1
[ 18.95] INFO [mavlink] Ignore command 512 from 255/190 to 6/1
[ 18.96] INFO [mavlink] Ignore command 512 from 255/190 to 6/1
[ 18.96] INFO [mavlink] Ignore command 512 from 255/190 to 6/1
[ 18.96] INFO [mavlink] Ignore command 512 from 255/190 to 6/1
[ 18.96] INFO [mavlink] Ignore command 512 from 255/190 to 6/1
[ 18.97] INFO [mavlink] Ignore command 512 from 255/190 to 6/1
[ 18.97] INFO [mavlink] Ignore command 512 from 255/190 to 6/1
[ 18.97] INFO [mavlink] Ignore command 512 from 255/190 to 6/1
[ 18.97] INFO [mavlink] Ignore command 512 from 255/190 to 6/1
[ 18.98] INFO [mavlink] Ignore command 512 from 255/190 to 6/1
[ 18.98] INFO [mavlink] Ignore command 512 from 255/190 to 6/1
[ 19.02] INFO [mavlink] Ignore command 512 from 255/190 to 6/1
[ 19.03] INFO [mavlink] Ignore command 512 from 255/190 to 6/1
[ 19.04] INFO [mavlink] Ignore command 512 from 255/190 to 6/1
[ 19.04] INFO [mavlink] Ignore command 512 from 255/190 to 6/1
[ 19.04] INFO [mavlink] Ignore command 512 from 255/190 to 6/1
[ 19.04] INFO [mavlink] Ignore command 512 from 255/190 to 6/1
[ 19.04] INFO [mavlink] Ignore command 512 from 255/190 to 6/1
[ 19.04] INFO [mavlink] Ignore command 512 from 255/190 to 6/1
[ 19.05] INFO [mavlink] Ignore command 512 from 255/190 to 6/1
[ 19.05] INFO [mavlink] Ignore command 512 from 255/190 to 6/1
[ 19.05] INFO [mavlink] Ignore command 512 from 255/190 to 6/1
[ 19.05] INFO [mavlink] Ignore command 512 from 255/190 to 6/1
[ 19.05] INFO [mavlink] Ignore command 512 from 255/190 to 6/1
[ 19.05] INFO [mavlink] Ignore command 512 from 255/190 to 6/1
[ 19.05] ERROR [mavlink] vehicle_command_ack lost, generation 4 → 16
[ 19.06] ERROR [mavlink] vehicle_command_ack lost, generation 4 → 16
[ 19.06] ERROR [mavlink] vehicle_command_ack lost, generation 4 → 16
[ 19.06] ERROR [mavlink] vehicle_command_ack lost, generation 4 → 16
[ 19.06] INFO [mavlink] Ignore command 512 from 255/190 to 6/1
[ 19.06] ERROR [mavlink] vehicle_command_ack lost, generation 20 → 22
[ 19.06] ERROR [mavlink] vehicle_command_ack lost, generation 20 → 22
[ 19.06] ERROR [mavlink] vehicle_command_ack lost, generation 20 → 22
[ 19.07] ERROR [mavlink] vehicle_command_ack lost, generation 20 → 22
[ 19.09] INFO [mavlink] Ignore command 401 from 255/190 to 6/1
[ 19.27] INFO [mavlink] Ignore command 512 from 255/190 to 4/1
[ 19.58] INFO [tone_alarm] notify positive
[ 19.82] INFO [mavlink] Ignore command 512 from 255/190 to 2/1
[ 19.84] INFO [mavlink] Ignore command 512 from 255/190 to 6/1
[ 19.88] INFO [mavlink] Ignore command 512 from 255/190 to 5/1
[ 20.28] INFO [mavlink] Ignore command 512 from 255/190 to 4/1
[ 20.57] INFO [commander] e[32mReady for takeoff!e[0m
[ 20.81] INFO [mavlink] Ignore command 512 from 255/190 to 2/1
[ 20.85] INFO [mavlink] Ignore command 512 from 255/190 to 6/1
[ 20.88] INFO [mavlink] Ignore command 512 from 255/190 to 5/1
[ 21.28] INFO [mavlink] Ignore command 512 from 255/190 to 4/1
[ 21.82] INFO [mavlink] Ignore command 512 from 255/190 to 2/1
[ 21.87] INFO [mavlink] Ignore command 512 from 255/190 to 6/1
[ 21.89] INFO [mavlink] Ignore command 512 from 255/190 to 5/1
[ 22.29] INFO [mavlink] Ignore command 512 from 255/190 to 4/1
[ 22.54] INFO [mavlink] Ignore command 512 from 255/190 to 4/1
[ 22.57] INFO [mavlink] Ignore command 521 from 255/190 to 4/1
[ 22.84] INFO [mavlink] Ignore command 512 from 255/190 to 2/1
[ 22.87] INFO [mavlink] Ignore command 512 from 255/190 to 6/1
[ 22.91] INFO [mavlink] Ignore command 512 from 255/190 to 5/1
[ 23.04] INFO [mavlink] Ignore command 512 from 255/190 to 5/1
[ 23.04] INFO [mavlink] Ignore command 512 from 255/190 to 2/1
[ 23.05] INFO [mavlink] Ignore command 521 from 255/190 to 5/1
[ 23.05] INFO [mavlink] Ignore command 521 from 255/190 to 2/1
[ 23.31] INFO [mavlink] Ignore command 512 from 255/190 to 4/1
[ 23.54] INFO [mavlink] Ignore command 512 from 255/190 to 1/1
[ 23.59] INFO [mavlink] Ignore command 521 from 255/190 to 1/1
[ 23.86] INFO [mavlink] Ignore command 512 from 255/190 to 2/1
[ 23.91] INFO [mavlink] Ignore command 512 from 255/190 to 6/1
[ 23.93] INFO [mavlink] Ignore command 512 from 255/190 to 5/1
[ 24.04] INFO [mavlink] Ignore command 512 from 255/190 to 6/1
[ 24.04] INFO [mavlink] Ignore command 521 from 255/190 to 6/1
[ 24.33] INFO [mavlink] Ignore command 512 from 255/190 to 4/1
[ 24.88] INFO [mavlink] Ignore command 512 from 255/190 to 2/1
[ 24.91] INFO [mavlink] Ignore command 512 from 255/190 to 6/1
[ 25.48] INFO [mavlink] Ignore command 512 from 255/190 to 1/1
[ 26.48] INFO [mavlink] Ignore command 512 from 255/190 to 1/1
[ 26.60] INFO [mavlink] partner IP: 127.0.0.1
[ 26.61] ERROR [mavlink] unknown param: CBRK_AIRSPD_CHK
[ 26.61] ERROR [mavlink] param types mismatch param: FD_FAIL_R
[ 26.61] ERROR [mavlink] param types mismatch param: FD_FAIL_P
[ 27.48] INFO [mavlink] Ignore command 512 from 255/190 to 1/1
[ 28.49] INFO [mavlink] Ignore command 512 from 255/190 to 1/1
[ 29.52] INFO [mavlink] Ignore command 512 from 255/190 to 1/1
[ 29.53] INFO [mavlink] Ignore command 512 from 255/190 to 4/1
[ 29.59] INFO [mavlink] Ignore command 521 from 255/190 to 4/1
[ 30.52] INFO [mavlink] Ignore command 512 from 255/190 to 1/1
[ 30.54] INFO [mavlink] Ignore command 512 from 255/190 to 5/1
[ 30.54] INFO [mavlink] Ignore command 512 from 255/190 to 2/1
[ 30.57] INFO [mavlink] Ignore command 521 from 255/190 to 2/1
[ 30.57] INFO [mavlink] Ignore command 521 from 255/190 to 5/1
[ 31.04] INFO [mavlink] Ignore command 512 from 255/190 to 1/1
[ 31.04] INFO [mavlink] Ignore command 521 from 255/190 to 1/1
[ 31.54] INFO [mavlink] Ignore command 512 from 255/190 to 6/1
[ 31.58] INFO [mavlink] Ignore command 521 from 255/190 to 6/1
[ 41.04] INFO [mavlink] Ignore command 512 from 255/190 to 4/1
[ 41.05] INFO [mavlink] Ignore command 521 from 255/190 to 4/1
[ 41.98] INFO [mavlink] Ignore command 512 from 255/190 to 5/1
[ 41.98] INFO [mavlink] Ignore command 512 from 255/190 to 2/1
[ 41.98] INFO [mavlink] Ignore command 521 from 255/190 to 5/1
[ 41.99] INFO [mavlink] Ignore command 521 from 255/190 to 2/1
[ 42.03] INFO [mavlink] Ignore command 512 from 255/190 to 1/1
[ 42.04] INFO [mavlink] Ignore command 521 from 255/190 to 1/1
[ 42.78] INFO [commander] Armed by external command
[ 42.78] INFO [tone_alarm] arming warning
[ 43.04] INFO [mavlink] Ignore command 512 from 255/190 to 6/1
[ 43.04] INFO [mavlink] Ignore command 521 from 255/190 to 6/1
[ 44.60] INFO [commander] Takeoff detected
[ 60.03] INFO [mavlink] Ignore command 512 from 255/190 to 4/1
[ 60.04] INFO [mavlink] Ignore command 521 from 255/190 to 4/1
[ 61.04] INFO [mavlink] Ignore command 512 from 255/190 to 5/1
[ 61.04] INFO [mavlink] Ignore command 512 from 255/190 to 2/1
[ 61.04] INFO [mavlink] Ignore command 512 from 255/190 to 1/1
[ 61.04] INFO [mavlink] Ignore command 521 from 255/190 to 2/1
[ 61.05] INFO [mavlink] Ignore command 521 from 255/190 to 5/1
[ 61.05] INFO [mavlink] Ignore command 521 from 255/190 to 1/1
[ 62.04] INFO [mavlink] Ignore command 512 from 255/190 to 6/1
[ 62.04] INFO [mavlink] Ignore command 521 from 255/190 to 6/1
[150.51] INFO [commander] Landing at current position
[152.43] INFO [commander] Landing detected
[157.48] INFO [commander] Disarmed by landing
Hey Shoe Marker.
I did something very similar to what you dod several times with no issues. But only After making minor adjustmetns to the PX4 I found a major issue had to do with the Gimbal model. I only got networked Gimbaled drones working after significant work on the config files for the camera. I can’t remember anymore what I had to do…..and it was a defense thing so I could not share with you the solution anyway. I reached the conclusion it was a waste of time anyway since the networked drone device and the vision of these drones was more easily solved by seperating the problem alltogether.
do you have the issue when you spawn the 10 nodes as generic GZ500?
look for namspace / mavlink ID
Sorry I can’t explain this better but there is an ID for the namespace for each drone and then there is the mavlink ID itself in the simulator. I recall I had to make sure the start up script had to make a unique of each. I also had ot ensure the UDP/TCP ports I communicated on are unique.
On occasion the simulator bugs out…especially with the camera model. it is for the same reason the camera links were not set up with multi agent systems in mind.
