Swarm simulation instability

What is your system RAM?
Try again by setting speed factor flag. This will give simulation more time to connect to xrce agent and get sensor data to publish.
Check this.