Once CPU load is higher, the px4 code or simulator code will miss cycles and at some point this can lead to control instabilities which is probably what you're seeing.
The solution to this is lockstep SITL which means px4 and the simulator wait on each other for the next step. I'm currently working in this pull request and you can already try it out with jMAVSim.
I don't know if it works for multiple drones though, that's something I have not tested, and also I don't think the simulators are synchronized when using multiple drones, so simulation time between the simulator instances can drift.
A workaround for your problem is to do profiling and find out what uses most CPU (or GPU?) resources, then optimize that while also using a faster computer.