Unstable flight with coaxial octocopter

PX4 Autopilot
1

Hi,

I’m designing a coxial octocopter which needs to be used for a research project. To try and get understand coaxial octocopter better I made a simulation in Gazebo using one. I managed to get it to fly, but it flies very unstable. I noticed in the flight log that the roll and pitch angle estimate is different from the ground truth. Not sure what can cause this, so any ideas is appreciated. I have also linked the flight log.
https://review.px4.io/plot_app?log=f9d2c7a4-a7bf-4722-9775-4d43d2a818b3

Furthermore, I also discovered that if I lower the weight of the simulated drone a bit it will start flying even more unstable and crash. The drone will also takeoff even just using the arm command. Again any ideas for the reason are appreciated.

2

Here is another update about the drone becoming very unstable if the weight of the drone is changed from 4 kg to 3.5 kg. It starts oscillating in several different areas. Not sure of the course of this.
https://review.px4.io/plot_app?log=f1eeb4a9-1259-40a2-a6f2-18470c216958

An identical flight with all the same parameters except the weight changed from 3.5kg to 4kg is shown here. https://review.px4.io/plot_app?log=17995461-71ba-42f2-865b-90f565a7a4c7
It produces a very different flight and is able to safely hover.