Stabilizing a multicopter is much easier than what you are trying to do, that's most likely why you have issues.
From what I understood, your vehicle is something like a thrust vectoring ducted fan (but uses fins instead of moving the nozzle). I also assume that the CG is above the thrust point.
This is a similar problem than balancing an inverted pendulum which is less actuated and more unstable than a standard quadcopter. The rate controller should then be tightly tuned since any growth in attitude error is more and more challenging to recover. The maximum tilt you could recover might be something like 10 degrees sloppy tuning is not an option.
A few things to consider:
- The torque you can produce with your system highly depends of the position of the CG, if you hold the drone on the CG point and that it can barely produce torque, move the CG higher
- You first have to tune the inner (rate loop) correctly, I recommend to hold it on the CG on a test bench with one free rotation at a time.
- The integral gain might need to be quite high