Custom mixer not working

IIRC there are only 8 main outputs so you’d have to transfer at least one to the aux output.

In my thesis I developed a tilt-quadrotor (4motors + 4 servos) and I used the motors on the MAIN and the servos on the AUX.

In your case I’d move the servo to the aux and maybe the ruddervators aswell.