I will comment to what I know so far. The VTOL controller relies on separate multicopter and fixed-wing control loops and blends them during a discrete VTOL transition. Both of these loops are running at all times, and outputs from them are sent to the actuators based on the blending that takes place in the VTOL logic and in the mixer.
To understand behavior better, I suggest reviewing the code directly. The multicopter code is in mc_att_control.cpp, and the fixed-wing code is in FixedwingAttitudeControl.cpp. The blending takes place in vtol_att_control_main; which uses a “vtol_type” object to effect control. This object’s code is in vtol_att_control_main.cpp.
The vtol_type is actually the parent class of tailsitter, tiltrotor, and standard subclasses, which determine the behavior of, for example, the fill_actuator_outputs() method. This is an example of where blending occurs. So, for example, if your vehicle were a tailsitter, your vtol_att_control vtol_type object would be of the tailsitter subclass. Then the fill_actuator_outputs() method of tailsitter would describe the blending of your vehicle.
For the tailsitter, the elevons always have 100% fixed-wing, and the motors have multicopter inputs faded out as airspeed increases. When transition is complete, the motors are left with only thrust. Then the multicopter thrust is changed to fixed-wing thrust when the transition state is changed (at the full transition airspeed).
I don’t know the behavior of other vtol types, but encourage you to go look into the vtol_type subclasses.