Great to hear! Yeah there are a lot of issues with tiltrotor.cpp, I think it was made with a particular airframe in mind and not the general case, so if you do anything slightly custom you get all sorts of strange/volatile behavior. Looks like you've already come across at least one of those strange behaviors but here are the main ones:
1- MC yaw weight goes instantly to 0 after 7m/s (hardcoded). Can result in sudden yaw/roll changes depending on the state of transition
2- FW throttle only applied in FW mode. Means that in a mission, your aircraft won't accelerate into the correct transition speed because it's only reading MC throttle inputs
3- In the back transition the throttle is commanded to cut to 0! (That's what you saw!). This to prevent sudden thrust inputs in strange directions (I've crashed an aircraft when I changed it). But like you said it leads to the aircraft losing a lot of altitude, and isn't great if you suddenly need to get back into hover mode (nearly crashed an aircraft because of that)
4- Again in the back transition, the MC weighting is set at 0 throughout. When I first edited the back transition I only removed the throttle cut line, but left this line. If this line is kept unchanged and the throttle line is removed (so you can have a powered back transition) you get a horrible mix of MC commands when the aircraft is not ready for MC flight (so MC roll = FW yaw with diff thrust, therefore input =/= output, autpilot can't control the aircraft). Long story short, if you want a powered transition you'll need to progressively change the MC/FW weighting according to how far through the transition it is. I've never tested this in flight so I can't guarantee it will work.
5- If you transition to FW mode while in MANUAL flight mode, the aircraft appears to lose ALL control! I've lost an aircraft to this, nosed dived from 50m, not good! I don't know why this is the case (it's somewhere else in the firmware), or if this happens every time, but you want to make sure you're in STABILIZED mode before taking off. Unfortunately, regardless of what you've set for your flight modes, your aircraft will ALWAYS start in MANUAL mode. To fix this, make sure "PERMANENT STABILIZATION" is ENABLED. It seems to have worked me for the time being.
I've nearly finished making an improved tiltrotor.cpp which should work better for all tiltrotor/tiltwing aircraft (includes fixes for the above problems related to the code), but I haven't worked out how to upload it to github yet. I'll let you know when I do, we might even be able to get multiple groups working on it so it's optimized for different scenarios.
As to your question, I'm not quite sure myself either, a couple of thoughts though:
Issue #1 - Best guess is it's due to issue #5 I mentioned above, if in manual mode it might just be cutting all control (but it seems that you get roll and pitch control?)
Issue #2 - Could be something with the yaw scaling/weighting. making sure the logic is correct so that the actuator output is correctly applying yaw in FW mode
Issue #3 - Sounds like you're using servos for tilt and pitch (/forward movement) in MC mode? I'd use a "S: 1 4 ..." command for the tilt control, and "S: 0 1 ...." for MC pitch. If you don't want the aircraft to pitch forward in MC mode, set your motor command line such that there is no pitch control using thrust, ei:
R: 8x 10000 0 10000 0
Then, in QGroundControl, you may want to change the parameter that limits pitch angle to something small (<10deg). I tried this a while ago and found I lost all pitch command control, but I hadn't learnt I could stop my motors from applying pitch commands.
That said, if you work out a way to do it with another strategy I'm keen to give it a shot!