Good to know regarding saving the HILStar via the plane editor, I'll try that.
The remapping for the VTOL airframe mixer is not going well... I've found the GQC source code responsible for this and have been able to recompile QGC with my changes, but X-Plane seems to not be receiving some of the data I send (thrust vector) and receiving the rest all scrambled (thrust to each of the motors, roll, pitch, yaw).
I believe the data rates and ports are set up correctly because I've been able to control a standard AERT X-Plane model using the HILStar airframe, and I know what each of my VTOL airframe's actuator control outputs are--but I don't know how those control outputs are getting pulled into the "QGCXPlaneLink::updateActuatorControls(....)" function, nor whether I'm sending them to the correct data groups in X-Plane.
Here is my current setup: In that updateActuatorControls function, I'm sending my motor control signals 1-4 to indices 0-3 of data set 25 (throttle command), my tilt servo command to index 0 of data set 9 ("other flight controls", 0 is thrust vector which the X-Plane model uses to tilt the motors), and aileron, elevator, and rudder to indices 0-2 of data set 8 (joystick ail/elv/rud). Does that seem like it could work?
Another issue I'm having is that the X-Plane VTOL model wants to determine differential throttle to each motor from roll, pitch and yaw commands--whereas I want the Pixhawk to do that, and X-Plane to just spin each motors at the speed the Pixhawk says. Anyone have any ideas for a work-around? Disabling differential throttle for the X-Plane model in Plane maker just makes all the motors spin at the same rate. I'm stumped!