This would be great!
I think the way you’d want to do this is with MAV_CMD_DO_LAND_START. (http://mavlink.org/messages/common)
MAV_CMD_DO_LAND_START can be in the mission plan (marking the start of a landing sequence) and sent as a command that jumps the mission to that point. The missing piece is tying that functionality to RTL.
This is the RTL code in navigator - https://github.com/PX4/Firmware/blob/master/src/modules/navigator/rtl.cpp#L153
and this is where the command MAV_CMD_DO_LAND_START is handeled https://github.com/PX4/Firmware/blob/master/src/modules/navigator/navigator_main.cpp#L495
Please feel free to ping me (@dagar on discuss, github, gitter, etc) if you have any questions.