SET_POSITION_TARGET_LOCAL_NED Issues with setting target positions

I have an Intel Aero connected to an Optitrack system in a GPS-denied environment. I am trying to make the drone take off in OFFBOARD mode using the SET_POSITION_TARGET_LOCAL_NED message.

I am building the message using dronekit.

msg = vehicle.message_factory.set_position_target_local_ned_encode(0, 0, 0, mavutil.mavlink.MAV_FRAME_LOCAL_NED,0b111111111000, north, east, down, 0, 0, 0, 0, 0, 0, 0, 0)
vehicle.send_mavlink(msg)

Initially, I set north and east to 0, and down to -0.3m. I have also set north and east to the initial positions of x0 and y0 after the drone is armed.

I can successfully transition to OFFBOARD mode, and I indeed send target setpoints to the drone and it begins to take off. However, I only want the target setpoint in the Z direction to change. What ends up happening is all three directions get some change in their targets and the drone moves along the -y direction (with respect to the drone meaning always in the direction opposite the onboard camera).
I have tried masking out the north and east targets, setting them to 0, setting them to the initial positions, changing the frame from MAV_FRAME_LOCAL_NED to BODY_NED, LOCAL_OFFSET_NED, and BODY_OFFSET_NED. The behavior is always the same, a drift backwards at a very low height.

Could it be that you need to “receive” the drone’s position first and then add your position setpoint on top of the current drone position? My hunch is that the drone does not actually start and initialize at 0, 0 but is a bit offset, and so you’re commanding it to go to 0, 0.