Quadcopter switch to offboard mode losse balance

Hi everyone, i am doing some task envole to autonomous quaddrone control

i am using gazebo simulation, the drone has forward camera attached. my purpose is control drone to to hit some target on the ground. (just for the research purpose)

the problem is, when drone in a mission, i start tracking object and send the object position (by convert object position to target local_position_ned message) to the drone. but when i switch from mission mode to offboard mode. it suddenly lost balance - pitch down too much or pitch up too much

please help me

This means there is a bug in your controller/tracking.
As this is not part of PX4, it could be quite hard to get help with that.

For the best change of getting help, also consider using the PX4 ROS 2 Control Interface | PX4 Guide (main), instead of Offboard mode.