I set up the PX4 simulation using the ROS 2 User Guide and Gazebo Simulation guide and I was able to add my own sensors and fly via GPS without much problems but for both the micro-xrce-dds-agent and mavros I have problems feeding in a external position. When I feed in the Gazebo ground truth position via the odometry plugin though /fmu/in/vehicle_visual_odometry when using micro-xrce-dds-agent or /mavros/vision_pose/pose when using mavros I get huge drifts in position and orientation even though the input position is constant.
Note I followed Using Vision or Motion Capture Systems for Position Estimation for setting the EKF2 parameter and for micro-xrce-dds-agent converted the input from FLU to FRD.
What would be a general approach for finding out the cause of the drift and afterwards tuning to ensure good performance?