PX4 Offboard: Drone fails to track and maintain following distance behind a visual target

PX4 Autopilot
1

Hi everyone,

I’m stuck with my simulated drone project and could use some help. I’ve been wrestling with tracking and control for days, and I’m no longer sure if the issue is my state estimator (IMM), my controller (PID/MPC), or the PX4 integration itself.

My Stack (based on stability recommendations):

  • OS: Ubuntu 22.04

  • Middleware: ROS 2 Humble

  • Autopilot: PX4 v1.14

  • Bridge: Micro XRCE-DDS (uXRCE-DDS)

  • Simulator: Gazebo Garden

  • Sensors: Depth Camera (publishing PointCloud2 and RGB images)

System Pipeline:

  1. Vision: HSV filter on the RGB image to isolate a red sphere. I extract the corresponding points from the PointCloud2 data and calculate the centroid.

  2. Estimation: I feed the centroid into an IMM (Interacting Multiple Model) filter to estimate the target’s position and velocity.

  3. Control: The IMM output feeds a controller (I’ve tried both PID and MPC), which generates Offboard mode setpoints for PX4.

The Problem (Symptoms): The drone flies very erratically:

  • Sometimes it crashes into the sphere (fails to regulate distance).

  • Sometimes it tracks it, but suddenly loses the target and hovers in place or drifts away.

  • Other times it detects the sphere but doesn’t move towards it, or moves in the wrong direction entirely.

Main Suspicion: I strongly suspect the IMM is the culprit. I feel it might be introducing too much noise or delay into the velocity estimation, causing both the PID and MPC to make violent or incorrect decisions. However, I’m not sure how to isolate it to confirm this.

What I’ve Tried:

  • Tuning PID gains and MPC horizons (both fail similarly).

  • Modifying the IMM’s covariance and transition probability matrices.

  • Verifying that the TF tree is correct (camera_link to base_link).

Specific Questions:

  1. Diagnosing the Estimator: How can I diagnose if the IMM is to blame? Do you recommend disabling it temporarily and using just the raw centroid or a simple Kalman Filter to see if the drone’s behavior improves?

  2. IMM + MPC Dynamics: Is MPC generally too sensitive to noisy IMM estimates? Would it be better to use a position PID and leave the IMM velocity solely as a feedforward term?

  3. Offboard Latency: Processing PointCloud2 + IMM + PID/MPC can take a long time. Does anyone know the minimum safe frequency limit for Offboard mode in PX4 v1.14, and how to avoid timeout issues? (Would you recommend applying a VoxelGrid filter to downsample the point cloud?).

  4. Known Issues: Has anyone successfully integrated an IMM with PX4 in Gazebo Garden? Are there any known issues with this specific Gazebo Garden version and the uXRCE-DDS bridge on ROS 2 Humble?

Thanks in advance for any insights!

2

I would avoid assuming the IMM is the root cause yet. Since both PID and MPC show similar failures, I would first isolate the common pipeline and identify where the first divergence appears.

A useful test would be to record, on the same clock:

image/depth capture → detected centroid → transformed target position → IMM state → controller setpoint → PX4 vehicle state

Then run a few deliberately simplified experiments.

First, keep the target stationary and command no vehicle motion. Check whether the transformed centroid remains stationary in the frame expected by the controller. This can expose frame/sign/axis errors before involving either IMM or control.

Second, bypass the IMM and feed a filtered centroid position to a very conservative position controller. If the wrong-direction or violent-motion behaviour remains, the IMM is probably not the initiating fault.

Third, keep the IMM but log both its input and output timestamps. I would look at measurement age, not only computation time. A perfectly stable estimate that represents the target 200–300 ms in the past can still produce poor following behaviour.

I would also separate two different rates:

  • the rate at which perception/IMM produces a new target estimate
  • the rate at which the Offboard control interface is kept alive / setpoints are published

PX4’s Offboard requirements should not be used as the target frequency for the perception pipeline. Keep the PX4 control/heartbeat path deterministic and sufficiently frequent even when a new camera estimate is not available.

For debugging, I’d build one timeline from a failing run and find the first observable divergence:

centroid → TF → IMM → control error/setpoint → PX4 response

In particular, for a run where the vehicle moves in the wrong direction, check the signs and coordinate frames of the target vector immediately before the first incorrect setpoint. That case may be more diagnostic than a crash because it can distinguish perception/TF errors from estimator lag and controller instability.

If you can share a ROS 2 bag plus the corresponding PX4 ULog from one short failing run, especially one where the sphere is detected but the vehicle initially moves in the wrong direction, it should be possible to narrow this down much more systematically.