UAV drifts away when hovering based on optical flow


Situation & Issues:
The sensors all work perfectly fine, required topics (e. g. OPTICAL_FLOW_RAD & DISTANCE_SENSOR) can be monitored and show reasonable values, and UAV seems to hold position after takeoff. Altitude is held perfectly over entire flight duration. The problem is, that it still has a slight drift (roughly 1 meter per minute), which is not sufficient for mission target. The log files show that the estimator assumes the drone to oscillate with minor amplitudes around the initial position while the GPS sensor measures a significant drift. The drift direction is fixed in body frame, it is independent of wind direction. The log files also show that the estimator is good when the drone is moving: when manually flown back to its initial position the position shift looks similar recognized by estimator and by GPS (flight path has same shapes, only a GPS offset is given).

Internet research leads to the following results for Troubleshooting:

  1. “Calibrate the sensor.” (usually leads to here: Optical Flow Sensor Testing and Setup — Copter documentation).
    As I am using PX4, not Ardupilot, I could not find a way to perform the calibrations – I could not find a PX4 equivalent for parameters like FLOW_FXSCALER etc. Also, HereFlow setup does not include any calibrations (see link above)

  2. “Exact values for Offset are very important.”
    I used very exact values, but also tried to strongly detune via setting big offset values – the effects of high offsets were too neglegtible to explain the drift by having not exact enough offset values

  3. “The ground surface plays an important role.”
    I performed the tests above an asphalt surface with many features and no elements moving due to downwash / wind in an adequate altitude (ca. 5 m above ground).
    At testing I found the following:
    a. Position offset parameters do barely affect the drift
    b. (Not) including Multi rotor drag fusion in the EKF has no effect on the drift
    c. Enabling UAVCAN_SUB_IMU (using the IMU of the HereFlow Sensor) leads to an instable flight behavior (strong swings / oscillations of the drone in xy-plane around setpoint position).
    d. Recalibration of level horizon had no effect on the drift
    e. PX4Flow roation parameter has no effect when HereFlow is used (as expected)
    f. Slight adjustments to the orientation / attachment of the sensor on the drone show significant effects on the drift
    Changing the pitch angle and the yaw angle of the sensor relatively to the Pixhawk by beveling by clamping thin objects under the sensor’s circuit board affected the drift direction and magnitude. The problem here is that I could not find parameters or calibration methods to compensate for this. Also, even the smallest changes create visible effects on the drift, which is why I can’t imagine that perfect physical alignment is achievable when mounting.

My questions:
I. Is there a way to manually calibrate the sensor / compensate for sensor orientation offsets when used on a PX4 setup?
II. Is there a tutorial or best practice for perfectly mounting the Sensor that has proven to be successful?
III. Can this be solved by adapting the EKF and how can this be achieved?
IV. What is another potential source of error?

Please find some additional information below.
Many thanks in advance for your contribution,

Add-on: More detailed information:
Here is some more information on the test flights and the log files.
As I can not post enough links in this post, I put the descriptions of setup changes and the links to the logfiles in the feedback section of this flight log:
Sorry for the inconvencience.

Here is some additional plots of flight data. If not stated otherwise, the x-axis shows the frames. As the optical flow (OF) sensor has double the frame rate as the GPS sensor, those value ranges resemble the same timespan for both sensors. Positions are given in meters, velocities in m/s. Major position and velocity changes are due to manual inputs to compensate the drift. Unfortunately, number of embedded files to post is limited, but further plots can be provided on request.

Here you can see the drift, and how drift direction turns after the 90 deg yaw:

Shows the drift:

Shows, how the OF-based estimator brings a velocity which results in drift: