Holybro H-FLOW - highly inaccurate + drift

Hi all.

We have a drone that we are trying to fly in offboard mode using the Holybro H-Flow sensor with our custom flight controller.

We are having some issues with the accuracy of the flow sensor - I did a test flight where I was manually flying the drone North → West → South → East in a square - But the drone came out around 45 degrees offset with the flow sensor and with a large offset between take off and landing spot (IRL it was within 1-2 meters). The sensor it self is mounted 90 degrees rotated, and SENS_FLOW_ROT was set to 90 degrees (option 2). When we are flying, we also see the position drifting a lot.

There is a flight log available here:

Here is the yellow square how we approximately flew, and the red line shows how the drone thinks it flew.

I have tried a indoor test with the H-Flow sensor, and sometimes I get some kind of state, where the sensor is constantly drifting.

In this picture, I manually held the drone and moved it back and forward in a north-south direction - On the distance sensor, we can see the movement as I cant keep the drone in a constant height, but on the LOCAL_POSITION_NED, it have drifted 44 meters north? We are using EKF2_OF_QMIN of 1, so it should always try to use the OF sensor.

We have followed the guide here: