From my understanding of PX4 and how sensors work, without some kind of distance_to_ground metric, your UAV will drift.
For one, GPS height is much less accurate than XY position estimation. It can fluctuate more than 1m in the Z axis. If your scenarios define flight at high altitude, this mostly isn’t a concern.
Barometer, on the other hand, is much more precise (+/- 10cm, or maybe even less), but it is vulnerable to static pressure changes. Let’s say you take off from some location at time X. After a while, the atmospheric pressure at the starting location has changed (if not, you are living in a low-resolution simulation xD). So, this also changed your internal height above the home position - even if the drone is still hovering at the same height. There is a lot to discover about it and how EKF tries to compensate for that (I’m still trying to fully understand it).
In short, the barometer will drift over time. GPS would not be accurate enough.
I encountered this issue before as well. Later, during flight tests, I made it a practice to keep the drone stationary on the ground for at least 30 seconds before arming it. After arming, I waited another 5 seconds before operating the joysticks to take off. This was done to allow sufficient time for the barometer and EKF2 to complete their calibration and correction processes, which helped alleviate the problem to a certain extent.
