I'm getting very similar results to you, which makes me think it's not a hardware issue.
I'm using a DJI Flamewheel with PX4Flow and LidarLite v3.
I find that even when my drone is at rest (nothing armed, no motors on) with all sensors seeing the ground, I get a constant drift in the LPE.
I've started looking at the raw sensor data from each sensor that's going into the LPE. One thing I've found is that even when my vehicle is level, it puts out a constant X acceleration of about 0.05. It's noisy, as expected, but I would expect the noise to be centered at zero. The offset is high enough that none of the noisy points get below zero. I re-calibrated the IMU and it did not help.
I think the LPE is a pretty new addition to the PX4 family, so maybe there's a bug that hasn't been found yet? I haven't been able to find such a bug, if it exists.