Pricise altitude using baro

Dear all,
I want to use baro to obtain a precise altitude, and I use position_estimate_inav. I find that the altitude drifts severly during flight, even if I hold it still (never move) and watch the local_position_ned.z in QGC, it could drift -2 meters after a couple of minutes, and continue drifting to even -5 meters. I think this value must be precise if I want to get the absolute altitude during flight. Is there any way to make local_position_ned.z stable?
Thanks in advance.

We have switched the default to LPE. INAV is limited in precision.

Thanks for the reply. I tried LPE, but in QGC, I see no param representing the absolute altitude.

Was your QNH changing during your observation interval?

Sorry for my unfamiliarity. What is QNH? Where can I see it?

Thanks for the reply.
Then, how can I see the QNH? Is it observable on QGC?

I use this service to get the barometric pressure at a particular location: http://www.aviationweather.gov/metar. QGC doesn’t currently do it for you, though it would be a very useful capability.