Quadcopter Suddenly Losing Altitude When Hovering for an Extended Time

I have not been having this problem until recently, or I just didn’t notice. After I take my quad off and bring it over 50ft (where lidar is no longer fused) and I allow the craft to hover for about 30 seconds and it began to lose altitude, until it seemed like the barometer data caught up with it, then the quad shot up very quickly, overshot the altitude setpoint and came down quickly again, finally regaining the altitude. What I have tried so far:
Increasing EKF2_BARO_GATE = 15 (from 5) - seemed to help some
Decreasing IMU_GYRO_CUTOFF = 40 (from 80) - Made it hold altitude for about 5 minutes before losing altitude.

I should mention that there are 20+MPH winds so the following logs are pretty noisy (about 20% over normal)
I am using firmware version 1.8.2 and flying on a Pixhawk 2.1 Cube.

I have read that vibrations in the Z directions can cause this but I’m unsure of the boundaries. Any Insight would be appreciated.

log with altitude loss (20+mph winds):

normal log with 5-10mph winds (flying on lidar):

Today I tested the situation again. I flew the drone to 100ft and let it hover. It seemed to be doing well until it began to lose altitude. Updon losing altitude, the drone shot up to 400ft indefinately, and was issued an RTL (via geofence) but came down way too fast, hit the ground and shot off into some grass. The vibration looked fine although, the vibration became worse and worse in the logs. Can someone give me a little insight on troubleshooting this issue?