EKF fuses vision data, but drifts and ignores vision in around a minute

Using an external vision sensor (HTC Vive) to provide position estimate to pixhawk EKF2.
The EKF fuses the vision information for a few tens of seconds, but the estimate starts drifting and resets to the x=y=~0 origin.
The logs of a test are here: https://logs.px4.io/plot_app?log=f16ec023-ecad-43cc-975e-2e7bfaf7deca .
The specific test sequence was
Pitch up once
Roll right thrice
Yaw right 4 times.

A video of Rviz can also be here: https://streamable.com/m1201

@Paul_Riseborough Is this something you can comment on?

Any help on how to debug this ?