Offboard hovering issue (with good position estimation)

I encountered the hovering issue with offboard flight mode. My test is to let the drone take off at (0, 0, 1) m position and keep hovering. I have made sure that the localization data is accurate and correct. However, when I use the offboard mode to take off the drone, it will go back and forth in all x, y, and z directions.

I have three different drone models with the same sensor, flight controller, and VIO algorithms. They all experience the same issues, so I posted all three videos and data here.

MODEL #1 (CUAV Nora Flight controller + D435i for VIO):
Flight record Video:

Localization Video:

Flight log link:
https://review.px4.io/plot_app?log=20665e05-fd91-432a-b294-23e16341107b

MODEL #2 (CUAV Nora Flight controller + D435i for VIO):
Flight record Video:

Localization Video (CUAV Nora Flight controller + D435i for VIO):

Flight log link:
https://review.px4.io/plot_app?log=cf5d3bbf-d06d-4493-a24e-4962719e1e9f

MODEL #3
Flight record video:

Localization video:

Flight log link:
https://review.px4.io/plot_app?log=303570ce-f902-46c7-b47c-270de1c6d733

I think the localization is fine, and it might be the flight controller’s problem?
THanks in advance!