Delayed response to RC sticks (Roll/Pitch/Yaw) in Position mode only

Hi,

Drone Setup:

Autopilot: Pixhawk 6C
Firmware: PX4 1.14.0
GPS: None
Local Positioning Sensor: ARKFlow (connected at CAN port)
Range Finder: LIDAR-Lite V3

Today, I just tuned my new drone frame in “Altitude” mode.
It went well and I can fly freely in Altitude mode.

But, as soon as I started flying in Position mode, the response to RC sticks is very bad. Upon doing the movements from RC sticks, the drone response is delayed it takes almost 1-2 seconds to the response and the response is also not seems stable.

Altitude flight mode log: (Control is almost fine)
https://review.px4.io/plot_app?log=63a3002b-acdd-448f-b6f1-64aa8f335151

Position flight mode log: (delayed response to RC sticks)
https://review.px4.io/plot_app?log=df54291a-da49-439d-90e6-acc6e7f6f850

Position flight mode log: (delayed response to RC sticks)
https://review.px4.io/plot_app?log=3b2384e8-8aba-4b11-926c-ddf89fb29799

My Hunch!

Is this something related to any of the following?
1- Attitude Controller

2- Velocity Controller

3- MC Filter Tuning & Control Latency