Delayed response to RC sticks (Roll/Pitch/Yaw) in Position mode only


Drone Setup:

Autopilot: Pixhawk 6C
Firmware: PX4 1.14.0
GPS: None
Local Positioning Sensor: ARKFlow (connected at CAN port)
Range Finder: LIDAR-Lite V3

Today, I just tuned my new drone frame in “Altitude” mode.
It went well and I can fly freely in Altitude mode.

But, as soon as I started flying in Position mode, the response to RC sticks is very bad. Upon doing the movements from RC sticks, the drone response is delayed it takes almost 1-2 seconds to the response and the response is also not seems stable.

Altitude flight mode log: (Control is almost fine)

Position flight mode log: (delayed response to RC sticks)

Position flight mode log: (delayed response to RC sticks)

My Hunch!

Is this something related to any of the following?
1- Attitude Controller

2- Velocity Controller

3- MC Filter Tuning & Control Latency