What optical flow sensor are you using? I have a similar issue which still is unsolved but seems to be sensor related and not due to any parameter / setup settings. You can find a detailed description about the issue and any troubleshooting done so far here: UAV drifts away when hovering based on optical flow
