Deon,
Questions and answers below:
Q) Could you send me any mathematical papers you have describing the maths behind EKF2?
Ans) The Khosrovian paper listed in the documentation was the original inspiration for the output observer, however the output observers current implementation has been modified substantially from that. For the state equations, covariance prediction, etc, see the derivation script that tumbili has listed .
Q) Should we also be looking at changing EKF2_TAU_POS & EKF2_TAU_VEL ? (this seems to be a complimentary filter time-constant that combines previous state and current state measurements in order to get rid of some of the delays, but I’m not 100% sure)
Ans) The role of the output filter is two-fold. Firstly to predict forward from the EKF’s fusion time horizon to current time and secondly to filter out observation noise. I would be looking to reduce the time constants if you have reasonably low noise aiding data (good GPS). The EST4.e1 EST4.e2 and EST4.e3 show the magnitude of the angular, velocity and position tracking error of the output predictor measured at the fusion time horizon. You can use the velocity (EST4.e2) and position (EST4.e3) errors to tune your time constants for the required compromise between tracking accuracy and observation noise.
Q) What other parameters would you suggest we should change in order to optimize our EKF2 for the uBlox RTK system?
Ans) One thing to try for better accuracy is to tune the value of EKF2_GPS_DELAY to match your receiver. The default is value is 200msec which was the value for the old UBlox 6 receiver hardware. If your receiver has lower delay, then you can alse reduce the size of the EKF time horizon delay by reducing https://github.com/PX4/ecl/blob/master/EKF/estimator_interface.h#L277 from the current value of 300msec (30 samples) to a value about 20msec larger than your GPS delay. That will tighten output predictor tracking significantly.
Finding the optimum value of EKF2_GPS_DELAY is best done using the EKF2 replay, adjusting the value of the parameter and picking the value that minimises the RMS GPS velocity innovations.
Q) I have looked at your InertialNav repository on GitHub, is this something we could use to optimize our EKF2 parameters?
Ans) I second the recommendation posted by tumbili for this one. Using the replay tool is a good option, but you will need a good SD card. This page has some information on different card brands: http://dev.px4.io/advanced-logging.html