Attitude control anomaly in auto mode

We were out doing some test flights today and were encountering some strange behavior in auto mode… Whenever there was a yaw movement the pitch and roll would become unstable.

Take time 10:49.2 from this log as an example:

https://review.px4.io/plot_app?log=5e2c766b-6fe4-4da4-b367-5a918da86a3b

Zoom into the roll angle and roll angular rate charts. At this instant a significant positive change is commanded on roll angle setpoint, but a significant negative change is commanded on roll angular rate? How does this make sense?

Also looking at the time from 10:49.2 to 10:51.2 the roll setpoint is consistently above the roll estimate, yet the commanded angular rate oscillates up an down. Why?

Same instant has a similar problem on pitch. Feel free to check any other times the yaw changes when in auto mode, there is likely a similar problem on pitch/roll.

This is custom branch based on 1.9.0. We have not made any changes to the attitude control system.

This system flies well in position mode and altitude mode.

What could be the problem here?