Hello,
I’m encountering an issue where Ulog should be streamed over mavlink from PX4 to an off board system, sometimes do not. This issue surfaces due to offboard mode sending an unconventional sequence of logging commands to PX4 - stopping logging immediately followed by starting logging again.
After investigating the problem, it appears that this command sequence exposes an underlying synchronization issue in PX4’s mavlink logging functionality. PX4 handles mavlink logging commands through three separate threads - the mavlink receiver thread, the mavlink main thread, and the logger task.
The root cause of the issue is that there is a lack of synchronization between the mavlink receiver thread and the mavlink main thread when handling the logging start/stop commands. Specifically:
- When PX4 receives a “stop logging” command:
- The logger task acknowledges and stops its logging.
- The acknowledgment is forwarded to the mavlink main thread.
- However, the mavlink main thread has not yet stopped its own logging.
- If offboard mode then sends a “start logging” command:
- The logger task will restart its logging.
- But the mavlink main thread will finish stopping its logging from the previous command, prematurely stopping the logging before it could be restarted.
This race condition between the two mavlink threads leads to the failure to properly handle the logging start/stop sequence, resulting in the missing Ulog files on offboard.
A contributing factor to this issue is that acknowledgments for the logging commands are sent before the threads fully complete their work.
Is this a known issue? Might there be a requirement to wait a certain time between sending a stop logging command and a start logging command?
Simple solutions are also welcome
Thanks!