Too much warning messages while controlling multiple vehicles through MAVSDK on PX4-SITL-Gazebo

At the beginning, I want to express that I am facing this issue with Mavsdk version:0.48 and PX4 firmware version: 1.11.0.

I am starting multiple quadrotors on PX4-Sitl-Gazebo via gazebo_multi_vehicle.sh script. Later I am connecting to the autopilots and controlling the vehicles in offboard mode. When I do this with 1 to 4 drones, I do not receive any warning messages. But when I work with more drones, I am constantly receving following messages:

[12:53:32|Info ] heartbeats timed out (system_impl.cpp:268)
[12:53:32|Debug] Discovered 1 component(s) (system_impl.cpp:525)
[12:53:32|Warn ] command denied (512). (mavlink_commands.cpp:179)
[12:53:33|Debug] Falling back to gimbal protocol v1 (mission_impl.cpp:131)

My script remains connected to the autopilot and offboard connection failsafe is not being triggered, but it is still annoying to get so many warning messages.

I am performing following actions in order:

  1. Arm
  2. Takeoff to a specific altitude and wait until takeoff completes
  3. Start offboard mode
  4. Command position and heading in 10Hz
  5. When my mission is completed, I stop offboard mode
  6. Command land
  7. Disarm

During these operations I am periodically receiving telemetry messages the autopilot.

Can anyone help me about this issue? Am I doing anything wrong or do you have any suggestion for me? For temporary solution for me to observe my own debug messages, is there any way to suppress the warning messages of MavSDK?

Ok, let’s see what we can do here. First, we can suppress them, as you suggest, so treat the symptom.

This can be done with an interface that allows you to process the debug messages before they are published:

So you need to implement to call this subscribe function and give it a callback. This callback will then be called for each printf with the log level, message, file name, and line number. If your callback returns true` but doesn’t print anything, it would basically silent all output.
(If you don’t have this API in your version, you would have to update.)


Now, what I would recommend instead is to actually treat the condition, rather than just the symptom. Let’s look at the messages:

[12:53:32|Info ] heartbeats timed out (system_impl.cpp:268)
[12:53:32|Debug] Discovered 1 component(s) (system_impl.cpp:525)

You say that you see them more often with multiple vehicles. The reason is very likely that the simulation runs slower on your computer as you add more vehicles, so then things take longer and time out every now and then. That’s somewhat expected. If you want to prevent that, you would have to change the timeouts or use a faster computer (or simulation that takes CPU, etc…
If you build mavsdk yourself, you could just increase this timeout:

[12:53:32|Warn ] command denied (512). (mavlink_commands.cpp:179)

Command 512 requests a message: Messages (common) · MAVLink Developer Guide
Presumably it’s one that is not yet supported by your PX4 version. In this case it probably does not support the gimbal v1 protocol yet. This is no problem as such but I can understand how it’s a bit annoying. I will put it on my list to get rid of that one for this case to avoid confusion.


My script remains connected to the autopilot and offboard connection failsafe is not being triggered, but it is still annoying to get so many warning messages.

I’m not sure what you mean with this one. Is this an actual question/problem as well, or just a remark?

2 Likes

Hi @JulianOes ,

Thanks for your solutions.
I have implemented your solution (using log_callback) for filtering and suppressing the debug logs. It helped me a lot.

I was suspicious about dependency on the computer performance while running multiple SITL instances and communication with autopilots via MAVSDK. I think your suggestion will help me on this issue. I will try adjusting HEARTBEAT_TIMEOUT_S value and write back the result here.

About the last part, that was not an actual problem, instead it was just a remark.