Too much warning messages while controlling multiple vehicles through MAVSDK on PX4-SITL-Gazebo

gwn · March 9, 2022, 7:26am

At the beginning, I want to express that I am facing this issue with Mavsdk version:0.48 and PX4 firmware version: 1.11.0.

I am starting multiple quadrotors on PX4-Sitl-Gazebo via gazebo_multi_vehicle.sh script. Later I am connecting to the autopilots and controlling the vehicles in offboard mode. When I do this with 1 to 4 drones, I do not receive any warning messages. But when I work with more drones, I am constantly receving following messages:

[12:53:32|Info ] heartbeats timed out (system_impl.cpp:268)
[12:53:32|Debug] Discovered 1 component(s) (system_impl.cpp:525)
[12:53:32|Warn ] command denied (512). (mavlink_commands.cpp:179)
[12:53:33|Debug] Falling back to gimbal protocol v1 (mission_impl.cpp:131)

My script remains connected to the autopilot and offboard connection failsafe is not being triggered, but it is still annoying to get so many warning messages.

I am performing following actions in order:

Arm
Takeoff to a specific altitude and wait until takeoff completes
Start offboard mode
Command position and heading in 10Hz
When my mission is completed, I stop offboard mode
Command land
Disarm

During these operations I am periodically receiving telemetry messages the autopilot.

Can anyone help me about this issue? Am I doing anything wrong or do you have any suggestion for me? For temporary solution for me to observe my own debug messages, is there any way to suppress the warning messages of MavSDK?

JulianOes · March 23, 2022, 7:05am

Ok, let’s see what we can do here. First, we can suppress them, as you suggest, so treat the symptom.

This can be done with an interface that allows you to process the debug messages before they are published:

github.com

mavlink/MAVSDK/blob/53dbea82a3f62c123a6323ab5815bdec023987a8/src/mavsdk/core/include/mavsdk/log_callback.h#L18

    
      
          // defined numeric values can be useful for filtering message by severity
          enum class Level : int { Debug = 0, Info = 1, Warn = 2, Err = 3 };
          
          
/** @brief User-defined callback for logging. Returning true from this callback
           * prevents default mavsdk`s logging to stdout. Returning false keeps it.
           */
          using Callback =
              std::function<bool(Level level, const std::string& message, const std::string& file, int line)>;
          
          
extern Callback& get_callback();
          extern void subscribe(const Callback& callback);
          
          
} // namespace mavsdk::log

So you need to implement to call this subscribe function and give it a callback. This callback will then be called for each printf with the log level, message, file name, and line number. If your callback returns true` but doesn’t print anything, it would basically silent all output.
(If you don’t have this API in your version, you would have to update.)

Now, what I would recommend instead is to actually treat the condition, rather than just the symptom. Let’s look at the messages:

[12:53:32|Info ] heartbeats timed out (system_impl.cpp:268)
[12:53:32|Debug] Discovered 1 component(s) (system_impl.cpp:525)

You say that you see them more often with multiple vehicles. The reason is very likely that the simulation runs slower on your computer as you add more vehicles, so then things take longer and time out every now and then. That’s somewhat expected. If you want to prevent that, you would have to change the timeouts or use a faster computer (or simulation that takes CPU, etc…
If you build mavsdk yourself, you could just increase this timeout:

github.com

mavlink/MAVSDK/blob/53dbea82a3f62c123a6323ab5815bdec023987a8/src/mavsdk/core/system_impl.h#L420

    
      
          std::atomic<bool> _hitl_enabled{false};
          bool _always_connected{false};
          
          
std::atomic<Autopilot> _autopilot{Autopilot::Unknown};
          
          
MavsdkImpl& _parent;
          
          
std::thread* _system_thread{nullptr};
          std::atomic<bool> _should_exit{false};
          
          
static constexpr double HEARTBEAT_TIMEOUT_S = 3.0;
          
          
std::mutex _connection_mutex{};
          std::atomic<bool> _connected{false};
          System::IsConnectedCallback _is_connected_callback{nullptr};
          void* _heartbeat_timeout_cookie = nullptr;
          
          
std::atomic<bool> _autopilot_version_pending{false};
          
          
static constexpr double _ping_interval_s = 5.0;

[12:53:32|Warn ] command denied (512). (mavlink_commands.cpp:179)

Command 512 requests a message: Messages (common) · MAVLink Developer Guide
Presumably it’s one that is not yet supported by your PX4 version. In this case it probably does not support the gimbal v1 protocol yet. This is no problem as such but I can understand how it’s a bit annoying. I will put it on my list to get rid of that one for this case to avoid confusion.

My script remains connected to the autopilot and offboard connection failsafe is not being triggered, but it is still annoying to get so many warning messages.

I’m not sure what you mean with this one. Is this an actual question/problem as well, or just a remark?

gwn · March 30, 2022, 7:48am

Hi @JulianOes ,

Thanks for your solutions.
I have implemented your solution (using log_callback) for filtering and suppressing the debug logs. It helped me a lot.

I was suspicious about dependency on the computer performance while running multiple SITL instances and communication with autopilots via MAVSDK. I think your suggestion will help me on this issue. I will try adjusting HEARTBEAT_TIMEOUT_S value and write back the result here.

About the last part, that was not an actual problem, instead it was just a remark.