Unstable or not connecting companion computer on Mavlink, using mavlink-routerd

Hi,
on 2 drones, that both make use of px4 1.14 on Holybro6x, i want to exchange mavlink messages between the FMU, an embedded companion computer and a remote GCS. This sounds standard but i face some issues or unstable behaviors.

Here are some details, as a new user, i must have done some basic mistakes, if you see this, tanks you !

First, Mavlink-routerd ahes been compiled and installed on the companion computer that is connected over ethernet to the FMU. In /etc/mavlink-router/main.conf, i got this kind of config:

[General]
TcpServerPort=5760
ReportStats=false

[UdpEndpoint fmu]
Mode=normal
Address=192.168.0.9
Port=14540

[UdpEndpoint onboardserver]
Mode=server
Address=0.0.0.0 #=> i had to use it, 127.0.0.1 could not make it work
Port=14540

[UdpEndpoint companion]
Mode=normal
Address=0.0.0.0 #=> also tried with 127.0.0.1, same results
Port=14550 #=> one specific port for the companion

[UdpEndpoint remote1]
Mode=normal
Address=192.168.x.z
Port=14540

[UdpEndpoint remote2]
Mode=normal
Address=192.168.x.y
Port=14540

When connecting to GCS, Qgroundstation always sees the FMU data so the FMU-companion(mavlink-routerd)-GCS works fine, every member communicates on port 14540 over its interface: FMU-companion communicate over ethernet while companion-GCS comunicate other wifi on different IP ranges.
Even if sometimes, the connexion of GCS is slow, sometimes waiting for somethink before the FMU data becomes available)

However, when i just need to create a python script, on the companion computer to listen to mavlink messages, troubles start and are unstable. I first need a listener as the basic MAVSDK official example https://github.com/mavlink/MAVSDK-Python/blob/cd941eb6dacd297c4ec5567e9a20ce5fcd3dfde0/examples/telemetry.py. Here are the observed behaviors:

  • On the first drone (Holybro6X+rpi CM4), it works most of the time but in some cases i have to restart mavlink-routerd service to see the basic mavlink messages from the script. When there is an issue, this is actually an infinite waiting on the connection command ( await drone.connect(system_address=“udp://:14550”)
  • On the second one (Holybro6X+Nvidia jetson nano), the python script never connects. As for the first drone, the connect command fails, everytime in ths case.

Then, How to solve this issue ?
Using the same config, on the first drone, i had this always not connect issue that i could solve by adding a delay when starting the mavlink-routerd service. Maybe this is necessayr to let the FMU start before being troubled by the mavlink-routerd messages ?
Here is my service setup file:

IN /lib/systemd/system/mavlink-router.service
[Unit]
Description=MAVLink Router

[Service]
Type=simple
ExecStartPre=/bin/sleep 20 # => delay added here, facilitates FMU connexion on the first drone (only)
After=network-online.target
ExecStart=/usr/bin/mavlink-routerd
Restart=always
StartLimitIntervalSec=10
StartLimitBurst=10
StartLimitInterval=20

[Install]
WantedBy=multi-user.target

But in the second drone, increasing this delay does not solve the issue. My understanding of the problem is limited. I am sure, i missed something basic.

Thanks for your help

@JulianOes Anything obvious here?

Do both drones have different MAVLink system IDs?

MAV_SYS_ID

Hi, yes then do, moreover, i test them separately.
Actually, each has a different companion computer. Each one is connected to its own holybro 6x with px4 1.14 over ethernet and for both mavlink message streaming to GCS over wifi through mavlink-routerd installed on the companion works fine.
First drone’s companion is a pi cm4 for which the mavsdk telemetry.py example script mostly works (sometimes cannot connect to the stream for unknown reason). On the second drone, companion is a nvidia jetson nano avec the telemetry script never wants to connect to the stream.

Looking for the solution… problem may be simple but cannot figure out.

To clarify, each drone is used separately, on a different wifi network so potential MAV_SYS_ID related issues should not occur. I consider a common network architecture with :

onboard[FMU<-ethernet->companion(hosts mavilnk-routerd)] <-wifi-> GCS

In this config Mavlink messages exchange between FMU and GCS are ok. The main issue is at the companion level :

Companion runs :

  1. mavlink-routerd, connected on all interfaces on port 14540, see config above, This seems to work fine
  2. a local script (typically the telemetry.py mavsdk example script) that should read mavlink messages from port 14550 issues are here. On drone 1(CM4 companion), sometimes it cannot connect, on drone 2(Jetson Nano companion), it never connects.

Hope this is clearer.
Thanks

There is a connection debug guide for mavsdk Python scripts:
http://mavsdk-python-docs.s3-website.eu-central-1.amazonaws.com/#debug-connection-issues

Hi,
actually, i obtain the same result as shown on the link you provided when the connection is waiting indefinitely:

/usr/local/lib/python3.7/dist-packages/mavsdk/bin/mavsdk_server  udp://:14550
[07:11:57|Info ] MAVSDK version: v1.4.17 (mavsdk_impl.cpp:20)
[07:11:57|Info ] Waiting to discover system on udp://:14550... (connection_initiator.h:20)

No more information. this looks strange and… blocking. What to check to get further ?
Thanks.

So nothing is connected just yet.

So mavlink-router is forwarding to UDP port 14550 on localhost, the same host that mavsdk_server is running on?

Hi, yes, mavlink-routerd and the monitoring script i want to run (or here mavsdk_server) are both running on the same companion computer (nvidia jetson nano, connected on the ethernet port). I use the same mavlink-routerd config as shown in my first script, the address of the companion UdpEndpoint can be 0.0.0.0, 127.0.0.1 or other, same issue…
but it works on my other test plateform, the companion being a rpi CM4 (but sometimes, no connection also occurs).

Thanks for your help

Solved, this was all related to Nvidia Jetson nano default installation. Communication port has to be explicitly declared (contrary to the rpi debian install). Here are the guidelines for memory:

  1. First check if port is open
    sudo iptables -L -n | grep 14550

  2. if not, create/update /etc/iptables using below command:
    sudo mkdir /etc/iptables

  3. execute below commands to open port 14550 (as for my use case) to transmit and receive data:

sudo iptables -A INPUT -p udp --dport 14550 -j ACCEPT
sudo iptables -A OUTPUT -p udp --dport 14550 -j ACCEPT
sudo iptables-save | sudo tee /etc/iptables/rules.v4
  1. reboot and test
1 Like