Unstable or not connecting companion computer on Mavlink, using mavlink-routerd

Hi,
on 2 drones, that both make use of px4 1.14 on Holybro6x, i want to exchange mavlink messages between the FMU, an embedded companion computer and a remote GCS. This sounds standard but i face some issues or unstable behaviors.

Here are some details, as a new user, i must have done some basic mistakes, if you see this, tanks you !

First, Mavlink-routerd ahes been compiled and installed on the companion computer that is connected over ethernet to the FMU. In /etc/mavlink-router/main.conf, i got this kind of config:

[General]
TcpServerPort=5760
ReportStats=false

[UdpEndpoint fmu]
Mode=normal
Address=192.168.0.9
Port=14540

[UdpEndpoint onboardserver]
Mode=server
Address=0.0.0.0 #=> i had to use it, 127.0.0.1 could not make it work
Port=14540

[UdpEndpoint companion]
Mode=normal
Address=0.0.0.0 #=> also tried with 127.0.0.1, same results
Port=14550 #=> one specific port for the companion

[UdpEndpoint remote1]
Mode=normal
Address=192.168.x.z
Port=14540

[UdpEndpoint remote2]
Mode=normal
Address=192.168.x.y
Port=14540

When connecting to GCS, Qgroundstation always sees the FMU data so the FMU-companion(mavlink-routerd)-GCS works fine, every member communicates on port 14540 over its interface: FMU-companion communicate over ethernet while companion-GCS comunicate other wifi on different IP ranges.
Even if sometimes, the connexion of GCS is slow, sometimes waiting for somethink before the FMU data becomes available)

However, when i just need to create a python script, on the companion computer to listen to mavlink messages, troubles start and are unstable. I first need a listener as the basic MAVSDK official example https://github.com/mavlink/MAVSDK-Python/blob/cd941eb6dacd297c4ec5567e9a20ce5fcd3dfde0/examples/telemetry.py. Here are the observed behaviors:

  • On the first drone (Holybro6X+rpi CM4), it works most of the time but in some cases i have to restart mavlink-routerd service to see the basic mavlink messages from the script. When there is an issue, this is actually an infinite waiting on the connection command ( await drone.connect(system_address=“udp://:14550”)
  • On the second one (Holybro6X+Nvidia jetson nano), the python script never connects. As for the first drone, the connect command fails, everytime in ths case.

Then, How to solve this issue ?
Using the same config, on the first drone, i had this always not connect issue that i could solve by adding a delay when starting the mavlink-routerd service. Maybe this is necessayr to let the FMU start before being troubled by the mavlink-routerd messages ?
Here is my service setup file:

IN /lib/systemd/system/mavlink-router.service
[Unit]
Description=MAVLink Router

[Service]
Type=simple
ExecStartPre=/bin/sleep 20 # => delay added here, facilitates FMU connexion on the first drone (only)
After=network-online.target
ExecStart=/usr/bin/mavlink-routerd
Restart=always
StartLimitIntervalSec=10
StartLimitBurst=10
StartLimitInterval=20

[Install]
WantedBy=multi-user.target

But in the second drone, increasing this delay does not solve the issue. My understanding of the problem is limited. I am sure, i missed something basic.

Thanks for your help

@JulianOes Anything obvious here?

Do both drones have different MAVLink system IDs?

MAV_SYS_ID

Hi, yes then do, moreover, i test them separately.
Actually, each has a different companion computer. Each one is connected to its own holybro 6x with px4 1.14 over ethernet and for both mavlink message streaming to GCS over wifi through mavlink-routerd installed on the companion works fine.
First drone’s companion is a pi cm4 for which the mavsdk telemetry.py example script mostly works (sometimes cannot connect to the stream for unknown reason). On the second drone, companion is a nvidia jetson nano avec the telemetry script never wants to connect to the stream.

Looking for the solution… problem may be simple but cannot figure out.

To clarify, each drone is used separately, on a different wifi network so potential MAV_SYS_ID related issues should not occur. I consider a common network architecture with :

onboard[FMU<-ethernet->companion(hosts mavilnk-routerd)] <-wifi-> GCS

In this config Mavlink messages exchange between FMU and GCS are ok. The main issue is at the companion level :

Companion runs :

  1. mavlink-routerd, connected on all interfaces on port 14540, see config above, This seems to work fine
  2. a local script (typically the telemetry.py mavsdk example script) that should read mavlink messages from port 14550 issues are here. On drone 1(CM4 companion), sometimes it cannot connect, on drone 2(Jetson Nano companion), it never connects.

Hope this is clearer.
Thanks

There is a connection debug guide for mavsdk Python scripts:
http://mavsdk-python-docs.s3-website.eu-central-1.amazonaws.com/#debug-connection-issues

Hi,
actually, i obtain the same result as shown on the link you provided when the connection is waiting indefinitely:

/usr/local/lib/python3.7/dist-packages/mavsdk/bin/mavsdk_server  udp://:14550
[07:11:57|Info ] MAVSDK version: v1.4.17 (mavsdk_impl.cpp:20)
[07:11:57|Info ] Waiting to discover system on udp://:14550... (connection_initiator.h:20)

No more information. this looks strange and… blocking. What to check to get further ?
Thanks.

So nothing is connected just yet.

So mavlink-router is forwarding to UDP port 14550 on localhost, the same host that mavsdk_server is running on?

Hi, yes, mavlink-routerd and the monitoring script i want to run (or here mavsdk_server) are both running on the same companion computer (nvidia jetson nano, connected on the ethernet port). I use the same mavlink-routerd config as shown in my first script, the address of the companion UdpEndpoint can be 0.0.0.0, 127.0.0.1 or other, same issue…
but it works on my other test plateform, the companion being a rpi CM4 (but sometimes, no connection also occurs).

Thanks for your help

Solved, this was all related to Nvidia Jetson nano default installation. Communication port has to be explicitly declared (contrary to the rpi debian install). Here are the guidelines for memory:

  1. First check if port is open
    sudo iptables -L -n | grep 14550

  2. if not, create/update /etc/iptables using below command:
    sudo mkdir /etc/iptables

  3. execute below commands to open port 14550 (as for my use case) to transmit and receive data:

sudo iptables -A INPUT -p udp --dport 14550 -j ACCEPT
sudo iptables -A OUTPUT -p udp --dport 14550 -j ACCEPT
sudo iptables-save | sudo tee /etc/iptables/rules.v4
  1. reboot and test
1 Like

Hello, I am also experiencing issues while trying to connect my brand new Pixhawk 6X to my Jetson Xavier NX board. I can ping the FMU, but I am unable to get the Mavlink communication working.

Here is my setup: PX4 is connected to the Jetson with the ethernet cable. I completed the PX4 Ethernet Setup guide using the proposed settings.

From the Jetson I can ping successfuly the FMU at 10.41.10.2 and here is the result of ifconfig:

eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 10.41.10.2  netmask 255.255.255.0  broadcast 10.41.10.255
        inet6 fe80::5e3e:7753:360:25a5  prefixlen 64  scopeid 0x20<link>
        ether 48:b0:2d:2f:7e:41  txqueuelen 1000  (Ethernet)
        RX packets 2  bytes 92 (92.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 557  bytes 26198 (26.1 KB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

But using MAVSDK like this doesn’t connect to the board:

await drone.connect(system_address="udp://10.41.10.2:14540")

no more chance using mavlink-routerd to broadcast the mavlink messages to QGroundControl on my laptop (I am actually not sure this is a way to test the communication though…)

mavlink-routerd -e <qgc_computer_ip>:14540 10.41.10.2:14540 -r

Note: I can see that QGroundControl and the Jetson communicate through Mavlink as the heart beat counter increase, which is not the case with the FMU.

I tried @albenoit solution to open the port 14540, but it didn’t solve the issue. I’ve noticed that after reboot sudo iptables -L -n | grep 14540 give me nothing, probably meaning that the settings get overridden during boot?

Note: I followed blindly the section Ubuntu Ethernet Network Setup to use netplan to configure the network (copy pasted the /etc/netplan/01-network-manager-all.yaml). I have no clue what it does and I don’t get why the IP is: 10.41.10.1 in that file instead of the IP of the FMU 10.41.10.2.

I would be very glad to hear some suggestion as I am stuck here.
Thanks.

Hi,
on which udp port have you connected your Jetson companion with the flight controller.

I followed the official guide and set the flight controller with the following settings:
MAV_2_MODE = 2 (onboard mode)
MAV_2_REMOTE_PRT = 14540 (as advised in the doc)
MAV_2_UDP_PRT = 14540 (as advised in the doc)

So to answer your question, I am trying to connect my Jetson to the udp port 14540 at the flight controller’s IP address → 10.41.10.2:14540.

Ex with MAVSDK, on the Jetson, I am trying to do this:

await drone.connect(system_address="udp://10.41.10.2:14540")

Does that make sense?

I think that could work.

Can you ping the Pixhawk from the companion?

And can you try the connection debugging as explained here?
http://mavsdk-python-docs.s3-website.eu-central-1.amazonaws.com/#debug-connection-issues

Actually the connection between FC and Jetson works only the first time either with mavproxy.py --master=udpout:10.41.10.2:14540 --out=udp:10.41.10.3:14550 or with mavsdk using: await drone.connect(system_address="udp://10.41.10.2:14540").
After issuing a ctrl-c, I need to restart the FC to be able to connect again…
Does it sound familiar?

Ok, this could be something regarding UDP ports changing.

Can you paste the output of mavsdk_server for the case where the connection breaks? Or could you check what messages you can see using Wireshark?