MicroDDS agent causing weird behavior gazebo

jkampia · October 4, 2024, 4:23am

Hi,

My setup is as follows:
Ubuntu 22.04
ROS2 Humble
PX4 Autopilot 1.14 (latest vers cloned from github)
Micro XRCE-DDS Agent (latest vers cloned and built from source)
px4_sitl gazebo

I am running px4_sitl gazebo with the x500 model. I installed and built the MicroDDS agent as per the px4 wiki, and am trying to use it to link some offboard code I wrote to the sim.

The sim works perfectly fine (i.e. the drone moves as it should, I can fly it in position mode with a controller and everything looks good in QGC) until I enable the MicroDDS agent. As soon as I enable it, the drone starts faltering and sinking while still in position hold. I was also getting the error message: " [[timesync] time jump detected. Resetting time synchroniser] periodically (only after I enabled the MicroDDS agent) until I disabled the time sync parameter in QGC in an attempt to fix the issue, which didn’t work.

I can attach a video if you want, but basically it’s preventing me from doing any meaningful testing with ROS2 integration. As soon as I start the agent, the sim becomes useless. Altitude starts drifting, etc. It seems to be related to whatever root issue was causing the time sync error.

I also get a strange “multicast failed” error that spams my terminal sometimes, coming from the sim. I assume that might be my wifi dropping or something, but maybe it’s related to the issue causing the sim to fail.

Giorgio-P98 · October 9, 2024, 1:41pm

Can you explain better what is the drone doing? I do not understand if you launch the MicroDDS agent while the drone is flying in position, or when the drone is still in the spawn position.

Theoretically, when you launch the agent it should not send anything to the controller. Thus, the drone should not move. Conversely, if you are flying while launching the MicroDDS agent, maybe problems could occur.

timesync error is because PX4 and the MicroDDS agent use different clocks. As explained here [Bug] Time jump detected. Resetting time synchronizer · Issue #22522 · PX4/PX4-Autopilot · GitHub PX4 uses the gazebo clock, while the MicroDDS agent uses the wall clock. This same issue with gz_x500 was reported and solved in the above link.

They linked the ROS 2 User Guide | PX4 Guide (main) at ROS, Gazebo and PX4 time synchronization.

The tymesync problem, could interfere with the functioning of the MicroDDS bridge, and maybe if you follow what is in the links i provide you, you can solve the problem.

Anyway, try run the MicroDDS agent before doing anything with the drone (so do not change to position mode or takeoff the drone).

Also make sure you have installed the correct ROS2-gazebo bridge. In your case of ROS2 Humble and gazebo harmonic, the command is:

sudo apt install ros-humble-ros-gzharmonic

if you are using gazebo garden, change gzharmonic with gzgarden

Let me know if this solves the problem

jkampia · October 14, 2024, 2:57am

Hey, sorry for the late reply. I fixed the offboard control mode issue – I believe the issue was that sending uOrb offboard messages in position mode interfere with the position controller and cause it to fight itself.

However, I am now having bigger issues with the simulator itself. I am still on the same setup (latest version of PX4_Autopilot repository, and installed the correct gz bridge via the command you gave. Whenever I launch gazebo through the command ‘make px4_sitl gz_x500’, I get this output:

make px4_sitl gz_x500
[0/1] cd /home/velocitylab/PX4-Autopilot/build...b/PX4-Autopilot/build/px4_sitl_default/bin/px4

______  __   __    ___ 
| ___ \ \ \ / /   /   |
| |_/ /  \ V /   / /| |
|  __/   /   \  / /_| |
| |     / /^\ \ \___  |
\_|     \/   \/     |_/

px4 starting.

INFO  [px4] startup script: /bin/sh etc/init.d-posix/rcS 0
INFO  [init] found model autostart file as SYS_AUTOSTART=4001
INFO  [param] selected parameter default file parameters.bson
INFO  [param] importing from 'parameters.bson'
INFO  [parameters] BSON document size 404 bytes, decoded 404 bytes (INT32:14, FLOAT:6)
INFO  [param] selected parameter backup file parameters_backup.bson
INFO  [dataman] data manager file './dataman' size is 7872608 bytes
INFO  [init] Gazebo simulator
INFO  [init] starting gazebo with world: /home/velocitylab/PX4-Autopilot/Tools/simulation/gz/worlds/default.sdf
WARN  [init] PX4_GZ_MODEL_POSE not set, spawning at origin.
INFO  [gz_bridge] world: default, model name: x500_0, simulation model: x500
INFO  [gz_bridge] Requested Model Position: 0,0,0,0,0,0
INFO  [gz_bridge] Model position z is less or equal 0.0, moving upwards
Warning: Ignoring XDG_SESSION_TYPE=wayland on Gnome. Use QT_QPA_PLATFORM=wayland to run on Wayland anyway.
Unknown message type [8].
Unknown message type [8].
Unknown message type [9].
Unknown message type [9].
Unknown message type [9].
Unknown message type [9].
ERROR [gz_bridge] Service call timed out. Check GZ_SIM_RESOURCE_PATH is set correctly.
WARN  [gz_bridge] Service call timed out as Gazebo has not been detected.
Unknown message type [8].
Unknown message type [8].
Unknown message type [9].
Unknown message type [9].
Unknown message type [9].
Unknown message type [9].
Unknown message type [9].
Unknown message type [9].
Unknown message type [9].
Unknown message type [9].
Unknown message type [9].
Unknown message type [9].
Unknown message type [9].
Unknown message type [9].
Unknown message type [9].
Unknown message type [9].
Unknown message type [9].
Unknown message type [9].
Unknown message type [9].
Unknown message type [9].
Unknown message type [9].
Unknown message type [9].
Unknown message type [9].
Unknown message type [9].
Unknown message type [9].
Unknown message type [9].
Unknown message type [9].
Unknown message type [9].
Unknown message type [9].
Unknown message type [9].
libEGL warning: egl: failed to create dri2 screen
libEGL warning: egl: failed to create dri2 screen
INFO  [lockstep_scheduler] setting initial absolute time to 1252000 us
INFO  [commander] LED: open /dev/led0 failed (22)
INFO  [uxrce_dds_client] init UDP agent IP:127.0.0.1, port:8888
INFO  [mavlink] mode: Normal, data rate: 4000000 B/s on udp port 18570 remote port 14550
INFO  [tone_alarm] home set
INFO  [mavlink] mode: Onboard, data rate: 4000000 B/s on udp port 14580 remote port 14540
INFO  [mavlink] mode: Onboard, data rate: 4000 B/s on udp port 14280 remote port 14030
INFO  [mavlink] mode: Gimbal, data rate: 400000 B/s on udp port 13030 remote port 13280
INFO  [logger] logger started (mode=all)
INFO  [logger] Start file log (type: full)
INFO  [logger] [logger] ./log/2024-10-14/02_46_53.ulg	
INFO  [logger] Opened full log file: ./log/2024-10-14/02_46_53.ulg
INFO  [mavlink] MAVLink only on localhost (set param MAV_{i}_BROADCAST = 1 to enable network)
INFO  [mavlink] MAVLink only on localhost (set param MAV_{i}_BROADCAST = 1 to enable network)
INFO  [px4] Startup script returned successfully
pxh> INFO  [mavlink] partner IP: 127.0.0.1
WARN  [health_and_arming_checks] Preflight Fail: ekf2 missing data
WARN  [health_and_arming_checks] Preflight Fail: No connection to the ground control station

After the simulation runs for some time (it seems to be random each time), it starts printing the error message:

Exception sending a multicast message:Network is unreachable
Exception sending a multicast message:Network is unreachable
Exception sending a multicast message:Network is unreachable
Exception sending a multicast message:Network is unreachable
Exception sending a multicast message:Network is unreachable
Exception sending a multicast message:Network is unreachable
Exception sending a multicast message:Network is unreachable
Exception sending a multicast message:Network is unreachable
Exception sending a multicast message:Network is unreachable
Exception sending a multicast message:Network is unreachable
Exception sending a multicast message:Network is unreachable

After this error starts printing, I can no longer communicate with the simulator over the ros2 topics published by the MicroXRCE agent.

The issue is not caused by the MicroXRCE agent, as I can still produce the error when I run the sim by itself. Grepping the PX4_Autopilot repo for this error message returns nothing, so have basically no idea where this error is originating or why.

Jmavsim seems to work fine, so I guess I’ll just use that for now.

Giorgio-P98 · October 14, 2024, 1:13pm

Just to understand better. Are you sure you have the last version of PX4-Autopilot? i.e. the main? Because v1.14 is not the latest. I don’t know if this is the case, but since the MicroAgent and PX4 use a common set of ROS2 msgs for the communication, maybe the problem is in some incompatibility between the two.

Also the first error that rises, is:

ERROR [gz_bridge] Service call timed out. Check GZ_SIM_RESOURCE_PATH is set correctly.

I really don’t know why this error shows up, since it can rise only when the PX4_GZ_STANDALONE var is set. If this is the case, make sure you have followed what is reported at Gazebo Simulation: Standalone Mode

Instead if you are not trying to launch in standalone mode, check why this variable is set to true (i.e. =1), since it should be =0 by default. Also check in the .bashrc file, if you have somehow set it like:

export PX4_GZ_STANDALONE=1

A rapid workaround could be to set the variable =0 when launching px4. Thus, when running the px4_sitl, run:

PX4_GZ_STANDALONE=0 make px4_sitl gz_x500

Try checking the possible causes that I have listed