Lidar sensor makes gz_bridge timed out

Hi,
Since I added a lidar to my model, PX4 fail waiting for clock message through gz_bridge.

[launch_px4.sh-8] ERROR [gz_bridge] timed out waiting for clock message
[launch_px4.sh-8] ERROR [gz_bridge] Task start failed (-1)
[launch_px4.sh-8] ERROR [init] gz_bridge failed to start
[launch_px4.sh-8] ERROR [px4] Startup script returned with return value: 256

launch_px4.sh is a simple script launched right after the node ros_gz_bridge::parameter_bridge, that contains make px4_sitl_default and "$PX4_PATH"/build/px4_sitl_default/bin/px4 with param.

I solved the problem by adding a sleep 3 before px4 run.

Did I do something wrong ?
Have you ever heard of a problem like this one ?

Using:
Ubuntu 22.04
Gazebo Garden
Ros2 Humble
PX4 : up to date main branch
ros_gz : humble branch

same problem, I already have the fix from [Bug]: gz_bridge gets wrong world name when GZ_VERBOSE=1 · Issue #21727 · PX4/PX4-Autopilot · GitHub as I just pulled the repo from main.

Thanks for your reply but my pb is not the same.
Your correction is already in the code, the world name is found.

[launch_px4.sh-8] INFO  [init] gazebo already running world: warehouse_lanes
[launch_px4.sh-8] INFO  [gz_bridge] world: warehouse_lanes, model name: cis_drone_1, simulation model: 
[launch_px4.sh-8] ERROR [gz_bridge] timed out waiting for clock message

I tried to set GZ_VERBOSE to 0, it changes nothing.

Following the code you highlighted, I found the pb comes from this line:

if gz_bridge start -n "${PX4_GZ_MODEL_NAME}" -w "${PX4_GZ_WORLD}"; then

→ command returns code 255.

Looking in GZBridge.cpp, I found that there is a sleep to wait for initial clock set.
cf #if defined(ENABLE_LOCKSTEP_SCHEDULER)
I won’t go any further, and will leave my sleep command in my script.

Hello,
From my experience, the gz_bridge fails to launch if the simulation is not in a “running” state (i.e. not “paused”).
So, if the px4 instance is launched too quickly after the Gazebo server (or if you have a heavy world to load), the px4 instance will fail to connect to Gazebo and abort.

The sleep 3 works but is admittedly clumsy.

In my setup I do something kind of hacky but it work : I use ROS2 launch system with event handlers to launch px4 instances when I detect that the Gazebo simulation has started. I can send you snippets of my ROS2 launch files if you need.

Hi @Theotime_Balaguer !
Could you share it here?

@SebastienL , you can also increase the connection timeout here

Hi @Benja !

Sure, I will share the whole launch file, even though it is quite a mess.

Explanation :
I have several ROS2 Nodes that gets launched by my launch file, one of the nodes launch the Gazebo server and outputs a particular string when the server is completely initialized (with std::cout, not the ROS2 logging system). ROS2 launch system is able to detect this output with an OnProcessIO event handler, created as follow (in LaunchDescription) :

            RegisterEventHandler(
                OnProcessIO(
                    target_action=ns3_simulator,
                    on_stdout=lambda info: spawn_px4 if(info.text.decode().strip() == 'Network simulator ready') 
                    else LogInfo(
                        msg='ns-3 says "{}"'.format(info.text.decode().strip())
                    )
                )
            ),

Note: My setup involves a network simulator on top of Gazebo, and I need to make sure the network simulator is also running before starting the px4, which is why the names refer to a network simulator. It would work the same with any other program outputing a string, just make sure to write on the standard output !

The full launch file (I didn’t manage to attach a .py or .zip file :frowning: ) :

# ROS2 Launch program that starts a co-simulation of Gazebo and NS-3 along with the Gazebo GUI.
# It sets-up a virtual network infrastructure based on the file setup_network.bash and launches PX4-Autopilot instances.

import os
from launch import LaunchDescription
from launch.actions import ExecuteProcess, DeclareLaunchArgument, LogInfo, RegisterEventHandler, SetEnvironmentVariable
from launch.event_handlers import OnProcessStart, OnProcessIO, OnExecutionComplete, OnProcessExit, OnShutdown
from launch_ros.actions import Node
from launch.substitutions import LaunchConfiguration, LocalSubstitution
from launch.conditions import IfCondition

import yaml


def generate_launch_description():
    ws_path = '/home/theotime/simulation_ws'
    config_path = ws_path + '/src/config/example_config.yaml'

    # copy all the environment variables of the user
    node_env = os.environ.copy()

    # read the config file (Need pyYaml >= 5.1)
    with open(config_path) as f:
        config = yaml.load(f, Loader=yaml.FullLoader)

    # Boolean parameter to launch the Gazebo GUI or not
    use_gz_gui = LaunchConfiguration('use_gz_gui', default=True)
    use_gz_gui_arg = DeclareLaunchArgument(
        'use_gz_gui',
        default_value=use_gz_gui,
        description='If true, launch the Gazebo GUI')

    # ROS2 Action: run a bash script as sudo to configure the virtual network (require sudo)
    setup_network = ExecuteProcess(
        cmd=['sudo', ws_path + '/src/launch/setup_network.bash', str(config['robots_number'])],
        name='setup_network'
    )

    # Sets the environment variable GZ_IP, needed to communicate with Gazebo server from different namespaces (Gazebo's transport library)
    # [config] bridge_ip: IP of the bridge providing direct communication with the network namespaces
    set_GZ_IP = SetEnvironmentVariable(
        name='GZ_IP',
        value=[str(config['bridge_ip'])]
    )

    # ROS2 Node: Launch the network coordinator node. 
    # This python file is a modified version of the network coordinator from the ROS-NetSim framework
    # Require sudo mode to access the network namespaces
    network_coordinator = Node(
            package='network_coordinator',
            executable='network_coordinator',
            name='net_coord',
            arguments=[config_path],
            prefix=["sudo -E env \"PATH=$PATH\" \"PYTHONPATH=$PYTHONPATH\" python3 "],
            shell=True
        )

    # ROS2 Node: Launch the robotics coordinator node.
    # This c++ file is a modified version of the robotics coordinator from the ROS-NetSim framework
    # The Gazebo server (i.e. the robotics simulator) is started by this Node, thus the name "gazebo"
    robotic_coordinator = Node(
            package='robotics_coordinator',
            executable='robotics_coord',
            name='gazebo',
            parameters=[
                {"config_file": config_path},
                {"verbose": False}
            ]
        )

    # ROS2 Node: Launch the network simulator node.
    ns3_simulator = Node(
            package='ns-3_sim_ros',
            executable='ns-3_sim_ros',
            name='ns3',
            parameters=[
                {"config_file": config_path},
                {"verbose": False}
            ]
        )

    # ROS2 Action: Runs a command to launch an empty Gazebo GUI. It connects automatically to a running Gazebo Server 
    gazebo_gui = ExecuteProcess(
        cmd=['gz', 'sim', '-g', '>', '/dev/null'],
        name='gazebo_gui',
        condition=IfCondition(LaunchConfiguration('use_gz_gui'))
    )

    # ROS2 Action: Runs a bash script to launch N PX4-Autopilot instances from the ~/PX4-Autopilot directory
    # [config] robots_number: Number of px4_Autopilot to launch
    # [config] robots_model: name of the model that will be spawned in Gazebo (a model with this name must exist in PX4-Autopilot/Tools/simulation/gz/models)
    spawn_px4 = ExecuteProcess(
        cmd=['/home/theotime/simulation_ws/src/launch/spawn_px4.bash', str(config['robots_number']), str(config['robots_model'])],
        name='spawn_px4',
        env=node_env
    )

    # ROS2 Node: Launch the tf2 broadcaster node
    # [config] robot_model: name of the robot model used in Gazebo
    def tf2_broadcaster(i):
        return Node(
            package='tf2_cpp',
            executable='uav_tf2_broadcaster',
            name='uav_tf2_broadcaster_'+str(i),
            parameters=[
                {'robot_name': config['robots_model']+'_'+str(i)}
            ]
        )

    # ROS2 Action: Runs a script that destroys the virtual network infrastructure 
    # Require sudo mode
    # Note: we wrap this script in a function in a clumsy way because of a bug in ROS2 launch system: 
    # it appears the OnShutDown event handler doesn't work with any action other than LogInfo() 
    # see https://robotics.stackexchange.com/questions/24908/how-to-execute-a-script-at-shutdown-of-a-launch-process
    def remove_network(event, context):
        os.system('sudo '+ws_path+'/src/launch/remove_network.bash '+str(config['robots_number']))
        return [
            LogInfo(msg=['Launch was asked to shutdown: ', LocalSubstitution('event.reason')])
        ]
    
    # The main ROS2 LaunchDescription object
    ld = LaunchDescription([
        use_gz_gui_arg,
        setup_network,
        set_GZ_IP,
        RegisterEventHandler(
            OnProcessExit(
                target_action=setup_network,
                on_exit=[
                    network_coordinator,
                    ns3_simulator,
                    robotic_coordinator,
                    gazebo_gui
                ]
            )
        ),
        RegisterEventHandler(
            OnProcessIO(
                target_action=ns3_simulator,
                on_stdout=lambda info: spawn_px4 if(info.text.decode().strip() == 'Network simulator ready') 
                else LogInfo(
                    msg='ns-3 says "{}"'.format(info.text.decode().strip())
                )
            )
        ),
        RegisterEventHandler(
            OnShutdown(
                on_shutdown=remove_network
            )
        )
    ])

    # Add N tf2_broadcaster nodes to the LaunchDescription
    for i in range(1, config['robots_number']+1):
        ld.add_action(tf2_broadcaster(i))

    return ld
1 Like

Thanks for this! I was having some similar issues out of the blue and increasing this worked!

Is there a definite cause for this though?