Upgrading from 1.8.x to 1.9.x: problems with onboard computer

PX4 Autopilot
1

Hi there!

We had the working installation:

  • Pixhawk CUBE (FMUv2(3))
  • px4 1.8.2
  • Onboard computer
  • mavros node
  • Flight controller connected to the onboard computer directly, without FTDI.

And it works pretty well.

We’ve upgraded px4 to 1.9.2 and change all parameters, according to this manual: https://dev.px4.io/master/en/companion_computer/pixhawk_companion.html

MAV_1_CONFIG=TELEM 2
MAV_1_FORWARD=Disabled (also tried Enabled)
MAV_1_MODE=Onboard
MAV_1_RATE=80000 B/s (we've tried also to set 0)
SER_TEL2_BAUD=921600 8N1

But mavros didn’t connect to the flight controller.
After we changed the value of MAV_1_MODE to normal it started to work, but partially. When we’ve tried to change a flight mode to ‘OFFBOARD’ we received FCU: REJECT OFFBOARD.
Here is two logs from mavros node, with normal and onboard values: Pixhawk CUBE, px4 1.9.2, Onboard communication problems · GitHub

Could anyone help, please, what should we change to make it working?
Thanks!

2

@szobov Could you check if you are sending companion process messages to the flight controller?

3

Thank you for your replay, but actually I didn’t catch what you mean.
As I said above, the environment is the same, what means that I started the mavros with the sys_status plugin enabled. So, the mavros node tries to send a heartbeat message, but only to a connected device. If I start this node with pixhawk with 1.8.2 connected, all works well. With the same node, the same onboard computer, the same pixhawk, but with 1.9.2 it doesn’t work.

you are sending companion process messages to the flight controller

Which messages, except heartbeat, should I check?

Also, here is the screenshot from QGC:

Selection_087
Selection_087633×647 73.5 KB

4

You need to actively send companion process status messages to the flight controller. It is on /mavros/companion_process/status. Otherwise the flight controller will think that the companion computer has died and will ignore all the setpoints

5

Hm… I’m not quite sure, that I understand you properly.
Could you please, share with me the manual, where I can find something about this changes?
We’ve never sent any messages to this topic by our code before.
I’ve also checked this manual and haven’t noticed any mentions of /mavros/companion_process/status: https://dev.px4.io/v1.9.0/en/ros/mavros_offboard.html
The same with: http://wiki.ros.org/mavros

6

@szobov This might help: https://github.com/PX4/avoidance/blob/master/local_planner/src/nodes/local_planner_node.cpp#L54

I think somehow this is not in the documentation. I think we definitely need to have it. The mavros wiki seems to be missing quite some stuff too.

7

Thanks for this link!
Actually, I’ve found the plugin, that should send this message: https://github.com/mavlink/mavros/blob/30bd87e17d369a3c3b1b01e18a8478eeaa14ec8d/mavros_extras/src/plugins/companion_process_status.cpp
But even with this plugin included, mavros and px4 still doesn’t communicate with each other.
It’s a tad disappointing, by the way.
I’ll try to look into px4 source code to find the answer. Thanks.

8

@szobov Including the plugin is not enough, you need to actively send the heartbeats over to the flight controller

9

@Jaeyoung-Lim is there the required frequency for publishing into this topic? And what the required content for this messages?
I wrote a code, that publishes into /mavros/companion_process/status, but still has no response from the flight controller:

...
[ INFO] [1564495261.708245048] /mavros: Plugin companion_process_status loaded
[ INFO] [1564495261.747007020] /mavros: Plugin companion_process_status initialized
...
$ rostopic echo -n1 /mavros/companion_process/status 
header: 
  seq: 414
  stamp: 
    secs: 1564495187
    nsecs: 826671561
  frame_id: ''
state: 4
component: 240

$ rostopic hz  /mavros/companion_process/status 
subscribed to [/mavros/companion_process/status]
average rate: 24.999
	min: 0.040s max: 0.040s std dev: 0.00008s window: 24
average rate: 25.000
	min: 0.040s max: 0.040s std dev: 0.00006s window: 49
average rate: 25.000
	min: 0.040s max: 0.040s std dev: 0.00005s window: 74
average rate: 25.000
	min: 0.040s max: 0.040s std dev: 0.00005s window: 99
^Caverage rate: 25.000

$ rostopic echo -n1 /mavros/state
header: 
  seq: 0
  stamp: 
    secs: 1564495172
    nsecs: 124132898
  frame_id: ''
connected: False
armed: False
guided: False
mode: ''
system_status: 0
---

What do I do wrong?

10

@szobov can you share the flight controller log trough flight review?
You don’t need to send /mavros/companion_process/status unless you have set the parameter COM_OBS_AVOID to true.

11

Hi @mrivi ,
Here is the log: https://review.px4.io/plot_app?log=32c2dcfc-5e25-463a-8be3-e9b8ca07a8fa
Hope it helps somehow.
Thanks!

12

@szobov, have you noticed that the mpu9250 driver failed to start?

13

@mrivi yes, but I’ve never seen this message on 1.8.2, so I’ve decided, that it’s OK.
To be honest, I don’t know, how it’s related to this issue. Could you explain it to me a ted, please?

–
EDIT
I’ve tried to arm the copter and fly in poshold and it’s flying as usual. Nothing bizarre.
:thinking: But before the flight the copter wasn’t calibrated, in QGC has been a message about uncalibrated magnetometer. Maybe this error message relates to uncalibrated mag.

14

That’s the IMU driver. Have you ever flown any other mode other than stabilized?
What message are you sending from ROS to control the vehicle? How are you filling it? At which rate is it sent?

15

Have you ever flown any other mode other than stabilized?

On the previous firmware version, 1.8.2, yes, in poshold, althold, auto-takeoff, auto-land and mission modes.
But on 1.9.2 only in the stabilized mode.

What message are you sending from ROS to control the vehicle?

~setpoint_raw/local (mavros_msgs/PositionTarget)
http://wiki.ros.org/mavros#mavros.2BAC8-Plugins.setpoint_raw

How are you filling it?

With velocity vector, yaw, stamp, seq, frame_id, coordinate_frame and type_mask.
To clarify, we didn’t change the code, that send this message, so it works stable on 1.8.2.

At which rate is it sent?

~30hz

16

The Flight Controller cannot switch into offboard because it isn’t receiving the setpoint that you’re publishing from ROS. I would check

  • physical connection between companion and autopilot
  • setpoints are published by your node as you expect
  • mavlink stream configuration
17

@mrivi yes, but the problem isn’t in the OFFBOARD mode, because we can’t receive anything from autopilot before posting any setpoints.
If you’ll look back onto my previous messages, you’ll see, that when I configure MAV_*_MODE as onboard I receive only:

$ rostopic echo -n1 /mavros/state 
header: 
  seq: 121
  stamp: 
    secs: 1564406488
    nsecs: 440964735
  frame_id: ''
connected: False
armed: False
guided: False
mode: ''
system_status: 0

And, yes, it means:

it isn’t receiving the setpoint that you’re publishing from ROS

This log message with FCU: REJECT OFFBOARD appears only in case of MAV_*_MODE=normal, but I don’t think it should work this way, because normal mode is for telemetry, not for onboard computer, right?

Also, I just want to say, that I’m absolutely sure, that everything is connected normally, because I can just downgrade the firmware version to 1.8.2 and return SYS_COMP_ID and everything will start to communicate as expected.

Thanks.

18

Try with mavlink mode ONBOARD

19

@mrivi mavlink mode ONBOARD is tried already, look at the thread head, please.

I digged a litte into the issue and problem could be in documentation.
If you look into these two manuals https://dev.px4.io/master/en/companion_computer/pixhawk_companion.html and https://docs.px4.io/v1.9.0/en/peripherals/mavlink_peripherals.html , you could see, that they have a difference.

In the first manual, you set parameter SER_TEL2_BAUD to set baudrate.

Otherwise, in the second - you set baudrate with MAV_2_RATE, as I guess by the value 912000 in that manual (it looks like one of standard baudrate of serial port), but… that conjecture is in conflict with words in documentation in https://github.com/PX4/Firmware/blob/19ad80385e80273120469e423f5cded5e8c182f7/src/modules/mavlink/module.yaml#L47. And there also saying about MAV_2_FORWARD=True, but nothing about that parameter in the first manual. It confuse.

@mrivi could you, please, say, what documentation is correct?

P.S. Guess, the basic idea of version 1.9.x is to switch to configure serial ports with MAV_X_* parameters. Is it?

20

Oh, I’ve just tried to disable mavlink instance for onboard computer and started it from mavlink console and, wow, it works!
Look:

Selection_089
Selection_0891645×971 212 KB

I’ll try to investigate it deeper, but at least it works somehow:

[ INFO] [1565018693.256706699]: Built-in MAVLink package version: 2019.7.7
[ INFO] [1565018693.256896532]: Known MAVLink dialects: common ardupilotmega ASLUAV autoquad icarous matrixpilot paparazzi slugs standard uAvionix ualberta
[ INFO] [1565018693.257032032]: MAVROS started. MY ID 1.240, TARGET ID 1.1
[ INFO] [1565018693.291988017]: IMU: High resolution IMU detected!
[ INFO] [1565018693.292366767]: IMU: Attitude quaternion IMU detected!
[ WARN] [1565018693.298403480]: GP: No GPS fix
[ INFO] [1565018693.301963191]: CON: Got HEARTBEAT, connected. FCU: PX4 Autopilot
[ERROR] [1565018693.346367807]: ODOM: Ex: The tf tree is invalid because it contains a loop.
Frame local_origin_ned exists with parent local_origin.
Frame fcu_frd exists with parent fcu.
Frame fcu exists with parent fcu_frd.


[ INFO] [1565018693.354693730]: IMU: High resolution IMU detected!
[ INFO] [1565018693.354946688]: IMU: Attitude quaternion IMU detected!
[ INFO] [1565018694.310437857]: VER: 1.1: Capabilities         0x000000000000e4ef
[ INFO] [1565018694.310608107]: VER: 1.1: Flight software:     010902ff (106905871d000000)
[ INFO] [1565018694.310783691]: VER: 1.1: Middleware software: 010902ff (106905871d000000)
[ INFO] [1565018694.310909357]: VER: 1.1: OS software:         071c00ff (423371c7d4012e72)
[ INFO] [1565018694.311030774]: VER: 1.1: Board hardware:      00000011
[ INFO] [1565018694.311176899]: VER: 1.1: VID/PID:             26ac:0011
[ INFO] [1565018694.311398191]: VER: 1.1: UID:                 3136510934313630
[ WARN] [1565018694.311625566]: CMD: Unexpected command 520, result 0
[ INFO] [1565018703.303135296]: HP: requesting home position
[ INFO] [1565018708.315184333]: WP: item #0* F:3 C: 16 p: 0.000000 0.000000 0.000000 0.000000 x: 47.397800 y: 8.545350 z: 3.000000
[ INFO] [1565018708.319870462]: WP: item #1  F:3 C: 16 p: 0.000000 0.000000 0.000000 0.000000 x: 47.397800 y: 8.545350 z: 3.000000