PX4 offboard mode( EKF2) not working with px4 mini fmuv5 - ERROR in odometer message, and also cannot see vision_pose(mocap data is published and advertised) topic in uorb command

Hello, I am facing below issue where px4 log reads “No GPS Fix” & ERROR message when i am communicating rpi3 with px4 mini ( i have telemetry_2 configured for MAV_1_CONFIG onboard companion with baud rate of 921600Byts/sec ). I am advertising /relaying motion capture readings at 120 hz from optitrack motive vrpn server to mavros client running on rpi3 via ( /mavros/vision_pose/pose). And when i look at mavlink inspector LOCAL position NED is read at 0.4 HZ as seen in below image. Can any one please let me know why px4 is reading LOCAL position NED from mavros/pose at very low frequency hz of 0.4 when i am actually sending data ta 120 HZ ?. Is this an issue with px4 firmware ???
LOGS

[ INFO] [1455208868.648885761]: MAVROS started. MY ID 1.240, TARGET ID 1.1
[ INFO] [1455208868.655187917]: IMU: Attitude quaternion IMU detected!
[ INFO] [1455208868.661861685]: IMU: High resolution IMU detected!
[ INFO] [1455208868.683556680]: RC_CHANNELS message detected!
[ERROR] [1455208868.686124115]: ODOM: Ex: The tf tree is invalid because it contains a loop.
Frame local_origin_ned exists with parent local_origin.
Frame fcu_frd exists with parent fcu.
Frame fcu exists with parent fcu_frd.

[ INFO] [1455208868.690640655]: CON: Got HEARTBEAT, connected. FCU: PX4 Autopilot
[ INFO] [1455208868.696894790]: IMU: Attitude quaternion IMU detected!
[ INFO] [1455208868.701559402]: IMU: High resolution IMU detected!
[ INFO] [1455208868.783650862]: RC_CHANNELS message detected!
[ WARN] [1455208868.786806003]: GP: No GPS fix
[ INFO] [1455208869.703263563]: VER: 1.1: Capabilities 0x000000000000e4ef
[ INFO] [1455208869.704099809]: VER: 1.1: Flight software: 010900ff (ea48cd4970000000)
[ INFO] [1455208869.704671212]: VER: 1.1: Middleware software: 010900ff (ea48cd4970000000)
[ INFO] [1455208869.705255480]: VER: 1.1: OS software: 071c00ff (423371c7d4012e72)
[ INFO] [1455208869.706020789]: VER: 1.1: Board hardware: 00000032
[ INFO] [1455208869.706638234]: VER: 1.1: VID/PID: 26ac:0032
[ INFO] [1455208869.707510938]: VER: 1.1: UID: 3138511935343633
[ WARN] [1455208869.708301819]: CMD: Unexpected command 520, result 0

image
https://review.px4.io/plot_app?log=e3591764-c0e5-4e47-b50e-1d9a63f9a41d

LOCAL_POSITION_NED is not supposed to be the messages from your mocap, but mavlink coming out of the local position estimator. I believe this can be increased by chaning the MAV_1_MODE to something like Onboard. However, this doesn’t mean that your mocap data is properly being streamed to your vehicle.

image

Above image shows my MAV_1 parameters and I tried bumping up the _rate to 1500000B/s. But still Hz rate is very low. Could you please let me know which message on mavlink inspector should i be looking at to verify my mocap data?

I also want to let you know that my connection between pixhawk mini4 and RPI3 is as show in below image.

Does it have/got to do anything with above issue.

@scalnoor The USB connector is not TELEM2 as you are trying to configure. You need to connect to the TELEM2 port, or configure your serial port to TELEM1

@jay, thank you.Could you please let us know an recommended way to connect pixhawk telemetry to raspberry-pi, like specified cable/port or it requires customized cable.? And if connected in the said specified manner what would it recognize as on /dev/* device mount point?

@scalnoor I am not familiar with the pixhawk4 mini, but I believe the USB cable is connected to TELEM1, or you need to connect to the TELEM2 UART port, possibly with a ftdi

thank you. But I really did not get what this error means. “[ERROR] [1455208868.686124115]: ODOM: Ex: The tf tree is invalid because it contains a loop.
Frame local_origin_ned exists with parent local_origin.
Frame fcu_frd exists with parent fcu.
Frame fcu exists with parent fcu_frd.”

Can you please let us know as why this could be caused or happening?

@scalnoor I would first make sure that the communication is being done properly.

@Jaeyoung-Lim thank you, And yes I shall try with getting the connection right and let you know.

Hello @Jaeyoung-Lim I connected the pixhawk 4 mini telemetry-1 pins to the RPI3-GPIO respectively and I was able to login successfully. configured the MAV_0 parameters respectively for teml1, baud-rate of 921600 b/s and onboard. Verified the mocap data being published is sent at 120HZ via mavros by issues command(.:. rostopic hz /mavros/vision_pose/pose).

Next part is I have the mavros script just to take off and land at 0, 0 , 1( Z = 1M above ground). And result is it drifts drastically and upon time progress it swings like crazy not at all holding position wish I had video recording to show you. But have uploaded the logs to flight review and below is the link to it. https://review.px4.io/plot_app?log=53a74ab1-9f63-4447-8c88-fa6b125c0b24

we are completly clueless as why the above behavior from logs. And all we have been trying to get it to set at 1M above the ground. NOTE mocap data is fed at 120 HZ verified coordinates at MAVLink inspector on LOCAL_POSITION_NED message. I have been setting single waypoint/setpoint at 0, 0, 1 by publish at 20 HZ using setpoint_postion/local topic. Please kindly do help us at the earliest as it seem a bit blocker from past month. Appreciate all your help here.

Hey scalnoor

I’m doing something similar right now with a pixhawk 2.1 - RPI3 - Vicon(MotionCapture). Did you check your EKF2 Outputs and Configuration? For example EKF2_AID_MASK to 24 and stuff like that?

What I think is a great way to check your connection is to run a rostopic echo on /mavros/local_position/pose which is the output of your EKF2 and check the outputs with you /mavros/vision_pose/pose to see problems/differences there.
I also realized the EKF2 will just work on 30Hz so I don’t think you need to run your Mocap on 120Hz all the time but I recommend to check the output frequency via rostopic hz /mavros/local_position/pose!
I had with another companion computer the problem the EKF2 ran on 8Hz or even less (I guess it might depends on the power supply of your companion computer).

Also do you power your PI via the 5V output of the pixhawk pins or via an addtional power supply? In our case the 5V output of the Telm2-UART port was not stable enough for our PI so I connected it via a 5V converter straight to the battery.

Hello @hsu-ret, @Jaeyoung-Lim,

I have double checked the EKF2 AID mask parameter it has been set to 24. And Im also redirecting the mocap reading via ( rostopic echo on /mavros/local_position/pose) and found the readings were intact. Well we even tried at point sending data rate at 30HZ, but I will again re-check on it.

Yes, we are powering up the PI via +5v leads straight from the battery, we doubled checked this by measuring via volt-meter.

@hsu-ret did this work for you, can you please share me the mavros script or explain the way your sending/publishing the data so we can cross verify. And also share all the parameter you set in pixhawk configuration to get it working.

@Jaeyoung-Lim , I also wanted ask jay from my previous interaction now that I have the proper connection b/w PI & Pixhawk, I still wonder why it drifts. And IMPORTANTLY I wanted to know if the pixhawk get GPS lock (green solid light) when I start feeding Mocap data ??? because in my case I dint find the green solid light, as I have blue solid light. NOTE im not using GPS. I have disabled the GPS 1 config.

@scalnoor External vision is not related to the GPS fix, so you should not see a green light.

You should verify that your mocap data is being properly forwarded to the EKF. Your observation of the system drifting simply shows that your mocap data pipeline is either not properly configured or your EKF is not tuned properly. I would say it is more probable that there is a problem in the data pipeline.

As I have commented above, the LOCAL_POSITION_NED is not coming from your mocap data. Therefore this does not verify that you are pumping in the right data. Have you tried moving your vehicle around (without flying) and see if the local position estimate is correct?

@Jaeyoung-Lim yes we did the capture volume test by moving around the drone in hand and verified the local_position/pose topic. Thank you for confirming about GPS not playing any role in external position.

I check the data rate for both mavros/local_position/pose (which I believe is the EKF output ) and mavros/vision_pose/pose are at 29.9 HZ thought the flight duration. And I also echoed the setpoint_position/local which reflected at 20HZ with fixed set point of (0,0,1).

I’m curious to know :

  1. What exactly does the EKF2 take as input along with mocap/vision data?
  2. What data of published mavros topics does the EKF2 LOOKUP or READ from, for the position and orientation ?
  3. finally, Where can I READ the EKF2 output? so I can plot the same against set points/local position. Is it the ekf2_innovation.csv file ?

And please kindly review the latest logs https://review.px4.io/plot_app?log=0acefeb9-3f54-4e62-b632-982cfce968cc from which the flight takes of and does the circle(seeking pattern) and then goes about 2M from x origin set point which is very wacky.

But one thing I noticed from attached flight review is that X Y positions follow the set point for very few seconds like about 3 seconds and but later the graph is off variedly. Im not sure why the X, Y varies when I’m setting way point to (0,0,1) at 20 HZ ? Why is the estimated setpoint( believing it is derived from EK2) varying so drastically?

NOTE :
I am just using below command to send Mocap data to pixhakw in order below, please let us know if these are correct:
roslaunch mavros px4.launch
roslaunch vrpn_client_ros sample.launch server:=192.168.1.143(my mocap pc streaming ip)
rosrun topic_tools relay /vrpn_client_node/<rigid_body_name>/pose /mavros/vision_pose/pose
are the above command correct?

I also pulled the .csv files from Ulog via ulog2csv . And notice there is nothing related to vision topic .csv files but instead notice that my mocap data was reflecting is vehicle_visual_odometry is this different from vision topic or related to it ??. Finally, makes me so confused to understand what topic is the EKF2 subscribing itself to read data upon ??? Wish I can send the csv files to you but not sure how to send it.

We appreciate all you help here. Looking forward to good crucial important clarification from you.

@scalnoor

  1. Here is a documentation of what the EKF uses to estimate the position of the vehicle.

  2. This is the mavlink which the vision_pose plugin sends

  3. LOCAL_POSITION_NED is the output of the EKF2

@Jaeyoung-Lim, thank you jay. I have more question. Is it necessary to set EKF2_EV_POS_X, EKF2_EV_POS_Y, EKF2_EV_POS_Z. If needed what should these value be set. I learnt that it should be the mocap reading of drone attached with markers on its origin. If so, should I just set the values directly as read by the mocap rigidbody centroid Or is there an specific format to be set for this ??? Can you please kindly let us know about it.

@scalnoor I don’t think it is related to your issue

@Jaeyoung-Lim thank you. With help of EKF2 tuning on GYRO and ACC noise vehicle parameters I was able to get little better result compared to last time as seen in the below logs please kindly request you to review the logs bit carefully. But I still face the issue where local position x & y does not follow the set point. Form the graph seen in flight review logs it is seen that the drone is making an CIRCLE like pattern flying. I am not pretty familiar, but sure as I have flew this for about ten time it makes the same behavior of CIRCLES. I would believe any one across the forum has come across similar issue can you please kindly help us out. Looking forward to your reply.

https://review.px4.io/plot_app?log=ff4c1a1e-b212-436c-9ad7-50b47b922984