PX4 Vision Estimation: Questions on Integration with Pixhawk

Hello Pixhawk Community,

I am asking for some advice on integrating an external vision position estimation with PX4 (on a pixhawk 2) via MavROS on behalf of our company, Airlitix. We have several questions about the integration, and proper channels for debugging.

1.) We currently generate a position estimation as a ROS PoseStamped message, broadcast to the “/mavros/vision_pose/pose” topic (since we lack covariance information). By what I’ve found, this is right, but is this a channel that will get information to the PX4 appropriately?

2.) What is the best standard practice for checking whether vision data is reaching the Pixhawk? Can we see the data the pixhawk gets?

3.) What is the best practice for ensuring the vision pose is being fused into the estimator appropriately? Which estimator works best for fusing vision data?

4.) Our application requires flying in a compass- and GPS-denied environment. Is there a way to use orientation information from vision estimation as a global frame direction? In other words, can we completely replace the compass direction with our vision’s direction estimation?

5.) In the case that the above points can’t be addressed simply, and require changes to the firmware, what files in the firmware file structure are primarily involved with reading and utilizing an external vision estimation?

Thank you in advance for your help.

1 Like

I assume, you’ve read External Position Estimation (Vision/Motion based) · PX4 Developer Guide. If not, you should probably do so :slight_smile:

Yes, this should work, in fact, we use it exactly this way. One thing to consider is the coordinate frame. PX4 uses NED and ROS uses ENU axes direction. However, mavros handles that for you (mavros/mavros_extras/src/plugins/vision_pose_estimate.cpp at master · mavlink/mavros · GitHub), so you should use ROS conventions in your ROS code.

VISION_POSITION_ESTIMATE messages are “echoed” via mavlink by PX4, so you can see them in “mavlink inspector” in QGroundControl. You can also check “vehicle_vision_position” uORB topic in PX4 logs (it’s not shown in flight review app, but you can examine all uORB topics by processing logs with ulog2csv).

EKF2 is the preferred estimator, we use it with vision position fusion and it works for us as far as I know :slight_smile:

I haven’t tried using VPE in both GPS and compass-denied environment (only in GPS-denied), but you should be able to enable fusion of yaw data from VPE (check EKF2_AID_MASK parameter) and ignore yaw data from magnetometer (see EKF2_MAG_TYPE). I haven’t tried this myself so I am unaware of any problems this might introduce.

We are using VPE to perform position-controlled flight in GPS-denied environments and we didn’t make any changes to PX4 firmware to achieve that. Hooray to PX4!

1 Like

Thanks for the detailed and helpful response, shylent! Unfortunately, I’ve been running through these steps, and it seems that something in our setup is atypical, as vision estimation still does not seem to work.

I took the following steps:

  • I reviewed the top-level link, and made sure I set parameters appropriately. I made sure to enable EKF2, and I set EKF2_AID_MASK to 24, which should be fusing vision position and vision yaw. I set EKF2_HGT_MODE to 3 for vision-based height mode.
  • I ensured the companion computer and the pixhawk are communicating via MavProxy.py.
  • I ran vision position estimation on our companion computer, and ensured it was being sent over the topic /mavros/vision_pose/pose: no problem here. When I searched for VISION_POSITION_ESTIMATE messages through the mavlink inspector on QGC though, no such topics exist.
  • I checked logs from our flight tests to see if vehicle_vision_position was being logged, but no such CSV file is created after processing with ulog2csv. These files were generated before making the above changes, so this is probably expected though.

I’m at a bit of a loss at what our next steps should be. :confused: I do have a couple of questions though:

1.) My colleagues found information about uORB messaging (https://dev.px4.io/en/middleware/uorb.html), but uORB is not installed on our companion computer (as we haven’t installed the PX4 Firmware there). Is it necessary to install uORB on our companion computer?

2.) Even if we have uORB on the companion computer, I believe that would only let us subscribe to vision estimation topics in our C++ code, but it wouldn’t change the fact that no vision position estimation is being seen on the Pixhawk, correct?

3.) Any other thoughts on why this might not be working?

Are you shure in commuication between Pixhawk and companion computer? In some cases i faced issues with start of mavlink instance on Pixhawk, you can check it by typing in QGC(in console) command mavlink status - normally you should have two instances - one for radiomodem on TELEM1 port, and second - on TELEM2, communicating with companion computer.

I’m pretty sure they communicate fine. The Mavlink Inspector is definitely showing a lot of other parameters that are being communicated, but I will follow-up on whether I have two mavlink status lines. I’ve also tested some basic commands, like rebooting the board, through MavProxy… it would be strange if that worked but the mavlink inspector wasn’t for some reason.


EDIT: I’ve looked into your suggestion, but it does seem the communication is fine. I can see the companion computer over telem2, and it seems like there’s not packet loss. I also reaffirmed that the Mavproxy connection works. I’m uploading a screenshot here of the output of mavlink status and the topics in the mavlink inspector.

@Andrew_at_Airlitix Is there a reason you are using mavproxy? I think that might be the problem. Why not directly connect mavros to the flight controller?

Hi Jaeyoung,

I was just using Mavproxy to verify that a connection was happening, as a bit of a sanity check to make sure there wasn’t a wiring issue.

While we’re running our vision software, we are using MavROS for the communication.

ekf2 is consuming the topic “vehicle_visual_odometry”. If you want to figure out if this topic is published correctly you either need to check a log file for that message or you use the command “listener vehicle_visual_odometry” in the system console. If you are using QGC you can access the system console directly.
You can also do “listener estimator_status” which will print the topic that tells you all about the status of the estimator. You can look at the message definition to figure out what the entries of the topic mean.

Hey tumbili, thanks for the response.

I’ve been doing testing myself to see if I could rule out other hardware issues (I thought it might be an issue with our carrier board; however, the behavior is the same with two different boards).

I did test out listening for vehicle_visual_odometry in the console, but it just returned that the the topic was “never published”. What could be the cause for this topic to not be publishing, even though we are sending data over /mavros/vision_pose/pose?

@Andrew_at_Airlitix Even though you are publishing data over the topic, you need to verify if the message is correctly reaching the flight controller

Does this mean you are using the Jetson? How are you connected to the flight controller?

@Jaeyoung-Lim We are using the Jetson TX2. I’ve been testing using both the DevKit board, and the Orbitty. We’re connecting using a UART connection on the carrier board to the telem2 port on the Pixhawk board.

Right, that’s… what I’m trying to figure out how to do. Following the advice of @tumbili, I can see that vehicle_visual_odometry is not published, so it seem that clearly, even though we’re publishing data, it’s not reaching the flight controller for some reason. I have no idea what that reason might be though. MavROS isn’t throwing any errors or giving any indication that it can’t communicate with the Pixhawk.

@Andrew_at_Airlitix There are two UARTs on the Orbitty carrierboard as far as I know.

Is there any chance that you are connected to UART0 on the orbitty carrier board?

If so, it is used for serial console on the carrier board and your mavlink stream will be corrupted.

@Jaeyoung-Lim :open_mouth: Yes, yes we were using UART0. That’s annoying: I never saw something on the ConnectTech documentation indicating that UART0 was in any way special… I will have to try with UART1.

Do you know if there is something similar with the DevKit? I was trying this on the devboard through UART1 (pins 8/10 on GPIO expand 1), but I know this maps to the same device (/def/ttyS0) as UART0 of the Orbitty. Is there another UART I can test on the DevBoard as well? I’m trying to pour over the schematics of the devkit, but I’m not finding a different open UART that I can plug into.

@Andrew_at_Airlitix I think you will find it somewhere if you look carefully. But it is not obvious at all. I had the same issue on the orbitty and that is how I found out.

I have never used the DevKit, so I am not sure. An easy way to find out is to just see what is coming out of it using a serial console.

@Jaeyoung-Lim Huh, well right now I can’t seem to get any communication to happen over UART1 with the orbitty, but I don’t really have proper soldering tools here. I will have to get back to this after I can test with a more proper and clean connection.

@Jaeyoung-Lim I’ve had some time with the setup and we’ve tried a few things, but we’re still having problems.

We tried the connection over UART1 rather than UART0, but it didn’t change the outcome: there’s still no message passing happening. I even checked to see if IMU data was coming back to the TX2, and we weren’t getting it. We decided to try to rule out UART issues, trying with a USB connection. It turns out, with this set up, vision messages are still not being received by the Pixhawk, but we can get IMU data to the TX2. It seems that in part, there are some issues with UART.

We’ve also been trying to rule out the configuration of the Pixhawk as an issue, trying to see if we can use a different CC (an ODroid). So far, I haven’t been able to get the ODroid and Pixhawk talking properly.

What really puzzles me is the TX2-Pixhawak communication with USB apparently only having single-directional communication. Anybody have ideas?

@Andrew_at_Airlitix It is wierd that you are having issues with the UART. I was able to connect to the flight controller with the UART on the TX2. The fact that you say you can’t even talk with the Odroid means that your configuration is wrong on either side (px4 or companion computer)

Going through USB changes the uart port on the pixhawk side. It means that you are going through telem1 on the pixhawk

Are you sure you have the correct device name and baudrate settings matching the fcu settings?

@Jaeyoung-Lim I think the ODroid issues are separate, and probably due to (surprise!) a bad connection, because the pins are just not snug. Standardized sizes is just too much to ask for it seems. Also the ODroid seems to think there’s something bad about our USB cable, so it won’t even populate on the device list.

Going through USB changes the uart port

Wait, so, if I have a UART->USB connection, I need to do that through Telem1? Everything I’ve seen says the companion computer must be connected through Telem2, but it does seem we can change parameters on the pixhawk to treat Telem1 as a CC link. Should I be doing that?

And yes, I’ve been being careful about device names and baudrates. If I didn’t have the right device names, nothing would communicate at all.

@Andrew_at_Airlitix by USB, I thought you meant by connecting to the USB port on the pixhawk side. Does this mean you were using a FTDI device on the USB on the jetson side?

I still don’t understand why you can’t get connection on the UART for the orbitty carrier board. Is there any chance you are trying to connect to /dev/ttyS1 (sorry if this is a dumb question) because UART1 is not /dev/ttyS1[quote=“Andrew_at_Airlitix, post:18, topic:10237”]
Wait, so, if I have a UART->USB connection, I need to do that through Telem1? Everything I’ve seen says the companion computer must be connected through Telem2, but it does seem we can change parameters on the pixhawk to treat Telem1 as a CC link. Should I be doing that?
[/quote]

No, its better you stay with Telem2

@Jaeyoung-Lim Ah, yeah sorry. No, our hardware guy made a cable that has the 6 pin FTDI connection on one end, and a USB on the other. We are plugging into Telem2 on the Pixhawk, and then the USB side goes into a USB port on the Companion Computer. I know it’s a little confusing to keep track of, but I’ve been trying both with the TX2/Orbitty and with an ODroid, because at this point we’re getting desperate for some answers. Edit: and to clarify, the use of this USB cable with the Orbitty had some progress (connection device in this case is /dev/ttyACM0). We were able to get IMU data to the companion computer with this connection, but no evidence of vision data reaching the pixhawk.

You and I are both quite confused why the UART is not behaving. And no, connecting to /dev/ttyTHS2 as per the documentation. If I didn’t have the right device, when I go to check with MavProxy, I’d never get a heartbeat and the MANUAL> mode prompt, or when running MAVROS, it would just straight up crash if it can’t open the device.

No, it's better you stay with Telem2

Cheers to that. Saves me a lot of pointless trial and error. :slight_smile: