VIO vs Optical Flow

PX4 Autopilot Drivers & Sensors 🔧
1

Both use cameras, both measure difference between frames. One is downward facing, the other uses a stereo camera or a 45 degree tracking camera.

For a local position estimate accuracy, assuming perfect calibration of camera and sensors, is one better than the other and why.

2

I asked the two smartest people I know

James Strawson at ModalAI (implemented VIO on VOXL2)
Alex Klimaj (created the ArkFlow board)

Would like to note that BOTH indicated that the best outcome would be a combination of both, and Alex indicated there are people flying with both today.

James

Optic flow requires an accurate measurement of the distance to the ground and also assumes a planar surface. If both of those are met then optic flow can be just as accurate and more reliable than VIO. Ideally one would use a combination of both. it’s more robust due to having fewer states, VIO is very complicated with a lot of states.
Yes, one could contrive some scenarios where a camera could see texture that could be tracked with patch-based optic flow but does not have point features, but our experience flying high up outdoors is that the real world almost always has features to track.
This reminds me of the Multi-IMU / Multi EKF2 thing in PX4. It’s all based on the idea of “what would happen if an IMU fails”, yet I’ve never once in 12 years of making drones had an IMU fail in flight. Sensor choice should be based on specific environments not wild guesses.

Alex

Probably a combination of the two is best. Downward facing optical flow gives you a planar velocity thats corrected for angular velocity with the gyro. Then you can fuse external vision odometry on top of that. I think they each have their place. You can’t do any obstacle avoidance or mapping with optical flow.
I know a number of companies using my flow with VIO as well. At the very least the flow allows you to hold position when the VIO falls apart

Takeaway

These two are not that different. Just like anything, both of these need cameras to understand localization. There are instances where optical flow can be more reliable, assuming accurate height and a flat surface. Probably great for low flying drones indoors. Like most things, VIO is more complex, but you can do more with it…especially use it in more outdoor settings as backup incase of GPS loss.

3 Likes
3

I love these kind of questions - enough that I’m injecting it into the docs: computer vision - vio vs optical flow by hamishwillee · Pull Request #2795 · PX4/PX4-user_guide · GitHub

The both said “a combination is probably best”, but what are the tradeoffs if you can only have one, and when do you make them?

My interpretation was that

  • if you think your surface can only be planar then optical flow is viable, but maybe not in very hilly or high foliage terrain?
  • VIO is probably going to work everywhere as long as you have some features you can track in the field of view. In practise it sounds like there always are.

So it sounds like VIO is overall the best choice if you can only have one?

Then what are the tradeoffs - cost, finding a VIO library that will do the job? Or might it be something like speed you can move at before it all falls over?

4

Trade off is simplicity. Vio requires separate compute, calibration, software, configuration etc. Flow (in most cases) is simply plugging in a flow sensor, a rangefinder, and setting up a few parameters.

1 Like
5

I would agree with @auturgy that the tradeoff is simplicity, and with VIO you need more powerful onboard compute

I would also agree with your bullet points @hamishwillee

All depends on operating environment.

If you plan on flying outdoors without GPS → VIO
If you plan on flying indoors without GPS → optical flow
If you plan on flying indoors & outdoors without GPS → VIO
If you care about costs → optical flow (ark flow is $250, ModalAI VOXL 2 computer+tracking ~$1600 but has WAY more capability)
If you care about building more complex features ontop → VIO

6

Thank you @auturgy and @AWilkins_Ascend - That’s really helpful. I’ve made the costs and decision path clear in computer vision - vio vs optical flow by hamishwillee · Pull Request #2795 · PX4/PX4-user_guide · GitHub and merged.

7

I like to mention something about Optical Flow.

You said that optical flow is great for low-flying drones indoors. Still, actually, because of the assumption of a flat surface, it will probably be better to fly outdoors and at high altitudes. The differences in the objects’ heights in the image, have a minor effect compared to the altitude of the drone.

In addition, usually indoors you have fluorescent lights that can make the optical flow impossible because fluorescent lights operate on alternating current, causing them to flicker at a frequency typically around 100-120hz.
That means the optical flow will probably see the same scene with different lights, and usually, optical flow makes an assumption that the colors of the pixels are the same between the pairs of the images it calculates on, so it is very bad for the optical flow and can cause poor performance.

8

Interesting! Thank you for your input.

Do you have any real world trials of these that we can reference?