Discussion: A Layered Architecture for UAV Autonomy Beyond Basic Flight Control

PX4 Autopilot
1

Hi everyone,

I would like to share some practical thoughts from a recent UAV autonomy integration project and discuss a broader architectural question with the PX4 community:

How should we structure the relationship between low-level flight control, onboard perception, mission planning, and AI-based decision-making in a real UAV system?

This post is not intended as a product announcement. I will use one of my own implementation cases, called FlyCore, only as a reference example. The main topic I want to discuss is the system architecture behind autonomous UAVs, especially when we move beyond stable flight and start integrating perception, localization, mapping, recognition, planning, and mission execution into one platform.

1. Background: Flight control is necessary, but not sufficient for autonomy

Traditional flight controllers are very good at what they are designed to do:

  • attitude estimation
  • sensor fusion for flight stabilization
  • motor control
  • failsafe handling
  • basic navigation
  • real-time control loop execution

In this layer, deterministic timing and reliability are more important than raw computing power.

However, once the UAV needs to operate in more complex scenarios, the system usually needs much more than stable flight. For example:

  • visual / LiDAR-based localization
  • SLAM or real-time mapping
  • target detection and recognition
  • obstacle understanding
  • autonomous path planning
  • GNSS-denied navigation
  • mission-level task allocation
  • onboard AI inference

These workloads have very different characteristics from low-level flight control. They are computationally heavier, less deterministic, and often depend on multiple high-bandwidth sensors.

This creates a common architectural conflict:

The flight controller needs strict real-time behavior, while perception and AI workloads need flexible high-performance computing.

In my view, trying to force all of these tasks into a single computing layer is not ideal. A more practical approach is to separate them into different layers with clear responsibilities.

2. A layered autonomy architecture

In my recent implementation, I used a three-layer structure:

  1. Perception Layer
  2. Decision / Planning Layer
  3. Execution / Flight-Control Layer

The idea is not to replace the flight controller, but to make the autonomy stack more modular and predictable.

3. Execution Layer: keep low-level control deterministic

The execution layer should remain focused on the core flight-control tasks.

This layer is responsible for:

  • attitude control
  • motor output
  • low-level stabilization
  • IMU-based state estimation
  • real-time control loops
  • safety-critical execution

For this part, I believe the design principle should be conservative:

Do not overload the flight-control layer with heavy perception or AI tasks.

In a PX4-based system, this would usually mean that PX4 continues to handle the real-time flight-control responsibilities, while higher-level autonomy modules communicate with it through well-defined interfaces.

The flight controller should receive commands that are already abstracted into a suitable form, such as:

  • position setpoints
  • velocity setpoints
  • attitude setpoints
  • trajectory references
  • mission commands
  • failsafe triggers

This separation helps preserve the reliability of the flight-control loop.

4. Perception Layer: localization, mapping, and environment understanding

The perception layer handles sensor-heavy workloads.

In my implementation case, this layer processes data from sensors such as:

  • cameras
  • LiDAR
  • RTK / GNSS
  • IMU
  • possibly other payload sensors

The main responsibilities include:

  • visual perception
  • LiDAR processing
  • SLAM
  • mapping
  • localization
  • environment reconstruction
  • obstacle information extraction

This layer typically requires more computing power than a traditional microcontroller-based flight controller can provide.

One practical design choice is to deploy the perception workload on an onboard computer that is separate from the flight-control unit. In the FlyCore implementation, this role is handled by a perception and localization computing unit based on an Intel Ultra platform.

From an engineering perspective, the key question is not just computing power. The harder problem is usually data consistency.

For example, if the camera frame, LiDAR scan, IMU data, and GNSS timestamp are not aligned properly, the fused localization result may become unstable. This is especially obvious during high-speed motion, vibration, or rapid attitude changes.

So for this layer, I think three points are especially important:

  • stable sensor drivers
  • accurate timestamping
  • reliable time synchronization across sensors

Without these, even a strong SLAM algorithm may perform poorly in real flight.

5. Decision / Planning Layer: from perception result to executable motion

The decision layer sits between perception and execution.

It does not directly control motors. Instead, it interprets the environment and generates higher-level motion or mission commands.

Typical tasks include:

  • target recognition
  • semantic understanding
  • path planning
  • obstacle avoidance strategy
  • mission allocation
  • task-level decision-making
  • trajectory generation

In the FlyCore case, this layer is handled by a separate recognition and planning unit based on an RK3588 processor. The software stack includes visual recognition modules and planning-control algorithms such as SpireCV and Prometheus.

The reason for separating this layer from perception is that the two workloads are related but not identical.

The perception layer answers:

Where am I, and what does the environment look like?

The decision layer answers:

What should the UAV do next?

The execution layer answers:

How do I safely execute the command in real time?

This distinction makes the system easier to debug and extend.

6. Why timing and synchronization matter more than they appear

One issue that is easy to underestimate is sensor clock synchronization.

In simulation or slow ground testing, timestamp errors may not look serious. But in real flight, even small timing offsets can affect localization and control quality.

For example:

  • the camera captures the environment at time T1
  • the IMU reports attitude at time T2
  • the LiDAR scan corresponds to a slightly different time window
  • the planner uses fused data that may already be delayed

If these data streams are not aligned, the system may produce a localization result that is mathematically valid but physically inconsistent.

This can cause problems such as:

  • map distortion
  • localization drift
  • unstable obstacle avoidance
  • delayed control response
  • inaccurate trajectory tracking

For this reason, I think hardware-level or system-level synchronization should be treated as part of the autonomy architecture, not as a minor implementation detail.

In my case, FlyCore uses a hardware-level synchronization design to reduce the time-base mismatch among vision sensors, LiDAR, IMU, and GNSS.

I would be interested to hear how others in the PX4 community handle this in their own systems, especially when combining PX4 with external perception computers.

7. Communication between autonomy stack and flight-control stack

Another important topic is the communication boundary between the autonomy computer and the flight controller.

In a layered architecture, the upper-level autonomy system should not send low-quality or unstable commands directly to the flight controller.

Some practical questions include:

  • Should the planner output position setpoints, velocity setpoints, or full trajectories?
  • Where should trajectory smoothing be handled?
  • How much authority should the companion computer have?
  • How should failsafe logic be divided between PX4 and the companion computer?
  • How should communication delay be measured and bounded?
  • What is the best way to handle degraded perception or localization failure?

In my opinion, the flight controller should always maintain a clear safety boundary. The companion computer can provide intelligent commands, but the flight-control layer should still handle safety-critical behavior.

This is especially important in GNSS-denied or partially observable environments.

8. Open-source toolchain and engineering reproducibility

Another lesson from this implementation is that the software toolchain matters as much as the hardware architecture.

For UAV autonomy development, we need more than one algorithm demo. A useful engineering system should support:

  • modular perception nodes
  • reusable planning modules
  • standardized communication
  • data logging
  • replay and debugging
  • simulation-to-real validation
  • fast replacement of sensors and algorithms

In my case, the architecture uses a combination of:

  • Prometheus for autonomous UAV planning and control logic
  • SpireCV for visual recognition workflows
  • SpireMS as a lightweight messaging layer

The purpose is to make perception, planning, streaming, saving, and control modules easier to split into independent nodes.

This node-based design is useful because it allows developers to replace or modify one part of the system without rebuilding the whole autonomy stack.

For example:

  • replace the detection model without changing the flight controller
  • modify the planning algorithm without changing the perception pipeline
  • add a new sensor without rewriting the whole mission logic
  • test a module offline before deploying it to the UAV

This is important for research platforms and industrial UAV development, where iteration speed matters.

9. Practical validation scenarios

The architecture was mainly designed for scenarios where a UAV needs to operate with limited or unreliable GNSS.

Typical test conditions include:

  • indoor environments
  • low-light areas
  • narrow spaces
  • partially occluded scenes
  • dense forest-like environments
  • night operation
  • inspection-type missions

These scenarios are difficult because they combine multiple problems:

  • degraded GNSS
  • unstable visual features
  • complex obstacle geometry
  • limited lighting
  • sensor noise
  • narrow flight space
  • high requirement for localization stability

In these cases, the UAV cannot rely only on a traditional flight controller and GNSS navigation. It needs a more complete autonomy stack that combines perception, localization, planning, and control.

10. Discussion points for the PX4 community

I would like to use this case to raise several technical questions for discussion:

  1. Where do you draw the boundary between PX4 and the companion computer?

    Should PX4 only handle low-level control, or should more autonomy logic be pushed into the flight-control side?

  2. What command interface do you prefer for autonomy systems?

    Position setpoints, velocity setpoints, trajectory setpoints, or mission-level commands?

  3. How do you handle time synchronization in multi-sensor UAV systems?

    Especially when combining camera, LiDAR, IMU, GNSS, and onboard computing.

  4. What is your preferred architecture for GNSS-denied flight?

    Visual-inertial odometry, LiDAR-inertial odometry, multi-sensor fusion, or other approaches?

  5. How should failsafe responsibility be divided?

    If the companion computer loses localization or planning confidence, should PX4 take over immediately, or should the companion system handle a degraded mode first?

  6. How do you validate perception-to-control delay?

    Do you measure end-to-end latency from sensor input to control command output? If so, what tools or methods do you use?

11. Conclusion

From this implementation, my main takeaway is that UAV autonomy should not be treated as a simple extension of the flight controller.

A more robust approach is to separate the system into layers:

  • real-time flight control
  • perception and localization
  • decision-making and planning
  • communication and synchronization
  • safety and fallback logic

FlyCore is only one practical implementation of this idea. The broader question is how to design a reproducible and reliable autonomy architecture around a flight-control system such as PX4.

I would be very interested to hear how other developers structure their systems, especially in real-world UAV projects involving SLAM, onboard AI, GNSS-denied navigation, and autonomous inspection.

2

I would like to ask whether the laboratory currently has interns or autumn recruit budget, I am currently in Chengdu, have rich experience in px4 secondary development, the current direction of navigation and positioning, I am very interested in your current project, as a drone enthusiast, very yearning for the development atmosphere of the laboratory, or as a enthusiast, you can carry out technical exchanges.

3

Thank you for your interest in our work and for sharing your background. Your experience in PX4 secondary development, navigation, and localization is closely related to the areas we are currently working on. At this stage, I cannot yet confirm whether there will be formal internship or autumn recruitment openings, as this depends on our project and hiring plans. Please feel free to send me a private message with your CV, a brief introduction to your PX4 projects, and any GitHub repositories or demonstration links. I can share your information with our team for review. Even if there is no suitable position immediately, we would still be happy to stay in touch and exchange ideas on PX4, SLAM, navigation, and autonomous UAV development.

4

Thank you for your sharing. I have two questions regarding these topics:

  1. On time synchronization: How is time synchronization implemented between camera‑captured image data and PX4 attitude data? After all, these two sensors run on separate hardware and even different operating systems.
  2. Regarding latency: In our camera deployment, we notice considerable latency from image processing. Computing flight‑controller control commands also takes time, and latency exists across the entire communication link. Do you have any practical experience to share for resolving such latency‑related issues?
5

Thank you—these are two very practical questions.

  1. Time synchronization

In our implementation, we do not associate images and PX4 attitude data by message arrival time. On the FlyCore perception side, IEEE 1588-2008 PTP and PPS are used as timing references where supported. If the camera provides hardware trigger or exposure feedback, the frame is timestamped at the actual exposure event rather than when the image reaches the application.

PX4 and the companion computer still have different clock domains, so we estimate the offset through MAVLink TIMESYNC or the automatic time synchronization provided by uXRCE-DDS. The camera timestamp is then converted to the PX4 time base. We maintain a short buffer of PX4 vehicle_attitude data and interpolate the attitude at the frame’s exposure timestamp using timestamp_sample.

Therefore, running different operating systems is not the main problem. The important points are having an observable exposure timestamp and mapping all timestamps onto a consistent time base. Timestamping only when the frame is received by the driver introduces USB, network, scheduling, and buffering jitter.

  1. End-to-end latency

Our first step is always to measure the pipeline rather than assume a fixed delay. We record timestamps for:

camera exposure;
frame arrival;
completion of perception/inference;
planner or controller output;
command reception by PX4.

We then align the companion-computer log with the PX4 ULog and examine not only the average delay, but also the 95th/99th percentile and maximum jitter.

In practice, the following measures are useful:

process data according to its capture timestamp, not its arrival time;
buffer historical attitude/state data and propagate the perception result from capture time to the current time using IMU and velocity information;
when external vision is fused by EKF2, compensate the measured vision delay and verify it through estimator innovations;
use asynchronous processing with a queue depth of one and drop stale frames instead of allowing latency to accumulate;
separate video traffic from the flight-control link, avoid unnecessary encoding or serialization, and optimize the slowest processing stage;
keep fast and safety-critical control loops inside PX4, while the companion computer sends position, velocity, or trajectory-level setpoints;
define a latency/jitter threshold. If it is exceeded, reduce speed, hold position, or trigger the configured Offboard failsafe.

A fixed and measured delay can often be compensated; large and unpredictable jitter is much more difficult. There is also no universal acceptable latency value—it depends on vehicle speed and dynamics. As a simple reference, the spatial error caused by stale data is approximately vehicle speed multiplied by total latency.