Hi everyone,
I would like to share some practical thoughts from a recent UAV autonomy integration project and discuss a broader architectural question with the PX4 community:
How should we structure the relationship between low-level flight control, onboard perception, mission planning, and AI-based decision-making in a real UAV system?
This post is not intended as a product announcement. I will use one of my own implementation cases, called FlyCore, only as a reference example. The main topic I want to discuss is the system architecture behind autonomous UAVs, especially when we move beyond stable flight and start integrating perception, localization, mapping, recognition, planning, and mission execution into one platform.
1. Background: Flight control is necessary, but not sufficient for autonomy
Traditional flight controllers are very good at what they are designed to do:
- attitude estimation
- sensor fusion for flight stabilization
- motor control
- failsafe handling
- basic navigation
- real-time control loop execution
In this layer, deterministic timing and reliability are more important than raw computing power.
However, once the UAV needs to operate in more complex scenarios, the system usually needs much more than stable flight. For example:
- visual / LiDAR-based localization
- SLAM or real-time mapping
- target detection and recognition
- obstacle understanding
- autonomous path planning
- GNSS-denied navigation
- mission-level task allocation
- onboard AI inference
These workloads have very different characteristics from low-level flight control. They are computationally heavier, less deterministic, and often depend on multiple high-bandwidth sensors.
This creates a common architectural conflict:
The flight controller needs strict real-time behavior, while perception and AI workloads need flexible high-performance computing.
In my view, trying to force all of these tasks into a single computing layer is not ideal. A more practical approach is to separate them into different layers with clear responsibilities.
2. A layered autonomy architecture
In my recent implementation, I used a three-layer structure:
- Perception Layer
- Decision / Planning Layer
- Execution / Flight-Control Layer
The idea is not to replace the flight controller, but to make the autonomy stack more modular and predictable.
3. Execution Layer: keep low-level control deterministic
The execution layer should remain focused on the core flight-control tasks.
This layer is responsible for:
- attitude control
- motor output
- low-level stabilization
- IMU-based state estimation
- real-time control loops
- safety-critical execution
For this part, I believe the design principle should be conservative:
Do not overload the flight-control layer with heavy perception or AI tasks.
In a PX4-based system, this would usually mean that PX4 continues to handle the real-time flight-control responsibilities, while higher-level autonomy modules communicate with it through well-defined interfaces.
The flight controller should receive commands that are already abstracted into a suitable form, such as:
- position setpoints
- velocity setpoints
- attitude setpoints
- trajectory references
- mission commands
- failsafe triggers
This separation helps preserve the reliability of the flight-control loop.
4. Perception Layer: localization, mapping, and environment understanding
The perception layer handles sensor-heavy workloads.
In my implementation case, this layer processes data from sensors such as:
- cameras
- LiDAR
- RTK / GNSS
- IMU
- possibly other payload sensors
The main responsibilities include:
- visual perception
- LiDAR processing
- SLAM
- mapping
- localization
- environment reconstruction
- obstacle information extraction
This layer typically requires more computing power than a traditional microcontroller-based flight controller can provide.
One practical design choice is to deploy the perception workload on an onboard computer that is separate from the flight-control unit. In the FlyCore implementation, this role is handled by a perception and localization computing unit based on an Intel Ultra platform.
From an engineering perspective, the key question is not just computing power. The harder problem is usually data consistency.
For example, if the camera frame, LiDAR scan, IMU data, and GNSS timestamp are not aligned properly, the fused localization result may become unstable. This is especially obvious during high-speed motion, vibration, or rapid attitude changes.
So for this layer, I think three points are especially important:
- stable sensor drivers
- accurate timestamping
- reliable time synchronization across sensors
Without these, even a strong SLAM algorithm may perform poorly in real flight.
5. Decision / Planning Layer: from perception result to executable motion
The decision layer sits between perception and execution.
It does not directly control motors. Instead, it interprets the environment and generates higher-level motion or mission commands.
Typical tasks include:
- target recognition
- semantic understanding
- path planning
- obstacle avoidance strategy
- mission allocation
- task-level decision-making
- trajectory generation
In the FlyCore case, this layer is handled by a separate recognition and planning unit based on an RK3588 processor. The software stack includes visual recognition modules and planning-control algorithms such as SpireCV and Prometheus.
The reason for separating this layer from perception is that the two workloads are related but not identical.
The perception layer answers:
Where am I, and what does the environment look like?
The decision layer answers:
What should the UAV do next?
The execution layer answers:
How do I safely execute the command in real time?
This distinction makes the system easier to debug and extend.
6. Why timing and synchronization matter more than they appear
One issue that is easy to underestimate is sensor clock synchronization.
In simulation or slow ground testing, timestamp errors may not look serious. But in real flight, even small timing offsets can affect localization and control quality.
For example:
- the camera captures the environment at time T1
- the IMU reports attitude at time T2
- the LiDAR scan corresponds to a slightly different time window
- the planner uses fused data that may already be delayed
If these data streams are not aligned, the system may produce a localization result that is mathematically valid but physically inconsistent.
This can cause problems such as:
- map distortion
- localization drift
- unstable obstacle avoidance
- delayed control response
- inaccurate trajectory tracking
For this reason, I think hardware-level or system-level synchronization should be treated as part of the autonomy architecture, not as a minor implementation detail.
In my case, FlyCore uses a hardware-level synchronization design to reduce the time-base mismatch among vision sensors, LiDAR, IMU, and GNSS.
I would be interested to hear how others in the PX4 community handle this in their own systems, especially when combining PX4 with external perception computers.
7. Communication between autonomy stack and flight-control stack
Another important topic is the communication boundary between the autonomy computer and the flight controller.
In a layered architecture, the upper-level autonomy system should not send low-quality or unstable commands directly to the flight controller.
Some practical questions include:
- Should the planner output position setpoints, velocity setpoints, or full trajectories?
- Where should trajectory smoothing be handled?
- How much authority should the companion computer have?
- How should failsafe logic be divided between PX4 and the companion computer?
- How should communication delay be measured and bounded?
- What is the best way to handle degraded perception or localization failure?
In my opinion, the flight controller should always maintain a clear safety boundary. The companion computer can provide intelligent commands, but the flight-control layer should still handle safety-critical behavior.
This is especially important in GNSS-denied or partially observable environments.
8. Open-source toolchain and engineering reproducibility
Another lesson from this implementation is that the software toolchain matters as much as the hardware architecture.
For UAV autonomy development, we need more than one algorithm demo. A useful engineering system should support:
- modular perception nodes
- reusable planning modules
- standardized communication
- data logging
- replay and debugging
- simulation-to-real validation
- fast replacement of sensors and algorithms
In my case, the architecture uses a combination of:
- Prometheus for autonomous UAV planning and control logic
- SpireCV for visual recognition workflows
- SpireMS as a lightweight messaging layer
The purpose is to make perception, planning, streaming, saving, and control modules easier to split into independent nodes.
This node-based design is useful because it allows developers to replace or modify one part of the system without rebuilding the whole autonomy stack.
For example:
- replace the detection model without changing the flight controller
- modify the planning algorithm without changing the perception pipeline
- add a new sensor without rewriting the whole mission logic
- test a module offline before deploying it to the UAV
This is important for research platforms and industrial UAV development, where iteration speed matters.
9. Practical validation scenarios
The architecture was mainly designed for scenarios where a UAV needs to operate with limited or unreliable GNSS.
Typical test conditions include:
- indoor environments
- low-light areas
- narrow spaces
- partially occluded scenes
- dense forest-like environments
- night operation
- inspection-type missions
These scenarios are difficult because they combine multiple problems:
- degraded GNSS
- unstable visual features
- complex obstacle geometry
- limited lighting
- sensor noise
- narrow flight space
- high requirement for localization stability
In these cases, the UAV cannot rely only on a traditional flight controller and GNSS navigation. It needs a more complete autonomy stack that combines perception, localization, planning, and control.
10. Discussion points for the PX4 community
I would like to use this case to raise several technical questions for discussion:
-
Where do you draw the boundary between PX4 and the companion computer?
Should PX4 only handle low-level control, or should more autonomy logic be pushed into the flight-control side?
-
What command interface do you prefer for autonomy systems?
Position setpoints, velocity setpoints, trajectory setpoints, or mission-level commands?
-
How do you handle time synchronization in multi-sensor UAV systems?
Especially when combining camera, LiDAR, IMU, GNSS, and onboard computing.
-
What is your preferred architecture for GNSS-denied flight?
Visual-inertial odometry, LiDAR-inertial odometry, multi-sensor fusion, or other approaches?
-
How should failsafe responsibility be divided?
If the companion computer loses localization or planning confidence, should PX4 take over immediately, or should the companion system handle a degraded mode first?
-
How do you validate perception-to-control delay?
Do you measure end-to-end latency from sensor input to control command output? If so, what tools or methods do you use?
11. Conclusion
From this implementation, my main takeaway is that UAV autonomy should not be treated as a simple extension of the flight controller.
A more robust approach is to separate the system into layers:
- real-time flight control
- perception and localization
- decision-making and planning
- communication and synchronization
- safety and fallback logic
FlyCore is only one practical implementation of this idea. The broader question is how to design a reproducible and reliable autonomy architecture around a flight-control system such as PX4.
I would be very interested to hear how other developers structure their systems, especially in real-world UAV projects involving SLAM, onboard AI, GNSS-denied navigation, and autonomous inspection.