Sindre
March 3, 2025, 2:30pm
2
I have worked around the same problem with the TFLM library, you can see how I’ve solved it (very hacky solution) in the CMakeLists.txt files of this PR:
main ← SindreMHegre:neural_control_with_tflm
opened 12:23PM - 19 Feb 25 UTC
### Solved Problem
Using neural networks for various tasks on UAVs in research … is becoming more and more commonplace. My aim is to provide a module that can be used to replace the MC controller with NN to lower the barrier for using RL and NN on UAVs in general, and give researchers and hobbyist a place to start for NN use.
### Solution
- Add TensorFlow Lite Micro as an inference lib into PX4
- Make custom board configs with neural controllers
- Make a module for NN control
### Changelog Entry
For release notes:
```
Feature/Bugfix XYZ
New parameter: XYZ_Z
Documentation: Need to clarify page ... / done, read docs.px4.io/...
```
### Alternatives
We could also look into using Eigen or executorch
### Test coverage
- Built and tested SITL with the x500
https://review.px4.io/plot_app?log=52502d02-009e-4d1f-baac-a49dd51cfdef (check neural_control topic)
- Built and tested HITL with pixracerpro and jMAVSim Hitl, note that the sim itself seemed a bit unstable and the network is not optimized for this model.
https://review.px4.io/plot_app?log=43d2ba6c-3d05-4d7f-a31b-490a57b3e647
- Plan on testing on the x500 v2 kit with the 6C
### Context
Help:
- I'm not very experienced with CMakeLists.txt, so any pointers in how to improve the code quality there is much appreciated.
- Also for the TFLM submodule, should I make a PX4 branch for it? With the dependencies already downloaded and the standard c++ header files extracted. Should it be on a fork or try to get it upstream?
- Would also like it to have documentation on the main docs, where do I start writing that?
- Should it be merged into main or as a separate branch upstream?
Other feature requests:
- Put in different controller parameters with possibilities to replace different parts: Full e2e, only control, only allocation, Allocation and control as two separate networks.
- Script to give thrust and delay parameters from a flight log
Info:
The networks I'm flying with right now is not mine and is being used for research, so they will be open sourced at some point, but can't give them away at the moment
Inference times on the pixracer pro:
control net: 70 microseconds
allocation net: 13 microseconds
Full NN controller: 141 microseconds
You also have to make sure to exclude tests, I had this problem with TFLM, and I know another who have used Eigen for other hardware who had problems with tests there as well
