Apparent catastrophic PX4IO failure

Hello, I have encountered a very troubling issue that I believe is related to the PX4IO driver, where everything basically ‘freezes’ and all control is lost.

I have been working on a customized version of PX4 v1.9.0 on the Pixhawk 2.1 to implement a dynamic mixer, similar to what @simopanzer posted in this thread. I have tested my dynamic mixer in SITL, and that works.

When I tried HITL, I initially had issues where PX4IO would fail to get uploaded properly as indicated by the buzzer warning message and bootlog, but I debugged those (the issues came from the fact that I had tried to modify the ‘Rotor’ struct so that it was no longer defined as a const which had involved modifying px_generate_mixers.py). After doing a few more hacky things to solve this, this no longer was an issue and PX4IO apparently appeared to work fine.

However, the real frightening problems showed up when I tried running the code on a real hexacopter. I armed the vehicle with no propellers on to make sure that everything was working fine (i.e. checking that RC inputs were properly affecting the motors), and it was working great for about a minute, until suddenly the vehicle stopped responding to any sort of input. The motors were just sort of ‘stuck’ at the same PWM. I would have thought this was related to an FMU issue, but also, the RC inputs were not appearing in QGroundControl, leading me to believe that somehow PX4IO failed. Here is a log of that test. The logging apparently continued after the failure occurred (which was at 1:47).

I am able to repeat this by restarting the vehicle. It consistently freezes either from the moment I turn the vehicle on (before arming), or after a few minutes but with no apparent trigger or root cause. Sometimes it will freeze even with absolutely no input. I don’t believe it is a memory allocation issue (I ran the ‘top’ command, and didn’t see any abnormalities), though wouldn’t be surprised if it was related to it.

I of course know that customizing the firmware means that the autopilot is at risk for failure, so I am definitely not posing this as an issue with PX4, but I would greatly appreciate if anyone could help me understand why this may be happening, or at least point me in the direction of being able to debug or solve it. I have been browsing the forums and debugging-related documentation for answers, but I am a mediocre coder at best, and debugging this kind of semi-low-level issue is difficult for me.

Thanks!
Gus