Set RC to Param - Temporary loss of control and data loss

Hi All,

I have attempted to tune the various PID loops on my PX4 multicopter using the Set RC to Param feature in QGroundControl. I am able to see it working while on the ground, and verify that turning the knob results in the parameter changing through the range of values that I desire. However, while actually trying to tune my multicopter in the air, I will lose control of the vehicle momentarily.

The behavior during the period ranges from continuing to move in one direction while sticks off in altittude mode, loss of all propulsion resulting in a crash, and doing a complete flip and falling to the ground. This only appears to happen while I have a parameter set to control from an RC input, and only when I try to adjust the knob. I can also replicate this behavior on the ground with props off. I notice that the lights on the pixhawk turn off momentarily, and the motors continue moving. When the lights come back on, the motors all seem to make very large adjustments.

Upon analyzing the ULog data, there is a noticeable gap in the data where the even occurs.

Has anyone else had this experience?

I’m using PX4 firmware, Pixhawk 2, QGroundControl and have a octo_cox frame (also tested with hex frame).

Thanks,
Jim

Just to follow up on this, has anyone ever used the RC to Param feature at all? Did it work on previous versions of the software? Does anyone have any advice for it to act in a more stable manner?

This would be extremely helpful for my application, but as of now it is unusable since it causes my vehicle to crash every time.

Can you share the log on flight review? http://logs.px4.io/

What was the cpu usage like during the param changes? I have a feeling that may be the underlying issue.

Here is the log: http://logs.px4.io/plot_app?log=90f7f217-124f-4e75-ad28-beb40b2f8f4c

It looks like the CPU usage was reported around 80% from the log.

Thanks!

One problem is the cpu spikes during param change that concern me. We fixed this on master yesterday and it will be in the next stable point release (v1.6.4). Could you try again?

The other is the MC_PITCHRATE_P initially goes to 0 which is quite dangerous. Is this value what you intended to set with the knob?

I’ll give it another try with the updated code shortly.

That was not the intended setpoint for the MC_PITCHRATE_P parameter. Normally, when I map the parameter to an RC knob, I have to change the scale value to give it a usable range of motion. The QGroundControl GUI mentions not to change this parameter, but it’s the only way to make this feature usable. I did not change the scale parameter this time and a small knob adjustment makes the parameter change to the endpoints. I’ll set it correctly so it does not go to zero when I test with the updated code.

Ok that makes sense. You might consider doing some bench testing first with SDLOG_MODE 1 (log from boot).

Hi,

I was finally able to test this again with the latest stable release, and saw somewhat similar results. I lost control of the vehicle at one point during the flight while slowly adjusting the knob. On the ground, you can see the status led does not fade smoothly while turning the knob indicating that the function that controls this is being blocked by another process.

https://review.px4.io/plot_app?log=3f8bbef5-b791-4f88-8019-27a81cf9b20b
looking at the plot, you can see that I lost RC multiple times during the flight (the long period of lost RC at the end is a result of shutting off the RC)

I don’t think that I will be using this function any more, but just wanted to let you know that the issue could still be present.

Thanks!

Hi all,

I can confirm that the issue is still present on my multicopter with the latest stable release 1.6.5 on Pixhawk 1 (FMUv2). I get CPU loads of over 90% and the RC is lost repeatingly. I did not test it in flight because this would be too dangerous. The issue can easily be reproduced on the ground. BTW, I wanted to set the Parameter MPC_XY_VEL_MAX.

Is there any update maybe for the next stable release?

Kind regards,
Thomas

@VertLim can you share the log (https://logs.px4.io/)?

https://logs.px4.io/plot_app?log=5a983629-e335-44df-bf4c-eba988655847

As you can see, the problems start at around 5:30min. This was the time when I enabled the RC to param function in QGC 3.2.4 for the parameter MPC_XY_VEL_MAX (scale 6, center 8, min 2, max 14). CPU load increases significantly, RC is lost continuously and you can visually see, that the Pixhawk status LED is not flashing smoothly any more. Exactly as @JRD described the issue some time ago.

Thomas

Thanks for sharing the log. You can really see it under cpu usage, with spikes corresponding with param changes.

Can you try again with current PX4 master? Flash through QGC under advanced.

Hi,

not looking good. Unfortunately, I experienced very similar issues with the current master FW.

https://logs.px4.io/plot_app?log=74c8b434-eccf-4af9-bef9-74b6a8647be7

Seems to be a dangerous issue when using the RC to param functionality. Please advice how I could help to solve this problem.

Thomas

Thanks for sharing the log. The issue is that when you change a param it notifies every module to update. This is acceptable when manually changing a param, but falls apart when rapidly changing them.

I’ll try reproducing this locally and see if I can find an acceptable solution.

Hi Dagar,

any news on the issue? I would like to tell you why this is a problem also when not using the RC to param functionality. We are using a flight planning software, which is writing some parameters before every mission upload (e.g. MIS_YAWMODE, RTL_RETURN_ALT, NAV_RCL_ACT, etc.). Mission uploads are not only done on ground but also in the air.

We have developed some electronics/logics, detecting issues partly also on the actuator output PWM signals. This failure detection is quite often triggered during mission upload, or in more detail during the parameter writing before a mission upload. When analysing the load using top on NSH you can clearly see, that lpwork is using a large amount of the CPU resources.

Although it was written in the discussion High lpwork CPU % at v1.6.0, that this does not affect the real-time tasks, such as the rate controller, this cannot be entirely true as you can see in the following log. You can easily see, that the actuator_outputs hang for a couple of milliseconds during the parameter write process. This is enough for our system to trigger a fatal error.

Although we could of course increase the detection trigger time, I would really prefer to find and fix the source of issue. And what about the SYS_FMU_TASK parameter? Can this influence the observed behaviour/issue?

Many thanks for your support!
Thomas

I have an idea that will significantly decrease the time a param save takes. Can you help with testing?

Yes, of course! Anytime.

Hi @dagar,

any news on the issue? Should I help testing your idea or is there already something to expect with the upcoming release 1.7.0?

Kind regards,
Thomas

I was going to do it right after the v1.7.0 release.

The idea is to buffer the tiny bson writes. If you take a look at the current implementation (starting https://github.com/PX4/Firmware/blob/master/src/modules/systemlib/param/param.c#L1028) you’ll see that it writes every single little piece individually. As a quick test I buffered a couple hundred bytes per write and it decreased the param save time from around 400ms to less than 100ms. It depends how many parameters you have set of course.

@VertLim PR here - https://github.com/PX4/Firmware/pull/8391