How does people deal with large ULog file?

PX4 Autopilot
1

Hi, I have a ULog file of about 2G and when I try to load it in PlotJuggler, it takes up all the memory that my machine have (16G). So I’m wondering what do people do to deal with large ULog file that can’t be seen with the tools at hands?

Here’s the log file if you want to try to load it: 20_04_20.ulg - Google Drive

Thanks!

1 Like
2

@bkueng How should people deal with very large ulog files? Can they be split safely - i.e. if I split at some point in the file, can I then plot the split parts?

Are there tools for this? There are some scripts in GitHub - PX4/pyulog: Python module & scripts for ULog files for extracting some parts, but nothing that seems to do file splitting.

I figure this would be useful to add to Logging | PX4 User Guide

3

I usually convert the ulog file to csv files using the pyulog python package, and then plot the data with Matlab. Hope it helps.

4

@Ke2023 humm the issue is more with PlotJuggler I guess. Since when trying to load them I was having issue. So I don’t think that having the CSV instead would help. I ought to try though. Although I just tried today with a ULog of 2Go and didn’t have a problem. I need to close application first, but that’s it.

1 Like
5

@hamishwillee I was trying to think of a solution to that problem. And I thought, maybe adding a parameter in PX4 which tell how big each log can be would help with that. Or yeah, just add one script to the PX4Tools repository that split the log in the size that you want would also work.

6

Thanks @Ke2023 - certainly an option.

humm the issue is more with PlotJuggler I guess.

My take on this is that every tool and computer will have limits. Splitting a file might become a good alternative when that happens. You might already run into this if you tried to upload to FlightReview.
In any case, knowing splitting the file is possible/reasonable would be good, so it can be discarded as an option if not.

1 Like
7

There’s currently no tool (that I’m aware of) to split logs. It would be possible to add one in pyulog though.

In general I’d rather aim for reducing the logged data in the first place, as large files are generally more difficult & expensive to handle. ulog_info tells you which data points need the most space.

1 Like
8

Hi @bkueng thanks for the input, I’ll look into the ulog_info and check which one data point needs the most space and try to optimize.

Also, our use case of PX4 is a bit different from normal fixed-wing or drone operation, we are using it inside a High Altitude Balloon, which can be in the air for multiple hours. I think we are using the normal profile for the recording and ought to change it to something more lean.

Thanks again for the input.

9

Thanks @bkueng - makes sense. Though I am pretty sure someone will come up with a case where an impossibly big log is needed even with good design of data points.

@LudovicVanasse If you learn something from this that isn’t easily inferable from the docs and of broad interest would we great if we could have a paragraph or two on the right approach. Having not done it, I can’t be sure whether this is needed or not, but it would be good the next time someone asks to be able to point to a section in the docs that helps answer your question.

10

@hamishwillee I think one interesting thing that I have learn is that the limit of the log file at the moment is about 2 GB and only in the main branch at the moment that it will be restarting the log. So in previous version of the code it will simply stop logging. Here’s the PR that mention and fix this: logger: restart on file write error by bkueng · Pull Request #21389 · PX4/PX4-Autopilot · GitHub

If you find that interesting, let me know and tell me where to put it in the doc, I was thinking maybe Logging section. And I’ll create a PR for that. :slight_smile:

11

Thanks @LudovicVanasse!

Yes, I think that we should add (from here)

File size limitations

The maximum file size depends on the file system and OS.
The size limit on NuttX is currently around 2GB.

This could go in the logging topic, perhaps just before the dropouts section - seem reasonable?

The information about restarting is also interesting. Perhaps we should add a line in the usage section before the last line “For a list of all supported logger commands and parameters, use:”

Something like

If logging stops due to a write error, or reaching the maximum file size [insert link down to the new section], PX4 will automatically restart logging (to a new file).

12

@hamishwillee sounds good. I just created the PR for it: Add information for the logger module error behavior and file size limitations by lvanasse · Pull Request #2616 · PX4/PX4-user_guide · GitHub

I wasn’t too original and pretty much took what you said :sweat_smile:, I thought it was good.

Please let me know if it is OK or there’s something else I need to do/change.

13

Thanks very much @LudovicVanasse ! I’ll check it out on Wednesday when I’m next in. Much appreciated.

14

No worries @hamishwillee, you did much of the work :sweat_smile:. Please let me know if you need some more help with the documentation :slight_smile:.

15

Thanks @LudovicVanasse - PR was good.

Please let me know if you need some more help with the documentation

Always. Add anything that should be answered by docs here in discord :-).

16

I am not sure what the limitation is, but for files too large for the online FlightReview I open them in an offline version (local Python) if I am not interested in using PlotJuggler or if it is for some small check. It is both faster and handles larger files than the online version, but I haven’t tried it with 2GB files (350M is the largest I have needed).