Data Comets: A new flight log analysis tool

@apaulsen thank you for that, I just pushed a quick fix which should resolve these issues, the overview tab was throwing errors when it did not find the data it was configured to plot, it will now just ignore that data and move on.

This should also help older logs @rollys @mlsky

Please pull the new changes and let me know if it works.

@dsaffo Looks like it’s working for me now, thanks!

1 Like

@dsaffo I am not sure if this was an issue for me, but I am still left with some issues. I get an error like this “TypeError: undefined is not an object (evaluating ‘data[‘vehicle_local_position’][0]’)”

PS: I might add that I am generating logs from PX4 v1.6.5.

@mlsky I think I found the cause of this error, from that message it seems like there is not a reference lat/lon in the log file or at least in the same place as others. I use this data to center the map on the flight, but I added a check and a fallback to just use the first GPS coordinate instead.

I’ve pushed these changes but I do not have a way of testing this myself as I have not found another log with this characteristic. Please let me know if that got it :slightly_smiling_face:

@dsaffo sounds great, I will test first thing in the morning. Btw, I am 99% certain that it was map.js around line 41 or 60.

@dsaffo Good morning. I tested the code and it didn’t fly. However, I fixed it. The problem was, that you have used “let” to define refLon/refLat. Such a variable is living within blocks, so between { let }. I declared refLon and refLat above the try logic, using “var refLon;” and “var refLat;” and removed let in lines below, which is using the variable.

If you would like, I can make a pull request.

@dsaffo Another “issue” might be the performance when handling large log-files. We have some log-files that contains >70MB of data (> 1 hour of flight). Uploading is inconsistent and rendering performance requires some patience :slight_smile:

@mlsky Ah! Thank you for that, that is some of the new changes I didn’t get to test. Please go ahead with a pull request and I will merge it.

As for the performance with those larger logs I am aware of the issue.
The slow down comes from a few places load times being mostly the time it takes to parse the log and render everything. Really big logs 100mb+ are around a gb parsed and decompressed and it is very likely the network will just give up on them…
A javascript Ulog Parser would solve a lot of this but not sure when I will get the time myself to implement that.

Since I’m using svg graphics those larger logs just produce so many elements it runs up to the limit of svg performance. Switching to canvas graphics has been on my wish list for the future, but there are some things I can do now to make it more usable.

I’ve pushed a small tweak to only update the linecharts when brushing is done for logs >50mb. This way the path still updates while brushing to show motion but the line charts don’t slow everything down. It made it a good bit smoother for the large logs I tested with. Switching the data encoded on the path is still a little slow though, there is a lot that happens to align the data with the right path segments.

I intended to improve performance overtime, I’ve created a project board on the GitHub page to help track progress.

@mlsky I’ve been working on improvements to the net code to help with stability I will start to roll out those improvements soon. In the mean time I would recommend tweaking the settings on gunicorn file to include a higher timeout (-t or --timeout int) the default is 30 (seconds) which is too short for larger logs I would set it to 120 (seconds) just to be safe.

@dsaffo Sounds really great. I will try tweak the settings while I eagerly await your updates.

Hello, I am using this tool very usful. Thank you!
But I need to check two logs. For this, I want to run two server simultaneously.
I tried to change the port number but it didn’t work.
Can you tell me how to change the port number?

→ Sorry, I did another port number. And it worked!!

1 Like