fw2kml - A Tool to Convert Featherweight GPS Data to .kml files for Google Earth!

[Arpak]

Hi all,

It's been a bit since I posted on TRF, been catching up on life (I graduated and got a big boy engineering job!). I had a discussion last night with my buddy Adam and he mentioned to me that he wished he could view his featherweight GPS data in Google Earth, apparently that's not a released feature yet. Turns out its not too difficult to convert it, but it would be incredibly tedious by hand.

So I made a python script! I've been calling it fw2kml (FeatherWeight To(2) KML), and it seems to work fairly well. It turns this....


Into this!


Right now its feature set is brief but as follows;

- Drag a featherweight .csv data file onto the script to generate a [filename]_fw2kml.kml file of that data
- Automatically filters out "bad" data (altitude is below ground)
- Nifty icon

I'd like to maintain this for a bit until it gets stable enough to work in practically every scenario with the featherweight, but I need your help for this. If you own a featherweight GPS, please try this tool out and let me know if you hit any snags.

How to Use

1. Download the "fw2kmldrag.exe" file from the link below
2. Drag a Featherweight .csv data file onto the .exe, it will automatically generate the output .kml and place it in the same location as the .exe
3. Open Google Earth Pro (its free)
4. Go to File>Open
5. Select the generated .kml file
6. Gawk and Awe
7. Upload a screenshot of your flight here!

Download

I've made the source code and built .exe file available on github, for you code-savvy folks out there its written in python and generated into a .exe using the py2exe library. To download the tool, go to the following link and download "fw2kmldrag.exe".

https://github.com/pirate21213/fw2kml/releases/tag/Latest

Note: You may/will probably get a warning that the file is malicious. This is because it doesn't have proper certificates and whatnot and for all intents and purposes looks to windows like a virus. I can assure you, its not. The source code is up. Its basically a highschool CompSci project. Don't believe me? I guess don't use it... This is the bulk of the logic.

    outfile_name = str(droppedFile).replace('.csv', '_fw2kml.kml')
    coords = []
    for row in rows:
        if int(row[5]) <= 0:
            print("Bad data, skipping", row[5])
        else:
            coords.append("{},{},{}".format(row[4], row[3], (int(row[5]) * 0.3048)))

    coordstring = ' '.join(coords)

If you run into an issue using the file, let me know what happened! I'd love to catch all the strange edge cases and make this fairly straight forward.

Happy flying!
-Patrick

 

[Adrian A]

Thanks so much for doing this! When I try it with my files, I see a very faint (nearly invisible) red line of the ground track but it's not above ground. Here are three files that are all doing this for me:

 

[Arpak]

Thanks so much for doing this! When I try it with my files, I see a very faint (nearly invisible) red line of the ground track but it's not above ground. Here are three files that are all doing this for me:

Aha! This is exactly why I wanted others to try it out

I'll take a look at it later today and see if I can figure out why its acting strange.

 

[Arpak]

Actually, I couldn't wait. Turns out the columns aren't standardized

Here you go @Adrian A, thanks for the test case and thanks for creating the device!




V1.1 Is now available from the same download location, thanks for the help!
https://github.com/pirate21213/fw2kml/releases/tag/Latest

 

[Adrian A]

When the data is downloaded, sometimes re-transmits are necessary that put the data out of order. Could you sort the data on time before converting to KML?

 

[Arpak]

When the data is downloaded, sometimes re-transmits are necessary that put the data out of order. Could you sort the data on time before converting to KML?

Didn't know that, I definitely could (but that one will have to wait until I get off work ). Any other quirks I should know about? Also do you have an example file with a misordered time?

 

[Adrian A]

Actually, I couldn't wait. Turns out the columns aren't standardized

Here you go @Adrian A, thanks for the test case and thanks for creating the device!

View attachment 555219


V1.1 Is now available from the same download location, thanks for the help!
https://github.com/pirate21213/fw2kml/releases/tag/Latest

The new version is working for me now. Thanks!

Didn't know that, I definitely could (but that one will have to wait until I get off work ). Any other quirks I should know about? Also do you have an example file with a misordered time?

I'll look for one.

 

[Adrian A]

Here's one with some minor back-steps:

 

[thzero]

Mmm... Have to try with mine... I've used it multiple times with flights and video, even for my l2

 

[Buckeye]

Nice. Thanks for the utility.

Yes, the time sort is definitely needed.

Slightly off topic, but do you have a good way to parse and format the UTC time stamp into something useable? I like to plot the GPS data vs. flight time (just like an altimeter) where the time is simply formatted as xxxx.x seconds. I have to go through a bunch of gyrations in Excel to get it looking right, and I never remember the steps.

 

[Adrian A]

Nice. Thanks for the utility.

Yes, the time sort is definitely needed.

Slightly off topic, but do you have a good way to parse and format the UTC time stamp into something useable? I like to plot the GPS data vs. flight time (just like an altimeter) where the time is simply formatted as xxxx.x seconds. I have to go through a bunch of gyrations in Excel to get it looking right, and I never remember the steps.

It's easiest to use the Unix Time for this.
I insert a new blank column, and then use an equation that subtracts off the first entry from the later entries. So if the unix time time is in column B, I'd make a column C and put in =B1-B$1 into the C1 cell, and then fill the rest of column C with that equation.

 

[Arpak]

Nice. Thanks for the utility.

Yes, the time sort is definitely needed.

Slightly off topic, but do you have a good way to parse and format the UTC time stamp into something useable? I like to plot the GPS data vs. flight time (just like an altimeter) where the time is simply formatted as xxxx.x seconds. I have to go through a bunch of gyrations in Excel to get it looking right, and I never remember the steps.

Do you have an example file with bad time stamps? The one Adrian sent is only off by a second or two every once in a while, I'm curious if worse cases are out there.

I'm stewing on the best way to do time sorting, as it is the polling rate is faster than Unix time, so there's multiple entries per Unix time stamp making sorting a bit more nuanced.

 

[Adrian A]

Do you have an example file with bad time stamps? The one Adrian sent is only off by a second or two every once in a while, I'm curious if worse cases are out there.

I'm stewing on the best way to do time sorting, as it is the polling rate is faster than Unix time, so there's multiple entries per Unix time stamp making sorting a bit more nuanced.

I thought the UNIX time already has sub-seconds, but I'll double check.

Edit: I see the UNIX time is already whole seconds, so parsing the UTCTime would be necessary for a true sort. IIRC, Python can be a pain with how it handles formatted times.

 

[Arpak]

I thought the UNIX time already has sub-seconds, but I'll double check.

Edit: I see the UNIX time is already whole seconds, so parsing the UTCTime would be necessary for a true sort. IIRC, Python can be a pain with how it handles formatted times.

Scratch that, you are totally right. Sub seconds are there but excel just likes to hide them.

 

[Arpak]

Update V1.2

• Added time jitter correction if UNIXTIME field is present

Download
https://github.com/pirate21213/fw2kml/releases/tag/Latest

 

[Arpak]

Update V1.3

• Added automatic time conversion from (I'm assuming) older DATE + TIME formats to UNIXTIME
• Added time jitter correction for DATE + TIME formats
• Added new flight detection, if a >1 minute off period is detected it creates a new flight track with a new color. Flights can be toggled on/off on the side panel

Download
https://github.com/pirate21213/fw2kml/releases/tag/Latest


-----------------------------------------------------------------------------------------------

Flight Detection
This was a recommendation passed to me by Adam from "SLAP" in the Hot Nozzle Society, it was a fun one to implement! If there is a >1 minute period between updates the program will now consider that the start of a new flight. Each flight is assigned a color (right now its just red, green, and blue cycling) as well as a unique name. This lets you view and sort flights separately if you did more than one on a single day.


Figure 1: Different flights are now color coded


Figure 2: Different flights listed in the sidebar


Happy flying! As always, let me know if you run into an issue or have an idea for a feature.
-Patrick

 

[manixFan]

Update V1.3

• Added automatic time conversion from (I'm assuming) older DATE + TIME formats to UNIXTIME
• Added time jitter correction for DATE + TIME formats
• Added new flight detection, if a >1 minute off period is detected it creates a new flight track with a new color. Flights can be toggled on/off on the side panel

Download
https://github.com/pirate21213/fw2kml/releases/tag/Latest


-----------------------------------------------------------------------------------------------

Flight Detection
This was a recommendation passed to me by Adam from "SLAP" in the Hot Nozzle Society, it was a fun one to implement! If there is a >1 minute period between updates the program will now consider that the start of a new flight. Each flight is assigned a color (right now its just red, green, and blue cycling) as well as a unique name. This lets you view and sort flights separately if you did more than one on a single day.

View attachment 555546
Figure 1: Different flights are now color coded

View attachment 555545
Figure 2: Different flights listed in the sidebar


Happy flying! As always, let me know if you run into an issue or have an idea for a feature.
-Patrick

Hah! I was just asking for this feature (multiple flight detection) when you posted this!! Thank you for this great work, super handy utility. I was doing it manually and it was a pain. I normally use a Mac, so eventually I'll have to see if I can run the script without the .exe, but in the meantime, well worth the trouble of firing up my Windows laptop.


Tony

EDIT: Added the screenshot below - the reddish colored flights are from BALLs XXX (2022), the blueish are from the year before. I had three flights in one file, to separate out those by hand and massage them Excel was a huge pain before. What a timesaver!

 

[Buckeye]

It's easiest to use the Unix Time for this.
I insert a new blank column, and then use an equation that subtracts off the first entry from the later entries. So if the unix time time is in column B, I'd make a column C and put in =B1-B$1 into the C1 cell, and then fill the rest of column C with that equation.

Yes, the decimal seconds were hidden for me, too. Now I see how Unix Time can be useful. Thanks.

 

[Arpak]

Update V1.3

Hah! I was just asking for this feature (multiple flight detection) when you posted this!! Thank you for this great work, super handy utility. I was doing it manually and it was a pain. I normally use a Mac, so eventually I'll have to see if I can run the script without the .exe, but in the meantime, well worth the trouble of firing up my Windows laptop.


Tony

I've been thinking of the Mac issue for a bit... There's technically nothing stopping you from running the python script in a python environment on Mac, but I've also been looking into a similar utility that just creates a mac app. I have absolutely no idea if the drag and drop thing would work on Mac.

 

[manixFan]

I've been thinking of the Mac issue for a bit... There's technically nothing stopping you from running the python script in a python environment on Mac, but I've also been looking into a similar utility that just creates a mac app. I have absolutely no idea if the drag and drop thing would work on Mac.

In the past I've been able to drag and drop files into the terminal window which automatically adds the file's pathname, so I should be able to do that here, although in a terminal window after typing your script name. So not a huge deal for now. But that assumes the script will run right from the command line without a lot of parameters other than the filepath. I can create a simple AppleScript that works with drag and drop very easily, so I don't think you'll have any trouble.

Thanks again for your efforts!


Tony

 

[Arpak]

In the past I've been able to drag and drop files into the terminal window which automatically adds the file's pathname, so I should be able to do that here, although in a terminal window after typing your script name. So not a huge deal for now. But that assumes the script will run right from the command line without a lot of parameters other than the filepath. I can create a simple AppleScript that works with drag and drop very easily, so I don't think you'll have any trouble.

Thanks again for your efforts!


Tony

Yea its just using the dropped files path as the only argument, the next issue though is the py2app library needs a Mac to build it for a Mac. My wife has a one but it suffered from flexgate and the screen does backlight, but I've got a spare monitor....

I'm also going to look into hosting it on Google Collab or something similar, it's just a python script after all.

 

[BDB]

This is excellent. Thanks!

 

[Arpak]

Update V1.3

Hah! I was just asking for this feature (multiple flight detection) when you posted this!! Thank you for this great work, super handy utility. I was doing it manually and it was a pain. I normally use a Mac, so eventually I'll have to see if I can run the script without the .exe, but in the meantime, well worth the trouble of firing up my Windows laptop.


Tony

EDIT: Added the screenshot below - the reddish colored flights are from BALLs XXX (2022), the blueish are from the year before. I had three flights in one file, to separate out those by hand and massage them Excel was a huge pain before. What a timesaver!
View attachment 555568

Incredible flights!

 

[Arpak]

Update V1.4 - Notebook
- Added google colaboratory notebook version (fw2kml.ipynb)


Google Colaboratory Notebook
Don't have a windows PC handy or you don't feel comfortable downloading a random .exe? You can now run the script in a browser on a provided google colaboratory notebook, it's fairly straight forward and walks you through it but here is a visual guide as well.

• Click on fw2kml.ipynb, click the "Open in Colab" button and run the script on Google Colaboratory! or go to this link and run the notebook https://colab.research.google.com/drive/1I2dGwhLMUKx1wPfJg8Pk0KMp6fMBAA5s?usp=sharing

• Open the file pane on the left side

• Click the upload files button and select as many featherweight .csv data files as you'd like, it will process them all. (Ignore the sample_data folder, its boilerplate garbage from google). Press ok on the following prompt.

• Click on the "Run Cell" icon, you can click "show code" if you want to see whats underneath the hood.

• You'll get a text response with some info about the run, namely which files it found, which it created, and a total number when it finishes. The script automatically deletes the .csv from its folder and in its place will be a .kml.
• If you let it sit long enough after a run the file pane will automatically update, but to get your file immediately click on the "refresh" button and download the file

• Download the file and open it with Google Earth Pro

• Enjoy your data!

As always, feel free to give me feedback,
-Patrick

 

[thzero]

I have something similiar...





Relevant kml files attached

 

[Arpak]

I have something similiar...

View attachment 555869

View attachment 555870

Relevant kml files attached

Nice flights! I have no doubt folks have figured out their own solutions to the problem (after all, that's exactly how I stumbled onto it). I just wanted to share an openly available, easily usable option for those who are code inclined and don't want to spend a day manually sifting data.

I like the launch/touchdown pins, I considered trying to grab the flags field from the data and give the readout a similar context..

 

[thzero]

Yeah, mine is built in a website that I hope to release soon. But I too stumbled on it somewhere and was like thats a good idea, and then struggled through figuring it out. I had an Electron app that would do it once I started using it for each of my flight videos, but Electron apps are just painful to update and there isn't anything that really required running on desktop, so switched gears.

Yeah, the pins were a best guess based on data.

I don't have the 1 minute break; probably should add that.

 

[manixFan]

When it rains, it pours! Very cool that folks are working on utilities to greatly ease the amount of work it takes otherwise to produce the KML files. It's not hard doing it by hand, but it's pretty tedious. Having multiple solutions is even better!

Just a note about the launch/landing flags, really only one is needed, and that would reduce clutter. For multiple launches, the landing flag might be best.

What would be super fantastic is a flag that is tagged to the highest reported altitude for each flight, or the time the flight was launched, to make it easy to identify flights. And, while I'm dreaming, the ability to set the default style of the path - color, fill, opacity, etc., perhaps through a simple config file for the Python version.

Super cool all around, thanks for the hard work.


Tony

 

[Arpak]

When it rains, it pours! Very cool that folks are working on utilities to greatly ease the amount of work it takes otherwise to produce the KML files. It's not hard doing it by hand, but it's pretty tedious. Having multiple solutions is even better!

Just a note about the launch/landing flags, really only one is needed, and that would reduce clutter. For multiple launches, the landing flag might be best.

What would be super fantastic is a flag that is tagged to the highest reported altitude for each flight, or the time the flight was launched, to make it easy to identify flights. And, while I'm dreaming, the ability to set the default style of the path - color, fill, opacity, etc., perhaps through a simple config file for the Python version.

Super cool all around, thanks for the hard work.


Tony

I think I'm going to just maintain the colab notebook from now on, and with that a config section would be super easy to implement.

Currently it just separates flights by giving it an index (Flight 1, Flight 2, ...) but that could be changed to give the launch time?

These are good ideas..

 

[thzero]

I have another tool that reads through altimeter readings (just Eggtimer right now) and was going to use that to pull in some of the drogue and main events, etc. and then try and label them on here too.