fw2kml - A Tool to Convert Featherweight GPS Data to .kml files for Google Earth!

Arpak

Well-Known Member
Joined
Nov 23, 2020
Messages
219
Reaction score
210
Location
Kirkland, WA
fancyicon.png

Hi all,

It's been a bit since I posted on TRF, been catching up on life (I graduated and got a big boy engineering job!). I had a discussion last night with my buddy Adam and he mentioned to me that he wished he could view his featherweight GPS data in Google Earth, apparently that's not a released feature yet. Turns out its not too difficult to convert it, but it would be incredibly tedious by hand.

So I made a python script! I've been calling it fw2kml (FeatherWeight To(2) KML), and it seems to work fairly well. It turns this....
1673042815928.png

Into this!
1673042762706.png

Right now its feature set is brief but as follows;

- Drag a featherweight .csv data file onto the script to generate a [filename]_fw2kml.kml file of that data
- Automatically filters out "bad" data (altitude is below ground)
- Nifty icon

I'd like to maintain this for a bit until it gets stable enough to work in practically every scenario with the featherweight, but I need your help for this. If you own a featherweight GPS, please try this tool out and let me know if you hit any snags.

How to Use
  1. Download the "fw2kmldrag.exe" file from the link below
  2. Drag a Featherweight .csv data file onto the .exe, it will automatically generate the output .kml and place it in the same location as the .exe
  3. Open Google Earth Pro (its free)
  4. Go to File>Open
  5. Select the generated .kml file
  6. Gawk and Awe
  7. Upload a screenshot of your flight here!

Download

I've made the source code and built .exe file available on github, for you code-savvy folks out there its written in python and generated into a .exe using the py2exe library. To download the tool, go to the following link and download "fw2kmldrag.exe".

https://github.com/pirate21213/fw2kml/releases/tag/Latest

Note: You may/will probably get a warning that the file is malicious. This is because it doesn't have proper certificates and whatnot and for all intents and purposes looks to windows like a virus. I can assure you, its not. The source code is up. Its basically a highschool CompSci project. Don't believe me? I guess don't use it... This is the bulk of the logic.

Python:
    outfile_name = str(droppedFile).replace('.csv', '_fw2kml.kml')
    coords = []
    for row in rows:
        if int(row[5]) <= 0:
            print("Bad data, skipping", row[5])
        else:
            coords.append("{},{},{}".format(row[4], row[3], (int(row[5]) * 0.3048)))

    coordstring = ' '.join(coords)


If you run into an issue using the file, let me know what happened! I'd love to catch all the strange edge cases and make this fairly straight forward.

Happy flying!
-Patrick
 

Adrian A

Sponsor
TRF Sponsor
TRF Supporter
Joined
Jan 21, 2009
Messages
2,647
Reaction score
1,069
Location
Lakewood, CO
Thanks so much for doing this! When I try it with my files, I see a very faint (nearly invisible) red line of the ground track but it's not above ground. Here are three files that are all doing this for me:
 

Attachments

  • 2022-12-03_(FthrWt00489)_19-21-32.csv
    26.6 KB · Views: 1
  • 2022-11-20_(Shot3)_15-21-30.csv
    41.4 KB · Views: 0
  • 2022-11-20_(Shot3)_18-58-28.csv
    64.9 KB · Views: 0

Arpak

Well-Known Member
Joined
Nov 23, 2020
Messages
219
Reaction score
210
Location
Kirkland, WA
Thanks so much for doing this! When I try it with my files, I see a very faint (nearly invisible) red line of the ground track but it's not above ground. Here are three files that are all doing this for me:
Aha! This is exactly why I wanted others to try it out :)

I'll take a look at it later today and see if I can figure out why its acting strange.
 

Arpak

Well-Known Member
Joined
Nov 23, 2020
Messages
219
Reaction score
210
Location
Kirkland, WA
Actually, I couldn't wait. Turns out the columns aren't standardized :p

Here you go @Adrian A, thanks for the test case and thanks for creating the device!

1673044684290.png


V1.1 Is now available from the same download location, thanks for the help!
 

Adrian A

Sponsor
TRF Sponsor
TRF Supporter
Joined
Jan 21, 2009
Messages
2,647
Reaction score
1,069
Location
Lakewood, CO
When the data is downloaded, sometimes re-transmits are necessary that put the data out of order. Could you sort the data on time before converting to KML?
 

Arpak

Well-Known Member
Joined
Nov 23, 2020
Messages
219
Reaction score
210
Location
Kirkland, WA
When the data is downloaded, sometimes re-transmits are necessary that put the data out of order. Could you sort the data on time before converting to KML?
Didn't know that, I definitely could (but that one will have to wait until I get off work ;) ). Any other quirks I should know about? Also do you have an example file with a misordered time?
 

Adrian A

Sponsor
TRF Sponsor
TRF Supporter
Joined
Jan 21, 2009
Messages
2,647
Reaction score
1,069
Location
Lakewood, CO
Actually, I couldn't wait. Turns out the columns aren't standardized :p

Here you go @Adrian A, thanks for the test case and thanks for creating the device!

View attachment 555219


V1.1 Is now available from the same download location, thanks for the help!
The new version is working for me now. Thanks!
Didn't know that, I definitely could (but that one will have to wait until I get off work ;) ). Any other quirks I should know about? Also do you have an example file with a misordered time?
I'll look for one.
 

Buckeye

Well-Known Member
TRF Supporter
Joined
Sep 5, 2009
Messages
3,032
Reaction score
801
Nice. Thanks for the utility.

Yes, the time sort is definitely needed.

Slightly off topic, but do you have a good way to parse and format the UTC time stamp into something useable? I like to plot the GPS data vs. flight time (just like an altimeter) where the time is simply formatted as xxxx.x seconds. I have to go through a bunch of gyrations in Excel to get it looking right, and I never remember the steps.
 

Adrian A

Sponsor
TRF Sponsor
TRF Supporter
Joined
Jan 21, 2009
Messages
2,647
Reaction score
1,069
Location
Lakewood, CO
Nice. Thanks for the utility.

Yes, the time sort is definitely needed.

Slightly off topic, but do you have a good way to parse and format the UTC time stamp into something useable? I like to plot the GPS data vs. flight time (just like an altimeter) where the time is simply formatted as xxxx.x seconds. I have to go through a bunch of gyrations in Excel to get it looking right, and I never remember the steps.
It's easiest to use the Unix Time for this.
I insert a new blank column, and then use an equation that subtracts off the first entry from the later entries. So if the unix time time is in column B, I'd make a column C and put in =B1-B$1 into the C1 cell, and then fill the rest of column C with that equation.
 
Last edited:

Arpak

Well-Known Member
Joined
Nov 23, 2020
Messages
219
Reaction score
210
Location
Kirkland, WA
Nice. Thanks for the utility.

Yes, the time sort is definitely needed.

Slightly off topic, but do you have a good way to parse and format the UTC time stamp into something useable? I like to plot the GPS data vs. flight time (just like an altimeter) where the time is simply formatted as xxxx.x seconds. I have to go through a bunch of gyrations in Excel to get it looking right, and I never remember the steps.
Do you have an example file with bad time stamps? The one Adrian sent is only off by a second or two every once in a while, I'm curious if worse cases are out there.

I'm stewing on the best way to do time sorting, as it is the polling rate is faster than Unix time, so there's multiple entries per Unix time stamp making sorting a bit more nuanced.
 

Adrian A

Sponsor
TRF Sponsor
TRF Supporter
Joined
Jan 21, 2009
Messages
2,647
Reaction score
1,069
Location
Lakewood, CO
Do you have an example file with bad time stamps? The one Adrian sent is only off by a second or two every once in a while, I'm curious if worse cases are out there.

I'm stewing on the best way to do time sorting, as it is the polling rate is faster than Unix time, so there's multiple entries per Unix time stamp making sorting a bit more nuanced.
I thought the UNIX time already has sub-seconds, but I'll double check.

Edit: I see the UNIX time is already whole seconds, so parsing the UTCTime would be necessary for a true sort. IIRC, Python can be a pain with how it handles formatted times.
 

Arpak

Well-Known Member
Joined
Nov 23, 2020
Messages
219
Reaction score
210
Location
Kirkland, WA
I thought the UNIX time already has sub-seconds, but I'll double check.

Edit: I see the UNIX time is already whole seconds, so parsing the UTCTime would be necessary for a true sort. IIRC, Python can be a pain with how it handles formatted times.
Scratch that, you are totally right. Sub seconds are there but excel just likes to hide them.
 

Arpak

Well-Known Member
Joined
Nov 23, 2020
Messages
219
Reaction score
210
Location
Kirkland, WA
Update V1.3
  • Added automatic time conversion from (I'm assuming) older DATE + TIME formats to UNIXTIME
  • Added time jitter correction for DATE + TIME formats
  • Added new flight detection, if a >1 minute off period is detected it creates a new flight track with a new color. Flights can be toggled on/off on the side panel
Download
https://github.com/pirate21213/fw2kml/releases/tag/Latest


-----------------------------------------------------------------------------------------------

Flight Detection
This was a recommendation passed to me by Adam from "SLAP" in the Hot Nozzle Society, it was a fun one to implement! If there is a >1 minute period between updates the program will now consider that the start of a new flight. Each flight is assigned a color (right now its just red, green, and blue cycling) as well as a unique name. This lets you view and sort flights separately if you did more than one on a single day.

1673124831177.png
Figure 1: Different flights are now color coded

1673124801299.png
Figure 2: Different flights listed in the sidebar


Happy flying! As always, let me know if you run into an issue or have an idea for a feature.
-Patrick
 

manixFan

Not a rocket scientist
Joined
Feb 15, 2009
Messages
2,612
Reaction score
1,846
Location
TX
Update V1.3
  • Added automatic time conversion from (I'm assuming) older DATE + TIME formats to UNIXTIME
  • Added time jitter correction for DATE + TIME formats
  • Added new flight detection, if a >1 minute off period is detected it creates a new flight track with a new color. Flights can be toggled on/off on the side panel
Download
https://github.com/pirate21213/fw2kml/releases/tag/Latest


-----------------------------------------------------------------------------------------------

Flight Detection
This was a recommendation passed to me by Adam from "SLAP" in the Hot Nozzle Society, it was a fun one to implement! If there is a >1 minute period between updates the program will now consider that the start of a new flight. Each flight is assigned a color (right now its just red, green, and blue cycling) as well as a unique name. This lets you view and sort flights separately if you did more than one on a single day.

View attachment 555546
Figure 1: Different flights are now color coded

View attachment 555545
Figure 2: Different flights listed in the sidebar


Happy flying! As always, let me know if you run into an issue or have an idea for a feature.
-Patrick
Hah! I was just asking for this feature (multiple flight detection) when you posted this!! Thank you for this great work, super handy utility. I was doing it manually and it was a pain. I normally use a Mac, so eventually I'll have to see if I can run the script without the .exe, but in the meantime, well worth the trouble of firing up my Windows laptop.


Tony

EDIT: Added the screenshot below - the reddish colored flights are from BALLs XXX (2022), the blueish are from the year before. I had three flights in one file, to separate out those by hand and massage them Excel was a huge pain before. What a timesaver!
BALLs-plots.jpg
 
Last edited:

Buckeye

Well-Known Member
TRF Supporter
Joined
Sep 5, 2009
Messages
3,032
Reaction score
801
It's easiest to use the Unix Time for this.
I insert a new blank column, and then use an equation that subtracts off the first entry from the later entries. So if the unix time time is in column B, I'd make a column C and put in =B1-B$1 into the C1 cell, and then fill the rest of column C with that equation.

Yes, the decimal seconds were hidden for me, too. Now I see how Unix Time can be useful. Thanks.
 

Arpak

Well-Known Member
Joined
Nov 23, 2020
Messages
219
Reaction score
210
Location
Kirkland, WA
Update V1.3

Hah! I was just asking for this feature (multiple flight detection) when you posted this!! Thank you for this great work, super handy utility. I was doing it manually and it was a pain. I normally use a Mac, so eventually I'll have to see if I can run the script without the .exe, but in the meantime, well worth the trouble of firing up my Windows laptop.


Tony
I've been thinking of the Mac issue for a bit... There's technically nothing stopping you from running the python script in a python environment on Mac, but I've also been looking into a similar utility that just creates a mac app. I have absolutely no idea if the drag and drop thing would work on Mac.
 

manixFan

Not a rocket scientist
Joined
Feb 15, 2009
Messages
2,612
Reaction score
1,846
Location
TX
I've been thinking of the Mac issue for a bit... There's technically nothing stopping you from running the python script in a python environment on Mac, but I've also been looking into a similar utility that just creates a mac app. I have absolutely no idea if the drag and drop thing would work on Mac.
In the past I've been able to drag and drop files into the terminal window which automatically adds the file's pathname, so I should be able to do that here, although in a terminal window after typing your script name. So not a huge deal for now. But that assumes the script will run right from the command line without a lot of parameters other than the filepath. I can create a simple AppleScript that works with drag and drop very easily, so I don't think you'll have any trouble.

Thanks again for your efforts!


Tony
 

Arpak

Well-Known Member
Joined
Nov 23, 2020
Messages
219
Reaction score
210
Location
Kirkland, WA
In the past I've been able to drag and drop files into the terminal window which automatically adds the file's pathname, so I should be able to do that here, although in a terminal window after typing your script name. So not a huge deal for now. But that assumes the script will run right from the command line without a lot of parameters other than the filepath. I can create a simple AppleScript that works with drag and drop very easily, so I don't think you'll have any trouble.

Thanks again for your efforts!


Tony
Yea its just using the dropped files path as the only argument, the next issue though is the py2app library needs a Mac to build it for a Mac. My wife has a one but it suffered from flexgate and the screen does backlight, but I've got a spare monitor....

I'm also going to look into hosting it on Google Collab or something similar, it's just a python script after all.
 

Arpak

Well-Known Member
Joined
Nov 23, 2020
Messages
219
Reaction score
210
Location
Kirkland, WA
Update V1.3

Hah! I was just asking for this feature (multiple flight detection) when you posted this!! Thank you for this great work, super handy utility. I was doing it manually and it was a pain. I normally use a Mac, so eventually I'll have to see if I can run the script without the .exe, but in the meantime, well worth the trouble of firing up my Windows laptop.


Tony

EDIT: Added the screenshot below - the reddish colored flights are from BALLs XXX (2022), the blueish are from the year before. I had three flights in one file, to separate out those by hand and massage them Excel was a huge pain before. What a timesaver!
View attachment 555568
Incredible flights!
 

Arpak

Well-Known Member
Joined
Nov 23, 2020
Messages
219
Reaction score
210
Location
Kirkland, WA
Update V1.4 - Notebook
- Added google colaboratory notebook version (fw2kml.ipynb)


Google Colaboratory Notebook
Don't have a windows PC handy or you don't feel comfortable downloading a random .exe? You can now run the script in a browser on a provided google colaboratory notebook, it's fairly straight forward and walks you through it but here is a visual guide as well.
1673148213281.png 1673148238925.png

  • Open the file pane on the left side
1673148335781.png

  • Click the upload files button and select as many featherweight .csv data files as you'd like, it will process them all. (Ignore the sample_data folder, its boilerplate garbage from google). Press ok on the following prompt.
1673148368513.png

  • Click on the "Run Cell" icon, you can click "show code" if you want to see whats underneath the hood.
1673148471736.png

  • You'll get a text response with some info about the run, namely which files it found, which it created, and a total number when it finishes. The script automatically deletes the .csv from its folder and in its place will be a .kml.
  • If you let it sit long enough after a run the file pane will automatically update, but to get your file immediately click on the "refresh" button and download the file
1673148637825.png

  • Download the file and open it with Google Earth Pro
1673148689887.png

  • Enjoy your data!
1673148734911.png

As always, feel free to give me feedback,
-Patrick
 
Last edited:

thzero

Well-Known Member
TRF Supporter
Joined
Jun 4, 2018
Messages
768
Reaction score
435
I have something similiar...

1673281264473.png

1673281291727.png

Relevant kml files attached
 

Attachments

  • flight-path-9-1-2023.kml
    21.8 KB · Views: 0
  • flight-path-9-1-2023 (2).kml
    35.9 KB · Views: 0
  • flight-path-9-1-2023 (1).kml
    12.8 KB · Views: 0

Arpak

Well-Known Member
Joined
Nov 23, 2020
Messages
219
Reaction score
210
Location
Kirkland, WA
I have something similiar...

View attachment 555869

View attachment 555870

Relevant kml files attached
Nice flights! I have no doubt folks have figured out their own solutions to the problem (after all, that's exactly how I stumbled onto it). I just wanted to share an openly available, easily usable option for those who are code inclined and don't want to spend a day manually sifting data.

I like the launch/touchdown pins, I considered trying to grab the flags field from the data and give the readout a similar context..
 

thzero

Well-Known Member
TRF Supporter
Joined
Jun 4, 2018
Messages
768
Reaction score
435
Yeah, mine is built in a website that I hope to release soon. But I too stumbled on it somewhere and was like thats a good idea, and then struggled through figuring it out. I had an Electron app that would do it once I started using it for each of my flight videos, but Electron apps are just painful to update and there isn't anything that really required running on desktop, so switched gears.

Yeah, the pins were a best guess based on data.

I don't have the 1 minute break; probably should add that.
 

manixFan

Not a rocket scientist
Joined
Feb 15, 2009
Messages
2,612
Reaction score
1,846
Location
TX
When it rains, it pours! Very cool that folks are working on utilities to greatly ease the amount of work it takes otherwise to produce the KML files. It's not hard doing it by hand, but it's pretty tedious. Having multiple solutions is even better!

Just a note about the launch/landing flags, really only one is needed, and that would reduce clutter. For multiple launches, the landing flag might be best.

What would be super fantastic is a flag that is tagged to the highest reported altitude for each flight, or the time the flight was launched, to make it easy to identify flights. And, while I'm dreaming, the ability to set the default style of the path - color, fill, opacity, etc., perhaps through a simple config file for the Python version.

Super cool all around, thanks for the hard work.


Tony
 

Arpak

Well-Known Member
Joined
Nov 23, 2020
Messages
219
Reaction score
210
Location
Kirkland, WA
When it rains, it pours! Very cool that folks are working on utilities to greatly ease the amount of work it takes otherwise to produce the KML files. It's not hard doing it by hand, but it's pretty tedious. Having multiple solutions is even better!

Just a note about the launch/landing flags, really only one is needed, and that would reduce clutter. For multiple launches, the landing flag might be best.

What would be super fantastic is a flag that is tagged to the highest reported altitude for each flight, or the time the flight was launched, to make it easy to identify flights. And, while I'm dreaming, the ability to set the default style of the path - color, fill, opacity, etc., perhaps through a simple config file for the Python version.

Super cool all around, thanks for the hard work.


Tony
I think I'm going to just maintain the colab notebook from now on, and with that a config section would be super easy to implement.

Currently it just separates flights by giving it an index (Flight 1, Flight 2, ...) but that could be changed to give the launch time?

These are good ideas..
 

thzero

Well-Known Member
TRF Supporter
Joined
Jun 4, 2018
Messages
768
Reaction score
435
1673382736988.png

I have another tool that reads through altimeter readings (just Eggtimer right now) and was going to use that to pull in some of the drogue and main events, etc. and then try and label them on here too.
 
Top