blankblank blank

Archive for the 'KML' Category

Crowdsourcing And Coordinating Data Collection With Handheld GPS Units

Published by
Leszek Pawlowicz
 in GIS, GPS, Google Earth, Google Maps, KML, MapWindow, Web apps, data, shapefile and spreadsheet. 1 Comment

Got an email today from someone with a data collection/coordination/assembly problem. They have multiple people out in the field with Garmin GPS units recording data, and bringing it back to a central location for collation/combination when they’re done. Since Garmin GPS units have limited capabilities for data storage – coordinates, name, comment, date and time, elevation, and that’s pretty much it – additional data associated with a point has to be recorded by hand. Once back from the field, data has to be downloaded from individual units, associated with the additional attribute data from forms, all the data combined together into a single dataset, then converted into GIS-friendly format. And the process they had come up with wasn’t really working well for them.

I think this highlights some of the major limitations of classic stand-alone handheld GPS units. They’re really designed for us in personal data collection, not combined data collection; assembling data from multiple units can take a lot of work. Plus, their limited data collection capabilities require offloading data attribute acquisition to other formats (e.g. pencil and paper), adding the addition problem of associating that data with coordinates later on. There are lots of professional solutions for these problems, like Trimble or MobileMapper GPS units, Terrasync and ArcPad software, but these can be complicated and expensive. I challenge you to find anyone with enough patience to use Terrasync for a single day without swearing at least once; I know I can’t  ;-).

For a few years, I’ve had CyberTracker on my list of potential topics to post on. CyberTracker is a terrific data acquisition and collation tool for field data with tons of great features, like custom data acquisition form design for easy data entry, moving maps, easy data collation and conversion to GIS-friendly format. Plus, the software is free, and the hardware is (relatively) affordable. But unfortunately, I think time and technology are passing Cybertracker by:

  • While the central data software runs on Windows, the field data collection software runs on old-school Palm OS and Windows Mobile. Palm OS is dead; Windows Mobile development has stopped with version 6.5, and only maintenance updates are scheduled. So the useful lifetime of any data collection system built around CyberTracker is limited. If you need a fast/cheap/short-term solution, CyberTracker is free for non-profit use, and you could probably pick up the hardware very cheaply on eBay. Long-term, I don’t see it having much of a future in its current form.
  • The data collation model is also becoming obsolete. With CyberTracker, you have to bring all the data units to a single data download/collation computer. While there are still places in the world where this might make sense, an increasingly-connected world means that data download/collation to the cloud instead of an individual computer makes more sense.

I’m really excited about the possibilities for crowdsourced geographic data collection and collation using portable devices running Android OS (yeah, iOS too). There are already several interesting apps for doing this, and I hope to cover some of those soon on my AndroGeoid website. And I suspect that the small number of apps that can currently do this will be quickly joined by far more apps, and far more capable apps, in the very near future. But that doesn’t help with my emailer’s current problem. I suspect there are many different ways you could do this, but here’s the first approach that came to me using all-free software and services.

1. Make sure that everyone on the project has a Google account (i.e. Gmail); completely free.

2. Have the project leader created a single main data spreadsheet on Google Docs, with all the desired data attributes (e.g. point name, coordinates, comments, additional data fields from the paper forms, etc.) and share a link to that spreadsheet with other project members so that they can edit it as well.

3. Project members can download data from their Garmin units using DNRGarmin, and then export the data from DNRGarmin in CSV format.

4. Load the data into the spreadsheet program of your choice, and add/edit data from data forms to make it conform to the data structure of the main Google Docs spreadsheet.

5. Copy the data cells in the spreadsheet program, and paste them into the main Google Docs spreadsheet. Note: Use Ctrl-C and Ctrl-V to copy and paste cells into Google Docs instead of using the Google Docs Edit menu to perform those operations, as the latter doesn’t seem to work for pasting data from different applications into Google Docs. You no longer have a single computer as a choke point for data entry; multiple people can add data to a spreadsheet at the same time, and Google Docs will coordinate data entry so that nothing is lost. And if you save the spreadsheet data from steps 3 and 4 as separate files, you’ll have backup copies of the original data as well.

For those who aren’t comfortable with working with spreadsheets, Google Docs lets you set up a “Form” to let anyone add data to a Google Docs spreadsheet directly; however, this increases the chances of coordinate data entry error.

6. Now that you have all the data centralized in Google Docs, you have lots of flexibility in how you can handle it:

  • Export the data in CSV format, and you can then import it into any GIS program that supports CSV data. If your GIS program doesn’t, use MapWindow to convert your CSV file into shapefile format first.
  • Unlike shapefile attribute tables, where adding/removing/re-ordering attribute data columns can be a pain, you can easily perform those operations in Google Docs and then re-export the data in CSV format.
  • Convert the data directly into a continuously-updated KML network link for display in Google Earth or Maps using Google’s Spreadsheet Mapper tool.
  • Use any of Google Docs built-in tools to analyze/plot/sort/visualize data, including their way-cool Fusion Tables.
  • And I’m sure there are more options I haven’t thought of.

Know an alternate approach? Have any additional ideas?  I welcome your links and suggestions in the Comments section below.




Removing Time Data Out Of A GPX Or KML File To Make It Work Right In Google Earth

Published by
Leszek Pawlowicz
 in GPS, GPX, Google Earth and KML. 1 Comment

In the process of writing yesterday’s post on Garmin Basecamp, I found an annoying flaw on how Google Earth handles GPX files. In a recent GPS talk I gave, I surprised some people when I told them that Google Earth can open some GPS-related formats like GPX, LOC and others directly; you just need to select the type of files you want to open with the drop-down in the lower-right corner:

open_gpx

But if the waypoints in GPX files come time-stamped, either with the time you created them in the field or in a program, Google Earth assumes that you want to use this time-related data, and brings up a time slider in the upper-left-hand corner:

timeslider

The first time you open the file, the time slider will run from start to finish, with waypoints popping up and disappearing as the time indicator hits their creation time. You only see all the waypoints when the time slider has run all the way through to the end. Very annoying.

But it gets worse. If you uncheck the GPS data box in the Places pane to hide the data, then check it again, you’ll see nothing at all except the time slider:

timeslider1

Hitting the play button on the time slider will make the waypoints appear and disappear quickly, and at the end the only waypoint visible will be the last one created; the time slider will look like this:

timeslider2

To see all the waypoints, you’ll need to move the “start-time-extent” slider all the way to the left:

timeslider3

Same behavior if you save the data permanently to “My Places”. This is pretty retarded behavior; I hope Google adds the option to turn off time-related data display when it’s not wanted. Until then, I banged together a simple Windows-only program called GPXTimeStripper that will remove all time-related data from a GPX file (KML files, too, although this may not work in every case).  Download the file at this link; it’s a zipped stand-alone executable. Run the program (won’t win any interface awards):

Click on the main button, choose the GPX or KML file you want to process, and the program will remove the time data from the file, and save it with “_TS” appended to the filename. You’ll get a pop-up box with the name and location of the new file, which should always be the same location as the input file.  If the pop-up becomes annoying (which it will), check the box in the lower-left-hand corner to turn it off. Help button takes you to this page; Exit does what you’d expect.

No real checks for overwriting older files, and may not work on every file correctly (report bugs). Use at your own risk. BTW,  Google, would it kill you to have KML files created in Google Earth terminate in CR-LF instead of just LF?  It would simplify the job of reading them in correctly.




Online Elevation Profiler

Published by
Leszek Pawlowicz
 in Google Maps, KML and Web apps. 3 Comments

Krystian Pietruszka emails about his new Geocontext Profiler site, based on the new Google Maps API version 3. Place two markers on a Google Maps view by clicking, and get the elevation profile between them:

geocontext

geoprofile

You can add as many additional markers as you want in any direction; to fine-tune a marker position, just click and hold on it, and drag it to the desired spot. You have multiple options for the lines between points:

  • Direct – Straight line between points
  • Driving – Elevation along roads from point to point (like this preset example for Death Valley):

deathvalley

deathvalleyprofile

  • Bicycling/walking

Click on the Geolocation IP link at right to jump to your local area; click on the double-arrow icon in the upper right corner to go to full-screen-width for the app (and back again). You can also link to a profile, or embed it on your site:

Note: The “Import KML” link in the upper left of the map will let you load and display KML data on the map, but doesn’t create an elevation profile between points or along a line.




Another KML Circle Creator

Published by
Leszek Pawlowicz
 in KML. 2 Comments

Both this site and KML-Circles appear to be down; try the GE Path program as an alternative.

Posted a long time ago about the KML-Circles site, which lets you generate polygons/quasi-circles centered around arbitratry points in KML format for Google Earth and Google Maps. Lots of options at that site, perhaps too many if you just want a simple circle. For that, there’s the very-stripped-down KML Circle Generator site. Enter the center point for the circle in lat/long, and either the desired radius in meters or a lat/long position you want the circle to pass through:

KMLcirclegenerator

Press the Go button, and get a link to the KML file:

link

Depending on how your browser is set up, clicking on the link will either download it to your computer, or open it automatically in Google Earth. Right-clicking on the link and choose “Save As” is the safest, as it will definitely download the file to your computer. Open it in Google Earth:

image

Default is a red circle; if you don’t like that, you can always right-click on the KML file listing at left, choose Properties, and change the style to whatever you want:

style

imagegreen

BTW, that big hole in the ground inside the circle is Meteor Crater, just outside Winslow, AZ.




Convert An Excel Address Spreadsheet Into A KML File (And Then Into A Geocoded CSV Text File) With KMLGeocode And KMLReport

Published by
Leszek Pawlowicz
 in Google Earth, KML and geocoding. 3 Comments

KMLGeocode (full UTK download page here) takes an address file in Excel (or XML) format, and creates a Google Earth KML file that plots geocodable addresses in the correct position. Load an address XLS file into the program (like this sample data also available on the download page):

KMLGeocode

You have to specify the column names that contain the key address fields required for geocoding, as well as the output KML filename. Once done, create the KML output file and open it in Google Earth. Addresses that could be geocoded will be plotted with orange pushpins:

geocoderesults

Clicking on  pushpin will bring up all the data fields for a record in the spreadsheet, not just the specified ones. Addresses that couldn’t be geocoded are given yellow pushpin designations, and will either show up plotted in a general location, or not plotted at all in Google Earth (though they will be listed in the Place pane under the KML file listing:

geocodeplaces

To embed the actual geographic coordinates for each of these addresses, right-click on the KML file listing (people.kml in the above example), and save it as another KML file with a different name. You can now open this different KML file in the KML Geocode Report program:

geocodereport

Specify the name of a text output file; this output will be a spreadsheet-readable CSV file that contains the original spreadsheet data with the longitude and latitude of geocodable addresses appended at the end as X and Y coordinates:

ID,Name,Group,Street,Street2,City,State,Zip,Phone,Email,X,Y


 11, Suzanne White, Office of Information Resources, 312 8th Ave North, Suite 1600, Nashville, TN, 37243, 615-253-4799, suzanne.white@state.tn.us,-86.784031,36.164133


 12, Kurt Snider, US Fish and Wildlife Service, 446 Neal St, , Cookeville, TN, 38501, 931-528-6481, kurt_snider@fws.gov,-85.497903,36.138056


 14, David Tirpak, Comptroller of the Treasury, 505 Deaderick St, Suite 1700, Nashville, TN, 37243-0277, 615-401-7820, david.tirpak@state.tn.us,-86.781603,36.165238

The “UnMatched” file is supposed to contain the non-geocodable addresses in KML format (at least, according to the PDF manual for these programs), but it didn’t show up for the sample data when I tried it. When you “Process” the files, in addition to the output file above, you’ll get a pop-up telling you the overall geocoding success percentage:

matchpercent

Note: When installing both programs, the installation directory is listed simply as “C:\Program Files\”; however, unless you specify the new name, it is installed in the directory “C:\Program Files\BRalston”, and “BRalston” is the folder in the Start Menu where you’ll find the program shortcut icons.




GISCloud – An Online Geographic Information System Application

Published by
Leszek Pawlowicz
 in GIS, KML, Web apps and shapefile. 1 Comment

I’ve been playing around a bit with GISCloud, a web-based GIS program. Looks interesting, and promising, but still has serious limitations. Feature set includes:

  • Raster and vector data display
  • Vector layer import and editing, including shapefiles, MapInfo, KML, tab-delimited and GPX
  • Built-in datafile coordinate reprojection; recognizes prj files, and lets you select the output coordinate system (including the one for your current project)
  • Advanced GIS analysis tools, including buffering, spatial selection by analysis, layer comparison (e.g. intersection), and area calculations.
  • Export vector data layers in shapefile, MapInfo, CSV or KML; this makes it a handy online format converter.
  • Share map editing with other users (or just publish it for viewing)
  • Easy-to-decode classic-looking GIS interface:

GISCloud

On the downside:

  • Only data layers imported into or through a PostGIS connection can be used for analysis and editing. I’m not up on PostGIS so I couldn’t test these functions, and none of the data sample sets have PostGIS data, either. You can apparently upload data and import it into a PostGIS database, though.
  • Uploading files can take a while (though they’re saved locally); GISCloud recognizes files in compressed archives, so you’re best off zipping up your data before you upload it.
  • For PostGIS data, I suspect that upload times are going to limit the size of the vector/database layers you can use.
  • Apparently no thematic (attribute-based) display colors yet; this is a major drawback.
  • Raster imagery updates during zooms can be slow.
  • Not all raster image types supported (e.g. 8-bit indexed GeoTiffs don’t work, 24-bit do).
  • It’s not exactly the zippiest GIS platform I’ve ever worked with.
  • It’s Flash-based, so Flash haters and iPad users should avoid, and everyone should expect CPU/performance/crashing issues. Flash 10.1 is supposed to be out soon, and supposedly will address some of those problems.

An interesting early effort in cloud-based GIS, and I’ll be watching its development. But unless you have access to, and/or experience with, PostGIS, of limited utility for now. And I don’t expect to see cloud-based GIS replacing dedicated GIS programs any time in the near future – too slow, and too limited.




Draw GPX Data Directly On Web Maps Using The GPS Visualizer Freehand Drawing Tool

Published by
Leszek Pawlowicz
 in GPS, GPX and KML. 0 Comments

I’ve posted recently about ScribbleMaps and ScribbleMaps Pro, web apps that let you draw features on a number of web maps (like Google Maps, OSM, and ESRI), and export the data in GPX format for upload to a GPS. Stopped by the GPS Visualizer website the other day, and just noticed that they have a similar application now, the GPS Visualizer Freehand Drawing Tool. On the down side, it doesn’t have nearly as many tools, editing options and output options as ScribbleMaps; on the plus side, because it doesn’t have a large number of tools and options, it’s easy and intuitive to use. And in addition to Google Maps and OSM Maps, it offers a number of background map options that ScribbleMaps doesn’t:

Continue reading ‘Draw GPX Data Directly On Web Maps Using The GPS Visualizer Freehand Drawing Tool’




Natural Earth Data In Google Earth

Published by
Leszek Pawlowicz
 in Google Earth, KML and cartography. 0 Comments

Last week, I posted about the new Natural Earth datasets, free medium-scale vector data for the earth to go along with the previously-available Natural Earth physiographic raster data. The Google Earth Library now has a KML network link that lets you view much of this data in Google Earth as layers, like here with lakes and rivers:

lakesrivers

Obviously, most of these features will be visible in Google Earth as soon as you zoom in, but these datasets let you view features with names at lower zoom levels, and more distinct colors. Plus, this offers a quick preview of the data available from Natural Earth, so you can see whether it meets your needs for use in a GIS or mapping program in shapefile format. There’s also an overlay with the raster physiographic imagery:

him

All of the data is in KML network links in subfolders of the main link, and some datasets will take a while to load. It’s also probably a good idea to not check the box next to the main network link, as this will load all the data into Google Earth, which will take a while.




« Previous Entries