blankblank blank

Archive for the 'shapefile' Category

Crowdsourcing And Coordinating Data Collection With Handheld GPS Units

Published by
Leszek Pawlowicz
 in GIS, GPS, Google Earth, Google Maps, KML, MapWindow, Web apps, data, shapefile and spreadsheet. 1 Comment

Got an email today from someone with a data collection/coordination/assembly problem. They have multiple people out in the field with Garmin GPS units recording data, and bringing it back to a central location for collation/combination when they’re done. Since Garmin GPS units have limited capabilities for data storage – coordinates, name, comment, date and time, elevation, and that’s pretty much it – additional data associated with a point has to be recorded by hand. Once back from the field, data has to be downloaded from individual units, associated with the additional attribute data from forms, all the data combined together into a single dataset, then converted into GIS-friendly format. And the process they had come up with wasn’t really working well for them.

I think this highlights some of the major limitations of classic stand-alone handheld GPS units. They’re really designed for us in personal data collection, not combined data collection; assembling data from multiple units can take a lot of work. Plus, their limited data collection capabilities require offloading data attribute acquisition to other formats (e.g. pencil and paper), adding the addition problem of associating that data with coordinates later on. There are lots of professional solutions for these problems, like Trimble or MobileMapper GPS units, Terrasync and ArcPad software, but these can be complicated and expensive. I challenge you to find anyone with enough patience to use Terrasync for a single day without swearing at least once; I know I can’t  ;-).

For a few years, I’ve had CyberTracker on my list of potential topics to post on. CyberTracker is a terrific data acquisition and collation tool for field data with tons of great features, like custom data acquisition form design for easy data entry, moving maps, easy data collation and conversion to GIS-friendly format. Plus, the software is free, and the hardware is (relatively) affordable. But unfortunately, I think time and technology are passing Cybertracker by:

  • While the central data software runs on Windows, the field data collection software runs on old-school Palm OS and Windows Mobile. Palm OS is dead; Windows Mobile development has stopped with version 6.5, and only maintenance updates are scheduled. So the useful lifetime of any data collection system built around CyberTracker is limited. If you need a fast/cheap/short-term solution, CyberTracker is free for non-profit use, and you could probably pick up the hardware very cheaply on eBay. Long-term, I don’t see it having much of a future in its current form.
  • The data collation model is also becoming obsolete. With CyberTracker, you have to bring all the data units to a single data download/collation computer. While there are still places in the world where this might make sense, an increasingly-connected world means that data download/collation to the cloud instead of an individual computer makes more sense.

I’m really excited about the possibilities for crowdsourced geographic data collection and collation using portable devices running Android OS (yeah, iOS too). There are already several interesting apps for doing this, and I hope to cover some of those soon on my AndroGeoid website. And I suspect that the small number of apps that can currently do this will be quickly joined by far more apps, and far more capable apps, in the very near future. But that doesn’t help with my emailer’s current problem. I suspect there are many different ways you could do this, but here’s the first approach that came to me using all-free software and services.

1. Make sure that everyone on the project has a Google account (i.e. Gmail); completely free.

2. Have the project leader created a single main data spreadsheet on Google Docs, with all the desired data attributes (e.g. point name, coordinates, comments, additional data fields from the paper forms, etc.) and share a link to that spreadsheet with other project members so that they can edit it as well.

3. Project members can download data from their Garmin units using DNRGarmin, and then export the data from DNRGarmin in CSV format.

4. Load the data into the spreadsheet program of your choice, and add/edit data from data forms to make it conform to the data structure of the main Google Docs spreadsheet.

5. Copy the data cells in the spreadsheet program, and paste them into the main Google Docs spreadsheet. Note: Use Ctrl-C and Ctrl-V to copy and paste cells into Google Docs instead of using the Google Docs Edit menu to perform those operations, as the latter doesn’t seem to work for pasting data from different applications into Google Docs. You no longer have a single computer as a choke point for data entry; multiple people can add data to a spreadsheet at the same time, and Google Docs will coordinate data entry so that nothing is lost. And if you save the spreadsheet data from steps 3 and 4 as separate files, you’ll have backup copies of the original data as well.

For those who aren’t comfortable with working with spreadsheets, Google Docs lets you set up a “Form” to let anyone add data to a Google Docs spreadsheet directly; however, this increases the chances of coordinate data entry error.

6. Now that you have all the data centralized in Google Docs, you have lots of flexibility in how you can handle it:

  • Export the data in CSV format, and you can then import it into any GIS program that supports CSV data. If your GIS program doesn’t, use MapWindow to convert your CSV file into shapefile format first.
  • Unlike shapefile attribute tables, where adding/removing/re-ordering attribute data columns can be a pain, you can easily perform those operations in Google Docs and then re-export the data in CSV format.
  • Convert the data directly into a continuously-updated KML network link for display in Google Earth or Maps using Google’s Spreadsheet Mapper tool.
  • Use any of Google Docs built-in tools to analyze/plot/sort/visualize data, including their way-cool Fusion Tables.
  • And I’m sure there are more options I haven’t thought of.

Know an alternate approach? Have any additional ideas?  I welcome your links and suggestions in the Comments section below.




World Vector Data (VMAP0 and VMAP1) In Shapefile Format

Published by
Leszek Pawlowicz
 in GIS, data and shapefile. 1 Comment

About three years ago, I posted about Penn State’s Digital Chart Of The World Server, which hosted old 1991/1992 VMAP0 world vector data from the National Geospatial Agency in E00 format. More recent data is available directly from the NGA, some of it in the higher-resolution VMAP1 dataset not available from Penn State, but the NGA data is in the uncommon VPF format that many GIS programs can’t handle. The Russian GIS Lab website has converted more recent VMAP0 and VMAP1 data into the more-commonly-used shapefile format, and made it openly available for download. You can get VMAP0 data for the whole world at this link, and the higher-resolution VMAP1 data for selected areas at this link.

While you can get similar and much fresher shapefile data for some of this info from the CloudMade OSM shapefile data site, and from the Natural Earth vector dataset, the VMAP0/VMAP1 files include data like railroads, utilities, cultural landmarks and others that can be difficult to track down.




GISCloud – An Online Geographic Information System Application

Published by
Leszek Pawlowicz
 in GIS, KML, Web apps and shapefile. 1 Comment

I’ve been playing around a bit with GISCloud, a web-based GIS program. Looks interesting, and promising, but still has serious limitations. Feature set includes:

  • Raster and vector data display
  • Vector layer import and editing, including shapefiles, MapInfo, KML, tab-delimited and GPX
  • Built-in datafile coordinate reprojection; recognizes prj files, and lets you select the output coordinate system (including the one for your current project)
  • Advanced GIS analysis tools, including buffering, spatial selection by analysis, layer comparison (e.g. intersection), and area calculations.
  • Export vector data layers in shapefile, MapInfo, CSV or KML; this makes it a handy online format converter.
  • Share map editing with other users (or just publish it for viewing)
  • Easy-to-decode classic-looking GIS interface:

GISCloud

On the downside:

  • Only data layers imported into or through a PostGIS connection can be used for analysis and editing. I’m not up on PostGIS so I couldn’t test these functions, and none of the data sample sets have PostGIS data, either. You can apparently upload data and import it into a PostGIS database, though.
  • Uploading files can take a while (though they’re saved locally); GISCloud recognizes files in compressed archives, so you’re best off zipping up your data before you upload it.
  • For PostGIS data, I suspect that upload times are going to limit the size of the vector/database layers you can use.
  • Apparently no thematic (attribute-based) display colors yet; this is a major drawback.
  • Raster imagery updates during zooms can be slow.
  • Not all raster image types supported (e.g. 8-bit indexed GeoTiffs don’t work, 24-bit do).
  • It’s not exactly the zippiest GIS platform I’ve ever worked with.
  • It’s Flash-based, so Flash haters and iPad users should avoid, and everyone should expect CPU/performance/crashing issues. Flash 10.1 is supposed to be out soon, and supposedly will address some of those problems.

An interesting early effort in cloud-based GIS, and I’ll be watching its development. But unless you have access to, and/or experience with, PostGIS, of limited utility for now. And I don’t expect to see cloud-based GIS replacing dedicated GIS programs any time in the near future – too slow, and too limited.




Free Vector Country Borders, Administrative Boundaries, And Soon More

Published by
Leszek Pawlowicz
 in GIS, data and shapefile. 4 Comments

Wish I’d known about this site when I was hunting for a high-res border outline for the province of Umbria in Italy. The Global Administrative Boundaries website offers free vector GIS data for country borders and administrative areas within those borders. Data is available in geographic coordinates (lat/long), WGS84 datum, in multiple formats:

Coverage is worldwide for level 1 data (country borders), but diminishes as you go down as far as level 4 (sub-administrative units), though for the latter there are some countries for which such a data level isn’t relevant. You can see maps of current level coverage here, and missing data will likely be added in the future. Data is licensed under Creative Commons for the US; other unspecified terms may apply for the rest of the world. Here’s the Italian province of Umbria, with the best free data I could initially find plotted in purple, and the high-res GADM data in blue:

umbria

Via Slashgeo.

Just around the corner, scheduled for unveiling at the October 2009 NACIS conference, is Natural Earth Vector. While there’s some overlap with GADM at the upper boundary levels, Natural Earth Vector may not include some of the finer administrative boundaries available in the GADM . But it will have a wide variety of other geographic features not available in GADM in vector format at multiple zoom levels (list from the web post):

  • Continents (North America, Europe, Asia, etc)
  • Cultural regions (South Asia, West Africa, etc)
  • Countries (US, Canada, Mexico, etc)
  • Country sub-divisions (for the US, states, semi-independent territories, dependencies, associations)
  • Disputed territories (like Kashmir, Northern Cyprus)
  • 1st order admins (states, provs)
  • Bathymetry
  • Lakes
  • Lake Center Lines
  • Rivers (including attributes that allow easy “tapering” of drains)
  • Islands
  • Glaciers
  • Populated places (urban boundaries, not city points)
  • Cities (point locations)
  • Physical features like peaks, ranges, valleys, plains

I’ll try to post again when it’s officially released.

Via Kelso’s Corner.




Habitat Digitizer Extension For ArcGIS

Published by
Leszek Pawlowicz
 in ESRI, GIS and shapefile. 0 Comments

Stumbled across this the other day, and while I don’t currently have the need for it (or a copy of ArcGIS to try it out on), it sounds useful enough to post about. NOAA’s Center for Coastal Monitoring and Assessment offers a free extension for ArcGIS called the Habitat Digitizer:

The Habitat Digitizer Extension is designed to use a hierarchical classification scheme to delineate habitats by visually interpreting georeferenced images such as aerial photographs, satellite images, and side scan sonar. The extension allows users to create custom classification schemes and rapidly delineate and attribute polygons, lines, and points using simple menus. The extension allows new hierarchical classification schemes to be easily created, modified, and saved for use on future mapping projects. There are several advantages to using classification schemes with a hierarchical structure including: the detail of habitat categories can be expanded or collapsed to suit user needs, the thematic accuracy of each category/hierarchical level can be determined, and additional categories can be easily added or deleted at any level of the scheme to suit user needs.

Habitat_Digitizer

While written for habitat digitization, it could prove useful for any digitization projects with extensive classification requirements. Versions are available for ArcView 3.0, ArcGIS 9.0, and ArcGIS 9.2 from the product page; download files include a PDF manual.




Convert Shapefiles Into SVG Format With Thematic Coloring

Published by
Leszek Pawlowicz
 in cartography and shapefile. 2 Comments

Not too long ago, I covered an online application from indiemapper that converts shapefiles into SVG format, a standard format for vector illustration programs like Adobe Illustrator or the open source Inkscape. One drawback of that site was the inability to color/shade the SVG output based on shapefile attribute data. Ken Hardy has a Perl utility called shptosvg that will take shapefiles and convert them into SVG format, and includes the ability to color files based on their attributes:

410px-Ogallala

Features (from the website):

  • Accepts source SRS specs on the command line, allowing multiple input files to have disparate coordinate systems.
  • Generates the results projected to any SRS using Geo::Proj4.
  • Works on multiple shapefiles at a single invocation, layering them in order into the SVG map.
  • Supports specification of color and other rendering options, per source file, on the command line.
  • Optionally select subset of shapes by regex on arbitrary attribute fields.
  • Support coloring of shapes by value of an attribute field.
  • Allow specification of a shapefile to use in scaling but not in rendering. That supports independent production of layers to be later combined in an SVG editor; they can all be scaled and translated according to the geometry of the base/largest layer to aid layering after the fact.

You’ll need to have Perl installed on your system; see the download page on the official Perl site for links to several different versions. And you’ll need to be comfortable working in a command-line interface; no GUI available yet. Ken has a page documenting usage, along with an example page that describes how he created the image above.

HT to Ken (of course).




MapTool Goes To Version 2– Elevations, Areas, And Distances In Google Maps

Published by
Leszek Pawlowicz
 in GPX, Google Maps and shapefile. 1 Comment

I’ve covered Zonums Software free online application MapTool before; the original version let you determine the elevation of a point in a Google Maps interface, as well as measuring the length of a multi-segment path or drawn polygon, but not much more. Zonums has recently updated this application to  MapTool 2, which adds some really nice new functions (most accessible by clicking on the Options link at top):

  • Create vertices by manually entering coordinates, in addition to previous option of clicking on map (use Input link at the top for this)
  • Continuous coordinate read-out in both latitude/longitude and UTM
  • Divide a drawn line into equal parts by automatically adding equally-spaced vertices, or add vertices with a user-defined spacing
  • Convert a drawn polygon into a polyline, and vice versa
  • Edit individual vertices in a line or polygon
  • Get a count of the total number of vertices in the drawn feature
  • Get an elevation profile for all the vertices on a line

profile

  • Determine the extents (bounding box) for the drawn feature
  • Export the vertex coordinates as a text file
  • Export the line or polygon directly as a GPX, DXF or shapefile in latitude/longitude/WGS84, with the option to include vertex elevations



LandSerf – Google Earth And GPS Functions

Published by
Leszek Pawlowicz
 in GPS, GPX, Google Earth, KML and shapefile. 2 Comments

In previous posts, I reviewed LandSerf’s raster terrain analysis functions and vector functions. Today I’ll wrap up with a short review of LandSerf’s Google Earth and GPS functions.

Google Earth: The most notable Google Earth functionality is the ability to export vector data, either imported or generated by the program, into KML vector files. So I can open a DEM, like the sample one of Mt. Rainier included with the program:

lsdem

Generate a set of flow vectors for the DEM:

flowvectors

And then export the vector data as a KML file:

rainier

You could do something similar with a shapefile opened in LandSerf, but just be sure the shapefile is in NAD83 or WGS84. As I mention in the post on LandSerf vector functions, LandSerf has problems re-projecting from one datum to a significantly different second datum, and Google Earth data needs to be in WGS84 (NAD83 is almost the same). A search for “KML shapefile” on this blog will bring up a number of other programs that do a better job of converting shapefiles to KML, including preserving attribute data, which LandSerf doesn’t. But all of those are Windows-based; LandSerf runs on Macintosh and Linux, so it might be a useful option for those OSes.

Note: You should save vector data as a KML file; the KMZ file format is supposedly reserved for saving raster data as Google Earth image overlays, but I’ve been unsuccessful in getting that to work.

GPS: The list of useful GPS functions in LandSerf is a bit longer:

  • Convert loaded shapefiles or other vector data to GPX format (data must be in WGS84/NAD83 datum)
  • Load a GPX file as vector data, then save it in any of the vector formats LandSerf supports
  • Interface directly with a GPS. You’ll need to have the GPS connected to your computer and turned on before starting up LandSerf. To establish a connection to the GPS, use Configure=>GPS and scan for a connected unit. Once you do this, you can import waypoint data using the File=>Import from GPS function. While the dialog box gives you the option to import waypoints, tracks and routes:

gpsimport

I’ve had difficulty getting tracks to import successfully – the program just shuts down on me. This may just be an issue with my model GPS (Garmin 60Cx); f you have better luck, let me know. You can import the data in the default lat/long coordinate system. or re-project it on import to OSNG or UTM. Once there, you can export it in GPX, shapefile or other vector format.

  • You can also export point data (no lines or polygons) directly from LandSerf to a connected GPS as waypoints.

If you can get track data to import successfully from your GPS to LandSerf, which I couldn’t, there are several other functions that might prove useful:

  • On importing the data, you have the option of saving all the track data in spreadsheet format with position, elevation and time
  • As with other vector data, if you have a digital elevation model loaded in the program, you can get a elevation profile plot; see this previous LandSerf post for more info
  • If you have a GPS track that is broken into .segments because you lost the GPS signal, LandSerf has the ability to join all of the line segments into a single line, which you can then save as a GPX track file, using the Edit => Join vector lines function. AFAIK, it’s the only free program that can do this.

As with the Google Earth functions, there are other programs that can do most of this in Windows, but far fewer free options in Macintosh and Linux.




« Previous Entries