Getting started takes some doing.  First of all, we need to perform geocoding of  our records.  It is worth noting that not every record needs to be geocoded – ones that aren’t coded just won’t appear in proximity searches.  To get up and running quickly, we used a postal code database to get a rough location of the records that weren’t already geocoded.

For the changes to the Solr installation, I referred to the helpful tutorial at GISSearch.com.  Once the changes had been made, I reindexed our records.  This was the longest part of the process – even though I only processed the Canadian records for this test, there are still over a million to go through.  If the server wasn’t being used I could have shut it down, deleted the indexes and rebuild to save some time.

Along the way, a few problems came up.  First was that a version build with the latest sources didn’t work, I had to revert to some earlier stable versions.  At GISSearch there is an example package that has a compiled solr that works, so that is a good place to start if you are having issues there.  The other big problem was that a bug in the phps output writer was preventing the searches from running.  Switching to xml or json output solves that.

Using Local Solr instead of writing our own solutions has saved a lot of development time.  We still need to do some performance testing to see how it will hold up under heavy usage, but so far it looks like with a dedicated server for geo searching we will be able to keep up with the loads.

Since the Google API is easier to use, I started with that one.  The function is written in PHP so that it will be easy to call in in response to searches from a web site if need be.  The function presented here is only using Google’s map API, it will be expanded to add an the Virtual Earth version to it, so that the caller can decide which service to use.

In my case, I only want to do lookups on records that actually have street addresses. Records that only specify a city, or maybe a PO box I am not interested in getting coordinates for, so that is why the test for record['street'] is there.  Google API allows a country bias value – this helps to resolve ambiguities in the address.  This value is the top level domain for the country, which is usually the country code, but not always.  Here I just want to make sure that Canadian records are recognized, so the test.  If this routine needs to deal with all countries, we would need to look up a table converting country codes to tlds, or else just assign the country code to the gl value and put in tests for the exceptions.  The sensor=false tells the API that this request is not coming from a device that can determine it’s own position.

define ("GOOGLE_KEY","yourapikeyforgoogle");
// geocode the supplied record using $service
function geoCode($record,$service="Google"){
    if ($service=="Google"){
        if ($record['street']){
            $address = $record['street'];
            $address .= " ,".$record['city'];
            $address .= " ,".$record['state'];
            $address .= " ,".$record['postalcode'];
            $url = "http://maps.google.com/maps/geo?q=".urlencode($address);
            $url.= "&output=csv&oe=utf8&sensor=false&key=".GOOGLE_KEY;
            if ($record['countrycode']=="CA") $url .= "&gl=ca";
            $result = file_get_contents($url);
 
            $parts = explode(",",$result);
            list ($status,$accuracy,$latitude,$longitude)=$parts;
            if ($status=="200"){
                $record['latitude']=$latitude;
                $record['longitude']= $longitude;
                $record['geo-accuracy']=$accuracy;
                $record['geo-service']="Google";
            }
        }
    }
    return $record;
}

Google allows a maximum of 15000 queries per day from a single IP. To code my values, I use a cron job that runs once per minute and looks up 10 at a time, with a 1 second interval between each. That gives 14400 per day, slightly below the limit.

For either Google MAP or Microsoft VE API’s you will need a key. Google will give you one instantly when you sign up, and Microsof will let you request an Evaluation Developer Acount - a slightly longer process.

Using a list of all zip codes I could just brute force my way through it, querying every code in order.  With over 40000 codes in the US, this wouldn’t be too efficient – something like half a day at 1 query / second.  If there are 100 national chains that I wanted to do this for, it could take months.  It could be shortened by running several in parallel, maybe even using extra machines – but it’s still not a very elegant way of doing it.

Instead, knowing the coordinates of the zip code center, and the radius of the search I am using it should be possible to build a list of zip codes that cover the entire area, with as little overlap as possible.  Since I can’t be sure that the coordinates in my list of Zip Codes will be exactly the same as the coordinates used by the store locators, I’ll have to allow some overlap range.

For the zip code database, I am using one from http://www.free-zipcodes.com – this is a 2006 database, and while it’s not as up to date as some of the commercial ones, it should be fine for my purposes.  I find it strange that the us post office doesn’t have a zip code database for download.  There doesn’t even seem to be anything convenient to crawl to build one.

Future versions could use data from the 2000 US census to predict which zip codes are more likely to have businesses near them, so that the searches could start with the highest density areas first and then fill in the other spots later.