ODM Technology

Learning iPhone development

Starting to develop on the iPhone is surprisingly straightforward.  Having previously developped on the palm platform, I expected the SDK to be similar in complexity and learning time.  I was happy to find that the iPhones SDK and tools are remarkably quick to learn.

It had been some time since I did any C or C++ coding, so I wasn’t sure how I’d like learning objective-C.   It turned out to be easy enough, once I got used to the [object message] syntax.

For a quick guide to getting started, take a look a Furbo.org – Boostrap article. I ordered several of the books recommended there, including Beginning iPhone Development: Exploring the iPhone SDK, and I find this to be an excellent book. Not only does it teach iPhone coding in detail,but it also covers using XCode and Interface Builder.  Since I had never used a Mac before for anything, the explanations of the development tools were very welcome.

Getting a developer account under for my corporation has been taking some time, but I was able to get a personal account in 30 minutes.  I finally received a request from Apple to fax them my articles of incorporation.  I am not sure how long it will take from then, and I am not sure if I will bother with the corporate account since it will make very little difference in my case.

It will still take a few more days to come up to speed, and then app development can start.

Geocoding in PHP with Microsoft’s API

Yesterday I looked at using the Google Map API to geocode some addresses – today I examined the same using Microsoft’s API.

The Microsoft API is somewhat more complicated to use, as it is a SOAP API we can’t just fetch and parse HTTP as we could with Google.

To use these services with PHP, we need SOAP functions.  The server I had available to test had an old version of PHP (4.4), which doesn’t have this functionality built in.  It is easily added through the nusoap package.  Even with a later version of PHP that inclused SOAP functionality we might want to use nusoap anyway.

Microsoft has a technical article showing exactly how to geocode an address using  their MapPoint web services in PHP.  I think it’s actually the only piece of PHP sample code in the whole Virtual Earth documentation, and it happens to be for Geocoding – lucky me.   The code as is runs fine, and it’s easy to understand.

Before doing any of this, you will need to get an evaluation developers account opened.

Geocoding in PHP with Google’s API

Today’s project was geocoding entries in a database of businesses, where we have the street address, city and postal code, but need latitude and longitude for mapping purposes.  There are a number of web sites that will perform this service for a fee, but it can also be done without charge using either the Google Maps API or Microsoft Virtual Earth API.  Both of these APIs limit the number of queries per day which can be looked up without charge.   Also, both have terms which put some limitations on what can be done with the data, the Mircosoft one being much more restrictive.

Since the Google API is easier to use, I started with that one.  The function is written in PHP so that it will be easy to call in in response to searches from a web site if need be.  The function presented here is only using Google’s map API, it will be expanded to add an the Virtual Earth version to it, so that the caller can decide which service to use.

In my case, I only want to do lookups on records that actually have street addresses. Records that only specify a city, or maybe a PO box I am not interested in getting coordinates for, so that is why the test for record['street'] is there.  Google API allows a country bias value – this helps to resolve ambiguities in the address.  This value is the top level domain for the country, which is usually the country code, but not always.  Here I just want to make sure that Canadian records are recognized, so the test.  If this routine needs to deal with all countries, we would need to look up a table converting country codes to tlds, or else just assign the country code to the gl value and put in tests for the exceptions.  The sensor=false tells the API that this request is not coming from a device that can determine it’s own position.

define ("GOOGLE_KEY","yourapikeyforgoogle");
// geocode the supplied record using $service
function geoCode($record,$service="Google"){
    if ($service=="Google"){
        if ($record['street']){
            $address = $record['street'];
            $address .= " ,".$record['city'];
            $address .= " ,".$record['state'];
            $address .= " ,".$record['postalcode'];
            $url = "http://maps.google.com/maps/geo?q=".urlencode($address);
            $url.= "&output=csv&oe=utf8&sensor=false&key=".GOOGLE_KEY;
            if ($record['countrycode']=="CA") $url .= "&gl=ca";
            $result = file_get_contents($url);
 
            $parts = explode(",",$result);
            list ($status,$accuracy,$latitude,$longitude)=$parts;
            if ($status=="200"){
                $record['latitude']=$latitude;
                $record['longitude']= $longitude;
                $record['geo-accuracy']=$accuracy;
                $record['geo-service']="Google";
            }
        }
    }
    return $record;
}

Google allows a maximum of 15000 queries per day from a single IP. To code my values, I use a cron job that runs once per minute and looks up 10 at a time, with a 1 second interval between each. That gives 14400 per day, slightly below the limit.

For either Google MAP or Microsoft VE API’s you will need a key. Google will give you one instantly when you sign up, and Microsof will let you request an Evaluation Developer Acount - a slightly longer process.

Twittering

Ok, so after hearing about twitter for a while, I decided to try it out.  As I’m not normally particularly interested in communicating with anyone, the idea of broadcasting my comings and goings to the world at large seemed a little odd.  I did however go ahead and set up an account (@ordmillar), and all of a sudden there I was, tweeting away!

What makes it really interesting (apart from the people) is the twitter api.  Pretty much everything is exposed, so apps can do just about anything that a user could do through the web site.  I’ve been trying out some of the clients made with the api, but so far none of them has really impressed me.  The closest to what I want is one called Alert Thingy, which is nice in that it will show notices from twitter, facebook, flikr, RSS feeds and more.  It has worked well, execpt for one unexplained crash.  The flickr feed isn’t great – it sends a notice whenever anyone posts a picture, but doesn’t show any sort of preview so I am forced to visit the site if I want to see if I am interested in it.

With so many platforms out there, and so many social networking sites I can see that there is a need for the “grand unification tool”.  Almost like the grand unification theory of physics, execpt that the universe won’t end if we get it wrong.

Scraping with Solvent

The Solvent extension for FireFox lets us write  screen scrapers that turn a web page into RDF data.  The data in this format is intended to feel into their Piggy Bank package, but for my purposes I will be using the generated scraping script to covert data read by the Crowbar application into RDF.  From there, it becomes easy to convert it to any format I need and save it to the database.

Unfortunately Solvent requires Piggy Bank, which doesn’t work with FireFox 3.  Not being willing to remove my 3.0.6 installation, I needed a way to get a version 2 installed at the same time.  I found some simple instructions:  Firefox 2 and 3 Living Together in Harmony.  My install was slightly different than the one described because I wanted to keep my existing profile for version 3 and use the newly created one for version 2.   Also, I did this with the latest version of 2.x, which you can find at www.mozilla.com/en-US/firefox/all-older.html.

After making a test scraper and uloading to a web server, I tried it with Crowbar using the -mode=scrape option, and sure enough it returned an RDF/XML document.

There are a few bugs with Crowbar – the one that immediately caused me problems was non-encoded ampersand characters.  It’s otherwise such a useful tool that I will definitely be spending some time on it.

HTML Parsers

Once again, I find that I need to parse some HTML pages.  I’ve looked at difference parsers in the past, and always ended up writing some string or pattern matching routines to get the data I wanted off the page.  This time, the data I want isn’t actually on the HTML pages, it gets written by a Javascript that is included from a third party.  So, I need a parser/fetcher that understands Javascript, can load files, and can wait for the scripts to finish running before giving me the output.

The Cobra toolkit, part of the Lobo Browser project looked ideal.   The toolkit itself looked great, and was fairly easy to get set up and testing.  It generally fetched and parsed pages as I would expect, although often with a lot of warnings.  Unfortunately, the pages I was actually wanting to parse didn’t work.  I decided to try the whole Lobo Browser to see how it would display those pages, but the installer gave a permissions error at the beginning of the install.  I decided to put this aside for now.  When I have to start troubleshooting installers so that I can test a program to find out if a library can do what I want, I see a huge risk of lost time.

The Mozilla Parser looked like another good option.  This package is essentially a java interface to the engine that is used in the Firefox browser.  Since I knew that Firefox rendered the pages I wanted properly, this seemed like a good choice.   Again I had some issues getting this installed onto the 64 bit vista platform.  The documentation is limited, and the project seems to have not been touched in over a year.  Rather than spend a lot of time to find out why that wasn’t running, I looked at other options for using the Mozilla engine.

Gecko, the engine behind Mozilla’s browsers has a lot of information about embedding it.   There is a whole SDK available, and if it were a bigger project I would probably take this route.  They also provide XULRunner, which allows running of XUL applications outside of the browser.  (Firefox 3 can also be used to run XUL on windows, by invoking with the -app parameter)

The Crowbar program is an XUL app that fetches and parses web pages, and as it uses the same Firefox engine, it does parse the pages I need without any problem.  This runs as a server, it accepts HTTP requests and returns the data.  I will do some work with Crowbar, it looks like just a few small changes will make it the ideal tool for this project.

Distance betwen points in Java

I needed to calculate the distance between two postals codes, in the case where I have latitude and longitude for each one.  It took some searching around the net to find the right formula, although along the way I found dozens of pages that would calculate the distances for me.

The method milesBetween in my ZipCode class reports the number of miles between this Zip Code and the supplied argument.

double milesBetween(ZipCode zip){
  double distance;
  distance = ( 3958 * Math.PI * Math.sqrt(
    (this.latitude - zip.latitude) * (this.longitude - zip.longitude) +
    Math.cos( Math.toRadians(this.latitude)) * Math.cos( Math.toRadians(zip.latitude)) *
    ( (this.longitude - zip.longitude) * (this.longitude - zip.longitude) )
    )/180);
 
  return distance;
}

The constant 3958 is the (average) radius of the earth in miles.  Chaging to 6369 would give us the distances in Kilometers instead.

US Zip Codes Database

More and more, businesses have “store locators” on their web sites.  Most of them want either a State and City, or a Zip Code.  These are handy for finding the location of the nearest Starbucks, but they really suck when I want to get a list of all the ones in Alaska.  Considering that I will want to repeat this for many different chains, I need a solution that will let me automate it.

Using a list of all zip codes I could just brute force my way through it, querying every code in order.  With over 40000 codes in the US, this wouldn’t be too efficient – something like half a day at 1 query / second.  If there are 100 national chains that I wanted to do this for, it could take months.  It could be shortened by running several in parallel, maybe even using extra machines – but it’s still not a very elegant way of doing it.

Instead, knowing the coordinates of the zip code center, and the radius of the search I am using it should be possible to build a list of zip codes that cover the entire area, with as little overlap as possible.  Since I can’t be sure that the coordinates in my list of Zip Codes will be exactly the same as the coordinates used by the store locators, I’ll have to allow some overlap range.

For the zip code database, I am using one from http://www.free-zipcodes.com – this is a 2006 database, and while it’s not as up to date as some of the commercial ones, it should be fine for my purposes.  I find it strange that the us post office doesn’t have a zip code database for download.  There doesn’t even seem to be anything convenient to crawl to build one.

Future versions could use data from the 2000 US census to predict which zip codes are more likely to have businesses near them, so that the searches could start with the highest density areas first and then fill in the other spots later.

Blogging I go

Ok, after several years of saying ” I gotta get that blog thing going soon” here I am, blogging your interwebs.

Anyway, I will be doing my best to keep this updated with current projects and so on.