Mashups using Google Mashup Editor and Yahoo pipes

Introduction

Google have released a mashup editor, that makes it easy to create small functional web apps using their apis.

Yahoo have pipes which allows you to create, manipulate, and aggregate rss feeds.

The Mash Up

I want to create a mash up with the contents of the RTE news feed, hooked up with maps so the location of the news topic is pinpointed.

Use Yahoo Pipes

First I use yahoo pipes to suck in the RTE feed. Once complete I add a location extractor. This will scan the feed items and add a y:location element if it can successfully determine location from the content.

See the pipe output here.

Use Google Mashup Editor

I can then use google mashup editor (GME) to create a list of items from the Yahoo pipe.

The items when imported to GME get converted to atom format, and the y:location added by Yahoo pipes is added to the item's geo:lat and geo:long elements respectively.

Each item has an event handler when selected. When fired an item's geo location is parsed and used to pinpoint the location on the map.

You can see the end result here http://rtenewsgeocoded.googlemashups.com/ (currently if no location can be established the map point is random). If you select a news item the subsequent location will be pinpointed on the map.

Update: Use Yahoo Maps with pipes

Google maps with GME doesn't work well when only a few of the items are geocoded. It still plots them on the map at an arbitrary point.

I started looking at this from a number of angles, one being that the location is being determined from the feed content. When the feed contains only a title and one sentence summary, that makes it it increasingly difficult.

My first pass was to obtain the full content for the news article from another mashup tool: Dapper.

Once Dapper is setup to extra the full content, I need to loop through each item in the original feed and construct the URL to call the dapper service. The result contains the full news article content, which I use to replace the original item's description. This increases the complexity of the pipe.

You can see the updated pipe output as html here.

Once I have the detailed content it should mean a higher hit ratio for geo locations.

Pipe + Map output

Finally, rather than using Google Maps, the Yahoo map and pipes integration works better. See the final result here. The map tab shows these items that could be geocoded, while the list tab shows all items.

Embeddable Maps by Google

If you use google maps you'll know they've always offered a link to this map option.

Now in a much anticipated move they have launched embeddable maps. It gives you the ability to embed maps in any web page with a snippet of HTML.

Here's an example:

View Larger Map

Excel Spreadsheet for getting geo locations

Geocoding

Ireland was recently added to Google Maps Geocoder.

What I didn't know was that Yahoo already had geocoding for Ireland. In fact as far as simplistic interfaces Yahoo wins hands down. They have a simple rest interface for geocoding.

The parameters are listed on their site, but it means I can pass the following query and receive a response.

http://local.yahooapis.com/MapsService/V1/geocode?appid=YahooDemo&location=128+East+Wall+Road,Dublin+3,Ireland

Using in Excel

I started looking at using this from Excel and found that Jon Udell had already started something.

He used AutomateExcel.com and Juice Analytics as a starting point and then rolled his own.

I did similar using Jon's as the basis but reintroducing some aspects of the others. in particular removing the dependency on XML parsing.

The changes I have made are to include some extra columns. One, a very important one in my mind is a check link column. It autogenerates a link to Google Maps to the returned Geo code for manual confirmation. Also the precision, warning, and error columns are included to give back as much info as possible as the user.

I also elected to use the Yahoo parameter location which allows comma separated free text for the address, rather than the more specific params street,city,start,zip.

How it works

Simple really! Add one or more addresses from row 4 onwards, into columns 2-6 and hit Auto Geocode.

It only sends a request to Yahoo if Address 1 is filled in, and Auto Location is blank.

Before

After

Getting the spreadsheet

You can download it from here.

If decide to use it, please link back to this post.

Ireland added to Google Maps geocoder

See the link here

http://googlemapsapi.blogspot.com/2007/08/looking-for-somewhere-in-india-hong.html

Enter an address without commas e.g. '128 East Wall Road Dublin 3 Ireland'

See an example

http://maps.google.com/maps?f=q&hl=en&geocode=&q=128+East+Wall+Road+Dublin+3+Ireland&sll=53.356421,-6.231713&sspn=0.016367,0.057335&ie=UTF8&z=15&iwloc=cent&om=1

Mongrel and Capistrano 2.0

I use Capistrano, and Mongrel. Mongrel comes prepackaged with capistrano 1.4 recipes for stopping and starting, making it a cinch to roll a new version and stop and start your service.

Capistrano 2.0 has just been released, and mongrel hasn't yet been updated. The solution is to roll your own tasks for mongrel. You can do this by adding the following to your project's deploy.rb file.

Update: minor bug fixed in code. Thanks to funkdoobiest for pointing it out.

Update: mongrel_conf may need to be defined in your deploy.rb as meekish points out in the comments.

set :mongrel_conf, "#{deploy_to}/current/config/mongrel_cluster.yml"


namespace :deploy do
  namespace :mongrel do
    [ :stop, :start, :restart ].each do |t|
      desc "#{t.to_s.capitalize} the mongrel appserver"
      task t, :roles => :app do
        #invoke_command checks the use_sudo variable to determine how to run the mongrel_rails command
        invoke_command "mongrel_rails cluster::#{t.to_s} -C #{mongrel_conf}", :via => run_method
      end
    end
  end

  desc "Custom restart task for mongrel cluster"
  task :restart, :roles => :app, :except => { :no_release => true } do
    deploy.mongrel.restart
  end

  desc "Custom start task for mongrel cluster"
  task :start, :roles => :app do
    deploy.mongrel.start
  end

  desc "Custom stop task for mongrel cluster"
  task :stop, :roles => :app do
    deploy.mongrel.stop
  end

end


Capistano already has tasks for deploy:restart, deploy:start, and deploy:stop. What the code above does is add the following tasks:


deploy:mongrel:start
deploy:mongrel:restart
deploy:mongrel:stop


It then overrides the default deploy:restart, deploy:start, and deploy:stop tasks to call the mongrel ones.

So now the following commands: stop, start and restart your mongrel servers respectively.

cap deploy:stop
cap deploy:start
cap deploy:restart