Mashing Without Code

Faced with finding a place to live in Dublin, I decided to quickly attempt to create a mashup of potential properties mapped throughout the city. This is, I sense, an increasingly common sort of mashup. But when I did a quick scan, I couldn’t find anything that accomplished this for the area I wanted. Yes, you could plot each place manually in either the My Maps section of Google Maps or create a KML overlay for Google Earth. Viable, but I wanted a tool that would allow the list to be dynamically generated and capture the list current to when I was looking at it. I would note that many sites have their own spatial displays, and there are a lot of mashups involving craigslist, but none for me and the Dublin scene. The site I was using to look for rental opportunities is daft.ie. It’s got a great search engine, and it will map your selections on a rental by rental basis, or will present all (unfiltered) listings in the area of the listing you have chosen. You can additionally select a particular agency and have their listings plotted on a Google Map, but this was not quite what I wanted. My objective was to create a custom search, take the detailed results, identify the location, geocode it, and then pipe it into Google Maps. In concept simple – in practise subject to the vagaries of daft and the tools used.

So, what tools do we need to build such a map without worrying about code?:

• Daft.ie for data
• Dapper.com to create a dynamic RSS feed
• Yahoo Pipes to geocode and output a further feed
• Yahoo Maps because that’s what Pipes works with

The first step is to construct the target properties from daft.ie. To do this, use their search form and have it return a paged series of search results matching your criteria. I have not been able to crack their paging code and am limited to the default 10 results per page. I tried to aggregate the paged results in Dapper, but to no avail. The work around for this is to create an account on daft and have it add listings to this as they are created. Viewing the My Daft page then provides non-paged view of your properties and gives the data (albeit in a relatively unstructured form) needed to build a custom feed from.

The next step is to log into your Dapper account and make a new Dapp. For those who have not used Dapper, it allows you to take unstructured data on a web page and turn it into structured data through some pattern matching exercises. It’s entirely graphical and calls for no knowledge of what’s going on behind the scenes. What you do is choose example pages containing the data you want to grab, teach Dapper how to recognize different types of data, provide this data with descriptors and then choose the format for the resulting structured data. Dapper is very slick and consistently reliable.

I pointed Dapper at ‘My Daft’ and it displayed the page with a list of target properties. I clicked on all the location phrases for each entry and then told it that these contained location info. I added separate fields for details about the property, a picture, the contact number and most importantly – the price. Once satisfied with the constructed feed, I told Dapper I wanted this as a public filtered RSS feed. Dapper then provides a custom URL containing this information linked back to the original source. Thus, when new things are added at the Daft end, they are picked up by Dapper and fed through to our next step.

Armed with a feed of structured data I turned to Yahoo PIpes to pick out the locational information, geocode this and plot it on a map. Yahoo Pipes provides a graphical editing environment into which you drag objects that represent your input, the operations you perform on this input and the way in which you want to view the output. So, I chose a URL feed input widget and pointed it at the custom Dapper feed. Now I have the property information flowing in from Daft. I then choose the widget ‘ location extractor’ that will scan any feed and attempt to locate geospatial references. In this case, I don’t even have to specify that the locational information is enclosed by a <location> bracket. When it finds this information, it then attempts to geocode it. Thus if we supply a location of “Long Boat Quay North, Sir Rogerson Quay, Dublin Ci, Dublin 2, Dublin City Centre – Apartment to let” it will take the information, access a geocoding service and return a longitude and latitude: 53.339688,-6.216545. This is very cool. Note that it is not stripping out information, but adding new information to the existing property reference. Now we can point this output at a map and each packet of information is plotted using the precise spatial information that has been determined. Working in Yahoo PIpes is quite cool. The graphical editor allows you to pipe your information between sources and operators by dragging pipes between them. Great for visual thinkers.

When you run the Yahoo Pipes, it pulls all these pieces together and presents a map cleverly dotted with pinpoints indicating the properties that you are interested in. Clicking on any property marker then brings up the details about that property. This works very well, but its not perfect and some times the geodcoder simply can’t find the location from the text provided. Sometimes it also guesses rather badly and you can end up with a property marker on the other side of the world. However, it works in most cases and gives you a great up to the minute view of rental opportunities meeting your search criteria in the city of choice.

A next step would be to start to provide some logic to deal with locations which cannot be geocoded. It might also be useful to colour the pins by price range, number of bedrooms, etc. This is all accomplishable within PIpes once we have this great custom feed coming from Dapper/Daft.

If you would like to experiment further with map mashups, here’s a couple good references:
Erle, Schuyler and Rich Gibson, Google Maps Hacks, O’Reilly, 2006 – 0-596-10161-9
Lewis, Andre, Michael Purvis, Jeffrey Sambells and Cameron Turner, Beginning Google Maps Applications with Rails and Ajax From Novice to Professional , Apress, 2007 – 1-59059-787-7
Lenssen , Philipp, Google Apps Hacks, O’Reilly, 2008 – 978-0-596-51588-1
Shu-Wai Chow, PHP Web 2.0 Mashup Projects, PAckt, 2007 – 978-1-847190-88-8