Faced with find­ing a place to live in Dub­lin, I decided to quickly attempt to cre­ate a mashup of poten­tial dubMap.jpg prop­er­ties mapped through­out the city. This is, I sense, an increas­ingly com­mon sort of mashup. But when I did a quick scan, I couldn’t find any­thing that accom­plished this for the area I wanted. Yes, you could plot each place manu­ally in either the My Maps sec­tion of Google Maps or cre­ate a KML over­lay for Google Earth. Viable, but I wanted a tool that would allow the list to be dynam­ic­ally gen­er­ated and cap­ture the list cur­rent to when I was look­ing at it. I would note that many sites have their own spa­tial dis­plays, and there are a lot of mashups involving craigslist, but none for me and the Dub­lin scene. The site I was using to look for rental oppor­tun­it­ies is daft.ie. It’s got a great search engine, and it will map your selec­tions on a rental by rental basis, or will present all (unfiltered) list­ings in the area of the list­ing you have chosen. You can addi­tion­ally select a par­tic­u­lar agency and have their list­ings plot­ted on a Google Map, but this was not quite what I wanted. My object­ive was to cre­ate a cus­tom search, take the detailed res­ults, identify the loc­a­tion, geo­code it, and then pipe it into Google Maps. In concept simple — in prac­tise sub­ject to the vagar­ies of daft and the tools used.

So, what tools do we need to build such a map without wor­ry­ing about code?:

The first step is to con­struct the tar­get prop­er­ties from daft.ie. To do this, use their search form and have it return a paged series of search res­ults match­ing your cri­teria. I have not been able to crack their paging code and am lim­ited to the default 10 res­ults per page. I tried to aggreg­ate the paged res­ults in Dap­per, but to no avail. The work around for this is to cre­ate an account on daft and have it add list­ings to this as they are cre­ated. View­ing the My Daft page then provides non-paged view of your prop­er­ties and gives the data (albeit in a rel­at­ively unstruc­tured form) needed to build a cus­tom feed from.

The next step is to log into your Dap­per account and make a new Dapp. For those who have not used Dap­per, it allows you to take unstruc­tured data on a web page and turn it into struc­tured data through some pat­tern match­ing exer­cises. It’s entirely graph­ical and calls for no know­ledge of what’s going on behind the scenes. What you do is choose example pages con­tain­ing the data you want to grab, teach Dap­per how to recog­nize dif­fer­ent types of data, provide this data with descriptors and then choose the format for the res­ult­ing struc­tured data. Dap­per is very slick and con­sist­ently reliable.

I poin­ted Dap­per at ‘My Daft’ and it dis­played the page with a list of tar­get prop­er­ties. I clicked on all the loc­a­tion phrases for each entry and then told it that these con­tained loc­a­tion info. I added sep­ar­ate fields for details about the prop­erty, a pic­ture, the con­tact num­ber and most import­antly — the price. Once sat­is­fied with the con­struc­ted feed, I told Dap­per I wanted this as a pub­lic filtered RSS feed. Dap­per then provides a cus­tom URL con­tain­ing this inform­a­tion linked back to the ori­ginal source. Thus, when new things are added at the Daft end, they are picked up by Dap­per and fed through to our next step.

Armed with a feed of struc­tured data I turned to Yahoo PIpes to pick out the loc­a­tional inform­a­tion, geo­code this and plot it on a map. Yahoo Pipes provides a graph­ical edit­ing envir­on­ment into which you drag objects that rep­res­ent your input, the oper­a­tions you per­form on this input and the way in which you want to view the out­put. So, I chose a URL feed input wid­get and poin­ted it at the cus­tom Dap­per feed. Now I have the prop­erty inform­a­tion flow­ing in from Daft. I then choose the wid­get ’ loc­a­tion extractor’ that will scan any feed and attempt to loc­ate geo­spa­tial ref­er­ences. In this case, I don’t even have to spe­cify that the loc­a­tional inform­a­tion is enclosed by a <loc­a­tion> bracket. When it finds this inform­a­tion, it then attempts to geo­code it. Thus if we sup­ply a loc­a­tion of “Long Boat Quay North, Sir Roger­son Quay, Dub­lin Ci, Dub­lin 2, Dub­lin City Centre — Apart­ment to let” it will take the inform­a­tion, access a geo­cod­ing ser­vice and return a lon­git­ude and lat­it­ude: 53.339688,-6.216545. This is very cool. Note that it is not strip­ping out inform­a­tion, but adding new inform­a­tion to the exist­ing prop­erty ref­er­ence. Now we can point this out­put at a map and each packet of inform­a­tion is plot­ted using the pre­cise spa­tial inform­a­tion that has been determ­ined. Work­ing in Yahoo PIpes is quite cool. The graph­ical editor allows you to pipe your inform­a­tion between sources and oper­at­ors by drag­ging pipes between them. Great for visual thinkers.yahooPipes.jpg

When you run the Yahoo Pipes, it pulls all these pieces together and presents a map clev­erly dot­ted with pin­points indic­at­ing the prop­er­ties that you are inter­ested in. Click­ing on any prop­erty marker then brings up the details about that prop­erty. This works very well, but its not per­fect and some times the geod­coder simply can’t find the loc­a­tion from the text provided. Some­times it also guesses rather badly and you can end up with a prop­erty marker on the other side of the world. How­ever, it works in most cases and gives you a great up to the minute view of rental oppor­tun­it­ies meet­ing your search cri­teria in the city of choice.

A next step would be to start to provide some logic to deal with loc­a­tions which can­not be geo­coded. It might also be use­ful to col­our the pins by price range, num­ber of bed­rooms, etc. This is all accom­plish­able within PIpes once we have this great cus­tom feed com­ing from Dapper/Daft.

If you would like to exper­i­ment fur­ther with map mashups, here’s a couple good ref­er­ences:
Erle, Schuyler and Rich Gib­son, Google Maps Hacks, O’Reilly, 2006 — 0–596-10161–9
Lewis, Andre, Michael Pur­vis, Jef­frey Sam­bells and Cameron Turner, Begin­ning Google Maps Applic­a­tions with Rails and Ajax From Novice to Pro­fes­sional , Apress, 2007 — 1–59059-787–7
Lenssen , Phil­ipp, Google Apps Hacks, O’Reilly, 2008 — 978–0-596–51588-1
Shu-Wai Chow, PHP Web 2.0 Mashup Pro­jects, PAckt, 2007 — 978–1-847190–88-8