The Data Source Handbook by Pete Warden provides a concise and handy guide to some of the main sources of public data accessible on the web today. It’s a very short book of 40 pages. This in itself does not stand against the book. These sources are rapidly changing and compiling and committing an exhaustive survey to a printed volume would damn it to almost instant obsolescence. It would also prevent any treatment of individual datasources in any useful detail.

As it is, Warden is able to pick a select few and identify strengths and available APIs in a useful fashion. He organises the type of sources into logical categories and identifies some key sources for each:

• Websites
• People
• Search terms
• Locations
• Companies
• IP Addresses
• Books, films, movies, music and products

He selects the key open providers of data in these areas and systematically shows how to access the information along with simple programmatic instructions. In a volume of such limited length you would not expect to find extensive instructions or discussion — and you won’t. What you have is a very concise survey identifying the key players and giving a nutshell indication of what you can use the datasources for.
This is a useful and quick reference for anyone routinely accessing, compiling, aggregating or augmenting their own datasets. Although very few of the sources identified would be new to most people in the data analysis space, this does provide a useful compilation and also handy concise reminder of how one might augment a limited dataset quickly in an automated fashion.
This is an easily accessible volume, well organized and with the only major failing that it will be become dated in a published form. However, as an eBook it is ideal and I would recommend it to anyone new to the area of adata visualisation looking for some great sample data to access, or to the more seasoned data traveller looking to keep their familiarity with the wide variety of available data current.

a

In Mining the Social Web Matthew A Russell offers to instruct in identifying social connections, trends in discussion and locations by tapping into social media data. He succeeds in spades. This fast-paced and rich handbook jumps right into the fray and provides an immediate and useful exercise in accessing the Twitter API using python and doing a very quick visualisation of trending subjects. I was hooked and greedily and immediately consumed a few more of his lessons. His approach is to go direct to real world applications of why you’d want to mine data from social media such as Twitter, Buzz, Facebook and utilise other freely available tools such as Google Maps to look for patterns and present solid research findings.

As he asserts, all the user needs is some programming background and a willingness to learn basic Python tools. Fair requirements. I am no python wizard, but I found clear instructions and come away feel far more comfortable with python and available frameworks after proceeding through exercises in the book. Russell takes you through exercises step by step and provides all the instruction necessary to guide you along. The book is hands-on heavy and you do have to be willing to play along with goodly chunks of code. This is not a light browsable book. Although it does allow for some skipping about, it is the sort of volume that engages with the reader/practitioner and naturally leads you along. The exercises do assume some degree of background with data analysis. Although expressed in as simple a language as possible, certainly when getting into natural language processing for example, some background in general terms, processes and methodologies in NLP are expected. However, this should not put anyone off and I was impressed that this formed a quick and useful introduction to the craft because Russell does such a superb job of rooting the theory in everyday language and practice.

I was similarly impressed with the authors inclusion of exercises in mining your own mailbox for useful analysis and how to conduct natural language processing on blog feeds, not just sticking with the sexier and trendier flashy social media tools today. This is a very comprehensive and thoughtful approach to the avalanche of explorable data available from our social existence and Russell provides an extremely approachable and superbly crafted volume. For anyone interested in stepping beyond simple participation and taking a thoughtful view of how social media is changing our lives, this is the book of reference. What’s more I really just simply enjoyed this book. Highly recommended!

a

The consistently thought-provoking Chris Brogan explores how current digital tools provide for greater freedom in employment and life in general. Chris makes some prescient referrals to technologies and ponders why one should focus on being more mobile or consider being more nomadic. His post explores the equation from the perspective of the nomad. I wonder what the perspective is from the other side — from those that would consider the nomad’s services. He raises the critical question about data security and I wonder if this doesn’t extend to a larger question of trust. I have only rarely been on the nomad employing side of the equation, but even by appreciating the nomadic perspective, I am challenged to feel comfortable with the nomad. It’s not really about the results — or about my level of trust. I agree with Chris and with Mark Harrison who affirmed that the nomad should be paid for delivering results. What concerns me is the breadth of digital relationships.
I have always shied away from contractual arrangements personally. I believe (perhaps naively) that if a relationship works for both parties, then it will be perpetuated. Maybe that’s overly Utopian thinking. Nonetheless, it does boil down to a trust issue and in a complex world, trust has to be measurable, demonstrable and to extend outside of the immediate relationship.
Increasingly, digital transactions have attempted to commodify trust. eBay is a good example. The traditional auction, mediated only by the slightly familiar auctioneer has become more information rich based on transparent historical transaction data and more visible reputation. This has theoretically allowed for more informed transactions. The visible social graph beyond eBay has extended this into the employment sphere. The traditional (perhaps mythical) one-on-one employment relationship has become more rare. Employers often must account for their business actions to investors, vcs, outside auditors and the like. Thus, while I may have implicit trust in your ability to deliver results, that trust cannot be tangibly extended/conveyed to third parties. The current inability to commodify this trust gets in the way of digitally nomadic employment schemes. I am not arguing that they are not potentially far more efficient for both parties, only that we don;t have a means to adapt them to employment infrastructures that remain traditional.
I guess this is really starting to range into a discussion of social capital. I wonder whether this is something that OpenSocial has any promise of addressing more immediately. I think it’s quite possible that an informal measure of social capitalization (I use that phrase deliberately) can be garnered through casual observation of wall posts in Facebook or posted application responses.
Wouldn’t it be cool if a digital scoreboard existed that allowed for a more granular demonstration of achievements that provided a measurability of personal trust and professional competence. Wow…that’s way too Orwellian.

a

I have a few friends on Facebook. Last week at the Social Network/ing Conference, I was reminded that the Many Eyes application has a Facebook application that quickly grabs your social network and allows you to paste it into Many Eyes to get a quick visualization of your social network. I finally got around to trying mine.

This is actually rather cool. Facebook doesn’t make the network graphically visible. If you think about it a bit, read the friends lists of your friends you can get that extra degree of connection. This graphic though makes it immediately and visibly clear. I have to imagine myself at the centre of this group, but it is obvious that I am tied in through association to 3 clear networks with two outliers. If you click on the embedded Many Eyes image above you can go to the site and play with this nodes and stretch the edges. The three groups are multimedia/communication studies, fencing and soccer. Clearly they have interconnections. Some of the connections are surprising…as are some of the connections not made ;-) It’s all about the social.

The next dimension here would be to get a feel for the heat of these edges. If we could take the wall postings or messages between nodes and add this as an edge strength visualization it would give a better indication of either which of these networks tend to make use of social networking technology or of realistically which groups I might be more involved with. In all, intriguing and worth some pondering on my part.

For the Facebook users out there…here’s a quick and fun way to do a little social network analysis!

a

What a treat! I had had the honour of meeting and spending the last two days chatting with Fernanda Viégas from the Visual Communications Lab. Her work has been and continues to be inspirational for me personally and to the information visualisation community more substantially. She presented a tantalizing talk at the Social Network/ing conference at OISE/UofT. ‘Visualizing and Analyzing Social Networks’ quickly demonstrated a small facet of Many Eyes to a new audience and gave us a sneak preview of a new tool soon to be available through ManyEyes called PivotGraph. The logic of the PivotGraph is one of those ah-ha moments — it makes all the sense in the world, but leave it to Fernanda and Martin Wattenberg to visualize the problem, and come up with a brilliant way to solve it. Consider that social networks have traditionally been visualized in two ways: the node-link map and the matrix. The common to node-link method is very intuitive, but also becomes quickly cluttered and loses visualization value as the scale of the network being mapped grows. The second is the representative matrix, which scales very well, but sacrifices intuition for clarity. Realizing that there had to be a way of combining the strengths and minimizing the weaknesses, the PivotGraph hybridize these two forms using a collapsible node-link metaphor that, interactively aggregates like nodes and allows for focus on individual vectors. It’s nothing short of amazing to see in action! As always, the carefully visualized transition animations of the Visual Communications Lab make the products particularly visually and experientially impressive. I feel compelled to note that this is not to detract from the intrinsic analytical power of these applications, or to suggest visual superficialities, but instead to observe and emphasize the importance of experience within a visual analysis product. This is something that is reinforced throughout the infoviz community in particular the powerful effect of tweening and time lapse in Hans Rosling’s gapMinder to appreciate change over time.

Fernanda also gave a quick user story that demonstrates a Visual Communication Lab success. One of their early users (two days after they went live) posted a dataset and visualization of name collocation from the New Testament.

He was quite taken with the results and reported these on his blog with accompanying screen shots. He was shortly overwhelmed with hits and story picked up on the traditional media. In fact, there was even a video uploaded to YouTube demonstrating a another user’s interaction with the visualisation. Fernanda emphasized that this was not because of the novelty of the tool or its product, but because of the content and result of the visualization itself. She stresses, and I can certainly attest to this myself, that we are truly starting to see the tremendous value of information visualization when we make new realizations because we are able to see the data in new ways. When we are able not just to present findings in the most clear and effective manner, but when we are able to use the visual tools for analysis. ManyEyes is pushing the envelope of PopViz!

More information on the PivotGraph is available at the Visual Communication Lab site — and keep your eyes on the ManyEyes toolset for availability — I sense that this is a tool that is crossing an interesting visual boundary and will shift the way in which we consider the object being represented as much as how we choose to represent it.

I apologize to Fernanda for my rather gushing post about her talk, as she is very earnestly seeking constructive criticism of ManyEyes — but I do promise to share thoughts on the effectiveness of PivotGraphs when I get my chance to play with it online. As for ManyEyes current offerings — they simply beg us to come up with data to visualize and if you haven’t already, I hope that you will take an opportunity to enjoy some cool data visualization with any sets that you happen to have kicking around.

Update: BTW, Fernanda, as I mentioned in a previous entry, blogged a number of the talks from InfoVis this year. Her recap of her own panel is well worth a look.

a

Avi Goldfarb presented a fast, concise and effective discussion of what conclusions could be drawn about multi-institutional collaboration between US universities during the era of BitNET adoption, 1981 — 1990. A bit of internet history, my ears perked up immediately. His more general framing question: How do changes in collabouration cost change how we produce knowledge.
His study examined 270 institutions as they connected to the BiTNET during this period and cross-indexed this with the number of coauthored journal articles subsequently produced. Goldfarb’s paper ‘Restructuring Research: Communication Costs and the Democratization of University Innovation’ concludes that collaboration was enhanced, but that the gain to institutions was not uniformly realized and physical distance between collabourators remained a factor.Goldfarb’s approach top answering these questions was simple: compare pair-quaility of the research collabourations to determine whether BiTnet encouraged or improved lateral collabouration (between 1st Tier schools) or whether it enabled vertical collabouration between first tier and lesser schools. In other words, did technology adoption allow allow lesser schools a way into the system.
Goldfarb’s study demonstrated that there was a 40% increase in collabourative publishing amongst first tier schools (those receiving the most NSF prior to this period). However, amongst middle tier schools, the increase was more than 130%, thus suggesting that the benefits to publishing were disproportionately realized. The study systematically explored publication data and found that the number of publications remained constant, but there was a significant shift from single authored to multi-authored articles (the author cannot control for other factors that might have necessitated or encouraged this shift). One of the beauties of this data was that the date of connection to BiTnet is known and can be immediately correlated to the publishing history. Goldfarb further indexed the date by measuring the quality of the school by amount of research funding received and evaluated the physical distance between collabourating researchers.
Clearly Bitnet did facilitate collaboration. When both author’s schools have bitnet, there is a positive and significant statistical relationship. But, does bitent amplify existing collabourations, or create new ones? Ultimately the simple answer is that it increased the amount of existing collabouration especially middle tier schools. However, is BiTNet-enabled collabouration a substitute for face-to-face collabouration? The answer to this question is a qualified no. The collabourations tended towards collocated institutions.

a

Despite technical difficulties (presenter’s worst nightmare — LCD projector bulb burnout), Steve Easterbrook demonstrated the usefulness of comparing software structures to social networks of developers to measure operational effectiveness. His well argued and logical presentation ‘Increasing Shared Understanding in Software Teams through Informal Knowledge Transfer Networks’ extended Conway’s Law to social network analysis. This technique of measuring socio-technical congruence is especially valuable in larger scale development projects, where it is probably less obvious about whether a development process is functioning effectivelly. By mining the data rich environment of communication and revision logs, it is possible to generate a social network map of developer interaction that can be connected to a software development schematic to determine Socio-Technical congruence.
Conway’s Law posits that “the structure of a software system will reflect the structure of the organization that builds it”. Today’s best practices for object oriented development emphasizes information hiding and effective modules that demonstrate minimized coupling (information flow between modules) and maximal cohesion (the amount of communication within a module).
Although software engineers typically feel that this modularity is important because it reflects a logical view of how the software works, Easterbrook suggest that it is the social structure of the team itself that forces modularity. Because software continually evolves as features are added, correctives applied and preventative maintenance undertaken, there are many pressures on the system and the largest cost and effort is handling this evolution. With modularity, structure can represent an assignment of responsibility and the most effective way to measure this is by determining the linkage between the social nature of the development team and the structure of the software system. This congruence can be measured by two means: Arc Mirroring…dependancy at the software level reflecting relationship at the social level…good system Similarly, Node Ties can demonstrate that people working on same module are actually talking to one another as they should. If node ties are weak or distant, then communication inefficiencies can be identified and rectified.This analysis can present a very clear picture of how well the team is working. If modules are interdependant and people are talking to one another, then development is efficient.

Effective development is enhanced by distributed cognition and conservation of knowledge. Nodes within the social graph should minimize the amount of knowledge transfer outside of the immediate sphere of involvement to operate most effectively.

To explore the effectiveness of measuring congruence, Easterbrook studied the email logs of open source development. Using SNA techniques it was possible to systematically determine who specific bugs should be referred to and who is the expert on a particular module.
One of the exciting possibilities that Easterbrook raises for this technique of socio-technical congruence is being able to extend it from the realm of the development team to being able to measure congruence with the structure of the customer organization itself to measure software system effectiveness.

a

Social Network/ing Week at the University of Toronto kicked off tonight with a fascinating keynote by Cornell’s Jon Kleinberg. ‘The Geography of Social and Information Networks,’ was one of the most fascinating applied mathematical lectures I can say to having ever attended (and before I go too far I will stress that the math was made very, very approachable for a layperson such as myself). His introducer indicated that he invented algorithmic sociology and although this sounded rather presumptuous (an Al Gore and the Internet sort of thing?), I can’t help but be quite willing to give this some credence after listening to this presentation.
Kleinberg opened with a quote from Jim Gray, that “the emergence of cyberspace and the world wide web was like the discovery of a new continent.” Kleinberg was quite deliberate in this juxtaposition of the geographic with the technological and he then teased this into a further merge with the social. But he questioned whether maps are actually an appropriate metaphor for something as aphysical as social networks — but chose to let this stand on the need to have some common vocabulary with which to be able to relate.

Kleinberg considered the evolution from Milgram’s Small World Experiment of 1968, through the more recent Watts and Strogatz’s work with the MSN network which attempted a more algorithmic approach, but largely substantiated the earlier work. Kleinberg’s own work with LiveJournal attempted to introduce less randomness to the hypothesis and move away from simple closeness to a ranked approach that has applications to the spread of epidemics.

The real meat of the talk was in the conclusion (and I think that this has now been thrown out there for the rest of the conference to further discuss). No one is questioning that social networks exist and have existed throughout time. Nor are we questioning that technologies have enabled new and potentially more effective forms of communication. What is really quite stunning, and paradigm-shifting is that these networks are now made visible. He quoted himself from an interview in the Globe and Mail in which he claimed that “MySpace is awkward because it makes visible the invisible. It makes public what should be private. It doesn’t just create social networks, it anatomizes them and spreads them out like a digestive track on the autopsy table so that you can see what is connected to what and who is connected to whom.” And this is where I suddenly clued in. The network isn’t new, but the fact that it is now made visible and that this can then have impact on how it is constructed and functions is radically new. The network then becomes the object and self-reflective. Soon he posits, software will know more about you then you know about yourself and not only does this demand a need for better computational models to analyse the network, but more importantly we will learn a lot more about large populations and about individual behaviour raising huge concerns about institutionalized lack of privacy.

a

One of the more intriguing social networking applications that I have been enjoying over the last year has been Plazes.com. I blogged about my initial experiences with this spatial addition to the social sphere. Plazes uses your cyberspace IP to place you in physical space. If you are at a previously defined Plaze, then you are pinpointed. If you have discovered a new place, you supply some info about the place, refine the location and it is stored for future reference. You can discover if there are other plazers in your nearby space or plazes that have been recommended and you can also get a Traze (a spatial and temporal indication of where you have been over time). You can also use your mobile phone to plaze yourself or to find nearby plazes. The system works, is a hoot to use and you can even provide a little map to your blog readers showing where you are in real time –-> see my own sidebar.

Last week, the Plazes folks opened up their final beta testing on a new version that will shortly appear. With the new version, not only can you identify your plaze to the world, you can also share information about what you are doing there– ala Twitter. Additionally you can add past plazes to your locations and indicate where you will be in the future. All of this is accomplished with little effort on your part and takes the social promise to a new level. The service is designed for arranging meet-ups, indicating wifi availability or a good restaurant amongst a circle of trust. It also feeds the current fascination with sharing your current emotions, locations, thoughts etc. with your friends and the world at large ( eg. Twitter, Facebook).

The spatial dimension fascinates me. Wouldn’t it be cool to have a real time moving map plazing your acquaintances with respect to your current plaze (this all presuming that you have friends that use Plazes). Imagine if you will a scalable map of your neighbourhood/world in which your friends are all pinpointed with a small bright object and their name or picture, all moving around like a radar display…oh the pursuit of omniscience (I guess it’s a matter of time before we all have embedded RFID ;-) I like the new Plaze and it opens up some new avenues to how I would consider benefiting from Plazes.

Hearkening to Kevin Lynch’s mental maps of personal perception of space, the ability to use Trazes (past record of where you have been, how long you have been there and what you have been doing) to learn more about your habits and how you arrange your time and define your space. Privacy factors aside and I admit that there are many, Plazes allows for learning as much about yourself as your friends.

ps. Had to also see how effectively this little ‘compose in Word and post directly to your blog’ function actually works. Nearly flawless as it turns out…just missed the more tag that I inserted…can probably escape that out to make it work.

a

My attention was drawn to a new Fortune Magazine initiative called the Corporate Org Chart Wiki. It bills itself as in early beta and clearly experimental. It claims to seek to ‘tap the collective knowledge’ of the community and to collect and share enterprise organizational charts. Its collaborativity certainly marks it as a wiki. Unfortunately it seems overly open to the abuse that has been associated with many of the public wikis existent today. There’s no authentication, nor any sort of transparent versioning that I can find. Its a nice little flash app and it functions efficiently. It allows a user to draw relationships and add nodes visually and relatively intuitively. It allows an observer to gain a quick appreciation of the organizational structure.
However, I wonder what the objective of this exercise is? If it is to do what it claims to be attempting, i.e. provide a publicly accessible repository of org charts for publicly traded companies, I wonder if there are not better means. I am reminded of LinkedIN, which attempts to basically accomplish a similar thing, although in a less visual manner. LinkedIN however, ensures a substantial data integrity through some form of authentication that you are who you say you are. It also leverages the fact that everyone is part of some web of interaction, whether business, personal, academic or otherwise. While I applaud fortune for the graphical approach to visualization of the org chart data that they provide, the low level of authentication seems to be a huge issue. The apparent failure of the experiment seems to be all to evident for all to see. I note on cursory examination that someone has made themselves CEO of Apple. I wonder if they can trace that self-promotion?
I wonder what the actual object of this experiment is? Maybe its about something less transparent that what it appears to be? It is an interesting foray into visual social network analysis, but I wonder whether it would really be rather more effective to screen scrape corporate sites to create such charts and perhaps allow modification as a means of verification. Interestingly, the visuals and the principle itself remind me of GENI which allows for interactive genealogy and which I blogged on earlier this year.
So I am left wondering why Fortune magazine has gotten into digital interactivity research. Don’t misunderstand me, its rather cool that they have. But I question what the research question is here. If it was merely to start to build a repository of corporate org charts, which are super fluid by nature, there is definitely more effective means to accomplish this and to ensure that these actually have some factual truth.

a