by moleitau (cc)

It has taken me a couple of days to write this post, because my brain is still crunching on its contents. It touches on so many different areas of the technology and business areas that I am passionate about, that I’ve had to give up covering them all in one post.

The historic destiny of data (and it’s big parent, knowledge) has been to be locked up; the constant struggle has been to set it free. From the formation of  the university to the API-rich Web 2.0 world, people have postulated ways to make it more available. Of course, not all data is going to be ‘free’ (as in beer), but much of it should at least be accessible and usable (‘free’ as in freedom). The trouble is that much of today’s digital information is trapped in non-portable and hard to process formats.

It’s a vision at the heart of the semantic web, championed by Sir Tim Berners-Lee – particularly under the banner of Web Science, down the M3 from where I am writing, at the University of Southampton (this TED video is a useful primer).

Time to join some dots. Prime Minister Gordon Brown appointed Sir Tim to work on opening up government data, and last month Sir Tim talked to the Cabinet about a goal of delivering a single online access point to Government information, similar to the one introduced by the Obama administration in the US, and to what the Australian government has been doing.

Back in August I interrupted the weekend to visit Google’s London office and see a bunch of young folks, with a bit of help from some not-so-young ones, hacking together web-based services under the banner of Young Rewired State. It really was inspiring stuff. There were some familiar faces (including @grantbell) and plenty of new ones, all working on an impressive list of apps, adding to the list of Rewired state projects.

At the end of the weekend, an impressive list of judges (@craigelder @jobsworth @BenHammersley @helenmilner @marxculture and @danielheaf) judged the apps, and there are some good blog posts on what went off, as well as my usual collection of photos:

As a side note, I like the idea of hack days. Development sprints where you focus on one thing are a great thing – every business should have “hack days” – pick some key problems then get everyone together to spend a day working on fixing them. Hack days aren’t just for coders.

Back to that data, or more specifically the mountains of anonymised data that the UK government collects every year. Putting aside the usual “we paid to collect it, so we should have it” argument, so often espoused, there is a much better reason for putting all of that data “out there” – or there is to me at least. There are a thousand creative and useful things that could be done with it, most of which fall under the umbrella of  micro-business or hobbyist. These are things that the government, and most businesses, could never justify funding, but which a sea of enthusiastic developers could make happen – tackling all sorts of problems along the way.

Enter data.gov.uk. A surprise email last week meant, literally a few hours later, I was huddled around a projector looking at one of the most exciting things I’ve seen on the Internet in a very long-time. A very long time. With a very exciting team of people too. The previous day, via the Digital Engagement blog, the Cabinet Office issued a call for help:

From today we are inviting developers to show government how to get the future public data site right – how to find and use public sector information.

The developer community through initiatives such as Show Us a Better Way, the Power of Information Taskforce, MySociety and Rewired State have consistently demonstrated their eagerness and abilities to “Code a Better Country“.  You have given us evidence and examples to help drive this forward within government.

The new site will provide a way to access (eventually and hopefully) most of the UK government’s published data. There are already over a thousand datasets, in differing levels of accessibility from CSVs, to SPARQL end points, as well as XML and JSON. If those sound like alphabet soup, don’t worry, just be as excited as the developers who know what they mean.

The potential result is a wave of new applications, based on government data, that could do a wealth of things, from relating performance with class sizes in your local school to understanding how your local farming community is faring. Data.gov.uk is a very non-trivial project, and there is a long way to go, but what I was a very promising start. The early developer community is already very active, even though the site won’t really be in beta until the end of the year. As Harry Metcalfe puts it, the wraps are off.

There are four aims behind the project and opening up the data: 1) Aid transparency and accountability. 2) Empower citizens to drive public service reform. 3) Unlock the social and economic value in the data. 4) Stimulate the UK’s digital economy, with regard to technology and research in the web domain.

Certainly there are big commercial uses of the data, and I am sure big business will muscle in on the act, but underneath that, what a great opportunity to give a new generation of digital talent something to dig their teeth into. My hope is that the data will provide a platform for an ecosystem of businesses and micro-businesses, as well as non-profit organisations, to create value for UK Plc, both for the public good and for economic good. It will also be a proving ground for a new generation of geeks who can work with massive datasets and produce insights from them. Exactly the kind of folks the knowledge-based business of the future will need.

Related Posts

• Open Data Opens Up Gov
• Tim Berners-Lee @ The Innovation Edge
• Gordon Brown Announces “Second Generation” Government
• Future of The Web – Part I – A History

My first encounter with Sir Tim’s work was back in 1990. I was at Kent University at the time, and got involved in the trial of something called “Hypertext”. Quite frankly, I didn’t get it. A page of text where you could click on a word and another page of text would come up. What was anyone going to do with something like that? I took the pay and went back to circuit boards and DJing at the college radio station. Two years later I was back working on Internet technologies. The experience has made me much more thoughtful when I encounter new technologies!

As Sir Tim recounted the foundations of the web, it was curious how accidental and casual the whole thing sounded. The project was a ‘back-room’ effort, carried out during a lull in the work on the Cern accelerator. As Tim put it, giving staff a long leash, giving them space, is where innovation comes from. It brought to mind Google’s 20% time. You could hear the pens of the managing directors in the audience scribbling furiously.

The point of innovation is that you don’t know the end product before you start, sometimes you don’t even know the problem. It is a big risk, and that makes it tricky for a traditional management mindset. The web (or rather hypertext) was driven by the challenge of dealing with lots of documents and having to get others up to speed on them quickly. Necessity is the mother of Invention, which is a theme that came back later during the conference.

He described blogs as a social machine, which is an interesting metaphor – I guess we are still cogs in a machine, even in the world of social media. I have seen what this social machine can create form the huge diversity that it brings together. For me, two stand out quotes from the talk:

“The people doing the really interesting things tended to fall between two stools”

“The web really has to be thought of not as a system of connections between computers, or even as links between web pages, but really as humanity connected.”

The first made me think of the Medici effect, and is the reason for The Web Science Initiative, which sounds very interesting. The second was the final take away for me: The future of the web, and of innovation, is in individuals working in collaboration. There again, that is how all the greatest achievements have come about.

Related Posts

• Set our Data Free and Create a (Digital) Economy
• Future of The Web – Part I – A History
• Future of The Web – Part II – The Future
• Communicating Feelings on the Future of the Web
• Gordon Brown at NESTA – The Innovation Edge