by moleitau (cc)

It has taken me a couple of days to write this post, because my brain is still crunching on its contents. It touches on so many different areas of the technology and business areas that I am passionate about, that I’ve had to give up covering them all in one post.

The historic destiny of data (and it’s big parent, knowledge) has been to be locked up; the constant struggle has been to set it free. From the formation of  the university to the API-rich Web 2.0 world, people have postulated ways to make it more available. Of course, not all data is going to be ‘free’ (as in beer), but much of it should at least be accessible and usable (‘free’ as in freedom). The trouble is that much of today’s digital information is trapped in non-portable and hard to process formats.

It’s a vision at the heart of the semantic web, championed by Sir Tim Berners-Lee – particularly under the banner of Web Science, down the M3 from where I am writing, at the University of Southampton (this TED video is a useful primer).

Time to join some dots. Prime Minister Gordon Brown appointed Sir Tim to work on opening up government data, and last month Sir Tim talked to the Cabinet about a goal of delivering a single online access point to Government information, similar to the one introduced by the Obama administration in the US, and to what the Australian government has been doing.

Back in August I interrupted the weekend to visit Google’s London office and see a bunch of young folks, with a bit of help from some not-so-young ones, hacking together web-based services under the banner of Young Rewired State. It really was inspiring stuff. There were some familiar faces (including @grantbell) and plenty of new ones, all working on an impressive list of apps, adding to the list of Rewired state projects.

At the end of the weekend, an impressive list of judges (@craigelder @jobsworth @BenHammersley @helenmilner @marxculture and @danielheaf) judged the apps, and there are some good blog posts on what went off, as well as my usual collection of photos:

As a side note, I like the idea of hack days. Development sprints where you focus on one thing are a great thing – every business should have “hack days” – pick some key problems then get everyone together to spend a day working on fixing them. Hack days aren’t just for coders.

Back to that data, or more specifically the mountains of anonymised data that the UK government collects every year. Putting aside the usual “we paid to collect it, so we should have it” argument, so often espoused, there is a much better reason for putting all of that data “out there” – or there is to me at least. There are a thousand creative and useful things that could be done with it, most of which fall under the umbrella of  micro-business or hobbyist. These are things that the government, and most businesses, could never justify funding, but which a sea of enthusiastic developers could make happen – tackling all sorts of problems along the way.

Enter data.gov.uk. A surprise email last week meant, literally a few hours later, I was huddled around a projector looking at one of the most exciting things I’ve seen on the Internet in a very long-time. A very long time. With a very exciting team of people too. The previous day, via the Digital Engagement blog, the Cabinet Office issued a call for help:

From today we are inviting developers to show government how to get the future public data site right – how to find and use public sector information.

The developer community through initiatives such as Show Us a Better Way, the Power of Information Taskforce, MySociety and Rewired State have consistently demonstrated their eagerness and abilities to “Code a Better Country“.  You have given us evidence and examples to help drive this forward within government.

The new site will provide a way to access (eventually and hopefully) most of the UK government’s published data. There are already over a thousand datasets, in differing levels of accessibility from CSVs, to SPARQL end points, as well as XML and JSON. If those sound like alphabet soup, don’t worry, just be as excited as the developers who know what they mean.

The potential result is a wave of new applications, based on government data, that could do a wealth of things, from relating performance with class sizes in your local school to understanding how your local farming community is faring. Data.gov.uk is a very non-trivial project, and there is a long way to go, but what I was a very promising start. The early developer community is already very active, even though the site won’t really be in beta until the end of the year. As Harry Metcalfe puts it, the wraps are off.

There are four aims behind the project and opening up the data: 1) Aid transparency and accountability. 2) Empower citizens to drive public service reform. 3) Unlock the social and economic value in the data. 4) Stimulate the UK’s digital economy, with regard to technology and research in the web domain.

Certainly there are big commercial uses of the data, and I am sure big business will muscle in on the act, but underneath that, what a great opportunity to give a new generation of digital talent something to dig their teeth into. My hope is that the data will provide a platform for an ecosystem of businesses and micro-businesses, as well as non-profit organisations, to create value for UK Plc, both for the public good and for economic good. It will also be a proving ground for a new generation of geeks who can work with massive datasets and produce insights from them. Exactly the kind of folks the knowledge-based business of the future will need.