Redcatco » opendata

Open Data Opens Up Gov

Benjamin Ellis — Thu, 21 Jan 2010 08:37:49 +0000

Today sees the launch of data.gov.uk. Over the last few months I’ve had some privileged peaks behind the scenes, and I’m very excited to see it all now live. The front paragraphs on the site put it well:

Advised by Sir Tim Berners-Lee and Professor Nigel Shadbolt and others, government are opening up data for reuse. This site seeks to give a way into the wealth of government data and is under constant development. We want to work with you to make it better.

We’re very aware that there are more people like you outside of government who have the skills and abilities to make wonderful things out of public data. These are our first steps in building a collaborative relationship with you.

A Short Long Journey

It feels like a long journey since October (“Set our Data Free and Create a (Digital) Economy“), but it reality it has only been a dozen weeks and some snow. In between-times, civil servants and some very civil technologists have been hard at work making thousands of sets of data accessible. I’ve just glanced back at my email folder – there are a good few thousand messages that have gone backwards and forwards between a large community of developers and the folks in Whitehall who have been making things happen. Amazing stuff.

It Matters Because…

I tuned into Radio 4′s Today program – normally bastion of great Radio – and was very disappointed to hear an odd piece which implied civil servants were battling to avoid releasing the data, and that the ordinance survey data might not get published. While I am sure that there are some who are, it’s the very opposite of what I have seen.

Why does data.gov.uk matter? It matters because:

Open data encourages transparency in government. I see that as a very-good-thing.
The datasets will stimulate innovation in services – from mapping accident black spots to finding cross-service opportunities.
Data.gov.uk will be a nursery for a new generation of semantic-web software developers. If the community isn’t where the next Google comes from (it might well be!), it will at least nurture a pool of developers who will bring great data processing and visualisation skills to business.
Supporting a digital Britain. The initiative provides a first step in helping to UK catch up and over take countries like Australia and others who are a long way down the track. Knowledge-based services are a big part of the future.

Just the Start

There is a better piece on the BBC website (it might be by Rory Cellan-Jones), talking with Sir Tim Berners-Lee, who has been helping to drive the initiative:

…from traffic statistics to crime figures, for private or commercial use. The target is to kickstart a new wave of services that find novel ways to make use of the information.

The official press release is on the COI site:

“Making public data available for re-use is about increasing accountability and transparency and letting people create new, innovative ways of using it. Government data should be a public resource. By releasing it, we can unlock new ideas for delivering public services, help communities and society work better, and let talented entrepreneurs and engineers create new businesses and services. ” Sir Tim Berners-Lee

Conversations are flowing on Twitter under the #opendata hashtag. Sir Tim and Nigel Shadbolt have also written a longer piece in The Guardian and the data.gov.uk site has a list of apps that have been written using the data already. I suspect we’ll see many more in the coming months.

[a couple of additions, post launch event]

There is now a set of resources listed in a blog post on the data.gov.uk site, including some background on SPARQL (the query language used to access the data) and how to list and search the datasets. The site itself is built using open source software – the main stay of what we work with here – under the action plan announced by Tom Watson last year: Open Source, Open Standards and Re–Use: Government Action Plan. There is a brilliant long post by Paul Clarke on his blog, which provides some good context and outlines the next set of challenges

It was interesting to read the US perspectives on the announcement. The US have their own initiative and there is some controversy about who is furthest ahead! The OPSI posted explaining the licensing terms for the data and how these relate to creative commons in the UK. All in all, a great achievement for Professor Shadbolt and Sir Tim.

Set our Data Free and Create a (Digital) Economy

Benjamin Ellis — Mon, 05 Oct 2009 21:03:22 +0000

by moleitau (cc)

It has taken me a couple of days to write this post, because my brain is still crunching on its contents. It touches on so many different areas of the technology and business areas that I am passionate about, that I’ve had to give up covering them all in one post.

The historic destiny of data (and it’s big parent, knowledge) has been to be locked up; the constant struggle has been to set it free. From the formation of the university to the API-rich Web 2.0 world, people have postulated ways to make it more available. Of course, not all data is going to be ‘free’ (as in beer), but much of it should at least be accessible and usable (‘free’ as in freedom). The trouble is that much of today’s digital information is trapped in non-portable and hard to process formats.

It’s a vision at the heart of the semantic web, championed by Sir Tim Berners-Lee – particularly under the banner of Web Science, down the M3 from where I am writing, at the University of Southampton (this TED video is a useful primer).

Time to join some dots. Prime Minister Gordon Brown appointed Sir Tim to work on opening up government data, and last month Sir Tim talked to the Cabinet about a goal of delivering a single online access point to Government information, similar to the one introduced by the Obama administration in the US, and to what the Australian government has been doing.

Back in August I interrupted the weekend to visit Google’s London office and see a bunch of young folks, with a bit of help from some not-so-young ones, hacking together web-based services under the banner of Young Rewired State. It really was inspiring stuff. There were some familiar faces (including @grantbell) and plenty of new ones, all working on an impressive list of apps, adding to the list of Rewired state projects.

At the end of the weekend, an impressive list of judges (@craigelder @jobsworth @BenHammersley @helenmilner @marxculture and @danielheaf) judged the apps, and there are some good blog posts on what went off, as well as my usual collection of photos:

As a side note, I like the idea of hack days. Development sprints where you focus on one thing are a great thing – every business should have “hack days” – pick some key problems then get everyone together to spend a day working on fixing them. Hack days aren’t just for coders.

Back to that data, or more specifically the mountains of anonymised data that the UK government collects every year. Putting aside the usual “we paid to collect it, so we should have it” argument, so often espoused, there is a much better reason for putting all of that data “out there” – or there is to me at least. There are a thousand creative and useful things that could be done with it, most of which fall under the umbrella of micro-business or hobbyist. These are things that the government, and most businesses, could never justify funding, but which a sea of enthusiastic developers could make happen – tackling all sorts of problems along the way.

Enter data.gov.uk. A surprise email last week meant, literally a few hours later, I was huddled around a projector looking at one of the most exciting things I’ve seen on the Internet in a very long-time. A very long time. With a very exciting team of people too. The previous day, via the Digital Engagement blog, the Cabinet Office issued a call for help:

From today we are inviting developers to show government how to get the future public data site right – how to find and use public sector information.

The developer community through initiatives such as Show Us a Better Way, the Power of Information Taskforce, MySociety and Rewired State have consistently demonstrated their eagerness and abilities to “Code a Better Country“. You have given us evidence and examples to help drive this forward within government.

The new site will provide a way to access (eventually and hopefully) most of the UK government’s published data. There are already over a thousand datasets, in differing levels of accessibility from CSVs, to SPARQL end points, as well as XML and JSON. If those sound like alphabet soup, don’t worry, just be as excited as the developers who know what they mean.

The potential result is a wave of new applications, based on government data, that could do a wealth of things, from relating performance with class sizes in your local school to understanding how your local farming community is faring. Data.gov.uk is a very non-trivial project, and there is a long way to go, but what I was a very promising start. The early developer community is already very active, even though the site won’t really be in beta until the end of the year. As Harry Metcalfe puts it, the wraps are off.

There are four aims behind the project and opening up the data: 1) Aid transparency and accountability. 2) Empower citizens to drive public service reform. 3) Unlock the social and economic value in the data. 4) Stimulate the UK’s digital economy, with regard to technology and research in the web domain.

Certainly there are big commercial uses of the data, and I am sure big business will muscle in on the act, but underneath that, what a great opportunity to give a new generation of digital talent something to dig their teeth into. My hope is that the data will provide a platform for an ecosystem of businesses and micro-businesses, as well as non-profit organisations, to create value for UK Plc, both for the public good and for economic good. It will also be a proving ground for a new generation of geeks who can work with massive datasets and produce insights from them. Exactly the kind of folks the knowledge-based business of the future will need.