In his third and final post of this series inspired by a visit to Silicon Valley in the U.S., Paul Maltby, Director of Open Data and Government Innovation in the Cabinet Office, envisions a data revolution and what that might mean in the future for government.
In my previous post I set out how some larger tech companies were leveraging the power of their data by making it discoverable and accessible internally across their business. Of course Government is different to a new tech company in Silicon Valley, but are there lessons to be learned?
Much of the data governments hold is acquired through the force of law, not consumer choice, and there are good reasons why there are strong legal constraints on our ability to share data across government Departments - few citizens would want their mental health records or personal tax history made easily available across swathes of the civil service.
Yet a huge proportion of the data within government is nowhere near as sensitive as this. The UK’s leading work on open data has opened up more than 13,000 datasets to the public via data.gov.uk, and our recent inventories show where there are other data sources that have not (yet) been made available. This work means that for the first time civil servants can, like any other member of the public, easily search and gain access to data from other government departments.
Sharing data
Even where data is too sensitive to be made open in its raw form, there are sometimes arrangements that legally enable data to be shared with other analytical teams across government. These often take the form of bilateral arrangements and sometimes they require separate legal arrangements, or ‘gateways’.
However, our current legislative framework is a confusing mess with many anomalies and significant bureaucratic hurdles; for example Local Authorities can share Housing Benefit data with the Department for Work and Pensions and can use it to investigate and pursue Housing Benefit fraud, but they are not able to share the same data with their own internal tenancy teams in order to detect and investigate tenancy fraud.
There is work underway in the Cabinet Office to collaborate with civil society organisations to understand how legislative barriers to sharing might be simplified in the future to help enable more sophisticated research and statistical work within government. And possibly to enable services that are more tailored around the individual, as we have seen in recent work on Troubled Families.
Frictionless internal data?
Our progress on open data in the UK is world leading – but the type of frictionless internal data system we saw in Silicon Valley, even for non-sensitive data, seems a long way off.
But imagine a future where the current work of the CTOs across government to untangle government data from restrictive contracts and locked-down proprietary systems was successful. Data would be increasingly held in the cloud, its physical location no longer the restricting primary criteria for deciding who had access.
Imagine if the hard slog endured by Amazon and other tech companies to create common dictionaries for data was used across government, and if the core quality of the data we hold was seen as an important objective so there were fewer erroneous records.
Imagine if, whether through APIs or an internal version of data.gov.uk, civil servants could call up the non-sensitive data from teams and services across government – perhaps in real time – and start to use the data to help form policy proposals, improvements to our existing processes, or even to create new products and services.
We are perhaps closer than we think. One of the most forward-looking government agencies, the DVLA, recently announced an API service available to insurers to check previous convictions. And in HMRC there have already been moves by the CTO to move towards the use of open source Hadoop-based cloud.
Next-generation skills
But to realise the opportunity of this data-rich future, there is a need to develop the next-generation analytical skills within government using tools such as Python and R, and a more widespread familiarity with lower-level coding skills that mean a wide range of civil servants are comfortable starting to hack around with data.
There are indications that the desire to embrace this new world within government is strong. A joint team from my Government Innovation Group, the Government Digital Service and Government Office for Science are leading a project on Data Science. In addition to producing a number of early alpha projects, there is keen interest to embrace these reforms across Whitehall, including from the Policy and Analytical professions.In another example, in the Cabinet Office our first go at a Code Club for civil servants saw 60 available places run out in just 30 minutes after registration opened, with the experiment likely to be repeated and extended.
Choices and challenges
In Silicon Valley we peeked into the tech crystal ball for how government could be run in the next five to ten years. Doing things in this way is obviously a choice, but it is one that successful organisations in other fields, and indeed in other countries, are already starting to embrace from the White House to Etalab in France. A data confident Government would be different – perhaps more permeable, hopefully seriously more efficient, more able to customise services and anticipate citizen needs. The civil service itself would need to be differently skilled, more appreciative of the potential power of data, and with a greater tolerance of a more experimental and creative attitude to working with data.
Of course it almost goes without saying that there are plenty of practical challenges before we’ve even considered public and political appetite or the ethical and policy consequences that will need to be worked through before anything like this comes into being at scale. Government analytical work has a long and noble history, but the type of rapid data-led decision making we saw first-hand in Silicon Valley is likely to prove challenging to any remaining civil servants that model themselves on Sir Humphrey.
Some of the activity required to bring about this change is already underway, and some is just beginning. Some, such as creating common taxonomies and dictionaries, will require a hard grind and an expectation that benefits would accrue to others through the following decade.
Yet this vision of a future data confident government could be well worth grasping, and indeed it may become one of the leading contenders for how we can reduce costs and improve the citizen’s experience of public services in years to come.
Follow Paul on Twitter and don't forget to subscribe to the Government Technology blog.