This is the first in a series of posts about technology trends and seeing them in action in the US. Here, Andy Beale, Director of Common Technology Services within the Office of the CTO, talks about big data and analytics.
In 2012 we published the digital strategy which set out how government would become digital by default. This strategy was partly informed by a visit to Silicon Valley to learn from the digital innovators there.
Last month, we went back. We wanted to look beyond transactions and publishing to see, first hand, how organisations are changing the way they operate by harnessing the value of their data. And how this can be applied in government. For example, how they are using APIs to get access to data and, looking behind the hype around “Big Data”, to see how that data can be better used.
We saw many companies but this list should give you a flavour:
For a great overview read Kevin Cunnington’s recent blog post on the trip.
These companies cover almost the full stack of technology in this area. We started in the middle with Revolution Analytics.
Revolution helps companies exploit the R language extensively used in predictive analytics. They describe themselves as the “Red Hat of the R community”. Dave Rich, their CEO, talked about “getting insights to action” - which is the the point of having this data after all. He suggests you start by asking yourself the following questions:
- What? (What do we want to know?)
- So what? (What does it mean?)
- Now what? (What decisions can we make now we’ve got it?)
Dave went on to illustrate what the platform or stack of technology looks like to make this work:
The illustration shows the four layers of the stack. Here's an explanation of each layer:
- The ETL (extract, transform, load) and external sources boxes: we need to get our data and take external data in
- The database box: we need somewhere to put it
- The analytics box: we need something to analyse it
- The end user consumption box: we need tools for users to work with and visualise the data
This is what the companies listed above do: Basho, Platfora, Cloudera and Pivotal in the database and storage layer and Apigee in the “sources” box. Tableau was also referenced frequently as a company working in the “end user consumption” layer (though we didn’t meet them).
The future of government IT
For me, the trip overall reinforced that we should be doing more of this kind of thing - technology that can have far greater impact on a business, rather than (and I am obviously generalising) continuing to see IT in terms of machines, wires, cost and risk. This is the future of government IT. Why? Because access to data and analytics help make better decisions. Better decisions mean better policy making, better operational effectiveness and better services for users.
In later blogs we will talk about other aspects of the trip and what these developments might mean for government technology.
In the meantime, what do you think about big data: marketing hype or a new technology era?