This is the first in a series of posts about technology trends and seeing them in action in the US. Here, Andy Beale, Director of Common Technology Services within the Office of the CTO, talks about big data and analytics.
In 2012 we published the digital strategy which set out how government would become digital by default. This strategy was partly informed by a visit to Silicon Valley to learn from the digital innovators there.
Last month, we went back. We wanted to look beyond transactions and publishing to see, first hand, how organisations are changing the way they operate by harnessing the value of their data. And how this can be applied in government. For example, how they are using APIs to get access to data and, looking behind the hype around “Big Data”, to see how that data can be better used.
We saw many companies but this list should give you a flavour:
For a great overview read Kevin Cunnington’s recent blog post on the trip.
Technology stack
These companies cover almost the full stack of technology in this area. We started in the middle with Revolution Analytics.
Revolution helps companies exploit the R language extensively used in predictive analytics. They describe themselves as the “Red Hat of the R community”. Dave Rich, their CEO, talked about “getting insights to action” - which is the the point of having this data after all. He suggests you start by asking yourself the following questions:
- What? (What do we want to know?)
- So what? (What does it mean?)
- Now what? (What decisions can we make now we’ve got it?)
Dave went on to illustrate what the platform or stack of technology looks like to make this work:
The illustration shows the four layers of the stack. Here's an explanation of each layer:
- The ETL (extract, transform, load) and external sources boxes: we need to get our data and take external data in
- The database box: we need somewhere to put it
- The analytics box: we need something to analyse it
- The end user consumption box: we need tools for users to work with and visualise the data
This is what the companies listed above do: Basho, Platfora, Cloudera and Pivotal in the database and storage layer and Apigee in the “sources” box. Tableau was also referenced frequently as a company working in the “end user consumption” layer (though we didn’t meet them).
The future of government IT
For me, the trip overall reinforced that we should be doing more of this kind of thing - technology that can have far greater impact on a business, rather than (and I am obviously generalising) continuing to see IT in terms of machines, wires, cost and risk. This is the future of government IT. Why? Because access to data and analytics help make better decisions. Better decisions mean better policy making, better operational effectiveness and better services for users.
In later blogs we will talk about other aspects of the trip and what these developments might mean for government technology.
In the meantime, what do you think about big data: marketing hype or a new technology era?
2 comments
Comment by Tieracker posted on
New Technology Era, definitely!!!!
Comment by simonfj posted on
Well "Big Data" is certainly marketing hype; a bit like "the cloud". It just addresses a different market; the buyers of Big Iron.
Seems a little strange though Andy, especially considering your background with the Guardian. Firstly there's no mention of the C in ICT, so that's one half of the equation that's often left out. The other paradigm is the (media) convergence of networks - particularly broadcast and interactive. Now we can talk back to the box (TV, radio), if only we were directed to an online place where we could get/give an education.
As for "... access to data and analytics help make better decisions". We all understand the belief system. But in reality you're talking about some policy maker, who may have little background in a particular field, running an inquiry for a short time, and coming up with a few immediate responses from which policy might be fashioned, in an isolated department.
Yes big data/analytics has a place. But let's balance this with improving the communications between inter-institutional peers, inside and outside government departments. i.e. share not deliver. Otherwise we are likely to be (as my librarian friends say) "buried in the stacks". (As if we are not already)
I suppose it's this idea of "an end user consumption box" which leads us astray. What's the phrase? "Data so simple even a Minister can understand it".