There’s a small but growing band of government teams around the world dedicated to making experiments happen. The Cabinet Office’s What Works Team, set up in 2013, was the first of its kind. But you’ll now find them in Canada, the US, Finland, Australia, Colombia, and the UAE.
All of these teams work across government to champion the testing and evaluation of new approaches to public service delivery. This blog takes a look at the many ways in which we’re striving to make experimentation the norm in our governments.
Different contexts, common mission
Unsurprisingly we’re all operating in very different contexts. Some teams were set up in response to central requirements for greater experimentation. Take Canada, for instance. In 2016 the Treasury Board directed departments and agencies to devote a fixed proportion of programme funds to “experimenting with new approaches” (building on Prime Minister Trudeau’s earlier instruction to Ministers). An Innovation and Experimentation Team was then set up in the Treasury Board to provide some central support.
Finland’s Experimentation Office, based in the Prime Minister’s Office, is in a similar position. The team supports the delivery of Prime Minister Juha Sipilä’s 2016 national action plan that calls for “a culture of experimentation” in public services and a series of flagship policy experiments.
Others, like the US Office of Evaluation Sciences (OES) and the Behavioural Economics Team of the Australian Government (BETA), grew out of political interest in using behavioural science experiments in public policy. But these teams now run experiments in a much broader set of areas.
What unites us is a focus on helping public servants generate and use new evidence in policy decisions and service delivery.
Enabling experimentation
So how are we going about it? For me, a number of key approaches stand out:
-
Joint trials
For some teams, joint trials with departments have become a popular way of overcoming the barriers preventing more experiments from taking place (e.g. expertise gaps and the fear of failure).
Since its creation in 2015, OES has worked with over a dozen federal agencies on more than 60 projects. They’ve had some impressive results. For example, a trial with the US Department for Education showed that sending personalised text message reminders to low-income students increased rates of college enrolment by 8.6 per cent.
BETA has similarly tackled a range of pressing policy issues since its creation in 2016. In a trial with the Department of Health, for instance, they sought to reduce the over-prescription of antibiotics by GPs. So they tested sending high-prescribing GPs a personalised letter from the Chief Medical Officer with a simple graph comparing their prescribing behaviour to that of their peers. The result? In just six months, antibiotic prescription rates were cut by 12 per cent.
-
Pooling expertise
Leveraging expertise from elsewhere is another big focus of activity. While our own What Works Team advises departments on setting up trials, we are well aware that our ability to match-make is just as important if we want to satisfy demand.
From the outset we’ve sought to ensure that policy teams with limited experience of running trials have access to advice and support from those that do. That’s why we set up a Trial Advice Panel in 2015. It’s a free advisory service that any civil servant can use. It’s made up of 46 experts from across government and academia that have first-hand experience of running high-quality trials.
Teams receive the support they need to run trials in-house and we’ve seen many gain the confidence to design further experiments (with or without our support). We’ve now assisted almost 60 projects.
We’re not alone in doing this. Our US counterparts also make good use of academics. They run a fellowship scheme that brings academic experts into government on secondment to help federal agencies conduct experiments. These fellows work alongside OES staff as they develop new trials.
Finally, the Finnish Experimentation Office seeks support from an even wider pool of experts in civil society. They run Place to Experiment, a digital platform aimed at “transform[ing] ideas into concrete experiments”. Users can pitch ideas and crowdsource the support they need.
-
Open-by-default approach
Our respective teams are also embracing an open-by-default approach to our work. We want to raise the profile of experimentation, demonstrate its value, and build understanding. Some of us have set up websites and blogs to showcase our activities. Others have live-streamed events, published reports, or made use of social media.
Canada’s Experimentation Works initiative offers an especially innovative approach – one that’s well worth keeping tabs on. Their Innovation and Experimentation Team has brought together teams in four departments who are live-blogging the progress of their experiments. The teams are learning from each other and, crucially, their materials and the lessons they’re learning are openly available to other public servants.
Finally, there’s an important trend in teams not only publishing the results of trials they’ve been involved in (regardless of whether they were positive or not) but also in being transparent about the trials they’re currently working on. Both BETA and OES have gone the furthest in publishing details of registered trials. As OES put it: “this is a critical component of conducting transparent, replicable, and high-quality research”.
International learning
Our team now regularly exchanges experiences and ideas with our colleagues overseas. It has been incredibly gratifying to feel part of a growing, global evidence-based policy movement. But more importantly, the significant strides that experimentation teams have taken in recent years have created new opportunities for collaboration and learning about what works from around the world.
Leave a comment