Posts Tagged ‘policy making’

Test, Learn, Adapt

Michael Sanders

If there is a phrase more likely to attract rolled eyes than “Behavioural Economics”, it is probably “Evidence based policy” – one has in the past been derided for not really being economics, and the other for being more apt with the first and third words reversed. Supporters of both have often despaired. There is now, however, a glimmer of hope.

The Cabinet Office’s Behavioural Insights Team, charged by the coalition with bringing behavioural science to policymaking, this week launched a paper with the title “Test, Learn Adapt: Developing Public Policy with Randomised Controlled Trials”[1], which is part manifesto for randomised controlled trials (RCT), and half handbook on how to conduct them. Although not comprehensive, its nine steps offer a simple guide to the basics of designing and running a trial.

Co-authored by the Ben Goldacre of the Guardian[2], and Professor David Torgerson, director of the University of York’s Trial Unit, the paper gives every indication that government is, or at the least is trying to be, committed to the idea of running trials wherever possible, often with the help of the academic community – which will hopefully lead to a better understanding of which policies are beneficial and which are not. Although there are other[3], more comprehensive[4] and technical guides[5] to trial design, this is an important step for government.

The second step may be even more difficult – making sure that successful intervention are rolled out more widely when proved to be beneficial.  For example an RCT by the University of Cambridge’s experimental criminologists has provided the first robust evidence of the benefits of ‘hotspot’ policing in the UK[6]. Another, by Robert Metcalfe of the University of Oxford, provides clear implications about how to reduce household energy consumption in both the short and the long term[7]. Both of these trials, although excellent, will be of very limited practical use if their findings are not used to inform future policy – this will require continued work both by policymakers, and by those running the trials[8].

[1] Haynes et al (2012): “Test, Learn Adapt: Developing Public Policy with Randomised Controlled Trials” Cabinet Office, available from:

[3] Angrist & Pischke (2008): “Mostly Harmless Econometrics” Princeton University Press

[4] Gerber & Green (2012): “Field Experiments: Design, Analysis and Interpretation” W. W. Norton & Company

[6] Presented by Barak Ariel at “Beyond Nudge” conference at the British Academy, June 2012

[7] Presented by Robert Metcalfe at “Beyond Nudge” conference at the British Academy, June 2012

EMA and good evidence-based policy making

December 15, 2010 Leave a comment

Paul Gregg and Lindsey Macmillan


Young people leaving school at age 16 with few or no qualifications face a bleak future. As with high school drop outs in the US, employment rates are low for most of their working lives, as are earnings when they work. So a programme that encourages people from low income families to stay on in school at age 16 would seem to be sensible policy making.

The Educational Maintenance Allowance is a benefit that 16 and 17 year olds from low income families can receive only if they continue in full-time education. To receive it, students are required to regularly attend college or the weekly payment is stopped and there are bonuses for course achievement encouraging completion. Unusually in the UK, EMA was fully trialled with a good study design to give clear answers to how much it changed behaviour. The study carried out by the IFS suggested that among those eligible, those staying on in school increased by 6 to 7 percentage points.

However, from January the government is going to abolish EMA for new starters which will have its main effect next September for new school leavers. The government argues that that 90% of EMA is deadweight and this justifies its abolition based on qualitative evidence that this 90% would have continued in education regardless of the payment. However, just because it changes the behaviour of just 1 in 10 of those eligible, this doesn’t mean it is not cost effective if the benefits to this group are large enough.

The IFS research shows that in areas where EMA was trialled, students as a whole were around 2 percentage points more likely to reach the thresholds for Levels 2 and 3 of the National Qualifications Framework and they also had A Level grades around 4 points higher on average. This was probably because of the attendance requirement and achievement incentives.  The benefits went wider than those who just attended school as a result of EMA. The value of these qualifications in terms of future earnings was greater than the cost of the programme. The IFS argue that even if this increase in participation is relatively small, the longer-term benefits to those affected by the policy in terms of future productivity more than out-weigh the cost of the scheme. Yet this doesn’t include the impact on reduced unemployment, greater well being and even the potential impact on the next generation of having better educated higher earning parents. An interesting piece of research by Leon Fienstein and Ricardo Sabetes  showed that in EMA areas crime generally fell and youth convictions for burglary fell significantly.

So all the evidence is that EMA was a well design policy that improved life chances and had wider ranging benefits. But the greatest paradox is that the government has kept payments for children attending college which go to nearly all children (Child Benefit) whilst slashing the targeted support for the poorest to attend school. Yet all the evidence is clear that it is those from the poorer families for whom the incentive effects are greatest and the longer term value to society is greater.   If the Government wants to save money in this area then reducing Child Benefit for post-16s would achieve the savings without the wider adverse consequences for the life chances of Britain’s poorest children.  As the one of the governments buzz words of the time is responsibility, it is counter-intuitive to dismantle a policy that encourages responsibility for these young adults, who face the biggest opportunity cost of further education, by making a direct payment, conditional on full attendance at school. It is heartening to see so many young people demonstrating against the abolition of EMA and it is important for academics to stand up for good evidence based policy making.