Author: Simon Burgess
RCT + NPD = Progress
A lot of research for education policy is focussed on evaluating the effects of a policy that has already been implemented. After all, we can only really learn from policies that have actually been tried. In the realm of UK education policy evaluation, the hot topic at the moment is the use of randomised control trials or RCTs.
In this post I want to emphasise that in schools in England we are in a very strong position to run RCTs because of the existing highly developed data infrastructure. Running RCTs on top of the census data on pupils in the National Pupil Database dramatically improves their effectiveness and their cost-effectiveness. This is both an encouragement to researchers (and funders) to consider this approach, and also another example of how useful the NPD is.
A major part of the impetus for using RCTs has come from the Education Endowment Foundation (EEF). This independent charity was set up with grant money from the Department for Education, and has since raised further charitable funding. Its goal is to discover and promote “what works” in raising the educational attainment of children from disadvantaged backgrounds. I doubt that anywhere else in the world is there a body with over £100m to spend on such a specific – and important – education objective. Another driver has been the Department for Education’s recent Analytical Review, led by Ben Goldacre, which recommended that the Department engage more thoroughly with the use of RCTs in generating evidence for education policy.
It is probably worth briefly reviewing why RCTs are thought to be so helpful in this regard: it’s about estimating a causal effect. There are of course many very interesting research questions other than those involving the evaluation of casual effects. But for policy, causality is key: “when this policy was implemented, what happened as a result?” The problem is that isolating a causal effect is very difficult using observational data, principally because the people exposed to the policy are often selected in some way and it is hard to disentangle their special characteristics from the effect of the policy. The classic example to show this is a training policy: a new training programme is offered, and people sign up; later they are shown to do better than those who did not sign up; is this because of the content of the training programme … or because those signing up evidently had more ambition, drive or determination? If the former, the policy is a good one and should be widened; if the latter, it may have no effect at all, and should be abandoned.
RCTs get around this problem by randomly allocating exposure to the policy, so there can be no such ambiguity. There are other advantages too, but the principal attraction is the identification of causal effects. Of course, as with all techniques, there are problems too.
The availability of the NPD makes RCTs much more viable and valuable. It provides a census of all pupils in all years in all state schools, including data on demographic characteristics, a complete test score history, and a complete history of schools attended and neighbourhoods lived in.
This helps in at least three important ways.
First, it improves the trade-off between cost and statistical power. Statistical power refers to the likelihood of being able to detect a causal effect if one is actually in operation. You want this to be high – undertaking a long-term and expensive trial and missing the key causal effect through bad luck is not a happy outcome. Researchers typically aim for 80% or 90% power. One of the initial decisions in an RCT is how many participants to recruit. The greater the sample size, the greater the statistical power to detect any causal effects. But of course, also, the greater is the cost, and sometimes this can be considerable. These trade-offs can be quite stark. For example, to detect an effect size of at least 0.2 standard deviations at standard significance levels with 80% power we would need a sample of 786 pupils, half of them treated. If for various reasons we were running the intervention at school level, we would need over 24,000 pupils.
This is where the NPD comes in. In an ideal world, we would want to be able to clone every individual in our sample and try the policy out on one and compare progress to their clone. Absent that, we can improve our estimate of the causal effect by getting as close as we can to ‘alike’ subjects. We can use the wealth of background data in the NPD to reduce observable differences and improve the precision of estimate of intervention effect. Exploiting the demographic and attainment data allows us to create observationally equivalent pupils, one of whom is treated and one is a control. This greatly reduces sampling variation and improves the precision of our estimation. This in turn means that the trade-off between cost and power improves. Returning to the previous numerical example, if we have a good set of predictors for (say) GCSE performance, we can reduce the required dataset for a pupil-level intervention from 786 pupils to just 284. Similarly for the school-cohort level intervention, we can cut back the sample from 24,600 pupils and 160 schools to 9,200 pupils and 62 schools. The relevant correlation is between a ‘pre-test’ and the outcome (this might literally be a pre-test, or it can be a prediction from a set of variables).
Second, the NPD is very useful for dealing with attrition. Researchers running RCTs typically face a big problem of participants dropping out of the study, both from the treatment arms and from the control group. Typically this is because the trial becomes too burdensome or inconvenient, rather than on principle because they did sign up in the first instance. This attrition can cause severe statistical problems and can jeopardise the validity of the study.
The NPD is a census and is an administrative dataset, so data on all pupils in all (state) schools are necessarily collected. This obviously includes all national Keystage test scores, GCSEs and A levels. If the target outcome of the RCT is improving test scores, then these data will be available to the researcher for all schools. Technically this means that an ‘intention to treat’ estimator can always be calculated. (obviously, if the school or pupil drops out and forbids the use of linked data then this is ruled out, but as noted above, most dropout is simply due to the burden).
Finally, the whole system of testing from which the NPD harvests data is also helpful. It embodies routine and expected tests so there is less chance of specific tests prompting specific answers. Although a lot about trials in schools cannot be ‘blind’ in the traditional way, these tests are blind. They are also nationally set and remotely marked, all of which adds to the validity of the study. These do not necessarily cover all the outcomes of interest such as wellbeing or health or very specific knowledge, but they do cover the key goal of raising attainment.
In summary, relative to other fields, education researchers have a major head start in running RCTs because of the strength, depth and coverage of the administrative data available.
Author: Paul Gregg
How should long-term unemployment be tackled?
Earlier in the week, George Osbourne announced new government plans for the very long term unemployed. The government flagship welfare to work programme, the Work Programme, lasts for two years and so there has been a question about what happens to those not finding work through it. Currently only 20% of those starting the Work Programme find sustained employment, although many more cycle in and out of employment.
Very long-term unemployment (2+ years) is strongly cyclical, almost disappearing from 1998 to 2009, but has returned with the protracted period of poor economic performance. This cyclicality is a strong indicator that it is not driven by a large group of workshy claimants. Rather the state of the economy leaves a few who unable to get work quickly face ever increasing employer resistance to higher them. Faced with ample choice of newly unemployed these people look like unnecessary risks with outdated skills.
Very long-term unemployment is thus not a new phenomenon and a large range of policies have been tried before and hence we have a very good idea of what does and does not work. The proposals had three elements. The first which got the headlines was that claimants would be made to ‘Work for the Dole’. The effects of requiring people to go into work placements depends a lot on the quality of the work experience offered. Such schemes have three main effects: first, some people leave benefits ahead of the required employment. This is called the deterrent effect and is stronger the more unpleasant and low paid (eg work for the dole) the placement is. Then, whilst on the placement, job search and job entry tend to dip as the person’s time is absorbed by working rather than applying for jobs. Finally, the gaining of work experience raises job search success on completion of the placement. This is stronger for high-quality job placement in terms of the experience gained and being with a regular employer who can give a good reference if the person has worked well.
The net effect of many such programmes, including work for the dole, has often been little or even negative. Australia and New Zealand have all tried and abandoned Work for the Dole policies because they were so ineffectual in getting people into work. The best effects from work experience programmes come where job search is actively required and supported when on a work placement, where the placement is with a regular employer rather than a “make work” scheme and where the placement provider is incentivised to care about the employment outcomes of the unemployed person after the work placement ends. The Future Jobs Fund under the previous labour government, which placed young people into high quality placements and paid a wage, was clearly a success in terms of improving job entry although the government cut it.
This element of the government’s plans has little chance of making a positive difference. However, the other elements maybe more positive. Some, the mix across elements is not clear yet, of the very long-term unemployed will be required to do daily signing. This probably means that the claimant will have to attend a Job Centre Plus office every day and look for and apply for jobs on the suite of computers. This is very similar to the Work Programme but more intense and perhaps with less support for CV writing and presentation etc. This may enhance the frequency of job applications but perhaps not the quality and may prove no more successful than the Work Programme. The third element is to attend a new as yet unspecified programme. As there are few details as yet it is hard to comment on this part.
The overall impression is that the announcement is of a rehashed version of previous rather unsuccessful programmes founded on a belief that the long-term unemployed are workshy rather unfortunates needing intensive help to overcome employer resistance and return to work.
Author: Michael Sanders
University, Gambling, and the Greater Fool
The betting company Ladbrokes have begun offering students (and their parents) the opportunity to bet on their eventual university degree classifications. This, as may have been predictable (and may have been the intention) has attracted a level of opprobrium from groups concerned about youngsters gambling away their student loans foolishly.
What does economics tell us about this? To begin with, this looks like a fairly standard asymmetric information problem, from which students can only benefit. In general, it is not sensible to make a bet with someone who has more information with you, or who has control over the outcome of that bet. For example, I bet you a million pounds that the next sentence will contain the word banana. Clearly, you won’t take the bet because I can control the outcome banana.
For students, the deal is a good one. They know how clever they are, and they know how hard they will work. Even if there is some noise associated with their outcome (bad days, sick pets, or grandmother fatalities), it is a fair bet that the people beginning their university lives this week have more control over the outcome than Ladbrokes do. So why are Ladbrokes taking these bets (and actively encouraging them)?
One possibility is that Ladbrokes are cash poor, and want to raise finance quickly. They take money in now from students placing their bets, but don’t need to pay out for three years. A perfectly sound theoretical argument, but it seems unlikely, either (a) that Ladbrokes can’t find better rates on what is essentially a loan on the open market, or (b) that students are so cash rich that they’re making long term investments.
A second possibility is that Ladbrokes is a ‘greater fool’ – a person who buys high and sells cheap, so that the rest of us can profit. Given their track record, I suspect not.
More likely, they are relying on students being greater fools. Where traditional economic theory tends to assume that agents observe their own quality with certainty (or, in English, what we know how good we are), behavioural economics suggests otherwise.
Overconfidence is an issue across many dimensions. It leads us to pay for expensive gym contracts we’ll never use and to drive less carefully than we should . Even among hyper-rational investors, it leads to over-investment in our own firms. So, even though we know that only 5% of students will get a first class degree, we rate our own chances at 10%. For some people this may be true, but for most it will not, and so firms like Ladbrokes can profit from our misconception.
Behavioural Economics offers useful tips on self control, and I’d encourage anyone at the beginning of their university career (or later in), to think about them seriously. There are times when it is good to be a greater fool, this is not one of them.
Post Script: A Rational Bet
On circulating this post internally, I’ve been asked under what circumstances you should take this bet. For almost everyone at Bristol, studying the social sciences, the odds you’ll get betting on a 2:1 are probably about 5/6 (Bristol isn’t one of those featured on the Ladbrokes site), so you’d lose money whatever you do. If you’re confident of getting a 2:1, however, you might be interested to know what happens if you work a bit less hard and bet on yourself getting a 2:2. Here the odds are better, probably about 12:5 – so you’ll get your initial investment back, plus an additional 140%. A recent working paper from the LSE finds that the return to a 2:1 is 2040 a year. If we extrapolate this for a 45 year career, that’s an extra £91800 over the course of your lifetime. Assuming a constant rate of inflation at 3% over that time, you’d need £181,516 now in order to maintain the same standard of living for your entire life. To win that, you’d need to bet £129,654 right now in order to be indifferent between getting a 2:2 and winning the bet, or getting a 2:1 and not betting. I’d still recommend against it, though.