Author: Simon Burgess
RCT + NPD = Progress
A lot of research for education policy is focussed on evaluating the effects of a policy that has already been implemented. After all, we can only really learn from policies that have actually been tried. In the realm of UK education policy evaluation, the hot topic at the moment is the use of randomised control trials or RCTs.
In this post I want to emphasise that in schools in England we are in a very strong position to run RCTs because of the existing highly developed data infrastructure. Running RCTs on top of the census data on pupils in the National Pupil Database dramatically improves their effectiveness and their cost-effectiveness. This is both an encouragement to researchers (and funders) to consider this approach, and also another example of how useful the NPD is.
A major part of the impetus for using RCTs has come from the Education Endowment Foundation (EEF). This independent charity was set up with grant money from the Department for Education, and has since raised further charitable funding. Its goal is to discover and promote “what works” in raising the educational attainment of children from disadvantaged backgrounds. I doubt that anywhere else in the world is there a body with over £100m to spend on such a specific – and important – education objective. Another driver has been the Department for Education’s recent Analytical Review, led by Ben Goldacre, which recommended that the Department engage more thoroughly with the use of RCTs in generating evidence for education policy.
It is probably worth briefly reviewing why RCTs are thought to be so helpful in this regard: it’s about estimating a causal effect. There are of course many very interesting research questions other than those involving the evaluation of casual effects. But for policy, causality is key: “when this policy was implemented, what happened as a result?” The problem is that isolating a causal effect is very difficult using observational data, principally because the people exposed to the policy are often selected in some way and it is hard to disentangle their special characteristics from the effect of the policy. The classic example to show this is a training policy: a new training programme is offered, and people sign up; later they are shown to do better than those who did not sign up; is this because of the content of the training programme … or because those signing up evidently had more ambition, drive or determination? If the former, the policy is a good one and should be widened; if the latter, it may have no effect at all, and should be abandoned.
RCTs get around this problem by randomly allocating exposure to the policy, so there can be no such ambiguity. There are other advantages too, but the principal attraction is the identification of causal effects. Of course, as with all techniques, there are problems too.
The availability of the NPD makes RCTs much more viable and valuable. It provides a census of all pupils in all years in all state schools, including data on demographic characteristics, a complete test score history, and a complete history of schools attended and neighbourhoods lived in.
This helps in at least three important ways.
First, it improves the trade-off between cost and statistical power. Statistical power refers to the likelihood of being able to detect a causal effect if one is actually in operation. You want this to be high – undertaking a long-term and expensive trial and missing the key causal effect through bad luck is not a happy outcome. Researchers typically aim for 80% or 90% power. One of the initial decisions in an RCT is how many participants to recruit. The greater the sample size, the greater the statistical power to detect any causal effects. But of course, also, the greater is the cost, and sometimes this can be considerable. These trade-offs can be quite stark. For example, to detect an effect size of at least 0.2 standard deviations at standard significance levels with 80% power we would need a sample of 786 pupils, half of them treated. If for various reasons we were running the intervention at school level, we would need over 24,000 pupils.
This is where the NPD comes in. In an ideal world, we would want to be able to clone every individual in our sample and try the policy out on one and compare progress to their clone. Absent that, we can improve our estimate of the causal effect by getting as close as we can to ‘alike’ subjects. We can use the wealth of background data in the NPD to reduce observable differences and improve the precision of estimate of intervention effect. Exploiting the demographic and attainment data allows us to create observationally equivalent pupils, one of whom is treated and one is a control. This greatly reduces sampling variation and improves the precision of our estimation. This in turn means that the trade-off between cost and power improves. Returning to the previous numerical example, if we have a good set of predictors for (say) GCSE performance, we can reduce the required dataset for a pupil-level intervention from 786 pupils to just 284. Similarly for the school-cohort level intervention, we can cut back the sample from 24,600 pupils and 160 schools to 9,200 pupils and 62 schools. The relevant correlation is between a ‘pre-test’ and the outcome (this might literally be a pre-test, or it can be a prediction from a set of variables).
Second, the NPD is very useful for dealing with attrition. Researchers running RCTs typically face a big problem of participants dropping out of the study, both from the treatment arms and from the control group. Typically this is because the trial becomes too burdensome or inconvenient, rather than on principle because they did sign up in the first instance. This attrition can cause severe statistical problems and can jeopardise the validity of the study.
The NPD is a census and is an administrative dataset, so data on all pupils in all (state) schools are necessarily collected. This obviously includes all national Keystage test scores, GCSEs and A levels. If the target outcome of the RCT is improving test scores, then these data will be available to the researcher for all schools. Technically this means that an ‘intention to treat’ estimator can always be calculated. (obviously, if the school or pupil drops out and forbids the use of linked data then this is ruled out, but as noted above, most dropout is simply due to the burden).
Finally, the whole system of testing from which the NPD harvests data is also helpful. It embodies routine and expected tests so there is less chance of specific tests prompting specific answers. Although a lot about trials in schools cannot be ‘blind’ in the traditional way, these tests are blind. They are also nationally set and remotely marked, all of which adds to the validity of the study. These do not necessarily cover all the outcomes of interest such as wellbeing or health or very specific knowledge, but they do cover the key goal of raising attainment.
In summary, relative to other fields, education researchers have a major head start in running RCTs because of the strength, depth and coverage of the administrative data available.
Author: Paul Gregg
How should long-term unemployment be tackled?
Earlier in the week, George Osbourne announced new government plans for the very long term unemployed. The government flagship welfare to work programme, the Work Programme, lasts for two years and so there has been a question about what happens to those not finding work through it. Currently only 20% of those starting the Work Programme find sustained employment, although many more cycle in and out of employment.
Very long-term unemployment (2+ years) is strongly cyclical, almost disappearing from 1998 to 2009, but has returned with the protracted period of poor economic performance. This cyclicality is a strong indicator that it is not driven by a large group of workshy claimants. Rather the state of the economy leaves a few who unable to get work quickly face ever increasing employer resistance to higher them. Faced with ample choice of newly unemployed these people look like unnecessary risks with outdated skills.
Very long-term unemployment is thus not a new phenomenon and a large range of policies have been tried before and hence we have a very good idea of what does and does not work. The proposals had three elements. The first which got the headlines was that claimants would be made to ‘Work for the Dole’. The effects of requiring people to go into work placements depends a lot on the quality of the work experience offered. Such schemes have three main effects: first, some people leave benefits ahead of the required employment. This is called the deterrent effect and is stronger the more unpleasant and low paid (eg work for the dole) the placement is. Then, whilst on the placement, job search and job entry tend to dip as the person’s time is absorbed by working rather than applying for jobs. Finally, the gaining of work experience raises job search success on completion of the placement. This is stronger for high-quality job placement in terms of the experience gained and being with a regular employer who can give a good reference if the person has worked well.
The net effect of many such programmes, including work for the dole, has often been little or even negative. Australia and New Zealand have all tried and abandoned Work for the Dole policies because they were so ineffectual in getting people into work. The best effects from work experience programmes come where job search is actively required and supported when on a work placement, where the placement is with a regular employer rather than a “make work” scheme and where the placement provider is incentivised to care about the employment outcomes of the unemployed person after the work placement ends. The Future Jobs Fund under the previous labour government, which placed young people into high quality placements and paid a wage, was clearly a success in terms of improving job entry although the government cut it.
This element of the government’s plans has little chance of making a positive difference. However, the other elements maybe more positive. Some, the mix across elements is not clear yet, of the very long-term unemployed will be required to do daily signing. This probably means that the claimant will have to attend a Job Centre Plus office every day and look for and apply for jobs on the suite of computers. This is very similar to the Work Programme but more intense and perhaps with less support for CV writing and presentation etc. This may enhance the frequency of job applications but perhaps not the quality and may prove no more successful than the Work Programme. The third element is to attend a new as yet unspecified programme. As there are few details as yet it is hard to comment on this part.
The overall impression is that the announcement is of a rehashed version of previous rather unsuccessful programmes founded on a belief that the long-term unemployed are workshy rather unfortunates needing intensive help to overcome employer resistance and return to work.
Author: Michael Sanders
University, Gambling, and the Greater Fool
The betting company Ladbrokes have begun offering students (and their parents) the opportunity to bet on their eventual university degree classifications. This, as may have been predictable (and may have been the intention) has attracted a level of opprobrium from groups concerned about youngsters gambling away their student loans foolishly.
What does economics tell us about this? To begin with, this looks like a fairly standard asymmetric information problem, from which students can only benefit. In general, it is not sensible to make a bet with someone who has more information with you, or who has control over the outcome of that bet. For example, I bet you a million pounds that the next sentence will contain the word banana. Clearly, you won’t take the bet because I can control the outcome banana.
For students, the deal is a good one. They know how clever they are, and they know how hard they will work. Even if there is some noise associated with their outcome (bad days, sick pets, or grandmother fatalities), it is a fair bet that the people beginning their university lives this week have more control over the outcome than Ladbrokes do. So why are Ladbrokes taking these bets (and actively encouraging them)?
One possibility is that Ladbrokes are cash poor, and want to raise finance quickly. They take money in now from students placing their bets, but don’t need to pay out for three years. A perfectly sound theoretical argument, but it seems unlikely, either (a) that Ladbrokes can’t find better rates on what is essentially a loan on the open market, or (b) that students are so cash rich that they’re making long term investments.
A second possibility is that Ladbrokes is a ‘greater fool’ – a person who buys high and sells cheap, so that the rest of us can profit. Given their track record, I suspect not.
More likely, they are relying on students being greater fools. Where traditional economic theory tends to assume that agents observe their own quality with certainty (or, in English, what we know how good we are), behavioural economics suggests otherwise.
Overconfidence is an issue across many dimensions. It leads us to pay for expensive gym contracts we’ll never use and to drive less carefully than we should . Even among hyper-rational investors, it leads to over-investment in our own firms. So, even though we know that only 5% of students will get a first class degree, we rate our own chances at 10%. For some people this may be true, but for most it will not, and so firms like Ladbrokes can profit from our misconception.
Behavioural Economics offers useful tips on self control, and I’d encourage anyone at the beginning of their university career (or later in), to think about them seriously. There are times when it is good to be a greater fool, this is not one of them.
Post Script: A Rational Bet
On circulating this post internally, I’ve been asked under what circumstances you should take this bet. For almost everyone at Bristol, studying the social sciences, the odds you’ll get betting on a 2:1 are probably about 5/6 (Bristol isn’t one of those featured on the Ladbrokes site), so you’d lose money whatever you do. If you’re confident of getting a 2:1, however, you might be interested to know what happens if you work a bit less hard and bet on yourself getting a 2:2. Here the odds are better, probably about 12:5 – so you’ll get your initial investment back, plus an additional 140%. A recent working paper from the LSE finds that the return to a 2:1 is 2040 a year. If we extrapolate this for a 45 year career, that’s an extra £91800 over the course of your lifetime. Assuming a constant rate of inflation at 3% over that time, you’d need £181,516 now in order to maintain the same standard of living for your entire life. To win that, you’d need to bet £129,654 right now in order to be indifferent between getting a 2:2 and winning the bet, or getting a 2:1 and not betting. I’d still recommend against it, though.
Author: Simon Burgess
Threshold measures in school accountability: asking the right question
We are in the midst of a significant upheaval in the setting and marking of exams, and the reporting of school exam results. One feature of the system has been the centre of a lot of criticism and highlighted for reform: the focus on the percentage of a school’s pupils that achieve at least 5 GCSEs at grades C to A*, including the scores on English and maths. This is typically the most-discussed metric for (secondary) school performance and is the headline figure in the school league tables.
The point is that this measure is based on a threshold, a ‘cliff-edge’. Get a grade C and you boost the school’s performance; missing a C by a lot or a little are the same, and just scraping a C is the same as getting an A*.
This has been described as distorting schools’ behaviour, forcing schools to focus on pupils around this borderline. The argument is seen as obviously right and strong grounds for change. In this post I want to make two counter-arguments, and to suggest we are asking the wrong question.
First a basic point. One central goal of any performance measure is to induce greater or better-targeted effort. This might just mean “working harder” or it might mean a stronger focus on the goals embodied in the measure at the expense of other outcomes. The key for the principal is to design the best scheme to achieve this. A very common scheme is a threshold one – this can be found for example in the Quality and Outcomes Framework for GPs, service organisations with a target number of clients to see, and of course schools trying to help pupils to achieve at least 5 grades of C or better. An organisation working under a threshold scheme faces very different marginal incentives for effort. Considering pupils: the most intense incentives relate to pupils just below the line: this is where the greatest payoff is to schools to devote the most resources.
The first counter argument starts by noting that the asymmetry in the incentive is not a newly-discovered flaw, it is a design feature which can be very powerful. If there is a level of achievement that is extremely important for everyone to reach, then it makes sense to set up a scheme that offers very strong incentives to do that – that focusses the incentive around that minimum level. This is precisely what a threshold scheme does.
So rather than simply pointing out that threshold designs strongly focus attention (which is what they’re supposed to do), the questions to ask are: is there some level of attainment that has that characteristic of being a minimum level of competence? And if so, what is it? If society feels that 5 grade C’s is a fair approximation to a minimum level that we want everyone to achieve, then it is absolutely right to have a ‘cliff-edge’ there because inducing schools to work very hard to get pupils past that level is exactly what society wants. It may be that we are equally happy to see grades increase for the very brightest children, those in the middle or those at the lower end of the ability distribution. Or not: all the main political parties express a desire to raise attainment at the lower end and narrow gaps.
The argument should be about where to put the threshold, not whether to have one or not. Perhaps we are starting to see a recognition of this in the recent policy announcement that all pupils will have to continue studying until they have passed English and Maths.
The second counter-argument is based on a scepticism of what is likely to happen without the 5A*-C(EM) threshold acting as a focal point.
The core strategic decision facing a headteacher is how best to deploy her main resource: the teachers. Specifically: how best to assign teachers of varying effectiveness to different classes. It has been said that schools will be free to focus equally on all pupils.
Well, maybe. Or perhaps we should think of the pressures on the headteacher, in this instance from teachers themselves. Effective teachers are very valuable to a school and any headteacher will be keen to keep her most effective teachers happy and loyal. It seems likely (I have no evidence on this, and would be keen to hear of any) that top teachers would typically prefer to teach top sets. If so, we might see a drift of the more effective teachers towards the more able classes in a school (and therefore on average, the more affluent pupils). The imperative of the C/D threshold gave headteachers an unanswerable argument to push against this.
So threshold metrics have an important role to play in communicating to schools where society wants them to focus their effort. The current threshold, at 5 C grades, may or may not be at the right level; but discussing what the right level is, is a more useful debate to have.
Author: Michael Sanders
Arguing about funding obscures important issues of quality research
Richard Thaler, the Chicago professor of economics and incoming president of the American Economic Association, has as one of his many mantras the truism that “we can’t do evidence based policy without evidence”. The government’s recent decision to establish a number of “What Works Centres” to collate, analyse and, in some cases, produce, evidence on a number of policy areas seek to address the very problem of a lack of evidence.
Evidence itself, however, is not in short supply. Newspapers fill their pages, day after day, with the results of studies into some facet of human behaviour, or statistics on the state of the world. So, there need to be two other criteria for evidence than mere ‘existence’ – goodness, and usability. I should be clear at the outset that when I say “Good”, I mean “Capable of determining a causal relationship between an input and an output”. Sadly, not all evidence which is good is useable, and often tragically, not all evidence that is usable is good.
As Ben Goldacre points out in his recent paper for the department for education, many researchers in that field like qualitative work, and use this as the basis for their findings. As an economist, I have a natural scepticism for such research, but I cannot dispute that it is eminently useable. The arguments constructed by such research are easily and well presented. They offer solutions which are simple, and neat. However, as H.L. Mencken said, these arguments are almost always also wrong. This research is usable but very much of it is not good.
On the other side, much research which is good, and detailed, and thorough, presents complicated and nuanced answers which reflect reality but whose methods are impenetrable to anyone who might actually have the power to change policy accordingly.
Randomised Controlled Trials (RCTs) are both useable, with the majority of results presentable in an easily understood way and the methodology being simple enough to explain to a lay person in about five minutes. As the recognised ‘gold standard’ of evaluation, they are also indisputably good.
In a blog post for the LSE impact blog, Neil Harris, a researcher at Bristol’s Centre for Causal Analysis in Translation Epidemiology, argues that education research is a public good and needs to be funded by the state, as, unlike in medicine, there is not money to be made by researchers through patent development, education being a public good. He is, of course, absolutely right. The structure of his argument implies however, that in order to get good evidence, it will need to be paid for – i.e. that RCTs are expensive, while qualitative research is cheap. If the government wants better education research, they should give researchers more money. But, well, we would say that, wouldn’t we?
The argument that RCTs are expensive is a well-worn one, but is not helpful, and often dangerously distracting. Saying that an RCT is expensive is akin to saying “Vehicles are expensive”. If one chooses to put up Ferraris as an example, then of course they are. A scooter, however, is not. Both are better than walking.
A good quality, robust RCT need not be outlandishly expensive, and certainly not any more so than qualitative analysis. Unlike medical trials, the marginal cost of interventions in policy is often not far above that of treatment as usual (the most logical control condition). Teaching phonics in 50 schools and not in 50 others should not require vast resources once allocation has taken place. Although the government does not spend as much money on policy research as it does on medicine, it spends a lot of money gathering data on the outcomes many researchers are interested in. At the end of a child’s GCSEs, finding out how well they did does not require specialist staff to draw their blood and perform expensive tests on them. The school knows the answer.
It is important not to downplay the risks or costs associated with RCTs, but nor is it possible to present these costs as a reason for conducting, or accepting, substandard research. As researchers, if our work is of low quality, there is only so far the buck can be passed.
School meals and packed lunches – How important is government policy?
Last week’s government-commissioned school food review showed that the nutritional quality of school food has improved substantially since 2005, when Jamie Oliver started its campaign to improve the nutritional value of school meals. Nevertheless, take-up of school meals remains low, at 43%. In other words, 57% of children are not eating school lunches, but bring a packed lunch, have snacks, or buy their food elsewhere. The report shows that the majority of these meals are unhealthy. In fact, in contrast to what most parents think, only 1% of packed lunches meet the nutritional standards, as they tend to include sweets, sugary drinks, and savoury snacks.
In addition to affecting child health, there is substantial evidence that poor nutrition affects cognitive performance. Michèle Belot and Jonathan James show in their study that the Jamie Oliver campaign led to a significant increase in children’s test scores in primary schools (Key Stage 2), as well as a drop in authorised absences (i.e. those that are mostly linked to illness and health).
Figure 1 is taken from last week’s school food review (p.42), showing that school meal take-up dropped from around 70% in the early 1970s to just over 40% in the late 1980s. Why has this take-up declined so dramatically? The authors of the review state that the reasons for this decline are complex. They argue that it may be partly driven by the rampant inflation of the mid-seventies, but also by the removal of the national fixed pricing of school meals.
In a recent study, I specifically focus on the latter: I investigate how two Acts of Parliament, introduced in the 1980s, affected school meal take-up in primary schools. Prior to 1980, schools had to provide free meals to all children in low-income families, whilst others paid a fixed price of 35p per day, set by the government. With the election of the Conservative administration in 1979, the government attitude to the service shifted. It was viewed as too expensive and the government wanted to introduce more choice and parental responsibility. Two reforms were introduced, which radically altered the school meal service.
The first, the 1980 Education Act, ended the fixed pricing of school meals, and abolished the minimum nutritional standards. Although schools were still obliged to provide free lunches to those eligible for free meals (i.e. those in low-income families), they could now set the price for all those not eligible. This caused a price increase of around 43%: the most common price in 1981 was 50p per day.
The main effect of second reform, the 1988 Local Government Act, was a tightening of eligibility rules: children in families receiving Family Credit (FC) were no longer eligible for free meals, with only those on Income Support (IS) remaining eligible.
I evaluate whether households change their behaviour due to these reforms, and find evidence of a considerable household response. Rather than plotting the school meal take-up rates of all children, as in Figure 1, Figure 2 shows the trends in the take-up rates of school meals by whether the child was affected by the reforms. The solid line denotes those not on benefits (those affected by the 1980 Act). Prior to the 1980 Act, their school meal take-up rates remained relatively constant. With the introduction of the reforms in 1980, however, their take-up rates drop considerably, with little change among those not affected (the dashed and dotted lines).
Figure 3 shows that this drop is compensated by an increase in the consumption of packed lunches.
Looking at the effects of the 1988 reform, the data show a very similar pattern. School meal take-up rates drop dramatically among those on FC (those affected by the 1988 reform), with no changes in the take-up rates among those not affected. Again, the drop in the take-up of school meals is compensated by an increase in the consumption of packed lunches.
I next investigate whether the reforms affected children’s Body Mass Index (weight in kilograms divided by height in metres squared). The results, however, show no consistent evidence that the reforms resulted in changes in children’s body weight.
With almost one in five 11-year-old children currently obese, this is obviously good news. However, as the data are not representative of today’s society, with large changes in the nutritional environment since the 1980s, we cannot extrapolate these results to the current context, nor can we argue that today’s school meals or packed lunches do not affect children’s weight. In addition, although I find no effects on child body weight, there may have been other advantages of the government-provided school meals, such as better academic attainment, or better health and nutritional outcomes later in life. Unfortunately, I have no data to explore this further.
To conclude, as suggested by the school food review, inflation may certainly have played a role in the downward trend in school meal take-up in the 1970s and 1980s. However, the analyses above show that exploring the trend in school meal take-up among all children may conceal potential differential trends between different subgroups of children. Indeed, distinguishing between those affected and those not affected by the reforms highlights clear differences in their response to government policy. It shows the extent to which households change their behaviour in response to such policies, suggesting that the reforms were an important cause of the decline in school meal take-up during the 1980s.
Author: Michael Sanders
Non-standard matches and charitable giving
The use of incentives to encourage charitable donations is commonplace. The governments of most developed countries (including the UK), as well as many large employers, offer match rates, or rebates to encourage donations. The principle behind such match rates is simple. With Gift-Aid, the UK’s main form of tax effective giving, a donation of £1 from net-of-tax income by a taxpayer attracts basic rate tax relief, which goes straight to the charity – hence, the charity receives £1.25 for every £1 donated.
The effect of this is to reduce the price of donating a given amount – if I am a basic rate taxpayer, and think “I want the British Heart Foundation to receive £100”, making that happen will only cost me £80 from my net of tax spending.
There is much discussion, both in the academic economics literature and in the public sphere, about the effectiveness of such matches, particularly in the aftermath of last year’s proposed £50,000 cap on charitable tax relief – although most studies find that people’s donations are relatively unresponsive to changes in the match rate (they are price inelastic), CMPO research has shown that high value donors may be more responsive to changes in their match (or rebate) than are smaller donors.
This mixed evidence on the effectiveness of matches at increasing out-of-pocket donations suggests that alternatives to the standard match may be more effective. Options for non-standard matches are summarized in our new working paper, which draws on the behavioural economics and psychology literatures to several such possibilities, including, non-linear matches, where the more an individual donates, the higher the match rate; social and team matches, in which the match rate received by one donor depends on the donations of others, giving them an incentive to ‘crowd in’ their friends and colleagues; competitive matches , when only the most successful fundraisers receive a match); and lottery matches, where each donation increases the chance of a donor’s chosen charity receiving a large windfall match.
Although these suggestions are sound in principle, and supported by both theory and empirical evidence, none has been experimentally tested in a real world setting. We would encourage anybody interested in testing one or more of these novels matches to contact us.
Author: Simon Burgess
Education spending, pupil attainment and causality
In these hard times, spending government money effectively is more important than ever. Last week Fraser Nelson challenged the effectiveness of spending in schools, one of the areas relatively protected from Coalition cuts. He said: “The biggest surprise, though, was the money: no matter how you split the figures, the amount spent didn’t seem to make the blindest bit of difference”, his reading of a report by Deloitte commissioned by the Department for Education.
What is the evidence? In fact, it is surprisingly difficult to establish the impact of spending more money on student achievement. This is partly shortage of data (researchers always want more data), but there is a more fundamental reason too.
Perhaps inadvertently, Fraser Nelson illustrated the difficulty in his first paragraph. He noted the variation in per-pupil expenditure “ranging from £4,500 in Lyme Regis to £10,000 in Salford.” This is absolutely right – there are very significant variations in revenue per pupil. But the key point is that these are not random: extra resources are explicitly and systematically directed towards schools in poorer neighbourhoods. The mechanism, accreting the new schemes of each successive government, may be incomprehensibly complex, but the intent is surely right.
Getting back to our question, on the one hand we have this systematic distribution of resources towards poorer neighbourhoods. On the other hand we know that pupil attainment is typically lower in schools in such neighbourhoods; not for every pupil, not in every school, but on average. So if money has no impact on attainment, and we line up pupil attainment and school expenditure, we will tend to see a negative relationship. This derives solely from the way that money is distributed to schools. The fundamental problem is that there are two things going on with opposite effects: low attainment is associated with more money (via the schools funding system) and more money may be associated with high attainment (via the education process). With no other information, there is simply no way of disentangling these two opposing effects, and by itself these numbers can tell us nothing about the causal impact of school expenditure on pupil attainment.
So the view that “the amount spent didn’t seem to make the blindest bit of difference” cannot be supported by this evidence.
What of the wider research evidence, based on studies with a plausibly causal research design? One of the most prominent economists in the field of education, Rick Hanushek from Stanford, is famously sceptical of the value of greater resources for schools. There certainly are studies that show money can matter, but it is probably fair to say that the majority view among economists is that simply providing more resources for schools is not the best option.
The really interesting question is this: why doesn’t more money raise attainment? More money usually helps most things. Either there simply is nothing that schools can buy that raises attainment. This seems unlikely, and would certainly be a surprise to parents paying many thousands of pounds to send their children to private schools. Or there are features of the system which lead schools to spending extra resources on the ‘wrong’ things – things that have little impact on attainment. This might be the manner in which the money is distributed by government (typically short-term, making long-term expenditure decisions risky); or the regulations and agreements governing its spending by schools; or other factors. We have speculated a little about this here.
Coincidentally, the Department for Education has just opened a consultation on school efficiency – they await your views.