Education spending, pupil attainment and causality

Author: Simon Burgess

Education spending, pupil attainment and causality

In these hard times, spending government money effectively is more important than ever. Last week Fraser Nelson challenged the effectiveness of spending in schools, one of the areas relatively protected from Coalition cuts. He said: “The biggest surprise, though, was the money: no matter how you split the figures, the amount spent didn’t seem to make the blindest bit of difference”, his reading of a report by Deloitte commissioned by the Department for Education.

What is the evidence? In fact, it is surprisingly difficult to establish the impact of spending more money on student achievement. This is partly shortage of data (researchers always want more data), but there is a more fundamental reason too.

Perhaps inadvertently, Fraser Nelson illustrated the difficulty in his first paragraph. He noted the variation in per-pupil expenditure “ranging from £4,500 in Lyme Regis to £10,000 in Salford.” This is absolutely right – there are very significant variations in revenue per pupil. But the key point is that these are not random: extra resources are explicitly and systematically directed towards schools in poorer neighbourhoods. The mechanism, accreting the new schemes of each successive government, may be incomprehensibly complex, but the intent is surely right.

Getting back to our question, on the one hand we have this systematic distribution of resources towards poorer neighbourhoods. On the other hand we know that pupil attainment is typically lower in schools in such neighbourhoods; not for every pupil, not in every school, but on average. So if money has no impact on attainment, and we line up pupil attainment and school expenditure, we will tend to see a negative relationship. This derives solely from the way that money is distributed to schools. The fundamental problem is that there are two things going on with opposite effects: low attainment is associated with more money (via the schools funding system) and more money may be associated with high attainment (via the education process). With no other information, there is simply no way of disentangling these two opposing effects, and by itself these numbers can tell us nothing about the causal impact of school expenditure on pupil attainment.

So the view that “the amount spent didn’t seem to make the blindest bit of difference” cannot be supported by this evidence.

What of the wider research evidence, based on studies with a plausibly causal research design? One of the most prominent economists in the field of education, Rick Hanushek from Stanford, is famously sceptical of the value of greater resources for schools. There certainly are studies that show money can matter, but it is probably fair to say that the majority view among economists is that simply providing more resources for schools is not the best option.

The really interesting question is this: why doesn’t more money raise attainment? More money usually helps most things. Either there simply is nothing that schools can buy that raises attainment. This seems unlikely, and would certainly be a surprise to parents paying many thousands of pounds to send their children to private schools. Or there are features of the system which lead schools to spending extra resources on the ‘wrong’ things – things that have little impact on attainment. This might be the manner in which the money is distributed by government (typically short-term, making long-term expenditure decisions risky); or the regulations and agreements governing its spending by schools; or other factors. We have speculated a little about this here. 

Coincidentally, the Department for Education has just opened a consultation  on school efficiency – they await your views.

Profits in schools – response

Author: Simon Burgess

Profits in schools – response

There were some interesting comments on the recent post I wrote on profit-making schools. This post offers a brief reply to those points.

First, one comment was that allowing profit-making is a solution to the lack of capital for schools:

“advocates see profit-making as a way to tap the private finance that might allow supply-side liberalisation, which would in turn allow choice to operate more effectively than it does at present. Theoretically, of course, this boost to capacity could be done with public finance. But it’s questionable whether the necessary level of spare capacity would be politically sustainable given all the other calls on public spending (especially now). So private finance is (arguably) one solution to that problem.”

It may be a solution to that problem, but it is not a necessary solution, there are other ways. The PFI programme has been funding capital spending on schools for over a decade now.  Nor is it just a thing of the past: in 2011 Michael Gove announced capital expenditure through PFI of around £2bn to rebuild 300 schools. The latest estimates are that PFI expenditure on education will top £260m in 2012-13, and the whole programme has generated over £7bn for school building. The PFI obviously utilises the profit motive in the capital market to get funds into school building without needing profits in the schools themselves.

Second is the question of just how profits can be made. Given fixed revenue per student, it is not possible to directly make a greater rate of return by raising quality (the indirect route is discussed below). Profits can be made by reducing costs. This may be possible without reducing quality, or not. That possibility is that other agents can come in, re-arrange the budget, reduce costs and maintain quality by raising quality per pound spent. The comment was:

“You also argue that ‘outsiders’ are unlikely to know best how best to deploy their budgets. This seems like an odd argument. The market’s virtue is supposed to be innovation and the ability to scale good practice quickly through incentives to mimic the best. If you don’t think that works then I can’t see why you’d be interested in the practical aspects of for-profit schools, since there wouldn’t even be any benefits in principle.”

It is certainly true that schools are unlikely to be making completely optimal decisions. Our own work shows a huge degree of heterogeneity in schools’ financial decisions which is very unlikely all to be optimal. So they certainly have scope for learning. And schools may be able to learn from each other: a lot of people interpret the success of London schools as down to ‘London Challenge’ – and a lot of people interpret the success of that to collaboration, to learning from other schools. In fact, we are in the design stage of a large-scale RCT to test this out. But the key point is that with the current system for school revenue, allowing profit-making provides incentives to reduce costs but no direct incentive to raise quality. So again profits might be a way of encouraging collaboration, but there are other, easier, ways of doing the same thing.

The indirect channel for profit making to affect quality is a dynamic one. The third comment is:

“Presumably if you designed the admission and information systems properly then schools in which children make more progress will expand (either on site, or on another site) due to increased demand This could either come from parents choosing higher performing schools or commissioners awarding contracts/charters to higher performing schools. Then, assuming the school makes a fixed profit on each student they ‘process’, they will increase their profit through increased market share. Student progress up > Market share up > Profit up.”

The key here is the word “presumably”. Yes – this is the standard dynamic market process. If this worked in schools, then this would make choice and competition more effective in raising quality. But it does not appear to work well, as we described here. Understanding the best way to reform the revenue stream for schools to encourage expansion is the important part; profit-making may eventually be part of an incentive mechanism, but is currently tangential to the main problem.

I’m an economist, I believe that incentives matter hugely. Indeed, many of the things that I write or say to the Department for Education involve the phrase “you need to make it matter more”. But that is about individual incentives: perhaps making the pay of Headteachers contingent on school outcomes, perhaps introducing some form of performance incentive for teachers.  These people can raise quality, and can be rewarded for doing so.

Within the present rules of the game, schools cannot be rewarded for raising quality, because the revenue they would receive is independent of quality. Clearly, profit-making schools can introduce individual performance incentives; but so can – and have – non-profit making schools. Again profit-making is a side issue. It’s the wrong battle to fight.

Should we have profit-making schools?

Author: Simon Burgess

Should we have profit-making schools?

Profit-making schools have returned to the education debate in England. This is an emotive issue for many, but an economic analysis is useful in defining the real issues.

There are some simple claims that can be quickly dealt with.

• “Education is far too important to be left to the mercy of profit-making companies.” Education is undoubtedly very important, for long-run growth, for social mobility, and for personal well-being. But think about possibly the most elemental of human needs, the production and distribution of food. While this is regulated by government, we are happy to leave all the decisions to profit-making companies. No-one seriously advocates the nationalisation of food.
• “It just won’t work.” It clearly does at a general level. Countries around the world, including those with well-regarded education systems such as Sweden, allow profit-making schools.
• “No-one should make money out of education.” Obviously they do at the moment: schools buy things from profit-making companies. This obviously has to be the case unless schools are going to start making their own books, desks and computers. So the real issues are (1) what kind of deal can schools get to minimise profiteering, and (2) what services are best bought in from outside as opposed to provided by the school itself

The appeal of allowing profit is the view that it makes decisions matter more. It provides strong rewards to organisations to innovate, to raise quality, and to do things more efficiently. Crudely, on a per-unit basis, organisations are pushed to improve quality and therefore revenue, or to reduce cost.

What would be the effects of this in the current education system in England? To answer this, we need to think about the parameters of the market.

Start with revenue. Schools get revenue for having students on the books. It is more or less a per-capita fee, albeit with some extras and some adjustment by the LA (for community schools). But to adopt the language of business, this money is for processing the students. The revenue that the school receives for each student depends not at all on the progress that the student makes.

This is central to the issue. Given the current system, there is nothing that profit maximising schools could do to raise their revenue per student by raising quality. Immediately, a great deal of the appeal of profit-making is removed.

The only way that schools could make profits is by driving down costs. This may be fine; it may be that this doesn’t really affect the quality of education if done in a smart way. If not done in a smart way, the quality of education would suffer and attainment would fall. It is clear that even the optimistic scenario does not improve education systemically in any way, either statically or dynamically through encouraging entry. The quality of education is the same, and the overall cost to the taxpayer is necessarily the same.

The counter-argument is that the pressure for profit might reduce slack enough so that the fall in costs allowed for profits and an increase in money spent wisely so that attainment increased. For this to work, it has to be that school budgets are spent very unwisely, and that an outside organisation could identify and cut ‘bad’ spending, take some profit and raise ‘good’ spending. It is certainly true that there is a huge amount of idiosyncratic variation in school financial decisions, variation that is unlikely to all be the result of optimal decision-making. Schools either know how to better deploy their budgets but are not sufficiently incentivised to do so, or they do not know. If they do not know, it is unlikely that outsiders will do (other schools may know; but that is another issue, only very clumsily mimicked by profit-making). Profit-making may answer the first point, but so do two other approaches, discussed below.

So profit-making is pointless at best: under the current market set-up, improvements in attainment would not make money (so would not happen) with profit-making schools, and cutting costs would make money but would either reduce attainment or leave it unchanged.

There are alternative strategies that might get some of the benefits of the innovative drive that profits might unleash, but in a more productive way: paying for attainment and incentivising cost reductions through resources for the school.

Paying for attainment. A positive step that keeps the current non-profit system intact but provides some of the same incentive is tying schools’ revenue to their pupils’ attainment. This would be straightforward to administer in principle, but there are some critical issues to resolve before it could be implemented. Chief among these is: should we pay for the simple ‘output’ of the school (GCSE points) or for pupil progress? There are good arguments both ways, to be visited in another post.  Of course, schools do much more than produce attainment, but this is the focus of policy.

Incentivising greater efficiency in other ways.   What if any surplus generated by this process had to be re-invested in the schools? Perhaps schools need some strong incentive to reduce costs. This might well be true, but this is not profit-making: profit-making by definition means the taking of monetary reward out of the school. An alternative scheme would be essentially equivalent to a team (school)-based incentive scheme in which the incentive is not money for the teachers, but resources for the school – resources saved are kept in the school. This is again potentially a good idea, worth looking at and some way short of profit-making.

Profit making in schools would either solve all schools’ problems nor signal the end of civilisation; the issue provokes strong feelings, but largely misses what should be the central policy concerns. Big gains in levels of attainment depend on raising average teacher effectiveness and big gains in equity depend on weakening the importance of proximity as an admissions rule and on changing the allocation of effective teachers across schools.  None of these would be strongly or directly affected by for-profit schools. However, there are certainly merits in piloting policies that link school’s revenue per student to the progress of that student, and incentivising cost reductions through keeping the surplus in the school.

Teacher performance pay without performance pay schemes

Author:  Simon Burgess

Teacher performance pay without performance pay schemes

Amid the macroeconomic gloom, the Autumn Statement contained a line about teachers’ pay. The School Teachers’ Review Body recommends “much greater freedom for individual schools to set pay in line with performance”. Consultations and proposals are expected in the near future.

But simply giving schools the freedom to do this may be a rather forlorn hope of anything much happening. It is not clear that there is a substantial demand from schools for performance-related pay (PRP) schemes that has only been thwarted by bureaucratic restrictions. It is hard to see high-powered, tough-minded PRP schemes being introduced by more than a handful of schools, not least because we have not seen large scale deviations from national pay bargaining in academies in England despite their new freedoms to do so.

If that path seems unpromising, there are other ways of facilitating a greater reflection of performance in pay, discussed shortly. But first – is PRP for teachers a good idea in the first place? Does it raise pupil attainment? What are the ‘side effects’?

This is a question that economists have produced a good deal of research on. And to summarise a lot of diverse work briefly, the international evidence is mixed. Those on both sides of the argument can point to high quality studies by leading researchers that find substantial positive effects, or no effects. In both cases, interestingly, there appeared to be little evidence of gaming or other unwanted effects of the incentives.

There is little evidence specifically for England. Our own research found a substantial positive effect of the introduction of a PRP scheme, but given the varied results found elsewhere it would seem unwise to place too much weight on this one study. The underlying performance pay scheme was poorly designed but nevertheless had a positive effect on the progress of pupils taught by eligible teachers relative to ineligible ones.

And design is key. There are many reasons why a simple high-powered incentive pay scheme might be detrimental to pupil progress, which we have discussed here and here. These include the fact that teachers have multiple tasks to do, the problems of measuring the outcomes of some of those tasks, the complex mixture of team and individual contributions, and the potential impacts on implicit motivation. The overall message is that incentives work, but schemes have to be very carefully designed to achieve what the schemes’ proponents truly intend.

There is another way to facilitate a closer link between pay and performance that does not require any school to introduce a performance pay scheme.

Published performance information in a labour market can change the way that the market rewards that performance. The critical features are first that the organisation’s own output depends in an important way on this performance characteristic of an individual; second that the organisation has some discretion in the pay offers it can make to new hires; and thirdly that the performance information is public – is available and verifiable outside the current employer. In this case, the pay structure of the market will reflect the performance rankings: high-performing individuals will be paid more.

In teaching, the first two of these three conditions are met: teacher quality matters hugely for schools, and schools have some discretion over pay. Now, suppose we had a simple, useful and universal measure of each teacher’s performance in raising the attainment of her pupils (obviously we don’t at the moment; I come back to this below), and that this was published nationally, primarily for the attention of Headteachers. The idea is that Headteachers trying to improve the attainment of their pupils would be on the look-out for high performing teachers when they had a vacancy to fill. Armed with this performance information, they might try offering a higher wage (or something else – it doesn’t have to be money) to tempt them to join their own school. Equally, the teacher’s current school may respond by raising the offer there.  Over time, this process will tend to raise the relative pay of high-performing teachers relative to low-performing ones, whom no-one is trying to bid for.

This idea should not be a strange one. A number of professions have open measures of performance. Just today it is reported that performance measures for more surgeons will be made public in the summer of 2013; this is already true for heart surgeons.

It is well-known that PRP does two things: it motivates and it attracts. The outcome for pay described here will tend to make teaching more attractive to people who are excellent teachers and less attractive to those who aren’t.

There are a number of problems with this idea, though perhaps less than might appear at first glance.  First, it could be argued that a performance measure derived from teaching in one school is not relevant to teaching in another school. Obviously each child and each school is unique, but it seems very unlikely that there is no commonality of context between one school and the next. Observation suggests this: teachers moving from one school to another are not counted as having zero experience, and Headteachers are often appointed from outside a school.

Second, there might be a fear that the teacher labour market would become chaotic, with everyone churning around from school to school in search of a quick gain. We have to recognise that there is substantial turnover of teachers now < http://www.bristol.ac.uk/cmpo/publications/papers/2012/wp294.pdf >. But the main point is that it does not require much actual movement to make the market work. Schools can make counter offers to try to retain their star teachers and the end result is the same – higher salaries for high-performing teachers.

Third, any measure would be noisy, partial and imperfect. Of course, all such measures are. Whether a measure is perfect is not really the question, the question is how noisy and imperfect is it, and whether it contains enough information to be useful. One advantage in this case is that the consumers of these performance indicators are the people best able to judge their usefulness and their shortcomings: Headteachers. If such metrics are not useful, Headteachers will simply ignore them; there would be no compulsion to use them.  Even in labour markets with some of the most detailed and finely measured performance indicators (for example, football or baseball) there are many moves between employers that do not work out. It is worth re-emphasising that these performance measures are bound to be imperfect and incomplete, but broad measures of performance may nevertheless be very useful.

There are useful parallels to be drawn from another profession: academics. For academics, the combination of very detailed and public performance information and a context where research performance matters a great deal to universities seems to have had a substantial effect on academics’ pay.

The Research Assessment Exercise (RAE) and more recently the Research Excellence Framework (REF) have made a strong research performance very important to a university’s standing and its income. But the critical factor for academics is that an individual’s research performance is public knowledge, through very detailed recording of the impact of their research papers. Departments and universities aiming to improve their ranking seek out star researchers and attempt to bid them away with higher salaries (plus other things such as research facilities). These offers may well be matched by their current employer, but the end result is that salaries now seem to be much more closely correlated with research productivity than before the RAE/REF (I say “seem” as there does not appear to be any evidence on this, so this is casual empiricism). This is a lot of what drives many young researchers to put in very long work hours: having a paper published in a top scientific journal early in a career has a substantial lifetime payoff even in a world with few or low-powered incentive schemes. If you check out academics’ websites you will invariably see their academic output prominently displayed.

Again, an important feature is that these indices of research output are largely consumed by other academics who are aware of their strengths and weaknesses. So although they are far from perfect, they are used by precisely the people best placed to calibrate their usefulness appropriately.

If we are to go down a path of tying teacher pay more closely to performance, and yet respect the rights of increasingly autonomous schools to determine their own pay systems, then this might be an option to consider.  The challenge is to devise a measure that is simple, useful and universal. It would measure the progress made by the pupils that teachers taught, it would have to deal with normal variations in performance by averaging over a number of classes and a few years, and be on a common metric.  This is not straightforward, but if it gave rise to a robust broad measure of performance it could form a part of performance pay for teachers, and performance management more broadly. It could also have substantial effects on the pay of high-performing teachers.

Who fails wins? The impact of failing an Ofsted Inspection

Rebecca Allen and Simon Burgess

What is the best way to deal with under-performing schools? This is a key policy concern for an education system. There clearly has to be a mechanism for identifying such schools. But what should then be done with schools which are highlighted as failing their pupils? There are important trade-offs to be considered: rapid intervention may be an over-reaction to a freak year of poor performance, but a more measured approach may condemn many cohorts of students to under-achieve.

This is the issue that Ofsted tackles. Its inspection system identifies failing schools and supervises their recovery. How effective is this? Is it even positive, or does labelling a school as failing push it to ever lower outcomes for its students?

It’s not clear what to expect. Ofsted inspections are often dreaded, and a fail judgement seen as being disastrous. It has been argued it triggers a ‘spiral of decline’, with teachers and pupils deserting the school, leading to further falls in performance. But it might also be a fresh start, with renewed focus on teaching and learning, leading to an improvement in exam scores. Equally, we might expect nothing much to happen: after all, the policy ‘treatment’ for those schools given a Notice to Improve is very light touch. It is neither strongly supportive (typically no or few extra resources) nor strongly punitive or directive (schools face no sanctions nor restrictions on their actions). Schools are instructed to focus intensively on pupil performance, and are told to expect a further inspection within a year. In addition – and possibly the most important factor – the judgement that the school is failing is public one, usually widely reported in the local press.

Our research shows that the Ofsted inspection system works. Schools that just failed their Ofsted significantly improved their performance over the next few years, relative to schools that just passed. The impact is statistically significant and sizeable. In terms of the internationally comparable metric of effect sizes, our main results suggest an improvement of around 10% of a standard deviation of pupil scores. This is a big effect, with a magnitude similar to a number of large-scale education interventions. Translated into an individual pupil’s GCSE grades, this amounts to a one grade improvement (for example, B to A) in one or two GCSEs. From the school’s perspective, the gain is an extra five percentage points in the proportion of pupils gaining five or more GCSEs at grades A*-C.

Our findings suggest that the turn-around arises from proper improvements in teaching and learning, not gaming to boost exam performance through switching to easier courses. First, the impact is significantly higher in the second year post visit than the first, and remains level into the third and fourth year after the inspection. So it is not simply a quick fix to satisfy the inspectors when they return twelve months later. Second, we find a stronger effect on the school’s average GCSE score than on the headline measure of the percentage of students gaining at least 5 good passes; if the schools’ responses were aimed at cosmetic improvement, we would expect the reverse. We also find similar positive effects on maths results and on English results.

It could be argued that these results are implausibly large given that the ‘treatment’ is so light touch and schools are given no new resources to improve their performance. The instruction to the school to improve its performance may empower headteachers and governors to take a tougher and more proactive line about school and teacher performance. This may not be a minor channel for improvement. Behavioural economics has provided a good deal of evidence on the importance of norms: the school management learning that what they might have considered satisfactory performance is unacceptable may have a major effect. The second part of the treatment derives from the fact that the judgement is a public statement and so provides a degree of public shame for the school leadership. Ofsted fail judgements are widely reported in local press and this is usually not treated as a trivial or ignorable announcement about the school. It seems plausible that this too will be a major spur to action for the school.

Where do we go from here? Our results suggest Ofsted’s identification of just-failing schools and the use of Notice to Improve measures is an effective policy, triggering the turn-around of these schools. We need to be clear that our research does not address the question of what to do about schools that comprehensively fail their Ofsted inspection. Possibly this light-touch approach can be extended. Since leaving the Headship of Mossbourne school to become the new Director of Ofsted, Sir Michael Wilshaw has argued that schools just above the fail grade should also be tackled: that ‘satisfactory’ performance is in fact unsatisfactory. Such interventions in ‘coasting’ or ‘just-ok’ schools are very likely to be of the same form as Notice to Improve. Our results suggest that this is potentially a fruitful development with some hope of significant returns.

This research is available on the CMPO website and the IoE website.

“Make Bradford British”: the parallel lives of the younger generation

Ellen Greaves  (IFS)

The high level of residential segregation for people from different backgrounds was highlighted as a factor contributing to the tension between people from different ethnic groups in the Cantle Report (2001) in response to the race riots in the same city in 2001. The Cantle Report identified that many people living in the same city led “parallel lives”; with little interaction with, or understanding of, people from different backgrounds.

Last night a Channel 4 programme explored the relationships between 8 residents of Bradford, brought together to interact with people from different backgrounds for the first time. The programme addressed the difficulties and benefits of over-coming entrenched prejudices, underlying (perhaps unacknowledged) racism in the city, and the similarities and differences between people from different cultures. The majority of participants in “Make Bradford British” found it a positive experience. Future improvements in relationships between different communities are likely to come through increased daily interaction, however, as well as the moving personal experiences witnessed on screen, and integration in schools can be a starting point.

Segregation in schools commonly results from residential segregation (as admission to schools is usually determined by proximity), but parents’ preferences can also play a role. Research by CMPO shows that pupils from most ethnic groups are more segregated in school than where they live, suggesting that there are factors, either parents’ preferences or other constraints, that restrict integration in England’s schools.

Pupils’ current level of integration in each area in England can be viewed on a website created by CMPO, funded by the ESRC and using national administrative data provided by the Department for Education. The figures for Bradford are startling: in 2009 around 80% of Pakistani pupils of primary school age attended a primary school that was primarily non-white, while just under 70% of white pupils attended a school that was primarily white. This suggests that these two groups of pupils are highly segregated in their primary schools, and the figures suggest that they remain so in secondary schools. Bradford is not alone in this high level of segregation (Oldham has similarly high levels), but the figures are higher than in other areas of the country with similar demographics; in Manchester under 60% of Pakistani pupils attended a school that was primarily non-white in 2009 and around only around 30% of white pupils attended a school that was primarily white.

Other indicators of segregation can also be compared: the dissimilarity index is a summary measure designed to capture the degree of “unevenness” between two groups of pupils (that is, the extent to which the school population reflects the wider population in the local area). Its value (between 0 and 1) represents the proportion of the group population that would have to change schools to achieve the same distribution of that of the local area, where, in general, a dissimilarity index of less than 0.3 is considered low, between 0.3 and 0.6 as moderate and above 0.6 as high (Massey and Denton, 1988).

Bradford has a decisively high level of segregation according to the dissimilarity index; in 2009 the figures show that around 70% of white and Pakistani primary pupils would have had to change schools in order for the distribution to reflect the wider population of the area, and the figure is similarly high for secondary schools.

There are encouraging signs, however; segregation according to the dissimilarity index has decreased marginally in primary schools since 2002, suggesting that the younger generation at least is becoming slightly more integrated, although levels remain higher than almost anywhere else in England.

The high level of segregation in Bradford’s schools suggest that integration and understanding between people from different backgrounds are unlikely to be fostered from direct experience in the education system, although there are some signs that the situation is marginally improving. What can make “Bradford British” and united remains unclear, but further integration of the city’s young people is not likely to harm the city’s prospects.

Notes on the author:

Ellen Greaves is a Research Economist in the education, employment and evaluation research sector at the IFS.

Why the new school league tables are much better … but could be better still

Rebecca Allen (IOE) and Simon Burgess (CMPO)

Tomorrow the new school league tables are published, with the usual blitz of interest in the rise and fall of individual schools. The arguments for and against the publication of these tables are now so familiar as to excite little interest.

But this year there is a significant change in the content of the tables.  For the first time, GCSE results for each school will be reported for groups of pupils within the school, groups defined by their Keystage 2 (KS2) scores. Specifically, for each school the tables will report the percentage of pupils attaining at least 5 A* – C grades (including English and maths) separately for low-attaining pupils, high attaining pupils and a middle group.  This change has potentially far-reaching implications, which we describe below.

This is a change for the better, one that we have proposed and supported elsewhere.  Why? We believe that in order to support parents choosing a school, league tables need to be functional, relevant and comprehensible. The last of these is straightforward (though not all league table measures in the past have been comprehensible: Contextualised Value-Added (CVA) being the perfect example). ‘Relevant’ means that a measure has some relevance to the family’s specific child. A simple school average, such as the standard whole-cohort % 5 A* to C, is not very informative about how one specific pupil is likely to get on there. By ‘functional’ we mean a measure that does actually help a family to predict the likely GCSE attainment of their child in different schools. If a measure is not functional it should not be published at all.

The new group-specific component is comprehensible and is more relevant than the whole-cohort %5 A* to C measure.  In our analysis of functionality, we show that it is as good as the standard measure, and much better than CVA.

It also addresses in a very straightforward way the critique of the standard league tables that they simply reflect the ability of the intake into schools, and not the effectiveness of the school.  By reporting the attainment of specific groups of students of given ability, this measure automatically corrects for prior attainment, and in a very transparent way. This is therefore much more informative to parents about the likely outcome for their own children than a simple average.  This of course is what value-added measures are meant to do, but they have never really become popular, and as we show they are not very functional.

However, the details of the new measure now published are problematic in one way. The choice of groups is important. We defined groups by quite narrow ten percentile bands, the low attaining group lying between the 20^th and 30^th percentiles in the KS2 distribution, the high attaining group between the 70^th and 80^th percentiles, and the middle group between the 45^th to 55^th percentiles. While clearly there is still variation in student ability within each band, it is second order and the main differences between schools in performance for any group will come from variation in schools’ teaching effectiveness.

However, the DfE has chosen much broader bands, and have defined the groups so that they cover the entire pupil population: the low attaining group are students below the expected level (Level 4) in the KS2 tests; the middle attaining are those at the expected level, and the high attaining group comprises students above the expected level.

This has one significant disadvantage, set out in detail by Rebecca Allen here. The middle group contains around 45% of all pupils, and so there is very significant variation in average ability within that group across schools. This in turn means that differences in league table performance between schools will reflect differences in intake as well as effectiveness, even within the group, thus partly undermining the aim of group-specific reports.

The chart below illustrates this for the middle attainment group (see here for more details).  Each of the three thousand or so tiny blue dots shows the capped GCSE attainment for a group of mid-attaining pupils (on the DfE’s measure of achieving at the expected level at KS2) against the average KS2 score (i.e. prior attainment) of pupils at the school. The red dots plot the same relationship for our narrow group of middle attainers (the 45^th to the 55^th percentile). The chart shows very clearly that the performance among our narrow band is essentially unrelated to prior attainment, but the DfE measure for the very broad group does still favour schools with higher prior ability pupils.

We can speculate as to why the DfE chose to have much broader groups. There may be statistical reasons, pragmatic reasons or what can be termed “look and feel” reasons. Using narrow KS2 bands will correctly identify the effectiveness of the school, but will almost always be averaging over a small number of students. So the estimates will tend to be “noisy”, and induce more variation from year to year than averaging over bigger groups. The trade-off here is then between a noisy measure of something very useful against a more stable measure of something less useful. Our original measure was intended to balance these, the DfE have gone all the way to the latter.

A pragmatic reason is that some schools may not have any pupils in a particular narrow percentile band of the KS2 distribution. The narrower the band the more likely this is to be true. This would mean either null entries in the league tables, which might be confusing, or some complex statistical imputation procedure, which might be more confusing. The broad groups that cover the entire pupil population are likely to have very few null entries. Finally, the broad groups feel more ‘inclusive’, they report the performance of all of a school’s students. This is a red herring – the point of the tables is to inform parents in choosing a school, not to generate warm glows.

The new measures hold out the promise of improvements in two areas: choices by parents and behaviour by schools. Parents will have better information on the likely academic attainment of their child in a range of schools. Second, parents will be able to see more directly whether school choice actually matters a great deal for them: whether there are worthwhile differences in attainment within the ability group of their child.

The key point for schools is that performance measures have consequences for behaviour. If this new measure is widely used, it will give schools more of an incentive to focus across the ability distribution. It is still the %5 A* – C measure that is the focus of attention for each group, but now schools will have to pay attention to improving this metric for high and low ability groups as well as simply the marginal children with the highest chance of getting that crucial fifth C grade.

If one believes that gaming and focussing of resources within schools is a very big deal (and there is little quantitative evidence either way) then the new measures could have a major impact on such behaviour. Even if such resource focussing is second order, performance measures send signals on what is valued. These new league table measures will explicitly draw widespread media and public attention to the performance of low- and high-ability children in every school in England.

A Report of Two Halves

Simon Burgess

We published some research last Friday showing that students perform less well in their crucial GCSE exams in years when there is a major international football tournament taking place at the same time. For example, the FIFA World Cup in the summer of 2010, or the UEFA European Championship next summer, both overlap in part with the GCSE exam timetable.

With the draw for the groups in the European Championship taking place earlier that day, much of the comment naturally and sensibly focussed on the specific issue of the impact of next year’s tournament on exam scores. This is important: we estimate that the concurrence of the exams and saturation media coverage of the football reduces exam scores on average by around 0.12 standard deviations of pupil performance and by a lot more for some groups who reduce their effort a lot. These groups tend to be from poorer areas and predominantly (but by no means exclusively) male students.  Since these groups are already lower-performing groups, this means that education gaps will widen. We think of this impact arising through a reduction in student effort, with that time being spent instead on watching the football tournament. The variation in impact arises because of differing tastes for football, arising in turn from cultural norms and idiosyncratic factors, and from the differential effectiveness of an hour of study on exam performance.

However, there is also a broader significance to the research: finding that effort matters matters.

Recent research by economists has broadened out from the previous focus on cognitive ability, and a great deal of work has investigated the role of non-cognitive factors in educational attainment. Non-cognitive factors can be identified with personality traits (see Heckman), and one of the ‘big 5’ personality traits is ‘conscientiousness’, with the related traits of self-control, accepting delayed gratification, and a strong work ethic. Conscientiousness has been shown to be an excellent predictor of educational attainment and course grades. These aspects of self control and ability to concentrate are clearly related to the broad notion of effort we are using here. Our results on the importance of effort strengthen this evidence by isolating the effect of decisions on effort and time allocation in addition to the general ability to concentrate and exert self-control.

There is a great deal of policy interest in England arising from recent studies of US Charter schools with what is called a “No Excuses” ethos. This includes the KIPP (Knowledge Is Power Program) network of schools and schools in the Harlem Children’s Zone. These schools all feature a long school day, a longer school year, very selective teacher recruitment, strong norms of behaviour, as well as other characteristics. Some of the profession’s very top researchers have produced evidence showing that such schools produce very powerful positive effects on student achievement. While this overall effect could be due to different aspects of the KIPP/HCZ ethos, part of it is very likely to be increased effort from the students. Our results complement this by showing the impact of just a change in effort, and that that can have very substantial effects.

This matters for a number of reasons. First, unlike genetic characteristics, cognitive ability or non-cognitive traits, effort is almost immediately changeable. Our results suggest that this could have a big effect. The fact that we find changes in student effort to be very potent in affecting test scores suggests that policy levers to raise effort through incentives or changing school ethos are worth considering seriously. Such interventions would be justified if the low effort resulted from market failures due to lack of information on the returns to schooling, or time-inconsistent discounting.  Second, the importance of a manipulable factor such as effort for adolescents’ educational performance provides evidence of potentially high value policy interventions much later than “early years” policies. This is encouraging, offering some hope that low performing students’ trajectories in life can perhaps be effectively improved even after a difficult environment early in life.

What are Free Schools for?

Rebecca Allen and Simon Burgess

As children start their lessons in the 24 Free Schools opening this week, a new experiment begins in English education. The founders and staff will have been working hard for this day over many months and no doubt all will wish the pupils and staff well. There has been a lot of political passion on both sides of the debate, but what is the significance of the Free Schools experiment likely to be?

It is not an experiment because of the “free” part – Free schools will enjoy essentially the same freedoms as Academies do.  It is an experiment because now anyone can propose to set up a school, and attract the same per-pupil funding from the state as other local schools. Anyone can propose … but the vetting process to check whether the applicants are “fit and proper” to run a school is (rightly, in our view) so stringent that around 90% of the original applications were turned down.

Will the experiment work? Work for whom? The focus cannot just be on the 5000 or so pupils starting now, less than a twentieth of 1% of their cohort. If so, this would be an incredibly expensive and high profile way to change the education chances of such a small number.

The focus has to be on the systemic impacts of the reform. What impact will the new option of opening a Free School have on the local schools?

The main idea, the big hope for the policy, is that this will raise the “competitive threat” to under-performing local schools. Economists talk about market entry: the theory that the threat of new suppliers entering a market works to keep the incumbents efficient and responsive. Any un-served niches or excessive costs would draw in new companies to offer a better deal.  Arguably, the prime function of the Free Schools experiment is to make the schools market “contestable”. This might sharpen the efforts of schools to raise their game to avoid losing their pupils to the new schools.

This might work. It is conceivable that really poorly performing schools may not wish to see their school supplanted by a new entrant, and push to raise attainment standards. But the UK evidence on the effects of competitive pressure on school effectiveness is not encouraging. Put at its best, there is only very weak evidence for small effects of competition. This is not true in every country, nor in every service in the UK, and there appear to be important market structural reasons why competition does not work to raise standards in schools.

This is not at all to say that all competitive mechanisms are pointless. Clearly any school system needs a strong policy for dealing with failing schools, something to act as a discipline on the performance of Headteachers and school governors. Free schools offer this in principle: in extremis, parents very dis-satisfied with their state school can opt out and set up their own school. But there are two reasons why Free Schools are unlikely to be the best answer to this. First, there are very significant set-up costs, both in time and energy from the founders, but also in the straightforward sense of acquiring premises.  While currently these are being funded (very generously) by the government, this simply cannot continue if the policy matures and spreads. But secondly, it seems inconceivable that any local area with one Free School would be offered the resources for any others. So as  discipline device, this is a one-shot game, not an on-going continued pressure on low performing schools, which is what is needed.

Despite the first rush of enthusiasm, it seems unlikely that Free Schools will act as a major stimulus to systemic higher performance. Perhaps the main problem was not the barriers to entry preventing many highly competent and motivated school-starters from setting up. In which case, we will need other policies to address poorly performing schools in disadvantaged neighbourhoods.

There are other important potential systemic effects, interactions between Free Schools and their neighbouring schools.

The first is school admissions. Free Schools will take part in coordinated admissions and, although, like Academies, being their own admissions authority, they will have to adhere to the Admissions Code. The big issue is whether the founders have any special rights to get their own children into ‘their’ school. It may be that this is one aim of the clause in the new draft Admissions Code allowing schools to give admissions priority to the children of “staff”. This is a difficult issue: it is quite understandable that those putting in time and effort to set up a new school want to get their own children in. But there are dangers: if this becomes the norm, we have a new route for particular individuals to get their child into a good school, by founding or acquiring a stake in, a Free School.  In any case, the admissions criteria and the highly related question of how to pass on a “stake” in a Free School need to be clarified.

Another important spill-over effect concerns funding. The on-going current expenditure of Free Schools is funded on the same basis as Academies, so is not much different from maintained schools (just differentiated by not having to pay the local authority for central services). But the capital funding is another matter. Funding the acquisition and refurbishment of premises, estimated by the DfE at £120m, has been a substantial drain on a much-reduced schools capital budget. The funding for Free Schools’ buildings is a direct diversion of funds from other capital funding projects, many in far more needy schools and disadvantaged areas than the Free Schools.

But finally what Free Schools are likely to offer is scope for pedagogical innovation. This will not always be the case – some of the groups setting up schools may have rather conservative ideas on what constitutes a good education. But others will likely be set up by innovators with radical ideas on how to teach.

So perhaps that’s what Free Schools will end up being for: not really working as a spur to higher standards but acting as incubators for radical new teaching ideas. Very valuable indeed, but perhaps not what everyone was expecting.

Revising the Draft School Admissions Code

Rebecca Allen and Simon Burgess

From next week, officials in the Department for Education are going to be busy sifting through responses to the consultation exercise around the new School Admissions Code.

Two important issues in the proposed code relate to the priority given to school staff, and to random allocation. We believe that as they currently stand, these provisions will set back the goals that the Government has set for its education policy.

1. Prioritising the children of staff

Paragraph  1.33 of the code says: “If admissions authorities decide to give priority to children of staff, they must set out clearly in their admission arrangements how they will define staff and on what basis children will be prioritised.”

This suggests that admissions authorities are to be allowed to prioritise the children of staff, reversing the policy of recent Admissions Codes.

One group very likely to be included in most definitions of “staff” are teachers. For those teachers with children, this will add a new aspect to their decision on which school to seek a job at.  Like many other parents, teachers will be keen for their children to attend high-performing schools.

Following the White Paper “The Importance of Teaching”, one of the leading education policy issues is how to attract the particularly effective teachers into the more challenging schools. Research evidence does not tell us whether teachers who are parents are on average more effective teachers, but there are two points to make:

1. This policy change will differentially increase the flow of applicants to high-performing schools. If the Headteachers of those schools are skilled at spotting effective teachers, then simply having access to a much bigger applicant pool will raise the average effectiveness of teachers the hired at those schools.
2. They are less likely to be novices, which is one of the few clear findings on teacher effectiveness, so in that sense alone teachers who are parents are likely to be more effective.

Given that, this policy change is very likely to work against any efforts to attract effective teachers to challenging schools, and thus set back the Government’s stated educational policy goals of narrowing the outcome gap between affluent and disadvantaged pupils.

The proposed code change will also complicate disciplinary procedures because firing a teacher from a school would also have implications for her/his children. This is likely to make it even less likely that headteachers will engage in robust performance management.

We know that any work-based privileges that are specific to particular establishments tend to cement people in that job and reduce turnover. Such privileges include health insurance, pension rights, and so on. This reduces labour mobility and typically will make the labour market less efficient. This proposed change will have the same effect in the teacher labour market as teachers will be less willing to move as it will disrupt their children’s education.

The core issue is where the most effective school staff work. It would clearly be in high-performing schools’ interests to define staff quite widely to attract highly motivated and effective governors and other staff. School leadership is key to excellent schools, ever more so as more schools become autonomous. If the children of governors were also prioritised for admission, then we would likely see an improvement in the quality of governors at high-performing schools and a decline at more challenging schools.

These points are about the effectiveness and efficiency of schools. There is also a fairness point about equity of access to high-performing schools. Some schools might define “staff” quite widely to include Teaching Assistants, lunchtime supervisors, and governors as well as teachers. In such cases, becoming one of those staff is an attractive route of access into the school, as an alternative to buying a nearby house. In extreme circumstances, individuals could offer to work for free as lunchtime supervisors, or could offer to provide goods or services to the school to become governors. While many may apply for such positions, schools will naturally choose the more able and articulate applicants, particularly those with a professional background. This will not help to foster more equal access to high-performing schools.

It may be that the provision to allow this prioritisation of staff was a way of finessing the problem of how to guarantee that the founders of free schools can get their children into ‘their’ school. If so, it might solve a problem affecting possibly 5% of pupils at the cost of the problems outlined here for the rest.

2. Random allocation

Paragraph 1.28 says “Local authorities must not use random allocation as the principal oversubscription criterion for allocating places at all the schools in the area for which they are the admission authority.”

This paragraph bans the use of lotteries as the main mechanism for resolving over-subscription across an LA. The problem of who is admitted to the high-performing over-subscribed schools is highly relevant to the Government’s goal of closing the outcome gaps between affluent and disadvantaged pupils.

The Government has stated that the primary goal of its social policy is to raise social mobility. One of the central ways in which where you were born influences your life chances is through the assignment of children to schools, governed by the Admissions Code.

Currently, the main mechanism to resolve over-subscription is proximity. It is clear that the widespread use of the proximity rule widens the socio-economic gap in the available quality of schooling. In fact, the gap in accessible school quality between rich and poor families widens by over 50% once a proximity criterion is imposed.

We can illustrate this straightforwardly, using data from the annual Census of all state school pupils in England. We can approximate the neighbourhood of a pupil as the area within 3km of her home, and calculate the average difference in the academic performance of schools in the neighbourhood of poor and rich pupils. As we would expect, more affluent families tend to live in neighbourhoods with higher quality schools. We can also use the database to estimate an approximate proximity criterion, and to look at the schools that a particular pupil could reasonably expect to be admitted to. The difference in school “quality” is now much starker, over 50% higher in fact. This is the impact of the proximity rule, and operates over and above the simple fact that rich and poor live in different places.

This shows the mean academic quality of primary school attended (measured by the mean Keystage 2 score achieved in that school) available to high and low socio-economic status (SES) families in England. ‘Neighbourhood’ is defined simply as an area within 3km of the student’s house; ‘Catchment’ is defined as schools that the student would have a very high chance of getting into based on residential location and the school ’s catchment area. Standard error bars are displayed on the levels.

Clearly some criterion has to be used to resolve over-subscription. One possibility for use in cities is a lottery. But by its very nature, a lottery ensures that places are allocated in a way that ignores social background. All those families who have applied to an over-subscribed school are entered into a ballot, and as many names are drawn out as there are free places available (that is, once looked-after children, children with special educational needs, and siblings of current pupils have been assigned slots).

It is clear that lotteries are not a panacea and are not without difficulties, as our own research has shown. However, they do offer one potentially important way of raising social mobility.

We wait to see which way the new Admissions Code turns in the end.