Non-standard matches and charitable giving

Author: Michael Sanders

Non-standard matches and charitable giving

The use of incentives to encourage charitable donations is commonplace. The governments of most developed countries (including the UK), as well as many large employers, offer match rates, or rebates to encourage donations. The principle behind such match rates is simple. With Gift-Aid, the UK’s main form of tax effective giving, a donation of £1 from net-of-tax income by a taxpayer attracts basic rate tax relief, which goes straight to the charity – hence, the charity receives £1.25 for every £1 donated.

The effect of this is to reduce the price of donating a given amount – if I am a basic rate taxpayer, and think “I want the British Heart Foundation to receive £100”, making that happen will only cost me £80 from my net of tax spending.

There is much discussion, both in the academic economics literature and in the public sphere, about the effectiveness of such matches, particularly in the aftermath of last year’s proposed £50,000 cap on charitable tax relief – although most studies find that people’s donations are relatively unresponsive to changes in the match rate (they are price inelastic), CMPO research has shown that high value donors may be more responsive to changes in their match (or rebate) than are smaller donors.

This mixed evidence on the effectiveness of matches at increasing out-of-pocket donations suggests that alternatives to the standard match may be more effective. Options for non-standard matches are summarized in our new working paper, which draws on the behavioural economics and psychology literatures to several such possibilities, including, non-linear matches, where the more an individual donates, the higher the match rate; social and team matches, in which the match rate received by one donor depends on the donations of others, giving them an incentive to ‘crowd in’ their friends and colleagues; competitive matches , when only the most successful fundraisers receive a match); and lottery matches, where each donation increases the chance of a donor’s chosen charity receiving a large windfall match.

Although these suggestions are sound in principle, and supported by both theory and empirical evidence, none has been experimentally tested in a real world setting. We would encourage anybody interested in testing one or more of these novels matches to contact us.

 

Justifying ‘Never-Working Families’?

Author: Lindsey Macmillan

Justifying ‘Never-working Families’?

The Secretary of State for Work and Pensions, Iain Duncan Smith, has repeatedly made references to families where generations never work. When asked to justify these claims (to Paul Goggins MP), he concedes this is based on personal observations, not evidence.  His justification for this is that statistical information on the number of UK families that never work is not available.

A recent Freedom of Information (FOI) request makes the same argument – but then goes on to cite my own research;

“a research report was produced by the University of Bristol in 2011 
looking at this issue. This reports that there are pockets in Britain where there are two and three generations of families who are unemployed.”

In fact, there is clear evidence that shows how rare a phenomenon the never-working family is.

In my paper in Dec 2011, I looked at the number of households where two generations had never worked. Evidence from the Labour Force Survey, which is used by DWP in their labour market statistics analysis, showed that in Spring 2010, only 0.3% of multi-generational households were in a position where both generations had never worked. That’s just 15,000 households in the country. Of these, in 5,000 households the younger generation had only just left full time education, within the last year, and so had barely had a chance to work yet.

This story holds in data where the families don’t live together in the same house (as seen in the National Child Development Study, the British Cohort Study and the British Household Panel Survey). There is very little evidence of even two-generation-never-working families, driven by the fact that so few of the younger generation are never in work (less than 2% by age 23 and less than 1% by age 29 – see here). Instances of three-generation-never-working families would be even rarer.

A recent Joseph Rowntree Foundation Study took the more direct qualitative approach and sent researchers out into deprived neighbourhoods in Glasgow and Middlesbrough to look for families where three generations had never had a job. They couldn’t find a single one.

Iain Duncan Smith isn’t the only senior politician to make these claims – Tony Blair said in 1997 that “Behind the statistics lie households where three generations have never had a job” (see p16 of this). But my work and others’ shows the evidence doesn’t match the rhetoric.

Education spending, pupil attainment and causality

Author: Simon Burgess

Education spending, pupil attainment and causality

In these hard times, spending government money effectively is more important than ever. Last week Fraser Nelson challenged the effectiveness of spending in schools, one of the areas relatively protected from Coalition cuts. He said: “The biggest surprise, though, was the money: no matter how you split the figures, the amount spent didn’t seem to make the blindest bit of difference”, his reading of a report by Deloitte commissioned by the Department for Education.

What is the evidence? In fact, it is surprisingly difficult to establish the impact of spending more money on student achievement. This is partly shortage of data (researchers always want more data), but there is a more fundamental reason too.

Perhaps inadvertently, Fraser Nelson illustrated the difficulty in his first paragraph. He noted the variation in per-pupil expenditure “ranging from £4,500 in Lyme Regis to £10,000 in Salford.” This is absolutely right – there are very significant variations in revenue per pupil. But the key point is that these are not random: extra resources are explicitly and systematically directed towards schools in poorer neighbourhoods. The mechanism, accreting the new schemes of each successive government, may be incomprehensibly complex, but the intent is surely right.

Getting back to our question, on the one hand we have this systematic distribution of resources towards poorer neighbourhoods. On the other hand we know that pupil attainment is typically lower in schools in such neighbourhoods; not for every pupil, not in every school, but on average. So if money has no impact on attainment, and we line up pupil attainment and school expenditure, we will tend to see a negative relationship. This derives solely from the way that money is distributed to schools. The fundamental problem is that there are two things going on with opposite effects: low attainment is associated with more money (via the schools funding system) and more money may be associated with high attainment (via the education process). With no other information, there is simply no way of disentangling these two opposing effects, and by itself these numbers can tell us nothing about the causal impact of school expenditure on pupil attainment.

So the view that “the amount spent didn’t seem to make the blindest bit of difference” cannot be supported by this evidence.

What of the wider research evidence, based on studies with a plausibly causal research design? One of the most prominent economists in the field of education, Rick Hanushek from Stanford, is famously sceptical of the value of greater resources for schools. There certainly are studies that show money can matter, but it is probably fair to say that the majority view among economists is that simply providing more resources for schools is not the best option.

The really interesting question is this: why doesn’t more money raise attainment? More money usually helps most things. Either there simply is nothing that schools can buy that raises attainment. This seems unlikely, and would certainly be a surprise to parents paying many thousands of pounds to send their children to private schools. Or there are features of the system which lead schools to spending extra resources on the ‘wrong’ things – things that have little impact on attainment. This might be the manner in which the money is distributed by government (typically short-term, making long-term expenditure decisions risky); or the regulations and agreements governing its spending by schools; or other factors. We have speculated a little about this here. 

Coincidentally, the Department for Education has just opened a consultation  on school efficiency – they await your views.

Does ‘the gap’ matter to children eligible for free school meals?

Author: Rebecca Allen

Does ‘the gap’ matter to children eligible for free school meals?

David Laws, the Liberal Democrat Minister for Schools, has been making a series of speeches over the past month about “closing the gap” in the attainment between pupils from deprived and more affluent backgrounds. Yesterday, he warned that schools should not be judged as outstanding by Ofsted if they failed to close the gap, a goal that sounds fair and even laudable in principle, but I believe is rather unfair in practice.

The “gap” is the difference in GCSE achievement between the average for pupils who are eligible for free school meals and the average for those who are not. Pupils eligible for free school meals have similar characteristics across schools since they all come from families claiming some sort of benefit. The problem is that the background of pupils who are not eligible for free school meals (FSM) will vary considerably across schools, since the group includes both those with bankers and with cleaners as parents.

A school may substantially narrow the gap by working hard to improve the attainment of their most deprived children, or through the accident of the characteristics of their non- eligible children. I have written before that very deprived schools tend to have very small gaps, a quirk I first spotted as a governor of a school that was struggling to produce strong academic results but was very proud that its FSM gap was zero. All the students at the school came from low income families living on a very large and universally deprived council estate. Some of the families happened to claim benefits that made them eligible for free school meals (not necessarily the poorest), others didn’t or couldn’t. Not surprisingly, the GCSE performance of the FSM and non-FSM pupils in this school were no different, on average, because these pupils were no different in their social or educational background. Nothing the school was doing contributed to this supposed “success” in “closing the gap”.

Social class disadvantage is an important national policy problem but I do not believe that schools should be deciding policies based on the size of their FSM attainment gap. What matters to children from low-income families is that a school enables them to achieve a qualification to get on in life. If a low-income student gets a low quality education from a school, it is little consolation or use for them to learn that the school served the higher income students equally poorly (i.e. the school’s “gap” was small).

As it turns out, great schools tend to be great schools for all children in the school – the statistical correlation between who does well for FSM children and who does well for non-FSM children is very high. Moreover, schools can make a difference to the life chances of FSM children – there are huge differences in attainment for these children across schools, far larger than there are for children from wealthy backgrounds who do pretty well in all schools.

Should the pupil premium be used exclusively for FSM children?

In an earlier speech David Laws threatened to ring fence the pupil premium money to force schools to spend it on their FSM pupils, rather than throwing it into the general pot of schools money. (The pupil premium is the policy tool by which schools are supposed to “close the gap”). I worry about this type of restriction in spending on certain activities, not least because the FSM children are not always from the poorest families in the school – the very act of claiming benefits means some family income leapfrogs those of others that work. More practically, students eligible for free school meals are not segregated into special classrooms, so it is rather difficult to spend the pupil premium on important things such as teaching and learning without the benefits spilling over to other children who sit in the classroom with them.

Free school meals children do not have educational needs that are unique or particularly distinctive. Like other children, excellent teaching and support for their learning is a good place to start. I do not deny that social deprivation in the home spills into schooling and it is clearly possible to target this via attendance incentives, breakfast clubs, homework clubs and tutoring support. But I believe schools would find it morally and emotionally difficult to exclude a child who is not on FSM from accessing these schemes if they could clearly benefit from them.

I hope that my very specific criticisms of an admirable policy do not detract from the urgency of understanding and reducing the size of social class gaps in educational achievement in England. Schools must engage in the evidence on resourcing and pupil achievement and make careful, deliberate budget decisions that improve pupil learning, particularly for those who are most educationally disadvantaged in their school. This may mean spending the pupil premium to retain the best teachers, introduce policies to improve teaching practice or even an “approved” activity targeted very specifically on more deprived children in the school.

However, I suspect that in trying to support the attainment of FSM-eligible children, a school might inadvertently help those not eligible too. Is this such a bad outcome? And if not, why do we use a “gap” metric that “punishes” a school for improving the attainment of those not eligible, as much as it does allowing the attainment of more deprived children to fall behind?

This post was first published on the IOE London Blog 

Marathon Fundraising

Author: Sarah Smith

Marathon fundraising

The 2013 London marathon is in less than a week’s time. The hard slog of training behind them, most runners will be easing up, resting their legs before the real thing. Over the last few months, many will have sought training tips to improve their running, but what about the thousands of runners who are using the London marathon as a way of raining money for charity? What are some of the top tips for how they can achieve a fundraising personal best?

David Reinstein from Essex University has started an interesting discussion of research that fundraisers might find relevant. At CMPO, we have also begun looking at the rich data available from online fundraising website Justgiving.com to gain further insights into charitable fundraising.

The good news for marathon runners is that they have chosen a good event. Compared to fundraisers doing other events – cycling, parachuting, birthdays and other running events – London marathon fundraisers tend to have more donors (29 is the typical number) and raise more money overall (between £800 and £900 is typical).

Most marathon pages have a fundraising target. This seems to work. Compared to having no target, pages which specify a target amount get more donations and higher average donations. Donors clearly respond to the target – when the total amount raised gets close to the fundraiser’s goal, donors increase their donations to hit the target, but they give less once the target has been reached. With this in mind, raising the target could be one way to keep donations high.

Donors are also responsive to how much others have given. Arriving at a fundraising page, each donor can see how much other people have given and this information affects how much they give. A generous donation to a page – particularly made early on – can have a positive effect on later amounts given. Our results suggest that a one-off donation of £100 leads to around a £10 increase in the amounts given subsequently. Some of the effect comes through the fact that this kind of large donation triggers other people to match it; but there is also a wider, positive effect. Small donations, on the other hand, result in subsequent donors giving less. So, fundraisers have an additional incentive to encourage their friends, family and colleagues to be generous because of the knock-on effect that donations have on how much other people give.

Most people raising money by running in the London marathon will be relying on donations from their friends, family and colleagues. One question that Kimberley Scharf and I are looking at is how important the size of the person’s social group is – and whether you can be a successful fundraiser even if you have a relatively small social network. We capture the size of a fundraiser’s social network by the number of their Facebook friends. This may not be a perfect measure of the potential donor pool but many fundraisers do post to their Facebook pages. We find – perhaps surprisingly – that larger social networks do not translate into higher total amounts raised. Although fundraisers with more friends tend to get more donations, each donor in a large social network typically gives less, meaning that the total amount raised is roughly the same for those with a thousand-plus friends as it is for those with less than a hundred. This is true controlling for age and income and is also true for pages without a target.

There are a number of explanations for why this might be the case. One possibility is that, in a larger group of donors, each person’s donation matters less to the fundraiser and so the donors give less. It may also be the case that there are weaker relationships between the fundraiser and their friends in larger social networks. By digging further into the fundraising data, we hope to get a better understanding of the relationships between fundraisers and donors and provide further insights into what determines fundraising success.

Profits in schools – response

Author: Simon Burgess

Profits in schools – response

There were some interesting comments on the recent post I wrote on profit-making schools. This post offers a brief reply to those points.

First, one comment was that allowing profit-making is a solution to the lack of capital for schools:

“advocates see profit-making as a way to tap the private finance that might allow supply-side liberalisation, which would in turn allow choice to operate more effectively than it does at present. Theoretically, of course, this boost to capacity could be done with public finance. But it’s questionable whether the necessary level of spare capacity would be politically sustainable given all the other calls on public spending (especially now). So private finance is (arguably) one solution to that problem.”

It may be a solution to that problem, but it is not a necessary solution, there are other ways. The PFI programme has been funding capital spending on schools for over a decade now.  Nor is it just a thing of the past: in 2011 Michael Gove announced capital expenditure through PFI of around £2bn to rebuild 300 schools. The latest estimates are that PFI expenditure on education will top £260m in 2012-13, and the whole programme has generated over £7bn for school building. The PFI obviously utilises the profit motive in the capital market to get funds into school building without needing profits in the schools themselves.

Second is the question of just how profits can be made. Given fixed revenue per student, it is not possible to directly make a greater rate of return by raising quality (the indirect route is discussed below). Profits can be made by reducing costs. This may be possible without reducing quality, or not. That possibility is that other agents can come in, re-arrange the budget, reduce costs and maintain quality by raising quality per pound spent. The comment was:

“You also argue that ‘outsiders’ are unlikely to know best how best to deploy their budgets. This seems like an odd argument. The market’s virtue is supposed to be innovation and the ability to scale good practice quickly through incentives to mimic the best. If you don’t think that works then I can’t see why you’d be interested in the practical aspects of for-profit schools, since there wouldn’t even be any benefits in principle.”

It is certainly true that schools are unlikely to be making completely optimal decisions. Our own work shows a huge degree of heterogeneity in schools’ financial decisions which is very unlikely all to be optimal. So they certainly have scope for learning. And schools may be able to learn from each other: a lot of people interpret the success of London schools as down to ‘London Challenge’ – and a lot of people interpret the success of that to collaboration, to learning from other schools. In fact, we are in the design stage of a large-scale RCT to test this out. But the key point is that with the current system for school revenue, allowing profit-making provides incentives to reduce costs but no direct incentive to raise quality. So again profits might be a way of encouraging collaboration, but there are other, easier, ways of doing the same thing.

The indirect channel for profit making to affect quality is a dynamic one. The third comment is:

“Presumably if you designed the admission and information systems properly then schools in which children make more progress will expand (either on site, or on another site) due to increased demand This could either come from parents choosing higher performing schools or commissioners awarding contracts/charters to higher performing schools. Then, assuming the school makes a fixed profit on each student they ‘process’, they will increase their profit through increased market share. Student progress up > Market share up > Profit up.”

The key here is the word “presumably”. Yes – this is the standard dynamic market process. If this worked in schools, then this would make choice and competition more effective in raising quality. But it does not appear to work well, as we described here. Understanding the best way to reform the revenue stream for schools to encourage expansion is the important part; profit-making may eventually be part of an incentive mechanism, but is currently tangential to the main problem.

I’m an economist, I believe that incentives matter hugely. Indeed, many of the things that I write or say to the Department for Education involve the phrase “you need to make it matter more”. But that is about individual incentives: perhaps making the pay of Headteachers contingent on school outcomes, perhaps introducing some form of performance incentive for teachers.  These people can raise quality, and can be rewarded for doing so.

Within the present rules of the game, schools cannot be rewarded for raising quality, because the revenue they would receive is independent of quality. Clearly, profit-making schools can introduce individual performance incentives; but so can – and have – non-profit making schools. Again profit-making is a side issue. It’s the wrong battle to fight.

Money for Good?

Author: Sarah Smith

Money for Good? 

On Thursday, New Philanthropy Capital published a new report, Money for Good UK, about why people give and what might be done to encourage them to give more. The very next day, the 25^th Comic Relief raised a record-breaking £75 million for charity and in doing so, served to illustrate a number of the key themes from the report. One of the messages from the report is that around half of donors feel no underlying obligation (moral or otherwise) to give money to charity. 47 per cent of those asked (mainly people who had given money to charity in the previous year) agreed that people should donate money to charity if they have the means. But nearly the same number (44 per cent) agreed that people should not feel obliged to donate money to charity. Comic Relief is a testament to the efforts of literally thousands of people to persuade other people to give to good causes.

So, what does persuade people to give? The report found that the personal touch is important – whether personalized communications from charities that donors have relationships with or an approach from friends, family and colleagues – and perhaps the odd famous celebrity.

People also say they care about impact. When asked about what factors they pay attention to when they give to charity, the most important is how the organization will use their donation (with 63 per cent of donors pay close or extremely close attention) and evidence that the organization is having an impact (with 58 per cent paying close or extremely close attention. This mirrors similar findings from a recent survey done on Justgiving donors. It is certainly something that Comic Relief understands – interspersing the comedy with clips showing where the money is going and how it is going to be spent.

It is a commonly held view that giving isn’t a rational decision. Providing evidence that donors care about impact and evidence of impact is therefore important. And it helps charities and fundraisers think about what kind of messages they need to deliver. But, it needs probing a little further. What does impact mean to donors and what kind of evidence do they want? On one level, it is clear that donors don’t want their money to be wasted, but do they really want to be presented with detailed accounts or impact studies? The report is careful not to push this too far – they emphasize that few donors do detailed research before giving to charity. Indeed, one possible explanation for the importance of personal recommendations is that friends and family are effectively vouching for the quality of the charity and acting as a substitute for the donor’s own research.

Following the report, NPC have emphasized that charities need to do more to demonstrate impact. But, another important next step is to find out more about what kind of information donors want – and how they respond to messages about charity impact and different types of evidence. There has been some limited work in this area showing that donors respond better to simple messages about personal cases rather than detailed statistics. These findings come from simple field experiments in which donors are randomly given different information to test the effect of alternative messages. More worryingly, the experiments show that donations actually appear to decline if donors are given, alongside the personal cases of individuals who are going to be helped, statistical data about similar victims caught up in a larger pattern of illness, hunger or neglect. The authors of the research concluded that donors respond to real rather than statistical lives. So while it is important for charities to generate data about their impact, it would also be worth finding out how best to deliver effective messages about charity impact in ways that – like Comic Relief – elicit positive responses from donors.

Should we have profit-making schools?

Author: Simon Burgess

Should we have profit-making schools?

Profit-making schools have returned to the education debate in England. This is an emotive issue for many, but an economic analysis is useful in defining the real issues.

There are some simple claims that can be quickly dealt with.

• “Education is far too important to be left to the mercy of profit-making companies.” Education is undoubtedly very important, for long-run growth, for social mobility, and for personal well-being. But think about possibly the most elemental of human needs, the production and distribution of food. While this is regulated by government, we are happy to leave all the decisions to profit-making companies. No-one seriously advocates the nationalisation of food.
• “It just won’t work.” It clearly does at a general level. Countries around the world, including those with well-regarded education systems such as Sweden, allow profit-making schools.
• “No-one should make money out of education.” Obviously they do at the moment: schools buy things from profit-making companies. This obviously has to be the case unless schools are going to start making their own books, desks and computers. So the real issues are (1) what kind of deal can schools get to minimise profiteering, and (2) what services are best bought in from outside as opposed to provided by the school itself

The appeal of allowing profit is the view that it makes decisions matter more. It provides strong rewards to organisations to innovate, to raise quality, and to do things more efficiently. Crudely, on a per-unit basis, organisations are pushed to improve quality and therefore revenue, or to reduce cost.

What would be the effects of this in the current education system in England? To answer this, we need to think about the parameters of the market.

Start with revenue. Schools get revenue for having students on the books. It is more or less a per-capita fee, albeit with some extras and some adjustment by the LA (for community schools). But to adopt the language of business, this money is for processing the students. The revenue that the school receives for each student depends not at all on the progress that the student makes.

This is central to the issue. Given the current system, there is nothing that profit maximising schools could do to raise their revenue per student by raising quality. Immediately, a great deal of the appeal of profit-making is removed.

The only way that schools could make profits is by driving down costs. This may be fine; it may be that this doesn’t really affect the quality of education if done in a smart way. If not done in a smart way, the quality of education would suffer and attainment would fall. It is clear that even the optimistic scenario does not improve education systemically in any way, either statically or dynamically through encouraging entry. The quality of education is the same, and the overall cost to the taxpayer is necessarily the same.

The counter-argument is that the pressure for profit might reduce slack enough so that the fall in costs allowed for profits and an increase in money spent wisely so that attainment increased. For this to work, it has to be that school budgets are spent very unwisely, and that an outside organisation could identify and cut ‘bad’ spending, take some profit and raise ‘good’ spending. It is certainly true that there is a huge amount of idiosyncratic variation in school financial decisions, variation that is unlikely to all be the result of optimal decision-making. Schools either know how to better deploy their budgets but are not sufficiently incentivised to do so, or they do not know. If they do not know, it is unlikely that outsiders will do (other schools may know; but that is another issue, only very clumsily mimicked by profit-making). Profit-making may answer the first point, but so do two other approaches, discussed below.

So profit-making is pointless at best: under the current market set-up, improvements in attainment would not make money (so would not happen) with profit-making schools, and cutting costs would make money but would either reduce attainment or leave it unchanged.

There are alternative strategies that might get some of the benefits of the innovative drive that profits might unleash, but in a more productive way: paying for attainment and incentivising cost reductions through resources for the school.

Paying for attainment. A positive step that keeps the current non-profit system intact but provides some of the same incentive is tying schools’ revenue to their pupils’ attainment. This would be straightforward to administer in principle, but there are some critical issues to resolve before it could be implemented. Chief among these is: should we pay for the simple ‘output’ of the school (GCSE points) or for pupil progress? There are good arguments both ways, to be visited in another post.  Of course, schools do much more than produce attainment, but this is the focus of policy.

Incentivising greater efficiency in other ways.   What if any surplus generated by this process had to be re-invested in the schools? Perhaps schools need some strong incentive to reduce costs. This might well be true, but this is not profit-making: profit-making by definition means the taking of monetary reward out of the school. An alternative scheme would be essentially equivalent to a team (school)-based incentive scheme in which the incentive is not money for the teachers, but resources for the school – resources saved are kept in the school. This is again potentially a good idea, worth looking at and some way short of profit-making.

Profit making in schools would either solve all schools’ problems nor signal the end of civilisation; the issue provokes strong feelings, but largely misses what should be the central policy concerns. Big gains in levels of attainment depend on raising average teacher effectiveness and big gains in equity depend on weakening the importance of proximity as an admissions rule and on changing the allocation of effective teachers across schools.  None of these would be strongly or directly affected by for-profit schools. However, there are certainly merits in piloting policies that link school’s revenue per student to the progress of that student, and incentivising cost reductions through keeping the surplus in the school.

Free to choose?

Authors: Marty Gaynor, Carol Propper and Stephan Seiler

Free to choose?

Greater choice and competition in healthcare is a popular reform model. This column discusses recent research suggesting that once restrictions on choice in the UK’s NHS were lifted, patients receiving cardiac surgery became more responsive to the quality of their care. This saved lives and gave hospitals a greater incentive to improve quality.

A central plank of the NHS reforms implemented by the UK Labour government of the 2000s was the introduction of patient choice. For the first time in the history of the NHS it was mandated that patients should have a say in the choice of hospital when being referred for an elective treatment. Rather than relying entirely on their general practitioner (GP), patients were now offered a set of five hospitals to choose from. At the same time, GPs were no longer tied to a particular hospital through selective contracting agreements and could refer patients more easily to any available hospital in the country. The intention behind the reform was to make referrals more responsive to hospital quality. The argument was that this in turn would increase hospitals’ incentives to improve quality. Although most economists subscribe to the idea that more choice always constitutes an improvement, things are slightly more difficult in the case of healthcare. Contrary to most consumer goods markets, evaluating hospital quality is not a trivial task, and patients might indeed find it hard to pick the best hospital for a particular treatment. This is presumably even more of an issue in the UK, where patients are not accustomed to makingchoices in the context of healthcare.

Do referrals respond to quality?
We set out to analyse whether referral patterns did indeed become more responsive to quality after the introduction of the reform (Gaynor, Propper and Seiler 2012). As a first test for whether this is the case, we analyse whether relatively better hospitals attracted a larger number of patients for one particular procedure: elective coronary artery bypass graft (i.e., heart bypass surgery). We look at the relationship between hospital quality, as measured by patient survival rates, and market shares separately for the time periods before and after the reform. Interestingly, we find that market shares are not correlated with hospital quality pre-reform, however they show a significant correlation with patient survival in the post-reform period. This gives us a first piece of evidence suggesting that patients were indeed allocated to relatively higher-quality hospitals after the reform (but not before). The magnitude of the effect of quality on market shares is economically significant. Post-reform, a one percentage point lower mortality rate led to the hospital attracting 20 more patients every year. This corresponds to about a 5% increase in market share.

In a second step we model patient behavior at a more micro level. Instead of analysing only aggregate shares of patients at each hospital, we model hospital choice individually for each patient. This allows us to incorporate the effect of the patient’s location relative to the hospital as well as to analyse how reactions vary across different patient groups.

We find, also at this level of analysis, that patients became more sensitive to the quality of service as measured by patient survival. However the effect differs substantially across patient groups. Our results show that more severely ill patients react more strongly to the reform, i.e. they are even more likely than the average patient to end up at a high-quality hospital post-reform. In other words, we see the reform having the strongest effect for the group of patients that is presumably most in need of high-quality treatment. Similarly, we find a stronger effect of the reform on patients who reported in a survey that they were informed about the choice reform at the point of referral.

Finally, we also analyse whether poorer patients reacted differently to the reform. A major concern of skeptics of the reform was that only affluent, well-educated patients would be able to process the necessary information and make an educated choice. According to this logic the reform would therefore effectively increase inequality in quality of healthcare. Our analysis however, shows that fortunately these concerns were unfounded. We find that poorer patients reacted no differently from other income groups to the introduction of choice. They did indeed react slightly more strongly, but the difference is not statistically significant, i.e. we cannot confidently tell it apart from noise in our data.

Methodology
We then employ our statistical estimates to quantitatively evaluate the impacts of the reform. We perform the following thought experiment. We compare the actual hospital choices patients made post-reform with the hypothetical choices the same set of patients would have made prior to the reforms, i.e., had referral patterns not changed in response to the freeing of choice. Because we know the sensitivity of referrals to quality pre- and post-reform, we can calculate the probability of visiting each hospital that is available to the patient under either level of responsiveness to quality. Artificially depriving patients of the benefits of the reform allows us to see to what extent patients would have ended up in lower quality hospitals in the absence of patient choice. Using the hospitals’ patient survival rates (adjusted for differences in case-mix, i.e. the severity of the cases treated at each hospital) we find that nine fewer patients (relative to slightly over 300 deaths a year, i.e. around 3%) would have survived every year had the reform not been implemented.

Unexpected drop in mortality rates
The drop in mortality post-reform is an important effect of the introduction of patient choice that was not emphasised by policymakers. Even if the increased responsiveness of referrals to quality did not change hospitals’ performance in any way, the reallocation of patients still leads to better health outcomes. This is due to the fact that patients now visit on average a higher quality hospital from the existing distribution of quality across hospitals.

The hope of the reform was that the quality distribution itself would change due to hospitals’ increased incentives to improve quality in order to attract patients. We next analyse whether there is reason to believe that such a change in hospital quality might have happened. To get a sense of the effect of the change in referrals on hospitals, we compute the change in patient admissions to each hospital if patients had been choosing their hospitals pre-reform according to post-reform referral patterns. This thought experiment – what would have happened had the reform been adopted earlier – gives us a direct sense of how much competitive pressure hospitals found themselves facing when the choice reform kicked in. When calculating the change in market shares we find a very substantial impact for some hospitals, with one hospital losing almost 10% of its market share due to the effects of the reform. There is substantial heterogeneity in the impacts across hospitals however, with most hospitals experiencing more modest changes in market share of around 2% or 3%. There is therefore good reason to believe that at least a significant subset of hospitals had substantial incentives to improve quality in order to retain (or enhance) their market shares of patient admissions.

Do hospitals pay attention to the new system?
In a final step we directly analyse whether there is any evidence of hospitals reacting to the change in referrals by increasing quality, as measured by a fall in mortality rates. To this end we look at whether hospitals that faced the strongest pressure post-reform (as measured by the potential loss in admissions) saw a bigger decline in their mortality rates than other hospitals. We do indeed find that this is the case. Hospitals which had the biggest increases in the responsiveness of patient admissions to their mortality rates had the largest declines in mortality rates, and vice versa. This result closely mirrors related work by Cooper et al. (2011) and Gaynor et al. (2012) who show that areas with more competition experienced a larger increase in patient survival rates after the introduction of patient choice.

In summary, we assess that the reform reduced cardiac bypass surgery mortality by 3% by re-allocating patients to better hospitals. This is clearly a lower bound on the beneficial effect one might expect from allowing choice, as we look only at the effect for one particular procedure. Secondly, we find evidence suggesting that hospitals responded to increased choice by improving their quality. If this is mirrored as a hospital-wide effect, there may be substantial additional positive benefits for patients. Finally, our findings add support to earlier evidence that indicate that the choice reforms led to falls in mortality in other treatments and shorter lengths of stay without increasing hospital total costs (Gaynor et al., 2012; Cooper et al., 2011) and also to work by Bloom et al. (2010) that indicates that more competitive environments have better management practices that are in turn associated with better hospital performance.

References
Bloom, N, C Propper, S Seiler, J van Reenen (2010), “The Impact of Competition on Quality: Evidence from Public Hospitals”, NBER Working Paper, 1630.

Cooper, Z, S Gibbons, S Jones, and A McGuire (2011), “Does Hospital Competition Save Lives? Evidence from the English Patient Choice Reforms”, Economic Journal, 121 (554), F228-F260.

Gaynor, M, R Morena-Serra, C Propper (2012), “Death by Market Power: Reform, Competition and Patient Outcomes in the British National Health Service”, forthcoming in American Economic Journal: Economic Policy, See also University of Bristol Centre for Market and Public Organisation, Working Paper, 10/242.

Gaynor, Martin, Carol Propper and Stephan Seiler (2012), “Free to Choose? Reform and Demand Response in the English National Health Service”, CEP Discussion Paper, 1179, November.

Teacher performance pay without performance pay schemes

Author:  Simon Burgess

Teacher performance pay without performance pay schemes

Amid the macroeconomic gloom, the Autumn Statement contained a line about teachers’ pay. The School Teachers’ Review Body recommends “much greater freedom for individual schools to set pay in line with performance”. Consultations and proposals are expected in the near future.

But simply giving schools the freedom to do this may be a rather forlorn hope of anything much happening. It is not clear that there is a substantial demand from schools for performance-related pay (PRP) schemes that has only been thwarted by bureaucratic restrictions. It is hard to see high-powered, tough-minded PRP schemes being introduced by more than a handful of schools, not least because we have not seen large scale deviations from national pay bargaining in academies in England despite their new freedoms to do so.

If that path seems unpromising, there are other ways of facilitating a greater reflection of performance in pay, discussed shortly. But first – is PRP for teachers a good idea in the first place? Does it raise pupil attainment? What are the ‘side effects’?

This is a question that economists have produced a good deal of research on. And to summarise a lot of diverse work briefly, the international evidence is mixed. Those on both sides of the argument can point to high quality studies by leading researchers that find substantial positive effects, or no effects. In both cases, interestingly, there appeared to be little evidence of gaming or other unwanted effects of the incentives.

There is little evidence specifically for England. Our own research found a substantial positive effect of the introduction of a PRP scheme, but given the varied results found elsewhere it would seem unwise to place too much weight on this one study. The underlying performance pay scheme was poorly designed but nevertheless had a positive effect on the progress of pupils taught by eligible teachers relative to ineligible ones.

And design is key. There are many reasons why a simple high-powered incentive pay scheme might be detrimental to pupil progress, which we have discussed here and here. These include the fact that teachers have multiple tasks to do, the problems of measuring the outcomes of some of those tasks, the complex mixture of team and individual contributions, and the potential impacts on implicit motivation. The overall message is that incentives work, but schemes have to be very carefully designed to achieve what the schemes’ proponents truly intend.

There is another way to facilitate a closer link between pay and performance that does not require any school to introduce a performance pay scheme.

Published performance information in a labour market can change the way that the market rewards that performance. The critical features are first that the organisation’s own output depends in an important way on this performance characteristic of an individual; second that the organisation has some discretion in the pay offers it can make to new hires; and thirdly that the performance information is public – is available and verifiable outside the current employer. In this case, the pay structure of the market will reflect the performance rankings: high-performing individuals will be paid more.

In teaching, the first two of these three conditions are met: teacher quality matters hugely for schools, and schools have some discretion over pay. Now, suppose we had a simple, useful and universal measure of each teacher’s performance in raising the attainment of her pupils (obviously we don’t at the moment; I come back to this below), and that this was published nationally, primarily for the attention of Headteachers. The idea is that Headteachers trying to improve the attainment of their pupils would be on the look-out for high performing teachers when they had a vacancy to fill. Armed with this performance information, they might try offering a higher wage (or something else – it doesn’t have to be money) to tempt them to join their own school. Equally, the teacher’s current school may respond by raising the offer there.  Over time, this process will tend to raise the relative pay of high-performing teachers relative to low-performing ones, whom no-one is trying to bid for.

This idea should not be a strange one. A number of professions have open measures of performance. Just today it is reported that performance measures for more surgeons will be made public in the summer of 2013; this is already true for heart surgeons.

It is well-known that PRP does two things: it motivates and it attracts. The outcome for pay described here will tend to make teaching more attractive to people who are excellent teachers and less attractive to those who aren’t.

There are a number of problems with this idea, though perhaps less than might appear at first glance.  First, it could be argued that a performance measure derived from teaching in one school is not relevant to teaching in another school. Obviously each child and each school is unique, but it seems very unlikely that there is no commonality of context between one school and the next. Observation suggests this: teachers moving from one school to another are not counted as having zero experience, and Headteachers are often appointed from outside a school.

Second, there might be a fear that the teacher labour market would become chaotic, with everyone churning around from school to school in search of a quick gain. We have to recognise that there is substantial turnover of teachers now < http://www.bristol.ac.uk/cmpo/publications/papers/2012/wp294.pdf >. But the main point is that it does not require much actual movement to make the market work. Schools can make counter offers to try to retain their star teachers and the end result is the same – higher salaries for high-performing teachers.

Third, any measure would be noisy, partial and imperfect. Of course, all such measures are. Whether a measure is perfect is not really the question, the question is how noisy and imperfect is it, and whether it contains enough information to be useful. One advantage in this case is that the consumers of these performance indicators are the people best able to judge their usefulness and their shortcomings: Headteachers. If such metrics are not useful, Headteachers will simply ignore them; there would be no compulsion to use them.  Even in labour markets with some of the most detailed and finely measured performance indicators (for example, football or baseball) there are many moves between employers that do not work out. It is worth re-emphasising that these performance measures are bound to be imperfect and incomplete, but broad measures of performance may nevertheless be very useful.

There are useful parallels to be drawn from another profession: academics. For academics, the combination of very detailed and public performance information and a context where research performance matters a great deal to universities seems to have had a substantial effect on academics’ pay.

The Research Assessment Exercise (RAE) and more recently the Research Excellence Framework (REF) have made a strong research performance very important to a university’s standing and its income. But the critical factor for academics is that an individual’s research performance is public knowledge, through very detailed recording of the impact of their research papers. Departments and universities aiming to improve their ranking seek out star researchers and attempt to bid them away with higher salaries (plus other things such as research facilities). These offers may well be matched by their current employer, but the end result is that salaries now seem to be much more closely correlated with research productivity than before the RAE/REF (I say “seem” as there does not appear to be any evidence on this, so this is casual empiricism). This is a lot of what drives many young researchers to put in very long work hours: having a paper published in a top scientific journal early in a career has a substantial lifetime payoff even in a world with few or low-powered incentive schemes. If you check out academics’ websites you will invariably see their academic output prominently displayed.

Again, an important feature is that these indices of research output are largely consumed by other academics who are aware of their strengths and weaknesses. So although they are far from perfect, they are used by precisely the people best placed to calibrate their usefulness appropriately.

If we are to go down a path of tying teacher pay more closely to performance, and yet respect the rights of increasingly autonomous schools to determine their own pay systems, then this might be an option to consider.  The challenge is to devise a measure that is simple, useful and universal. It would measure the progress made by the pupils that teachers taught, it would have to deal with normal variations in performance by averaging over a number of classes and a few years, and be on a common metric.  This is not straightforward, but if it gave rise to a robust broad measure of performance it could form a part of performance pay for teachers, and performance management more broadly. It could also have substantial effects on the pay of high-performing teachers.