Author: Simon Burgess
Threshold measures in school accountability: asking the right question
We are in the midst of a significant upheaval in the setting and marking of exams, and the reporting of school exam results. One feature of the system has been the centre of a lot of criticism and highlighted for reform: the focus on the percentage of a school’s pupils that achieve at least 5 GCSEs at grades C to A*, including the scores on English and maths. This is typically the most-discussed metric for (secondary) school performance and is the headline figure in the school league tables.
The point is that this measure is based on a threshold, a ‘cliff-edge’. Get a grade C and you boost the school’s performance; missing a C by a lot or a little are the same, and just scraping a C is the same as getting an A*.
This has been described as distorting schools’ behaviour, forcing schools to focus on pupils around this borderline. The argument is seen as obviously right and strong grounds for change. In this post I want to make two counter-arguments, and to suggest we are asking the wrong question.
First a basic point. One central goal of any performance measure is to induce greater or better-targeted effort. This might just mean “working harder” or it might mean a stronger focus on the goals embodied in the measure at the expense of other outcomes. The key for the principal is to design the best scheme to achieve this. A very common scheme is a threshold one – this can be found for example in the Quality and Outcomes Framework for GPs, service organisations with a target number of clients to see, and of course schools trying to help pupils to achieve at least 5 grades of C or better. An organisation working under a threshold scheme faces very different marginal incentives for effort. Considering pupils: the most intense incentives relate to pupils just below the line: this is where the greatest payoff is to schools to devote the most resources.
The first counter argument starts by noting that the asymmetry in the incentive is not a newly-discovered flaw, it is a design feature which can be very powerful. If there is a level of achievement that is extremely important for everyone to reach, then it makes sense to set up a scheme that offers very strong incentives to do that – that focusses the incentive around that minimum level. This is precisely what a threshold scheme does.
So rather than simply pointing out that threshold designs strongly focus attention (which is what they’re supposed to do), the questions to ask are: is there some level of attainment that has that characteristic of being a minimum level of competence? And if so, what is it? If society feels that 5 grade C’s is a fair approximation to a minimum level that we want everyone to achieve, then it is absolutely right to have a ‘cliff-edge’ there because inducing schools to work very hard to get pupils past that level is exactly what society wants. It may be that we are equally happy to see grades increase for the very brightest children, those in the middle or those at the lower end of the ability distribution. Or not: all the main political parties express a desire to raise attainment at the lower end and narrow gaps.
The argument should be about where to put the threshold, not whether to have one or not. Perhaps we are starting to see a recognition of this in the recent policy announcement that all pupils will have to continue studying until they have passed English and Maths.
The second counter-argument is based on a scepticism of what is likely to happen without the 5A*-C(EM) threshold acting as a focal point.
The core strategic decision facing a headteacher is how best to deploy her main resource: the teachers. Specifically: how best to assign teachers of varying effectiveness to different classes. It has been said that schools will be free to focus equally on all pupils.
Well, maybe. Or perhaps we should think of the pressures on the headteacher, in this instance from teachers themselves. Effective teachers are very valuable to a school and any headteacher will be keen to keep her most effective teachers happy and loyal. It seems likely (I have no evidence on this, and would be keen to hear of any) that top teachers would typically prefer to teach top sets. If so, we might see a drift of the more effective teachers towards the more able classes in a school (and therefore on average, the more affluent pupils). The imperative of the C/D threshold gave headteachers an unanswerable argument to push against this.
So threshold metrics have an important role to play in communicating to schools where society wants them to focus their effort. The current threshold, at 5 C grades, may or may not be at the right level; but discussing what the right level is, is a more useful debate to have.