Who fails wins? The impact of failing an Ofsted Inspection

Rebecca Allen and Simon Burgess

What is the best way to deal with under-performing schools? This is a key policy concern for an education system. There clearly has to be a mechanism for identifying such schools. But what should then be done with schools which are highlighted as failing their pupils? There are important trade-offs to be considered: rapid intervention may be an over-reaction to a freak year of poor performance, but a more measured approach may condemn many cohorts of students to under-achieve.

This is the issue that Ofsted tackles. Its inspection system identifies failing schools and supervises their recovery. How effective is this? Is it even positive, or does labelling a school as failing push it to ever lower outcomes for its students?

It’s not clear what to expect. Ofsted inspections are often dreaded, and a fail judgement seen as being disastrous. It has been argued it triggers a ‘spiral of decline’, with teachers and pupils deserting the school, leading to further falls in performance. But it might also be a fresh start, with renewed focus on teaching and learning, leading to an improvement in exam scores. Equally, we might expect nothing much to happen: after all, the policy ‘treatment’ for those schools given a Notice to Improve is very light touch. It is neither strongly supportive (typically no or few extra resources) nor strongly punitive or directive (schools face no sanctions nor restrictions on their actions). Schools are instructed to focus intensively on pupil performance, and are told to expect a further inspection within a year. In addition – and possibly the most important factor – the judgement that the school is failing is public one, usually widely reported in the local press.

Our research shows that the Ofsted inspection system works. Schools that just failed their Ofsted significantly improved their performance over the next few years, relative to schools that just passed. The impact is statistically significant and sizeable. In terms of the internationally comparable metric of effect sizes, our main results suggest an improvement of around 10% of a standard deviation of pupil scores. This is a big effect, with a magnitude similar to a number of large-scale education interventions. Translated into an individual pupil’s GCSE grades, this amounts to a one grade improvement (for example, B to A) in one or two GCSEs. From the school’s perspective, the gain is an extra five percentage points in the proportion of pupils gaining five or more GCSEs at grades A*-C.

Our findings suggest that the turn-around arises from proper improvements in teaching and learning, not gaming to boost exam performance through switching to easier courses. First, the impact is significantly higher in the second year post visit than the first, and remains level into the third and fourth year after the inspection. So it is not simply a quick fix to satisfy the inspectors when they return twelve months later. Second, we find a stronger effect on the school’s average GCSE score than on the headline measure of the percentage of students gaining at least 5 good passes; if the schools’ responses were aimed at cosmetic improvement, we would expect the reverse. We also find similar positive effects on maths results and on English results.

It could be argued that these results are implausibly large given that the ‘treatment’ is so light touch and schools are given no new resources to improve their performance. The instruction to the school to improve its performance may empower headteachers and governors to take a tougher and more proactive line about school and teacher performance. This may not be a minor channel for improvement. Behavioural economics has provided a good deal of evidence on the importance of norms: the school management learning that what they might have considered satisfactory performance is unacceptable may have a major effect. The second part of the treatment derives from the fact that the judgement is a public statement and so provides a degree of public shame for the school leadership. Ofsted fail judgements are widely reported in local press and this is usually not treated as a trivial or ignorable announcement about the school. It seems plausible that this too will be a major spur to action for the school.

Where do we go from here? Our results suggest Ofsted’s identification of just-failing schools and the use of Notice to Improve measures is an effective policy, triggering the turn-around of these schools. We need to be clear that our research does not address the question of what to do about schools that comprehensively fail their Ofsted inspection. Possibly this light-touch approach can be extended. Since leaving the Headship of Mossbourne school to become the new Director of Ofsted, Sir Michael Wilshaw has argued that schools just above the fail grade should also be tackled: that ‘satisfactory’ performance is in fact unsatisfactory. Such interventions in ‘coasting’ or ‘just-ok’ schools are very likely to be of the same form as Notice to Improve. Our results suggest that this is potentially a fruitful development with some hope of significant returns.

This research is available on the CMPO website and the IoE website.

Work experience

Paul Gregg

Work experience programmes to help get the unemployed back into regular jobs have a long and somewhat chequered past. They have two distinct and offsetting effects; first the work experience itself is valuable. It gives an opportunity for the unemployed person to signal good work habits, including turning up on time and self-motivation when at work. These are the basics that employers want, after skills and experience. A decent reference from another employer is a big plus when looking for work. Obviously this is more important for those who have been out of work for a long time, or for young people with little previous work experience. However, whilst a person is on a work experience programme job search activity diminishes. The participant is working which simply reduces time available, but it is as important that they focus on trying to impress the firm they are based in, in the hope of being kept on. This is called ‘Lock In’ in the jargon of such schemes. These two effects are clearly visible in the Figure below (taken from the CESI website http://www.cesi.org.uk/blog/2012/mar/work-experience-more-be-done). It shows the proportion of participants in the government’s current Work Experience programme who are still claiming benefits week by week once they start participating (the blue line), compared with those who do not participate (the red line). For the duration of the placement, the first eight weeks, the proportion still claiming is higher by about 5% and then they rapidly catch up. So whilst they are on the experience programme job entry falls but once it has ended the positive effects start to kick in.

In this comparison the net effect is zero. This doesn’t definitively suggest that the programme has no beneficial effects for those participating, as we can’t be certain that those joining are not those who would have found securing work harder. They could of course be the more keen participants, or those in places where more firms offer placements, we simply can’t tell. However, this story of ambiguous effects with little overall impact on people’s chances of securing jobs has been the norm of Work Experience programmes since the 1970s. Unless, the design actively tries to minimise the adverse effects on job search and maximise the positive part of the work experience.

The first key ingredient is to maintain and support job search throughout the placement. That means requiring and monitoring the participant’s job search activity, as is the case when people are on unemployment benefits. Furthermore, the support services that welfare to work providers deliver to the long-term unemployed, such as CV writing and job interview techniques need to be in place to make the most of the experience. Finally, the agents delivering the placement, both the provider and the firm, need to care about the person getting a job after the placement. The easiest way to guarantee this is a bonus payment if the person is kept on by the firm or another job is found shortly after completing the placement. Small firms especially have a lot of contacts they can use to help place someone. Hence it is generally most valuable for private or third sector firms to house the participant rather than in the public sector.

Labour’s New Deal programme launched a decade ago now, had some of these ingredients but in sequence rather than all at the same time. There was a Gateway phase before the placement for support and intense job search, and again at the end in a Follow-through phase, but not when actually on the experience part. However, it didn’t have any outcome related payment for firms or charities providing the experience. By comparing those aged 24 who joined the scheme at 6 months, with those aged 25 who had to wait until 1 year, we got a sense of its effectiveness. A study by John van Reenen (LSE) showed clear job entry gains for participants, although it didn’t assess how long the gains lasted as the older group also joined the programme.

The current Work Experience programme for under 25s starts much earlier at just 3 months, which means there are more potential participants and many more will find work quickly without help. For this reason the government is keeping costs down to a minimum. The placements are not paid and no support services are provided, no job search monitoring is in place and there are no bonus payments for firms who take participants on. Hence it should be no surprise that the scheme appears to be making little or no difference.

Labour has just launched its ideas for a more intensive, 6 month paid programme starting after 12 months on unemployment benefits. These are a much smaller group with much greater need; people who go on to have very damaged working lives well in to their 40s unless they can get attached to long-term stable employment. With signs of intelligent design, the programme has all the key ingredients for making a difference, except outcome payments for the firms. Participants will have a support service provider whilst on the programme who is only paid on the basis of getting the person into work, the work experience is part-time to leave time for job search and as when receiving benefits normally, active job search is monitored and sanctions will apply if it is not undertaken. So the scheme has a decent chance of making a sufficient difference to cover its costs. But the other issue, apart from there being no bonus payment to firms getting participants into work, is that it is drawn very narrowly. By requiring a person to be on unemployment benefits for as long as a year means that many cycling between short term jobs and unemployment are missed. Those leaving school at age 16 or 17 who don’t get work are not entitled to benefits, and will still have to wait a year (until they are 19) before help kicks in. There is also no help for the 16 and 17 year old unemployed. So, it is a good first instalment but more will be needed to end long-term youth unemployment.

“Make Bradford British”: the parallel lives of the younger generation

Ellen Greaves  (IFS)

The high level of residential segregation for people from different backgrounds was highlighted as a factor contributing to the tension between people from different ethnic groups in the Cantle Report (2001) in response to the race riots in the same city in 2001. The Cantle Report identified that many people living in the same city led “parallel lives”; with little interaction with, or understanding of, people from different backgrounds.

Last night a Channel 4 programme explored the relationships between 8 residents of Bradford, brought together to interact with people from different backgrounds for the first time. The programme addressed the difficulties and benefits of over-coming entrenched prejudices, underlying (perhaps unacknowledged) racism in the city, and the similarities and differences between people from different cultures. The majority of participants in “Make Bradford British” found it a positive experience. Future improvements in relationships between different communities are likely to come through increased daily interaction, however, as well as the moving personal experiences witnessed on screen, and integration in schools can be a starting point.

Segregation in schools commonly results from residential segregation (as admission to schools is usually determined by proximity), but parents’ preferences can also play a role. Research by CMPO shows that pupils from most ethnic groups are more segregated in school than where they live, suggesting that there are factors, either parents’ preferences or other constraints, that restrict integration in England’s schools.

Pupils’ current level of integration in each area in England can be viewed on a website created by CMPO, funded by the ESRC and using national administrative data provided by the Department for Education. The figures for Bradford are startling: in 2009 around 80% of Pakistani pupils of primary school age attended a primary school that was primarily non-white, while just under 70% of white pupils attended a school that was primarily white. This suggests that these two groups of pupils are highly segregated in their primary schools, and the figures suggest that they remain so in secondary schools. Bradford is not alone in this high level of segregation (Oldham has similarly high levels), but the figures are higher than in other areas of the country with similar demographics; in Manchester under 60% of Pakistani pupils attended a school that was primarily non-white in 2009 and around only around 30% of white pupils attended a school that was primarily white.

Other indicators of segregation can also be compared: the dissimilarity index is a summary measure designed to capture the degree of “unevenness” between two groups of pupils (that is, the extent to which the school population reflects the wider population in the local area). Its value (between 0 and 1) represents the proportion of the group population that would have to change schools to achieve the same distribution of that of the local area, where, in general, a dissimilarity index of less than 0.3 is considered low, between 0.3 and 0.6 as moderate and above 0.6 as high (Massey and Denton, 1988).

Bradford has a decisively high level of segregation according to the dissimilarity index; in 2009 the figures show that around 70% of white and Pakistani primary pupils would have had to change schools in order for the distribution to reflect the wider population of the area, and the figure is similarly high for secondary schools.

There are encouraging signs, however; segregation according to the dissimilarity index has decreased marginally in primary schools since 2002, suggesting that the younger generation at least is becoming slightly more integrated, although levels remain higher than almost anywhere else in England.

The high level of segregation in Bradford’s schools suggest that integration and understanding between people from different backgrounds are unlikely to be fostered from direct experience in the education system, although there are some signs that the situation is marginally improving. What can make “Bradford British” and united remains unclear, but further integration of the city’s young people is not likely to harm the city’s prospects.

Notes on the author:

Ellen Greaves is a Research Economist in the education, employment and evaluation research sector at the IFS.