Gender, teacher assessment and stereotypes

By:  Simon Burgess

The gender gap in attainment is a key fact of our times, with girls now out-performing boys pretty much throughout the education system. Nevertheless, there are currently significant gaps in jobs: women are still under-represented in science, technology, engineering and mathematics. How the gap in qualifications plays out into jobs and pay over the next twenty years or so is going to have significant consequences for the nature of work, the composition of the leading professions, family life, bringing up children and more.

But that’s for the future. For now, we are still trying to understand the implications of the gender gap in schools. Last week a new report from the OECD uses the PISA data to shed more light on the gender gap across a large group of countries. The TES  highlights the conclusions drawn about teacher assessments and stereotyping:

“ … while teachers generally reward girls with higher marks in both mathematics and language-of-instruction courses, after accounting for their PISA performance in these subjects, girls’ performance advantage is wider in language-of-instruction than in mathematics. This suggests both that girls may enjoy better marks in all subjects because of their better classroom discipline and better self-regulation, but also that teachers hold stereotypical ideas about boys’ and girls’ academic strengths and weaknesses.” (OECD, p. 56)

These findings echo CMPO research by Ellen Greaves and I on teacher assessments and pupil ethnicity.  While we focussed on ethnicity, we also included gender and social status in the analysis.

We used data from the National Pupil Database (NPD) to compare written tests and teacher assessments of the same characteristic, namely the pupil’s ability in Maths, English and Science. The tests were nationally set and remotely marked; the teacher assessment was provided by the pupil’s subject teacher.

We can make this comparison because the end of Keystage 2 at age 11 has both these forms of assessments. There is no presumption that one form of the assessment is the Truth and one is biased. They are independent but noisy measures of the same underlying characteristic – just how good is this pupil at maths? But a comparison of the two across a large sample is revealing. Since we used all the eleven year-olds in England, our sample is big enough. Overall, the most common outcome is that the two estimates of ability agree, there is no difference between teacher assessment and remotely marked test.

But there are systematic patterns in the differences that are very interesting. In terms of gender, our findings for England are similar to the OECD, although since we use NPD data from the mid 2000’s, girls’ progress has moved on. We show that girls are “over-assessed” in English and “under-assessed” in maths. That is to say: the gaps between the test and the teacher assessment are on average positive in maths and negative in English for girls.

In terms of social class, we found that pupils eligible for free school meals were “under-assessed” on all three subjects.

Another way of saying the same thing is that poor pupils systematically and significantly out-performed what their teachers thought they would achieve.

We show the results for different ethnic, gender and social divides in the graph below. It shows very starkly that groups doing well in a test at a national level tend to be over-assessed by teachers; and equivalently, groups doing badly nationally tend to be under-assessed.

Note

None of this is to say that teachers are biased. Like everyone else all the time, they use stereotypes to help make decisions when their information is imperfect.

But there are consequences. It is important that we do not rely solely on teacher assessments and that we retain and use nationally set and remotely marked tests. Using teacher assessments rather than the test scores to define attainment would result in a much greater recorded gap between poor and non-poor pupils. Tests allow pupils to show what they can do independently of someone’s opinion of them, including that person being their teacher.

Categories: Uncategorized

Do place-based policies achieve their objectives?

By:  Helen Simpson

Place-based policies such as enterprise zones target specific geographic areas, rather than specific groups of people. Even so, their ultimate aim is often to create jobs and boost incomes of relatively disadvantaged residents. A new Federal Reserve Bank of San Francisco Economic Letter, by David Neumark and Helen Simpson, discusses the evidence on whether place-based policies meet their objectives.  The evidence on enterprise zones is mixed, with some studies finding no effects on employment and others finding positive effects on job creation. There is also no clear cut evidence that enterprise zones reduce poverty, and some evidence that they lead to house price increases, suggesting that the end beneficiaries may well differ from those the policy originally set out to help.

While targeting specific areas to take advantage of “agglomeration benefits” (that is, the productivity boost stemming from increased density of firms and workers), or purely on equity grounds is justifiable, the intended effects of policies that target “place” rather than “people” can be undone by geographic mobility. Firms may simply re-locate into subsidised areas, calling into question the nationwide benefits of an area-based subsidy if jobs are simply being geographically reshuffled, and people may also move. This means that the ultimate beneficiaries of any new jobs may not be the original disadvantaged residents, and that homeowners and landlords may benefit from increased property values. Given this, place-based policies aimed at specific locations should be closely evaluated to fully understand who exactly it is that gains any benefits, and whether these come at a cost to other individuals or areas.

Categories: Uncategorized

Will a decrease in stamp duty actually help first time buyers?

December 4, 2014 Leave a comment

By: Mike Peacey

Yesterday George Osborne delivered his autumn statement. Perhaps the most significant reform is that of stamp duty land tax. Unsurprisingly this tax has been criticized by numerous interested parties for some time. The infamous feature of stamp duty was that the marginal tax rate exceeds 100% every time the price crosses one of the thresholds, because a single tax rate applies to the whole transaction. This tax has been labelled “Britain’s worst tax”, being distortionary and unfair. Reforms to stamp duty have been in the pipeline for years (my two pence worth), and this year Scotland announced its own overhaul of the tax.

George Osborne’s reform eliminates the most objectionable feature – now the threshold will only affect the proportion of the property above it. The other change is to the rates of tax. The results are that most property sales below £1.125m will now pay a lower tax (and very expensive houses will now pay a much larger amount).

Final

On the face of it this reform appears to make sense. The purpose of most taxes is to raise revenue and redistribute wealth. Stamp duty does raise significant revenue – having recently overtaken revenue raised by tobacco taxes. Although this reform is expected to reduce government revenue by around £800m (approximately 12.5%), it is claimed to also benefit 98% of homebuyers.

A first year economics textbook teaches us two important ideas about taxation: deadweight losses and tax incidence. The purpose of trade is to reassign property to the party who values it most. However, if the increase in value the buyer places on the property (above the seller) is less than the tax amount the sale won’t happen. When this happens the tax has resulted in dead weight loss. Tax incidence refers to the subtle difference between who is legally responsible for payment (in the case the buyer) and who in practice picks up the bill. The distinction arises because the buyer will take payment of the tax into account when deciding on her offer.

If 98% of homebuyers pay less tax does this imply they are necessarily better off? Not necessarily. Focusing on these individuals (think 6, not 7, figure house prices), there are reasons which suggest this tax reduction might make them worse off.

Current first time buyers are credit constrained: Many could afford the loan repayments on a more expensive property, but not the deposit. Since stamp duty cannot be added to a mortgage, the buyer must have enough for this tax payment in addition to their deposit. If the tax reduction gives buyers the opportunity to increase their deposit, banks might respond by permitting them to increase their leverage. This raises the possibility that a relatively small tax reduction might, via a multiplier effect, significantly increase house prices – making those buyers worse off.

Best and Kleven (2013) have shown that house prices respond to changes in stamp duty by bunching at new thresholds. Removing the cliff edge will smooth the price distribution – and permit an increase in average house prices. Moreover, since the majority of transactions involve some bargaining, we should consider the introduction of taxes in this framework. Chae (2002) showed us there are situations in which parties can actually benefit from a higher tax! My own research into the UK housing market backs this up: suggesting that the cliff edge property of stamp duty provides such a benefit to buyers.

First time buyers rejoicing in the belief they can now afford a house might be disappointed. If the most objectionable feature of the tax – the cliff edge property – has been keeping prices (across the whole market) down, perhaps homeowners not homebuyers should be rejoicing. Economics 101 teaches us that the legal and economic incidence of a tax is not the same. A tax reform intended to help first time buyers might paradoxically make them worse off, especially if it fuels house price inflation.

Categories: Uncategorized

London, ethnicity and GCSEs: Response … and a question for you

November 14, 2014 Leave a comment

By:  Simon Burgess

One of the things to come out of the response to day before yesterday’s blog was a clear desire to hang on to the ‘London Effect’. This is the belief that the much higher GCSE scores in London than the rest of England are the result of some policy or practice in the capital’s schools.

As I show in the report and as Chris Cook demonstrated here and before, this can be done by focussing on a subset of GCSE exams and taking that as the outcome measure. This is currently the only way that that belief can be supported.

I will come back to this below.

But here is the main point: I think we are in danger of missing the big issue by focussing on a debate over vocational qualifications. In this rush to hang on to the effects of a slightly mysterious policy, we are just marching past a demonstrable achievement of London. Sustaining a large, successful and reasonably integrated multi-ethnic school system containing pupils from every country in the world and speaking over 300 languages is a great thing. The role of ethnic minorities in generating London’s premium shows that London is achieving this. How many of those are there? I don’t know enough about school systems around the world to say, but I’d guess it’s probably unique.

To my mind, this is a fact worth celebrating about the London education system.

Having dwelt on that thought, here are two points on vocational qualifications.

First, the comparison of London and the rest of England, with and without vocational qualifications (VQs), and across ethnic groups is quite complex. Here are some things that are not true and some that are true.

1. It is not the case that White British pupils make better progress than pupils from ethnic minorities if we exclude VQs. The performances on this measure line up very strongly with performances on the regular measure, as shown in Figure 1:

Pt 2 Figure 1

2. It is not the case that pupils from all ethnic minorities do more VQs than White British pupils. Some do, some don’t: see the final column of table 1. Bangladeshi, Black African, Chinese and Indian ethnicity do fewer VQs than White British pupils; Pakistani and Black Caribbean pupils do more.

Table 1

3. But it is the case that pupils of each ethnicity did more VQs in schools outside London than inside. See the first two columns of Table 1 above. All these differences are statistically significant. You can also see that in Figure 2 below, which also shows that outside London pupils did slightly more entries overall, meaning less study time per subject.

Pt 2 Figure 4

4. And it is the case that if we do exclude VQs, pupils from each ethnic group score higher on this progress measure in schools in London than outside. See table 2 below.

Table 2

So the key question is: why did schools in London enter their pupils for far fewer VQs? Was this a city-wide policy decision? Or more informal but still common across London?

This is the question to you.

There is more on equivalents etc in a useful blog by Dave Thomson of FFT. He finishes with the same question.

Second, is it legitimate to decide now, after the fact, that some qualifications count less or not at all in a measure of what schools do? Obviously some vocational qualifications were severely Wolf’ed but not all.

Chris Cook and I have swapped analogies on this:

Chris: It’s a 110m hurdles race and we let schools choose their own hurdles, and then only look at their times at the finish. But we know some chose lower hurdles.

Me: athletes running a 400m race to find a winner. After the race, someone says that actually the proper test of run ability is just the first 200m so we will declare the true winner as the person ahead after 200m.

I am sure there is some truth in both (Any one coming up with a shot put-based analogy wins a prize.)

Finally, a thought about why there is this desire to hang on to the London Effect. If the higher GCSE progress had been the result of a policy (about 8 GCSE grades or 9.8% of an SD more technically) then it would be one of the best large scale policies ever. One reason I guess why people want to believe in it is that we lack a portfolio of proven, large scale public policies to raise educational attainment. Reiterating my comment from the paper, there are no innate differences in ability between pupils from different ethnic groups, but higher aspirations and expectations and perhaps a strong social network encouraging success matter a great deal. How to encourage that in groups where it can be absent is currently unknown; that is where the research frontier is.

Categories: Uncategorized

London schools: the central role of ethnicity

November 12, 2014 Leave a comment

by: Simon Burgess

Urban areas are often associated with poor educational attainment. But London is different. Recent analysis suggests that the attainment and progress of pupils in London is the highest in the country. A leading education policy commentator argues that: “Perhaps the biggest question in education policy over the past few years is why the outcomes for London schools have been improving so much faster than in the rest of the country”. Some have argued that this is the result of policies and practices adopted by London schools. If so, identifying the key policies is a great prize, with the hope that they can be implemented more broadly. Another recent report emphasises more the importance of primary schools.

In this research I have set out the evidence that a big part, almost all in fact, of the answer lies in the ethnic composition of London’s pupils. More broadly, my interpretation of this leads to a focus on pupil aspiration, ambition and engagement. There is nothing inherently different in the educational performance of pupils from different ethnic backgrounds, but the children of relatively recent immigrants typically have greater hopes and expectations of education, and are, on average, more likely to be engaged with their school work. This is not by chance of course; a key part of the London effect is its attraction to migrants and those aspiring to a better life.

There are two key and indisputable facts that lie behind this: ethnic minority pupils make better progress through school than white British pupils do, and ethnic minority pupils make up a much higher fraction of pupils in London. London also has a lot more recent migrants into the country. We showed some time ago that ethnic minority pupils make better progress to GCSEs than white British pupils. Given that these pupils typically live in more disadvantaged neighbourhoods and come from poorer families, their advantages must be less material than books, museum visits and computers. It is argued that ethnic minority pupils have greater ambition, aspiration, and work harder in school.

Put simply, this is the story: London has more of these pupils and so has a higher average GCSE score than the rest of the country. In this research, I put these two facts together and show that this accounts for almost all of the London effect.

The best way to measure what a school, or a city-wide school system, adds to its pupils is to look at pupil progress. This is what I do here. I focus on progress through secondary school, partly because the attainment measure at the start of secondary school is better than that for primary, and partly because others are focussing on primary schools. Progress is simply and standardly defined as capped GCSE points score relative to Keystage 2 scores.

What do we see? First, there is a London premium in pupil progress of just over 8 GCSE grade points, where a grade point is the difference between an A and a B, or a D and an E, etc. (for international comparability this is 9.8% of a standard deviation (SD). This is a huge number: the difference between say getting 8 C’s rather than 8 D’s. Given this, it is unsurprising there has been speculation as to what lies behind it. There is also a very substantial gap in the fraction of pupils progressing to at least 5 A*-C grades, 2.5 percentage points more in London.

Second, ethnic composition matters a great deal. In fact, differences in composition account for all of the gap in the progress measure. If I assume that London had the same ethnic composition as the rest of England, then given the progress of each ethnic group in each place (London, not London), there would be no ‘London Effect’. This decomposition is discussed in the paper, but an easy way to see this is a simple regression. Estimate the ‘London effect’ over all the pupils in all state schools in England, first not taking account of ethnicity and then adding those controls in, and Figure 1 (in SD units) shows that the effect is wiped out.

Figure 1

If I analyse conditional progress – taking account of personal circumstances such as poverty, gender, month of birth etc. – a London premium of 11% of an SD is also entirely eliminated by controls for ethnicity; this is also robust to conditioning on neighbourhood disadvantage as well. The Figure also shows that there is no significant difference between the progress of white British pupils in London and in the rest of the country, nor between pupils of Asian ethnicity in London and outside.

Third, it may well be that this effect is not a pupil’s ethnicity per se, but a characteristic that is correlated with ethnicity for some ethnic groups – being the child of immigrants. There is no educational data with immigrant status, so we have to use imperfect approximations. One way is to look at language – whether the pupil speaks English at home, or whether the pupil has English as an additional language (“EAL”). I also try another approximation in the paper.

Figure 2 shows again the impact on the estimated London Effect of controlling for language:

Figure 2

Of course, there has been change over the last decade. In fact, repeating this analysis it is clear that the London progress premium has existed for the last decade and is statistically accounted for by ethnic composition in each year. Again, it is easy to see why: London has seen a relative decline in its fraction of low-scoring pupils and a corresponding increase in high-scoring pupils:

Figure 3

There are two important cases in which accounting for ethnicity reduces (halves) the London premium but does not entirely eliminate it. If we consider GCSE points excluding vocational qualifications, then there remains a small but significant London effect. This in turn arises because pupils in London were entered for significantly fewer of these ‘equivalent’ qualifications. The implications of this are unclear. It is certainly not appropriate to simply remove some subject scores from the total and claim that the remainder represents ‘true’ progress. Nevertheless, the fact that London schools systematically entered pupils for less of these qualifications may represent the outcome of a particular policy.

Second, a measure of very high exam performance also yields a small but significant London effect once ethnicity is included. While this may also be the result of policy, it is also plausible that it derives from the very high concentration of professional families in London and their high input into their children’s education (which is not captured by a socio-economic control variable that just measures eligibility for free school meals).

Many policy makers, school leaders and commentators enthuse about the major policy of the time, London Challenge, and view it as unambiguously improving schools in London. This unanimity carries weight, and no doubt London schools were improved in a number of ways. But so far at least, catching a reflection of this improvement in the attainment data is proving to be difficult.

It sounds somehow uninspiring and disappointing that the London attainment premium is largely “accounted for by demographic composition” rather than wholly caused by some innovative policy. I disagree. It can be seen as a story of aspiration and ambition. There is nothing inherently different about the ability of pupils from different ethnic backgrounds, but the children of immigrants typically have high aspirations and ambitions, and might place greater hopes in the education system.

London has a right to be pleased with itself in terms of the excellent GCSE performance of its pupils. These results help to explain the ‘London Effect’; they do not explain it away. My argument is that the London effect is a very positive thing, and much of the praise for this should be given to the pupils and parents of London for creating a successful multi-ethnic school system.

Why are pupils from disadvantaged families more often found studying in poorly performing schools?

October 6, 2014 Leave a comment

By:  Simon Burgess, Ellen Greaves (IFS) and Anna Vignoles (University of Cambridge)

Why are pupils from disadvantaged families more often found studying in poorly performing schools? Is it choice or is it constraint? Is it because the families choose local schools despite low performance? Or is it because the school admissions system which focusses on proximity to school works against poorer families?

New research from CMPO, IFS and Cambridge shows that differences in constraints in school choice between households of high and low socio-economic status drives the unequal allocation of pupils across schools much more than differences in preferences.

Our research looked at the choices of primary schools made on the Local Authority application form for thousands of families in England (using the Millennium Cohort Study). We first show that there are substantial differences in the academic quality of the local schools for families in different socio-economic groups. On average, the richest socio-economic quintile of families have schools nearby that get grades 46% of a standard deviation than those available to families in the poorest fifth of families.

On top of that is a second layer of difference. When popular schools have more applicants than places, there has to be some mechanism to ration those places. The most widespread criterion in England is proximity – families living closer to the school are given priority. Our study shows that this adds a substantial further difference to the quality of available schools – the gap in average school quality for the richest and poorest fifth of families is one third greater when considering schools feasible in terms of distance and the probability of admission. It is this criterion that is responsible for a significant component of inequality in access to high-performing schools.

The figure shows the mean school academic quality available to families across the socio-economic spectrum, differentiating between those that are nearby but not necessarily feasible (“local”) and those that are have a high probability of admission once we take account of the proximity rule (“proximity”).

GraphFINAL

Of course it could still be true that differences in preferences are much more important than these differences in available choices. But our analysis shows that this is not the case. Different families have different sets of schools to choose from: richer families choose between schools that on average have higher performance than poorer families. The differences in school choices made between households with higher and lower socio-economic status therefore largely reflects differences in the available schools.

Our research related families’ choices of school to the attributes of all local and feasible schools to estimate the strength of the families’ preferences for these attributes. We also investigated the variation in observed preferences between socio-economic groups. Our results show that families care about three main school attributes: the academic quality of the school, its socio-economic composition, and the home-school distance. The majority of households prefer schools with higher academic standards. On average, families prefer schools with fewer children living in low-income households. Almost all households have strong preferences for a school near where they live. Preferences differ across socio-economic (SES) groups: those in the lowest SES group in particular have distinct preferences, with a preference for lower academic quality and a higher proportion of pupils from low-income backgrounds, on average. Households from each SES group value proximity to the same extent, however.

In summary, households from different SES groups face differences in the type of schools available, but also have different preferences to some extent. Both factors may contribute to the unequal presence of pupils from different SES groups in different schools: we wanted to know which factor was more important. Because of the spatial clustering of our survey data, the majority of our sample share feasible school choice sets with at least one other household. This allows us to control for all the characteristics of the choice sets (technically, to define choice set fixed effects) and therefore to compare the choices of households from different SES groups confronted with exactly the same feasible school choices. This allowed us to decompose the overall relationship between SES and the academic quality of the chosen school into a component due to different preferences and a component due to differences in school quality across feasible choice sets. We show that the constraints account for two thirds of the overall observed difference. These constraints are largely driven by admissions criteria (principally the proximity criterion), which means that choice is restricted for some households.

To address these questions we assembled a unique dataset. We used survey information on parents’ primary school choices and a rich set of family socio-economic and neighbourhood characteristics. We linked this to administrative data on the characteristics of schools, and the nature of the local school choice mechanism. To identify parents’ constraints in terms of available school choices, we used the national pupil census with embedded spatial information to model de facto catchment areas around schools within which there is a high probability of admission.

The broader implications of our results for choice in education are mixed. Most parents in our data have a strong preference for schools with high academic attainment. This supports the idea that competition to meet these preferences should help to raise the standards in England’s schools. The measure of academic attainment we use is an absolute measure of test scores as this is what parents are most likely to be familiar with, and not an estimate of school effectiveness. How schools try to increase their test scores, either through increasing effectiveness or manipulating their intake of pupils, is therefore another important question in the chain between parental preferences and school effectiveness.

Our results confirm parents’ preference for a school near to home. We are confident that this is a true preference and not the result of proximity enabling entry, as all schools in the set we define as having a high probability of admission should be considered feasible by parents. This implies the existence of de facto local monopolies, not through the lack of choice, but through strong preferences for proximity amongst parents of primary school children, perhaps due to transport costs and practical considerations of travel with young children.

What can be done? Popular schools cannot take everyone who applies – there has to be some mechanism to ration places. If we want to break the link between access to high-performing schools and family income, then we need an alternative to proximity as a tie-breaker. A lottery for over-subscribed places is one idea used around the world, but only infrequently in England. Schools could set aside a fraction of places for applicants who live beyond the “catchment” area of the school. Alternatively, schools could apply a banding system often used in the past, whereby schools’ intake of pupils is spread across the distribution of prior attainment or socio-economic background.

The overall goal for policy is to make all schools excellent. But until that nirvana arrives we cannot ignore the question of how places in the better performing schools are allocated. And at the moment, the proximity criterion for admissions means that differences in family income have a substantial and regressive impact on that allocation.

Categories: Uncategorized

Social contagion in giving

August 14, 2014 Leave a comment

Sarah Smith

Nomakeupselfie showed how charitable giving can be contagious. It powerfully demonstrated the potential to harness social networks to spread giving to a good cause.

But recent online experiments suggest that nomakeupselfie may be the exception rather than the rule. Researchers looked at what happened when they gave donors the opportunity to ask others in their social network to join them in supporting the same charity. The donors could ask others by posting on their Facebook wall or sending a message to a single friend (the two options were offered randomly).

More donors chose to post to their wall than to send a private message – but very few did either (7% and 4% respectively). The response rate – measured by the number of those who received the suggestion who made a donation – was also very low at 1.2%, all from the wall posts. Putting the two together suggests that it would take more than 1,500 donors (and their social networks) to yield a single extra donation.

Replicating the nomakeupselfie may be hard, but there are ways that charities can encourage giving through social networks.

First, online individual fundraising is more effective than a simple “ask”. Most of the people who are asked for sponsorship are the fundraiser’s friends, family and colleagues. A survey of JustGiving donors showed that of those asked to sponsor, 96% had been asked by a friend (of whom 67% always gave), 89% had been asked by a colleague (48% always gave), 84% had been asked by a family member (87% always gave) and 70% had been asked by a charity representative (9% always gave). The typical JustGiving donor who links their fundraising page to their Facebook page has 251 Facebook friends and gets nine donations – an implicit “response rate” of 3.6%. In smaller networks, the response rate is even higher. Two things make online individual fundraising a more powerful ask – first, the fundraising activity itself (the fact that someone is not just asking for charity, but running a marathon) and second, the fact that the donations are observable (the fundraiser can see who has responded to their request).

Second, incentives matter. The researchers also looked at what happened when they offered donors incentives (in the form of extra donations) to ask their friends. A $1 donation increased the percentage posting to their wall to 17% (from 7%) and sending a message to 9% (from 4%). A $5 donation increased the percentages further to 19% and 14%, but wasn’t cost effective.

It could be possible to use incentives to greater effect. At a schools charity challenge we held with sixth formers last year, the single best idea was to “pass the match” – to allow a donor to leverage incentives by passing on a financial match to a friend as a way of encouraging them to donate. It’s a clever idea to combine financial incentives with social pressure and one that would be well worth testing in the field.

Categories: Uncategorized
Follow

Get every new post delivered to your Inbox.

Join 46 other followers