Statistical myopia, measures of unemployment and economic reporting
Today the ONS published a statistical bulletin of labour market statistics, which was widely reported as evidence of an increase in unemployment. However, the executive summaries provided by the ONS did not include sufficient information about the precision of the statistics. This led newspapers to report ‘changes’ in important features of the economy, such as the unemployment rate, that may in fact be due to chance.
Labour market statistics, provided by the ONS, are estimates, not exact measures. The statistics are calculated from surveys of thousands of people. This means that the statistics are subject to sampling variation. Therefore, if the ONS were to repeat their survey and calculate all their statistics on a different group of people we would not expect them to be the same. The ONS provide estimates of the size of this variation in additional tables: they estimate the range in which their statistics would be expected in 95% of samples. However, these ranges are not reported in the executive summaries, or in the much of the discussion of what the statistics show. The findings of the ONS bulletins are faithfully reported to the public by newspapers, but unfortunately without a measure of sampling variability, it is not possible to tell if the level of unemployment has changed, or if the reported change is consistent with chance.
The summary of latest labour market statistical bulletin reports the following statistics, to which I have added confidence intervals using the measures of sampling variation provided by ONS later in the document:
- The employment rate for those aged from 16 to 64 for the three months to November 2010 was 70.4 (70.0, 70.8) per cent, down 0.3 (0.0, 0.6) on the quarter. The number of people in employment aged 16 and over fell by 69,000 (-62,000, 200,000) on the quarter to reach 29.09 (27.53, 30.64) million.
- The unemployment rate for the three months to November 2010 was 7.9 (7.7, 8.1) per cent, up 0.2 (-0.1, 0.5) on the quarter. The total number of unemployed people increased by 49,000 (-37,000, 135,000) over the quarter to reach 2.50 (2.42, 2.58) million.
- There were 157,000 (137,000, 177,000) redundancies in the three months to November 2010, up 14,000 (-14,000, 42000) on the quarter.
- The inactivity rate for those aged from 16 to 64 for the three months to November 2010 was 23.4 (23.1, 23.7) per cent, up 0.2 (-0.1, 0.5) on the quarter. The number of economically inactive people aged from 16 to 64 increased by 89,000 (-26,000, 204,000) over the quarter to reach 9.37 (9.23, 9.50) million.
There are 7 estimates of change in levels. All seven of these changes are consistent with what might be expected from sampling variation, so the statistical bulletin provides no strong evidence of a quarter on quarter change in any of these statistics. The statistical bulletin contains further statistics that report the change in employment in sub-groups of population, by age or by region. Each of these sub-populations is smaller, and hence less precisely measured. Therefore estimated changes in unemployment in sub-populations, such as youth unemployment, are more likely to be due to sampling variation. However, we cannot know, as the sampling variation of these sub-populations are not provided by the ONS, so it is not possible to tell if these differences are important, or if they are just due to chance.
Understandably, given the executive summaries of the statistical releases, newspapers report that unemployment is rising, when the data do not support this. This leads newspapers to miss the bigger story: unemployment is still very high after the recession, and there is little evidence of falls in employment. The ONS could help by stating confidence intervals in their statistical bulletins to enable journalists to easily interpret whether the information in a report is consistent with chance, or if there is evidence of a genuine change. We know long term measures of unemployment, for example annual measures, with much more precision, because we have larger samples, so if measures of precision were to be more widely reported then debates over the economy might be slightly less myopic.