Author: Michael Sanders
Arguing about funding obscures important issues of quality research
Richard Thaler, the Chicago professor of economics and incoming president of the American Economic Association, has as one of his many mantras the truism that “we can’t do evidence based policy without evidence”. The government’s recent decision to establish a number of “What Works Centres” to collate, analyse and, in some cases, produce, evidence on a number of policy areas seek to address the very problem of a lack of evidence.
Evidence itself, however, is not in short supply. Newspapers fill their pages, day after day, with the results of studies into some facet of human behaviour, or statistics on the state of the world. So, there need to be two other criteria for evidence than mere ‘existence’ – goodness, and usability. I should be clear at the outset that when I say “Good”, I mean “Capable of determining a causal relationship between an input and an output”. Sadly, not all evidence which is good is useable, and often tragically, not all evidence that is usable is good.
As Ben Goldacre points out in his recent paper for the department for education, many researchers in that field like qualitative work, and use this as the basis for their findings. As an economist, I have a natural scepticism for such research, but I cannot dispute that it is eminently useable. The arguments constructed by such research are easily and well presented. They offer solutions which are simple, and neat. However, as H.L. Mencken said, these arguments are almost always also wrong. This research is usable but very much of it is not good.
On the other side, much research which is good, and detailed, and thorough, presents complicated and nuanced answers which reflect reality but whose methods are impenetrable to anyone who might actually have the power to change policy accordingly.
Randomised Controlled Trials (RCTs) are both useable, with the majority of results presentable in an easily understood way and the methodology being simple enough to explain to a lay person in about five minutes. As the recognised ‘gold standard’ of evaluation, they are also indisputably good.
In a blog post for the LSE impact blog, Neil Harris, a researcher at Bristol’s Centre for Causal Analysis in Translation Epidemiology, argues that education research is a public good and needs to be funded by the state, as, unlike in medicine, there is not money to be made by researchers through patent development, education being a public good. He is, of course, absolutely right. The structure of his argument implies however, that in order to get good evidence, it will need to be paid for – i.e. that RCTs are expensive, while qualitative research is cheap. If the government wants better education research, they should give researchers more money. But, well, we would say that, wouldn’t we?
The argument that RCTs are expensive is a well-worn one, but is not helpful, and often dangerously distracting. Saying that an RCT is expensive is akin to saying “Vehicles are expensive”. If one chooses to put up Ferraris as an example, then of course they are. A scooter, however, is not. Both are better than walking.
A good quality, robust RCT need not be outlandishly expensive, and certainly not any more so than qualitative analysis. Unlike medical trials, the marginal cost of interventions in policy is often not far above that of treatment as usual (the most logical control condition). Teaching phonics in 50 schools and not in 50 others should not require vast resources once allocation has taken place. Although the government does not spend as much money on policy research as it does on medicine, it spends a lot of money gathering data on the outcomes many researchers are interested in. At the end of a child’s GCSEs, finding out how well they did does not require specialist staff to draw their blood and perform expensive tests on them. The school knows the answer.
It is important not to downplay the risks or costs associated with RCTs, but nor is it possible to present these costs as a reason for conducting, or accepting, substandard research. As researchers, if our work is of low quality, there is only so far the buck can be passed.
Incentives matter. Our research has shown repeatedly that this is true for the public sector as it is for the private sector: for teachers and schools, for doctors and hospitals and for civil servants. It is very likely also to be true for universities and those of us who work in them.
For the past couple of decades, universities have been very strongly incentivised to improve their research profiles. The evolving formats of the Research Assessment Exercise (now the Research Excellence Framework) have rewarded Departments and universities on the basis of their research output in a high powered way. This has been ferociously effective. As a whole, UK universities have vastly improved the quality and quantity of their research and now stand close to the very top of the international rankings.
One key insight is that while the RAE/REF itself is a collective Department-level incentive, this has trickled down to incentives for individual lecturers and professors. Universities keen to improve their research rating have created a “transfer market” for star researchers, and this has meant that recruitment effort, salary and respect have been focussed overwhelmingly on research ability. Young academics, wanting to get on, are aware of this and so spend their scarce time and energy on research.
This is not necessarily a bad thing – research is extremely important to a nation’s prosperity and cultural wealth. But it does mean that universities and individual academics have been incentivised to spend more time and resources on research than teaching. Does the Browne Review change any of this?
One of the less discussed points in the Browne Review is that new institutions can provide higher education (HE). Obviously, a new start-up university may find it hard to develop credibility for its degrees, but David Willetts, the Minister for universities, has floated the idea that they could teach towards the degree exams of established universities. This has worked in the past, and would give instant credibility to the degrees.
This opens up a range of possibilities. It seems unlikely that any single new institution would attempt to offer degrees across the whole range of disciplines. Instead we might see institutions offering, say, just a BSc in Computer Science, or just a BA in Spanish. This is reminiscent of the Independent Treatment Centres that transformed outcomes in health care; centres just doing cataracts or just hips. Obviously this does not provide the breadth of three years spent in a traditional university – chatting to people outside your subject, quizzing the great researchers in your field – but it would allow students to choose between these options and put a price on those factors.
Would this affect traditional universities, and alter the incentive structure for lecturers there? After all, this is where the bulk of students will be taught for the immediate future. It might. A new source of demand for talented degree teachers would raise their outside option and might force the traditional universities to pay more. The outcome depends in part on the co-production of teaching and research. Are good teachers good researchers, and vice versa, or not? What evidence there is suggests no strong correlation either positive or negative. In which case, there will definitely be an overlap in demand between traditional universities and new providers.
Of course, traditional universities will respond and make clear that their products are different, are distinctive. But they are likely to be more expensive too, and this gives students choices. There is likely to be a lot of innovation in institutional form and contracts following this path. How this will all pan out is unclear – the market for higher education is a complex one.
But it also creates a new market for talented degree-level teachers, and this may spill over into the pay and status of good teachers in universities. This in turn will encourage a re-balancing of lecturers’ effort towards teaching at the margin, and may have a greater impact on the quality of teaching in universities than any increased resources that may flow into the sector.