I was at a meeting yesterday where we discussed our REF strategy. I probably shouldn’t say what it is (might be confidential), but it didn’t increase my confidence in the basic system. For those who don’t know, REF is the Research Excellence Framework and the basic idea is that all university departments will be assessed to determine the quality of their research, the wider impact of their research, and the vitality of their research environment.

In fairness, what the REF is attempting to do is not inherently bad. The precursor to REF was the RAE (Research Assessment Excecise). In RAE2001, if I remember correctly, individuals were assessed and given a score. They were not told (I think) what their scores were, but departments were then given a final score based on the scores of the individuals in that department. In RAE2008, rather than scoring individuals, outputs were scored. Each person who was included by a department would submit 4 papers (with brief descriptions), some invited talks and other forms of output. These were then ranked on a scale from 1* to 4*. The advantage of this (in my view) is that an individual could have some outputs that score 4* and some that score 1*, so many could contribute to the 4* outputs of a department. A department was then given a score that was essentially what fraction of their ouputs were 4*, 3*, 2* and 1*. If a department had 25% 4* it wouldn’t be known whether this was because only 25% of the individuals produced 4* outputs or if a quarter of everyone’s outputs were 4* (or somewhere inbetween, as is more likely).

The amount of money given to a university was then based on what fraction of the outputs were 1*, 2*, 3* and 4*. I forget the exact formula, but it was something like amount*[(fraction of 1*) + (fraction of 2*)*2 + (fraction of 3*)*5 + (fraction of 4*)*7] multiplied by the number of people submitted. The 3* and 4* outputs therefore counted much more than the 1* and 2* outputs. That money was given for 1* and 2* outputs was, recently, heavily criticised by Vince Cable who (incorrectly in my view) interpreted this as giving money for mediocre research.

As a consequence of the above view, it appears as though 1* and 2* outputs will not receive any funding from the upcoming REF. This, consequently, has implications for the strategy that departments might choose to use. The two strategies that I’m aware of are, firstly, to submit as many people as possible, which will dilute the fraction of 3* and 4* outputs but the reduction in amount per person may be more than compensated for by the fact that there are more people submitted. The second strategy is to submit fewer people so as to minimise the fraction of 1* and 2* outputs (and hence increase the fraction of 3* and 4* outputs) and hope that the reduction in the number of people submitted is compensated for by the fact that the amount per person increases sharply with increasing fraction of 3* and 4* outputs. The advantage of the latter strategy is that it is also likely lead to a higher place in the rankings table, which is often regarded as extremely important.

The problem that I have with the above is not the university departments are chosing strategies (which is largely because they don’t actually know how the assessment scores will translate into money), but that strategies are necessary at all. The REF is meant, in my opinion at least, to be an attempt to objectively assess the quality of research in UK universities. That two essentially identical departments could end up with different scores depending on their chosen strategies suggests that the process is flawed. It’s not meant to be about whether they can guess the best strategy or not. It’s meant to be producing a measure of their quality (relative to other departments in the same area). The future funding of UK universities should – ideally – be based on objective measures of quality, not on whether the stategy gamble paid off or not.