The more I learn about the Research Excellence Framework (REF) the less convinced I am about the merits of this whole exercise. That universities are assessed to get some idea of how best to distribute a pot of money is fine. The way in which it is done, and the “games” that appear to be played by universities and university departments, is what concerns me. For starters, something like 300 senior people are involved in actually carrying out the assessments and numerous others are involved in preparing the submissions. The cost of doing this must be substantial (plus these are meant to be our leading researchers who are spending a large fraction of their time assessing everyone else). Some might argue that the amount being distributed (billions) makes it worth spending all this money carrying out the assessment.
An alternative argument might be that if ever it was an appropriate occasion in which to use metrics, it would be when assessing a large diverse organisation like a university. The problem with metrics (like citations) is that comparing different fields (or even different areas within the same field) is difficult because there might different citation practices in different fields and the size of the field plays a role. A typical university, however, has so many different fields that these variations should – to a certain extent – cancel and one could probably get a pretty good idea of the quality of a university by considering citations statistics and other metrics (number of spin-out companies, patents, etc.). One could also be a bit cruder in the rankings. I don’t really believe that we can rank universities perfectly. Rather than first, second, third…, it could be top 3, next 5, next 5, etc.
What concerns me more are the implications of what universities and university departments seem to be willing to do to optimise their REF scores. You can include research fellows in REF submissions and so there will be lots of carrots dangled to try to ensure that no Fellows leave before the REF census date in October 2013. Some of these research fellows may also be offered permanent positions that will start when their Fellowships end, either to keep them or to attract them away from another university. These will clearly be very good researchers, but I have an issue with a hiring practice in which holding a Fellowship plays a significant role in whether or not you will be hired. Getting a Fellowship is a bit of a lottery in the first place and what about those whose Fellowships are just due to end. It becomes a bit of a career year lottery – if you have a number of years left on a Fellowship at the same time as a REF submission you are more likely to get a permanent academic job than if you don’t.
There are also other issues. Departments will potentially be creating a number of new posts at a very uncertain time. What if things do not work as expected. How do you pay these people once they come off their Fellowships. What about the stability of academic careers. A burst of hiring every 7 years to coincide with REF submissions doesn’t seem very sensible. I should add, however, that if anyone who actually reads this has managed to get a permanent job or a promise of a permanent job, well done to you. I should also add that my views are not really based on anything specific, just a sense that we are letting the REF dictate our behaviour in a way that may not be ideal and wouldn’t be how we’d behave if the REF wasn’t happening. You have to worry slightly about the validity of an assessment exercise that has such a potentially strong influence on the behaviour of the organisations it is trying to assess. Can’t really be regarded as independent.