So I recently had an interview with our Head of Department and our Director of Research to discuss my inclusion in the Research Excellence Framework (REF2014). Although I had expected some problems, my interview went fine and it looks as though I’m “good enough” or, more correctly, I have my name on 4 papers that are “good enough”. Some of my colleagues were not particularly happy after their interviews, but I don’t feel all that comfortable discussing their issues in any detail. As you may know, if you’ve read any of my other posts, I’m not a huge fan of REF. Even though my interview went fine, my views haven’t changed much and – if anything – it’s rather confirmed my general views of the issues with the process.
I don’t really want to go into specifics as maybe I shouldn’t be giving away our strategy. I thought, instead, that I would give some general comments. It’s clear that it’s a game, and everyone knows this. There are two parameters; money and league table ranking. Most institutions will be adjusting their submission to optimise between these two parameters. This is not an objective assessment of research quality. It’s a game to try and submit the optimum set of papers that will get you as high as possible on the League table and get as you much money as possible. Some institutions may determine that it will be better to sacrifice league table position for money and others may choose to prioritise league table position over money.
It was claimed that submissions (papers) would be read and assessed and that metrics would not be used. Noone believes this to be true. We’re clearly using Impact Factors and citations at some level. Papers in Nature or Science will be 4* simply because they’re in Nature or Science. Papers in other good Journals will be 4*, 3* or 2* depending on the number of citations. There’s also no rigorous assessment of ownership. As long as you can indicate that you contributed sufficiently to a paper, your institution will get full credit. Someone who makes modest contributions to 4 papers that are regarded as excellent, is more valuable to an institution than another person who writes 4 good papers, none of which are regarded as excellent.
You could argue that the 4 excellent papers are better than the 4 good papers, but there is a chance that our careers may depend on how our 4 papers are judged for REF. Is it really better to have average people who are good at getting their names on excellent papers, rather than good people who write their own. I may be slightly biased in that I’m probably more in the latter category than the former, but I would say that I don’t intend for my papers to only be “good”. I always aim to write papers that I think are tackling interesting and challenging problems and I always aim to be as careful and rigorous as I can reasonably be. I always try to write something that would be regarded by others as excellent. The citations, however, don’t always indicate that they have been received particularly well. Of course, the definition of excellence is very tricky and using simplistic metrics is not necessarily a good way to determine the quality of a piece of research. In discussing this, someone commented that it is not unreasonable to expect academics to write 4 good papers every 7 years. In some sense I agree, and I would certainly back myself to write 4 good (maybe even some excellent) papers every 7 years. What I have much less confidence in is the ability of those who are judging these (either on the REF panel or in my own Department) to do so properly.
It seems that there is a real chance that we will aim to adapt our research strategy to suit future REF exercises. It’s clear that independence and originality is no longer necessarily optimal. It might be much better to have people who work in collaborations that are likely to publish papers that will be highly cited. I could easily see people being encouraged to make sure that they belong to such a collaborations and that they aim to contribute sufficiently to at least 4 papers so that their institution can submit these 4 papers to REF. Maybe this won’t happen, but I find the possibility of this happening quite disturbing. What I find slightly more disturbing is that the senior people that I encounter acknowledge that it is a game, but seem to feel that it is a game we need to play. Noone seems that bothered that we might be adjusting our research strategies to suit what is essentially an assessment exercise. I appreciate that there is a lot of money involved but, as far as I’m concerned, either our research is valuable at some fundamental level, or we really shouldn’t be bothering. Maybe that’s a little extreme, but hopefully you know what I mean.
That’s probably all I was going to say. As I may have mentioned before, a concern of mine is that over time we will adapt so that top UK institutions are very good at scoring well on REF exercises but don’t really do research that has any particular value. Maybe I’ll be proven wrong and that once REF is finished we’ll forget about it for a while and everyone will be left alone to get on with whatever research they think is most interesting and valuable. I somewhat doubt that and I do worry slightly about the careers of those who are deemed to not have enough papers that are good enough for REF. Will they be marked and will their jobs be at risk. I think that would be awful if it did start happening. I have no real issue with people who are not contributing positively to the running of an academic being sanctioned. I do, however, have an issue with the possibility that some people’s careers could depend largely on an assessment exercise that, in my opinion, is horribly flawed.