Some more REF thoughts

The post about my REF interview seems to have generated a modest amount of interest in the last day or so. There were no comments, so I can’t tell if others identified with my experience and agreed with my general views, or disagreed and thought it was all a load of nonsense. However, seeing that post generate a little interest reminded me that I had seen some interesting data recently about REF outputs. For those that don’t know, REF is the Research Excellence Framework and is an exercise in which the quality of research in UK universities will be judged and the results will determine how to divide up a fairly substantial pot of money. What makes it more “interesting” is that the formula that decides how much money each university gets is highly non-linear. There is a big difference between doing “very well” compared to simply doing “well”.

What will be assessed will in general be papers published by academics in each institution. Typically, there will be 4 papers – published since 2008 – for each academic included in the submission. The intention is that each paper will be judged in terms of its originality, significance and rigour and will be given a score of either 4*, 3*, 2*, or 1*. The claim is that the panels doing the judging will not be using Journal Impact Factors or citations to make their assessment. It has, however, already been pointed out that this claim is unlikely to be credible. In Physics, there will probably be something like 6500 papers each of which will supposedly be read by 2 of the 20 panel members in a period of about 12 months. In other words, at least 2 papers per day each. Pretty difficult to do. Virtually impossible to make a credible judgement of each paper. The general view is that, despite what is claimed, Journal Impact Factors and citations will indeed be used to judge these papers.

Here’s what I found interesting. According to what I saw recently, a paper published since 2008 that is receiving about 8 citations a year will be in the top 10% according to citation numbers. I was a little surprised. I assumed that the top 10% of papers (according to citation numbers) would be receiving more than 8 or so citations a year. I decided to look into this myself using Web of Knowledge. If you search for all refereed articles published in the general area of Physics that also have “UK”, “United Kingdom”, “England”, or “Scotland” in the address you discover 38176 refereed articles published since January 2008. Web of Knowledge can’t do citation statistics on more than 10000 papers. I divided these papers into 5 categories (Condensed Matter, Astronomy & Astrophyscs, Particle Physics, Nuclear Physics, Mathematical Physics). I also included a randomly chosen sample of areas in Physics that, together, hadn’t published more than 10000 papers since 2008. The table below shows the average number of citations per paper, the number to be in the top 1%, the number to be in the top 10%, and the median.


Indeed, it seems that the average number of citations per Physics paper published since 2008 is about 10 and to be in the top 10% of all physics papers published since 2008 you need to be collecting fewer than 10 citations per year. Although it is different for different areas of physics, the difference isn’t particularly large. One issue with the above table is that the older papers will have collected more citations than the newer papers. I then repeated the above, but considered only papers published – with a UK author – in 2008, 2009, 2010, 2011, and 2012. In this case there are typically between 7000 and 7500 refereed articles published per year, so I didn’t divide it into different disciplines, but considered all articles in physics. The table below shows the result.


The result seems about the same. To be in the top 10% of papers published in any year since 2008 a paper needs fewer than 10 citations per year. Essentially, for a paper to be in the top 10% of cited papers, requires a fairly small number of citations per year. Alternatively, most papers seem to receive very few citations. What to make of this? Partly, I was just a little surprised. If asked, I would have guessed that to be in the top 10% of cited papers would require more than 10 citations per year. Also, what does the fact that a large fraction of refereed articles attract very few citations per year imply? Does it mean that much of what we publish isn’t particularly interesting. Although I think we probably publish too many papers, I don’t think that 90% of what we publish is worthless. Quite a large number of those papers receiving very few citations must be excellent bits of research that are worth publishing. Maybe they just haven’t been noticed. Maybe they’re what is referred to as slow-burners. Maybe it was a necessary step that has been superseded by a newer bit of research but that isn’t getting the citations that it might deserve. Maybe it’s something a researcher enjoyed doing, learned a lot by doing and that then allowed them to move on to something newer and more interesting.

What’s more interesting is how citations can then be used to judge these papers. We will presumably be submitting something like 6500 papers, so potentially 17% or so of all refereed physics papers published since 2008. Only papers judged to be 3* or 4* will attract money. One could assume that 3* and 4* papers will be those with much higher than average number of citations. This would then imply that a small number of papers will be used to determine how to divide up the large sum of money associated with REF. Small variations could then have a big effect. On the other hand, if 3* papers are not necessarily those with much more than the average number of citations, how do you then distinguish between 3* and 2* papers. Most papers are collecting fewer than 10 citations per year. Where’s the division? Is 3 a year 2* and 6 a year 3*? Alternatively, we shouldn’t really use citations and metrics and should judge each paper on it’s originality, significance and rigour (as suggested in the REF documentation). The problem is that very few, if any, believes that it is possible for a panel of 20, however distinguished, to do this.

The truth is probably that it will be a combination. The panel members will, I’m sure, try to read the papers and will then use metrics to fine tune their scores. However, combining two largely flawed processes to try and determine the quality of research activity in UK universities doesn’t really seem like much of an improvement. I suspect that, at the end of the day, a ranking will be produced that isn’t entirely unreasonable. However, as I’ve pointed out before, it should be possible to achieve a reasonable ranking in a manner that doesn’t use up quite as much time and effort as REF is currently doing.


3 thoughts on “Some more REF thoughts

  1. That’s an interesting analysis. I did already have some idea of the numbers. In choosing which of my papers to put forward for consideration, I picked one from 2012 which already has 17 citations, and is from one of the subject areas in your post that is on the lower side of citations. In fact, I could see from ISI that it was the second-most cited paper from 2012 in the particular Phys Rev journal.

    My university’s internal ranking procedure deemed the paper below threshold, though another from the same journal from the year before, with fewer citations was above. How they actually judged the papers is a mystery to me, but it sure wasn’t on metrics.

    • Interesting. We have had a similar things happen here, and with a similar sense of uncertainty as to what measure had been used to judge the papers. I guess my post was partly trying to illustrate how difficult it is. Using metrics is difficult because most papers don’t receive many citations a year. Expecting a small pool of scientist to assess – by reading – papers across a wide range of different areas is also difficult. My basic view is that the whole process is flawed and although the final rankings may be okay, it just seems silly to spend so much time and effort on something that is probably impossible to do with any real rigour.

  2. Pingback: REF again! | To the left of centre

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s