In a previous post (REF prediction) I looked up the h-indices and the citations per publication for all Physics and Astronomy departments included in RAE2008. I ranked them in terms of their h-index, in terms of their citations per publication, and as average of these two. It looked alright but I did comment that one could produce something more sophisticated. At the time I did worry that using just the h-index would disadvantage smaller departments, but I couldn’t really think of what else to do and it was just a very basic exercise.
Deevy Bishop has, however, suggested an alternative way of ranking the departments. This is to basically relate the income they get with their h-index. For example, in RAE2008 each department was ranked according to what fraction of their papers were 4*, 3*, 2*, 1* and U. The amount of funding they received (although I think it technically went to the university, rather than to the department) was then scaled according to N(0.1×2* + 0.3×3* + 0.7×4*) where N was the number of FTEs submitted. This data can all be downloaded from the RAE2008 website. Deevy Bishop did an analysis for psychology and discovered that the level of funding from RAE2008 correlated extremely well the department’s h-index. What was slightly concerning was that the correlation was even stronger if one also included whether or not a department was represented on the RAE2008 panel.
I’ve now done the same analysis for Physics and Astronomy. I’ve added various figures and text to my REF prediction post, but thought it worth making it more prominent by adding it to a new post. The figure showing RAE2008 funding plotted against h-index is below. According to my quick calculation, the correlation is 0.9. I haven’t considered how this changes if you include whether or not a department was represented on the RAE2008 panel. The funding formula for REF2014 might possibly be N(0.1×3* + 0.9×4*). I’ve redone the figure below to see what the impact would have been if this formula had been used instead of the RAE2008 formula. It’s very similar and – if you’re interested – it’s included at the bottom of my REF prediction post. It does seem that if all we want to know is how to distribute the money, relating it to a department’s h-index seems to work quite well (or at least it would have worked well if used for RAE2008). I’m not quite sure how easy it would be to produce an actual league table though. Given that the REF2014 formula may depend almost entirely on the fraction of 4*, one could simply divide the h-index by the number of FTEs to get a league table ranking, but I haven’t had a chance to see if this produces anything reasonable or not. Of course, noone really trusts league tables anyway, so it may be a good thing if we don’t bother producing one.