Kafkarna continues: REF gloves off at Lancaster University

I thought I would reblog this. Partly because I haven’t had a chance to write much recently and it gives me an opportunity to keep things ticking over, partly because it seems like something worth highlighting (although given my readership, this may not help much), and partly because I’ve written about the REF (and been quite critical) and this type of activity is one of the things that I expected to happen and illustrates the issues – in my opinion – with this type of assessment process. I particularly like, and agree with, this comment in the post Whatever this charade is, it is not a framework for research excellence and so I recommend giving this a good read.


Two days ago an open letter from Professor Paolo Palladino, former Head of the History Department at Lancaster University, appeared on the university’s internal History Staff list serve, with copies sent to senior administrators in the Faculty of Arts and Social Sciences (FASS) and Central Administration.  I responded the next day with a post supporting Professor Palladino’s position, sent to the same recipients. With Paolo’s permission, I am reproducing both of our letters here.  We both believe that the issues raised by Lancaster’s selective culling of research-active staff from submission in the 2014 REF–a practice in which it is not alone among British universities–deserve the widest possible public debate.  We would therefore urge anyone who shares our concerns to share this post through social media.

Professor Palladino’s letter:
Dear all,
Over the next few days, a number of colleagues across the university are to be informed that they will not…

View original post 1,060 more words

REF, QR funding and the science budget

I guess we should all be reasonably pleased that the science budget has remained ring-fenced. The reality, of course, is that this just means that it is ring-fenced in a flat-cash sense, not in an inflation adjusted sense. As scientists, maybe we should make sure they define the term more specifically in future. It does appear that the current definition would allow the government to claim that something has been ring-fenced, despite the spending power tending to zero. Logically, you mighty expect it to be defined in terms of spending power rather than pounds, but that would require deciding between RPI and CPI, and that is clearly for too difficult.

Anyway, enough cynicism. In truth, we should probably be grateful that the outcome hasn’t been worse. It certainly has been for some and I do feel that the current government has got its basic economic policies completely wrong. It seems like it’s time that someone explained to George Osborne that it’s not necessarily the size of the debt and deficit that matters. What matters is their size relative to the size of our economy. The way its going now it seems like he’s getting it wrong on both counts.

What is maybe more concerning, from a science funding perspective, is the possibility that the government may choose to axe the QR funding. This is the funding stream that comes from the Higher Education Funding Councils and how it is distributed is determined via the outcome of the Research Excellence Framework (REF) exercise. Now, it may well be part of the ring fence and it may well be safe, but I wouldn’t know whether to laugh or cry if it were cut. If you’ve read some of my earlier posts, you’ll know that I’ve been very critical of REF. This is both due to the manner in which it is implemented and due to the shenanigans taking place at UK universities; the potentially risking hiring, the morally/legally questionable redundancies, and the time and effort spent preparing for what is – in my opinion – a completely flawed exercise.

So, if it were to be cut, part of me would feel like saying “serves us right for taking something so silly so seriously, and for playing the kind of games that have not and will not benefit our fundamental role as teachers and researchers”. On the other hand, it is a lot of money (£1.5 billion I believe) and I certainly have no desire to see this money leave the Higher Education sector. As far as I can tell, some may struggle to survive as they are, even if there are no cuts to the QR funding. Well, I certainly hope that it isn’t cut but I also hope that in future, universities will find the backbone to tell the government that playing these kind of games is silly and that they should find a simpler and more effective mechanism for distributing this money (although I don’t think it should simply be given to the research councils, but that might be a topic for another post).

Declaration on Research Assessment

Just thought I would highlight the San Francisco Declaration on Research Assessment. It’s been a long day and I’m quite tired, so I don’t want to say too much. You can read most of this yourself, but basically a group of editors and publishers of scholarly journals met during the Annual Meeting of The American Society for Cell Biology (ASCB), and have come up with a set of recommendations about the use of journal-based metrics. The basic motivation was

  • the need to eliminate the use of journal-based metrics, such as Journal Impact Factors, in funding, appointment, and promotion considerations;

  • the need to assess research on its own merits rather than on the basis of the journal in which the research is published; and

  • the need to capitalize on the opportunities provided by online publication (such as relaxing unnecessary limits on the number of words, figures, and references in articles, and exploring new indicators of significance and impact).

  • A lot of it seemed to focus on the use of Journal Impact factors when assessing individual bits of research (which, as many have already pointed out, is horribly flawed) but there was indication that this was attempting to go further than just that. Citation metrics can be useful but can also be problematic. There can be a huge range of different practices even with the same basic area and so using them alone to assess an individual can disadvantage some potentially excellent researchers. It can also be advantageous to some who aren’t particularly good but who just happen to work in an area that is currently popular and in which papers are collecting citations easily.

    Anyway, this is an encouraging step and I hope it has some impact and that it’s taken seriously by funding agencies, interview panels and promotion boards. I suspect it’s too late to have much effect on the REF2014 panels, but maybe there’s hope for REF2021.

    The integrity of universities

    There’s a very interesting (in my opinion) article by George Monbiot in the Guardian today. The article is called Oxford University won’t take funding from tobacco companies, but Shell’s OK. The basic premise is that universities should be acting for the common good, or as George Monbiot puts it

    the need for a disinterested class of intellectuals which acts as a counterweight to prevailing mores.

    I have to say that I agree completely with this. It has always surprised me how disinterested UK academics can be. I had always assumed that university academics had a role to play in defining what is acceptable in our societies. They are meant to be the intellectual members of our society; the people who think. If academics are reluctant to be involved in this, then who else is going to do it? This isn’t to say that everyone should bow to the views of academics, simply that academics should feel free to question what is accepted in our societies.

    There are probably many reasons why UK academics are reluctant to engage in discussions about our society. One may simply be that academics have become very focused. They see themselves as experts in quite specific areas and so don’t see it as appropriate to engage in areas outside their expertise. There is some merit to this, but it is a bit disappointing – in my opinion. Another may be that there is now quite a lot of pressure on academics. Universities have become very bureaucratic and there is quite a strong publish-or-perish attitude. Academics don’t have much spare time to contemplate the merits – or lack thereof – of our societal mores. Universities have also become much more like businesses. The goal is to maximise teaching and research income and, hence, academics are discouraged from doing anything that doesn’t enhance a university’s ability to generate income.

    Personally, I think the latter is the main reason why universities (in the UK) are no longer hotbeds of dissent. We are publicly funded and hence need to do what is expected of us. I’m often very critical, for example, of the Research Excellence Framework (REF2014) and even though most seem to agree, the typical response is “we just have to do this”. Well, yes, but do we have to do it happily. If we think it is damaging what we regard as strengths of the UK Higher Education system, shouldn’t we be making it clear that we’re doing it under duress. There’s also this view that we have to do what is best for UK PLC (i.e., what will best help economic growth in the UK). In a sense, I agree with this. What I disagree with is how we’re influenced to do this. University research has, for many decades, had a very positive impact on economic growth. However, this didn’t happen because politicians told universities to do this. It’s because people recognised the significance of some piece of research and used it to develop something that had economic value. It’s also largely unpredictable. It’s certainly my view that telling us to start predicting the economic benefit of our research will do more harm than good.

    The final thing I was going to say regards the main thrust of George Monbiot’s article. If there is increasing evidence that global warming is happening (as there is) and if there is evidence that such warming could lead to life-threatening climate change, shouldn’t the universities where this research is taking place act as though it’s important. Maybe universities shouldn’t be accepting funding from oil companies, if their own research indicates that we should significantly reduce our use of fossil fuels. I have heard some argue that we shouldn’t worry about the provenance of our funding as we’ll typically do good things with whatever money we can get. I think this is naive. The idea that one can take funding from oil companies without being influenced by the source of this funding seems highly unlikely.

    REF and teaching

    There’s a recent article in the Guardian about the influence of the Research Excellence Framework (REF2014) on university teaching. The basic issue is that money will only be allocated on the basis of papers that score highly (3* and 4*) and that league table rankings will also be determined by these high-ranking papers. Therefore, there is an incentive for universities to only submit researchers who have enough (essentially 4) papers that will be judged to be 3* or 4*. The concern is therefore that those who do not qualify will be encouraged (forced) to focus primarily on teaching or (as in the case of Queen Mary, University of London) face redundancy.

    Many universities are making “clear pledges that not being entered to the REF in November will not damage an academic’s career“. There are others, however, where this is clearly already having an impact (Queen Mary, University of London, Kings College and Strathclyde are three that I’ve heard about). I personally think that it is potentially a real problem. There is a big difference between how research and teaching are evaluated at universities, with an individual’s contribution to the research ranking being much more obvious than an individual’s contribution to any teaching ranking. One concern is that it will create a hierarchy within universities with some able to focus more on research and others “encouraged” to focus primarily on teaching and administration. I don’t have an issue with different people contributing to an academic department in different ways. I just would rather it were dynamic and evolved in some “natural” way, rather than being forced upon us by an external assessment exercise.

    University leaders are trying, in general, to make it clear that research and teaching are both valued parts of an academic’s career. The problem is that they don’t get to decide if the staff regard them as being of similar value. It certainly seems that even students are concerned about the impact that REF might have on the motivation of staff who might be judged to be “unworthy” and hence encouraged into having a larger role in teaching. I certainly think that these concerns are justified, even if there isn’t any evidence that REF is having, in general, this kind of impact.

    There do seem to be two common views expressed by those who are more supportive of REF than maybe I am. One is that it is not unreasonable to expect academics to publish 4 good papers every 7 years. In general I agree with this, although there may be some exceptions. However, there is a difference between publishing 4 good papers and publishing 4 papers that will be judged (by a panel – many of whom may not be particular expert in your field) to be good. Maybe about one-quarter of my papers have done quite well (in terms of citations) but I don’t really have a good idea why they did well and why others didn’t. I can’t really look back and claim that I can now tell why some papers would be judged to be good, while others would not. I’m typically quite pleased with most of the papers I publish. Whether or not they do well (in metric terms) all seems a little random to me.

    The other claim that is often made is that REF has forced universities to take hiring more seriously and that hiring is now based on excellence. Firstly, this is presumably only “perceived excellence” in research. One of the perennial criticisms of university hiring has been that teaching ability hasn’t been taken seriously enough. I really can’t see that REF has helped here. My feeling is that it may have made the situation worse. The other issue I have with this claim is that it suggests that the typical academic today is somehow better (because of REF) than they were 20 or 30 years ago. Really? I thought universities in the UK have been world-class for decades. I’m sure many academics who were active in the 60s, 70s, 80s and 90s might be slightly insulted by this suggestion. I suspect there were issues with hiring practices in those days, but that was probably more to do with societal issues that have been remedied via equalities legislation, than via REF.

    It strikes me that there has been quite a lot of recent coverage about the negative aspects of REF, so maybe some of it will sink in. Not that hopeful though. Maybe I should be considering holding back some of my current work so as to publish papers that will qualify for REF2021.

    A new REF algorithm

    In a previous post (REF prediction) I looked up the h-indices and the citations per publication for all Physics and Astronomy departments included in RAE2008. I ranked them in terms of their h-index, in terms of their citations per publication, and as average of these two. It looked alright but I did comment that one could produce something more sophisticated. At the time I did worry that using just the h-index would disadvantage smaller departments, but I couldn’t really think of what else to do and it was just a very basic exercise.

    Deevy Bishop has, however, suggested an alternative way of ranking the departments. This is to basically relate the income they get with their h-index. For example, in RAE2008 each department was ranked according to what fraction of their papers were 4*, 3*, 2*, 1* and U. The amount of funding they received (although I think it technically went to the university, rather than to the department) was then scaled according to N(0.1×2* + 0.3×3* + 0.7×4*) where N was the number of FTEs submitted. This data can all be downloaded from the RAE2008 website. Deevy Bishop did an analysis for psychology and discovered that the level of funding from RAE2008 correlated extremely well the department’s h-index. What was slightly concerning was that the correlation was even stronger if one also included whether or not a department was represented on the RAE2008 panel.

    I’ve now done the same analysis for Physics and Astronomy. I’ve added various figures and text to my REF prediction post, but thought it worth making it more prominent by adding it to a new post. The figure showing RAE2008 funding plotted against h-index is below. According to my quick calculation, the correlation is 0.9. I haven’t considered how this changes if you include whether or not a department was represented on the RAE2008 panel. The funding formula for REF2014 might possibly be N(0.1×3* + 0.9×4*). I’ve redone the figure below to see what the impact would have been if this formula had been used instead of the RAE2008 formula. It’s very similar and – if you’re interested – it’s included at the bottom of my REF prediction post. It does seem that if all we want to know is how to distribute the money, relating it to a department’s h-index seems to work quite well (or at least it would have worked well if used for RAE2008). I’m not quite sure how easy it would be to produce an actual league table though. Given that the REF2014 formula may depend almost entirely on the fraction of 4*, one could simply divide the h-index by the number of FTEs to get a league table ranking, but I haven’t had a chance to see if this produces anything reasonable or not. Of course, noone really trusts league tables anyway, so it may be a good thing if we don’t bother producing one.

    A plot of h-index against the RAE2008 funding formula - N(0.1x2* + 0.3*3* + 0.7*4*).

    A plot of h-index against the RAE2008 funding formula – N(0.1×2* + 0.3×3* + 0.7×4*).

    REF again!

    Some interesting posts recently about the forthcoming Research Excellence Framework (REF2014). An interesting post by Dave Fernig called In Defence of REF. This post does make some valid points. REF, and previous RAEs, may well have encouraged more sensible hiring practices in which the quality of the applicant is taken more seriously than maybe it was in the distant past. Two comments I would make are that it I still think that teaching ability is still not taken seriously enough and, in my field at least, many places have adopted a very risky hiring strategy that – I hope – doesn’t come back to bite us in 5 years time. Dave Fernig also seems to feel that the panel, in his field, can distinguish between excellent, good and mediocre papers. This may well be true in his field, but I don’t think it is for my field (physics).

    Peter Coles, who writes the Telescoper blog, has written a new post called Counting for the REF. I won’t say much about this post as you can read it for yourself, but I agree with much of what is said. Maybe the most concerning comment in the post was the suggestion that the weighting – when determing the funding distribution – would be 9 for 4* papers and 1 for 3* papers. Essentially, most of the funding would be determined by 4* paper and a very small amount would be associated with 3* papers. Fundamentally I think this is unfortunate as it gives very little credit to some very good papers and absolutely no credit to what might be quite good papers (there is no funding associated with 2*).

    There is a more fundamental concern that is associated with what is discussed in Peter Coles’s post. In a recent post (Some more REF thoughts) I pointed out that in Physics fewer than 10% of all papers get more than 10 citations per year. The claim is that two members of the REF panel will read and assess each paper. However, as pointed out by others, this would require each panel member to read 2 papers per day for a year. Consequently, it is impossible for them to give these papers as much scrutiny as they would be given if they were being properly peer-reviewed. There is an expectation that metrics (citations for example) will play an important role in deciding how to rate the papers. How could you do this? You could set a threshold and say, for example, that since most papers get fewer than 10 citations a year that 4* papers will be those that receive more than 10 citations a year. The problem that I have (ignoring that citations are not necessarily a good indicator of quality) is that this would then be a very small fraction (about 5%) of all published papers. The distribution of REF funding would then be being determined by a minority of the work published since 2008. This means that small variations can have a big impact on how the money is distributed. One could imagine that just a few papers being judged 3* instead of 4* could have a massive impact on how much money a department gets (I accept that the money doesn’t actually go to the department, but you probably know what I mean).

    Alternatively, if you want to avoid small variations having a big impact you would need 4* papers to make up a reasonable fraction of the assessed papers (maybe 10 – 20%). The problem here is that you’re now getting down to papers that are only collecting a few (5-10) citations per year, so where do you draw the boundary. Is 3 per year too few, but 5 a year okay. You could argue that these are just being used to to guide the assessment and that the panels’ reading of the paper will allow a distinction to be drawn between 4* and 3* papers. This doesn’t, however, change the fact that the panel members have to read a massive number of papers. It feels more like combining two completely flawed processes and hoping that what pops out the other side is okay.

    I suggested in an earlier post (REF prediction) that, given the diverse nature of a typical academic department or university, that this might be an appropriate time to simply consider using some kind of metric. I did a quick analysis of all 42 physics departments’s h-indices and saw a reasonable correlation between their h-index and how they did in RAE2008. I noticed today that Deevy Bishop, who writes a blog called BishopBlog, has made a similar suggestion and carried out the same kind of analysis for psychology. Her analysis seems quite similar to mine and suggested that this would be An alternative to REF2014.

    Anyway, it’s quite good to see others writing about REF2104 (whether for or against). I think it is a very important issue and I’m just disappointed that it is probably too late to make any changes that would make the REF2014 process simpler and more likely to produce a reasonable ranking.