Kafkarna continues: REF gloves off at Lancaster University

I thought I would reblog this. Partly because I haven’t had a chance to write much recently and it gives me an opportunity to keep things ticking over, partly because it seems like something worth highlighting (although given my readership, this may not help much), and partly because I’ve written about the REF (and been quite critical) and this type of activity is one of the things that I expected to happen and illustrates the issues – in my opinion – with this type of assessment process. I particularly like, and agree with, this comment in the post Whatever this charade is, it is not a framework for research excellence and so I recommend giving this a good read.

coasts of bohemia

Two days ago an open letter from Professor Paolo Palladino, former Head of the History Department at Lancaster University, appeared on the university’s internal History Staff list serve, with copies sent to senior administrators in the Faculty of Arts and Social Sciences (FASS) and Central Administration.  I responded the next day with a post supporting Professor Palladino’s position, sent to the same recipients. With Paolo’s permission, I am reproducing both of our letters here.  We both believe that the issues raised by Lancaster’s selective culling of research-active staff from submission in the 2014 REF–a practice in which it is not alone among British universities–deserve the widest possible public debate.  We would therefore urge anyone who shares our concerns to share this post through social media.

Professor Palladino’s letter:
Dear all,
Over the next few days, a number of colleagues across the university are to be informed that they will not…

View original post 1,060 more words

Advertisements

REF, QR funding and the science budget

I guess we should all be reasonably pleased that the science budget has remained ring-fenced. The reality, of course, is that this just means that it is ring-fenced in a flat-cash sense, not in an inflation adjusted sense. As scientists, maybe we should make sure they define the term more specifically in future. It does appear that the current definition would allow the government to claim that something has been ring-fenced, despite the spending power tending to zero. Logically, you mighty expect it to be defined in terms of spending power rather than pounds, but that would require deciding between RPI and CPI, and that is clearly for too difficult.

Anyway, enough cynicism. In truth, we should probably be grateful that the outcome hasn’t been worse. It certainly has been for some and I do feel that the current government has got its basic economic policies completely wrong. It seems like it’s time that someone explained to George Osborne that it’s not necessarily the size of the debt and deficit that matters. What matters is their size relative to the size of our economy. The way its going now it seems like he’s getting it wrong on both counts.

What is maybe more concerning, from a science funding perspective, is the possibility that the government may choose to axe the QR funding. This is the funding stream that comes from the Higher Education Funding Councils and how it is distributed is determined via the outcome of the Research Excellence Framework (REF) exercise. Now, it may well be part of the ring fence and it may well be safe, but I wouldn’t know whether to laugh or cry if it were cut. If you’ve read some of my earlier posts, you’ll know that I’ve been very critical of REF. This is both due to the manner in which it is implemented and due to the shenanigans taking place at UK universities; the potentially risking hiring, the morally/legally questionable redundancies, and the time and effort spent preparing for what is – in my opinion – a completely flawed exercise.

So, if it were to be cut, part of me would feel like saying “serves us right for taking something so silly so seriously, and for playing the kind of games that have not and will not benefit our fundamental role as teachers and researchers”. On the other hand, it is a lot of money (£1.5 billion I believe) and I certainly have no desire to see this money leave the Higher Education sector. As far as I can tell, some may struggle to survive as they are, even if there are no cuts to the QR funding. Well, I certainly hope that it isn’t cut but I also hope that in future, universities will find the backbone to tell the government that playing these kind of games is silly and that they should find a simpler and more effective mechanism for distributing this money (although I don’t think it should simply be given to the research councils, but that might be a topic for another post).

Declaration on Research Assessment

Just thought I would highlight the San Francisco Declaration on Research Assessment. It’s been a long day and I’m quite tired, so I don’t want to say too much. You can read most of this yourself, but basically a group of editors and publishers of scholarly journals met during the Annual Meeting of The American Society for Cell Biology (ASCB), and have come up with a set of recommendations about the use of journal-based metrics. The basic motivation was

  • the need to eliminate the use of journal-based metrics, such as Journal Impact Factors, in funding, appointment, and promotion considerations;

  • the need to assess research on its own merits rather than on the basis of the journal in which the research is published; and

  • the need to capitalize on the opportunities provided by online publication (such as relaxing unnecessary limits on the number of words, figures, and references in articles, and exploring new indicators of significance and impact).

  • A lot of it seemed to focus on the use of Journal Impact factors when assessing individual bits of research (which, as many have already pointed out, is horribly flawed) but there was indication that this was attempting to go further than just that. Citation metrics can be useful but can also be problematic. There can be a huge range of different practices even with the same basic area and so using them alone to assess an individual can disadvantage some potentially excellent researchers. It can also be advantageous to some who aren’t particularly good but who just happen to work in an area that is currently popular and in which papers are collecting citations easily.

    Anyway, this is an encouraging step and I hope it has some impact and that it’s taken seriously by funding agencies, interview panels and promotion boards. I suspect it’s too late to have much effect on the REF2014 panels, but maybe there’s hope for REF2021.

    REF and teaching

    There’s a recent article in the Guardian about the influence of the Research Excellence Framework (REF2014) on university teaching. The basic issue is that money will only be allocated on the basis of papers that score highly (3* and 4*) and that league table rankings will also be determined by these high-ranking papers. Therefore, there is an incentive for universities to only submit researchers who have enough (essentially 4) papers that will be judged to be 3* or 4*. The concern is therefore that those who do not qualify will be encouraged (forced) to focus primarily on teaching or (as in the case of Queen Mary, University of London) face redundancy.

    Many universities are making “clear pledges that not being entered to the REF in November will not damage an academic’s career“. There are others, however, where this is clearly already having an impact (Queen Mary, University of London, Kings College and Strathclyde are three that I’ve heard about). I personally think that it is potentially a real problem. There is a big difference between how research and teaching are evaluated at universities, with an individual’s contribution to the research ranking being much more obvious than an individual’s contribution to any teaching ranking. One concern is that it will create a hierarchy within universities with some able to focus more on research and others “encouraged” to focus primarily on teaching and administration. I don’t have an issue with different people contributing to an academic department in different ways. I just would rather it were dynamic and evolved in some “natural” way, rather than being forced upon us by an external assessment exercise.

    University leaders are trying, in general, to make it clear that research and teaching are both valued parts of an academic’s career. The problem is that they don’t get to decide if the staff regard them as being of similar value. It certainly seems that even students are concerned about the impact that REF might have on the motivation of staff who might be judged to be “unworthy” and hence encouraged into having a larger role in teaching. I certainly think that these concerns are justified, even if there isn’t any evidence that REF is having, in general, this kind of impact.

    There do seem to be two common views expressed by those who are more supportive of REF than maybe I am. One is that it is not unreasonable to expect academics to publish 4 good papers every 7 years. In general I agree with this, although there may be some exceptions. However, there is a difference between publishing 4 good papers and publishing 4 papers that will be judged (by a panel – many of whom may not be particular expert in your field) to be good. Maybe about one-quarter of my papers have done quite well (in terms of citations) but I don’t really have a good idea why they did well and why others didn’t. I can’t really look back and claim that I can now tell why some papers would be judged to be good, while others would not. I’m typically quite pleased with most of the papers I publish. Whether or not they do well (in metric terms) all seems a little random to me.

    The other claim that is often made is that REF has forced universities to take hiring more seriously and that hiring is now based on excellence. Firstly, this is presumably only “perceived excellence” in research. One of the perennial criticisms of university hiring has been that teaching ability hasn’t been taken seriously enough. I really can’t see that REF has helped here. My feeling is that it may have made the situation worse. The other issue I have with this claim is that it suggests that the typical academic today is somehow better (because of REF) than they were 20 or 30 years ago. Really? I thought universities in the UK have been world-class for decades. I’m sure many academics who were active in the 60s, 70s, 80s and 90s might be slightly insulted by this suggestion. I suspect there were issues with hiring practices in those days, but that was probably more to do with societal issues that have been remedied via equalities legislation, than via REF.

    It strikes me that there has been quite a lot of recent coverage about the negative aspects of REF, so maybe some of it will sink in. Not that hopeful though. Maybe I should be considering holding back some of my current work so as to publish papers that will qualify for REF2021.

    A new REF algorithm

    In a previous post (REF prediction) I looked up the h-indices and the citations per publication for all Physics and Astronomy departments included in RAE2008. I ranked them in terms of their h-index, in terms of their citations per publication, and as average of these two. It looked alright but I did comment that one could produce something more sophisticated. At the time I did worry that using just the h-index would disadvantage smaller departments, but I couldn’t really think of what else to do and it was just a very basic exercise.

    Deevy Bishop has, however, suggested an alternative way of ranking the departments. This is to basically relate the income they get with their h-index. For example, in RAE2008 each department was ranked according to what fraction of their papers were 4*, 3*, 2*, 1* and U. The amount of funding they received (although I think it technically went to the university, rather than to the department) was then scaled according to N(0.1×2* + 0.3×3* + 0.7×4*) where N was the number of FTEs submitted. This data can all be downloaded from the RAE2008 website. Deevy Bishop did an analysis for psychology and discovered that the level of funding from RAE2008 correlated extremely well the department’s h-index. What was slightly concerning was that the correlation was even stronger if one also included whether or not a department was represented on the RAE2008 panel.

    I’ve now done the same analysis for Physics and Astronomy. I’ve added various figures and text to my REF prediction post, but thought it worth making it more prominent by adding it to a new post. The figure showing RAE2008 funding plotted against h-index is below. According to my quick calculation, the correlation is 0.9. I haven’t considered how this changes if you include whether or not a department was represented on the RAE2008 panel. The funding formula for REF2014 might possibly be N(0.1×3* + 0.9×4*). I’ve redone the figure below to see what the impact would have been if this formula had been used instead of the RAE2008 formula. It’s very similar and – if you’re interested – it’s included at the bottom of my REF prediction post. It does seem that if all we want to know is how to distribute the money, relating it to a department’s h-index seems to work quite well (or at least it would have worked well if used for RAE2008). I’m not quite sure how easy it would be to produce an actual league table though. Given that the REF2014 formula may depend almost entirely on the fraction of 4*, one could simply divide the h-index by the number of FTEs to get a league table ranking, but I haven’t had a chance to see if this produces anything reasonable or not. Of course, noone really trusts league tables anyway, so it may be a good thing if we don’t bother producing one.

    A plot of h-index against the RAE2008 funding formula - N(0.1x2* + 0.3*3* + 0.7*4*).

    A plot of h-index against the RAE2008 funding formula – N(0.1×2* + 0.3×3* + 0.7×4*).

    REF again!

    Some interesting posts recently about the forthcoming Research Excellence Framework (REF2014). An interesting post by Dave Fernig called In Defence of REF. This post does make some valid points. REF, and previous RAEs, may well have encouraged more sensible hiring practices in which the quality of the applicant is taken more seriously than maybe it was in the distant past. Two comments I would make are that it I still think that teaching ability is still not taken seriously enough and, in my field at least, many places have adopted a very risky hiring strategy that – I hope – doesn’t come back to bite us in 5 years time. Dave Fernig also seems to feel that the panel, in his field, can distinguish between excellent, good and mediocre papers. This may well be true in his field, but I don’t think it is for my field (physics).

    Peter Coles, who writes the Telescoper blog, has written a new post called Counting for the REF. I won’t say much about this post as you can read it for yourself, but I agree with much of what is said. Maybe the most concerning comment in the post was the suggestion that the weighting – when determing the funding distribution – would be 9 for 4* papers and 1 for 3* papers. Essentially, most of the funding would be determined by 4* paper and a very small amount would be associated with 3* papers. Fundamentally I think this is unfortunate as it gives very little credit to some very good papers and absolutely no credit to what might be quite good papers (there is no funding associated with 2*).

    There is a more fundamental concern that is associated with what is discussed in Peter Coles’s post. In a recent post (Some more REF thoughts) I pointed out that in Physics fewer than 10% of all papers get more than 10 citations per year. The claim is that two members of the REF panel will read and assess each paper. However, as pointed out by others, this would require each panel member to read 2 papers per day for a year. Consequently, it is impossible for them to give these papers as much scrutiny as they would be given if they were being properly peer-reviewed. There is an expectation that metrics (citations for example) will play an important role in deciding how to rate the papers. How could you do this? You could set a threshold and say, for example, that since most papers get fewer than 10 citations a year that 4* papers will be those that receive more than 10 citations a year. The problem that I have (ignoring that citations are not necessarily a good indicator of quality) is that this would then be a very small fraction (about 5%) of all published papers. The distribution of REF funding would then be being determined by a minority of the work published since 2008. This means that small variations can have a big impact on how the money is distributed. One could imagine that just a few papers being judged 3* instead of 4* could have a massive impact on how much money a department gets (I accept that the money doesn’t actually go to the department, but you probably know what I mean).

    Alternatively, if you want to avoid small variations having a big impact you would need 4* papers to make up a reasonable fraction of the assessed papers (maybe 10 – 20%). The problem here is that you’re now getting down to papers that are only collecting a few (5-10) citations per year, so where do you draw the boundary. Is 3 per year too few, but 5 a year okay. You could argue that these are just being used to to guide the assessment and that the panels’ reading of the paper will allow a distinction to be drawn between 4* and 3* papers. This doesn’t, however, change the fact that the panel members have to read a massive number of papers. It feels more like combining two completely flawed processes and hoping that what pops out the other side is okay.

    I suggested in an earlier post (REF prediction) that, given the diverse nature of a typical academic department or university, that this might be an appropriate time to simply consider using some kind of metric. I did a quick analysis of all 42 physics departments’s h-indices and saw a reasonable correlation between their h-index and how they did in RAE2008. I noticed today that Deevy Bishop, who writes a blog called BishopBlog, has made a similar suggestion and carried out the same kind of analysis for psychology. Her analysis seems quite similar to mine and suggested that this would be An alternative to REF2014.

    Anyway, it’s quite good to see others writing about REF2104 (whether for or against). I think it is a very important issue and I’m just disappointed that it is probably too late to make any changes that would make the REF2014 process simpler and more likely to produce a reasonable ranking.

    I wanted to reblog this partly because it’s good – in my view – to see more people writing about REF. I also think the post makes some interesting points. I think REF is horribly flawed, but maybe we have to also realise that self-regulation also has its problems. I would suggest that self-regulation might be the wrong term to use. Not all of our research money comes through REF. A big fraction comes through research grants that are competitive and are assessed by a reasonably knowledgeable panel (although I would suggest that the allocation of research council grants in the UK has its own problems). REF is essentially allocating QR money that goes – in the first instance – to the university, not to individuals. The university then decides how to divide this money up. It’s not clear to me that a simpler assessment exercise that used up less time, was less easy to game, and in which maybe the allocation was less non-linear (i.e., a weaker dependence on how well you do and how much money you get per person submitted) would be just as effective and less damaging. Part of me thinks that maybe the criteria should be secret until the process is finished, but I suspect many would feel that they then wouldn’t trust it at all and everyone would complain afterwards if they didn’t do as well as they thought they should have done.

    Corticalia

    When I was a medical student we were encouraged to conduct vaginal examinations on anaesthetized gynaecological patients, so that we could learn how to examine the reproductive system in a relaxed setting. The women did not know this was going to happen to them during their surgery, and did not sign any consent forms. Probably they would not have minded anyway (they were asleep after all, and educating the next generation of doctors is undoubtedly a good cause) but the possibility that they should be asked if they did mind did not occur to anybody, until my ultra-feminist friend kicked up a stink and organised a rebellion. The surgeons were genuinely surprised and mystified. Being well-meaning, it had not occurred to them that some people might see what they were doing as wrong.

    Fast-forward a few years to the Alder Hey scandal, in which it emerged that doctors at a children’s hospital had…

    View original post 584 more words