Kafkarna continues: REF gloves off at Lancaster University

I thought I would reblog this. Partly because I haven’t had a chance to write much recently and it gives me an opportunity to keep things ticking over, partly because it seems like something worth highlighting (although given my readership, this may not help much), and partly because I’ve written about the REF (and been quite critical) and this type of activity is one of the things that I expected to happen and illustrates the issues – in my opinion – with this type of assessment process. I particularly like, and agree with, this comment in the post Whatever this charade is, it is not a framework for research excellence and so I recommend giving this a good read.

coastsofbohemia

Two days ago an open letter from Professor Paolo Palladino, former Head of the History Department at Lancaster University, appeared on the university’s internal History Staff list serve, with copies sent to senior administrators in the Faculty of Arts and Social Sciences (FASS) and Central Administration.  I responded the next day with a post supporting Professor Palladino’s position, sent to the same recipients. With Paolo’s permission, I am reproducing both of our letters here.  We both believe that the issues raised by Lancaster’s selective culling of research-active staff from submission in the 2014 REF–a practice in which it is not alone among British universities–deserve the widest possible public debate.  We would therefore urge anyone who shares our concerns to share this post through social media.

Professor Palladino’s letter:
Dear all,
Over the next few days, a number of colleagues across the university are to be informed that they will not…

View original post 1,060 more words

REF, QR funding and the science budget

I guess we should all be reasonably pleased that the science budget has remained ring-fenced. The reality, of course, is that this just means that it is ring-fenced in a flat-cash sense, not in an inflation adjusted sense. As scientists, maybe we should make sure they define the term more specifically in future. It does appear that the current definition would allow the government to claim that something has been ring-fenced, despite the spending power tending to zero. Logically, you mighty expect it to be defined in terms of spending power rather than pounds, but that would require deciding between RPI and CPI, and that is clearly for too difficult.

Anyway, enough cynicism. In truth, we should probably be grateful that the outcome hasn’t been worse. It certainly has been for some and I do feel that the current government has got its basic economic policies completely wrong. It seems like it’s time that someone explained to George Osborne that it’s not necessarily the size of the debt and deficit that matters. What matters is their size relative to the size of our economy. The way its going now it seems like he’s getting it wrong on both counts.

What is maybe more concerning, from a science funding perspective, is the possibility that the government may choose to axe the QR funding. This is the funding stream that comes from the Higher Education Funding Councils and how it is distributed is determined via the outcome of the Research Excellence Framework (REF) exercise. Now, it may well be part of the ring fence and it may well be safe, but I wouldn’t know whether to laugh or cry if it were cut. If you’ve read some of my earlier posts, you’ll know that I’ve been very critical of REF. This is both due to the manner in which it is implemented and due to the shenanigans taking place at UK universities; the potentially risking hiring, the morally/legally questionable redundancies, and the time and effort spent preparing for what is – in my opinion – a completely flawed exercise.

So, if it were to be cut, part of me would feel like saying “serves us right for taking something so silly so seriously, and for playing the kind of games that have not and will not benefit our fundamental role as teachers and researchers”. On the other hand, it is a lot of money (£1.5 billion I believe) and I certainly have no desire to see this money leave the Higher Education sector. As far as I can tell, some may struggle to survive as they are, even if there are no cuts to the QR funding. Well, I certainly hope that it isn’t cut but I also hope that in future, universities will find the backbone to tell the government that playing these kind of games is silly and that they should find a simpler and more effective mechanism for distributing this money (although I don’t think it should simply be given to the research councils, but that might be a topic for another post).

I wanted to reblog this partly because it’s good – in my view – to see more people writing about REF. I also think the post makes some interesting points. I think REF is horribly flawed, but maybe we have to also realise that self-regulation also has its problems. I would suggest that self-regulation might be the wrong term to use. Not all of our research money comes through REF. A big fraction comes through research grants that are competitive and are assessed by a reasonably knowledgeable panel (although I would suggest that the allocation of research council grants in the UK has its own problems). REF is essentially allocating QR money that goes – in the first instance – to the university, not to individuals. The university then decides how to divide this money up. It’s not clear to me that a simpler assessment exercise that used up less time, was less easy to game, and in which maybe the allocation was less non-linear (i.e., a weaker dependence on how well you do and how much money you get per person submitted) would be just as effective and less damaging. Part of me thinks that maybe the criteria should be secret until the process is finished, but I suspect many would feel that they then wouldn’t trust it at all and everyone would complain afterwards if they didn’t do as well as they thought they should have done.

Corticalia

When I was a medical student we were encouraged to conduct vaginal examinations on anaesthetized gynaecological patients, so that we could learn how to examine the reproductive system in a relaxed setting. The women did not know this was going to happen to them during their surgery, and did not sign any consent forms. Probably they would not have minded anyway (they were asleep after all, and educating the next generation of doctors is undoubtedly a good cause) but the possibility that they should be asked if they did mind did not occur to anybody, until my ultra-feminist friend kicked up a stink and organised a rebellion. The surgeons were genuinely surprised and mystified. Being well-meaning, it had not occurred to them that some people might see what they were doing as wrong.

Fast-forward a few years to the Alder Hey scandal, in which it emerged that doctors at a children’s hospital had…

View original post 584 more words

Some more REF thoughts

The post about my REF interview seems to have generated a modest amount of interest in the last day or so. There were no comments, so I can’t tell if others identified with my experience and agreed with my general views, or disagreed and thought it was all a load of nonsense. However, seeing that post generate a little interest reminded me that I had seen some interesting data recently about REF outputs. For those that don’t know, REF is the Research Excellence Framework and is an exercise in which the quality of research in UK universities will be judged and the results will determine how to divide up a fairly substantial pot of money. What makes it more “interesting” is that the formula that decides how much money each university gets is highly non-linear. There is a big difference between doing “very well” compared to simply doing “well”.

What will be assessed will in general be papers published by academics in each institution. Typically, there will be 4 papers – published since 2008 – for each academic included in the submission. The intention is that each paper will be judged in terms of its originality, significance and rigour and will be given a score of either 4*, 3*, 2*, or 1*. The claim is that the panels doing the judging will not be using Journal Impact Factors or citations to make their assessment. It has, however, already been pointed out that this claim is unlikely to be credible. In Physics, there will probably be something like 6500 papers each of which will supposedly be read by 2 of the 20 panel members in a period of about 12 months. In other words, at least 2 papers per day each. Pretty difficult to do. Virtually impossible to make a credible judgement of each paper. The general view is that, despite what is claimed, Journal Impact Factors and citations will indeed be used to judge these papers.

Here’s what I found interesting. According to what I saw recently, a paper published since 2008 that is receiving about 8 citations a year will be in the top 10% according to citation numbers. I was a little surprised. I assumed that the top 10% of papers (according to citation numbers) would be receiving more than 8 or so citations a year. I decided to look into this myself using Web of Knowledge. If you search for all refereed articles published in the general area of Physics that also have “UK”, “United Kingdom”, “England”, or “Scotland” in the address you discover 38176 refereed articles published since January 2008. Web of Knowledge can’t do citation statistics on more than 10000 papers. I divided these papers into 5 categories (Condensed Matter, Astronomy & Astrophyscs, Particle Physics, Nuclear Physics, Mathematical Physics). I also included a randomly chosen sample of areas in Physics that, together, hadn’t published more than 10000 papers since 2008. The table below shows the average number of citations per paper, the number to be in the top 1%, the number to be in the top 10%, and the median.

PhysicsFields

Indeed, it seems that the average number of citations per Physics paper published since 2008 is about 10 and to be in the top 10% of all physics papers published since 2008 you need to be collecting fewer than 10 citations per year. Although it is different for different areas of physics, the difference isn’t particularly large. One issue with the above table is that the older papers will have collected more citations than the newer papers. I then repeated the above, but considered only papers published – with a UK author – in 2008, 2009, 2010, 2011, and 2012. In this case there are typically between 7000 and 7500 refereed articles published per year, so I didn’t divide it into different disciplines, but considered all articles in physics. The table below shows the result.

PhysicsYears

The result seems about the same. To be in the top 10% of papers published in any year since 2008 a paper needs fewer than 10 citations per year. Essentially, for a paper to be in the top 10% of cited papers, requires a fairly small number of citations per year. Alternatively, most papers seem to receive very few citations. What to make of this? Partly, I was just a little surprised. If asked, I would have guessed that to be in the top 10% of cited papers would require more than 10 citations per year. Also, what does the fact that a large fraction of refereed articles attract very few citations per year imply? Does it mean that much of what we publish isn’t particularly interesting. Although I think we probably publish too many papers, I don’t think that 90% of what we publish is worthless. Quite a large number of those papers receiving very few citations must be excellent bits of research that are worth publishing. Maybe they just haven’t been noticed. Maybe they’re what is referred to as slow-burners. Maybe it was a necessary step that has been superseded by a newer bit of research but that isn’t getting the citations that it might deserve. Maybe it’s something a researcher enjoyed doing, learned a lot by doing and that then allowed them to move on to something newer and more interesting.

What’s more interesting is how citations can then be used to judge these papers. We will presumably be submitting something like 6500 papers, so potentially 17% or so of all refereed physics papers published since 2008. Only papers judged to be 3* or 4* will attract money. One could assume that 3* and 4* papers will be those with much higher than average number of citations. This would then imply that a small number of papers will be used to determine how to divide up the large sum of money associated with REF. Small variations could then have a big effect. On the other hand, if 3* papers are not necessarily those with much more than the average number of citations, how do you then distinguish between 3* and 2* papers. Most papers are collecting fewer than 10 citations per year. Where’s the division? Is 3 a year 2* and 6 a year 3*? Alternatively, we shouldn’t really use citations and metrics and should judge each paper on it’s originality, significance and rigour (as suggested in the REF documentation). The problem is that very few, if any, believes that it is possible for a panel of 20, however distinguished, to do this.

The truth is probably that it will be a combination. The panel members will, I’m sure, try to read the papers and will then use metrics to fine tune their scores. However, combining two largely flawed processes to try and determine the quality of research activity in UK universities doesn’t really seem like much of an improvement. I suspect that, at the end of the day, a ranking will be produced that isn’t entirely unreasonable. However, as I’ve pointed out before, it should be possible to achieve a reasonable ranking in a manner that doesn’t use up quite as much time and effort as REF is currently doing.

Happy New Year

This will be my last post of 2012. It’s been an interesting year. I’ve written more posts than I expected to, and my readership has climbed a little (although is still not particularly high). A couple of the posts that have been popular (at least in terms of readership) have been REF2014: Good or bad? and More REF madness. This has been a bit of a theme for me this year. I’m quite concerned about the impact REF is having on UK universities and am rather disillusioned at the moment. I’m hoping the new year will reduce my cynicism and that I can start enjoying my career again.

Nothing else has had a particularly high readership. I’ve written more about income inequality and a post I wrote in 2009 about the The Gini coefficient continues to attract readers. I’ve also commented on the possible effective privatisation of the NHS and am certainly very concerned about the direction in which the coalition government appears to be taking the country. It still amazes me that the financial crisis – that started in 2008 – seemed to be a strong indicator that neo-liberal policies have real problems and yet the government appears to be continuing with these kind of ideas. I’m certainly not proposing that we should implement full socialism, but hoping that the free market will save us seems incredibly short-sighted. Some may well benefit, but most will almost certainly not. We live in a social democracy and pretending that the private sector can solve all problems, and that welfare benefits are largely given to those who are “scroungers”, is incredibly naive and simplistic.

Anyway, that’s all I was going to say. I hope everyone has a good new year and that 2013 is better for most than I suspect 2012 has been.

REF2014 Follow-up

Well, telescoper reblogging my post about REF2014 being good or bad has increased my readership by an order of magnitude or so for the last couple of days. The poll, in the post, about whether or not people thought the impact of REF2014 on university behaviour was positive, negative, or neutral has also been taken by considerably more people than was the case a few days ago.

Of those who took the poll – at this stage at least – a large majority (82%) were against REF2014 and thought it was having a damaging impact on university behaviour. Less than 10% thought it was having no impact or was having a positive impact. This largely confirmed my view that very few thought positively of REF2014. I need to be a little careful as clearly the poll was not a definitive poll on REF2014 and, as one commentator mentioned, there may be some positive aspects to REF2014. If it encourages us to publish fewer, better papers that would be a good thing.

On the other hand, it seems clear that REF2014 (which is meant to essentially be an assessment exercise) is having an impact on how universities behave. It does seem odd to have an assessment exercise that influences university behaviour and we should be very careful that we don’t end up with a university system that is optimised for performing well in REF exercises, rather than having an assessment system that is optimised for identifying universities that excel at research and scholarship.

So, what’s to be done. Probably nothing. It’s almost certainly too late to influence REF2014 and much of the negative aspects have already taken place (the hiring and firing for example). I did think that maybe negative publicity about REF2014 might change things in the future, but someone mentioned recently that there was a lot of negative publicity about RAE2001 and RAE2008 and that didn’t have much impact. Anyway, that’s all I have to say for now. If anyone would like to make a comment, please feel free to do so. Would be quite interested in the views of those who think REF2014 is having no impact or having a positive impact.

A REF conundrum

For those who have read any of my previous posts, you’ll know that I’m not a fan on the Research Excellence Framework (REF2014). I’ve blogged about the negative impact of REF before. Essentially, although it may aiming to do something quite reasonable, the way in which it is aiming to do this, and the impact it is having on the way universities are behaving, seem very negative to me. I did, however, think of something particular that I thought I would blog about here.

Basically, each university department in the UK will submit – to be assessed by the relevant REF panel – 4 refereed journal papers from all, or some, of their academics and research fellows. Each paper will be scored as either 1*, 2*, 3*, or 4*. The amount of money that the university will then get will depend on the average score and the number of people submitted. It’s still not quite clear if it’s better to submit fewer people, so as to get a higher score, or to simply submit as many eligible people as possible. However, I believe that someone cannot be submitted if they don’t have 4 refereed journal papers published between January 2008 and October 2013.

Here’s where I thought there could be a possible issue. Consider the situation in which there is someone in a university department who is the primary author on 4 refereed journal papers that are probably okay. They will probably score 3*. Imagine there is a second person in the same department who is the lead author on only 3 papers, but they’re fantastic papers and will probably score 4*. This second person, I think cannot be submitted to REF. However, if they happen to also be an author on one or more of the first person’s papers, one of these papers could be transferred to the second person who now has 4 papers (one scoring 3* and the others potentially scoring 4*). If this paper has 10 or fewer authors, the second person’s contribution does not even need to be justified. If it has more than 10, there would need to be some narrative explaining the second person’s contribution to the paper. The first person can now no longer, however, be submitted to REF.

In some sense, the fact that the first person can no longer be submitted to REF doesn’t matter. Individuals aren’t actually assessed. It’s simply that a subset of their papers are used to assess the research quality of a university department. However, an individual must be associated with each set of 4 papers. It’s in the department’s interest to submit the strongest set of papers. The first person is, however, someone who was not formally submitted and so this could disadvantage them (in a career sense) if people at their university don’t realise why. Also, if such a scenario were to occur, should the first person (and second I guess) approve the strategy. Is it acceptable for a university department to simply decide who should take credit for a particular paper? What if the first person objected and insisted that their 4 papers (which are good but not fantastic) be submitted to REF and refused to allow the department to transfer one of their papers to someone else?

I’m not sure if the above scenario is at all likely. I do think, however, that there will be situations (where more than one person in a department is an author on a paper) in which a decision will have to be made as to who should be credited with a particular paper and that it may well go to the person who played the less significant role. Given that individuals are not actually being assessed, it is logical that the optimal set of papers be submitted. However, it is an interesting issue as to whether or not it is acceptable for a university department to decide on who gets credit for a paper. Given that someone objecting to this strategy would disadvantage their department, I suspect that most will be largely happy with this. It does, however, seem to be something that could create some difficulties.