REF and teaching

There’s a recent article in the Guardian about the influence of the Research Excellence Framework (REF2014) on university teaching. The basic issue is that money will only be allocated on the basis of papers that score highly (3* and 4*) and that league table rankings will also be determined by these high-ranking papers. Therefore, there is an incentive for universities to only submit researchers who have enough (essentially 4) papers that will be judged to be 3* or 4*. The concern is therefore that those who do not qualify will be encouraged (forced) to focus primarily on teaching or (as in the case of Queen Mary, University of London) face redundancy.

Many universities are making “clear pledges that not being entered to the REF in November will not damage an academic’s career“. There are others, however, where this is clearly already having an impact (Queen Mary, University of London, Kings College and Strathclyde are three that I’ve heard about). I personally think that it is potentially a real problem. There is a big difference between how research and teaching are evaluated at universities, with an individual’s contribution to the research ranking being much more obvious than an individual’s contribution to any teaching ranking. One concern is that it will create a hierarchy within universities with some able to focus more on research and others “encouraged” to focus primarily on teaching and administration. I don’t have an issue with different people contributing to an academic department in different ways. I just would rather it were dynamic and evolved in some “natural” way, rather than being forced upon us by an external assessment exercise.

University leaders are trying, in general, to make it clear that research and teaching are both valued parts of an academic’s career. The problem is that they don’t get to decide if the staff regard them as being of similar value. It certainly seems that even students are concerned about the impact that REF might have on the motivation of staff who might be judged to be “unworthy” and hence encouraged into having a larger role in teaching. I certainly think that these concerns are justified, even if there isn’t any evidence that REF is having, in general, this kind of impact.

There do seem to be two common views expressed by those who are more supportive of REF than maybe I am. One is that it is not unreasonable to expect academics to publish 4 good papers every 7 years. In general I agree with this, although there may be some exceptions. However, there is a difference between publishing 4 good papers and publishing 4 papers that will be judged (by a panel – many of whom may not be particular expert in your field) to be good. Maybe about one-quarter of my papers have done quite well (in terms of citations) but I don’t really have a good idea why they did well and why others didn’t. I can’t really look back and claim that I can now tell why some papers would be judged to be good, while others would not. I’m typically quite pleased with most of the papers I publish. Whether or not they do well (in metric terms) all seems a little random to me.

The other claim that is often made is that REF has forced universities to take hiring more seriously and that hiring is now based on excellence. Firstly, this is presumably only “perceived excellence” in research. One of the perennial criticisms of university hiring has been that teaching ability hasn’t been taken seriously enough. I really can’t see that REF has helped here. My feeling is that it may have made the situation worse. The other issue I have with this claim is that it suggests that the typical academic today is somehow better (because of REF) than they were 20 or 30 years ago. Really? I thought universities in the UK have been world-class for decades. I’m sure many academics who were active in the 60s, 70s, 80s and 90s might be slightly insulted by this suggestion. I suspect there were issues with hiring practices in those days, but that was probably more to do with societal issues that have been remedied via equalities legislation, than via REF.

It strikes me that there has been quite a lot of recent coverage about the negative aspects of REF, so maybe some of it will sink in. Not that hopeful though. Maybe I should be considering holding back some of my current work so as to publish papers that will qualify for REF2021.

Advertisements

Some more REF thoughts

The post about my REF interview seems to have generated a modest amount of interest in the last day or so. There were no comments, so I can’t tell if others identified with my experience and agreed with my general views, or disagreed and thought it was all a load of nonsense. However, seeing that post generate a little interest reminded me that I had seen some interesting data recently about REF outputs. For those that don’t know, REF is the Research Excellence Framework and is an exercise in which the quality of research in UK universities will be judged and the results will determine how to divide up a fairly substantial pot of money. What makes it more “interesting” is that the formula that decides how much money each university gets is highly non-linear. There is a big difference between doing “very well” compared to simply doing “well”.

What will be assessed will in general be papers published by academics in each institution. Typically, there will be 4 papers – published since 2008 – for each academic included in the submission. The intention is that each paper will be judged in terms of its originality, significance and rigour and will be given a score of either 4*, 3*, 2*, or 1*. The claim is that the panels doing the judging will not be using Journal Impact Factors or citations to make their assessment. It has, however, already been pointed out that this claim is unlikely to be credible. In Physics, there will probably be something like 6500 papers each of which will supposedly be read by 2 of the 20 panel members in a period of about 12 months. In other words, at least 2 papers per day each. Pretty difficult to do. Virtually impossible to make a credible judgement of each paper. The general view is that, despite what is claimed, Journal Impact Factors and citations will indeed be used to judge these papers.

Here’s what I found interesting. According to what I saw recently, a paper published since 2008 that is receiving about 8 citations a year will be in the top 10% according to citation numbers. I was a little surprised. I assumed that the top 10% of papers (according to citation numbers) would be receiving more than 8 or so citations a year. I decided to look into this myself using Web of Knowledge. If you search for all refereed articles published in the general area of Physics that also have “UK”, “United Kingdom”, “England”, or “Scotland” in the address you discover 38176 refereed articles published since January 2008. Web of Knowledge can’t do citation statistics on more than 10000 papers. I divided these papers into 5 categories (Condensed Matter, Astronomy & Astrophyscs, Particle Physics, Nuclear Physics, Mathematical Physics). I also included a randomly chosen sample of areas in Physics that, together, hadn’t published more than 10000 papers since 2008. The table below shows the average number of citations per paper, the number to be in the top 1%, the number to be in the top 10%, and the median.

PhysicsFields

Indeed, it seems that the average number of citations per Physics paper published since 2008 is about 10 and to be in the top 10% of all physics papers published since 2008 you need to be collecting fewer than 10 citations per year. Although it is different for different areas of physics, the difference isn’t particularly large. One issue with the above table is that the older papers will have collected more citations than the newer papers. I then repeated the above, but considered only papers published – with a UK author – in 2008, 2009, 2010, 2011, and 2012. In this case there are typically between 7000 and 7500 refereed articles published per year, so I didn’t divide it into different disciplines, but considered all articles in physics. The table below shows the result.

PhysicsYears

The result seems about the same. To be in the top 10% of papers published in any year since 2008 a paper needs fewer than 10 citations per year. Essentially, for a paper to be in the top 10% of cited papers, requires a fairly small number of citations per year. Alternatively, most papers seem to receive very few citations. What to make of this? Partly, I was just a little surprised. If asked, I would have guessed that to be in the top 10% of cited papers would require more than 10 citations per year. Also, what does the fact that a large fraction of refereed articles attract very few citations per year imply? Does it mean that much of what we publish isn’t particularly interesting. Although I think we probably publish too many papers, I don’t think that 90% of what we publish is worthless. Quite a large number of those papers receiving very few citations must be excellent bits of research that are worth publishing. Maybe they just haven’t been noticed. Maybe they’re what is referred to as slow-burners. Maybe it was a necessary step that has been superseded by a newer bit of research but that isn’t getting the citations that it might deserve. Maybe it’s something a researcher enjoyed doing, learned a lot by doing and that then allowed them to move on to something newer and more interesting.

What’s more interesting is how citations can then be used to judge these papers. We will presumably be submitting something like 6500 papers, so potentially 17% or so of all refereed physics papers published since 2008. Only papers judged to be 3* or 4* will attract money. One could assume that 3* and 4* papers will be those with much higher than average number of citations. This would then imply that a small number of papers will be used to determine how to divide up the large sum of money associated with REF. Small variations could then have a big effect. On the other hand, if 3* papers are not necessarily those with much more than the average number of citations, how do you then distinguish between 3* and 2* papers. Most papers are collecting fewer than 10 citations per year. Where’s the division? Is 3 a year 2* and 6 a year 3*? Alternatively, we shouldn’t really use citations and metrics and should judge each paper on it’s originality, significance and rigour (as suggested in the REF documentation). The problem is that very few, if any, believes that it is possible for a panel of 20, however distinguished, to do this.

The truth is probably that it will be a combination. The panel members will, I’m sure, try to read the papers and will then use metrics to fine tune their scores. However, combining two largely flawed processes to try and determine the quality of research activity in UK universities doesn’t really seem like much of an improvement. I suspect that, at the end of the day, a ranking will be produced that isn’t entirely unreasonable. However, as I’ve pointed out before, it should be possible to achieve a reasonable ranking in a manner that doesn’t use up quite as much time and effort as REF is currently doing.

REF Interview

So I recently had an interview with our Head of Department and our Director of Research to discuss my inclusion in the Research Excellence Framework (REF2014). Although I had expected some problems, my interview went fine and it looks as though I’m “good enough” or, more correctly, I have my name on 4 papers that are “good enough”. Some of my colleagues were not particularly happy after their interviews, but I don’t feel all that comfortable discussing their issues in any detail. As you may know, if you’ve read any of my other posts, I’m not a huge fan of REF. Even though my interview went fine, my views haven’t changed much and – if anything – it’s rather confirmed my general views of the issues with the process.

I don’t really want to go into specifics as maybe I shouldn’t be giving away our strategy. I thought, instead, that I would give some general comments. It’s clear that it’s a game, and everyone knows this. There are two parameters; money and league table ranking. Most institutions will be adjusting their submission to optimise between these two parameters. This is not an objective assessment of research quality. It’s a game to try and submit the optimum set of papers that will get you as high as possible on the League table and get as you much money as possible. Some institutions may determine that it will be better to sacrifice league table position for money and others may choose to prioritise league table position over money.

It was claimed that submissions (papers) would be read and assessed and that metrics would not be used. Noone believes this to be true. We’re clearly using Impact Factors and citations at some level. Papers in Nature or Science will be 4* simply because they’re in Nature or Science. Papers in other good Journals will be 4*, 3* or 2* depending on the number of citations. There’s also no rigorous assessment of ownership. As long as you can indicate that you contributed sufficiently to a paper, your institution will get full credit. Someone who makes modest contributions to 4 papers that are regarded as excellent, is more valuable to an institution than another person who writes 4 good papers, none of which are regarded as excellent.

You could argue that the 4 excellent papers are better than the 4 good papers, but there is a chance that our careers may depend on how our 4 papers are judged for REF. Is it really better to have average people who are good at getting their names on excellent papers, rather than good people who write their own. I may be slightly biased in that I’m probably more in the latter category than the former, but I would say that I don’t intend for my papers to only be “good”. I always aim to write papers that I think are tackling interesting and challenging problems and I always aim to be as careful and rigorous as I can reasonably be. I always try to write something that would be regarded by others as excellent. The citations, however, don’t always indicate that they have been received particularly well. Of course, the definition of excellence is very tricky and using simplistic metrics is not necessarily a good way to determine the quality of a piece of research. In discussing this, someone commented that it is not unreasonable to expect academics to write 4 good papers every 7 years. In some sense I agree, and I would certainly back myself to write 4 good (maybe even some excellent) papers every 7 years. What I have much less confidence in is the ability of those who are judging these (either on the REF panel or in my own Department) to do so properly.

It seems that there is a real chance that we will aim to adapt our research strategy to suit future REF exercises. It’s clear that independence and originality is no longer necessarily optimal. It might be much better to have people who work in collaborations that are likely to publish papers that will be highly cited. I could easily see people being encouraged to make sure that they belong to such a collaborations and that they aim to contribute sufficiently to at least 4 papers so that their institution can submit these 4 papers to REF. Maybe this won’t happen, but I find the possibility of this happening quite disturbing. What I find slightly more disturbing is that the senior people that I encounter acknowledge that it is a game, but seem to feel that it is a game we need to play. Noone seems that bothered that we might be adjusting our research strategies to suit what is essentially an assessment exercise. I appreciate that there is a lot of money involved but, as far as I’m concerned, either our research is valuable at some fundamental level, or we really shouldn’t be bothering. Maybe that’s a little extreme, but hopefully you know what I mean.

That’s probably all I was going to say. As I may have mentioned before, a concern of mine is that over time we will adapt so that top UK institutions are very good at scoring well on REF exercises but don’t really do research that has any particular value. Maybe I’ll be proven wrong and that once REF is finished we’ll forget about it for a while and everyone will be left alone to get on with whatever research they think is most interesting and valuable. I somewhat doubt that and I do worry slightly about the careers of those who are deemed to not have enough papers that are good enough for REF. Will they be marked and will their jobs be at risk. I think that would be awful if it did start happening. I have no real issue with people who are not contributing positively to the running of an academic being sanctioned. I do, however, have an issue with the possibility that some people’s careers could depend largely on an assessment exercise that, in my opinion, is horribly flawed.

Unconscious bias

I attented, today, a very interesting presentation that was essentially about gender balance in academia, specifically in the sciences. There are a number of schemes, such as Athen SWAN and the Institute of Physics’s Project Juno, aimed at improving the gender balance in academia. These are schemes where a university department/school can qualify for a certain award if they satisfy certain criteria. There are different levels in each of the schemes and so a department can start with a basic award and build up to a higher award over time. Although the goal, in some sense, is to improve the gender balance in the physical sciences, ultimately it is meant to encourage a working environment and working practices that should benefit both men and women.

What I found interesting was a discussion about unconscious biases. Two researchers (Michelle Hebl and Randi Martin) from Rice University studied 624 reference letters for 194 applicants for junior faculty positions at US universities. They classified certain words as communal (social or emotive) and others as agentic (active or assertive). They found that communal words were used more often in reference letters for female applicants while agentic terms were more commonly used in letters for male applicants. What they then found is that communal terms were not valued by those assessing the applicants. The basic conclusion was that those writing reference letters unconsciously associate communal characteristics with women and agentic characteristics with men and that this then disadvantages women when their applications are assessed by those on hiring panels (whether those on the hiring panels were male of female).

The one interpretation that I have heard is that this means that we should be more careful, when writing reference letters, not to unconsciously use terms that have a gender bias. Of course, if we are unconsciously describing two equivalent people differently just because one is a women and one is a man, then I would agree. My personal view is slightly different. There is no reason why someone who is more communal (kind, sympathetic, tactful, agreeable) wouldn’t be an excellent academic. If these terms fairly describe someone, then it’s not clear that reference letter writers should describe them differently. It’s the job of those on hiring panels to not undervalue such characteristics.

It’s certainly my opinion that the reason that agentic characteristics (ambitious, aggressive, daring) are more valued than communal characteristics is not because they are better characteristics for an academic, it is simply because these have been – and still are – the characteristics of the typical academic (in the sciences at least). I would much rather see us recognise that the ideal academic department is made up of people with a wide range of different characteristics (communal and agentic) rather than suggesting that those who want to become academics should become more agentic.

Fellowships and Buy-Outs

I’ve always been a little worried about writing about Fellowships and buy-outs, partly because I don’t really have strong views about them (or rather I’m not really sure what I think) and partly because it may seem like sour grapes (which may indeed be somewhat true). This may end up seeming a little incoherent, but here we go anyway. When I first started working at a UK university there were postdoctoral-like Fellowships for researchers who didn’t have permanent jobs and a few Senior Fellowships that would allow academic staff to buy themselves out of teaching for a few years. Even back then I was concerned that Fellowships played too big a role in hiring practices. If two people were competing for a job, one with a Fellowship and one without, the one with the Fellowship might get the job because the University wouldn’t have to pay them directly for a few years. In fairness, having a Fellowship could also indicate research leadership and independence and so one might expect Fellowship holders to be more competitive in the job market.

Today, however, there seem to be more ways that academic staff can effectively pay themselves to focus mainly on research. There are European Research Council (ERC) Starting and Advanced grants that pay for a group of researchers plus the salary of the PI. I believe EPSRC has various Fellowships for academic staff and most UK Research grants now pay Full Economic Costing (fEC) which includes a portion of the PI and Co-I’s salaries. If a strong enough case is made, it could pay a significant fraction of someone’s salary. It seems – to me at least – that today a bigger fraction of academic staff are effectively bought out of teaching, than was the case 10 years ago.

Many people seem to think that this is all a good thing, but I’m not so sure. I would (and have done in the past) apply for such Fellowships and wouldn’t turn down the chance to buy myself out of some teaching and admin, but I’m still not convinced that it’s all good. Firstly, I took an academic job expecting to teach, do some administration and do research. It now seems, to a certain extent at least, that research is valued above all else and hence those who buy themselves out of teaching and admin are somehow “better” than those who don’t have buy-outs. I should add, however, that most of the people I know who have buy-outs do contribute something to admin and teaching, so my issue isn’t with the individuals, who often do more than is required. My issue is with the system that seems to value these buy-outs more highly than it values the contribution others do through teaching and administration. In some sense it seems odd to me that we have a system in which we’re hired to do a certain job (teaching, administration and research) and are then encouraged to find a way to avoid doing a big chunk of what you were hired to do.

There are other issues too. It feels a little like it can distort the system. Someone who has been bought out for a number of years will typically look stronger when it comes to promotion than someone who hasn’t. It seems obvious that the system should take into account how much teaching, administration and research someone has done when determining their contribution to the running of an academic department. It’s not obvious – to me at least – that it does. The perception is that research plays such a big role that those who’ve been bought out, and hence have been able to focus primarily on research, are more highly valued than those who haven’t.

Also, what do you do if a reasonable fraction of the academic staff are bought out? Typically, they do some teaching and/or have an administrative role, so it’s not quite as bad as it could be, but there will still be some requirement to take over the jobs that they would have done had they not been bought out. Either others have to pick up the slack or new staff need to be hired. This can introduce a level of risk. If you hire new staff you have to hope that you can maintain a reasonably constant buy-out rate. If not, you could suddenly have an increased salary cost when staff come off their buy-outs. As it is, the level of salary fEC on UK research grants has been dropping and there is currently a petition to secure the EU research budget and, presumably, protect the ERC budget. This presumably means that the number of available Fellowships may drop in the future. There therefore seems to be a reasonable chance that we won’t be able to maintain the current buy-out rate and, hence, there may well be people whose jobs will be at risk in the future. Who are these likely to be? I would guess, not those who have been bought out in the past.

In general, I don’t think it’s all bad. The idea that an academic could spend a few years focusing on research does seem good and I, personally, would be quite keen to do that. I think my main issue is whether or not the level is optimal. Are too many bought out and is it skewing the system in their favour? Can we maintain this? It seems to have increased quite dramatically and could, in principle, drop just as fast. Has it allowed us to take risks that will suddenly come back to bite us in a few years time. We also, in the past, had world-class research universities without needing a significant fraction to be bought out from teaching and administration. They coped fine back then, surely we can do the same now. Also, what do students think if they come to what they think is a world-class research university and then barely have any contact with the research leaders because they’re all bought out. This may all seem as though I’m against this, but I’m actually not really sure and I am pushing the implications to the extreme. I could probably be persuaded that actually everything is fine and that this is a really a good system that provides strength and depth in research and that it can be done without negatively impacting the other activities that need to take place in a World-class research university. If you have any thoughts, feel free to make a comment.

Latest AGP round

It seems that the latest STFC Astronomy Grants Panel (AGP) results are slowly being released. It also seems that a number of people are somewhat disappointed with the outcome. A recent post by Disturbing the Universe titled Doomed suggests that things are far from rosy (to say the least). Doomed may be a bit extreme, but I have to admit that I am quite worried about the future. The funding level has dropped dramatically since about 2006 and it seems to me that we now have a large mismatch between the size of the community and the amount of available funding. This may not have been too much of a problem 10 years ago when grants primarily covered direct costs with a modest amount of overhead. Today, however, the overheards (indirect costs, estates and building etc) are significantly higher than they once were, and grants now also cover a fraction of an academic’s salary. There is therefore a great deal of pressure on academics to get grants as this is now a significant income stream for universities. I’m not convinced that universities will be happy supporting large astronomy groups if they can’t maintain a decent level of grant income.

There are also other issues with such low funding levels. As hard as panels and referees may try, it is probably difficult to really rank proposals in a manner that everyone would agree was correct. If, however, a reasonable fraction of the community is funded at any one time and if those who are unsuccessful can resubmit their proposals on a reasonable timescale (say once a year) this isn’t too much of a problem. Those who were unsuccessful can learn from the feedback and can have a decent chance of getting funded after one or two resubmissions (I actually think that submitting grant applications is quite a good way to think of sensible and interesting research projects). The current situation seems to be that less than a third of academics will be funded at any one time and those who are unsuccessful cannot resubmit for 3 years. Such a low effective success rate means that uncertainties are magnified. We can’t really say that the projects that were regarded as just not good enough to fund were definitely worse than those that were on the other side of the boundary. However, those that are not funded are at a disadvantage compared to those that are, as they have to spend 3 years without funding and then compete against the same cohort when the grant cycle repeats.

There are also – in my view at least – other issues. The UK community is reasonably broad, but is dominated by some research areas. With a sudden change in the funding level (as has essentially happened), how do we ensure that the dominant areas don’t suddenly become even more dominant. They probably look stronger when metrics are taken into account (more people means more papers means more citations). Also, these areas will almost certainly dominate committees and decision making bodies (simply because there are more people in these areas than in others) and these people, with the best will in the world, are unlikely to be completely unbiased when deciding what is important and worth funding and what isn’t. It’s hard to see how the smaller areas will be able to survive when the funding levels are as low as they currently are.

I guess, it’s not yet all doom and gloom and maybe things will get better or maybe the future just isn’t as bleak as some may think. I just have trouble understanding how we can maintain the current community given the current funding level. Either a significant fraction of the community are doing research that isn’t worth funding and hence the community should really get smaller, or they are doing worthwhile research but we just can’t afford to fund it all at the moment (in which case we have to aim for an increase in the near future). There is also an alternative that I’m not completely against. As long as there is a bit of money for travel and computing, many academics will be happy to do research without necessarily having a postdoc or even a student. They may not get as much done, but it should still be worthwhile science. However, given the amount of extra money that grants are now bringing into the university system, I can’t see the university management being all that happy with this option.

EPSRC studentships

I’m somewhat uncertain about how to react to the news that EPSRC is cutting the number of studentships by about 30%. If you’ve read some of my earlier posts you’ll know that I feel that we may be producing too many PhD graduates. This isn’t because I don’t believe that PhDs are useful, it’s because I’m not sure it’s necessary to have as many PhD graduates as we currently have. It’s also because there are, in my opinion, real problems with the academic career structure that is exacerbated by the large number of PhD graduates.

Many who start a PhD do so because they would like to end up doing independent research. The main way to do this is to become an academic in a university. The fraction of PhD students who can, realistically, achieve this is now very small. Some argue that this doesn’t matter because those who don’t become academics go into industry and do very well. This is absolutely fine and I have no problem with someone doing a PhD and then choosing to go into industry. A concern I have, however, is that it can still be very disheartening for those who had hoped for an academic career and didn’t realise how difficult it was to do so. We also have to be careful that we don’t discourage, because of the difficulty of having an academic career, potentially excellent researchers from starting a PhD in the first place.

The other concern I have is that some feel that PhDs should become degrees in which people are taught research skills. I sat through a meeting recently where a concern was expressed that our PhD graduates typically were not competitive internationally. Someone then responded by saying that this didn’t matter as their PhD students were snapped up by industry. That’s great, except that in my opinion a PhD from a top UK university should typically allow that person to compete for research jobs anywhere in the world. They don’t have to do so, but it should allow them to do so if they so choose. If this is no longer then case, then our PhDs are no longer degrees in which students learn to undertake independent, world-class research. I think this is a crucial aspect of a PhD, otherwise we’re wasting everyone’s time.

The solution, in my view, is to expand the number of degrees. If we introduce a research Masters degree, students could learn, in a year or two, very useful research skills that will translate very well into industry. The tops students could then go on to do PhDs and the rest could go out into industry where they could contribute greatly. This would be more cost effective and those going into industry would do so a year or two earlier than they would do if they’d done a PhD and probably with most of the research skills they would need. If the cuts to EPSRC studentships was an attempt to rebalance the system, I might be quite pleased. However, it does seem to be purely a cut because of a reduction in their budget and does not appear to be based on any sense of attempting to produce a sensible system that will address issues relating to the academic career structure without reducing the number of research trained people going into industry. Admittedly EPSRC is a research council and doesn’t have any say in the structure of degrees at UK universities. It’s unfortunate, however, that there doesn’t appear to be any attempt to try and use this as an opportunity to address issues with the degree structure and career structure in UK universities.