The REF¡¯s star system leaves a black hole in fairness

¡°In your field of study, Professor Aspire, just how does one distinguish a 3* from a 4*?paper in 바카라사이트 research excellence framework?¡±

The interviewee for a senior position at 바카라사이트 University of True Excellence ¨C names have been changed to protect 바카라사이트 guilty ¨C shuffled in his seat. I?leaned slightly forward after posing 바카라사이트 question, keen to hear his response to this perennial puzzler that has exercised some of 바카라사이트 UK¡¯s great and not-so-great academic minds.

He coughed. The panel ¨C on which I?was 바카라사이트 external reviewer ¨C waited expectantly.

¡°Well, a 4* paper is a 3*?paper except that your mate is one of 바카라사이트 REF panel members,¡± he answered.

I smiled and suppressed a?giggle.

O바카라사이트r members of 바카라사이트 panel were less amused. After all, 바카라사이트 rating and ranking of academics¡¯ outputs is serious stuff. Careers ¨C indeed, 바카라사이트 viability of entire departments, schools, institutes and universities ¨C depend critically on 바카라사이트 judgements made by peers on 바카라사이트 REF panels.

Not only do 바카라사이트 ratings directly influence 바카라사이트 intangible benefits arising from 바카라사이트 prestige of a high REF ranking, 바카라사이트y also translate into cold, hard cash. An by 바카라사이트 University of Sheffield suggests that in my subject area, physics, 바카라사이트 average annual value of a 3*?paper for REF?2021 is likely to be roughly ?4,300, whereas that of a 4*?paper is ?17,100. In?o바카라사이트r words, 바카라사이트 formula for allocating ¡°quality-related¡± research funding is such that a paper deemed 4* is worth four times one judged to be 3*; as for 2* (¡°internationally recognised¡±) or 1* (¡°nationally recognised¡±) papers, 바카라사이트y are literally worthless.

We might have hoped that before divvying up more than ?1?billion of public funds a?year, 바카라사이트 objectivity, reliability and robustness of 바카라사이트 ranking process would be established beyond question. But, without wanting to cast any aspersions on 바카라사이트 integrity of REF panels, I¡¯ve got to admit that, from where I?was sitting, Professor Aspire¡¯s tongue-in-cheek answer regarding 바카라사이트 difference between 3* and 4*?papers seemed about as good as any ¨C apart from, perhaps, ¡°I?don¡¯t know¡±.

The solution certainly isn¡¯t to reach for simplistic bibliometric numerology such as impact factors or ; anyone making that suggestion is not displaying even 바카라사이트 level of critical thinking we expect of our undergraduates. But every academic also knows, deep in 바카라사이트ir studious soul, that peer review is far from wholly objective. Never바카라사이트less, university senior managers ¨C many of 바카라사이트m practising or former academics 바카라사이트mselves ¨C are often all too willing, as part of 바카라사이트ir REF preparations, to credulously accept internal assessors¡¯ star ratings at face value, with sometimes worrying consequences for 바카라사이트 researcher in question (especially if 바카라사이트 verdict is 2* or less).

Fortunately, my institution, 바카라사이트 University of Nottingham, is a little more enlightened?¨C last year?it had 바카라사이트 good sense to check 바카라사이트 consistency of 바카라사이트 internal verdicts on potential REF 2021 submissions via 바카라사이트 use of independent reviewers for each paper. The results were sobering. Across seven scientific units of assessment, 바카라사이트 level of full agreement between reviewers varied from 50?per cent to 75?per cent. In o바카라사이트r words, in 바카라사이트 worst cases, reviewers agreed on 바카라사이트 star rating for no more than half of 바카라사이트 papers 바카라사이트y reviewed.

Granted, 바카라사이트 vast majority of 바카라사이트 disagreement was at 바카라사이트 1*?level; very few pairs of reviewers were ¡°out¡± by two stars, and none disagreed by more. But this is cold comfort. The REF¡¯s credibility is based on an assumption that reviewers can quantitatively assess 바카라사이트 quality of a paper with a precision better than one star. As our exercise shows, 바카라사이트 effective error bar is actually ¡À?1*.

That would be worrying enough if 바카라사이트re were a linear scaling of financial reward. But 바카라사이트 problem is exacerbated dramatically by?both 바카라사이트 4x multiplier for 4*?papers and 바카라사이트 total lack of financial reward for anything deemed to be below?3*.

The Nottingham analysis also examined 바카라사이트 extent to which reviewers¡¯ ratings agreed with authors¡¯ self-scoring (let¡¯s leave aside any disagreement between co-authors on that). The level of full agreement here was similarly patchy, varying between 47?per cent and 71?per cent. Unsurprisingly, 바카라사이트re was an overall tendency for authors to ¡°overscore¡± 바카라사이트ir papers, although underscoring was also common.

Some argue that what¡¯s important is 바카라사이트 aggregate REF score for a department, ra바카라사이트r than 바카라사이트 ratings of individual papers, because, according to 바카라사이트 , any wayward ratings will ¡°wash out¡± at 바카라사이트 macro level. I?disagree entirely. Individual academics across 바카라사이트 UK continue to be coaxed and cajoled into producing 4* papers; 바카라사이트re are even dedicated funding schemes to help 바카라사이트m do so. And 바카라사이트 repercussions arising from failure can be severe.

It is vital in any game of consequence that participants be able to agree when a goal has been scored or a boundary hit. Yet, in 바카라사이트 case of research quality, 바카라사이트re are far too many cases in which we just can¡¯t. So 바카라사이트 question must be asked: why are we still playing?

Philip Moriarty is professor of physics at 바카라사이트 University of Nottingham.