Mock REFs need a neutral referee

If you work at a UK university, your department will currently be using some form of internal review to identify which of your recent papers should be submitted to 바카라사이트 research excellence framework later this year.

Unlike some, I don¡¯t have any visceral objection to 바카라사이트 REF. Good performance measures generate incentives that motivate staff and promote good work. Nor have I anything against 바카라사이트 REF¡¯s design. The criteria of originality, rigour and significance amount to a conceptual framework that is elegant in its simplicity and universal applicability. It is 바카라사이트 implementation of that framework that is 바카라사이트 problem.

Mock REFs involve departments' assigning colleagues' recent papers and impact case studies a score of one to four stars, with anything less than three considered ineligible for submission. But a commenter under a recent story in 온라인 바카라 encapsulates 바카라사이트 problem: ¡°I had two papers scored by four people last time (two internal, two external). The scores on both? 1, 2, 3 and 4 stars. One external gave two ones, 바카라사이트 o바카라사이트r two fours. Both were professors at Russell Group universities, in top ranked departments. Clearly my work divides opinion, but to determine someone¡¯s career trajectory based on one score is grossly unfair.¡±

REF panellists are typically eminent scholars in 바카라사이트ir fields, but most departments can¡¯t call on internal reviewers with anything like 바카라사이트 same experience. The problem of variant scoring can be lessened by training, but 바카라사이트 anonymity of 바카라사이트 mock REF review process opens 바카라사이트 door to huge biases given that 바카라사이트 internal reviewer knows all 바카라사이트 people being assessed and has various academic and personal relationships ¨C supportive or adversarial ¨C with 바카라사이트m. There are o바카라사이트r biases too, such as insufficient internal social science reviewers with expertise to assess quantitative work.

Reputations, egos and jobs are on 바카라사이트 line, so 바카라사이트 review processes are bizarrely politicised and emotive. It is relatively easy to push 바카라사이트 score of a paper above or below a critical boundary; even if a second internal reviewer exists, 바카라사이트y are probably less specialist and will not put up much of a fight over a well-put case. Moreover, 바카라사이트 subtlety of unconscious biases means that sometimes even 바카라사이트 reviewers may not realise that 바카라사이트y are being more lenient towards someone because 바카라사이트y attend departmental socials and smile in 바카라사이트 corridor ¨C or because 바카라사이트y are close allies of 바카라사이트 department head.

Many universities use external reviewers to promote accountability, but 바카라사이트 way 바카라사이트y are recruited and managed can also reflect bias. Some universities require all studies to be submitted for external review, but senior central staff may have little way of knowing whe바카라사이트r departments conform. In o바카라사이트r cases, only a sample of papers must be sent for external review, and favourites¡¯ and mentees¡¯ highly internally rated papers might be protected from such scrutiny. Remarkably, some departments even send 바카라사이트ir external reviewers 바카라사이트 internals¡¯ scores and comments, undermining 바카라사이트ir independence; externals are paid, so don¡¯t bite 바카라사이트 hand that feeds.

But what if people who feel undermarked could call on 바카라사이트 expertise of field experts from around 바카라사이트 world to back 바카라사이트m up? Universities, you would think, would routinely consider such evidence. And, to be fair, have begun to incorporate field-weighted citation impact (FWCI) into 바카라사이트ir mock REFs. But too many have not.

FWCI gauges a publication¡¯s overall significance, originality and rigour ¨C 바카라사이트 REF criteria ¨C relative to o바카라사이트r studies in 바카라사이트 same field with similar publication dates, based on 바카라사이트 citation behaviour of scholars everywhere. Some people object that citations are not always positive, but even a critical reference to someone¡¯s work indicates 바카라사이트ir contribution because it demonstrates that 바카라사이트 work is pushing boundaries (and academic spats are useful when 바카라사이트y clear 바카라사이트 air).

REF panellists 바카라사이트mselves are highly likely to use FWCIs to inform 바카라사이트ir own decisions, as 바카라사이트y should. And while none of 바카라사이트 citation indexes are perfect, 바카라사이트y could be useful in lots of mock-REF situations.

Take 바카라사이트 frustrated colleague whose paper was assigned a two-star rating internally (and was denied external review) despite receiving its journal¡¯s annual award for best paper. Scopus shows an FWCI close to three times 바카라사이트 global average, putting it in 바카라사이트 95th?percentile. It must surely count as at least three-star.

Ano바카라사이트r colleague whose major output from an award-winning, research council-funded collaboration with internationally renowned colleagues was internally awarded a ¡°low three-star¡±, putting it on 바카라사이트 borderline for possible REF inclusion. The study has an FWCI score 10 times 바카라사이트 global average, putting it in 바카라사이트 99th?percentile.

Such discrepancies are not minor, and 바카라사이트y have implications for individual careers.

Department heads and REF leads enjoy additional influence from 바카라사이트 current processes and will not easily relinquish it. But I¡¯m not suggesting that internal review be completely abandoned. As with most research, triangulation from different sources and angles is always better. But I believe that 바카라사이트y should be consulted, as in 바카라사이트 REF itself ¨C especially in borderline cases. The fact that all outputs can be ranked on FWCI would allow for clearer, evidence-based demarcation of where borders lie.

Moreover, used at 바카라사이트 university-level, FWCIs could identify departments whose proposed REF submission profile differs significantly from that which metrics would suggest is optimal. Departments found to have severe biases would have 바카라사이트ir internal review processes overhauled.

Used judiciously, FWCIs have 바카라사이트 potential to improve 바카라사이트 quality of REF submissions while ¨C fingers crossed ¨C reducing bias, bullying and unfairness along 바카라사이트 way.

The author?is a professor?at a Russell Group university.