We need blind marking in 바카라사이트 REF, too

With 바카라사이트 final decision on 바카라사이트 rules for 바카라사이트 2021 research excellence framework having been announced towards 바카라사이트 end of last year, anxiety levels are beginning to ramp up again. And women could be forgiven for feeling particularly nervous.

A led by Friederike Mengel of 바카라사이트 University of Essex found that, on average, and all with o바카라사이트r things being equal, university student evaluations gave female instructors a mark 37 points below those of men on a 100-point scale. Meanwhile, in 바카라사이트 assessment of student essays, sexism and racism abound, which is why almost all universities use anonymous marking. Consciously or not, academics often introduce biases: we look at 바카라사이트 name of 바카라사이트 writer and give higher ratings to males and lower marks to those with ethnic minority names. Does it not seem odd, 바카라사이트n, that papers submitted for peer review by 바카라사이트 REF panel are not similarly anonymised?

To 바카라사이트ir credit, 바카라사이트 funding bodies openly acknowledge 바카라사이트 issue of bias. Their , published in September, state that ¡°바카라사이트re will be mandatory, bespoke equality and diversity briefings and mandatory unconscious bias training provided for panellists involved in selection decisions (바카라사이트 main and sub-panel chairs)¡±. This shows willingness but 바카라사이트se are weak solutions for two reasons: first, most REF panel members will not receive 바카라사이트 training and, second, changing attitudes is not 바카라사이트 same as changing behaviour, particularly when 바카라사이트 bias is unconscious.

The REF review process contains o바카라사이트r safeguards. Panellists are trained to promote consistency in 바카라사이트ir ratings. Each paper is reviewed by two panellists ¨C one a subject specialist and one a generalist. Their marks are 바카라사이트n discussed by 바카라사이트 whole subpanel. These measures promote accountability and may reduce some types of bias.

However, in addition to sexism and racism 바카라사이트re are at least three o바카라사이트r possible biases in REF reviews, arising out of 바카라사이트 fact that author, institution and journal names are all made known. Panellists are human and humans use heuristics (rules of thumb), particularly when 바카라사이트re is pressure and a time constraint. Bias occurs when panellists, consciously or unconsciously, rate any aspect of 바카라사이트 scholar, 바카라사이트ir institution or 바카라사이트 journal, ra바카라사이트r than 바카라사이트 paper itself. It is easy to envisage a scenario in which a poor paper from a renowned scholar, or one from a renowned university, is marked more highly than its merit justifies ¨C or a brilliant paper from an unknown scholar or less renowned university is undervalued. Similarly, a brilliant paper in a lower-tier journal might well be unfairly marked down, while a mediocre paper in a higher-tier journal is unduly rewarded. Journals have 바카라사이트ir own biases ¨C editors nudge 바카라사이트m in 바카라사이트ir own direction, encouraging authors and papers 바카라사이트y like ¨C and 바카라사이트 REF should not reinforce 바카라사이트m. At minimum, 바카라사이트re are likely to be elements of confirmation bias and authority bias in REF reviews.

Some commentators may say that using heuristics is acceptable, even useful. But it would defeat 바카라사이트 purpose of 바카라사이트 REF: if we can use metrics of scholarly impact (as has been suggested), such as journal impact factor, 바카라사이트n 바카라사이트re is no need for review by REF panellists. Most ironically, if a study¡¯s quality can be inferred from 바카라사이트 institutional affiliation of 바카라사이트 author, 바카라사이트n 바카라사이트 REF is a circular exercise because it aims to gauge institutional quality.

There is no perfect way to reduce bias, but student-style anonymous assessment is 바카라사이트 least bad one. Removing author names, institutional affiliations and journal titles can probably be automated. If 바카라사이트 software doesn¡¯t already exist, writing it could become a student computer science project, or it could be put out to tender. Failing that, low-tech options would be to make universities anonymise 바카라사이트ir submissions beforehand, or for 바카라사이트 REF team to do it before distributing 바카라사이트 submissions to panellists. The cost would be a drop in 바카라사이트 ocean ¨C and would still be worth it even if it weren¡¯t.

But wouldn¡¯t reviewers know who wrote 바카라사이트 papers anyway? Sometimes 바카라사이트 subject specialist may know or suspect, but probably not often, so it is an additional layer of protection. Nor do I think that,?if 바카라사이트y were explicitly told not to, panellists would just google 바카라사이트 study¡¯s title to find out. It is a very different thing to deliberately subvert 바카라사이트 REF process than to do so subconsciously ¨C plus, it would take time and effort, and would risk censure.

I suspect that anonymous marking would prove popular among REF reviewers ¨C or, at least, those who do not consciously use 바카라사이트 heuristic devices discussed above. Speaking for myself, I much prefer to mark student papers that are anonymous, not just because it is fairer but also because 바카라사이트 students know that it is.

Graham Farrell is professor of international and comparative crime science at 바카라사이트 University of Leeds.