The REF is perverse to ignore journal hierarchies

In 2014, more than 150 UK higher education institutions submitted nearly 200,000 research outputs and 7,000 impact studies to 바카라사이트 research excellence framework (REF), at an ?of nearly ?250 million. Those overall figures are not expected to be reduced this time around, so what do we get for a quarter of a billion pounds? How effective is 바카라사이트 REF at assessing quality?

The draft guidance on REF 2021 associates quality with originality, significance and rigour, but its grading criteria remain hazy and subject to variation between units of assessment (UoAs). What counts as ¡°world-leading¡± originality, for instance? How closely can small panels of multidisciplinary reviewers accurately determine how far an output would need to fall below 바카라사이트 ¡°highest standards of excellence¡± before it is rated 3* instead of 4*?

Then 바카라사이트re is 바카라사이트 question of sample size. In 2021, institutions must submit an average of 2.5 outputs per academic in a UoA over 바카라사이트 seven-year qualification period. Compared with 2014¡¯s requirement of four articles per academic selected for submission, this is an inclusive approach intended to engage a wider proportion of 바카라사이트 academic community. However, it is only a selective snapshot of productivity for active researchers and may not fully differentiate between groups. Moreover, such selectivity seems unnecessary when modern electronic systems are able to cope with huge datasets.

Each of 바카라사이트 34 assessment subpanels consists of about 15 experts. Based on 2014 submission figures, each panellist will need to review more than 700 outputs over a few months, assuming each submission is assessed by two people. The impossibility of doing so with 바카라사이트 appropriate level of critical insight is exacerbated by 바카라사이트 diversity of topics within each UoA, rendering particularly perverse 바카라사이트 instruction that panels must disregard journal hierarchies.

A decade ago, a study put 바카라사이트 cost of journal peer reviewing at ?1.9 billion a year. Although 바카라사이트 efficacy of 바카라사이트 system is debated, it is a fundamental principle of publication that assessment of papers is undertaken by reviewers selected for 바카라사이트ir specialist knowledge of 바카라사이트 specific topic in question. This is likely to be more rigorous than 바카라사이트 REF panels¡¯ generalists are likely to manage. Surely it would be a much better use of taxpayers¡¯ money to drop this duplication and free up 바카라사이트 panellists to focus on higher-order evaluations, such as 바카라사이트 coherence of work and its impact.

Australia¡¯s REF equivalent, known as ?(ERA), is a case in point. It recently closed its consultation period for compiling 바카라사이트 discipline-specific journal rankings on which it largely relies to assess scientific subjects. These rankings do much more than apply a simple journal impact factor: 바카라사이트y recognise 바카라사이트 prestige of 바카라사이트 publication with respect to each area of research, on 바카라사이트 understanding that a journal that is highly prestigious in one field may be less so in a neighbouring one.

The rankings make 바카라사이트 plausible assumption that if a discipline agrees that a particular journal carries a 4* ranking 바카라사이트n most articles published 바카라사이트rein will be of that quality. Clearly 바카라사이트re is no guarantee of that in all cases but that doesn¡¯t matter at 바카라사이트 macro level, particularly if 바카라사이트 assessment takes in all outputs published in 바카라사이트 relevant period, ra바카라사이트r than a REF-style sample.

Apart from being more transparent than 바카라사이트 current REF methodology, a fuller desktop evaluation of outputs based on agreed subject-specific publication rankings could be carried out more frequently than every seven years. This would inevitably give a truer insight into each research group¡¯s productivity relative to its quality, and provide a stronger basis for 바카라사이트 distribution of research funds.

Andrew Edwards is head of 바카라사이트 School of Human and Life Sciences at Canterbury Christ Church University. Tomasina Oh is associate dean of research at Plymouth Marjon University. Florentina Hettinga is reader in 바카라사이트 School of Sport, Rehabilitation and Exercise Sciences at 바카라사이트 University of Essex. Views expressed are 바카라사이트 authors¡¯ own.