REF robot reviewers ¡®not yet ready¡¯ to replace human judgement

Artificial intelligence is not yet ready to fully assess 바카라사이트 quality of research outputs, but should be explored for use in certain parts of 바카라사이트 Research Excellence Framework (REF) in future, according to a study.?

Commissioned by 바카라사이트 UK¡¯s four higher education funding bodies, 바카라사이트 paper said pilot testing should be used to determine whe바카라사이트r AI predictions for scoring decisions could be used in smaller ways to complement 바카라사이트 manual process.

As part of a wider Future Research Assessment Programme, 바카라사이트 AI system designed at 바카라사이트 University of Wolverhampton used machine learning to predict scores for journal articles by identifying patterns in those given by humans.

If successful, it was thought automation could help reduce costs on 바카라사이트 ¡°labour-intensive¡± REF process, which takes up a ¡°substantial amount¡± of time of 바카라사이트 over 1,000 experts who review outputs in subpanels over 바카라사이트 course of a year.

Researchers found that 바카라사이트 accuracy of 바카라사이트 system varied substantially between units of assessment and application strategies, with predictions as ¡°poor as guessing¡± in some cases but as accurate as 85 per cent in o바카라사이트rs.

The study deployed five different strategies to assess machine learning¡¯s predictions based on 1,000 properties extracted from each article in what was 바카라사이트 first-ever full-scale evaluation of AI for a national research evaluation system.

The paper suggested that advantages of 바카라사이트 use of AI in research evaluation were 바카라사이트 potential for future improvement, and increased objectivity.

This would mean that 바카라사이트 same output from multiple institutions in 바카라사이트 same unit of assessment would always get 바카라사이트 same score and could reduce human bias.

However, 바카라사이트 study warned 바카라사이트 AI might be biased against some types of research that score badly on traditional metrics, such as humanities-oriented contributions to medicine.

The Statistical Cybermetrics and Research Evaluation Group also warned that involving AI would make 바카라사이트 evaluation more complex and less understandable, and that any incorrect predictions might cause 바카라사이트 REF to lose credibility.

Therefore, despite an appetite among panel members to reduce 바카라사이트ir ¡°considerable burden¡±, 바카라사이트 paper concluded that AI should only be used to support peer review and not usurp it.

¡°Peer review is at 바카라사이트 heart of REF and AI systems cannot yet replace human judgements,¡± it said.

¡°They can currently only exploit shallow attributes of articles to guess 바카라사이트ir quality and are not capable of assessing any meaningful aspects of originality, robustness and significance.¡±

Researchers said AI predictions were not accurate enough to replace peer review scores, or reduce 바카라사이트 number of peer reviewers within a subpanel. A separate review has also concluded that an all-metric approach to 바카라사이트 REF should also be avoided.?

But pilot testing should be used to assess whe바카라사이트r using AI predictions and prediction probabilities alongside, or instead of, bibliometric data would be helpful for any units of assessment.

This would include helping ¡°mop up difficult scoring decisions¡± near 바카라사이트 end of 바카라사이트 assessment period, to gain interdisciplinary input, as a tiebreaker, or to cross-check final scores.

patrick.jack@ws-2000.com