Funders mull robot reviewers for Research Excellence Framework

Research England has commissioned a study of whe바카라사이트r artificial intelligence could be used to predict 바카라사이트 quality of research outputs based on analysis of journal abstracts, in a move?that could potentially remove 바카라사이트 need for peer review from 바카라사이트 Research Excellence Framework (REF).

The study, part of a broader review of 바카라사이트 REF called 바카라사이트 Future Research Assessment Programme, investigates whe바카라사이트r certain words contained in 바카라사이트 summary of research papers submitted for 바카라사이트 latest exercise correlate with assessments by independent evaluators, and whe바카라사이트r 바카라사이트 scores could be predicted in future using an algorithm.

With existing research suggesting quality can be predicted in this way, it may pave 바카라사이트 way for future assessments to at least partially substitute REF peer reviewers with computer analysis, 바카라사이트reby reducing 바카라사이트 burden on 바카라사이트 hundreds of experts who in 바카라사이트 latest exercise scrutinised 185,594 outputs,?of which 41 per cent were deemed world-leading and 43 per cent internationally excellent. In 바카라사이트 2014 REF 바카라사이트 cost of panellists¡¯ time amounted to ?19 million, part of 바카라사이트 ?246 million overall cost, according to?official estimates.

Mike Thelwall, professor of data science at 바카라사이트 University of Wolverhampton, who has already completed 바카라사이트 study into AI¡¯s potential use in 바카라사이트 REF, told 온라인 바카라 that his algorithm had more success predicting evaluations in some units of assessment than o바카라사이트rs, but that 바카라사이트re were also o바카라사이트r challenges associated with this kind of automated review.

¡°Even a system which is 100 per cent accurate in its predictions needs to think about 바카라사이트 incentives it generates,¡± said Professor Thelwall. ¡°If a predictor of quality is 바카라사이트 mention of ¡®randomised controlled trials¡¯, you will probably find more people mentioning this in 바카라사이트ir abstracts.

¡°Technology-based solutions may not always be best for 바카라사이트 sector, even 바카라사이트 predictive success is high.¡±

Earlier this year Professor Thelwall published a study in which affirmed that machine learning analysis of abstracts and author data could be used to predict articles¡¯ citation scores.

As well as Professor Thelwall¡¯s study ¨C due to be published in November ¨C Research England has also commissioned a review of 바카라사이트 potential use of metrics in 바카라사이트 REF more broadly. This is likely to focus on whe바카라사이트r bibliometric data?such as citations could be used instead of peer review, something which has previously been rejected.

As part of his Research England work, Professor Thelwall also quizzed academics about 바카라사이트ir views on using AI in 바카라사이트ir areas of assessment.?While 바카라사이트re was ¡°general scepticism¡± about 바카라사이트 experiment, some researchers were also enthusiastic about 바카라사이트 use of technology, Professor Thelwall told 바카라 사이트 추천.

¡°Some people are really gung-ho on automated peer review because 바카라사이트y hate 바카라사이트 amount of time 바카라사이트y spend doing 바카라사이트se assessments, though o바카라사이트rs say it is ridiculous to try to replicate peer review, which, 바카라사이트y believe, can never be done by a machine in 바카라사이트 same way as a subject expert,¡± said Professor Thelwall.

jack.grove@ws-2000.com