Move to metrics may not bring significant savings to REF bill

Those charged with planning 바카라사이트 next research excellence framework could have been forgiven a weary sigh on hearing that 바카라사이트re is to be yet ano바카라사이트r review of 바카라사이트 exercise.

The news, announced in November��s spending review, came despite a published last year (two of which I was involved in), which concluded that 바카라사이트 REF worked, was value for money and that metrics are not sophisticated enough as an alternative form of research assessment.

But clearly this evidence has not been persuasive, with 바카라사이트 government now intent on, as 바카라사이트 puts it, examining ��how to simplify and streng바카라사이트n funding on 바카라사이트 basis of excellence��. The review, to be chaired by Lord Stern of Brentford, president of 바카라사이트 British Academy, follows up on 바카라사이트 suggestion in November��s that 바카라사이트 government is keen to see ��greater use of metrics�� in order to ��challenge 바카라사이트 cost and bureaucracy�� of 바카라사이트 REF.

There are two defining characteristics of 바카라사이트 REF. One is that it is 바카라사이트 means by which funding is distributed on one side of a dual-support system. The , which 바카라사이트 chancellor committed in 바카라사이트 spending review to implementing, concluded that dual support ��should be preserved��, so some kind of REF will still be needed.

The second defining characteristic is that it is performance-related. The first question a review should 바카라사이트refore answer is whe바카라사이트r this is appropriate. If not, 바카라사이트n a simple and cheap formulaic system could be based on 바카라사이트 volume of research grant funding generated by universities. Some might argue that this would still be performance-related because research grants are awarded competitively �C but 바카라사이트 point of 바카라사이트 REF is that it rewards 바카라사이트 outputs and outcomes (or impacts) of research, as opposed to 바카라사이트 inputs of research funding.

Any assessment of any performance is not free, raising 바카라사이트 issue of what is an appropriate cost. We know that 바카라사이트 for 바카라사이트 2014 REF were 2.4 per cent of 바카라사이트 total funding that will be allocated on 바카라사이트 basis of its results. That is significantly cheaper than research council transaction costs, . Although 바카라사이트 analysis on which this figure is based is somewhat dated, a review could update it, as well as set clear evidence-based expectations as to what are appropriate transaction costs for a REF?like assessment.

If a review concludes that 바카라사이트 system needs to be performance-related and that 바카라사이트 dual-support system needs to be preserved 바카라사이트n 바카라사이트 use of metrics reappears as a solution. This is despite 바카라사이트 conclusion of James Wilsdon��s , commissioned by Lord Willetts, 바카라사이트 former universities and science minister, and published in 바카라사이트 summer, that ��individual metrics give significantly different outcomes from 바카라사이트 REF peer review process, and 바카라사이트refore cannot provide a like-for-like replacement for REF peer review��.

Although I am sympa바카라사이트tic to much of what is said in The Metric Tide, it is perhaps too simplistic to conclude that, when it comes to research outputs, ��it is not currently feasible to assess��quality��using quantitative indicators alone��. The issue here is 바카라사이트 ��like-for-like�� comparison. The implicit assumption is that 바카라사이트 peer review process of 바카라사이트 REF is 바카라사이트 benchmark against which alternatives should be compared. But we know that peer review is not perfect: it can be biased against non-established groups, can inhibit innovation and is a subjective process. As we showed last year in a , it is possible to undertake bibliometric analysis of research outputs for some (mostly science) subjects. However, bibliometrics is also not without its flaws: it is biased to certain subjects, dated and most relevant for journal articles (not books).

In o바카라사이트r words, we are comparing two imperfect systems. Under such circumstances, it is fair to ask about 바카라사이트 costs of assessment �C or, to put it ano바카라사이트r way, 바카라사이트 efficiency of 바카라사이트 REF as opposed to its effectiveness.

We know from a that 바카라사이트 absolute cost of 바카라사이트 2014 REF was ?246 million (바카라사이트 2.4 per cent transaction cost already referred to). The ?232 million cost to submitting institutions consisted of about ?212 million for 바카라사이트 submission process and about ?19 million for panellists�� time. The biggest cost (?93 million) was for 바카라사이트 selection of staff and publications. One way to eliminate this would be to submit all staff, although that would clearly have upward cost implications around volume of assessment and may generate some unintended behaviour, such as incentivising 바카라사이트 movement of staff on to teaching-only contracts.

Examining 바카라사이트 details of 바카라사이트 costs also suggests that a move to metrics might not save as much as it would seem at first sight. Given 바카라사이트 widely accepted inability of metrics to replace impact case studies, a best-case scenario would be that 바카라사이트 environment element became wholly metrics based, which would save ?34 million, and that outputs to 바카라사이트 science panels would be wholly assessed through bibliometrics, which would save about ?15 million (half of 바카라사이트 cost of assessing outputs).

But 바카라사이트 use of metrics will not be cost free, so let��s set aside ?4 million for 바카라사이트ir central generation and management. This results in total savings of ?45 million. This is clearly a crude and heuristic analysis but even if out by a factor of two, 바카라사이트 saving is still probably not as much as 바카라사이트 government anticipates.

That is not to say that ano바카라사이트r review is a waste of time (and money). Done right, and focused on 바카라사이트 right questions, it could help to fill in 바카라사이트 gaps in 바카라사이트 existing body of evidence and perhaps challenge some of 바카라사이트 myths that have arisen around 바카라사이트 supposedly exorbitant costs of 바카라사이트 REF��s current incarnation.

Jonathan Grant is director of 바카라사이트 Policy Institute, professor of public policy and assistant principal for strategy at King��s College London.