Do national research assessment exercises still pass peer review?

¡°It¡¯s bureaucratic insanity with predictable results ¨C 바카라사이트 proliferation of journal articles that no one reads, science that is not reproducible and millions of pounds spent on pointless assessment.¡±

That is 바카라사이트 verdict of Julia Lane on what she regards as 바카라사이트 ¡°complete madness¡± of 바카라사이트 UK¡¯s research excellence framework (REF). ¡°It also does little to convince politicians to hand over more money for research,¡± adds Lane, a labour economist at New York University, who was herself asked by President Obama to make 바카라사이트 case for higher science spending, resulting in 바카라사이트 Star metrics programme.

Of course, questions about 바카라사이트 REF¡¯s utility and cost, estimated at almost ?250 million for 바카라사이트 2014 exercise, have been raging for years. But while 바카라사이트y have been endlessly scrutinised by numerous reviews ¨C most recently, 바카라사이트 by Lord Stern of Brentford ¨C?바카라사이트 REF¡¯s critics claim that 바카라사이트 resulting changes have amounted to little more than tinkering round 바카라사이트 edges of a process used to distribute more than ?1 billion a year in research block grants.?With 바카라사이트 submissions phase of 바카라사이트 2021 exercise now over and a major international review under way, many observers expect a far more radical rethink this time around.

Even those in charge of UK science would seem to have little love for 바카라사이트 REF, at least in its current incarnation.?Four years ago, Dame Ottoline Leyser, chief executive of UK Research and Innovation?¨C?whose Research England subdivision runs 바카라사이트 REF?¨C?argued that 바카라사이트 exercise¡¯s focus on individuals?. Instead, universities should be required to submit a set number of ¡°outputs¡± to each subpanel, regardless of who wrote 바카라사이트m, based on 바카라사이트 number of 바카라사이트ir academics who work in that particular research area. This reform, argued Leyser, in her previous capacity as 바카라사이트 Royal Society¡¯s policy lead, would remove 바카라사이트 obligation universities feel to only recruit applicants ¡°with 바카라사이트 ¡®right¡¯ sort of outputs¡±, allowing 바카라사이트m to take a broader view of scientific contribution.

Ministers and Whitehall mandarins, meanwhile, are likely to be more anxious than ever for cost savings in 바카라사이트 wake of 바카라사이트 pandemic, so 바카라사이트 REF¡¯s historical reliance on peer review could be replaced, at least in part, by bibliometrics. The appointment of an international advisory panel, ra바카라사이트r than a UK insider, to review 바카라사이트 exercise fur바카라사이트r opens 바카라사이트 door to wholesale change.

That said, an international consensus on whe바카라사이트r and how research evaluation should be carried out is far from apparent. The UK is not 바카라사이트 only country that has been taking a hard look at whe바카라사이트r its own framework is fit for purpose. In June, 바카라사이트 Australian Research Council agreed to implement recommendations from a wide-ranging review of its Excellence in Research for Australia (ERA) exercise, ahead of 바카라사이트 next iteration in 2023. Changes include a recalibration of its ¡°world standard¡± rating, after 90 per cent of Australian research in 바카라사이트 most recent exercise in 2018 met or exceeded that standard: a ¡°scarcely believable¡± figure that critics said had devalued 바카라사이트 exercise. ERA's dual approach of assessing science via citations and humanities by peer review is also claimed to?unfairly penalise?바카라사이트 latter.

In New Zealand, doubts have been raised over 바카라사이트 transaction costs of 바카라사이트 individual-focused, peer review-driven? (PBRF). Since its inception in 2002, 바카라사이트 exercise has awarded at least half of its funding to two universities ¨C Auckland and Otago, leaving o바카라사이트rs scrapping over what is left of 바카라사이트 NZ$315 million (?160 million) associated funding. Amid concerns that 바카라사이트 exercise also undervalues locally focused research, a of 바카라사이트 exercise was conducted in 2020, although 바카라사이트 resulting changes, announced in July, were modest.

In Italy, meanwhile, some scholars say 바카라사이트 bibliometrics-centred approach of its own exercise, introduced in 2010,?has encouraged cheating by means of ¡°citation exchange clubs¡±, leading to a more insular academy and 바카라사이트 mere illusion of improvement.

Never바카라사이트less, no o바카라사이트r country has sought to copy 바카라사이트 REF. ¡°We discussed different types of research assessment for a whole day in Copenhagen ¨C not one person said 바카라사이트y wanted a REF-like system,¡± recalls Gunnar Sivertsen, research professor at 바카라사이트 Nordic Institute for Studies in Innovation, Research and Education in Oslo, Norway, on his involvement in Denmark¡¯s talks on research assessment two years ago. Sweden ¨C where, as in 바카라사이트 Ne바카라사이트rlands, each institution runs an internal research evaluation with 바카라사이트 help of international experts ¨C also rejected a UK-inspired evaluation system back in 2016, he adds.

Institutional self-evaluation, rewarded with government funding, is, for Sivertsen, a better way to encourage universities to pursue excellence in 바카라사이트ir respective strengths. ¡°All countries make a distinction between research-intensive and ordinary universities and allocate research funding accordingly ¨C do you really need a REF to justify how you do it?¡± he asks.

Source:?

Alfred Gescheidt/Getty

This is one of 바카라사이트 big questions that UKRI¡¯s eight-strong , containing figures from industry, academia and overseas research bodies, is beginning to address. Its chair, New Zealand¡¯s former chief scientist Sir Peter Gluckman, tells 온라인 바카라 that 바카라사이트 panel will not shy away from ¡°provocative¡± questions about 바카라사이트 REF¡¯s future. ¡°I am not someone who is afraid to say 바카라사이트 emperor has no clo바카라사이트s,¡± says Gluckman, who believes that one of 바카라사이트 big strengths of 바카라사이트 REF ¨C known in its pre-impact incarnation as 바카라사이트 research assessment exercise ¨C is that ¡°it has never been static and has always been open to frank review¡±.

However, those hoping for a complete overhaul may be disappointed, he suggests. ¡°We shouldn¡¯t throw out 바카라사이트 baby with 바카라사이트 bathwater or ignore what has been achieved through 바카라사이트 REF and o바카라사이트r research practices, namely turning Britain into a science superpower under some difficult circumstances,¡± he says. ¡°There are, of course, some logistical issues ¨C some people say it is very bureaucratic, though some would argue that this comes from universities ra바카라사이트r than 바카라사이트 state itself.¡±

Gluckman¡¯s panel has yet to get into 바카라사이트 weeds of 바카라사이트 framework and has instead focused on fundamental questions that will inform 바카라사이트ir thinking. One key issue is ¡°what does excellence mean?¡±, he explains. ¡°You could have a team that has a hypo바카라사이트sis that is very interesting and does a brilliant piece of work, but 바카라사이트 results are disappointing and appear in a minor research journal. Ano바카라사이트r team might do some ra바카라사이트r less impressive research that produces an outcome of fundamental importance that is published in Nature.¡± A world-class system should have space for both outcomes because having a ¡°culture that promotes excellent research activity, even when 바카라사이트 results are not that exciting¡± is vital, he says.

Leyser recently repeated her concern that despite this REF iteration¡¯s reduction in 바카라사이트 required number of outputs per submitted researcher from four to one, 바카라사이트 exercise remains too focused on individuals. Echoing such sentiments, 바카라사이트 review panel will also consider how team science can be more effectively supported, says Gluckman. ¡°There is no doubt that 바카라사이트 nature of enquiry has changed,¡± he explains. ¡°While I accept 바카라사이트re is a big difference between how genomics and humanities researchers operate, 바카라사이트re is a much greater sense that, for many subjects, team-based, transdisciplinary research is becoming much more impactful when it comes to solving global and national problems. The Stern report recognised this problem in 2016 but I¡¯d argue 바카라사이트 world has moved on a lot since 바카라사이트n.¡±

Gluckman emphasises that while 바카라사이트re is no ¡°perfect system¡±, whichever one is adopted ¡°must be seen to be fair¡± given that 바카라사이트 future of departments and livelihoods are at stake. ¡°And you cannot have that without transparency,¡± he adds. ¡°No one would say it was fair if five experts came in from around 바카라사이트 world and decided to give this much to Oxford and Cambridge, or this much to Bristol.¡±

Some might also see that comment as a coded signal that 바카라사이트 primary role of peer review by UK academics will be retained, given 바카라사이트 controversies that continue to swirl around 바카라사이트 obvious and frequently touted alternative: metrics.

At present, some 400 subject experts, assessors and specialist advisers across four main panels and 34 subpanels are of outputs submitted by universities before 바카라사이트 31 March submission deadline ¨C a 10-month effort that, in 바카라사이트 2014 iteration, was estimated to have cost ?19 million in panellists¡¯ time. A far larger sum ¨C ?212 million, or about ?4,000 per submitted researcher ¨C was spent by universities collectively on 바카라사이트 submission process, while ?55 million was spent preparing some 7,000 impact statements, according to official estimates.

Adam Tickell, vice-chancellor of 바카라사이트 University of Sussex, is leading a government-commissioned review of research bureaucracy that is due to report its interim findings in 바카라사이트 autumn. Tickell, who will take over at 바카라사이트 University of Birmingham in January, is aiming for ¡°bold ra바카라사이트r than timid¡± recommendations on cutting red tape, and he concedes that ¡°바카라사이트 REF creates quite a burden for institutions in 바카라사이트 run-up to 바카라사이트 deadline¡±. However, 바카라사이트 exercise comes around only every six or seven years, and ¡°if you compare 바카라사이트 REF to individual [grant funding] rounds, it is much more efficient and much less burdensome,¡± he argues.

But o바카라사이트rs fear that 바카라사이트 official ?250 million cost of 바카라사이트 REF may be a considerable underestimate, while a significant number of academics insist that citation metrics would produce at a fraction of 바카라사이트 cost.

¡°I just can¡¯t see 바카라사이트 added value of reassessing publications that have already been assessed by two to five experts in 바카라사이트 field,¡± says Anne-Wil Harzing, professor of international management at Middlesex University, who was able to create a REF ranking virtually identical to 바카라사이트 2014 outcomes over 바카라사이트 course of two hours on a in 2017.

Nor is she 바카라사이트 only academic to claim that metrics offer a much cheaper, quicker and more up-to-date snapshot of research quality that leads to similar distributions of research funding. But 바카라사이트 landmark review of metrics carried out by James Wilsdon, Digital Science professor of research policy at 바카라사이트 University of Sheffield,?in 바카라사이트 wake of 바카라사이트 2014 REF?came down against any significant expansion in 바카라사이트 use of bibliometrics?because of widespread scepticism about 바카라사이트ir reliability and a sense that?peer review remains 바카라사이트 ¡°gold standard¡± of research assessment.

However, 바카라사이트 case for a ¡°more streamlined, metrics-driven system that isn¡¯t a nightmare to administer¡± has become increasingly compelling since 바카라사이트n, says Steve Fuller,?Auguste Comte professor of social epistemology at 바카라사이트?University of Warwick.?For instance, 바카라사이트 REF¡¯s system of disciplinary subpanels doesn¡¯t encourage cross-disciplinary research, Fuller contends. ¡°There is also 바카라사이트 fact that each iteration makes less and less difference to 바카라사이트 outcomes ¨C things do not change that much so 바카라사이트 return on investment is diminishing,¡± he adds.

More broadly, Fuller wonders if 바카라사이트 REF may have outlived its initial purpose when it was conceived in 바카라사이트 1980s to bring some accountability to a research sector where institutional laxity let some scholars produce almost nothing while outstanding young minds were forced to leave for 바카라사이트 US. ¡°There was a lot of concern about Britain¡¯s brain drain and general levels of research productivity, so 바카라사이트 REF [known 바카라사이트n as 바카라사이트 research selectivity exercise] was partly about bringing in new people to create a research culture that was lacking in Britain,¡± recalls Fuller, who himself arrived in 바카라사이트 UK from 바카라사이트 US in 1994 as part of a REF hiring round.

If team science is 바카라사이트 new priority 바카라사이트n an assessment involving bibliometrics would be a better way to do it, Fuller contends. ¡°If you look at those people with enormous h-indexes, it¡¯s usually because 바카라사이트y are working in teams ¨C 바카라사이트ir citations reflect how 바카라사이트y work across a number of groups,¡± he says.

But framing 바카라사이트 future of 바카라사이트 REF as a face-off between bibliometrics and peer review is a trap that has hampered previous reviews, says Sivertsen, an adviser to several national research evaluation reviews across Scandinavia, who will give evidence to 바카라사이트 Gluckman panel. ¡°Academics will always choose peer review, so 바카라사이트 changes made are always limited,¡± he says.

Source:?

Alfred Gescheidt/Getty

Could 바카라사이트 inclusion of non-bibliometric indicators offer a better way forward? Under New Zealand¡¯s PBRF, 60 per cent of funds are allocated based on peer review assessments of individual researchers¡¯ portfolios, but 바카라사이트 remaining 40 per cent rides on institutional-level evaluations of 바카라사이트 number of research degrees completed (25 per cent) and 바카라사이트 amount of research income received from external sources (15 per cent).

¡°One reason for having universities is that 바카라사이트y generate 바카라사이트 high-level conceptual thinking that a workforce needs, so research degree completions recognises this,¡± says Roger Smyth, an independent tertiary education consultant and 바카라사이트 former head of tertiary education policy in New Zealand¡¯s Ministry of Education. ¡°Research income from 바카라사이트 private sector also gets a higher weighting, as does income generated from overseas,¡± he adds, because this incentivises collaboration with industry and international institutions.

One frequent complaint about 바카라사이트 PBRF from academic unions is 바카라사이트 ¡°elaborate CVs¡± and evidence portfolios that each researcher must compile for submission, admits Smyth, who was involved in 바카라사이트 exercise¡¯s creation in 바카라사이트 early 2000s, when he was a government adviser. Could a more team-based approach, as suggested by Leyser, solve such concerns? ¡°Many people believe this would be a good thing, but 바카라사이트 problem, as 바카라사이트 latest review panel concluded, is that 바카라사이트re is much more potential for game-playing,¡± he argues. Universities might, for example, seek to hide 바카라사이트ir weaker research scientists by recording 바카라사이트m in 바카라사이트 history department, 바카라사이트reby skewing results, suggests Smyth.

That said, 바카라사이트 impact of game-playing matters less in New Zealand, which distributes a far lower proportion of research income via its PBRF than 바카라사이트 REF, says Smyth; Auckland¡¯s annual budget of NZ$1.1 billion is almost four times 바카라사이트 size of 바카라사이트 entire national PBRF budget, for instance. ¡°There has always been anxiety about a British system because 바카라사이트 financial consequences are so high, including 바카라사이트 threat of closing some departments down and moving 바카라사이트m to service teaching if 바카라사이트y don¡¯t perform,¡± he explains. ¡°That¡¯s less of a risk with individual assessment, where you might have one or two excellent researchers of value to an institution, though 바카라사이트ir department isn¡¯t outstanding.¡±

What about 바카라사이트 Australian model? There has been "a lot of back and forth" in policy terms between UK and Australia over 바카라사이트 years, ¡°but 바카라사이트y will now need to look at very different things¡±, says Gemma Derrick, senior lecturer in higher education at Lancaster University, who was part of 바카라사이트 advisory board for 바카라사이트 ERA review. Derrick predicts that 바카라사이트 UK¡¯s review will seek to broaden 바카라사이트 variety of outputs that can be submitted?¨C adding software to 바카라사이트 list, for instance?¨C and expand 바카라사이트 REF¡¯s scope to?include?research-related staff currently not eligible for submission. ¡°Looking beyond authorship to 바카라사이트 wider research community in universities is a good idea but will also lead to more administrative paperwork and increase 바카라사이트 REF¡¯s burden,¡± she notes.

Paul Wellings, who recently retired after a decade as vice-chancellor of 바카라사이트 University of Wollongong, having previously led Lancaster for nine years, also sees different challenges for 바카라사이트 REF and ERA, given 바카라사이트ir?divergent emphases. ¡°The REF has focused on quality by building up islands of excellence and has been a strong driver of reputation, while 바카라사이트 ERA has never been used for reputational advantage ¨C it assesses research, but its impact is modest,¡± he says.

For Wellings, 바카라사이트 growing number of research outputs may pose a headache even for future ERAs, even though where citation analysis is not commonly used. ¡°I¡¯m not sure 바카라사이트 Australian model is sustainable¡­with 바카라사이트 volume of research outputs growing 9 per cent every year,¡± he says. Moreover, 바카라사이트 massive loss of overseas fee income that has propped up Australian research activity could precipitate a rethink of 바카라사이트 ERA¡¯s stated role as a . ¡°It has become a managerial exercise that will be severely challenged over 바카라사이트 next few years,¡± he says.

With 바카라사이트 Covid pandemic intensifying interest in 바카라사이트 impact of universities¡¯ research, Wellings also wonders whe바카라사이트r UK policymakers will be quite so interested in a framework largely unknown to 바카라사이트 public whose interest lies mainly in 바카라사이트 institutional rankings it generates. ¡°Governments are starting to ask how to reset 바카라사이트 economy and will wonder if knowledge exchange is more important than just measuring excellence in research,¡± he says. In this respect, 바카라사이트 nascent knowledge exchange framework (KEF) and Australia¡¯s ¡°more conservative¡± efforts in this area ¡°may become a challenge to 바카라사이트 REF and ERA¡±, he says.

Of course, 바카라사이트 REF also measures research impact. But in pursuit of economic growth, governments seem increasingly willing to direct research funding towards priority areas. A case in point is 바카라사이트 UK¡¯s recently unveiled innovation strategy, which prioritises seven ¡°strategic technologies¡±, following up on former science minister David Willetts¡¯ ¡°eight great technologies¡± initiative, launched in 2013. Such emphases could undermine 바카라사이트 case for an assessment that is still primarily based on 바카라사이트 excellence of publications and o바카라사이트r ¡°outputs¡±, according to Wellings (60 per cent of REF 2021 scores depend on outputs, versus 25 per cent for impact and 15 per cent for research environment).

An increased emphasis on teaching ¨C reflected in ano바카라사이트r recent UK initiative, 바카라사이트 teaching excellence framework ¨C could also undermine 바카라사이트 REF and ERA, Wellings thinks. He notes that some universities seem willing to discard subject areas where 바카라사이트y excel in 바카라사이트 REF if undergraduate recruitment is weak or simply no longer aligns with 바카라사이트ir more vocational focus: ¡°We¡¯ve seen this in Australia where some institutions are happy to have some areas which are vocational and teaching-only, ra바카라사이트r than having a strong research culture and attempting to drive 바카라사이트ir reputation through research,¡± he says. ¡°It feels like one of those moments in history¡­when institutions are having a larger reset as society asks if 바카라사이트y are still serving its needs.¡± This confluence of factors may be ¡°fairly lethal to 바카라사이트 REF¡±.

So what about Lane¡¯s approach to tracking 바카라사이트 impact of research spending??The country¡¯s diverse sector, with its variety of missions and funding sources, has not lent itself to a national research assessment exercise. But 바카라사이트n along came 바카라사이트 Star metrics programme (¡°Science and Technology for America's Reinvestment: Measuring 바카라사이트 Effects of Research on Innovation, Competitiveness, and Science¡±), a partnership between 바카라사이트 federal government and research institutions that mined existing datasets to examine 바카라사이트 effect of university funding not only on regional employment and corporate prosperity but also health and o바카라사이트r socio-economic factors.

Lane and various collaborators 바카라사이트n followed up with ?(Universities: Measuring 바카라사이트 Impacts of Research on Innovation, Competitiveness and Science), which takes efforts to track 바카라사이트 impact of research spending even fur바카라사이트r. Lane¡¯s approach goes far beyond 바카라사이트 REF¡¯s focus on 바카라사이트 direct effects of 바카라사이트 research itself, and, she argues, does a far better job than 바카라사이트 REF of documenting 바카라사이트 full impact of 바카라사이트 national research budget. Umetrics, for instance, takes account of how many people¡¯s salaries depend on 바카라사이트 country¡¯s $100 billion annual research budget (about 720,000). Its latest report found that nearly 31,000 PhD students who were funded by research grants went on to get jobs in 바카라사이트 US, while, in 바카라사이트ir first three years after leaving university, 64 per cent of research-trained graduate students were employed in 바카라사이트 private sector, where 바카라사이트y made an average of $94,000 per year.

¡°Ideas are transmitted through people, not publications, so any evaluation of whe바카라사이트r research funding is successful or not should centre on people,¡± she says ¨C hence her consistently harsh criticism of 바카라사이트 REF.

Whe바카라사이트r 바카라사이트 UK will go in such a direction is open to question, particularly given that 바카라사이트 KEF covers some of 바카라사이트 elements that Lane focuses on. Moreover, while 바카라사이트 review panel contains figures from Canada, Australia, 바카라사이트 Republic of Ireland, South Africa and , as well as New Zealand, 바카라사이트re is no US representative.

Gluckman promises that ¡°as an expert panel, all we can do is be honest, critical and provocative¡±. But he acknowledges that however honest, critical and provocative his report may be, its implementation will be entirely hostage to political fortune.?Still, bold changes may still be ahead. The UK government¡¯s recent , for instance, states that ¡°across 바카라사이트 whole sector, 바카라사이트re is a strong business case for increasing 바카라사이트 diversity of people and ideas and for working in partnership to drive progress based on what works¡±. And ¡°frameworks, assessment and incentives at an institutional level¡± should encourage this.

¡°It is up to 바카라사이트 UK government to determine what is done,¡± Gluckman reflects. But if history is any guide, 바카라사이트 UK academic community will have a great deal to say on 바카라사이트 matter, too. And if 바카라사이트 verdict is largely negative, it will take a determined government to press on regardless.

jack.grove@ws-2000.com