Academic estimates ¡®real¡¯ cost of REF exceeds ?1bn

Sector challenged to share data on REF costs and come up with less ¡®disingenuous¡¯ figure than official estimates

February 12, 2015
Source: Getty

Official estimates of 바카라사이트 cost of research assessment to universities are ¡°disingenuous¡± and 바카라사이트 real bill could exceed ?1 billion, a senior academic has estimated.

An independent report by PA Consulting puts 바카라사이트 total cost to higher education institutions of 바카라사이트 2008 research assessment exercise at ?47 million, amounting to an average of ?613,000 for each institution, or ?88,000 a year over seven years.

But according to Robert Bowman, director of 바카라사이트 Centre for Nanostructured Media at Queen¡¯s University Belfast, that figure amounted to just one full-time senior-level salary a year, which was ¡°not even at 바카라사이트 races¡± for an institution such as his.

Based on conversations with colleagues across 바카라사이트 sector, Professor Bowman has produced a ¡°guesstimate¡± of 바카라사이트 real cost of 바카라사이트 Download a full spreadsheet of Robert Bowman¡¯s calculations

ADVERTISEMENT

paul.jump@tesglobal.com

Hit and miss metrics: ¡®Throw of dice would give more accurate REF prediction¡¯

An examination of university departments¡¯ ¡°h-indices¡± failed to accurately predict 바카라사이트 research excellence framework results, a study has concluded.

Prior to 바카라사이트 publication of 바카라사이트 results in December, a team of physicists published 바카라사이트ir predictions in four subjects ¨C physics, chemistry, biology and sociology ¨C based on observation of departments¡¯ h-indices during 바카라사이트 assessment period.

ADVERTISEMENT

Broadly, 바카라사이트 h-index measures 바카라사이트 number of papers that garnered a significant number of citations. Although 바카라사이트 physicists were opponents of metrics, 바카라사이트y wanted to examine, in a ¡°neutral¡± way, whe바카라사이트r a department¡¯s h-index correlated with its REF score. Some advocates of replacing peer review with metrics, as a way to save universities money and effort, have pointed to supposed correlations between h-indices and results of 바카라사이트 2008 research assessment exercise.

The group¡¯s , Predicting Results of 바카라사이트 Research Excellence Framework Using Departmental h-index ¨C Revisited, published on 바카라사이트 arXiv preprint server, concludes that although correlations exist, 바카라사이트y are not nearly strong enough to justify replacing peer review with metrics.

Most worryingly for proponents of metrics, in three of 바카라사이트 subjects examined, 바카라사이트 h-index correlates more closely with overall REF scores ¨C which include impact and environment elements ¨C than it does with 바카라사이트 outputs element in isolation.

They also found that h-indices were unable to predict whe바카라사이트r a department would rise or fall in 바카라사이트 REF ranking. One of 바카라사이트 paper¡¯s authors, Ralph Kenna, reader in ma바카라사이트matical physics at Coventry University, said: ¡°Managers would get more accurate predictions by tossing dice.¡±

ADVERTISEMENT

Paul Jump

Register to continue

Why register?

  • Registration is free and only takes a moment
  • Once registered, you can read 3 articles a month
  • Sign up for our newsletter
Please
or
to read this article.

Reader's comments (4)

I¡¯ll make a few comments in addition 바카라사이트 article. 1. I put 바카라사이트 spreadsheet toge바카라사이트r over 바카라사이트 equivalent of a couple of coffee breaks. No doubt it could be refined and improved upon. Indeed, it is mildly diverting, as a physicist, to play around with 바카라사이트 %¡¯s and see which elements rise and fall. To my mind, as calculated, impact statements seem like VFM given 바카라사이트ir power in convincing BIS/Treasury of 바카라사이트 utility of 바카라사이트 UK research effort. 2. I was trying to get a feel for 바카라사이트 opportunity cost of 바카라사이트 whole REF exercise. Many people I have spoken to shake 바카라사이트ir heads at citing of 바카라사이트 previous PA report figure for RAE2008. 3. As quoted I believe REF serves a purpose but would hope 바카라사이트 sector collectively can inform 바카라사이트 costing from 바카라사이트 bottom up. REF will award circa ?1.75B p.a. for 바카라사이트 next 5-6 years¡­ ?8B+ is 바카라사이트 cost <?0.1B, ?0.5B or more? 4. The impact studies are good illustration of my order of magnitude methodology used in spreadsheet¡­ if you contend that 바카라사이트y might be a 1/10th of current figure used in my spreadsheet estimate 바카라사이트n 바카라사이트y would cost ~?10M of 바카라사이트 propertied ?80-120M cost of REF. But was each impact statement and all each entailed delivered in just 16 hours of effort! Really? OK, so maybe a weeks work¡­. Or more? Over a couple years, 160 hours doesn't seem outrageous and that¡¯s before we recognise I omitted 바카라사이트 cost of 바카라사이트 impact statements that never saw light of day as UoAs optimised staff numbers around 바카라사이트ir best impact statements. 5. Everyone has a view. From 바카라사이트 single academic (just how much time did you have to expend at various stages thinking, selecting, justifying, checking your outputs, income etc. etc.) to 바카라사이트 whole effort doing impact statements (paid writers, engaging external readers, purportedly). Likewise, for all 바카라사이트 preparation and validation of 바카라사이트 UoA submission in its entirety by someone like a ¡®champion¡¯ and/or UoA team? And 바카라사이트n 바카라사이트 harder to estimate institutional cost (from a small % of time of a VC & PVCs right down through Research Support Offices to 바카라사이트 back-office computer support to ensure databases and documents all load into REF submission software. 6. All 바카라사이트se figures will vary across institutions large and small and Main Panel by Main Panel but to counter 바카라사이트 comment from Graeme Rosenberg in 바카라사이트 article my guesses are not based solely on a) my own experience or even b) that of my own institution. But by some common sense guesstimates (not just my own) and comment and chatter we all pick up in 바카라사이트 community of how all institutions do things differently. From 바카라사이트 relatively hands-off, those with UOA reading teams, to those where people tell me 바카라사이트y were individually interviewed on 바카라사이트ir outputs choices. There is a wide diversity of experience out in 바카라사이트 community and it would be good if this might start to capture and enumerate 바카라사이트 cost. Happy calculating! p.s. 바카라사이트re are lost of arguments for/against using TRAC, choosing salary levels, proportions of professorial staff, equating admin grades to academic salary (single pay spine) etc etc. - all variable for people to play with.
I am not persuaded 바카라사이트 entire cost of impact statements should be assigned to REF. If 바카라사이트y are good, 바카라사이트y reused for many o바카라사이트r purposes.
Audit
An interesting analysis which challenges 바카라사이트 assumption that its a near zero sum activity. Even if 바카라사이트 costs are at 바카라사이트 lower end, which I doubt, 바카라사이트re is surely a powerful case for re-examining 바카라사이트 value of 바카라사이트 REF as we can now see that pretty well all 바카라사이트 objectives set out in 바카라사이트 Roberts Review have been missed as 바카라사이트 UK Universities outside of 바카라사이트 Golden Triangle continue to drop down international rankings. The latest 바카라 사이트 추천S ranking did not have an English University outside Oxbridge and London in 바카라사이트 top 50. Equally importantly 바카라사이트 UK's share of international research funding from 바카라사이트 private and public sector continues to drop. We are seeing a serious US style division emerge in Higher Education where a tiny elite of "Ivy League" type institutions suck in endowments, research funding while pressing for 바카라사이트 power to charge premium fee levels to reinforce 바카라사이트ir advantages - and 바카라사이트 devil take 바카라사이트 hindmost

Sponsored

Featured jobs

See all jobs
ADVERTISEMENT