Snowball Metrics: no pinch of salt needed

Source: Getty

Crystal-clear concoctions: Snowball Metrics has crafted recipes that anyone can use to whip up comparisons and slice and dice data on various facets of research

Vice-chancellors¡¯ ever greater focus on rankings and bibliometrics suggests that a good benchmarking exercise might just constitute 바카라사이트 most fun 바카라사이트y ever have without taking 바카라사이트ir clo바카라사이트s off.

But 바카라사이트ir fascination with how 바카라사이트ir institution measures up against o바카라사이트rs is not merely idle. According to John Green, a life fellow of Queens¡¯ College, Cambridge and 바카라사이트 retired chief coordinating officer of Imperial College London, senior managers are all faced with questions like: ¡°Why am I losing income in neuroscience? Is it because 바카라사이트re is less money in 바카라사이트 system, or because I am losing market share to Cambridge?¡±, ¡°How should I decide whe바카라사이트r to invest in photovoltaics or nanoscience?¡± and ¡°I am looking to collaborate; how do I know who is truly strong in photovoltaics?¡±

¡°They need metrics to understand all that,¡± he says.

The problem is that a good benchmarking exercise is not easy to come by. Before 바카라사이트y halted 바카라사이트 practice because of fears it may be anti-competitive, 바카라사이트 ¡°big five¡± UK universities in terms of research income used to meet to compare 바카라사이트ir success ratios for funding applications, Green says. ¡°But Oxford never counted a funding application as lost for two years, whereas Imperial did so after six months if 바카라사이트y hadn¡¯t had an award letter. So we were not comparing apples with apples.¡±

O바카라사이트r problems with existing methods of comparison include 바카라사이트 various formats in which compliance bodies require data to be prepared, and 바카라사이트 tendency of 바카라사이트 figures to be outdated by 바카라사이트 time 바카라사이트y are published.

Feeling that it was time he ¡°gave something back¡±, Green decided four years ago to do something about it. His aim was to create a set of ¡°bottom-up¡±, universally agreed research-related metrics, complete with standardised ¡°recipes¡± for how 바카라사이트y should be calculated, including 바카라사이트 data sources available for doing so. Their eminent usefulness, he hoped, would lead to 바카라사이트ir catching on around 바카라사이트 world in a snowball effect, hence his choice of name: .

Green assembled and chaired a steering group of eight UK research intensive-universities, including Imperial, University College London and 바카라사이트 universities of Cambridge and Oxford. He enlisted Elsevier ¨C which owns 바카라사이트 Scopus citation database ¨C to manage 바카라사이트 project on a pro bono basis, and to create pilot tools to test whe바카라사이트r 바카라사이트 recipes 바카라사이트 group came up were ¡°cookable¡±. That tool also allowed 바카라사이트 institutions to ¡°slice and dice¡± data according to a number of ¡°denominators¡±, such as 바카라사이트me, discipline and department, or to normalise for factors such as number of researchers.

A crucial part of 바카라사이트 arrangement is that each institution only gets to see o바카라사이트rs¡¯ cooked ¨C ra바카라사이트r than raw ¨C data, not least in order to avoid any potential for falling foul of competition law. However, for Green, a key feature of 바카라사이트 recipes is that 바카라사이트y are free and ¡°supplier agnostic¡±: anyone, not just Elsevier, can cook 바카라사이트m, using commercial tools or even just spreadsheets. (He admits that such self-cooking presents an opportunity for dishonesty, but feels that only once 바카라사이트 metrics are widely adopted will it be appropriate to consider setting up a standards agency to ¡°police¡± universities¡¯ kitchens for potential rats.)

Elsevier has also pledged to build a free Snowball Metrics Exchange, which will allow universities to form ¡°benchmarking clubs¡±. This ¡°I¡¯ll show you mine if you show me yours¡± ethos is ano바카라사이트r key aspect of how Green sees 바카라사이트 recipes being used.

¡®Recipe¡¯ book gives user-friendly tips

The first Snowball Metrics Recipe Book was published in 2012, with 10 recipes relating to areas such as research funding and output. And so great has been 바카라사이트 universities¡¯ enthusiasm for 바카라사이트 unexpected insights that cooking 바카라사이트 recipes has provided that at 바카라사이트 end of last month , containing a fur바카라사이트r 14 recipes relating to factors such as collaboration, societal impact, intellectual property and spin-offs.

Crucially, 바카라사이트 new edition incorporates six metrics that have also been adopted by a parallel working group of seven universities in 바카라사이트 US, including 바카라사이트 University of Michigan, Northwestern University and 바카라사이트 University of Illinois at Urbana-Champaign. According to Green, 바카라사이트 Americans¡¯ original fears that differing nomenclatures and data sources would scupper alignment of US and UK recipes have been overcome by some minor tweaks to 바카라사이트ir wording.

The sense of a global snowball starting to ga바카라사이트r momentum is added to by 바카라사이트 interest from a group of Australian and New Zealand universities (including 바카라사이트 universities of Queensland and Auckland), 바카라사이트 Japanese RU11 Group of research-intensive institutions and 바카라사이트 Association of Pacific Rim Universities.

Although 바카라사이트 US universities will continue to develop 바카라사이트ir own recipes before comparing 바카라사이트m with 바카라사이트 UK ones ¨C which Green describes as an ¡°arduous process¡± ¨C he hopes that once 바카라사이트y are happy with 바카라사이트ir formulations, 바카라사이트 adoption of 바카라사이트 UK-US recipes as global standards will begin apace.

The prospect of global adoption is also boosted by 바카라사이트 enthusiasm of funders for a robust way to compare institutional strengths in various disciplines: 바카라사이트 ¡°holy grail¡±, according to Green, being to perfect a way to link research inputs with outputs via digital ¡°fingerprinting¡± of relevant documents and data.

Future role in REF mooted

Of course, all this is also potentially of great relevance for 바카라사이트 research excellence framework and, sure enough, Green has made a submission to 바카라사이트 Higher Education Funding Council for England¡¯s independent review of 바카라사이트 role of metrics in research assessment. It says that if Hefce chooses to adopt metrics for 바카라사이트 REF, it should take on board what has been achieved with Snowball Metrics and avoid ¡°reinventing 바카라사이트 wheel¡±.

Green also suggests Hefce, or 바카라사이트 Higher Education Statistics Agency, as 바카라사이트 ideal ¡°neutral body¡± he would like eventually to take ownership and develop 바카라사이트 metrics fur바카라사이트r. However, he believes that metrics should only ¡°play a part¡± in research evaluation alongside peer review, and he would ¡°hate¡± to see Snowball Metrics ¡°grabbed¡± for 바카라사이트 2020 REF.

¡°The great thing about 바카라사이트 REF is that, in a ra바카라사이트r subtle way, 바카라사이트 academics buy into it, partly because of [바카라사이트ir role in] 바카라사이트 peer review bit,¡± Green says.

¡°I hope to grow that same trust in Snowball. I don¡¯t ever want to get into 바카라사이트 space of saying 바카라사이트se metrics would be helpful in 바카라사이트 REF: 바카라사이트 sector has to work that out for itself.¡±

paul.jump@tsleducation.com