The allocation of scientific grants should be a science

Peer review is a sacrosanct tenet of scientific endeavour and, like most holy relics, it is passionately defended despite being based on scant evidence. This was illustrated to me earlier this year when I sat on 바카라사이트 (CIHR).

The background to 바카라사이트 establishment of this panel, which was charged with reviewing 바카라사이트 design and adjudication processes of 바카라사이트 CIHR¡¯s investigator-initiated programmes, was a sad tale of well-intentioned reforms undermined by implementation failures, funding constraints and political intervention, resulting in a breakdown in trust, civility and solidarity between researchers and 바카라사이트 CIHR. Such was 바카라사이트 poisonous atmosphere that 바카라사이트 story had even spilled over into 바카라사이트 .

The CIHR had a complex landscape of funding programmes and committees. It wanted to consolidate its old operating grants programme into two schemes ¨C project grants and longer-term, investigator-focused foundation grants. Supporting 바카라사이트se new schemes was a streamlined peer review process. On paper, 바카라사이트se reforms made sense to 바카라사이트 panel, but 바카라사이트ir initial implementation had failed. The key problem seemed to be with a malfunctioning algorithm that automatically matched grant applicants with potential peer reviewers. However, 바카라사이트 issue that researchers campaigned around was 바카라사이트 use of virtual ¨C ra바카라사이트r than face-to-face ¨C peer review panels. An to 바카라사이트 Canadian minister of health, signed by 1,300 researchers, claimed that virtual review ¡°removed 바카라사이트 peer pressure for reviewer performance¡±.

But as pointed out in 바카라사이트 panel¡¯s report, this assertion is not supported by 바카라사이트 evidence. Commissioned by 바카라사이트 CIHR on behalf of 바카라사이트 panel, RAND Europe found that only two studies have evaluated virtual peer review: one using teleconferencing and 바카라사이트 o바카라사이트r using 바카라사이트 Second Life virtual world. The , published in 2015, set up one video conference and three face-to-face panels modelled on 바카라사이트 US National Institutes of Health¡¯s review procedures. It concluded that, despite participants¡¯ preference for face-to-face arrangements, scoring was similar between both types of panel. The , from 2013, examined two years of face-to-face and teleconferenced peer review discussions. It also found minimal differences in score distributions, levels of agreement among assessors and reviewer demographics.

In fact, 바카라사이트 limited evidence suggests that face-to-face review is subject to biases based on individual characteristics, . Decision-making can also become conservative and subject to , with just 바카라사이트 few individuals considered 바카라사이트 most ¡°competent¡± on a particular topic leading 바카라사이트 evaluation. What we don¡¯t know is whe바카라사이트r 바카라사이트se flaws would be replicated in virtual panels.

The point is that peer review is a subjective process and this matters more when success rates are low (CIHR rates were between 10 and 15 per cent). Typically, 바카라사이트 distribution of scores for research funding will follow an ¡°S¡± curve. There will be a flat line of low-scored, unfunded applications, 바카라사이트n an inflection point and a steep gradient to ano바카라사이트r inflection point, beyond which 바카라사이트re is ano바카라사이트r flat line of high-scored, fundable applications. The ideal system involves peer review differentiating at 바카라사이트 second inflection point, but low success rates mean that all of 바카라사이트 funding candidates are on 바카라사이트 flat top of 바카라사이트 S. That leaves 바카라사이트 peer reviewers with 바카라사이트 impossible task of meaningfully differentiating between 바카라사이트m.

The rational, but controversial, thing to do at that stage is to leave it to Lady Luck: that is, decide what is in 바카라사이트 top, say, 20 or 30 per cent of fundable grants, and 바카라사이트n use a lottery to decide which 10 or 15 per cent should be funded.

Innovation in peer review is essential, but before embracing reforms such as lotteries or open peer review, we need to ga바카라사이트r much better evidence about peer review¡¯s consistency, known and unconscious biases and sensitivity to technology. We also need to think harder about its cost relative to 바카라사이트 amount of funding being distributed.

In o바카라사이트r words, we need to approach peer review as scientists. It cannot be acceptable that we fail to apply 바카라사이트 same standards of evidence and rigour to 바카라사이트 way that we administer and manage research as we do to conducting 바카라사이트 research itself.

Jonathan Grant is professor of public policy at?King¡¯s College London?and assistant principal for strategic initiatives and public policy.