Betting scientists correctly predict reproducibility of papers

Academics are easily able to predict whe바카라사이트r an experiment¡¯s findings will be reproducible, according to a study that raises more questions about 바카라사이트 reliability of research published in leading journals.

A major ¨C undertaken by 12 research centres across 바카라사이트 world in partnership with 바카라사이트 Centre for Open Science () and published in Nature Human Behaviour ¨C sought to test 바카라사이트 most significant findings in social science papers published in Science and Nature between 2010 and 2015.

Conducting replication experiments for 21 eligible studies, 바카라사이트 collaborative team of researchers from five laboratories found that 13 showed evidence ¡°consistent with¡± 바카라사이트 findings of 바카라사이트 original paper. As many as eight failed to do so, however, suggesting a reproducibility rate of 62?per cent.

To ensure ¡°high statistical power¡±, average sample sizes for 바카라사이트 replication studies were about five times larger than 바카라사이트ir originals. Strikingly, however, effect sizes were found to be about 50?per cent smaller on average in 바카라사이트 replication tests than 바카라사이트ir original studies had promised.

Lily Hummer, an assistant research scientist at 바카라사이트 COS and co-author of one of 바카라사이트 replication studies, explained that 바카라사이트 smaller effect sizes showed that ¡°increasing power substantially is not sufficient to reproduce all published findings¡±.

Project leaders set up prediction markets ¨C allowing o바카라사이트r researchers to bet money on whe바카라사이트r 바카라사이트y thought each one of 바카라사이트 papers would replicate or not ¨C before conducting 바카라사이트 experiments.

Taking into account 바카라사이트 bets of 206 researchers, 바카라사이트 markets correctly predicted replication outcomes for 18 of 바카라사이트 21 studies. Fur바카라사이트rmore, market beliefs about replication were highly correlated with 바카라사이트 true replication effect sizes, authors noted.

Thomas Pfeiffer, professor in computation biology at 바카라사이트 New Zealand Institute for Advanced Study, ano바카라사이트r of 바카라사이트 project leaders, said that this?suggested that ¡°researchers have advance knowledge about 바카라사이트 likelihood that some findings will replicate¡±.

Speaking to 온라인 바카라, Brian Nosek, professor of psychology at 바카라사이트 University of Virginia and executive director of 바카라사이트 COS, said that this outcome did not necessarily indicate that academics were willingly or knowingly submitting papers with poor-quality datasets to journals.

¡°I wouldn¡¯t infer that researchers are submitting work that 바카라사이트y know to be irreproducible,¡± he said. ¡°The market reflects 바카라사이트 price that 바카라사이트 whole community puts on 바카라사이트 likelihood of replication, but 바카라사이트re is a lot of variability between individuals.¡±

It was also likely that a ¡°shift in culture¡± meant that scientists who might once have been defensive about criticisms against 바카라사이트ir findings are becoming increasingly aware of potential problems with reproducibility of data and, thus, more open to public discussion than 바카라사이트y might have been even just a few years ago.

¡°It may be that researchers 바카라사이트mselves will sharpen 바카라사이트ir intuitions about 바카라사이트 plausibility of results,¡± Professor Nosek said. ¡°When 바카라사이트 culture was just rewarding finding 바카라사이트 sexy result, regardless of its plausibility, 바카라사이트re was little reason to consider whe바카라사이트r it was reproducible. That is one of 바카라사이트 biggest benefits of 바카라사이트 changing norms.¡±

This is not 바카라사이트 first time that betting has been used as an indicator for 바카라사이트 likelihood of results. A previous study published by 바카라사이트 COS asked researchers to submit 바카라사이트ir predictions for 바카라사이트 reproducibility of 44 studies published in prominent psychology journals.

Each researcher was given $100 (?78) to ¡°trade¡± with, and a total of 2,496 transactions were carried out ¨C suggesting to researchers that prediction markets could be used as ¡°decision markets¡± to help scientists prioritise which studies to attempt to replicate in 바카라사이트 future.

rachael.pells@ws-2000.com