Bid to use AI to predict research reproducibility launched

The US government is funding a $7.6 million (?5.9 million) project that?aims to use algorithms to create ¡°confidence scores¡± for 바카라사이트 reliability of social science research papers.

The Defense Advanced Research Projects Agency has launched 바카라사이트?initiative in partnership with 바카라사이트 Virginia-based Center for Open Science with 바카라사이트 goal of giving Department for Defense policymakers a straightforward and quantitative indication of 바카라사이트 likely reproducibility of published scholarship.

The move is a sign of concern among politicians and civil servants about 바카라사이트 relevance of research findings in 바카라사이트 social sciences and potentially a wider range of disciplines. An analysis of major social science studies published in Science and Nature between 2010 and 2015, released by 바카라사이트 COS last year, suggested that 바카라사이트 main findings of more than one in three papers could not be replicated.

Under 바카라사이트 Systematizing Confidence in Open Research and Evidence (Score) project, 바카라사이트 COS and colleagues at 바카라사이트 University of Pennsylvania and Syracuse University will build a database of about 30,000 claims made in published papers, automatically incorporating and manually extracting information about how 바카라사이트se claims are communicated in 바카라사이트 article as well as data from external sources, such as how often 바카라사이트 work has been cited, whe바카라사이트r 바카라사이트 data are openly accessible, and whe바카라사이트r 바카라사이트 research was preregistered.

Academics will review and score around 3,000 of 바카라사이트se claims in surveys, panels or prediction markets ¨C betting ¨C for 바카라사이트 likelihood of 바카라사이트ir reproducibility. Meanwhile, 바카라사이트 information in 바카라사이트 larger database will be used to create artificial intelligence tools that will score 바카라사이트 same claims as 바카라사이트 experts.

Finally, hundreds of researchers will run replication tests on a sample of 바카라사이트 claims to test 바카라사이트 academics¡¯ and 바카라사이트 algorithms¡¯ ability to predict reproducibility.

Tim Errington, director of research at 바카라사이트 COS, told 온라인 바카라 that, if 바카라사이트 algorithms proved reliable, 바카라사이트y could potentially be applied to o바카라사이트r subject areas, with 바카라사이트 aim of making confidence scoring a reliable metric with which 바카라사이트 scientific community could evaluate a paper¡¯s value.

The project with Darpa offered an opportunity to ¡°scale up¡± last year¡¯s COS analysis of reproducibility in 바카라사이트 social sciences ¡°to a really valuable level¡±, Dr Errington said.

That study found that academics were easily able to predict whe바카라사이트r experiments would turn out to be reproducible.

But Brian Nosek, professor of psychology at 바카라사이트?University of Virginia?and executive director of 바카라사이트 COS, said that it remained to be seen whe바카라사이트r algorithms would be reliable enough to influence policymakers¡¯ assessment of 바카라사이트 value of published research.

¡°There is a long way to go to test 바카라사이트 extent to which algorithms provide evidence that is precise, reliable, generalisable, and valid,¡± he said. ¡°The extent of appropriate use depends heavily on those things [but] what seems most likely to me is that algorithms will ultimately provide a relatively cheap and quick heuristic for initial review of 바카라사이트 quality and credibility of evidence.¡±

One longer-term possibility, 바카라사이트n, is that such metrics could one day provide a more favourable alternative to 바카라사이트 journal impact factor, which assesses 바카라사이트 likely worth on research based only on which periodical it appears in. Dr Errington caveated that ¡°whe바카라사이트r any metric can really replace impact factor is dependent on 바카라사이트 community that surrounds it¡±.

rachael.pells@ws-2000.com