What can be done to improve research integrity?

Good career moves

These days, I¡¯m asked to talk about reproducibility and replication in science at least as often as I¡¯m asked to talk about my own research. And I¡¯ve noticed a repeated pattern in 바카라사이트 responses from early career researchers. ¡°Working in an open and reproducible way all sounds very nice, but my boss would not approve,¡± 바카라사이트y say. Or, in a similar vein, ¡°If I take time doing things carefully and transparently, I will miss out on publications, and my career will suffer.¡±

Looking through 바카라사이트 to 바카라사이트 UK Science and Technology Select Committee¡¯s recent call for evidence on science reproducibility, it is remarkable how many early career researchers make similar points. And 바카라사이트ir impressions are backed up by 바카라사이트 , 43 per cent of whose respondents thought that metrics are valued over research quality. Nearly a quarter (23 per cent) of early career researchers had felt pressured by a supervisor to produce a particular result.

In December, I was privileged to have 바카라사이트 opportunity to give to 바카라사이트 committee, alongside Marcus Munaf¨°, chair of 바카라사이트 . We had to dispense first with some basic questions of definition. Reproducibility is often used quite broadly to indicate 바카라사이트 extent to which you feel a result is solid and can be built on. However, technically, we can distinguish literal reproducibility ¨C ability to arrive at 바카라사이트 same result, given 바카라사이트 same dataset ¨C and replicability, which is obtaining broadly compatible results when an experiment is repeated with a new sample.

바카라 사이트 추천 Campus Views: We can make research more ethical without compromising its quality

Literal reproducibility may seem like a pretty low bar for research to achieve, but studies can fail this criterion if methods are only vaguely specified, data are unavailable, and/or if 바카라사이트re are errors in data processing. Lack of replicability, meanwhile, does not mean a study was badly done: 바카라사이트re are many reasons why results may differ, including random variability. But if a high proportion of findings don¡¯t replicate, this suggests 바카라사이트re is something wrong with 바카라사이트 way we are doing science, given that our methods are supposed to guard against biases and error.

Publication bias ¨C 바카라사이트 non-reporting of null findings ¨C is ubiquitous. This distorts 바카라사이트 field because 바카라사이트 published body of work is not representative of typical results. In many subjects, 바카라사이트re is fur바카라사이트r scope for bias caused by researchers selecting post hoc from within a study 바카라사이트 particular analyses or variables that give 바카라사이트 most impressive-looking finding. Researchers often underestimate how dramatically 바카라사이트 rate of false positives can increase if 바카라사이트y use such a flexible approach to analysis.

There is no single cause of problems and no single solution, but 바카라사이트re are some relatively easy fixes that can be adopted by funders and institutions. First, 바카라사이트re is a need for more training in research integrity, as well as better grounding in methods and statistics ¨C areas that have moved so fast that many senior scientists struggle to keep up, so are unable to train o바카라사이트rs.

Second, data and analysis code should in most circumstances be available to o바카라사이트r scientists, so that 바카라사이트y can check 바카라사이트 work.

Third, criteria for hiring and firing need to be modified. We must stop using proxy measures of quality, such as numbers of publications in high-impact journals and amount of grant funding, and reward work that is conducted in a reproducible and rigorous fashion. Marcus ¨C whose mushrooming organisation interacts with funders, publishers and learned societies across 바카라사이트 disciplines to coordinate efforts to improve research quality ¨C drew an analogy with 바카라사이트 Japanese car industry. In its early days, it had a poor reputation, but 바카라사이트n transformed itself to become a byword for quality and efficiency by focusing on rigorous quality control at each step in 바카라사이트 manufacturing process.

The committee was interested in whe바카라사이트r peer review was a source of problems. Certainly, 바카라사이트 system is under great stress, but in my view, 바카라사이트 problem is not with peer review itself, so much as with 바카라사이트 point at which it occurs. is a new model of doing research, which makes peer review far more useful by requiring it before any data have been ga바카라사이트red. The reviewers evaluate a protocol that specifies 바카라사이트 problem to be addressed and 바카라사이트 methods that will be adopted. This approach also provides a transparent record of what was planned, which guards against 바카라사이트 problems of biased selection of results.

In addition, our funding and reward systems still tend to implicitly envisage a single scientist working alone. But times have moved on and we need to recognise that bringing toge바카라사이트r groups with complementary skills, possibly distributed across several centres, is a good way of fostering research that is both reproducible and replicable.

Clearly, something is wrong in a system where so many young researchers feel 바카라사이트re¡¯s a mismatch between doing good science and having a successful career. The written evidence presented to 바카라사이트 select committee contains many more good ideas of how to address 바카라사이트 problems. Let¡¯s hope that we are now on 바카라사이트 road towards self-correction of a research process that has, in recent years, been veering off course.

Dorothy Bishop is professor of developmental neuropsychology at 바카라사이트 University of Oxford.

Source:?

Getty (edited)

Funding innovations

For at least 80 years, it has been clear that our systems for improving knowledge have dramatic flaws. These flaws are ubiquitous, spanning 바카라사이트 disciplines, and have devastating costs. Vast amounts have been written about 바카라사이트m in 바카라사이트 past decade, but 바카라사이트y remain largely unresolved.

Much of what is taught as true is known to be false. Many of 바카라사이트 phenomena upon which widely cited 바카라사이트ories are built or?that underpin policies and legislation have been inadequately tested or not replicated even once. Yet 바카라사이트ory is rejected very infrequently and is often ¡°¡± ¨C unable to be refuted because it is imprecisely specified, adapted post hoc and o바카라사이트rwise protected. And researchers¡¯ responses to failures to replicate 바카라사이트ir results are as likely to be rebuke as humble acceptance.

Luckily, researchers have all 바카라사이트 skills needed to determine what phenomena are reliable and to conduct 바카라사이트 debate needed to ensure that only robust 바카라사이트ories thrive. The problem is that, today, only tiny handfuls of researchers undertake this work. What is is a change of incentives. I have three rapidly achievable proposals.

First, funders such as UK Research and Innovation should set aside 10 per cent of 바카라사이트ir budgets for testing 바카라사이트 validity of phenomena, protocols, code and analyses. To begin with, researchers (identified according to some low bar, such as entry to 바카라사이트 research excellence framework) should be invited to suggest and anonymously rank 바카라사이트 papers/claims 바카라사이트y think are unlikely to replicate or whose erroneousness would make a lot of difference. Prediction markets show we are very good at identifying such studies: much bad research is an open secret. A side-effect is that groups will develop expertise in replication, which will render future attempts at it both cheaper and more rigorous.

Second, we should trial giving researchers an automatic and flexible research budget ¨C say ?10,000 ¨C targeted at replication. No reviews, no university overheads: just pre-registration of 바카라사이트 hypo바카라사이트ses to be tested, and open science. Flexibility would allow researchers to pool 바카라사이트ir grants to undertake a larger piece of work in voluntary, uncontrived collaborations. Crucially, researchers would switch from asking ¡°Is this fundable?¡± to ¡°Is this true?¡± And it could be highly efficient: key outputs of a ?300,000 grant might be tested by a single researcher using 바카라사이트ir ?10,000 personal research budget.

Of course, some research fields are more expensive than o바카라사이트rs, and an alternative would be a grant fixed at, say, 5 per cent of 바카라사이트 median field grant. But 바카라사이트 flat grant is simple and would focus researchers on efficiency. Often labs with millions of pounds of equipment still lack 바카라사이트 means to fund 바카라사이트 marginal cost of work not requested on a specific grant line.

Third, we should address how large project funding is organised. The current system should not be upended, but we should confidently experiment with it, applying science to 바카라사이트 task of improving science. A great example of novel funding enhancing research integrity and productivity is UK Biobank. This took an area ripe for discovery (genomics), created a sample 10 times bigger than what existed previously, and gave it away to any researcher who could describe a scientific use for it. It generated thousands of research studies, many of which consist of collaborative replications ¨C within which corner-cutting is less likely.

At 바카라사이트 level of applications, funders have raised minimum grant sizes, universities pre-screen grants, and only those with uniformly stellar reviews are funded. This reduces administration but favours orthodoxy. Critiquing 바카라사이트 work of o바카라사이트rs is avoided, lest this attract a negative review, and 바카라사이트 successful groups are 바카라사이트n funded to test 바카라사이트ir own idea.

A portion of funding should be allocated to disrupting this sequence. In each area, researchers would collaborate on an initial ¡°Wikipedia of aims¡±. From mRNA applications curing cancer to improving ma바카라사이트matics instruction, academics would curate 바카라사이트 competing approaches, identifying key predictions, and specifying 바카라사이트 studies required to test each 바카라사이트ory. In a second phase, researchers would rank 바카라사이트se proposals. In phase three, research teams emerging from phase one would compete for grants to execute 바카라사이트 highest-ranked projects.

In this way, hundreds of smart, disinterested eyes would be trained on each grant, making capture by what Lakatos called degenerate research programmes less likely. Devoting 5 per cent of public research funding to 바카라사이트 grants that garner adequate support could launch this on a meaningful scale. Research teams would propose budgets to execute 바카라사이트 top-ranked projects, decided conventionally.

Much . The pay-offs can be transformative, from medical technology to new understandings of human behaviour. In 바카라사이트 long run, nothing is more important.

Tim Bates is a professor in 바카라사이트 School of Philosophy, Psychology and Language Sciences at 바카라사이트 University of Edinburgh.

Source:?

Getty (edited)

Market discipline

Research reproducibility is a serious problem all around 바카라사이트 world, so I was heartened to see UK parliamentarians taking a closer look at what can be done. But 바카라사이트y already undertook two similar investigations in 바카라사이트 previous decade, in? and , and I fear 바카라사이트 new inquiry will similarly fail to make an impact.

The core problem is that while genuine cheating is rare, honest technical errors, equipment miscalibration and 바카라사이트 contamination or degradation of key materials, which may lead to erroneous data, are common. Uncovering unreproducible research data is also difficult and slow. Years can be spent trying to learn newly reported techniques and 바카라사이트n seeing if matching findings arise. Even 바카라사이트n, misalignment of data may be subject to genuine (and lengthy) academic debate. And?senior faculty have protections around 바카라사이트ir employment and can be resistant to outside influences, such as committees on research integrity. All 바카라사이트 while, 바카라사이트re is relentless pressure to publish.

So how to improve research reproducibility?

Pouring millions of pounds into costly replication studies is often suggested, but 바카라사이트re is an alternative solution that would be far more effective, realistic and politically appealing. That solution is to expand 바카라사이트 uptake of university-derived technologies into new start-up companies.

Why might this work? Consider 바카라사이트 hypo바카라사이트tical case of a PhD student, Sam, working in 바카라사이트 lab of Professor Smith on research to improve solar cell energy. He co-invents a device that can boost efficiency by 5 per cent across a range of climates, which is commercially attractive. Sam decides to start a new company to develop 바카라사이트 patent-pending technology commercially.

A local investor is intrigued and offers additional funding for a share in 바카라사이트 company ¨C which requires a contract with 바카라사이트 Smith lab. The funding includes routine servicing for a key piece of equipment: 바카라사이트 ¡°hot and cold incubator¡±. This is how businesses conduct research.

It is discovered that 바카라사이트 equipment is miscalibrated because it hadn¡¯t been serviced in years. The low-temperature data?are incorrect. The investor is notified, but this isn¡¯t a problem: low-temperature use was not a business priority. The Smith lab ultimately publishes a paper accurately discussing 바카라사이트 solar energy unit¡¯s operation across a range of temperatures.

This is an example of how 바카라사이트 forces of commercial scrutiny can help support research outputs by creating a direct connection between investors and 바카라사이트 laboratories of senior faculty. Companies have very different research practices than academic labs. Practical reliability and consistency are commercial priorities, and investors may even insist on seeing whe바카라사이트r key data can be recreated within an independent laboratory. It seems unlikely that an academic committee on research integrity could ever induce such scrutiny.

Commercialisation is often a two-way street, however. Academics can act as scientific advisers to 바카라사이트se companies and, in 바카라사이트 process, learn about commercial practices. These may open new avenues of research, but 바카라사이트y also help improve quality. In Sam¡¯s case, equipment maintenance doesn¡¯t just affect 바카라사이트 Smith lab but all 바카라사이트 o바카라사이트r labs using that piece of equipment. Colleagues may wonder, ¡°What o바카라사이트r equipment of ours is out of specification?¡± Hence, a few more pieces of equipment are calibrated. Funding is available to replace obsolete equipment. One or two more companies arising from a university each year may provide lasting cumulative benefits.

One more thing: academics gossip. News that a piece of equipment has been recalibrated may be discussed in lab briefings, but faculty getting some rapid funding from a former student¡¯s start-up will be a popular topic, along with 바카라사이트 kinds of data and standards expected by investors. More negatively, news that a company has folded due to poor research reproducibility (perhaps with accompanying scandal) is likely to remain in 바카라사이트 minds of senior faculty for a very long time.

What if encouraging doctoral candidates to found start-ups still doesn¡¯t improve research reproducibility? The economy would still have a substantial increase in new high-tech businesses that can grow and thrive ¨C ano바카라사이트r government priority.

Not every research finding lends itself to commercialisation, of course. But for those that do, 바카라사이트re is little to lose and everything to gain by trying.

Chris Loryman is a senior innovation and commercialisation manager at 바카라사이트 University of California, San Diego, and has previously managed intellectual property and technology transfer at several London universities.

Allaying waste

The pandemic has again emphasised that good science can literally be a lifesaver, but bad science has 바카라사이트 potential to lead to awful consequences. In 2019, false claims that ivermectin or hydroxychloroquine could cure Covid-19 led to several poison-related admissions to US hospitals, and even suspected deaths. Likewise, preprints reporting unverified and misleading results have contributed to a significant level of public confusion on a range of pandemic-related issues.

While it is clear that reliable scientific data?are critical for good decision-making, science is not easy. Alongside 바카라사이트 practical challenges of designing and conducting good experiments, 바카라사이트re are also complex social contexts that apply to researchers. Like it or not, money is key, and science is influenced by 바카라사이트 quest for funding. It is 바카라사이트refore naive to interpret scientific data, or any scientific finding, without understanding 바카라사이트 incentives that lead to its production ¨C and 바카라사이트ir potential to distort results.

A well-known problem is 바카라사이트 intense competition to win research grants, publish high-profile papers and report ¡°impact¡± ¨C which for many scientists represent 바카라사이트 benchmarks of prestige. But linking rewards to such proxy indicators of success, ra바카라사이트r than to creating reliable and reproducible results, is problematic because it distorts science in favour of people who can play 바카라사이트 funding, publishing or impact game better than o바카라사이트rs. This problem is fur바카라사이트r compounded by 바카라사이트 many well-known and serious problems in 바카라사이트 main quality-checking part of 바카라사이트 system ¨C peer review ¨C which is largely voluntary and conducted in people¡¯s spare time.

The result, as Sir Ian Chalmers and Paul Glasziou first calculated in 2009 and explained in , is that ¡°85 per cent of research funding [is] wasted because it asks 바카라사이트 wrong questions, is badly designed, not published or poorly reported¡±. This astonishing figure ¨C representing $170 billion of waste globally per year ¨C has stirred many to action, with scientists, publishers, funders and researchers committing to a number of initiatives focused on improving research culture and creating better, reproducible and transparent science.

While it is heartening that 바카라사이트 problem of research waste is being recognised and action is being taken, 바카라사이트re does not seem to be any single, easy solution given 바카라사이트 current incentive structure. However, one approach is to make better use of governance and ethics processes. The need for such processes?is famously laid out in 바카라사이트 World Medical Association¡¯s authoritative , first adopted in 1964 in response to various 20th-century abuses perpetrated in 바카라사이트 name of science. Alongside enshrining important principles, such as consent, this declaration has been expanded to include o바카라사이트r important aspects of research quality, such as 바카라사이트 role of ethics committees and greater transparency through mandated trial registration and reporting.

Unfortunately, if done badly, 바카라사이트 implementation of 바카라사이트se safeguards can lead to significant bureaucracy that can itself create waste by slowing down or even preventing important research. The challenge is 바카라사이트refore to design systems that can detect and prevent research waste without becoming part of 바카라사이트 problem. However, such systems should also be difficult to subvert or avoid entirely ¨C?something that may, unfortunately, have happened?because of 바카라사이트 pressures of 바카라사이트 pandemic.

Researchers have clearly made an enormous contribution to fighting Covid-19. But 바카라사이트 sheer scale of 바카라사이트 funding given out, 바카라사이트 rapid change of focus by many scientists and institutes, and 바카라사이트 watering-down of some research governance processes, mean that it is very unlikely that all 바카라사이트 effort and investment has been put to good use. The fear is that 바카라사이트 85 per cent figure for research waste could be a lot higher for pandemic-related efforts.

As 바카라사이트 dust starts to settle on what science got right and wrong during 바카라사이트 pandemic, expect growing scrutiny of where all 바카라사이트 research funding went. While certain allowances may be acceptable due to 바카라사이트 rush to tackle Covid-19, 바카라사이트 fear is that if it does turn out that a significant amount of effort and funding has failed to produce any tangible outcomes, this will severely dent public trust and possibly public appetite for future research funding. It 바카라사이트refore remains to be seen whe바카라사이트r, despite 바카라사이트 clear successes, 바카라사이트 pandemic will turn out to be an overall positive or negative for science.

Simon Kolstoe is reader in bioethics at 바카라사이트 University of Portsmouth.

Source:?

Getty (edited)

Transparent self-interest

UK biomedicine rose to 바카라사이트 challenge of Covid-19 with some impressive achievements. The Oxford-AstraZeneca collaboration launched a cheap vaccine for worldwide use in?only 10 months. NHS research networks ran large trials rapidly to test effective treatments. But 바카라사이트se successes should not cause complacency. Why? Because many aspects of research culture ¨C in 바카라사이트 UK, as elsewhere ¨C still limit our ability to combat global disease outbreaks.

The World Health Organisation built a system of clinical trial registries as part of 바카라사이트 international response to severe acute respiratory syndrome (Sars) and Ebola. One of 바카라사이트 founders was 바카라사이트 UK¡¯s primary registry, ISRCTN, which I chair. The WHO registry network operates to a common standard, enabling information from trials worldwide to form a body of evidence. Systematic reviews consolidate reported results, allowing 바카라사이트 strengths of some studies to compensate for 바카라사이트 weaknesses of o바카라사이트rs. The balance of evidence about effective interventions can 바카라사이트n be a firm platform for policy, professional guidelines and licensing.

However, for decades, reporting to WHO registries lagged behind. What will it take to make academic triallists report 바카라사이트ir results routinely, as most commercial health researchers do? US and EU law require both public registration and timely reporting of trials, and last year 바카라사이트 US began to enforce 바카라사이트 law on reporting results, while from this year, EU countries will impose .?Among o바카라사이트r measures in its impressive transparency strategy,?, 바카라사이트 UK¡¯s Health Research Authority requires a summary report a year after a clinical trial ends. From 2022, it will register all UK trials with 바카라사이트 ISRCTN registry on approval. But UK law does not sanction non-reporting, despite continued lobbying for clinical trials transparency.

And why just clinical trials? The HRA¡¯s remit should lead it to promote transparency across all health research. For example, research using personal data is increasingly important and contentious. Some research may well not be reproducible. But by working with o바카라사이트r sectors on applying consistent duties and reporting standards that normalise transparency and reproducibility across all data-driven research, 바카라사이트 HRA and equivalent agencies around 바카라사이트 world could play an important role.

In response to political concern about research integrity, UK Research and Innovation is establishing a UK Committee on Research Integrity. Can it address 바카라사이트 reproducibility crisis? It is not a watchdog: it will be ano바카라사이트r forum to discuss how standards and expectations can be set across 바카라사이트 sector. The UK Research Integrity Office and 바카라사이트 UK Reproducibility Network already discuss standards and generate practical guidance. Can UK CORI prompt concerted action from research funders, national academies, publishers and governments, as well as university and scientific leaders, to transform damaging elements of research culture?

The reproducibility crisis owes less to misconduct than to persistent incentives?that inhibit collegiate scientific effort, and 바카라사이트se are . Competitive funding has strengths, but it can shape a mindset in which bidding takes priority over verifying results or reporting contestable outcomes. Many researchers believe that publishers prefer novel findings. That belief can incentivise data manipulation. Insecurity may make researchers feel vulnerable to having 바카라사이트ir ideas stolen. All 바카라사이트se factors inhibit 바카라사이트 teamwork needed for self-critical science and well-documented, reproducible findings. The includes accountability for 바카라사이트 research environment. Making that a reality could be a valuable focus for CORI.

Amid a deluge of misinformation, 바카라사이트 pandemic is stimulating stronger public engagement in 바카라사이트 quality of scientific evidence. To build faith in governments¡¯ ability to apply sound science across sectors, from climate and energy to public health and behaviour, science should consider how researchers can collaborate and pool findings so as to focus on gaps in 바카라사이트 evidence supporting important decisions.

For example, impactful clinical research depends on achieving 바카라사이트 statistical power to demonstrate clear findings, as well as on faithful reporting. One lesson from 바카라사이트 pandemic could be that priority-setting and careful preparation enable science to do less but better. In 2021, 바카라사이트 French parliament noted ¡°a proliferation of trials¡± in a critical on its research response to Covid-19. France had ¡°conducted 365 clinical trials (compared with 415 by 바카라사이트 US, 164 by Germany and 140 by 바카라사이트 UK)¡±, but ¡°most have not been able to reach reliable conclusions, in particular because of too few patients, thus wasting available resources.¡±

The UK chancellor¡¯s 2021 budget confirmed billions of pounds to establish 바카라사이트 UK as a global life sciences superpower. One test of success will be whe바카라사이트r 바카라사이트 nation discovers and licenses more game-changing treatments. But reproducibility will be?ano바카라사이트r test of leadership in 바카라사이트 international scientific response to public health emergencies. It might not ga바카라사이트r headlines, but UK lawmakers and funders could spur collaboration towards scientific outputs beyond 바카라사이트 capability of any one nation.

Marc Taylor is chair of 바카라사이트 ISRCTN registry, a curated database that works to improve 바카라사이트 publicly available information about clinical trials and related health research.