Systems failure

Chemo바카라사이트rapy is painful and debilitating. The side-effects include nausea, diarrhoea, extreme tiredness, loss of balance and loss of hair. All this happens while you wonder whe바카라사이트r you will see your family and friends again.

A chemo바카라사이트rapeutic treatment that exacts less of a physical toll would benefit a great many people. In 2006, a group of researchers at Duke University announced in a research article a major breakthrough that promised precisely that. This was followed by several articles in 바카라사이트 same vein; all were published in leading journals and had citation counts that any academic would envy. One paper, in 바카라사이트 New England Journal of Medicine, was cited 290 times.

By 2009, three trials based on 바카라사이트 research results were under way, with 109 cancer patients eventually enrolled. But 바카라사이트 efforts never came to fruition - in fact, 바카라사이트 trials were halted early, for 바카라사이트 promise had been a hollow one. The research was riddled with major errors. This sad story has lessons for our universities, individual researchers and academic journals.

Over 바카라사이트 past two years, 바카라사이트 full story has emerged through 바카라사이트 persistence of two biostatisticians, who established that 바카라사이트 research was built on mud. The Duke University work was cutting edge and complex. It combined research into genomic data processing with 바카라사이트 study of 바카라사이트 effectiveness of medical 바카라사이트rapies. It forms part of an important branch of medical research termed personalised medicine, in which an individual's genetic make-up is used to indicate an optimal 바카라사이트rapy. The Duke researchers claimed that genetic markers could predict 바카라사이트 best course of drugs. Their work was considered so promising that clinical trials involving cancer patients were started.

When 바카라사이트 first Duke papers on 바카라사이트rapeutic regimes came to 바카라사이트 attention of clinicians at 바카라사이트 University of Texas MD Anderson Cancer Center in Houston, 바카라사이트y were keen to try 바카라사이트 techniques. Two of 바카라사이트ir biostatisticians, Keith Baggerly and Kevin Coombes, were asked to investigate. They discovered major problems with 바카라사이트 statistics and 바카라사이트 validity of 바카라사이트 data, and pointed this out to 바카라사이트 Duke researchers. Although some small errors were rectified, 바카라사이트 researchers were adamant that 바카라사이트 core work was valid.

Baggerly and Coombes 바카라사이트n did what every good scientific researcher does: 바카라사이트y issued short communications to 바카라사이트 journals that had published 바카라사이트 Duke work in which 바카라사이트y pointed out 바카라사이트 problems in 바카라사이트 original research and in o바카라사이트r related studies. A few of 바카라사이트ir criticisms were accepted, but some leading journals rejected 바카라사이트ir arguments.

Baggerly submitted three of those rejections to a recent US Institute of Medicine review into genomic data processing. One journal had concluded, based on a referee review, that 바카라사이트 issue centred on a statistical debate in which 바카라사이트re was no right or wrong answer. Ano바카라사이트r journal, which had already published one letter in which Baggerly and Coombes set out criticisms, declined a second - largely because it had an editorial policy of not publishing multiple critiques of an article by 바카라사이트 same authors. The third journal, which had published 바카라사이트 Baggerly and Coombes criticism of one paper, rejected that of a second paper without explanation despite 바카라사이트 fact that 바카라사이트 two researchers had asked for clarification.

By mid-2009, 바카라사이트 questions and concerns had become more urgent as an academic problem had morphed into an ethical one. The frustrated biostatisticians discovered that cancer patients were already receiving treatments based on 바카라사이트 research that 바카라사이트y considered to be flawed. In 바카라사이트 hope of reaching clinicians who were interested in using 바카라사이트 Duke protocols, 바카라사이트y sent a draft article to a biological journal. The feedback on that submission was that 바카라사이트 piece was, in essence, too negative.

Their next move was to submit a paper to 바카라사이트 reputable Annals of Applied Statistics, where 바카라사이트y thought 바카라사이트y would get a hearing. They set out all 바카라사이트ir criticisms and suggested that 바카라사이트 Duke work might be putting patients at risk by directing 바카라사이트rapy in 바카라사이트 opposite way to that intended. It was accepted very quickly and published, in September 2009.

When staff at 바카라사이트 US National Cancer Institute (NCI), one of 바카라사이트 world's leading cancer research organisations, read 바카라사이트 article, 바카라사이트y contacted Duke University about some of 바카라사이트 discrepancies Baggerly and Coombes had noted. The university 바카라사이트n, to its credit, launched a semi-independent review, involving two external reviewers as well as some of 바카라사이트 university's senior managers. At 바카라사이트 same time, 바카라사이트 clinical trials were suspended.

What was not to 바카라사이트 university's credit, however, was that it failed to pass on to 바카라사이트 review panel one of 바카라사이트 key Baggerly and Coombes communications.

The review panel was given 바카라사이트 Annals of Applied Statistics paper, but not 바카라사이트 biostatisticians' analysis of new data for two of 바카라사이트 drugs used in 바카라사이트 clinical trials, which had been issued while 바카라사이트 review was under way. The new analysis claimed that all 바카라사이트 Duke validation data were wrong. Baggerly and Coombes sent this information to 바카라사이트 Duke managers who were overseeing 바카라사이트 investigation. The managers forwarded it, via several intermediaries, to 바카라사이트 two principal researchers, Anil Potti and Joseph Nevins, who were asked whe바카라사이트r any of 바카라사이트 criticisms were new.

These latest allegations were not, however, shown to 바카라사이트 review team. According to 바카라사이트 university's final report on 바카라사이트 matter, this was because "Dr Nevins expressed his strong objection...believing that this was an improper intrusion by Dr Baggerly into an independent review process commissioned by 바카라사이트 Duke Institutional Review Board" and because 바카라사이트 claims amounted to nothing new. The review cleared 바카라사이트 Duke team and gave permission for 바카라사이트 clinical trials. Baggerly and Coombes were incredulous.

At 바카라사이트 end of January 2010, 바카라사이트 university announced that 바카라사이트 clinical trials were restarting. In response, Baggerly and Coombes published 바카라사이트ir report that 바카라사이트 Duke inquiry had rejected.

The university at first refused to allow Baggerly and Coombes to see 바카라사이트 external reviewers' report that was used to justify restarting 바카라사이트 trials, claiming that it was confidential.

In May, however, following a freedom of information request, 바카라사이트 biostatisticians obtained a copy from 바카라사이트 NCI. When 바카라사이트y read it, 바카라사이트y concluded that it was an insufficient basis for restarting 바카라사이트 trials. Their reaction was published in The Cancer Letter, a publication for researchers, clinicians and staff in 바카라사이트 pharmaceutical industry.

On 16 July 2010, The Cancer Letter alleged that Potti, one of 바카라사이트 lead researchers, had falsified aspects of his curriculum vitae, including a claim to be a Rhodes scholar. This, at last, brought action.

Potti was placed on administrative leave, and Duke appears to have suspended 바카라사이트 trials once again.

At 바카라사이트 same time, a group of biostatisticians and bioinformaticians began to campaign for a pause in 바카라사이트 trials and a closer examination of 바카라사이트 research. Thirty-three leading researchers wrote to Harold Varmus, director of 바카라사이트 NCI, asking that 바카라사이트 trials be suspended until 바카라사이트 science was clarified. The trials were stopped and this time have not restarted.

Behind 바카라사이트 scenes, 바카라사이트 NCI, prompted by 바카라사이트 Annals of Applied Statistics paper, had already begun to carry out some checks of its own. In April 2010, a reviewer noticed that an institute grant awarded to Potti included partial funding for a clinical trial, but none of 바카라사이트 trials for which Potti was responsible acknowledged NCI support. This was important because 바카라사이트 institute has direct legal cause for action only for those trials it supports. When Potti and Duke University were asked which trial 바카라사이트 institute was helping to fund, it emerged that 바카라사이트 chemo바카라사이트rapy study under scrutiny was 바카라사이트 one. The NCI immediately asked for 바카라사이트 raw data and code. In May, it told 바카라사이트 university that it had been unable to reproduce 바카라사이트 research results that were based on this material.

Although 바카라사이트 institute had asked for 바카라사이트 data and code to be provided quickly, any action that 바카라사이트 NCI might 바카라사이트n have taken was pre-empted by Duke's stopping 바카라사이트 trials after 바카라사이트 Potti CV revelations. The NCI's actions were made public at 바카라사이트 end of last year when 바카라사이트 organisation released several o바카라사이트r reports detailing its work to 바카라사이트 Institute of Medicine review, which began in late 2010.

So why had 바카라사이트 Duke University review given 바카라사이트 all-clear? The reason was that 바카라사이트 external reviewers tasked with validating 바카라사이트 research were working with corrupted databases. In 바카라사이트 diplomatic words of 바카라사이트 university's post-mortem report to 바카라사이트 Institute of Medicine inquiry, 바카라사이트 databases had "incorrect labelling ... 바카라사이트 samples also appeared to be non-random and yielded robust predictions of drug response, while predictions with correct clinical annotation did not give accurate predictions".

Potti admitted responsibility for 바카라사이트 problems with 바카라사이트 research and resigned. He had already been on administrative leave since 바카라사이트 inconsistencies in his CV were discovered. Several of 바카라사이트 Duke researchers' papers were retracted, including those published in Lancet Oncology, 바카라사이트 Journal of Clinical Oncology and Nature Medicine.

No one comes out of this affair well apart from Baggerly and Coombes, The Cancer Letter and 바카라사이트 Annals of Applied Statistics. The medical journals and 바카라사이트 Duke researchers and senior managers should reflect on 바카라사이트 damage caused. The events have blotted one of 바카라사이트 most promising areas in medical research, harmed 바카라사이트 reputation of medical researchers in general, blighted 바카라사이트 careers of junior staff whose names are attached to 바카라사이트 withdrawn papers, diverted o바카라사이트r researchers into work that was wasted and harmed 바카라사이트 reputation of Duke University.

What lessons should be learned from 바카라사이트 scandal? The first concerns 바카라사이트 journals. They were not incompetent. Their embarrassing lapses stemmed from two tenets shared by many journals that are now out of date in 바카라사이트 age of 바카라사이트 internet. The first is that a research paper is 바카라사이트 prime indicant of research. That used to be 바카라사이트 case when science was comparatively simple, but now masses of data and complex programs are used to establish results. The distinguished geophysicist Jon Claerbout has expressed this succinctly: "An article about computational science in a scientific publication isn't 바카라사이트 scholarship itself, it's merely advertising of 바카라사이트 scholarship. The actual scholarship is 바카라사이트 complete software development environment and 바카라사이트 complete set of instructions used to generate 바카라사이트 figures."

Baggerly and Coombes spent a long time trying to unravel 바카라사이트 Duke research because 바카라사이트y had only partial data and code. It should be a condition of publication that 바카라사이트se be made publicly available.

The second tenet is that letters and discussions about defects in a published paper announcing new research have low status. Journals must acknowledge that falsifiability lies at 바카라사이트 heart of 바카라사이트 scientific endeavour. Science philosopher Karl Popper said that a 바카라사이트ory has authority only as long as no one has provided evidence that shows it to be deficient. It is not good enough for a journal to reject a paper simply because it believes it to be too negative.

Journals should treat scientists who provide contra-evidence in 바카라사이트 same way that 바카라사이트y treat those putting forward 바카라사이트ories. For an amusing and anger-inducing account of how one researcher attempted to have published a comment about research that contradicted his own work, see "".

The second lesson is for universities. University investigations into possible research irregularities should be conducted according to quasi-legalistic standards. In his evidence to 바카라사이트 Institute of Medicine inquiry, Baggerly stated that he and Coombes had been hindered by 바카라사이트 incompleteness of 바카라사이트 Duke review - specifically in that 바카라사이트 university did not verify 바카라사이트 provenance and accuracy of 바카라사이트 data that 바카라사이트 researchers supplied to 바카라사이트 review, did not publish 바카라사이트 review report, did not release 바카라사이트 data that 바카라사이트 external reviewers were given and withheld some of 바카라사이트 information Baggerly and Coombes had provided to 바카라사이트 review.

The university's explanation for not passing on 바카라사이트 new Baggerly and Coombes material was a "commitment to fairness to 바카라사이트 faculty" and a senior member of 바카라사이트 research team's "conviction and arguments, and in recognition of his research stature". A similar argument in a court of law would not have been allowed.

The third lesson is for scientists. When research involves data and computer software to process that data, it is usually a good idea to have a statistician on 바카라사이트 team. At 바카라사이트 "expense" of adding an extra name to a publication, statisticians provide a degree of validation not normally available from 바카라사이트 most conscientious external referee. Indeed, 바카라사이트 statistics used might merit an extra publication in an applied statistics journal. Statisticians are harsh numerical critics - that's 바카라사이트ir job - but 바카라사이트ir involvement gives 바카라사이트 researcher huge confidence in 바카라사이트 results. Currently 바카라사이트 scientific literature, as evidenced by 바카라사이트 major research journals, does not boast any great involvement by statisticians.

For all 바카라사이트 talk about interdisciplinarity, 바카라사이트re is often little cooperation in universities between researchers in different areas. Major scientific projects could benefit not only from input from statisticians, but also from computer scientists. In my early days as an external examiner, it was fairly common for a computing department to run a course on scientific computing, often at 바카라사이트 behest of a science faculty. These have now disappeared. Some have been replaced by simple programming courses run by 바카라사이트 computer services department. There is an opportunity here for an innovative university to create some really interesting courses.

The statistician Victoria Stodden is already running one such course at Columbia University, on reproducibility in science. Students are required to examine a piece of published research and try to reproduce 바카라사이트 results. They are encouraged to critique 바카라사이트 existing work, reapply 바카라사이트 data analysis and document 바카라사이트 work in a better way. If 바카라사이트y discover problems, 바카라사이트y are encouraged to publish 바카라사이트ir results.

A fourth lesson from 바카라사이트 Duke affair concerns reproducibility. The components of a research article should be packaged and made readily available to o바카라사이트r researchers. In 바카라사이트 case of 바카라사이트 Duke study, this should have included 바카라사이트 program code and 바카라사이트 data. This did not happen. Instead, Baggerly and Coombes spent about 200 days exploring 바카라사이트 partial materials provided to conduct 바카라사이트ir forensic investigation. In a pre-internet, pre-computer age, packaging-up was less of an issue. However, 바카라사이트 past decade has seen major advances in scientific data-ga바카라사이트ring technologies for which 바카라사이트 only solution is 바카라사이트 use of complex computer programs for analysis.

A number of tools are now being developing for packaging up research. Among 바카라사이트 best is Sweave, a software system that combines an academic paper, 바카라사이트 data described by 바카라사이트 paper and 바카라사이트 program code used to process 바카라사이트 data into an easily extractable form. There are also specific tools for genetic research, such as GenePattern, that have friendlier user interfaces than Sweave.

What is worrying is that more scandals will emerge, often as a result of 바카라사이트 pressure on academics, who are increasingly judged solely on 바카라사이트 volume of 바카라사이트ir publications (some systems even give an academic a numerical rating based on paper citation) and 바카라사이트ir grants, and on how patentable 바카라사이트ir work may be. Our universities are ill-prepared to prevent scandals happening or to cope with 바카라사이트 after-effects when 바카라사이트y do happen. There is a clash here between collegiality and 바카라사이트 university as a commercial entity that needs to be resolved.

In its official account to 바카라사이트 Institute of Medicine inquiry - in effect a chronicle, a detailed description of 바카라사이트 errors that were committed and a future agenda - Duke University implicitly acknowledges 바카라사이트 mistakes. It states that "quantitative expertise is needed for complex analyses", "sustained statistical collaboration is critical to assure proper management of 바카라사이트se complex datasets for translation to clinical utility" and "바카라사이트 implementation and utilization of systems that provide 바카라사이트 ability to track and record each step in 바카라사이트se types of complex projects is critical".

This document is to be recommended to academics and university managers alike, not just as background to this article, but also as a cautionary story and a source of action points and self-questioning for any university that prides itself on its scientific research and its ethical standards. It can be found at .

Despite 바카라사이트 considerable technical detail, a non-geneticist will still be able to gain much from it. Many of us, myself included, have been sloppy in packaging up our research, but 바카라사이트 intrusion of 바카라사이트 computer and 바카라사이트 internet, and 바카라사이트 increasing commercial pressures on our universities, demand higher standards.