Maurie Markman, MD
Clinical oncologists may not be fully aware of shortcomings in the laboratory research paradigm that have the potential to distort the scientific underpinnings of vital cancer studies. Yet there are problems surfacing in existing protocols—and the disconcerting reaction of some academics to questions about these procedures— that are critical to the future development of innovative cancer therapeutics that clinicians will be employing in their practices.
Any objective observer would recognize that being able to replicate a scientific claim, including observations related to laboratory-based cancer investigative efforts, is a core component of the process of science.
Of course, there are different forms of replication within the scientific domain. One form is the requirement that an individual scientist or research laboratory repeat experiments several or even multiple times to be certain any observation is reproducible and not simply a laboratory error. Certainly, this process should be undertaken before any results, with their corresponding interpretation, are presented to an external audience or submitted for peer-reviewed publication.
However, a second essential and separate component of the replication process occurs when a completely independent research group subsequently examines any initial claims and confirms the findings, even if not necessarily the interpretation of the observations, made by the first group.
In the clinical investigative domain, this often translates to a request or requirement for a confirmatory study by a second collection of researchers, especially in the setting of unexpected trial results, where the sample size might have led to the outcome due solely to small patient numbers or when the study is being reported by a research unit without an established track record in clinical investigation.
Importantly, trials involving human subjects are—or at least absolutely should be—conducted under well-established and widely accepted national/international guidelines that include the development of a formal research protocol, scientific and ethical review, and regular reporting to an oversight body (an independent review board) on the progress of the investigative effort. In addition, prospective studies should include a formal statistical plan based on the study question being addressed.
Lack of Independent Oversight
But although laboratory based-research may be conducted under quite rigorous internal controls, there is no such established external oversight. Further, there is no mandate for how such efforts should be designed or conducted, for blinding or randomization of experiments, for the number of times a given experiment should be repeated to confirm reproducibility, or even for how outliers in any given test should be handled.
For example, it is understood in clinical research that all subjects who entered a trial must be reported even if they never received therapy and that outcomes must include patients who discontinued therapy for any reason.
By contrast, how does one know if a laboratory investigator simply elected to discard the results of several mice that died unexpectedly in the control or study arm of an individual experiment? Or if a particular experiment was repeated 10 times by an investigator and only once gave “favorable results”? What is to prevent an individual investigator or research unit from solely reporting that single “positive” experiment?
With this background in mind, is it really any wonder that when an independent attempt was made by a pharmaceutical company to replicate 53 landmark laboratory-based research results, conformation was only possible in six studies (11%)?1 A similar disquieting outcome was reported by a second pharmaceutical company attempting to replicate previously reported laboratory findings.1
These analyses have appropriately concerned laboratory-based academic leaders including top officials at the National Institutes of Health.2
Unfortunately, as a remarkable indication of the rather profound arrogance of some in the academic community, a number of investigators have questioned the need to be required to replicate their findings, apparently feeling they alone have the ability to conduct high-level quality research.3,4
Such hubris might be amusing if these self-righteous beliefs were not potentially translated into practices that have led to a profound waste of time, effort, and financial resources of many individuals and organizations.