Serendipity in Medical Breakthroughs
Many major medical breakthroughs, including cancer drugs and chemotherapy, have been discovered by chance rather than through directed research. These discoveries challenge the teleological view of science: serendipity matters in complex domains.
THE IDEA
THE CASE IN MEDICINE
Unlike technology, medicine has a long history of domestication of luck; it now has accepted randomness in its practice. But not quite.
Medical data allow us to assess the performance of teleological research compared to randomly generated discoveries. The U.S. government provides us with the ideal dataset for that: the activities of the National Cancer Institute that came out of the Nixon “war on cancer” in the early 1970s. Morton Meyers, a practicing doctor and researcher, writes in his wonderful Happy Accidents: Serendipity in Modern Medical Breakthroughs: “Over a twenty-year period of screening more than 144,000 plant extracts, representing about 15,000 species, not a single plant-based anticancer drug reached approved status. This failure stands in stark contrast to the discovery in the late 1950s of a major group of plant-derived cancer drugs, the Vinca Alcaloids—a discovery that came about by chance, not through directed research.”
John LaMatina, an insider who described what he saw after leaving the pharmaceutical business, shows statistics illustrating the gap between public perception of academic contributions and truth: private industry develops nine drugs out of ten. Even the tax-funded National Institutes of Health found that out of forty-six drugs on the market with significant sales, about three had anything to do with federal funding.
We have not digested the fact that cures for cancer had been coming from other branches of research. You search for noncancer drugs (or noncancer nondrugs) and find something you were not looking for (and vice versa). But the interesting constant is that when a result is initially discovered by an academic researcher, he is likely to disregard the consequences because it is not what he wanted to find—an academic has a script to follow. So, to put it in option terms, he does not exercise his option in spite of its value, a strict violation of rationality (no matter how you define rationality), like someone who both is greedy and does not pick up a large sum of money found in his garden. Meyers also shows the lecturing-birds-how-to-fly effect as discoveries are ex post narrated back to some academic research, contributing to our illusion.
In some cases, because the source of the discovery is military, we don’t know exactly what’s going on. Take for instance chemotherapy for cancer, as discussed in Meyers’s book. An American ship carrying mustard gas off Bari in Italy was bombed by the Germans 1942. It helped develop chemotherapy owing to the effect of the gas on the condition of the soldiers who had liquid cancers (eradication of white blood cells). But mustard gas was banned by the Geneva Conventions, so the story was kept secret—Churchill purged all mention from U.K. records, and in the United States, the information was stifled, though not the research on the effect of nitrogen mustard.
James Le Fanu, the doctor and writer about medicine, wrote that the therapeutic revolution, or the period in the postwar years that saw a large number of effective therapies, was not ignited by a major scientific insight. It came from the exact opposite, “the realization by doctors and scientists that it was not necessary to understand in any detail what was wrong, but that synthetic chemistry blindly and randomly would deliver the remedies that had eluded doctors for centuries.” (He uses as a central example the sulfonamides identified by Gerhard Domagk.)
Further, the increase in our theoretical understanding—the “epistemic base,” to use Mokyr’s term—came with a decrease in the number of new drugs. This is something Fat Tony or the green lumber fellow could have told us. Now, one can argue that we depleted the low-hanging fruits, but I go further, with more cues from other parts (such as the payoff from the Human Genome Project or the stalling of medical cures of the past two decades in the face of the growing research expenditures)—knowledge, or what is called “knowledge,” in complex domains inhibits research.
Or, another way to see it, studying the chemical composition of ingredients will make you neither a better cook nor a more expert taster—it might even make you worse at both. (Cooking is particularly humbling for teleology-driven fellows.)
One can make a list of medications that came Black Swan–style from serendipity and compare it to the list of medications that came from design. I was about to embark on such a list until I realized that the notable exceptions, that is, drugs that were discovered in a teleological manner, are too few—mostly AZT, AIDS drugs. Designer drugs have a main property—they are designed (and are therefore teleological). But it does not look as if we are capable of designing a drug while taking into account the potential side effects. Hence a problem for the future of designer drugs. The more drugs there are on the market, the more interactions with one another—so we end up with a swelling number of possible interactions with every new drug introduced. If there are twenty unrelated drugs, the twenty-first would need to consider twenty interactions, no big deal. But if there are a thousand, we would need to predict a little less than a thousand. And there are tens of thousands of drugs available today. Further, there is research showing that we may be underestimating the interactions of current drugs, those already on the market, by a factor of four so, if anything, the pool of available drugs should be shrinking rather than growing.
There is an obvious drift in that business, as a drug can be invented for something and find new applications, what the economist John Kay calls obliquity—aspirin, for instance, changed many times in uses; or the ideas of Judah Folkman about restricting the blood supply of tumors (angiogenesis inhibitors) have led to the treatment of macular degeneration (bevacizumab, known as Avastin), an effect that is more effective than the original intent.
Now, instead of giving my laundry list of drugs here (too inelegant), I refer the reader to, in addition to Meyers’s book, Claude Bohuon and Claude Monneret, Fabuleux hasards, histoire de la découverte des médicaments, and Jie Jack Li’s Laughing Gas, Viagra and Lipitor.
When to use it
A pharmaceutical company screening thousands of compounds for a specific target might miss unexpected uses that could be discovered through broader testing.
A researcher in a different field might stumble upon a drug for another disease, emphasizing cross-disciplinary exploration.
Policy-makers allocating research funds should consider balancing directed projects with exploratory research to allow for serendipitous discoveries.
In personal decision-making, being open to unexpected opportunities can lead to valuable outcomes that a rigid plan might miss.
FROM THEORY TO EVERYDAY LIFE
In practice
Vinca Alkaloids
Discovered by chance in the late 1950s from plant extracts, leading to major cancer drugs, in stark contrast to the failure of the National Cancer Institute's targeted screening of 144,000 plant extracts over 20 years.
Chemotherapy from Mustard Gas
An accidental discovery during WWII when a ship carrying mustard gas was bombed, leading to the development of chemotherapy for cancer, kept secret due to Geneva Conventions.
Aspirin's Changing Uses
Originally a painkiller, aspirin has found multiple new applications over time, illustrating the concept of obliquity in drug development.
Avastin for Macular Degeneration
A drug designed to restrict blood supply to tumors (angiogenesis inhibitor) proved more effective for treating macular degeneration than its original intended use.
A MOMENT TO REFLECT
Put it to work.
Can you identify a major discovery in your field that came about by chance rather than design?
How might an overemphasis on directed research hinder innovation in complex domains?
What are the implications of serendipity for how we fund and evaluate scientific research?
Often confused with
References
- Happy Accidents: Serendipity in Modern Medical Breakthroughs — Morton Meyers
- Fabuleux hasards, histoire de la découverte des médicaments — Claude Bohuon and Claude Monneret
- Laughing Gas, Viagra and Lipitor — Jie Jack Li
CONTINUE THE CONVERSATION
Source texts
- Antifragile (Nassim Nicholas Taleb)