Play It Again, Sam: Trying–and Failing–To Reproduce Scientific Results

With the release of Zyprexa, a second-generation antipsychotic, Eli Lilly was certain it had hit pay dirt.

The second-generation antipsychotics, known as the atypical antipsychotics, or atypicals, had an improved side effect profile over the older antipsychotics, and the original research on large numbers of schizophrenics in the 1990s showed great improvement for patients treated with Zyprexa and others like it (think also of Abilify, Seroquel, and Geodon as the highest profile atypicals).

By 2001, Zyprexa had surpassed even Prozac as Lilly’s top-grossing drug, and all looked rosy on the financial front.

However, something odd was happening when it came to the research support trials. Simply put: the results that so emphasized that effectiveness of the atypicals couldn’t be replicated.

Replicability [not the most graceful of terms, but it’ll have to serve] is the foundation stone of scientific studies. It refers to the degree to which a second study will repeat the same findings as the initial study. Strong research studies should be repeatable–and their results easily replicable.

So far, so good. If a reputable study, using rigorous methodology, finds a signficant correlation between left-handed people and higher scores on certain parts of the IQ test [I’m totally making this up, and, since I’m a dominant righty, I’m sure no such thing is true], other scientists in other labs should be able to replicate that finding in order for it to hold any validity.

But it turns out, over and over again, that strong, well-controlled research studies produce results–results, often, that the public has bought into and acted accordingly–which cannot later be replicated.

But by 2006, research seemed to indicate that the therapeutic value of the atypicals was not at all what had originally been asserted. A study that year produced an effect less than half of what was found in the first trials. It came to the point where a number of researchers asserted that the expensive new meds were hardly an improvement over the original antipsychotics–bad news, indeed, for the drug companies who saw these meds as tickets to the bank for years to come.

Recently, as a critique of the poor research and super-agressive marketing that keeps the atypicals as mooing cash cows despite the cheater, older antipsychotics, the British Journal of Medicine published a scathing editorial against their primacy entitled “The rise and fall of the atypical antipsychotics”  (October 2011).  “Evidence is growing,” says the editorial, “that the primary purpose of these fabricated classes [of second-generation antipsychotics] is for marketing. It is time we stopped using these expensive labels – they are all just antipsychotics.”

Well, they told them off!

But, lest you cry for poor old Eli Lilly, you should know–research replicability be-damned–Zyprexa is still their top grossing drug.

All’s well that ends well, I suppose.

But the lack of replicable scientific studies and crosses into non mental health domains.There are numerous examples–I just cherry-picked some juicy ones.

Remember vitamin E’s magic heart-related properties?

Well, take, for example, two large studies, the Health Professionals Follow-Up Study and the Nurses’ Health Study, both in 1993. Both studies found that vitamin E had a significant correlation with decreased heart disease. Further, a 1996 trial of 2002 patients indicated a  47% risk reduction for cardiovascular deaths or nonfatal myocardial infarction with vitamin E.

Clearly, vitamin E was the bomb when it came to protecting your heart.

And all that was well and good, particularly if you owned stock in Puritan’s Pride, and heart patients like my husband were happy they could swallow a pill and feel virtuous, especially if they, again like my husband, thought exercise was fine for Olympic athletes and marathoners, but really not their particular cup of tea.

However, a larger randomized trial in 2000 ripped the carpet out from under the vitamin E industry. The paper, published in the New England Journal of Medicine, found not just less of an effect, but rather absolutely no beneficial effect for vitamin E on heart disease at all. Nada, nothing, bubkis.

(from pharmagossip.blogspot.com)

And how about estrogen, anyone? As a female, I’m all for it.

But claims were it did more than ensure my femme fatale figure. A crucial study supported the idea that estrogen was  a strong neuroprotective agent that could decrease the risk of dementia.  A 1996 study discovered a 60% relative risk reduction in postmenopausal women who took estrogen. As of 2006 the study had received 915 citations.

But there was a problem.

Not only could the study not be replicated, but in fact the opposite results were found.  Finally, in 2004, the Women’s Health Initiative  Memory Study Random Controlled Trial published results indicating a tendency towards increased risk of dementia with supplemental estrogen in postmenopausal women and–get this–worsening of cognition with the hormone.

 The original study never could be replicated–to this day estrogen is not recommended as an intervention in the treatment of dementia.

A cure for cancer, anyone?

In 2009, a group of Boston researchers published a study that described how they had destroyed cancer tumors by specifically targeting  the protein STK33. This research was too good to pass up.

So the biotechnology firm Amgen Inc. quickly assigned two dozen researchers to try to replicate the experiment. End goal: turning the findings into a drug that could revolutionize cancer treatment.

After six months of intensive work, the scientists threw in the towel. There simply was no replicating the results.  As quoted in a December 2, 2011, Wall Street Journal article entitled “Scientists’ Elusive Goal: Reproducing Study Results,” this, apparently, came as no surprise to Amgen.

“‘I was disappointed but not surprised,'” says Glenn Begley, vice president of research at Amgen of Thousand Oaks, Calif. ‘More often than not, we are unable to reproduce findings’ published by researchers in journals.”

In 2005, John Ioannidis, somewhat of a replicability expert these days (and a guy who clearly doesn’t shun controversy) stated in an interview with the Atlantic Monthly that “as much as 90 percent of the published medical information that doctors rely on is flawed.”  He examined  frequently cited clinical research studies  in three of the premier medical  journals. Initially, 45 had ‘positive results,’ indicating that the particular drug or intervention tested was effective.

What did he find?

Well, researchers only attempted to replicate 34 of the claims [note that that leaves, if my math skills hold up, a surprising 11 studies that were never looked into further–a concerning fact if there is one].  Of those, a shocking 41% were either explicitly contradicted–or had their effect sizes significantly decreased.

Well–what can I say about what to believe and do for yourself based on the latest research?

Guess it’s back to a previous post–shall we try prayer?

References

Ionnidis, J. Contradicted and Initially Stronger Effects in Highly Cited Clinical Research. JAMA 2005; 294(2):218-228.

Kendall, T. The rise and fall of the atypical antipsychotics. British Journal of Psychiatry 2011; 199:266-26.