Hit-and-Miss Genetic Testing
The beleaguered defenders of direct-to-consumer genetic testing are now pressing their case in Nature. Robert Green and Nita Farahany complain that the FDA is "overcautious" and Virginia Hughes in National Geographic's blog Phenomena agrees. In this, they are following less temperate commentators who have used terms such as "outrageous" and "borderline absurd."
Critics of the FDA's move to require that 23andMe produce evidence that the health claims it makes about its DTC gene tests are warranted generally assume the technology works. (The moderates do include some caveats.) But in practice, it doesn't. Not reliably enough.
There have now been at least four experiments that involved having the same DNA analyzed by different companies, and in every case significant anomalies appeared. The classic report was the first, by David Ewing Duncan. In 2008, he had his DNA tested as part of his research for a book:
I was told by three companies — Navigenics, 23andme, and deCODE genetics — that my genetic risk for heart attack was high, medium, and low.
That part of the process that led to the book Experimental Man is detailed in an article titled:
You 2.0: I'm Doomed. Or Not.
In 2009, Craig Venter and colleagues ran a similar experiment comparing the reports of Navigenics and 23andMe on five individuals. The full analysis is behind a paywall (Duncan wrote a useful summary for MIT Technology Review) but the abstract notes:
For seven diseases, 50% or less of the predictions of two companies agreed across five individuals
In 2010, the U.S. Government Accountability Office ran a similar comparison:
GAO's donors often received disease risk predictions that varied across the four companies, indicating that identical DNA samples yield contradictory results. One donor was told that he was at below-average, average, and above-average risk for prostate cancer and hypertension.
The GAO uncovered much more, including unethical and even illegal comments and advice. And yet this neither ended the story nor ushered in any significant regulatory changes.
By 2013, several companies had left the industry or required a doctor's referral, but Kira Peikoff repeated the experiment and published her results in the New York Times:
I Had My DNA Picture Taken, With Varying Results
For example:
23andMe said my most elevated risks — about double the average for women of European ethnicity — were for psoriasis and rheumatoid arthritis, with my lifetime odds of getting the diseases at 20.2 percent and 8.2 percent. But according to Genetic Testing Laboratories, my lowest risks were for — you guessed it — psoriasis (2 percent) and rheumatoid arthritis (2.6 percent).
Bernard Munos took a different tack. He had his whole family tested by 23andMe and wrote about the results in Forbes:
My daughter, as expected, got half of her genes from my wife and half from myself. My son, however, got half of his from my wife, but only 47.5% from me. Now, he is a crafty guy, but how did he pull that off?
He also found some (relatively trivial) projections that do not accord with his experience, for instance about lactose intolerance and metabolizing caffeine. In yet another case, Lukas Hartmann wrote about a more disturbing finding in Quartz:
Why 23andMe has the FDA worried: It wrongly told me I might die young
Hartmann's experience was caused by a bug in 23andMe's analysis that confused two unrelated heterozygous mutations with a potentially lethal homozygous mutation. Hartmann painstakingly examined the detailed data dump of his genetic analysis, and found the bug. The company confirmed and removed it, and now merely notes that he is at risk of passing on the mutations.
Craig Venter, quoted in Peikoff's New York Times article, commented on the contradictions she found:
Your results are not the least bit surprising. Anything short of [whole genome] sequencing is going to be short on accuracy — and even then, there's almost no comprehensive data sets to compare to.
Given the state of the science — and perhaps the state of reality — even full-genome analysis may not be definitive, since the risks of most common diseases are caused by complex interactions of genes, gene regulation, and environment. Certainly, almost everyone needs the assistance of trained professionals to understand the results of comprehensive genetic tests, let alone to make significant decisions based on them.
The anomalies detailed above were not uncovered by campaigners concerned about privacy issues and other social, economic, philosophical or political concerns. They were reported by people who actively want genetic technologies to succeed or are concerned about consumer protection. The inescapable conclusion is that the direct-to-consumer genetic testing industry is still, as one of the co-founders of Navigenics told Michael Hiltzik in December 2013, "not ready for prime time." Whether it ever will be ready is yet to be determined.
Update, 1/30: For an excellent analysis of how unreliable predictions based on DTC test results are, see this article by Cecile Janssens, whose research [pdf] was previously discussed by my colleague Jessica Cussins.
Previously on Biopolitical Times: