Fertility app Glow accused of false advertising
Fertility app, Glow, claims it gives its users "at least a 40% greater likelihood of getting pregnant compared to if you weren’t as actively tracking and using Glow", data which, according to Wired, does not match up with a study presented at the annual meeting for the American Society of Reproductive Medicine in Baltimore. Researchers speaking to Wired criticise the data presented, saying that its methods may merely measure "the motivation to conceive, rather than helping to conceive". Doubtless, Glow has access to a superb dataset, but many are unconvinced that they have proved the efficacy of their ovulation-tracking app. 

But according to researchers not associated with Glow, the study has a chicken-and-egg problem. Is frequent app use driving behavior that supports fertility, or are people who are already aware of fertility facts more inclined to use Glow’s app regularly?

"The study is very crude in nature—very similar to an online survey, without controlling for many key factors," says Kutluk Oktay, a doctor and reproductive endocrinologist at New York Medical College. Perhaps women who use the app more frequently are more actively engaged in trying to conceive, while casual Glow users are less deeply invested in conceiving right now—so they’re not having sex as often, or aren't doing other things to improve their chances at conception. "It may just be measuring the motivation to conceive, rather than helping to conceive," says Oktay.

Glow did control for age in one of its secondary studies presented at ASRM, and its study population included only women who reported a first day trying to conceive and who conceived naturally. But according to Emory University's Kawwass, it would have been helpful if the authors controlled for variables like race, history of infertility, duration of unprotected intercourse without contraception, and prior use of fertility treatment—all of which could have an impact on the likelihood of conception.1

The study's results themselves support the idea that the observed fertility gains were caused by more than app usage. Vitaly Kushnir, a fertility doctor at the Center for Human Reproductive in New York, notes that the study reported improvement in fertility after just one ovulatory cycle—unlikely to be an effect of short-term tracking. “This is more likely to reflect selection bias than a change in behavior learned from fertility tracking,” Kushnir says.

The Potential for Better Data

Here's the frustrating part: Glow has the potential to design a truly excellent study based on the detailed fertility data it collects from its users. Because it has a lot of it.

Not every person fills out every blank in the app, but Glow allows users to aggregate daily information about everything from whether they had sex to basal body temperature to sleep duration. If enough of the 100,000-plus users in this study tracked data about their alcohol consumption or weight or cervical mucus, perhaps Glow would be able to control for those differences to pull out the real impact of app usage. (A perfect study would compare two groups that were biologically and behaviorally identical—except for how often they used the Glow app.) Glow's executives do say the company is still in its early days of data collection. But its data scientists also don't seem to be looking into the tough questions so far.

Asking those questions—and asking them the right way—is remarkably important given how difficult it is to collect clean data about fertility. The best data the scientific community has, Kawwass says, is from patients who have sought care from fertility clinics—which still isn’t a whole lot of people. Some landmark studies are decades old, conducted on the Hutterites, a population that believes in large families and abstains from contraception.

Here's the frustrating part: Glow has the potential to design a truly excellent study based on the detailed fertility data it collects from its users.

Now, Glow's user base isn't a perfect test ground for fertility research either—something the company readily admits. “The Glow population is not currently representative of the general United States population,” the study says, because, by definition, the app's users are concerned about pregnancy. But Glow expects that to change, and as more users join the app, its database could broaden and improve. "In two to three years I could see having data sets that we can trust, because we'll soon have the finest data set, but right now we're at first base, for the first inning, of a nine inning game," says fertility specialist Dr. Phil Chenette, a co-author of the paper and medical advisor to Glow.2

But their sheer numbers could allow researchers to ask novel questions about fertility, ones that are difficult to answer within the confines of academic clinical trials and fertility clinics. And the data could increase confidence in the one result Glow and its users are truly interested in: the app's ability to improve chances of conceiving.

At last year’s ASRM meeting, Glow said it had tested the assumption that the moon affected women’s menstrual cycles—and it found a correlation. At the time, the company rightly said that it needed “to do more studies to determine causality.” The same logic applies here: Correlation does not imply causation. The fact that women’s health is getting more attention among makers of apps is a good thing, as is helping women proactively manage fertility. But as Kawwass says: “We just have to be careful about what kind of conclusions we draw from the available data.”

1UPDATE 10/28/15 5:00 pm EDT: This story has been corrected to note that the total Glow population used in the study included only women who reported a first day trying to conceive and who conceived naturally.

2UPDATE 10/28/15 5:00 pm EDT: This story has been updated to include comment from a medical advisor to Glow.