How a statistical paradox can make research findings fall apart
Simpson’s paradox demonstrates how counterintuitive statistics can be
Andriy Onufriyenko/Getty Images
This article is from Proof Positive, our friendly newsletter that explores the joys and peculiarities of math. Sign up today for a weekly math essay and puzzle in your email inbox.
Statistics can produce real results that are hard to believe. As just one example, in the 1970s, the University of California, Berkeley, had to go to court for alleged discrimination against women. The admission rate in its graduate program was 44 percent for male students, but for female students, it was only 35 percent. From these figures, the plaintiffs concluded that men were given preference.
On supporting science journalism
If you’re enjoying this article, consider supporting our award-winning journalism by subscribing. By purchasing a subscription you are helping to ensure the future of impactful stories about the discoveries and ideas shaping our world today.
When applications for individual subjects were broken down, however, a completely different picture emerged. In four of the six largest departments, for example, more women were admitted than men. A closer look at the data by statistician Peter J. Bickel and his colleagues revealed that there was—if anything—a preference for female students.
How could this be? This counterintuitive finding is a consequence of what’s called Simpson’s paradox, a phenomenon that is now widely known in statistics. In 1899 mathematician Karl Pearson first described it in a paper, and four years later it was rediscovered by his colleague George Udny Yule.
But as is so often the case in science, the papers were forgotten until Edward Simpson dedicated a publication to the topic in 1951. According to this paper, trends involving different groups can differ depending on whether they are divided into subgroups. In another example of this paradox in action, data from 2021 showed that COVID was almost twice as deadly in Italy as it was in China despite the fact that every single Italian age group had a higher chance of survival. In such cases, the big-picture trend seems to reverse what smaller subgroups tell us.
Simpson’s paradox often appears when there are undetected factors that influence an outcome. In the case with the U.C. Berkeley data, more investigation revealed that women had low overall admission rates because they tended to apply to departments with higher rejection rates, whereas male students tended to apply to departments with many spots and few applicants. In other words, the women were seeking a place in more competitive departments. This case is a reminder to take a closer look at statistics and consider correlations that could influence results.
And sometimes unravelling these correlations can be challenging. Simpson’s paradox can occur in medical studies that revolve around the approval of a drug, for instance. An active ingredient may be more effective than a placebo for all subjects, but when the patients are divided into subgroups, such as men and women, the placebo turns out to be more effective for each. How should one proceed in such a situation? Should one allow the drug because it was shown to be effective—when considered for all subjects—or abandon the approach because it did not work better than a placebo for either women or men alone?
There is no universal answer. Scientifically speaking, the most sensible thing is to conduct further research to investigate the extent to which a factor such as gender influences efficacy and whether there may be other influencing factors. There’s no substitute for careful analysis when it comes to separating causal relationships from correlations.
It’s Time to Stand Up for Science
If you enjoyed this article, I’d like to ask for your support. Scientific American has served as an advocate for science and industry for 180 years, and right now may be the most critical moment in that two-century history.
I’ve been a Scientific American subscriber since I was 12 years old, and it helped shape the way I look at the world. SciAm always educates and delights me, and inspires a sense of awe for our vast, beautiful universe. I hope it does that for you, too.
If you subscribe to Scientific American, you help ensure that our coverage is centered on meaningful research and discovery; that we have the resources to report on the decisions that threaten labs across the U.S.; and that we support both budding and working scientists at a time when the value of science itself too often goes unrecognized.
In return, you get essential news, captivating podcasts, brilliant infographics, can’t-miss newsletters, must-watch videos, challenging games, and the science world’s best writing and reporting. You can even gift someone a subscription.
There has never been a more important time for us to stand up and show why science matters. I hope you’ll support us in that mission.