Interpreting Data with Simpson’s Paradox

In this article we have covered interpretation of data with Simpson’s Paradox. It also include topics like drawback of Simpson’s Paradox and how to Avoid Simpson’s Paradox.

Table of Contents

1. Introduction
2. Simpson’s Paradox: Meaning, Definition with Example
3. Lurking Variable
4. The drawback of Simpson’s Paradox
5. How to Avoid Simpson’s Paradox
6. History of Simpson’s Paradox

Introduction

Simpson’s Paradox is a situation that gives a new viewpoint or result by interpreting the same data. The increasing density of digital data with more dependency on data-driven decision-making enhanced the importance and requirement of Data Mining. 

Data Mining organizes extensive data into a structured form for further processing and identifying hidden patterns. It uses various tools and techniques to organize unstructured data. 

In this article, we will learn about Simpson’s Paradox and how to resolve it. This article covers many examples to define the meaning of Simpson’s Paradox for data interpretation.

Simpson’s Paradox: Meaning, Definition with Example 

Simpson’s Paradox is a situation that arises when grouping data into subgroups shows different relationships when all groups are clubbed together. It reverses the result of the same data by interpreting data through some hidden variables. Simpson’s Paradox shows its importance with real-world data and defines how we can view the same data with different aspects.

It might be confusing for you to understand this Simpson’s Paradox and how we can use it for data interpretation. 

Consider the most famous example of Simpson’s Paradox: the University of UC Berkeley’s gender basis admission. 

It was in 1973 when the school was going through admissions and saw that there were 44% male applicants and 35% female applicants. People started saying that school is favoring males over women i.e, gender discrimination. There were chances of lawsuits, and to overcome the situation, Peter Bickel, a statistician, examined the school data and got surprising results. 

He saw that the school was functional in 6 departments and 4 departments had more female admissions than men. In the remaining 2 departments, there was no gender discrimination. 

He concluded that females are interested in departments with fewer applications. Peter interpreted data based on the department and divided the school into different departments showing no sex favoring. The school came with admission into different departments. Peter resolved the problem of gender discrimination-based admission, which was not the real problem.

This is Simpson’s Paradox situation, where interpreting data into small groups gives completely different results than combining all data into one major group. Simpson’s Paradox helps in revealing real insight into the data.

Lurking Variable 

It is a hidden variable in data that helps you to identify the truth or mold the data as per your requirement. It is challenging to identify the lurking variable. If data scientists closely examine the data, they can define it and use it to reach conclusions.

Consider an example to understand the meaning of the Lurking variable in Simpson’s Paradox.

A soft drink company wants to launch 2 new soft drink flavors, one Mango and the other Pineapple. The audience tasted both flavors and voted. It was found that 75% of people liked the mango flavor, and 60% of people liked the strawberry flavor. From the results, the mango-flavored soft drink is the winner. But the company is still looking for the results. 

The data mining team rechecks the data and identifies sex can be a deciding variable to get the interested users of both soft drinks.

In the above example, male and female are the lurking variables giving a new result to the data. It was easy to identify the lurking variable of Simpson’s Paradox in the above situation, but it can be challenging.

The drawback of Simpson’s Paradox

Simpson’s Paradox does not result in one solid conclusion. You can interpret data as much as you can using different variables and viewpoints. It becomes difficult to conclude with complete data meaning. 

Using a data set, one can draw different conclusions that become challenging for strategy preparation. 

But, one can positively use Simpson’s Paradox by examining which conclusion you need to use to reach your final destination and reveal the truth.

How to Avoid Simpson’s Paradox

One can avoid Simpson’s Paradox by closely examining the experiment and all related variables. Simpson’s Paradox arises when there is a difference between summing all data versus data into small groups. Simpson’s Paradox interprets data in a completely new form with subgroups.

For example:

You and your partner are going for dinner but are confused about where to go. Your choice is “The Heaven.” Your partner is interested in “Foodie”; to resolve the situation, both of you decide to focus on the modern oracle, and that is customer reviews. “The heaven” is voted by 40% of men and 60% of women as a good dining area. In contrast, 30% of women and 70% of males voted for the “Foodie.” Again confusion, she used the same data interpretation and found a combined population of 65% interested in Foodie restaurants. 

Here, we have two different results for the same data interpretation. One can use the above results as per requirements. 

The restaurant selection situation results in Simpson’s Paradox, and it is unavoidable, but one can use it smartly to get real-world results.

It is easy to resolve Simpson’s Paradox by identifying the need for the result and closely examining the frequency tables. To reveal the truth and hidden data patterns, Simpson’s Paradox helps data engineers and statisticians.

History of Simpson’s Paradox

Simpson’s Paradox was introduced very early by different statisticians. Firstly in 1899, Karl Pearson, a great statistician, described a similar effect in his research.

In 1903, Udny Yule also gave proof of similar effects with the association of different variables.

Edward Simpson. In 1951, Edward H. Simpson, in this technical paper, described the Effect of Simpson’s Paradox, and due to this great statistician, Simpson’s Paradox got its name. 

Simpson’s Paradox is also known as Yule-Simpson’s Effect, Simpson’s Reversal, Reversal paradox, or the amalgamation paradox.

Simpson’s Paradox is a probability and statistical situation that gives different results when data is disaggregated, and the result from small groups disappear or reverses when groups are aggregated.

Conclusion

Simpson’s Paradox helps in understanding the true meaning of the data. It also defines the data limitations and finds the hidden variables which can change the whole meaning of the result. 

Simpson’s Paradox situation does not always arise, and it only introduces if we do not interpret the data from all possible angles. With more dependence on digital data-based results, Simpson’s Paradox helps get different results to meet the business target. 

We have reached the end of this article, and I hope that article serves your purpose of understanding Simpson’s Paradox.

Contributed by Sonal Meenu singh