How do I prove Simpson's Paradox, scilicet $P(A|B) > P(A|B^C)$?
$\forall a,b,c,d > 0, a<b, c<d \implies$ $0 \le ab < cd$.
-
The blue, orange and purple inequalities below are all in the OPPOSITE direction as $P(A|B)>P(A|B^C)$! I can multiply the blue and orange inequalities together, but their product is in the opposite direction too!
-
I can't simply multiply the purple inequality by the green inequality, because the green inequality's in the SAME direction as $P(A|B)>P(A|B^C)$.
I modified this example to make it easier to understand.
Example 2.8.3 (Simpson's paradox).
Two doctors, Dr. Good and Dr. Bad, each perform two types of surgeries: cardiac surgery and Band-Aid removal. Each surgery can be either a success or a failure. The two doctors' respective records are given in the following tables, and shown graphically in Figure 2.6, where white dots represent successful surgeries and black dots represent failed surgeries.
Dr. Good had a higher success rate than Dr. Bad in heart surgeries: 70 out of 90 versus 2 out of 10. Dr. Good also had a higher success rate in Band-Aid removal: 10 out of 10 versus 81 out of 90. But if we aggregate across the two types of surgeries to compare overall surgery success rates, Dr. Good was successful in 80 out of 100 surgeries while Dr. Bad was successful in 83 out of 100 surgeries: Dr. Bad's overall success rate is higher!
FIGURE 2.6
An example of Simpson's paradox. White dots represent successful surgeries and black dots represent failed surgeries. Dr. Good is better in both types of surgery but has a lower overall success rate, because he is performing the harder type of surgery much more often than Dr. Bad is.What's happening is that Dr. Good, presumably due to his reputation as the superior doctor, is performing a greater number of cardiac surgeries, which are inherently riskier than Band-Aid removals. His overall success rate is lower not because of lesser skill on any particular type of surgery, but because a larger fraction of his surgeries are risky.
Let's use event notation to make this precise. For events A, B, and C, we say that we have a Simpson's paradox if \begin{align*} P(A|B,C) &<P(A|B^C,C)\newline P(A|B,C^C) &<P(A|B^C, C^C);\newline \text{but }P(A|B) &>P(A|B^C). \end{align*} In this case, let A be the event of a successful surgery, B be the event that Dr. Bad is the surgeon, and C be the event that the surgery is a cardiac surgery. The conditions for Simpson's paradox are fulfilled because the probability of a successful surgery is lower under Dr. Bad than under Dr. Good whether we condition on heart surgery or on Band-Aid removal, but the overall probability of success is higher for Dr. Bad.
Blitzstein. Introduction to Probability (2019 2 ed). pp 76-78.
0 answers
0 comment threads