Hunting for Simpson’s Paradox (part 2)
(If you haven’t already, you should read part 1 of this article which was posted on April 13, 2011)
Simpson’s paradox illustrates an intuitive error in reasoning that's hard to accept without credible examples. It occurs when a correlation or trend that is present in groups is reversed when the groups are combined. The example we’ve been working with deals with batting averages. A hypothetical situation was presented in the first part of the article in which we stated that one batter had a higher batting average against left-handed pitchers and against right-handed pitchers, separately. However, when the overall batting average was calculated, that batter had a lower average.
Here’s the table once more just for reference:
Certainly it was an interesting example but it was entirely made-up. I just tweaked the numbers a bit to get the table to work. I set my sights on finding some of my own examples of Simpson’s paradox based on REAL data.
Attempt 1
Yahoo Sports provides some easily accessible baseball team split stats, e.g., home vs. road, pre-all star break vs. post-all star break, turf vs. grass, indoor vs. outdoor (http://yhoo.it/hKAiLn). I started by pulling the overall team batting averages and then their averages split by home games versus road games (for the 2010 season). By pasting into Excel and sorting, then manually checking for a good 15 minutes, I was able to establish that there was no example of Simpson’s paradox in this data set. Bummer.
Attempt 2
This process had to be automated because there was no way I was going to spend 15+ minutes checking every split, every season, until I found an example. While this could be fairly easily done with a VBA script (e.g. macro) in Excel, I decided to move over to MATLAB simply because I’m more well-versed in scripting there. Algorithmically speaking, it really is a piece of cake to automate.
NOTE: If the details of the script don’t interest you, you can skip down to the results. And yes, I found some examples. WOOHOOO!
First, you need your data read into the program. In Excel, I had a table with the first column as the team names, the second column was the overall batting average and the third and fourth column were the split stats (i.e., team batting averages for road vs. home). To move this into MATLAB, I created a cell array called teams which would contain team names.
teams = {}
Then, after opening the teams variable in the variable editor window (double-click it in the Workspace), I pasted in the team names (right-click, “Paste Excel Data”). Next, I created a variable x and pasted in the three columns of batting average data.
At this point a very simple little script (simphunt.m) tests every pair of teams for Simpson’s paradox. The main step of the script is to compare the overall averages for each pair and the split stats for each pair. If the relation of the split is reversed in the overall, then we have an example.
if (x(i,1)>x(j,1) && x(i,2)<x(j,2) && x(i,3)<x(j,3)) || ...
(x(i,1)<x(j,1) && x(i,2)>x(j,2) && x(i,3)>x(j,3))
Results
The first split I tested was, again, home vs. road (to which I already knew the answer):
There are no cases of Simpson's Paradox
The next was day vs. night. Disappointed again.
There are no cases of Simpson's Paradox
Fortunately, it was only taking a minute or so to fetch the data and move it into MATLAB. Nevertheless, I was still feeling a little disheartened. I was beginning to expect to run through dozens of splits, maybe from multiple seasons, or even moving on to player vs. player stats instead of teams.
Then I went with indoor vs. outdoor. JACKPOT!!!
There are 5 cases of Simpson's Paradox
' Chicago Cubs' ' Oakland Athletics'
' Cleveland Indians' ' Tampa Bay Rays'
' Colorado Rockies' ' Milwaukee Brewers'
' Los Angeles Angels' ' Tampa Bay Rays'
' New York Mets' ' Toronto Blue Jays'
Five Cases!!! Looking back at the data we can confirm:
I could clearly keep hunting but I’ve satisfied my desire for finding my own examples. Next time I bring up Simpson’s paradox in any of my classes, I’ll start with my manufactured example but quickly move on to these REAL examples.
Read More…
Find out more about Simpson’s Paradox at the following links:
Simpson's paradox - Wikipedia, the free encyclopedia
http://www.furtadoworld.com/handouts/SimpsonParadox.pdf
When Combined Data Reveal the Flaw of Averages
Hunting for Simpson’s Paradox, part 1
Let’s say we happen to know the batting average for two baseball players. Overall, player 1 has a higher average that player 2. However, if you consider only how each player hits against left-handed pitchers we find that player 2 actually has a better average player 1. In this hypothetical scenario, it also turns out that player 2 also has a better average against right-handed pitchers. How is that possible?
Doesn’t it make intuitive sense that if player 2 is better than player 1 against left and right handed pitchers separately, that he must be better than player 1 against all pitchers? While that may be what our intuition tells us, it turns out that it’s not necessarily true.
Consider the following table. Note that batting average is simply the ratio of a players number of hits over the number of at-bats.
This table presents exactly the hypothetical scenario described above. Separately, player 2 had a higher average than player 1 against left and right handed pitchers, but over all player 1 has a higher average than player 2.
This phenomenon is commonly known in statistics as Simpson’s paradox. It demonstrates how our intuition can get us into trouble. Briefly stated, Simpson’s paradox occurs when a correlation or trend that is present in groups is reversed when the groups are combined.
I was recently reminded of Simpson’s paradox when @Math_Bits posted a link on twitter to an article, “Instances of Simpson’s Paradox” by Thomas R. Knapp. It got me thinking. Sure I can manufacture an example and I’ve seen a few examples in papers, texts and even wikipedia. But I want to find my own examples. And of course, manufactured examples like the table above don’t count. I need real data.
I figured the best place to start for real data that’s easy to find is in sports, say baseball, while we’re thinking of it. I started simple and pulled up the first split data set I found. Over on Yahoo sports, I pulled up team statistics for the full 2010 season, see http://yhoo.it/hKAiLn. I pulled the data over into excel and began (manually, ug…) hunting for an example of Simpson’s paradox. Of course, I would start the hardest way possible. I looked at batting averages (overall, home and on the road). I made three lists, one for each category: overall, home and road. Then, I sorted the teams from highest to lowest and began looking one-by-one for pairs of teams where one had a higher average overall but lower both at home and on the road.
… to no avail. I even reversed the process by sorting from lowest to highest.
I’m beginning to believe that Knapp was right when he claimed that examples of Simpson’s paradox are extremely rare.
The next step was to automate this process. As a programmer, I began devising a simple code that will take an overall list and lists for each group and identifies all those pairs that satisfy Simpson’s paradox.
In the next post, I’ll walk through the progress I made using Matlab to do my dirty work.
Read More…
Find out more about Simpson’s Paradox at the following links:
Simpson's paradox - Wikipedia, the free encyclopedia
http://www.furtadoworld.com/handouts/SimpsonParadox.pdf
When Combined Data Reveal the Flaw of Averages