The Statistical World, part 4

From late 2011 through November 2012, my main math hobby was keeping track of the election data. On the morning of election day, I made a prediction as to how things would turn out in the 51 electoral college races (all 50 states and DC) and the 33 Senate races. I was using a mathematical system which gave a favorite in 83 of the 84 races and called one race (electoral college in Florida) a toss-up. I flipped a mental coin and gave Florida to Romney.

It turns out I'm not fantastic at flipping mental coins. Obama barely beat Romney in Florida. But in the other 83 races, my system picked the winner every time.

Nate Silver of the New York Times also made predictions in all 84 of these races. His system also called Florida a toss-up, which is a credit to both of us, but also a little lucky. In other elections, my system has called a race a toss-up and one side or the other won handily. In the other 83 races, Nate went 81-2, missing two Senate races in Montana and North Dakota, two results that my system got right.


So if we include my guessing call of Florida, I went 83-1 and Nate went 81-2. Our percentages are 98.8% and 97.6% respectively, both of which count as excellent when it comes to prognostication.

Are we geniuses or what?

Well, I'm going to say "or what". The general election polling data was non-stop for several months. Looking back at my records, there were 700 polls dealing with the 84 races in the last five weeks of the race. I started keeping daily track of the median electoral college result after Obama's disastrous first debate appearance, and his numbers did suffer. But then came the Biden-Ryan debate and second Obama-Romney debate and Romney finally repudiating his "47% comment" and the Obama advantage moved up to where it had been as of early October.  It was hard to pick winners because the races were not very close and the opinions were not taking huge swings, just small ones.

Here is the best data that shows Silver and I are not geniuses and that is the primary election season. This graph shows the ups and downs of the four candidates still in the race in February, Mitt Romney (green), Newt Gingrich (gray), Rick Santorum (brown) and Ron Paul (gold). I also tracked NONE OF THE ABOVE in black.

There were a lot of polls during this month, but not anywhere near the number there were in the general election. More than that, the Republican electorate was in an amazing state of flux. You can see Santorum climbed from third place to first place then back down to second in the space of four weeks. More than that, NONE OF THE ABOVE was holding steady at about 15% throughout the month.

In the primaries that month, Nate and I weren't scoring in the 98th or 99th percentiles. The data was sketchier and our predictions suffered. Predictions from polling data is a lot more accurate than predicting the results of sporting events, to give just one example, but even taking the average (or median) of a lot of polls can be shaky, especially when NONE OF THE ABOVE is well over 10% this close to the election.

Nate's book The Signal and the Noise is a study of why some predictions do well and others do not. He thinks that in the long run we are going to learn how to do better in general. I'm not convinced. Sometimes, the randomness inherent in a system will overwhelm the cleverest human prediction methods.