VA Governor's race:Election Eve update

Several elections are being held tomorrow. The polls say there is no suspense whatsoever in the governor's race in New Jersey or the mayor's race in New York, where repsectively Chris Christie and Bill De Blasio are expected to win by double digits. In Virginia, the governor's race between Terry McAuliffe and Ken Cuccinelli looks closer, but if my system hold true, there is extremely little chance of an upset. (I have notbeen checking the data for the down ticket races. Daily Kos says only the Attorney General race looks close.)

For a time in early September, Cuccinelli appeared to be gaining on McAuliffe, but that trend turned around and Cuccinelli's team has been starved for good news since. The last three updates on October 20, October 27 and November 3 have all put the Confidence of Victory number for McAuliffe at a formidable 99.4%. (In a strange coincidence, these three numbers came from three separate polls from three different companies, the well-known Quinnipiac and Rasmussen and the lesser known Newport College Polling Group.

My system does not predict the margin of victory, but except for one outlier predicting a close 2% margin, the rest of the polls in the past week make it look like the margin will be 6% to 7%.

I will report back on Wednesday about the results.
 

Results from the Democratic mayoral race in New York City.

Almost all the ballots have been counted in the New York City municipal elections held yesterday. My last post here was on Sunday, but there was a late poll from Public Policy Polling that changed my numbers. After working on how I was going to consider the "median result" from a set of four polls in a three person race where there was a 40% threshold, my last prediction on Monday evening was posted to Twitter.

Final call on NYC Dem Mayor's race, different from @SamWangPhD 
55% De Blasio 1st ballot 
30% De Blasio/Thompson 
15% De Blasio/Quinn

Professor Wang's last prediction was a 90% probability of De Blasio on the first ballot.

The current vote count has Bill De Blasio at 40.3% and his closest rival Bill Thompson at 26.2%. Because his margin over the 40% threshold is so slim, we will have to wait for absentee ballots and an automatic recount for a result this close. The reported estimate is next Monday for a conclusive result.

Here on the blog, I made no mention of the comptroller's race, but I did mention it on Twitter in a two part tweet.

NYC mayoral primary today. @SamWangPhD and I agree that De Blasio first ballot win is most likely as is Spitzer for Comptroller. [1/2]

If Spitzer loses the Comptroller race, full credit should go @QuinnipiacPoll, the only organization to favor his rival Stringer. [2/2]

There were several polls of this race as you can see at this link to the Real Clear Politics polling page. There was nothing that made this look like a close race until late August, with Eliot Spitzer's hopes for a political comeback looking very strong while Manhattan borough president Scott Stringer seemed unable to make any headway. But then Quinnipiac had a poll that said the race was tied, then another Quinnipiac poll had a small of 2% for Stringer on September 1, then a commanding 7% lead a week later.

The actual result was Stringer by 4%, 52% to 48%.

The reason my system - and Professor Wang's - favored Spitzer was that no other pollster agreed with Quinnipiac, not even once. After the first Quinnipiac poll that showed a close race, Siena, Marist and PPP all released polls, all of them favoring Spitzer.

So my system failed to predict this race. Often, when I get a race wrong, I try to find ways to adjust things to improve, but if I was given a similar situation tomorrow, I wouldn't change a thing. Here are my reasons.

1. I never weigh in more than a single poll from any polling company and always the most recent.

2. Because Quinnipiac only gets one result counted, they had one poll out of three in the final mix, along with PPP and Siena.

3. In a three poll situation, I take the median result, not the average of the three. Quinnipiac was the outlier, but outliers aren't often the closest to correct. Obviously, it will happen sometimes, but it is rare enough that I do not want to second guess myself in a situation like this. Polls miss the mark, sometimes by making bad assumptions, sometimes just by the fact that randomness is involved. For example, of the three polls Quinnipiac had the largest sample size, but that does not always (or even often) mean the most accurate. Looking at the mayoral polling and factoring out the undecided, Quinnipiac had De Blasio at 46% of the people who had a preference, while PPP had him at 42% and Siena had him at 39%. In this case, it's the low outlier that was closest.

So this means at least one more report on the mayor's race. It is widely agreed that the Republican primary winner Joe Llota will be very hard pressed to win in the general election. De Blasio got 260,000 votes in the Democratic primary, Llota finished first with over 50% in the Republican primary with about 30,000, which would have given him sixth place in the Democratic race.

And while I did not use their poll, I want to congratulate Quinnipiac for being alone with the correct result in the comptroller's race. It is a field filled with randomness, but they had Stringer with a chance or the lead three separate times in barely two weeks when no one else did. That's a record they can point to with pride.

New York City Mayoral Race: Democratic PrimaryThe Weekend before the election.

On Tuesday, New York City voters will go to the polls to decide the Democratic and Republican candidates to be the next mayor. The conventional wisdom is the Democratic nominee holds a huge advantage over whichever candidate the Republicans put forward, so the Democratic primary is being given the lion's share of interest.

Since mid-August, just after Anthony Weiner's habit of sending portraits of his penis to strangers became common knowledge, the front runner has been Bill De Blasio. It is widely agreed that the ads featuring his 15 year old son Dante, he of the remarkable adolescent baritone and even cooler Afro, have made quite the impact.

This Sunday, a new poll from Marist has been  published which agrees with the general trend if not the exact numbers.

Candidate recent % (previous %)

Marist - 9/6 recent, 8/14 previous
De Blasio 36% (24%)
Thompson 20% (16%)
Quinn 20% (24%)
Other 16% (21%)
None of the Above 8% (15%)

Quinnipiac - 9/1 recent, 8/12 previous
De Blasio 43% (30%)
Thompson 20% (22%)
Quinn 18% (24%)
Other 12% (17%)
None of the Above 7% (7%)

Siena - 8/28 recent, 8/7 previous
De Blasio 32% (14%)
Thompson 18% (16%)
Quinn 17% (25%)
Other 16% (19%)
None of the Above 17% (26%)

The agreement is across the board. De Blasio had a great September, Quinn took a beating and Thompson improved enough to probably be the favorite for second place if that matters. De Blasio is seen as the progressive candidate and the candidate for the boroughs other than Manhattan. Even the fact that De Blasio is a Red Sox fan may not be enough to stop him.

Of the three polls, Quinnipiac has been the one that has found the most support for De Blasio and Siena has lagged, but that may be due to Siena being first to polls in each case. That leaves Marist, last to poll and the median poll for De Blasio support both mid-August and early September.

If we accept Marist as the most reliable because of being the median, there are three outcomes that look possible on Tuesday.

De Blasio gets more than 40% on the first ballot: 34%
A De Blasio-Thompson run-off: 36%
A De Blasio-Quinn run-off: 30%

If I were putting a wager on this three way outcome, I'd put my buck on De Blasio-Thompson in a run-off, though any of these three results can hardly be called an upset. The only thing that would make me doubt my sanity or the validity of my methods is De Blasio finishing third. There hasn't been a poll in more than three weeks that shows that result to be even close to plausible.

I'll be back on Tuesday evening to report the actual vote.

Some thoughts about election prediction.

It's been nearly two weeks since there has been any new data on the New York City mayoral race. The last few polls agree the three at the front of the pack are Christine Quinn, Bill De Blasio and Bill Thompson. Unless one of them gets over 40% of the vote, and the recent polls make that look very unlikely, the top two will go into a runoff election, and the odds make it look like an uphill climb for Thompson.

The New York Times endorsed Quinn. Her bio is certainly compelling from a progressive standpoint. She would be the first female mayor of America's largest city and she would be the first openly gay mayor. Her record on the issues is not as compelling from a progressive standpoint. She is seen by many as a person who would continue the Bloomberg status quo. The Nation, a truly progressive periodical unlike the Times, has endorsed De Blasio.

Nate Silver has published numbers saying that history favors the long term front runner of the year, which would be Quinn in this case. She had competition from Anthony Weiner after he announced and before his spectacular crash, but for months before that she lead in every poll.

This is one of the many places in predictions where I part company with Silver. I assume that every election stands on its own and historical data should not outweigh current polling.

It's impossible to predict the big events and impossible to gauge the weight they will have. It looked like based on name recognition Anthony Weiner was a serious candidate, but the Carlos Danger situation cut his support in half, his position plummeting from first to a distant fourth. A judge deciding "stop and frisk" was unconstitutional gave De Blasio a big boost, but it's hard to say if it's a surge or a bounce until we get more data. The Times endorsement should be much more important that The Nation's, but are either of them really events that make a difference in the polls?

I have no idea.

When I say that the last polls look like an overwhelming likelihood of a Quinn/De Blasio runoff, I am completely willing to say something else if the next polls change. It's very hard to say what events will change the polling numbers, known in journalistic circles as "moving the needle".

In 2012, Romney's 47% comments moved the needle against him, but he was already in a horrible position when he said that. Obama's awful first debate definitely moved the needle in Romney's favor, but by the time of the Biden/Ryan and the second Obama/Romney debate, the movement towards Romney had stopped and the tide reversed. He never was favored to win.

Conservatives hoped Benghazi would move the needle. It didn't.

Silver went to Princeton and I went to Cal State Hayward. Silver was hired by the Times and I haven't been able to get anyone to pay me for my work. In my favor, I did better than he did in 2008 and 2012. (He missed Indiana in the electoral vote in 2008 and two Senate races in 2012 that I got right. One of them he missed by giving weight to historical data, something I would never do.)

Here's the big difference between us as far as I'm concerned. My education is in pure mathematics and his is in applied.

Election prediction is applied math. I deeply distrust it and I wait until I think the data really means something. I have arbitrarily chosen about a week before the polls close. By then, I figure enough mail-in ballots have been cast that the most important proviso of any opinion poll is actually true, that "the election is being held when the poll was taken".

Is seven days the right number? Maybe ten would be better, maybe four. If my predictions become less reliable in the future, changing this time period is one of my first places to look to improve my system.

This brings me to something else Silver has done that I disagree with vehemently. He is predicting the make-up of the Senate in 2014 and giving a margin of error. These numbers are being used as scare tactics by the Democrats begging for money.

As far as I'm concerned, these numbers are worse than nonsense. He does actual work, so I am not happy comparing him to lazy, ignorant anti-intellectuals like Dick Morris, Karl Rove, Jim Cramer or George F. Will, but predictions about a multi-part election more than fifteen months away from polls closing are mere balloon juice. Even if someone "crunches numbers", given how many unknowable events there will be in the next year and a quarter means his work has no more meaning than doing exact bio-rhythms or a complete astrological chart.

I do not know if Nate Silver will ever read what I write. If he does, I want to say to him, older nerd to younger nerd, stop doing the stuff that is most likely to hurt your reputation. A fifteen month prediction about elections is no better than your work in fantasy football or Oscar prediction, which you have been gracious enough to admit has been less than optimal.

Wait until just before the election. Then the numbers of people doing actual work, work like yours and mine and Sam Wang's, have meaning. Right now, your work in 2013 is like Gene Simmons of KISS going out on a limb in 2011 predicting President Rick Perry and giving us as the reason to believe him, the quote "I'm never wrong."

He was wrong. Even with your margin of error, saying there is 95% confidence in your numbers is plain silly.

Stop being silly. You are much better than that.

Here endeth the lesson.