VA Governor's race:Election Eve update

Several elections are being held tomorrow. The polls say there is no suspense whatsoever in the governor's race in New Jersey or the mayor's race in New York, where repsectively Chris Christie and Bill De Blasio are expected to win by double digits. In Virginia, the governor's race between Terry McAuliffe and Ken Cuccinelli looks closer, but if my system hold true, there is extremely little chance of an upset. (I have notbeen checking the data for the down ticket races. Daily Kos says only the Attorney General race looks close.)

For a time in early September, Cuccinelli appeared to be gaining on McAuliffe, but that trend turned around and Cuccinelli's team has been starved for good news since. The last three updates on October 20, October 27 and November 3 have all put the Confidence of Victory number for McAuliffe at a formidable 99.4%. (In a strange coincidence, these three numbers came from three separate polls from three different companies, the well-known Quinnipiac and Rasmussen and the lesser known Newport College Polling Group.

My system does not predict the margin of victory, but except for one outlier predicting a close 2% margin, the rest of the polls in the past week make it look like the margin will be 6% to 7%.

I will report back on Wednesday about the results.
 

Results from the Democratic mayoral race in New York City.

Almost all the ballots have been counted in the New York City municipal elections held yesterday. My last post here was on Sunday, but there was a late poll from Public Policy Polling that changed my numbers. After working on how I was going to consider the "median result" from a set of four polls in a three person race where there was a 40% threshold, my last prediction on Monday evening was posted to Twitter.

Final call on NYC Dem Mayor's race, different from @SamWangPhD 
55% De Blasio 1st ballot 
30% De Blasio/Thompson 
15% De Blasio/Quinn

Professor Wang's last prediction was a 90% probability of De Blasio on the first ballot.

The current vote count has Bill De Blasio at 40.3% and his closest rival Bill Thompson at 26.2%. Because his margin over the 40% threshold is so slim, we will have to wait for absentee ballots and an automatic recount for a result this close. The reported estimate is next Monday for a conclusive result.

Here on the blog, I made no mention of the comptroller's race, but I did mention it on Twitter in a two part tweet.

NYC mayoral primary today. @SamWangPhD and I agree that De Blasio first ballot win is most likely as is Spitzer for Comptroller. [1/2]

If Spitzer loses the Comptroller race, full credit should go @QuinnipiacPoll, the only organization to favor his rival Stringer. [2/2]

There were several polls of this race as you can see at this link to the Real Clear Politics polling page. There was nothing that made this look like a close race until late August, with Eliot Spitzer's hopes for a political comeback looking very strong while Manhattan borough president Scott Stringer seemed unable to make any headway. But then Quinnipiac had a poll that said the race was tied, then another Quinnipiac poll had a small of 2% for Stringer on September 1, then a commanding 7% lead a week later.

The actual result was Stringer by 4%, 52% to 48%.

The reason my system - and Professor Wang's - favored Spitzer was that no other pollster agreed with Quinnipiac, not even once. After the first Quinnipiac poll that showed a close race, Siena, Marist and PPP all released polls, all of them favoring Spitzer.

So my system failed to predict this race. Often, when I get a race wrong, I try to find ways to adjust things to improve, but if I was given a similar situation tomorrow, I wouldn't change a thing. Here are my reasons.

1. I never weigh in more than a single poll from any polling company and always the most recent.

2. Because Quinnipiac only gets one result counted, they had one poll out of three in the final mix, along with PPP and Siena.

3. In a three poll situation, I take the median result, not the average of the three. Quinnipiac was the outlier, but outliers aren't often the closest to correct. Obviously, it will happen sometimes, but it is rare enough that I do not want to second guess myself in a situation like this. Polls miss the mark, sometimes by making bad assumptions, sometimes just by the fact that randomness is involved. For example, of the three polls Quinnipiac had the largest sample size, but that does not always (or even often) mean the most accurate. Looking at the mayoral polling and factoring out the undecided, Quinnipiac had De Blasio at 46% of the people who had a preference, while PPP had him at 42% and Siena had him at 39%. In this case, it's the low outlier that was closest.

So this means at least one more report on the mayor's race. It is widely agreed that the Republican primary winner Joe Llota will be very hard pressed to win in the general election. De Blasio got 260,000 votes in the Democratic primary, Llota finished first with over 50% in the Republican primary with about 30,000, which would have given him sixth place in the Democratic race.

And while I did not use their poll, I want to congratulate Quinnipiac for being alone with the correct result in the comptroller's race. It is a field filled with randomness, but they had Stringer with a chance or the lead three separate times in barely two weeks when no one else did. That's a record they can point to with pride.

New York City Mayoral Race: Democratic PrimaryThe Weekend before the election.

On Tuesday, New York City voters will go to the polls to decide the Democratic and Republican candidates to be the next mayor. The conventional wisdom is the Democratic nominee holds a huge advantage over whichever candidate the Republicans put forward, so the Democratic primary is being given the lion's share of interest.

Since mid-August, just after Anthony Weiner's habit of sending portraits of his penis to strangers became common knowledge, the front runner has been Bill De Blasio. It is widely agreed that the ads featuring his 15 year old son Dante, he of the remarkable adolescent baritone and even cooler Afro, have made quite the impact.

This Sunday, a new poll from Marist has been  published which agrees with the general trend if not the exact numbers.

Candidate recent % (previous %)

Marist - 9/6 recent, 8/14 previous
De Blasio 36% (24%)
Thompson 20% (16%)
Quinn 20% (24%)
Other 16% (21%)
None of the Above 8% (15%)

Quinnipiac - 9/1 recent, 8/12 previous
De Blasio 43% (30%)
Thompson 20% (22%)
Quinn 18% (24%)
Other 12% (17%)
None of the Above 7% (7%)

Siena - 8/28 recent, 8/7 previous
De Blasio 32% (14%)
Thompson 18% (16%)
Quinn 17% (25%)
Other 16% (19%)
None of the Above 17% (26%)

The agreement is across the board. De Blasio had a great September, Quinn took a beating and Thompson improved enough to probably be the favorite for second place if that matters. De Blasio is seen as the progressive candidate and the candidate for the boroughs other than Manhattan. Even the fact that De Blasio is a Red Sox fan may not be enough to stop him.

Of the three polls, Quinnipiac has been the one that has found the most support for De Blasio and Siena has lagged, but that may be due to Siena being first to polls in each case. That leaves Marist, last to poll and the median poll for De Blasio support both mid-August and early September.

If we accept Marist as the most reliable because of being the median, there are three outcomes that look possible on Tuesday.

De Blasio gets more than 40% on the first ballot: 34%
A De Blasio-Thompson run-off: 36%
A De Blasio-Quinn run-off: 30%

If I were putting a wager on this three way outcome, I'd put my buck on De Blasio-Thompson in a run-off, though any of these three results can hardly be called an upset. The only thing that would make me doubt my sanity or the validity of my methods is De Blasio finishing third. There hasn't been a poll in more than three weeks that shows that result to be even close to plausible.

I'll be back on Tuesday evening to report the actual vote.

The New York City Mayor's race:One week from election day

There were a flurry of polls in the New York mayoral race in mid August and then nothing until last week. Now, we have four new polls, but two of them are from the same company, Quinnipiac, so I only count three, using the most recent Quinnipiac, one from amNew York-News 12 and a third from Siena, the polling company hired by The New York Times.
 

Both Quinnipiac and Siena polled mid-month and late month, so I will give their numbers in the form

Candidate recent% (previous %)

Quinnipiac - 9/1 recent, 8/12 previous
De Blasio 43% (30%)
Thompson 20% (22%)
Quinn 18% (24%)
Other 12% (17%)
None of the Above 7% (7%)

Siena - 8/28 recent, 8/7 previous
De Blasio 32% (14%)
Thompson 18% (16%)
Quinn 17% (25%)
Other 16% (19%)
None of the Above 17% (26%)

Both of these polls agree that August was a very good month for De Blasio and a bad month for almost everyone else. We also have a poll from amNew York-News12, who only polled once in August.

amNew York-News12 - 8/27 recent, no previous
De Blasio 29%
Thompson 24%
Quinn 17%
Other 17%
None of the Above 13%

The "median" poll can be a little difficult to judge in a multi-candidate contest. If we consider the race De Blasio vs. Thompson, the Siena poll is the median with a 14% lead.  In this case, De Blasio has about a 27% chance to cross the 40% threshold and win outright, and the Siena poll says the race for second is relatively close, but still gives Thompson a 65% Confidence of Victory over Quinn.

Here are the Confidence of Victory numbers from all three polls.

Quinnipiac
De Blasio outright win: 99.9%
De Blasio-Thompson run-off: 0.08% 
De Blasio-Quinn run-off: 0.02%

Siena
De Blasio outright win: 27.2%
De Blasio-Thompson run-off: 47.3% 
De Blasio-Quinn run-off: 25.5%

amNew York-News 12
De Blasio outright win: 0.1%
De Blasio-Thompson run-off: 99.9% 
De Blasio-Quinn run-off: 0%

There should be at least one more poll from Marist by the end of the week, but unless Siena is right, Quinn is pretty much out of the race.

 

Some thoughts about election prediction.

It's been nearly two weeks since there has been any new data on the New York City mayoral race. The last few polls agree the three at the front of the pack are Christine Quinn, Bill De Blasio and Bill Thompson. Unless one of them gets over 40% of the vote, and the recent polls make that look very unlikely, the top two will go into a runoff election, and the odds make it look like an uphill climb for Thompson.

The New York Times endorsed Quinn. Her bio is certainly compelling from a progressive standpoint. She would be the first female mayor of America's largest city and she would be the first openly gay mayor. Her record on the issues is not as compelling from a progressive standpoint. She is seen by many as a person who would continue the Bloomberg status quo. The Nation, a truly progressive periodical unlike the Times, has endorsed De Blasio.

Nate Silver has published numbers saying that history favors the long term front runner of the year, which would be Quinn in this case. She had competition from Anthony Weiner after he announced and before his spectacular crash, but for months before that she lead in every poll.

This is one of the many places in predictions where I part company with Silver. I assume that every election stands on its own and historical data should not outweigh current polling.

It's impossible to predict the big events and impossible to gauge the weight they will have. It looked like based on name recognition Anthony Weiner was a serious candidate, but the Carlos Danger situation cut his support in half, his position plummeting from first to a distant fourth. A judge deciding "stop and frisk" was unconstitutional gave De Blasio a big boost, but it's hard to say if it's a surge or a bounce until we get more data. The Times endorsement should be much more important that The Nation's, but are either of them really events that make a difference in the polls?

I have no idea.

When I say that the last polls look like an overwhelming likelihood of a Quinn/De Blasio runoff, I am completely willing to say something else if the next polls change. It's very hard to say what events will change the polling numbers, known in journalistic circles as "moving the needle".

In 2012, Romney's 47% comments moved the needle against him, but he was already in a horrible position when he said that. Obama's awful first debate definitely moved the needle in Romney's favor, but by the time of the Biden/Ryan and the second Obama/Romney debate, the movement towards Romney had stopped and the tide reversed. He never was favored to win.

Conservatives hoped Benghazi would move the needle. It didn't.

Silver went to Princeton and I went to Cal State Hayward. Silver was hired by the Times and I haven't been able to get anyone to pay me for my work. In my favor, I did better than he did in 2008 and 2012. (He missed Indiana in the electoral vote in 2008 and two Senate races in 2012 that I got right. One of them he missed by giving weight to historical data, something I would never do.)

Here's the big difference between us as far as I'm concerned. My education is in pure mathematics and his is in applied.

Election prediction is applied math. I deeply distrust it and I wait until I think the data really means something. I have arbitrarily chosen about a week before the polls close. By then, I figure enough mail-in ballots have been cast that the most important proviso of any opinion poll is actually true, that "the election is being held when the poll was taken".

Is seven days the right number? Maybe ten would be better, maybe four. If my predictions become less reliable in the future, changing this time period is one of my first places to look to improve my system.

This brings me to something else Silver has done that I disagree with vehemently. He is predicting the make-up of the Senate in 2014 and giving a margin of error. These numbers are being used as scare tactics by the Democrats begging for money.

As far as I'm concerned, these numbers are worse than nonsense. He does actual work, so I am not happy comparing him to lazy, ignorant anti-intellectuals like Dick Morris, Karl Rove, Jim Cramer or George F. Will, but predictions about a multi-part election more than fifteen months away from polls closing are mere balloon juice. Even if someone "crunches numbers", given how many unknowable events there will be in the next year and a quarter means his work has no more meaning than doing exact bio-rhythms or a complete astrological chart.

I do not know if Nate Silver will ever read what I write. If he does, I want to say to him, older nerd to younger nerd, stop doing the stuff that is most likely to hurt your reputation. A fifteen month prediction about elections is no better than your work in fantasy football or Oscar prediction, which you have been gracious enough to admit has been less than optimal.

Wait until just before the election. Then the numbers of people doing actual work, work like yours and mine and Sam Wang's, have meaning. Right now, your work in 2013 is like Gene Simmons of KISS going out on a limb in 2011 predicting President Rick Perry and giving us as the reason to believe him, the quote "I'm never wrong."

He was wrong. Even with your margin of error, saying there is 95% confidence in your numbers is plain silly.

Stop being silly. You are much better than that.

Here endeth the lesson.

Another New York City mayoral poll.

A new poll for the New York City races was released Thursday night, this one from Marist, yet another small East Coast college that does polling.(The earlier polls were from Qunnipiac and Siena, also small East Coast colleges.)

The latest poll, completed on August 14 sampling 679 Democrats, puts City Council Speaker Christine Quinn and public advocate Bill DiBlasio in a tie at 24% and former comptroller William Thompson in third with 16%.

The rules of the race are that if any single candidate polls over 40%, he or she is declared the winner and goes onto the general election. If not, the top two vote getters are in a run-off.

If the election were held when the poll was taken, one candidate getting over the 40% mark is highly unlikely, so it's really a race for the top two spots. A week ago, that looked like Quinn and Thompson, but the two polls this week say Quinn and DiBlasio. The event that is considered the turning point is the decision by a judge that "stop and frisk" policies are unconstitutional and DiBlasio's long held objection to the policy in New York City is his key advantage.

Marist also polled the Republican race, where Joe Lhota has a large lead but not polling over 40% yet. He could win without need for a run-off depending on how the undecided vote breaks, but he is not given a very good chance against whoever survives the Democratic race.

There will certainly be more polls between now and the election and I will give updates when available.

More election news:Results from New Jersey, new polls for New York City mayor

The results from the primary for the New Jersey special election came in and there were no surprises. Cory Booker won the Democratic nomination handily with 59% while his closest rival had 20%. Republican Steve Lonegan beat his single opponent Alieta Eck 79% to 21%.

The Booker vs. Lonegan election will be held on Wednesday, Oct. 16. Two months is a long time in politics, but currently Booker is leading comfortably in all the polls that have asked about this particular match-up.
  

The New York City mayoral race is much more up in the air. The Democratic primary polls taken this month by Siena and Quinnipiac don't agree on much except the fall of Anthony Weiner from the top tier of candidates.

The rules of the primary state that if a single candidate gets more than 40% of the vote, he or she is declared the winner. Otherwise the top two candidates are in a run-off. Here are the results for the top three candidates in the most recent polls.

Siena (8/2 to 8/7)
Quinn 25%
DiBlasio 14%
Thompson 16%

Quinnipiac (8/7 to 8/12)
DiBlasio 30%
Quinn 24%
Thompson 22%

My Confidence of Victory method does have ways of turning each of these sets of numbers into probabilities of victory for all three of these candidates, and as you might expect the Siena numbers from earlier in the month would make Quinn the prohibitive favorite while the Quinnipiac number would hand the big favorite role to DiBlasio.

Quinn has been the favorite for several months, never polling less than second. Similar to the 2012 Republican presidential primary, there is a remarkable amount of what I call "churn" in the numbers. If I stretched the analogy a little farther (perhaps too far, I can't be sure), this would put Christine Quinn, the openly gay speaker of the New York City Council in the role of Romney, while Weiner and DiBlasio, the  New York City public advocate and considered the most liberal candidate, in the roles of the many Not Romneys that the Republican field had to offer.

The last five elections for mayor have been won by a Republican (Giuliani) and a former Republican turned Independent (Bloomberg), but the current assumption is that the star power is gone from the Republican ranks currently and the Democratic primary winner will move to Gracie Mansion.

I've read analyses by Harry Enten and Nate Silver, both of whom are very keen on historical track records. Maybe it's because I'm older than either of them or I've just seen so many predictions done on flimsy data turn out so badly, but I'm not convinced the historical record is the right thing to look at. An issue like "stop and frisk" has a strong chance to be a game changer, and that would work to DiBlasio's advantage.

Enten believes it's all about ethnic politics, which gives Thompson,who is black, an edge over DiBlasio. Silver thinks Quinn's position as front runner for most of the year gives her the inside track. I claim no expertise about New York politics, but I believe more strongly in event driven elections being the correct modern model and that historical data is often so much balloon juice.

I'll post again when there is new data.

I will keep track of these races through the primary on September 10, the likely run-off on October 1 and the general election on November 5.