Polls Are Getting Harder and More Expensive to Get Right, and That’s Not Changing

On Tuesday, March 8th, Bernie Sanders defeated Hillary Clinton in the Democratic primary in Michigan, garnering 50 percent of the vote to Hillary’s 48. On its surface, that statement doesn’t seem shocking or surprising. But other than the continued success of Republican front-runner Donald Trump, Bernie’s win has perhaps been the most surprising result of this primary season, if not the most surprising in recent memory.

Why? Because across the board pollsters had Hillary winning Michigan, and winning big.

As Harry Enten notes on FiveThirtyEight:

“Not a single poll taken over the last month had Clinton leading by less than 5 percentage points. In fact, many had her lead at 20 percentage points or higher. Sanders’s win in Michigan was one of the greatest upsets in modern political history. Both the FiveThirtyEight polls-plus and polls-only forecast gave Clinton a greater than 99 percent chance of winning.”

There are numerous other recent errors in polling I could point out, from Gallup incorrectly predicting a Romney win in the 2012 presidential election to House polls that showed when one candidate was leading a contest by between 5 and 10 percentage points, they were wrong 23 percent of the time.”

In spite of this, major media outlets routinely insist on reporting on polling on an almost daily basis. This gives a horse race cadence of the election season, with media being the race announcers. While that may be fine if there wasn’t a real impact from it, polling data has at times influenced the amount and type of coverage candidates receive in the media, leading to real effects on how candidates are perceived and their exposure to the electorate.

It’s not just the media either. In the primaries, candidates and their supporters are already touting hypothetical general election polling as a reason that their candidate should be the party nominee over the other, and seemingly indiscriminately tossing around any other polling data they can find.

So what in fact is wrong with polling? The short answer is lots of things.

First of all, the predominant polling method in the modern era has been to call individuals on their landlines. Pollsters randomly select a set of survey respondents and then call them repeatedly in an effort to get an answer. If they can’t reach those voters by the sixth try they then have to move on to a new respondent. The problem is that this method is proving less and less effective because so few Americans actually have landlines anymore. At the beginning of 2014, 41% of US households were wireless only, meaning they didn’t have a landline telephone. So when pollsters used traditional methods to try to conduct polling, they weren’t able to access almost half of the country.

So why don’t they just switch to calling cellphones? Well, there are two main reasons. One, a lot of the tech advances that have kept telemarketing and polling cheap, like auto-dialing (in which a computer will automatically dial multiple numbers, connecting the caller to someone whenever they pick up) doesn’t apply to cellphone. Federal law requires that cellphones must be dialed by hand, which makes them much more expensive and time consuming to call. The law also bars pollsters who solely use cheaper automated-recording calls — “Press 1 if you are supporting Hillary Clinton” — from including cellphones in their polls.

This comes at a time when news budgets for the media organizations that underwrite much of the polling have shrunk considerably. Secondly, it’s harder to get a representative sample of the population in a given area using cellphones. Cellphones aren’t implicitly tied to an area code, and that creates issues for the modeling pollsters use. For example, while my cellphone area code identifies me as living in the Hudson Valley, I live in Brooklyn.

A second issue with polling is that, well, even when pollsters do connect with people, they aren’t responding or finishing the polls. As Cliff Zukin, a policy and political studies professor at Rutgers, writes in the New York Times:

“When I first started doing telephone surveys in New Jersey in the late 1970s, we considered an 80 percent response rate acceptable, and even then we worried if the 20 percent we missed were different in attitudes and behaviors than the 80 percent we got. Enter answering machines and other technologies. By 1997, Pew’s response rate was 36 percent, and the decline has accelerated. By 2014 the response rate had fallen to 8 percent. As Nate Silver of fivethirtyeight.com recently observed, ‘The problem is simple but daunting. The foundation of opinion research has historically been the ability to draw a random sample of the population. That’s become much harder to do.’”

He goes on to note that this has huge impacts on—once again—the costs of polling. If you’re a polling company that, because of the low response rate, now has to pay interviewers to complete “X” number of interviews, they’re going to be on phones longer when the response rate is only 8 percent, which means you need to pay them more.

It’d be great if you could use an auto-dial to make this process faster and cheaper, but as previously mentioned, then you can’t call cellphones. Mark Schulman, a co-founder and research chief at Abt SRBI, previously estimated that interviewing costs in 2016 would be more than twice what they were in 2008.

The seemingly obvious situation here would be to use the internet to conduct polling, as that would capture wireless households and also seems like it would be cheap, right? Wrong. Polling grew up around the same methods that were used for going door to door to measure public opinion, and that doesn’t translate well to the Internet. First off, they can’t call you; you have to come to them. Additionally, we run into a problem similar to the one we ran into with landlines. That is, not everyone uses the Internet, while those use it consistently are younger and more liberal than people who don’t.

Finally, and perhaps it seems cliché, but polls only reflect a single moment in time. They generally catch people at home, while they’re making dinner, the kid is screaming, and after they’ve finished a long day of work (trust me, I’ve been the one calling them before). What they are thinking at that point and time can be quite different from what they’ll decide in the voting booth, as polls don’t capture developments following them. For example, it’s not farfetched to argue that Bernie Sanders’ performance in the debate in Flint a few days before the Michigan primary shifted voters views on him.

So with all this in mind, why do we continue to pay so much attention to polls? They’re so much of a staple of elections that it’s hard to imagine life without them. It seems we need a winner and we need to know who that is, even before the race has been finished. As Zukin notes at the end of his article: “In short, polls and pollsters are going to be less reliable. We may not even know when we’re off base. What this means for 2016 is anybody’s guess.”

The lesson: Don’t trust the winner when it’s called in advance, because it’s not always going to be right. Now, back to the races.