Poll Reliability

[realtraveller]

Everyone wonders how reliable the polls are. The exit polls in 2004, for example, weren't reliable at all. Now there have been questions about whether cell phone only users are polled. That has been answered, yes, they are polled and they don't poll any differently than those with land lines.

The one group that can never be polled are the 'hang-ups'. People who are called by a pollster and hang up either because they are too busy, or are too annoyed (the Do Not Call effect) to answer the poll questions.

If anyone sees or has seen any reasonable discussion of this, please post it. Or any speculation by any of you. Are both Republicans and Democrats even split on the 'hang-up' effect, do you think? What about Independents? If you are undecided would you not bother to answer the poll questions, because you hadn't made up your mind?

I was called by Rasmussen a month or two ago and I took the time (they caught me at a good time) to answer the five or so questions. And then I am interested in politics, and so I was interested in what questions they were going to ask.

[emeleel]

I always say no unless they specifically say up front it's only a 1-3 question survey. And even then I might say no if they call at the wrong time - dinner or something.

[mjfrombuffalo[$]]

I've never been called. Regarding cell phone users, not all of the polls reach cell-only households. More details: fivethirtyeight.com, a site for people who really like statistics

[emeleel]

I've never gotten a survey on my cell, and I'll be really ticked if I ever do because I use a TracFone.

[jgibson2[$]]

Depending on my mood, I either hang up on most pollsters or lie to them.

Occasionally I'll answer the questions straight, but if they are using push-poll techniques, I ask if the person asking the questions knows the definition of a push poll and then tell them that I do not participate in that sort of crap.

[drmomentum[$]]

Is there a theory on the importance of hangups? I'm not familiar with that as a specific problem. It seems like the sort of thing that wouldn't necessarily affect the outcome of a poll (but could, in certain circumstances).

Who knows if polls are reliable? People only say the polls were unreliable in a certain year if, after the election, we see the election go a different way. Polls are based on history, which is the best measure we have, but sometimes pollsters are unprepared for changes in the landscape.

When a candidate is down in the polls, his or her supporters wring their hands over it and question poll numbers. When a candidate is up in the polls, supporters take comfort in it. Either way, people are watching the news.

This is the purpose of polls.

-JP

[realtraveller]

Polls can be wrong for a variety of reasons either:

1) People enjoy lying to pollsters. (Not an unreasonable assumption). To the extent that polls are linked to media organizations, i. e. a pollster calls a Democrat and says "This is a Fox News opinion poll" or a CBS (known as See B. S. News on conservative websites) calls a Republican the results could be skewed.

2) Polls could be off because those who choose not to participate in polls could be leaning on way or the other. And I have no idea how pollsters would even control or test for this effect. You can't do a control sample on those who hang up on pollsters to even find out the types of voters who hang up on pollsters.

[drmomentum[$]]

True. But both of those points are only reasons for inaccuracy if the pollsters are unaware of the extent to which they are occurring. Lying to pollsters and opting out can be adjusted for.

For instance, researchers can sometimes tell when a respondent is lying. Certain questions can be added to the poll which are indicators (when the responses are taken together) indicate a bad response.

In the end, we're still talking about the skill of pollsters, or lack thereof.

Some of this is also being realistic about the margin of error in polls. Aside from margin of error, changes in polls can be within the noise level. If a poll is not changing rapidly, then I sometimes find the trend more compelling than the actual numbers.

But in these presidential polls, they seem to fluctuate so much that I don't think you can necessarily read a reliable trend.

In general, I have a low opinion of poll accuracy.

[magenta321]

I've never been polled. If someone attempted to poll me, I think I'd be so afraid of it being a scam that I wouldn't talk to them.

I mean no offense when I say this, but I think elderly people are more likely to respond to polls. For one, they're easier to take advantage of in general (that's why so many phone scams target the elderly), and two, I think many are looking for someone to talk to, and much more likely to give the pollster the time of day.

Based on my own, non-scientific assumptions, I would think the data would be somewhat unreliable.

[jgibson2[$]]

Despite my tendency to lie to pollsters or just hang up on them, I do believe that if you ask ENOUGH people you can find out broad generalities. IOW, if a candidate is up by 10 points, the poll is probably showing a trend. If he's up by 2 points, things are close, but you can't really predict the election based on that.

Depends on how your poll is structured, of course.

[realtraveller]

When I was polled by Rasmussen, it was a recorded message. It was press 1 for this or 5 for that. I don't know if it's easier to lie to a recording or whether people would be more likely to tell the truth since they aren't giving opinions to another person. But in any event, there is no person on the other end to try and figure out if you are being truthful in your answers or not.

The technology does cut out those people who still have dial phones. Don't laugh. Some older people I know still have them.

[jgibson2[$]]

I hang up on recorded messages EVERY time. I doubt that's a trait which tends toward one political POV, though.

[brian_igo]

realtraveller said

Everyone wonders how reliable the polls are. The exit polls in 2004, for example, weren't reliable at all. Now there have been questions about whether cell phone only users are polled. That has been answered, yes, they are polled and they don't poll any differently than those with land lines.

The one group that can never be polled are the 'hang-ups'. People who are called by a pollster and hang up either because they are too busy, or are too annoyed (the Do Not Call effect) to answer the poll questions.

If anyone sees or has seen any reasonable discussion of this, please post it. Or any speculation by any of you. Are both Republicans and Democrats even split on the 'hang-up' effect, do you think? What about Independents? If you are undecided would you not bother to answer the poll questions, because you hadn't made up your mind?

I was called by Rasmussen a month or two ago and I took the time (they caught me at a good time) to answer the five or so questions. And then I am interested in politics, and so I was interested in what questions they were going to ask.

There are some differences with polling cell phone users. Federal law prohibits automated calls like your Rassmussen poll to cell phone numbers because of telemarketing and cell users being billed for received calls. If you want to telemarket to a cell user, or poll them, you have to use a live person and that greatly increases the cost.

Exit polls in 2004 were actually pretty good - when they were used to call a race before the actual outcomes made it obvious. Part of that is because every media outlet was gun shy to call a state unless the reported results matched up with their exit polling, even if the reported results showed clear leads throughout the night. You saw a lot of this in the primaries this year, too, and I'm sure you'll see a lot of it again on November 4.

Hangups aren't a problem in political polling. They don't represent one group alone so their refusal to participate doesn't skew the results. If voters of one party are so discouraged by their candidate, or their candidate's chances, that they as a group don't want to talk about it with a pollster - they probably don't have a prayer of winning in the first place. (For example, an Obama supporter in Utah has good reason to think talking to a pollster is a complete waste of time.)

The art of polling is in creating the model a pollster uses to calculate the results. You can break this down every way imaginable, but the big one in this cycle is the split between Republicans, Democrats and Independents. In 2000 and 2004 it was close to an even split between the three. But polling since 2004 and post-race analysis of the 2006 midterm polls show a substantial swing away from Republicans and towards Democrats.

But no one can say definitively say the new breakdown is x R/ x D/ x I until after the election. This is hugely important because it directly affects the outcome of the poll. If your poll says that the electorate is 32R/39D/29I, and my poll uses a model of 35R/38D/27I, I'm going to have more Republicans in my sample and it's likely that I'll think it's a close race.

A few times this year there have been huge swings in poll results generated by a change to the model. There was a CBS poll a couple of weeks ago that showed McCain going from a tie race to an eight-point lead in just a couple of days after his convention. I dug around in the .pdf's and found that the model had been adjusted between the two polls to give a six-point swing from Dems to Republicans in the model. I don't think the polling was malicious, but it was definitely sleazy for CBS to hype these results without also reporting that the model was changed.

Another question about polling is the breakdown between registered voters and likely voters. If you're polled by a company doing a likely voter poll and you haven't voted in the last couple of elections, your answers are probabaly going to get flushed. That's a problem in a year like this, when there is evidence of a record number of first time voters. In Indiana there have been more than 300,000 updated registrations since January 1, and over 200,000 new registrations. That's a huge number for a state that cast just 2.5 million votes in 2004. How do you poll that group? You don't know how many of them are going to actually vote.

My educated guess is the change in the party identification and Obama's voter registration campaign is going to turn some battleground states into 4-6 point wins that never show up in the polls. If the election were held today, I think Obama would win Indiana, Virginia and possibly North Carolina, which most polls currently show McCain leading by a couple of points. I don't see any states McCain can take from Obama.

[realtraveller]

brian_igo said

Hangups aren't a problem in political polling. They don't represent one group alone so their refusal to participate doesn't skew the results. If voters of one party are so discouraged by their candidate, or their candidate's chances, that they as a group don't want to talk about it with a pollster - they probably don't have a prayer of winning in the first place. (For example, an Obama supporter in Utah has good reason to think talking to a pollster is a complete waste of time.)

Which brings me to my original point. There is no way for pollsters to know if the hang-ups are skewed one way or another or if they aren't skewed any particular way. And then you undermine your own argument by saying that there may be instances in which one group simply doesn't want to talk to a pollster.

The early exit polls in 2004 were very flawed. I remember hearing that South Carolina (SOUTH CAROLINA!, the reddest of red states) was too close to call. In the days that followed I remember reading that people simply didn't want to talk to the exit pollsters.

[lynnzop[$]]

Wouldn't the liars be split somewhat equally on either side?

So if I lied about wanting to vote for candidate X and my husband lied about wanting to vote for candidate Y, we'd cancel each other out.

[brian_igo]

realtraveller said

Which brings me to my original point. There is no way for pollsters to know if the hang-ups are skewed one way or another or if they aren't skewed any particular way. And then you undermine your own argument by saying that there may be instances in which one group simply doesn't want to talk to a pollster.

The early exit polls in 2004 were very flawed. I remember hearing that South Carolina (SOUTH CAROLINA!, the reddest of red states) was too close to call. In the days that followed I remember reading that people simply didn't want to talk to the exit pollsters.

There are reason why one group might be predisposed to not talk to pollsters, and I think I covered a couple of good examples. But I'm still not aware of any plausible scenario where it might be a factor in a competitive race. When the contest is tight, people who care do not typically hang up the phone.

You also ignore the modeling that I explained. If a pollster is working with a model that California is 31% self-identified Republican, that's going to be the ratio of Republicans who completed the survey in the final model. If you hang up, they'll keep going until they find enough Republican respondents to fill out the model. A poll that uses 900 registered voters will probably require more than 3,000 calls.




About South Carolina, I don't know what you're talking about. According to this USA Today blog, the AP called South Carolina for Bush at 8:44pm - which was one hour and 45 minutes after the polls closed. That means <10% of precincts would have been reporting when the state was called for Bush, and I'm being generous. The actual returns that were in at the time the race was called was probably closer to five percent.

If SC was reported as too close to call, it wasn't reported that way for very long.

In any case, "too close to call" can also mean insufficient or conflicting data between the exit polling and bellwether results being reported from election officials. That's not blowing a call like happened in 2000, it's just saying that we need to watch the official results for a while longer.

Lynnzop said

Wouldn't the liars be split somewhat equally on either side?

So if I lied about wanting to vote for candidate X and my husband lied about wanting to vote for candidate Y, we'd cancel each other out.

Yeah, Lynn. And you're only one person out of a thousand. To have any statistical effect, you'd have to organize other liars in the pool, too.

There are also cross-checks pollsters run on the data. There can be big differences between the party affiliation samples in their models. But if you go one layer beneath that, to the support each party candidate has among self-identified members of that party, that's a very solid number. The support for McCain among self-identified Republicans is very consistent across different polls. Same for Obama and Democrats.

Pollsters use this as a check on the veracity of their data. A successful lying campaign would skew these results from the norm and that would be obvious red flag to the pollster. For example, if Californians wanted to change the polling results to make the race appear more competitive, they'd have to raise the support for McCain among Republicans and independents to levels that are going to be different from other polls - unless you get all of their respondents to lie the same way.

[lynnzop[$]]

Brian, this is exactly why I don't "do" math. In my brain, you'd have a somewhat equal number of liars on either side of the issues (I'm not going to debate whether Republicans or Democrats lie more), so the liars answering to one side of a poll are going to negate the liars on the other side.

I'm no statistician (obviously)

[jgibson2[$]]

Brian,
People who care about WHAT don't hang up?

People who care about the outcome of the election? I'd have to call bullshit onthat one.

People who care about having their dinner, quiet time, conversation, etc interrupted? Yep. we hang up. I get really tired of being called for push-polls and robo-polls. I will not participate with either. I hang up on robo-polls immediately and I give the push-poller an earful as soon as they go into push-poll mode which is generally about the third question.

I would speak to a live person who is asking reasonable questions, but I really cannot recall that happening. Ever. I know that live people are more expensive, but if you want to know what I think, look at the election results. That's the only poll that counts anyway. And sometimes even that doesn't.

[brian_igo]

jgibson2 said

Brian,
People who care about WHAT don't hang up?

People who care about the outcome of the election? I'd have to call bullshit onthat one.

People who care about having their dinner, quiet time, conversation, etc interrupted? Yep. we hang up. I get really tired of being called for push-polls and robo-polls. I will not participate with either. I hang up on robo-polls immediately and I give the push-poller an earful as soon as they go into push-poll mode which is generally about the third question.

I would speak to a live person who is asking reasonable questions, but I really cannot recall that happening. Ever. I know that live people are more expensive, but if you want to know what I think, look at the election results. That's the only poll that counts anyway. And sometimes even that doesn't.

Call whatever you want. People who live in a competitive state and who are engaged in politics typically have a very high response rate when polled. With all respect, you don't live in a competitive state and you lean towards supporting the party that usually loses that state. That's the hardest sample to fill because your reaction is typical. Same for Kathy, or the Democrat in Utah that I cited originally.

Beyond that, voice interviews do get higher response rates than automated calls, but automated calls still cost much less even though they have to make a lot more calls to fill out the sample. But there's no measurable difference in accuracy. One of the most reliable polls in the 2004, 2006 and 2008 cycles has been Survey USA, and they only do automated polling.

Push polling is something entirely different.

If you can't ever recall being asked reasonable questions, then you've probably never been polled by one of the national polling firms that work for the newspapers and networks. Here is the WaPo/ABC poll that's out today, with the exact language of the questions. This is typical for the national media-sponsored polls. Also note that the Post doesn't change their questions from poll to poll so they can establish the timeline.

[pisces]

The media is driving the polls. I have no proof, but I suspect that lots of polls are rigged for the Media's favorite candidate, especially when results are so close, and a biased Media can use a margin of error to toss out, or toss in, the number of votes need to put their favored Candidate over the top.

[jgibson2[$]]

brian_igo said

Call whatever you want. People who live in a competitive state and who are engaged in politics typically have a very high response rate when polled. With all respect, you don't live in a competitive state and you lean towards supporting the party that usually loses that state.

Only in the presidential election. We end up with a Republican governor from time to time and I live in a very red county when it comes to local government, representation in the state house and the US House of Representatives. As for engaged in politics, I KNOW out county executive, our county council representative, and all 4 of our representatives to the MD House and Senate. Know as in they introduce my husband and I to people when we run into them in public.