Over at CatholicCulture, Uncle Di has an interesting post about conflicting poll numbers in Massachusetts concerning which candidate is likely to win Ted Kennedy's former seat.
Last week, a Boston Globe poll of likely voters show the Democratic candidate, Martha Coakley, leading the Republican, Scott Brown, by a comfortable margin: 50- 35%.
A Public Policy Polling survey of likely voters, released the same day, showed Brown ahead, 48- 47%.
The Globe poll claimed a margin of error of +/- 4.2%; the PPP poll said its margin of error was 3.6%. Go ahead: try the numbers. They don't work.
Wait; there's a possible explanation. The Globe poll was taken January 2- 6; the PPP poll was January 7-9. So you might say that as a Little Christmas gift, Scott Brown got 13% of the likely voters.
Alternatively, you might say that there's a margin of error to the pollsters' margin of error.
Di is exactly right that there is a "margin of error" to the pollsters' margin of error–a margin that pollsters very seldom talk about.
What pollsters mean when they say that a poll has a margin of error of 4 percentage points (or whatever number) it does not mean that the true figure is really within 4 percentage points of the figure they name. They have no independent way of knowing what the true figure is. All they can do is estimate what the true figure is based on the sample of data they got.
But sometimes you get unrepresentative data, which is what the margin for error is supposed to allow for. It's a fudge factor that means, in essence, if we ran the same poll a bunch of times, the result would vary but would tend to remain within the stated margin of error.
Yet sometimes you get really unrepresentative data, and this is what pollsters don't generally point out.
In standard polling, the margin of error is based on what the result would be approximately 95% of the time, or nineteen out of twenty times. ( . . . keeping this simple so we don't have to get into standard deviations and normal distribution and confidence intervals and other technical minutiae).
So, for example, if pollsters went out and polled the right number of people to give them a 4 percent margin of error, and 48 percent of the people said that they'd vote for Candidate X then what this means is that if you re-ran the poll that nineteen out of twenty times the result you would get back should be between 44 and 52 percent, all else being equal.
But one time out of twenty the result you would get back would be wildly off, either below 44 or above 52.
So . . . bear that in mind when looking at poll numbers.
Even when the poll is properly done, one in twenty polls produces a reading so anomalous that it falls outside what the margin of error would be if you ran it another nineteen times.
On average.
We think.
There were big differences in the samples used for both polls. I think the number of independants was much higher in the PPP poll, for example. I read an article that broke down how the samples differed. I will post it here if I can find it.
Really, what it comes down to is which poll better reflects the population that will turn out to vote?
http://charliefoxtrotblog.blogspot.com/2010/01/boston-globe-coakley-up-big.html
Here is a blog post that breaks it down.
One mustn’t forget what we (or some of us) in Chicago know as the Royko factor. Mike Royko was a famous columnist for various daily papers over the years. The practice of exit polling by the networks and then calling the result of a given race after less than 2% of votes were counted (or even before the polls closed) burned Royko big time. His solution? Encourage his readers to lie to the pollsters. It messed up their polling and irritated them, since they then had to add a second fudge factor into their results.
It’s probably not an issue in Boston.
The “Royko factor” – I’ll have to remember that one. Though I don’t think there’s any way lies to pollsters can be considered morally justifiable.
Yeah, I know. But I don’t think we’re morally required to answer their questions if we so choose…
You’ve correctly described the coverage properties of confidence intervals. However, one of the major sources of unaccounted uncertainty in such estimates is the failure of the assumption of random sampling. It is surprisingly hard to obtain a true random sample of the population (or the population of voters)!
I would guess that polling during normal working hours versus polling at night, etc., should provide some great variation in the results. I don’t really know much about polling, but I’ve heard “stories” of polls being conducted in these ways.
Does anybody know how common that type of tactic is if its employed at all? Or if there are watch dog groups that try to keep polling somewhat neutral?
A corollary to the Royko factor that I’ve seen reported on from time to time here in Chicagoland is that people who are thinking of voting (or in exit polls, have voted) for the candidate from the party-out-of-power will sometimes lie to the pollster, lest the wrong people find out.
In this case, however, what Simon suggests about sample bias (unintentional or otherwise) is probably the biggest factor. There can be problems in how the sample is set up and how it is collected. It can be a problem getting people to answer the survey for a variety of reasons; the more difficult it gets, the more expensive the poll will be, so cost is a factor at least sometimes.
Jimmy, great job explaining something that I wish everyone else already knew. You made this political scientist quite happy!
Also, with regards to the Royko factor, don’t forget: it really wouldn’t matter much if people started lying en mass to pollsters, as long as both Republicans and Democrats were just as likely to lie at the same rates – they would just cancel each other out in the polls!
Actually, I think a study of whether Republicans or Democrats are more likely to lie to pollsters would be quite interesting, though rather paradoxical. Sort of like statistics on the inaccuracy of statistics.
“they would just cancel each other out in the polls”
Surely that would assume, Steve, that Democrats and Republicans were in the population in question in roughly equal numbers, which in many local races is not the case.
There’s also a common misunderstanding about “margin of error”. Naturally, if the margin of error were listed as a set value, the same way as the original percentages, there would have to be a margin of error applied to that margin of error, and so on, ad infinitum.
So there seems to be a misunderstanding of what the “margin of error” actually means. When a person applies error estimates to a sample, that person is fitting the sample (finding the mean value, or presuming that the random sampling has given a good mean value), and then saying that it’s 80% likely that the percentages quoted are within the error.
So saying 50% / 35% (+/- 4.2%) is like saying: X has about 50% of the voters, and there’s an 80% likeliness that the actual percentage of the voters is between 54.2% and 45.8%.
After all, even with purely random sampling, it would be statistically possible to by pure chance select only supporters of one candidate. The margin of error is thus almost always based on standard deviations of a Gaussian fit.
It isn’t about whether the sample is truly random/representative. Even if you poll 100% of a population, there will still be a margin of error – how big that margin is depends on things like how big the population is (and on the percentages of the population that come down on each side of the poll question). This despite the fact that obviously a “sample” that is in fact the whole population, is “representative” of that population.
Nooc: It seems as though you have missed my point. Maybe I didn’t communicate it very clearly.
My point is this: Margin of error says nothing about the bounds wherein the actual percentages must be, but the bounds wherein they are most likely to be.
James Akin’s observation, though maybe humorous, though maybe it suggests other errors weren’t accounted for, is most likely an observation of a statistical irregularity, for whatever reason.