Before We Start
Why "not all polls are equal" is the right starting frame
Public opinion polling is a genuine statistical methodology, not guesswork — but its validity depends entirely on following specific methodological requirements. Polls that cut corners on these requirements can produce genuinely misleading results, which is exactly why understanding the underlying methodology matters for interpreting any specific poll critically.
💡 The Only Valid Sampling Method
A random sample — where every person in the target population has an equal chance of being selected — is the only valid method for a scientifically sound poll. Non-random samples (like an online poll people opt into) don't have this property and can't reliably represent the broader population.
Mnemonic
The specific methodological factors
Random Sample
Every person has an equal chance of selection
The foundational methodological requirement — without genuine randomness, a poll's results can't be reliably generalized to the broader population it claims to represent.
Sampling Error
±3% margin of error at 95% confidence, for n=1,000
Even a properly conducted random-sample poll has an inherent margin of error — a specific, quantifiable range around the reported number reflecting the natural uncertainty of sampling a subset rather than surveying the entire population.
Question Wording Effects
Small wording changes can dramatically shift responses
How a question is phrased can significantly affect the responses it generates — even seemingly minor changes in wording can produce meaningfully different results, making question design a genuinely important methodological consideration.
Response Bias
People give socially acceptable answers
Respondents sometimes shade their answers toward what they perceive as more socially acceptable, rather than their genuine private view — a distortion that can affect poll accuracy on sensitive topics.
Likely Voter Screen
Likely voters differ from registered voters
Pollsters often apply a "likely voter" screen to narrow their sample from all registered voters down to those actually expected to vote — since these two groups can have meaningfully different characteristics and preferences.
💊 The margin of error figure is worth memorizing precisely: ±3% at 95% confidence for a sample size of about 1,000 — this specific figure appears constantly in discussions of poll reliability and is worth having ready as a concrete reference point.
⚖️ Applying the Framework — Evaluating a Specific Poll's Reliability
A poll conducted through an online opt-in survey reports a candidate leading by 5 percentage points. A separate poll, using a random-digit-dial telephone methodology with proper random sampling, shows the race essentially tied.
Evaluate the Methodology of Each Poll
The online opt-in poll lacks genuine random sampling — people who choose to participate in an online survey aren't representative of the broader population in the same way a properly randomized sample would be. The random-digit-dial poll, by contrast, more closely approximates the random sampling requirement for a methodologically valid poll.
Weight the Results Accordingly
Given this methodological difference, the random-sample poll's result deserves significantly more confidence than the opt-in poll's result — the 5-point lead in the opt-in poll may reflect selection bias in who chose to participate, rather than a genuine reflection of the broader electorate's preferences. This illustrates directly why "not all polls are equal," even when reporting on the exact same race.
📌 Exam Application
Polling questions test both the specific statistical concepts and methodological critique skills:
Margin of error: "What is the approximate margin of error for a well-conducted poll with a sample size of 1,000, at 95% confidence?" → ±3%.
Sampling requirement: "What is the only valid method for sampling in a scientifically sound poll?" → Random sampling, where every person has an equal chance of selection.
Methodological critique: "Why might an online opt-in poll produce less reliable results than a randomly sampled telephone poll?" → It lacks genuine random sampling — people who opt in aren't necessarily representative of the broader population.
⚠️ The Trap — Treating All Polls as Equally Credible Regardless of Methodology
Because polls are frequently reported in the news without much discussion of their underlying methodology, it's easy to treat all polls as roughly interchangeable sources of information. But methodology genuinely matters — a poll lacking proper random sampling, or one with poorly worded questions, can produce meaningfully misleading results even while looking similar to a well-conducted poll on the surface.
The safeguard: Always consider the underlying methodology (random sampling, question wording, likely voter screening) before treating a specific poll's results as reliable.
✓ Quick Self-Test
Answer before checking:
1. What is the only valid sampling method for a scientifically sound poll?
2. What is the approximate margin of error for a poll with n=1,000, at 95% confidence?
3. What is "response bias"?
4. What is a "likely voter screen," and why is it used?
Answers:
1. Random sampling — every person has an equal chance of being selected.
2. ±3%.
3. Respondents giving socially acceptable answers rather than their genuine private view.
4. A method for narrowing a sample from all registered voters to those actually expected to vote, since these two groups can differ meaningfully in characteristics and preferences.
Next Lesson
Dealignment and Realignment
→