Social Science; is there some test on the significance of survey questions that needs to be performed before a large-scale survey is conducted. Looking at the Alberta conultation results, it seems to me that a lot of these are simply the law of large numbers kicking in. People answering indifferently to 'stupid' questions, and hitting 50/50...
You're touching on a very real issue in survey methodology. Social scientists and pollsters do not simply write ten questions on a napkin and ask millions of people. There is usually a whole stage of question validation, because poorly designed questions often produce results that look meaningful statistically but are not meaningful substantively. A few concepts are relevant here: 1. Statistical significance is not the same as substantive significance Suppose you ask 1 million people: "Should governments be efficient?" and 52% say yes. With a huge sample, that 52% may be statistically very unlikely to have occurred by chance. But it doesn't tell you much because people may not have a clear, shared understanding of the question. A result can be: Statistically significant. Politically meaningless. These are different things. 2. Survey researchers pre-test questions Before a major survey, researchers often conduct: Cognitive interviews. Pilot surveys. Focus groups. Reliability testing. The goal is to determine whether respondents interpret the question in the same way. For example: "Should Alberta have greater authority over immigration?" sounds straightforward. But different respondents may hear: "Should Alberta decide who enters Canada?" "Should Alberta get a Quebec-style immigration agreement?" "Should immigration be reduced?" "Should Alberta control temporary workers?" If many interpretations are possible, the result becomes difficult to interpret. 3. The "non-attitude" problem Political scientist Philip Converse famously argued that many citizens do not have well-developed opinions on many policy questions. When confronted with a question they have never seriously considered, respondents may: Guess. Follow a vague ideological instinct. Pick "yes" because it sounds positive. Pick "no" because it sounds risky. In effect, they are constructing an opinion on the spot. This is one reason many public opinion questions produce surprisingly unstable results. 4. Why do so many referendum-style questions end up near 50-50? There are several possible reasons. One is exactly what you suggest: some respondents are essentially indifferent. If people have weak opinions, random influences can push them either way: Which side campaigned harder. Which wording was used. Current headlines. General mood toward government. When this happens, outcomes often drift toward the middle. However, a 50-50 result does not automatically mean people are answering randomly. Imagine a province where: 45% genuinely support something. 45% genuinely oppose it. 10% are indifferent. You would also get something close to 50-50. The closeness of the result alone doesn't tell us which explanation is correct. 5. Looking at the Alberta questions The pattern you described is interesting. If we look at the unofficial results discussed earlier: Q6: about 51.5% yes. Q8: about 48.7% yes. Q9: about 50.6% yes. These were highly constitutional and institutional questions. Most Albertans are not constitutional lawyers. Many likely had limited prior opinions on: Judicial appointments. Federal-provincial paramountcy. Constitutional division of powers. When ordinary voters encounter such questions, many political scientists would expect: Lower information. Weaker opinions. Greater sensitivity to wording. Results clustering around 50%. By contrast, the immigration questions produced clearer majorities, suggesting people arrived with more developed views. 6. The law of large numbers is not quite the culprit Technically, the law of large numbers says that as the sample gets larger, the average converges toward the true underlying population preference. It doesn't create a 50-50 result. What it does is reveal whatever the population's actual average preference is. If the true preference were 70-30, a very large vote would converge toward 70-30. If it converges near 50-50, there are two broad possibilities: The population is genuinely divided. Many respondents have weak or poorly formed opinions, leaving the aggregate result near the middle. To distinguish between those explanations, researchers often look at additional evidence: How many respondents say they are unsure. How strongly views are held. Whether answers change when wording changes. Whether the same people give consistent answers over time. Your instinct is actually shared by many survey scholars. One criticism of broad public consultations is that governments sometimes ask technically complex questions for which a substantial fraction of respondents may not possess stable, pre-existing opinions. In those circumstances, a result of 50.6% versus 49.4% can look more like a measure of mood or framing than a profound expression of public will. The statistic is precise; the underlying opinion may not be.
Copilot said:
You're touching on a very real issue in survey methodology.
Social scientists and pollsters do not simply write ten questions on a napkin and ask millions of people. There is usually a whole stage of question validation, because poorly designed questions often produce results that look meaningful statistically but are not meaningful substantively.
A few concepts are relevant here:
1. Statistical significance is not the same as substantive significance
Suppose you ask 1 million people:
"Should governments be efficient?"
and 52% say yes.
With a huge sample, that 52% may be statistically very unlikely to have occurred by chance. But it doesn't tell you much because people may not have a clear, shared understanding of the question.
A result can be:
- Statistically significant.
- Politically meaningless.
These are different things.
2. Survey researchers pre-test questions
Before a major survey, researchers often conduct:
- Cognitive interviews.
- Pilot surveys.
- Focus groups.
- Reliability testing.
The goal is to determine whether respondents interpret the question in the same way.
For example:
"Should Alberta have greater authority over immigration?"
sounds straightforward.
But different respondents may hear:
- "Should Alberta decide who enters Canada?"
- "Should Alberta get a Quebec-style immigration agreement?"
- "Should immigration be reduced?"
- "Should Alberta control temporary workers?"
If many interpretations are possible, the result becomes difficult to interpret.
3. The "non-attitude" problem
Political scientist Philip Converse famously argued that many citizens do not have well-developed opinions on many policy questions.
When confronted with a question they have never seriously considered, respondents may:
- Guess.
- Follow a vague ideological instinct.
- Pick "yes" because it sounds positive.
- Pick "no" because it sounds risky.
In effect, they are constructing an opinion on the spot.
This is one reason many public opinion questions produce surprisingly unstable results.
4. Why do so many referendum-style questions end up near 50-50?
There are several possible reasons.
One is exactly what you suggest: some respondents are essentially indifferent.
If people have weak opinions, random influences can push them either way:
- Which side campaigned harder.
- Which wording was used.
- Current headlines.
- General mood toward government.
When this happens, outcomes often drift toward the middle.
However, a 50-50 result does not automatically mean people are answering randomly.
Imagine a province where:
- 45% genuinely support something.
- 45% genuinely oppose it.
- 10% are indifferent.
You would also get something close to 50-50.
The closeness of the result alone doesn't tell us which explanation is correct.
5. Looking at the Alberta questions
The pattern you described is interesting.
If we look at the unofficial results discussed earlier:
- Q6: about 51.5% yes.
- Q8: about 48.7% yes.
- Q9: about 50.6% yes.
These were highly constitutional and institutional questions.
Most Albertans are not constitutional lawyers. Many likely had limited prior opinions on:
- Judicial appointments.
- Federal-provincial paramountcy.
- Constitutional division of powers.
When ordinary voters encounter such questions, many political scientists would expect:
- Lower information.
- Weaker opinions.
- Greater sensitivity to wording.
- Results clustering around 50%.
By contrast, the immigration questions produced clearer majorities, suggesting people arrived with more developed views.
6. The law of large numbers is not quite the culprit
Technically, the law of large numbers says that as the sample gets larger, the average converges toward the true underlying population preference.
It doesn't create a 50-50 result.
What it does is reveal whatever the population's actual average preference is.
If the true preference were 70-30, a very large vote would converge toward 70-30.
If it converges near 50-50, there are two broad possibilities:
- The population is genuinely divided.
- Many respondents have weak or poorly formed opinions, leaving the aggregate result near the middle.
To distinguish between those explanations, researchers often look at additional evidence:
- How many respondents say they are unsure.
- How strongly views are held.
- Whether answers change when wording changes.
- Whether the same people give consistent answers over time.
Your instinct is actually shared by many survey scholars. One criticism of broad public consultations is that governments sometimes ask technically complex questions for which a substantial fraction of respondents may not possess stable, pre-existing opinions. In those circumstances, a result of 50.6% versus 49.4% can look more like a measure of mood or framing than a profound expression of public will. The statistic is precise; the underlying opinion may not be.
No comments:
Post a Comment