Perhaps the most devastating problem with subjective [survey] questions, however, is the possibility that attitudes may not “exist” in a coherent form. A first indication of such problems is that measured attitudes are quite unstable over time. For example, in two surveys spaced a few months apart, the same subjects were asked about their views on government spending. Amazingly, 55% of the subjects reported different answers. Such low correlations at high frequencies are quite representative.
It's fitting that my least favorite writer nailed this point completely:
"But why should you keep your head over your shoulder? Why drag about this corpse of your memory, lest you contradict somewhat you have stated in this or that public place? Suppose you should contradict yourself; what then? It seems to be a rule of wisdom never to rely on your memory alone, scarcely even in acts of pure memory, but to bring the past for judgment into the thousand-eyed present, and live ever in a new day. In your metaphysics you have denied personality to the Deity: yet when the devout motions of the soul come, yield to them heart and life, though they should clothe God with shape and color. Leave your theory, as Joseph his coat in the hand of the harlot, and flee.
A foolish consistency is the hobgoblin of little minds, adored by little statesmen and philosophers and divines. With consistency a great soul has simply nothing to do. He may as well concern himself with his shadow on the wall. Speak what you think now in hard words, and to-morrow speak what to-morrow thinks in hard words again, though it contradict every thing you said to-day."
There are, of course, ways of overcoming the problems with surveys, such as doing pilot surveys and following up with interviews of respondents to see whether the question wording, response options etc. really tapped into the attitudes that the researcher was intending. The best surveys in the academic realm have had their "psychometric" properties validated in these ways, although probably most of the surveys we read about in the paper have not. I do a lot of surveying of the students at my small liberal arts college, and I never stake out serious claims about particular phenomena unless I see them repeatedly over time, can confirm them through qualitative methods like focus groups, interviews, or analyzing the content of open-ended responses (which also have their own methodological issues). That's about as much as you can hope to do, I think, even if the results are still imperfect.
"I’m not clear on just how far this effect goes, but one lesson is: you have fewer real opinions than you think."
I am very hesitant to extrapolate to whether the individuals actually have "fewer opinions". If the surveys mentioned here didn't explicitly mention a "or do you not have a strong opinion one way or another", the pollsters were likely trained to extract an opinion and avoid letting the respondent be neutral or "don't know". I would say the lesson is: respondents are often not taking surveys very seriously. This might not speak well for surveys (which is sad for my own research), but strikes me as the most accurate claim based on the information.
"Self-Reported" opinions (genuine or supposed) are indeed a big problem in Survey-Research of public opinion... but many other big problems heavily compete for that title of "most devastating".
For example, virtually 100% of the opinion polls we see trumpeted daily in the media... are non-scientific, due to severe lack of a random sample in the population under study. Response rates on major national polls are typically less than 10% these days.
Opinion survey methodology & assumptions are quite interesting and complex.
But all one needs to know is that almost all opinion-poll results are merely 'opinions' themselves-- not facts or scientific measurements in any way. (..the exceptions are so rare that they can be safely ignored).
humans don't even have coherent *goals* much less coherent opinions on how to reach goals. Humans have responses to various stimuli. We construct a narrative under which our responses in a variety of domains is consistent. Any close examination immediately reveals to this to be not just false, but ridiculously, trivially false.
Note that actions, as well as opinions, may also not be consistent through time. Though they are more likely to be so than opinions.
One reason for actions being more consistent (to the degree that they are) is the one Robin suggests - actions might be based on fewer underlying reasons than we think. Another is that we may simply not take actions when we don't have a strong reason to do so. A final reason is that we may adjust opinions to match actions, once taken, and future actions will be taken consistently both with memories of past actions and with these freshly formed opinions.
So more ultra-general questions from pollers (who can ask further questions -- following a flowchart, for example, depending on how many real opinions get expressed) would probably produce more accurate results?
It's fitting that my least favorite writer nailed this point completely:
"But why should you keep your head over your shoulder? Why drag about this corpse of your memory, lest you contradict somewhat you have stated in this or that public place? Suppose you should contradict yourself; what then? It seems to be a rule of wisdom never to rely on your memory alone, scarcely even in acts of pure memory, but to bring the past for judgment into the thousand-eyed present, and live ever in a new day. In your metaphysics you have denied personality to the Deity: yet when the devout motions of the soul come, yield to them heart and life, though they should clothe God with shape and color. Leave your theory, as Joseph his coat in the hand of the harlot, and flee.
A foolish consistency is the hobgoblin of little minds, adored by little statesmen and philosophers and divines. With consistency a great soul has simply nothing to do. He may as well concern himself with his shadow on the wall. Speak what you think now in hard words, and to-morrow speak what to-morrow thinks in hard words again, though it contradict every thing you said to-day."
Emerson, in Self Reliance
Contemplation and critical thinking are what give weight to opinion. Modern life is designed to eliminate both.
There are, of course, ways of overcoming the problems with surveys, such as doing pilot surveys and following up with interviews of respondents to see whether the question wording, response options etc. really tapped into the attitudes that the researcher was intending. The best surveys in the academic realm have had their "psychometric" properties validated in these ways, although probably most of the surveys we read about in the paper have not. I do a lot of surveying of the students at my small liberal arts college, and I never stake out serious claims about particular phenomena unless I see them repeatedly over time, can confirm them through qualitative methods like focus groups, interviews, or analyzing the content of open-ended responses (which also have their own methodological issues). That's about as much as you can hope to do, I think, even if the results are still imperfect.
"I’m not clear on just how far this effect goes, but one lesson is: you have fewer real opinions than you think."
I am very hesitant to extrapolate to whether the individuals actually have "fewer opinions". If the surveys mentioned here didn't explicitly mention a "or do you not have a strong opinion one way or another", the pollsters were likely trained to extract an opinion and avoid letting the respondent be neutral or "don't know". I would say the lesson is: respondents are often not taking surveys very seriously. This might not speak well for surveys (which is sad for my own research), but strikes me as the most accurate claim based on the information.
"Self-Reported" opinions (genuine or supposed) are indeed a big problem in Survey-Research of public opinion... but many other big problems heavily compete for that title of "most devastating".
For example, virtually 100% of the opinion polls we see trumpeted daily in the media... are non-scientific, due to severe lack of a random sample in the population under study. Response rates on major national polls are typically less than 10% these days.
Opinion survey methodology & assumptions are quite interesting and complex.
But all one needs to know is that almost all opinion-poll results are merely 'opinions' themselves-- not facts or scientific measurements in any way. (..the exceptions are so rare that they can be safely ignored).
then, there is also the issue of "preference falsification." See the work of Timur Kuran.
humans don't even have coherent *goals* much less coherent opinions on how to reach goals. Humans have responses to various stimuli. We construct a narrative under which our responses in a variety of domains is consistent. Any close examination immediately reveals to this to be not just false, but ridiculously, trivially false.
I'm not sure how I feel about all of this.
Note that actions, as well as opinions, may also not be consistent through time. Though they are more likely to be so than opinions.
One reason for actions being more consistent (to the degree that they are) is the one Robin suggests - actions might be based on fewer underlying reasons than we think. Another is that we may simply not take actions when we don't have a strong reason to do so. A final reason is that we may adjust opinions to match actions, once taken, and future actions will be taken consistently both with memories of past actions and with these freshly formed opinions.
So more ultra-general questions from pollers (who can ask further questions -- following a flowchart, for example, depending on how many real opinions get expressed) would probably produce more accurate results?