Don’t Ask, Can’t Tell

12. Don’t Ask, Can’t Tell

How many questionnaire and survey results about people’s beliefs, values, or behavior will you read during your lifetime in newspapers, magazines, and business reports? Thousands, surely. You may even create some of these surveys yourself in order to get information that is important for your business, school, or charitable organization. Most of us tend to read survey results rather uncritically. “Hmm, dear, I see in the Times that 56 percent of Americans favor tax increases for creating more national parks.” Ditto for questions we create ourselves and the answers our respondents give us. So far, all the methods I’ve discussed are applicable to pretty much everything—animal, vegetable, or mineral. We can do A/B testing on rats, learn from natural experiments about factors influencing corn yields, and do multiple regression studies of factors associated with water purity. Now I’d like to look at methodological difficulties in measuring specifically human variables. Unlike rats, corn, or water, people can tell you in verbal form (oral or written) about their attitudes, emotions, needs, goals, and behavior. And they can tell you what the causal influences on these variables are. In this chapter you’ll see just how misleading such reports can be, which won’t be surprising given what you read in Part I about our limited accessibility to the factors that influence our behavior. The chapter will show you how a variety of behavioral measures can provide much more trustworthy answers to questions about people’s attributes and states than their verbal reports can. You’ll also get some tips about experiments you can do on yourself to learn what kinds of things influence your attitudes, behaviors, and physical and emotional health.

Constructing Attitudes on the Fly

The following examples may make you pause before trusting self-reported verbal answers, and may help you consider how best to get useful information about people’s attitudes and beliefs.

• Q. Life Satisfaction and Events: Suppose I ask you about three positive events in your life and then ask you about your life satisfaction; or I ask you about three negative events and then about your life satisfaction. In which case do you report greater life satisfaction? Whatever you guessed, you’re likely wrong. It all depends on whether those events were recent or from five years ago. Recent negative events lower satisfaction, recent positive events raise it. But for events from five years ago, the opposite occurs: past negatives make current life seem good by comparison, while past positives make current life seem less great.
• Q. Weather and Mood: If your cousin from Omaha calls and asks how things are going, your answer is influenced by the weather—unless she first asks about the weather, in which case you discount the weather’s effect on your mood.
• Q. Marriage and Life Satisfaction: The correlation between satisfaction with marriage and satisfaction with life changes dramatically depending on question order. If asked about marriage first, the correlation is .67; if asked about life first, it’s .32.
• Q. Politician Ratings: If you are told others rate politicians highly (e.g., 5/6), you will rate them higher than if told others rate them low (2/6). This shifts your interpretation of “politicians”.
• Q. Death Penalty: In the abstract, a majority favor it. But when details of crime and perpetrator are given, support drops—even for heinous crimes.
• Q. Abortion Support: Depending on question wording, support can range from 42% (pro-choice) to 53% (legal under certain circumstances) to higher if specific scenarios are offered.

These examples show that attitudes are often constructed on the fly, not retrieved from a mental file drawer.

What Makes You Happy?

Verbal reports about attitudes are susceptible to social desirability bias. But even knowing what makes us happy is problematic. In a Harvard study, female students kept daily diaries of mood and various factors (work, sleep, health, weather, sex, day of week, menstrual cycle). At the end, they estimated how much each factor influenced their mood. The correlation between their beliefs and actual correlations was zero. People’s beliefs about what affects their mood are theories, not accurate introspection.

The Relativity of Attitudes and Beliefs

• Cultural Self-Reports: Chinese value choosing personal goals more than Americans, Japanese are less conscientious than Italians, etc. These surprising findings result from the reference group effect: respondents compare themselves to their own cultural group. When reference group is made explicit, the differences disappear.
• Self-Enhancement Bias (Lake Wobegon Effect): Most people rate themselves above average. 70% of American students rate themselves above average in leadership; 25% say they are in the top 1% in getting along with others. East Asians often show a modesty bias.
• Acquiescence Response Set: Some groups (e.g., polite East Asians) tend to agree with statements. Researchers counterbalance response categories to cancel this bias.

Talking the Talk Versus Walking the Walk

Behavioral measures (grades, punctuality, room neatness, postal speed, clock accuracy) are more trustworthy than self-reports. For cross-national comparisons, less conscientious nations by behavioral indices have citizens who rate themselves as more conscientious by self-report. General rule: Trust behavior > scenario responses > verbal reports.

Even preferences are unreliable. Focus groups can mislead: Ford’s focus groups rejected the center-post-free hardtop design, but GM launched it successfully. Ford then used focus groups for the Edsel—a famous failure.

Experiments on Yourself

You can do experiments on yourself to learn what works for you. Key principles:

• N of 1, but within-subject before/after design can be effective.
• Keep other factors constant.
• Use random assignment (e.g., flip a coin for coffee) and systematic recording.
• Simple three-point scales are adequate. Examples: Does morning coffee improve efficiency? Does a catnap after lunch help? Does yoga boost well-being?

Summing Up

Verbal reports are distorted by question wording, priming, context, reference groups, and social desirability. Attitudes are often constructed on the fly. Self-reports about causes of behavior are unreliable. Behavior is a better guide. Conduct self-experiments with random assignment and systematic recording to gain accurate personal insights.