What you'll understand by the end of this lesson
- Why survey responses systematically overstate demand and positive intent
- How social desirability bias distorts what people tell you about their preferences
- Why "would you use this feature?" surveys are unreliable predictors of actual use
- How to design research that gets closer to what people will actually do
The principle in plain English
Survey Bias is the systematic distortion of survey responses away from what's true and toward what's socially acceptable, flattering to the asker, or expected by the context.
The most common form in product and CRO work is social desirability bias: respondents give the answer that makes them look good, makes the researcher feel good, or that they believe is "correct" — rather than the honest answer.
Ask someone if they'd pay for a premium feature and they say yes — because saying no feels cheap or unhelpful. Ask someone how often they'd use an app and they overestimate — because admitting they'd use it rarely feels like criticism. Ask someone if they read terms and conditions and they say yes — because admitting they don't feels irresponsible.
In every case, the survey response is real. The respondent believes it. But it predicts future behaviour poorly, because it reflects an idealised self-image rather than actual behaviour.
A simple example
A team is deciding whether to build a new reporting feature. They survey 200 existing users: "Would you use a detailed weekly usage report if it were available?"
82% say yes.
They build the feature. At launch, 9% of users open their first report. After one month, 3% are regular users.
Nothing went wrong technically. The survey wasn't faked. 82% of respondents genuinely believed they would use it. But "would you use this?" is a question about a hypothetical future self — and that self is consistently more disciplined, more organised, and more interested than the actual person.
Why "would you use this?" surveys overstate demand
When you ask someone "would you use this feature?", you're asking them to predict their own behaviour in a hypothetical scenario. This prediction is unreliable for several reasons:
Social desirability. Saying "no, I wouldn't use that" can feel like criticism of the person asking. Respondents default to "yes" to be helpful.
Optimistic self-modelling. People imagine their best-case use of a product — the version of themselves who has time, interest, and discipline to use every feature. That person doesn't always show up.
Present bias. The feature doesn't exist yet. There's no cost to saying you'd use it. When the feature exists, the friction of actually using it is real and acts as a filter.
Hypothetical versus actual money. "Would you pay £5/month for this?" gets yes more often than the actual pricing screen does. The hypothetical costs nothing; the real payment costs £5.
A more reliable question than "would you use this?" is "how do you currently solve this problem?" That question asks about present behaviour, not future intent. If no one has a current workaround or existing solution, the demand may be lower than a direct "would you use it?" question suggests. If many people have built elaborate workarounds, the demand is real.
How to design surveys that minimise social desirability bias
Ask about past behaviour, not future intent. "Have you ever tried to export your data from this product?" is more reliable than "Would you export your data if we built an export feature?"
Make it socially acceptable to give negative answers. "Some of our users told us they wouldn't find this useful — what do you think?" normalises the negative response, reducing the pressure to say yes.
Use forced ranking over open agreement. Instead of "would you use feature X, Y, Z?" ask respondents to rank X, Y, Z in order of usefulness. Ranking forces trade-offs; open agreement allows inflated responses across the board.
Ask about frequency, not just yes/no. "If this feature existed, how often would you use it?" with response options from "never" to "daily" produces better signal than a yes/no response — people are less likely to select "daily" when they know that's an extreme claim.
Why behavioural data always beats attitudinal data
Attitudinal data is what people say they think, feel, or will do. Surveys collect attitudinal data.
Behavioural data is what people actually do. Analytics, session recordings, heat maps, and conversion tracking collect behavioural data.
The gap between the two is called the intention-behaviour gap, and it's well documented across psychology, economics, and public health research. People say they'll exercise more; they don't. They say they'll save money; they don't. They say they'll read more; they don't.
In product and CRO work, this means: a survey that says 82% of users want a feature is weaker evidence than an experiment where 12% of users actually click a prototype of that feature when it's real and available.
This doesn't mean surveys are useless — they're valuable for understanding how users describe problems in their own language, which problems feel most urgent to them, and what vocabulary they use for things they want. They're poor predictors of specific feature usage or willingness to pay. Use surveys for discovery and vocabulary; use behavioural experiments for validation.
The intention-behaviour gap in CRO
The most direct CRO application: don't build features because survey respondents said they would use them. Build a fake door — a button or link that looks like the feature — and measure how many users click it. That's behavioural data. The click costs the user something real (a moment of attention, the effort of deciding). The survey cost them nothing.
If 12% of real users click a fake door for a feature, that's a meaningful signal. If 82% of survey respondents say they'd want it but 1% click the fake door, the survey was telling you what people think they should want — not what they actually want.
The CRO audit
1. What research decisions are currently based on survey data?
List the features, page designs, or copy changes that were informed by "would you use/want/prefer" survey responses. Identify which of those decisions could be validated with a behavioural experiment instead.
2. Are any of your surveys asking future-intent questions?
Review your survey instruments. Replace "would you" with "do you" or "have you" wherever possible. Shift from future intent to present behaviour.
3. Is your team treating survey data and behavioural data as equally reliable?
If yes, establish a hierarchy: behavioural data (what people do) outranks attitudinal data (what people say) for predicting conversion. Both are useful, but they answer different questions.
A team surveys 500 users: 'Would you pay an extra £3/month for advanced reporting?' 74% say yes. They add the feature. Only 11% of users upgrade. What explains the gap?
You've seen how what people say they'll do differs from what they actually do. Now — what happens inside someone's mind when they do something that contradicts what they believe about themselves? The discomfort that creates, and the ways people resolve it, shape behaviour in surprising ways.