What you'll understand by the end of this lesson
- Why the brain seeks information that confirms existing beliefs — and ignores what contradicts them
- How confirmation bias kills otherwise solid CRO programmes
- The specific ways it shows up when reading test results and interpreting analytics
- The one practice that defends against it: pre-registration
The principle in plain English
People are not neutral processors of information. When we form a belief, we naturally begin to favour evidence that supports it — and downweight or dismiss evidence that challenges it.
This is Confirmation Bias. It isn't a sign of stupidity or laziness. It's a feature of how the brain handles information overload. We can't evaluate every data point equally — so we use our existing model of the world as a filter. Information that fits the model gets through easily. Information that challenges it requires more work to process, so it gets less attention.
In daily life, this produces views that are harder to change than they should be. In a CRO context, it produces A/B test results that are misread, analytics reports that tell flattering stories, and hypotheses that are confirmed regardless of what the data actually says.
A simple example
A test ran on a landing page. The overall result: a 2% lift in conversion rate, with a confidence interval that crossed zero — meaning the result was statistically inconclusive. The honest read was: we don't know if this variant is better.
But a mobile segment showed a +18% lift. That was a compelling story. The team was excited about the variant and wanted it to work. So the 18% mobile figure became the headline. "The test showed the variant lifts mobile conversion by 18%."
The test did not show that. The test showed a statistically insignificant overall result and a subgroup finding that could easily be noise. But the brain found the data that matched the preferred outcome, and that became the conclusion.
How it shows up in CRO practice
Cherry-picking segments
Every A/B test result contains dozens of segment cuts: desktop vs mobile, new vs returning, organic vs paid, by country, by device, by time of day. Some of those segments will show a lift by chance alone — not because the variant actually worked for that group.
Confirmation bias makes it easy to look at a failed test, find the one segment that performed well, and report that as the result. The problem: that segment was probably identified after the test, not before. If you didn't pre-specify that mobile organic users from the US were the target segment before the test ran, finding a lift in that group after the test is not a finding — it's a pattern that may well be noise.
Skipping statistical significance
Teams that believe in their variant often declare victory early — when the trend looks good but before statistical significance is reached. Confirmation bias makes the early trend feel like confirmation. It's not.
The reverse also happens: a variant the team doesn't believe in hits statistical significance against the control, and the team runs the test longer looking for the result to reverse. Confirmation bias shapes not just how results are read but how long tests are run.
Interpreting analytics reports
Analytics data is infinitely sliceable. A team that believes their new homepage is working will find the slice of data that supports it. A team that believes paid traffic is performing well will look at the sessions that converted, not the sessions that didn't.
This isn't malicious. It's automatic. The brain navigates toward confirming data because confirming data is easier to process and more comfortable to report.
The most effective defence against confirmation bias in CRO is pre-registration: before any test runs, write down the primary metric, the minimum detectable effect, the required sample size, and the planned analysis. When the test ends, evaluate against exactly what you wrote. The pre-registered plan forces you to commit to what "success" means before you can see the results.
Why pre-registration works
Pre-registration doesn't eliminate confirmation bias — it creates a structural barrier against it.
When you've written down "this test wins if overall conversion rate lifts by at least 3% at 95% confidence" before the test starts, you can't reframe the result as a win when overall conversion lifts 2% at 80% confidence but mobile lifts 18% in one segment. The written plan is the plan. The test is evaluated against it.
Pre-registration is standard practice in clinical trials and academic research for exactly this reason — the findings are more reliable when the hypothesis and success criteria are locked before the data is collected.
In CRO, a simple pre-registration document doesn't have to be long: hypothesis, primary metric, secondary metrics (if any), minimum sample size, significance threshold, and planned test duration. One page is enough.
The higher-level problem
Confirmation bias in CRO isn't just about misreading individual tests. It's about the cumulative effect on a testing programme over time.
A team that consistently interprets ambiguous results as supporting whatever they wanted to be true will build a false picture of what's working on their site. Pages will be "optimised" based on findings that aren't findings. The mental model of what converts will drift further from reality. Eventually the programme produces lots of tests and lots of "wins" but no actual lift in revenue.
The protection isn't just test-by-test rigour. It's a team culture where inconclusive is a valid result, where a well-run test that found nothing is considered as valuable as a winning test, and where the question "what was our pre-registered hypothesis?" is asked before any test result is discussed.
The most dangerous confirmation bias move in CRO is not reading the results wrong — it's running the test wrong to make the results easier to read wrongly. Stopping a test early because the trend looks good, extending a test that's going badly, or changing the primary metric mid-test are all ways of manufacturing the conclusion you wanted. They're hard to catch because they feel like reasonable decisions at the time.
The CRO audit
Look at your testing programme and ask:
1. Do you pre-register your hypotheses and success metrics before tests run?
If the answer is no — if your test conclusions are evaluated after seeing the results — your programme is vulnerable to confirmation bias at every test. Add a simple pre-registration step: before any test goes live, write down the primary metric and what a win looks like.
2. When a test is "inconclusive," how is it reported?
If inconclusive results are quietly filed away while only clear winners are shared with stakeholders, you're building a distorted picture of what's working. Inconclusive is valid and important. A well-run inconclusive test tells you something the site needs to know: this change didn't matter.
3. Have any recent "wins" been primarily segment findings rather than primary metric findings?
Revisit your last five test results. For each one: what was the primary metric? Did it hit significance? Or did the team declare success based on a segment — mobile, new users, a specific traffic source — that wasn't pre-specified? If so, those findings warrant re-testing with the segment as the intended primary audience, not retrospective validation.
An A/B test completes. Overall conversion rate: +2% lift, confidence interval crosses zero (statistically inconclusive). A post-hoc analysis finds that US mobile users on organic traffic converted at +22% in the variant. The team declares the test a win. What's the problem?
You've covered 18 principles of how people think and decide. The next chapter explores how the brain processes visual information — what it sees first, what it ignores, and what guides the eye across a page.