Law of the Instrument: when you have a hammer, everything looks like a nail

What you'll understand by the end of this lesson

• Why teams apply familiar research methods to problems those methods can't solve
• When A/B testing is the wrong tool — and what to use instead
• How heatmaps can mislead when used without the right interpretive context
• How to choose a research method based on the question you're trying to answer

---

The principle in plain English

The Law of the Instrument is the tendency to over-rely on a familiar tool, even when a different approach would serve the problem better.

The phrase is usually attributed to Abraham Maslow: "I suppose it is tempting, if the only tool you have is a hammer, to treat everything as if it were a nail."

In research and CRO, this shows up when teams default to their most familiar method — an A/B test, a heatmap, a survey — regardless of whether that method can actually answer the question at hand. The tool becomes the lens through which the problem is defined, and questions that don't fit the tool don't get asked.

The result is a body of research that looks thorough but is systematically biased toward whatever the team knows how to measure.

---

A simple example

A SaaS product has a 72% drop-off at the pricing page. The CRO team sets up an A/B test: they test a new headline on the pricing page.

The test runs for 3 weeks. No significant result. They test a different button colour. No result. They test a new layout. Still nothing.

The problem was never the headline, the button, or the layout. The problem was that visitors didn't understand what they were getting for the price. A 20-minute user interview would have surfaced this in the first conversation. The team ran three A/B tests because that's the tool they knew — and the tool defined what kind of answers they looked for.

---

When A/B testing is the wrong tool

A/B testing is excellent for answering: which of these two specific options produces a better outcome?

It is a poor tool for answering:

• Why are users dropping off here? — A/B testing can confirm which version reduces drop-off; it cannot explain why users are dropping off in the first place. For that, you need qualitative research: session recordings, user interviews, or on-page surveys.

• What do users think this product does? — A/B testing can't capture perception or mental models. Card sorting, first-click testing, and user interviews can.

• Why aren't these visitors converting? — Testing variations of existing copy can lift a page by 10%. Understanding the actual objection blocking conversion can lift it by 50%. You find the objection through interviews and surveys, not A/B tests.

• Low-traffic problems — A/B testing requires sufficient traffic to reach statistical significance. A page that receives 200 visits per month cannot produce a reliable A/B test result in any reasonable time frame. A team that defaults to A/B testing regardless of traffic volume will produce noisy, unreliable results and draw wrong conclusions from them.

---

How heatmaps mislead

Heatmaps are one of the most visually compelling research tools in CRO — and one of the most frequently misinterpreted.

A scroll map shows that 70% of users don't reach a section below the fold. Teams often interpret this as: "users aren't interested in that content." But there are other explanations: users already found what they needed, the page loaded too slowly below the fold, or users got the information they came for and left satisfied.

A click map shows that users are clicking an image that isn't a link. Teams often interpret this as: "users want to click through to a product page." But users may be clicking to dismiss a tooltip, trying to zoom, or simply mis-tapping on mobile.

Heatmaps show what users did with their cursor or finger. They cannot show why. Treating heatmap patterns as explanations — rather than as indicators of where to dig deeper — is Law of the Instrument in action: the tool provides data, and the data gets interpreted as if it answered a question it wasn't designed to answer.

---

Choosing the right tool for the question

A practical decision framework:

Question type	Right tool
Which version performs better?	A/B test
Why are users dropping off?	Session recordings, user interviews
What do users think this page is about?	First-click testing, user interviews
What are users clicking on?	Click heatmap (then interview to understand why)
What objections are blocking conversion?	On-page survey, exit survey, user interviews
Is the page readable and navigable?	Usability testing
What do users search for?	On-site search analysis

The key pattern: quantitative tools (A/B tests, heatmaps, analytics) tell you what happened. Qualitative tools (interviews, surveys, session recordings reviewed with context) tell you why.

---

The CRO audit

Look at your research and testing activity and ask:

1. What percentage of your research is quantitative vs qualitative?

If qualitative research (interviews, open-ended surveys, usability tests) accounts for less than 20% of your research activity, you are likely missing the "why" behind most of your data. The A/B tests you are running may be testing solutions to problems you haven't actually identified correctly.

2. Before your last three A/B tests: where did the hypothesis come from?

If the hypothesis came from another A/B test, a heatmap, or intuition — rather than from direct user feedback — trace it back further. The method that generated the hypothesis matters as much as the method that tests it.

3. Have you run user interviews on your highest-traffic, lowest-converting pages in the last 12 months?

If not, you may have extensive quantitative data on these pages and no understanding of why they convert the way they do. Five user interviews can reframe a conversion problem more effectively than 20 A/B tests.

---

---