Everyone said yes to the dashboard and kept opening the spreadsheet

Some time ago I talked to engineers at a company that sells market data. They told me about a product they had built for customers who want to know how their prices compare with the competition. I have changed the industry, so for this post the customers are pricing managers at fuel retailers. The story is theirs. The conversations below are my reconstruction of what they described, not transcripts.

The product was a large dashboard: a map of every competitor, price trends per region, alerts when a competitor moves and a view of where your own price sits in the market. Before building it, the team did what most teams do and asked customers about it.

Product manager: If you could see every competitor's prices on one map, with trends and alerts, would you use that?

Pricing manager: Yes, definitely. Right now I only see the numbers. I never see why they move.

Product manager: How often would you open it?

Pricing manager: Every day. It would be the first thing I look at.

The product manager writes down a strong yes and so do the other interviews. It feels like a green light. I understand why, because almost everyone asks this question at some point and it sounds like exactly what an interview is for.

That "yes" is worth less than it looks. The team built the dashboard and then looked at what customers actually opened.

Why they say yes

Saying yes costs the pricing manager nothing. They are being polite to someone who clearly worked hard, the insights sound valuable and nobody has to change anything today to agree with them. The question also asks for a prediction about a future that doesn't exist yet, which people are bad at because they imagine a version of themselves with more time and fewer habits.

The yes is also honest. The pricing manager does want to understand why prices move, and they would say so again tomorrow. What the question can't tell you is what they do with that wish when they sit down at their desk.

What they opened instead

The system also mailed customers a spreadsheet with the competitor prices automatically. Customers kept opening that spreadsheet and left the dashboard alone, which is hard to believe about a product they had asked for. Asking about the past would have shown it much sooner.

Product manager: Walk me through this morning. How did you start with competitor prices?

Pricing manager: Same as always. The mail is there when I sit down, I open the spreadsheet and filter on my region.

Product manager: And the dashboard?

Pricing manager: I know it's there. I looked at it once, when it launched. It's impressive.

Product manager: What happened after that?

Pricing manager: I'd have to log in, set my filters and find my stations again. The sheet is already in my inbox with exactly what I need.

Product manager: Anything else?

Pricing manager: My boss gets the same sheet. If I quote a number from the dashboard and it looks different, I'm the one explaining why.

Nothing is wrong with the dashboard. The pricing manager wants the insights and prefers the dashboard's view of them, but the spreadsheet has a time, a place and a person attached to it: it arrives on its own, it sits in the inbox and the boss reads the same one. The dashboard asks them to give all three up.

What to ask instead

Ask about the last time, because it is the one moment that really happened.

• Start with "Walk me through the last time you checked competitor prices." A real morning shows you what they open first, instead of what they think they would open.
• Ask what they did right before and right after. The habit is a sequence. Your product has to fit into it or replace a whole step.
• Ask who else sees it. The boss who reads the same spreadsheet is a reason to stay with it that no dashboard feature touches.
• When they hesitate, ask what they are worried about. The worry about explaining a different number is the real objection. Nobody raises it in a demo.
• After every answer, wait five to seven seconds. The silence helps them reconstruct the story because the useful detail usually comes in that gap.

Interviewers who work with Jobs to be Done call this a switch interview and call the four things above the forces: the push that starts the search, the pull of the new option, the anxiety about it and the habit of the old way. In this story the habit won against a product that was better in every way the customers had asked for.

If I were on that team, I would start from the spreadsheet. The one insight customers asked for could go inside the file they already open, with a link to the dashboard for the people who want to dig further. The sheet stays the entry point and the dashboard earns its place from there.

Compare it to cooking. Asking guests whether they would eat more vegetables gets you a yes every time, while watching what they leave on the plate gets you their diet behaviour.

What I take from it

The mistake is rarely the building. It is that a yes arrives, nobody asks what the yes would have to beat and the team builds against an opinion. In your next interview, the moment you hear yourself starting a question with "would you", stop and ask about the last time instead.

If you want to know how your own questions read, you can grade one interview for free (opens in a new tab). The report comes by mail, quotes your own lines and nobody but you sees it unless you share it. And if you have asked this question yourself and want to talk about what came after, write to me. I read every reply.