How to validate a startup idea with a survey (without fooling yourself)

A yes costs the person nothing. Michael Lynch, reviewing Rob Fitzpatrick's book The Mom Test, writes that "compliments cost nothing, so it's a cheap way for the customer to make the founder feel good" (Lynch). He also notes that "people are often overly optimistic about paying for a product when it's hypothetical" and are "less likely to follow through" in practice.

There is some research behind the worry. A study of wine buyers in the Journal of Consumer Marketing compared what people said they would pay with what they paid in an auction. The authors found that "the gap between stated willingness to pay and the actual price paid was wide," even for people who stated a high intention to buy (Barber et al., 2012). It was one product, and the authors say so, but the pattern is the one founders worry about.

So a survey that asks "would you buy this?" measures politeness. A survey that asks what happened last week measures the problem.

People who have the problem, not friends. Friends want you to succeed, and they were never going to be your buyers.

Paul Graham's advice on finding your first people applies here: you can't wait for them to come to you, and "you have to go out and get them" (Graham). For a survey, that means picking a place where the problem comes up, such as a forum, a Slack group or a newsletter, and asking in the thread where people already complain about it. Say who you are and what the survey is for.

Expect a small number of replies. That is fine for this job. You are reading written answers, not estimating a market share. Our post on what to do with only a few survey responses covers reading a small batch closely.

Write down first what result would make you stop. If you only decide after you see the replies, you will read them kindly. Then use four questions.

1. "When did you last run into [the problem]? What happened?" This asks for a specific event. Per Lynch's summary of the book, the advice is to "ask about specifics in the past instead of generics or opinions about the future" (Lynch).
2. "What did you do about it?" Someone who built a spreadsheet, hired a freelancer or paid for a tool has a real problem. Someone who did nothing may not.
3. "What did that cost you, in time or money?" Ask for what they gave up, not what they would give up. Lynch quotes the book: "the more they're giving up, the more seriously you can take what they're saying."
4. "Can I follow up with you? Leave an email." This is a small ask for commitment. An email is cheap, but it is more than a tick box.

Keep the problem in the question, not your solution. Our post on writing open-ended questions people answer has more on asking about a moment instead of an opinion.

Questions to leave out: "Would you use a product that did X?", "How much would you pay?" and "Do you think this is a good idea?" All three invite a guess.

Read every reply to question 1 and label it by what actually happened. Drop replies that describe nothing specific, such as "yes, this is a pain". Then count how many people did something about it in question 2.

The numbers below are invented to show the method. They are not from a real survey.

Say you are testing a tool that chases late invoices for freelancers, and 24 people replied. You decided in advance that you would go on only if at least a third had already paid or built something to solve it.

Nine people did something about it, which is 9 of 24, or 38%, worked out by dividing 9 by 24. That clears your line, but the work sits in one row. Late payment is the problem people act on. "Forgot to send it" is a small row where almost nobody did anything, so it is not a business yet. The idea survived, but a narrower version of it did. Open the replies under the first row and read them before you commit, since a label is a summary you wrote.

A survey can't show that people will pay. The next test is to ask the people who left an email for something real: a deposit, a pilot, or ten minutes on a call.

You have a few honest ways to run this.

• A spreadsheet. Paste the replies in, add a label column, count. It costs nothing and works for a few dozen replies. The cost is your time, and labels drift as you tire.
• A general AI chat. Paste the replies in and ask for groups. It is fast, but it can merge two problems into one, so give each reply an ID and read the replies under any group you act on. See our post on whether ChatGPT can analyze survey responses.
• A form tool. Google Forms collects the answers fine, though it doesn't count written ones. Here is how Enform compares with Google Forms.
• Enform. It writes the survey from a description, you share one link, it reads every reply, counts what people said, and every count opens to the replies behind it. The concept test template is a starting point, and pricing is here.

None of these replaces talking to five people from your list. A call finds things a form can't. The survey tells you which of those things many people hit.