Beta tester feedback: the 4 questions to ask

Testers volunteered. They like you, or at least the idea. So "looks great, maybe make it faster" is a common reply, and it gives you nothing to fix.

Two things help. Ask about something that already happened, such as the last time they opened the product, instead of what they think of it. And ask for the exact spot, not a score. You can fix "the import stopped halfway". You can't fix a 4 out of 5.

The GOV.UK Service Manual describes the aim of beta research as to "test the developing service with likely users to make sure it meets their needs" and to understand and resolve usability issues (GOV.UK). Both halves need testers to describe what they did, and a rating doesn't carry that.

Short. Centercode, which makes beta test software, says "10 questions is a good survey, 15 is long but acceptable, and 20+ is only really appropriate" later in a test. It also counts one free-form response as equivalent to three multiple choice questions in effort (Centercode). If you survey testers more than once a week, it suggests around 5 questions each.

Worked out with that rule: three written answers count as 9, plus one pick from a list makes 10. The four questions below sit right on Centercode's "good" line.

1. What were you trying to do the last time you opened it?

This ties everything else to a real task. "It was confusing" becomes "it was confusing while I was trying to invite a teammate". It also tells you if testers use the product for the job you built it for.

2. Where did it stop working the way you expected?

Ask for the screen or the step, and what they expected instead. This is where you find the bugs and the confusing spots, in the tester's words. Centercode's advice on useful feedback is that it should directly improve the quality of the product (Centercode). A named failure point does that. A general impression doesn't.

3. If one thing got fixed before launch, what should it be?

Forcing one pick turns a wish list into a vote. When you count the answers, the biggest group is your pre-launch list, and the long tail waits.

4. How would you feel if you could no longer use it?

Offer three choices: very disappointed, somewhat disappointed, not disappointed. This is the question Sean Ellis used when he benchmarked nearly a hundred startups. Per First Round Review, "companies that struggled to find growth almost always had less than 40% of users respond 'very disappointed'" (First Round Review).

Politeness has less room here, because "not disappointed" is an easy and honest thing to tick. If you want one more written answer, add "What's the main benefit you get from it?". Superhuman asked that and used it to decide whose feedback to weigh more.

Start with question 4. It tells you whether you have a fit problem or a polish problem. Then filter every other answer to the "very disappointed" group first, because they show what the product is for. Then count question 3.

The numbers below are invented to show the method. They are not from a real beta.

Say 30 testers finished the survey. On question 4, 11 said very disappointed, 12 somewhat, 7 not. That is 37% very disappointed, calculated as 11 divided by 30. It is under the 40% line, so fit work comes before polish.

Now read question 3 and label each reply by what went wrong:

Import has the biggest count. But the teammate invite has the most "very disappointed" testers behind it, 5 of 7. The people who would miss the product most are stuck at the invite. That is the first fix, and the counts alone would have pointed you at import. Read every reply under the invite label before you commit, since a label is a summary you wrote.

You have a few honest ways to run this.

• A spreadsheet. Paste the replies into Google Sheets, add a label column, and count. It costs nothing and works well up to a few dozen testers. The cost is your time, and the labels drift as you tire.
• A general AI chat. Paste the replies in and ask for groups. It is quick, but it can merge two problems into one, so give each reply an ID and read the replies under any group you act on. Our post on whether ChatGPT can analyze survey responses goes through what to check.
• A form tool. Google Forms or Typeform collect the answers fine. For the counting, see how Enform compares with Google Forms and with Typeform.
• Enform. It writes the survey from a description, you share one link, it reads every reply, counts what people said, and every count opens to the replies behind it. The beta feedback survey template has these questions ready to copy, and pricing is here.

Enform won't replace talking to your five most active testers. A call finds things a form doesn't. The survey is for knowing which of those things many people hit.