How to analyze open-ended survey responses, with a worked example

To analyze open-ended survey responses, give every reply a short label that says what happened to the person, count how many replies share each label, then read all the replies under the biggest counts before you decide anything. With fewer than a hundred replies, a spreadsheet and one focused afternoon is enough.

That's the whole method. The rest of this post is how to do each step well, and a worked example so you can see where it usually goes wrong.

Why the written answers are worth the effort

A multiple-choice question can only return the answers you thought of. Pew Research Center ran the same question two ways after the 2008 US election: "What one issue mattered most to you in deciding how you voted for president?" When the economy was on the list, 58% picked it. When people had to write their own answer, 35% named the economy, and 43% gave an answer that wasn't on the list at all (Pew).

That 43% is the reason to ask open questions. It's also why reading them takes work: nobody has sorted them for you.

Step 1: Read 20 replies before you label anything

Open the export and read the first 20 or so replies straight through. Don't label yet. You're looking for the different kinds of things people are saying, in their words.

Pew builds its labels the same way, at a much bigger scale. For one 2024 study, researchers drafted a list of labels, tested it on random samples of at least 75 replies, argued over every reply they labelled differently, and repeated that five times before using the list on everything (Pew). You don't need five rounds. You do need one round of reading before you commit to a list.

Step 2: Label what happened, not how they felt

This is where most analysis quietly fails. "Frustrated" and "pricing" feel like labels, but they fit half the replies and tell you nothing about what to fix. Usercall's guide puts it well: "If your codes can apply to half the dataset, they're not analytical codes. They're buckets" (Usercall).

A good label is specific enough that you could act on it:

• Weak: "pricing"
• Better: "price went up at renewal"
• Better: "didn't use it enough to justify the cost"

Those last two both sound like "too expensive". One is a pricing problem. The other is a problem with getting people to use the product, and a discount won't fix it.

A reply can carry more than one label. If someone says "the export broke and support took three days", that's two labels.

Step 3: Count the labels

Put the replies in column A and labels in column B (add a column C if replies need a second label). Sort by label and count each one. Now you have a short list with numbers next to it, and the biggest number is where to start.

Counting matters because memory lies. The reply that made you wince will feel like it came up ten times. The count tells you if it did.

Step 4: Read every reply under the top three counts

Don't skip this. A label is a summary you wrote, and summaries drift. Go back and read every reply under your three biggest labels, one after another. You'll usually find that one label is really two, or that a small label belongs inside a big one. Fix the labels, recount, and only then pick the quotes you'll share.

Pick quotes last, after the counts are stable. A quote chosen first tends to become the story, whether or not anyone else said it.

A worked example

The numbers and replies below are invented to show the method. They are not from a real survey.

Say you run a small invoicing app and sent a one-question survey to people who cancelled: "What was happening when you decided to cancel?" You got 64 replies.

On a first pass, you might label them like this:

That table looks finished. It isn't. "Too expensive" holds 22 replies, a third of the total, which is a sign it's a bucket. Reading those 22 again:

• "The price jumped when my plan renewed and I didn't see it coming." (9 replies like this)
• "I only sent four invoices all year, couldn't justify it." (8)
• "Needed a second user seat and that doubled the bill." (5)

Now the same 22 replies point to three different fixes: warn people before a renewal price change, point people who barely use it to a cheaper plan, and look at how extra seats are priced. The "Missing a feature" label splits the same way once you read it. 10 of the 15 asked for recurring invoices, and that's a roadmap item with a number attached.

The "Other" row is worth one more look too. If it holds more than about one reply in ten, some of it probably belongs to a label you haven't named yet.

Here's the second pass:

Every row on that list tells you what to do next. The first table only told you people left.

How many replies you can realistically read

Fewer than you'd hope. Wikimedia's survey guidance warns that when most of your questions ask for written sentences, "100 respondents may actually be more data than you can realistically synthesize" (Wikimedia).

Expect fewer written answers than finished surveys, too. On Pew's panel, people skip closed questions 1% to 2% of the time and open questions around 18% of the time on average. Questions that ask for several sentences get skipped more than ones that ask for a word or a phrase (Pew). If 100 people finish your survey, plan on 80 or so written answers per open question.

Where a tool helps, and where it doesn't

Steps 2 to 4 are mechanical once the labels are good, and that's the part software can take off your hands. Enform reads every reply, counts what people said, and every count opens to the replies behind it, so step 4 is one click instead of a filter in a spreadsheet. If you're comparing options, our roundup of survey tools that summarize open-ended answers covers how the main ones handle this, and pricing is here.

What no tool should do for you is the last call. Read the replies under the counts you're about to act on, every time. If you're writing the survey from scratch, our churn survey template and the post on why "why did you cancel?" is the wrong question cover how to ask so the replies are easier to label.

FAQ

How long does it take to analyze 100 open-ended responses?

As a rough guide, a focused afternoon for one question: an hour to read and draft labels, an hour to label, and an hour to recount and reread the top groups. It goes faster when the question asks about one specific moment.

Can I use ChatGPT to analyze open-ended survey responses?

It can suggest a first set of labels quickly. Give each reply an ID so you can match its labels back to the original text, and still read the replies under the counts you plan to act on, because a confident summary can merge two different problems into one.

Should one reply get more than one label?

Yes, when it describes more than one thing. Pew applies its labels to each piece of information in a reply, not to the reply as a whole. Count labels, not replies, and say so when you share the numbers.

What should go in the "Other" group?

Replies that don't fit any label yet. If "Other" holds more than about one reply in ten, read it again. There's usually a label hiding in it.

Sources

1. Writing Survey Questions, Pew Research Center
2. Why do some open-ended survey questions result in higher item nonresponse rates than others?, Pew Research Center (2021)
3. Appendix A: Coding methodology, What Can Improve Democracy?, Pew Research Center (2024)
4. How to Analyze Open-Ended Survey Responses (Without Reading Every One), Usercall (2026)
5. Learning patterns/Who should we survey?, Wikimedia Meta-Wiki