Yes, ChatGPT can analyze survey responses. It is good at sorting written answers into groups, and in published tests the best models come close to a trained human. It is weak at exact counts and at noticing when one group holds two different problems. Give every reply an ID, ask for the IDs under each group, and do the counting yourself.
The rest of this post covers what the research says, the four places it usually goes wrong, a setup you can copy, and a worked check that shows why the IDs matter.
What does the research say?
The most direct test comes from the British Election Study. Researchers asked six language models to sort "most important issue" answers into 50 categories, and compared them with a second human coder. With no existing training data, the best model scored 93.9% accuracy and the human 94.7%. In a second test on new text, the best model scored 80.9% and the human 88.6% (Mellon et al.).
So on a clear question with a fixed list of groups, the sorting is close to human. Two things weaken that result for a small team.
First, it varies a lot by model and by group. A 2025 study of German survey answers found that "only a fine-tuned LLM achieves satisfactory levels of predictive performance." The models were better at some categories than others, and that "results in different categorical distributions" (von der Heyde et al.). In plain words: the counts come out different from a human's, even when most single replies are sorted right.
Second, those studies had a fixed list of groups written by researchers. When you paste replies into a chat and ask "what are people saying?", the model writes the list too. Nobody has checked that list.
Pew Research Center now uses AI for "coding open-ended survey responses into categories," and says "humans oversee every aspect of our work" (Pew). That's the right model to copy: the tool sorts, a person checks.
Where does ChatGPT go wrong with survey replies?
1. The counts. In a 2023 thread on OpenAI's developer forum, someone asked the model to summarize 80 answers to a yes, no or maybe question. It reported on 64 of them: 52 yes, 8 no and 4 maybe. The real counts were 62, 11 and 7 (OpenAI forum). Models have improved since then, but a number that reads cleanly is still not a number that was counted. Ask for counts and you get an estimate that looks like a count.
2. Two problems under one label. "Pricing" and "too expensive" are the labels a summary reaches for first. They are also the labels most likely to hide two different stories, like a renewal price jump and a person who barely used the product. The summary reads fine. The fix it points to is wrong for half the people inside it.
3. Replies in the middle of a long paste. A 2023 study found that language models do best with information "at the beginning or end of the input context" and get worse when it sits in the middle of a long one (Liu et al.). Paste 400 replies in one go and the ones in the middle get less attention.
4. The label becomes the requirement. Once a group has a tidy name, people stop reading the replies under it. "Wants a mobile app" turns into a roadmap item, even if the replies say "I need to check one invoice on my phone." The name was a guess. The replies were the evidence.
How to set it up so you can check it
- Give every reply an ID. Put the replies in a spreadsheet, one per row, with a number in column A. In the forum thread above, the poster thought the chat version did better because each answer was numbered.
- Ask for groups from a sample first. Paste 50 replies and ask for a list of groups, each named after what happened to the person, not how they felt. Edit the list yourself.
- Sort in batches, and ask for IDs, not counts. Send 100 or so replies at a time with your list. A prompt like this works:
Below are survey replies, one per line, each starting with an ID.
Put each reply in one or more of these groups: [your list].
If a reply fits none, put it in "Other".
Return a table with two columns: ID, group. Do not summarize or count.
- Count in the spreadsheet. Paste the table back next to your replies and count with
COUNTIF. Now every number comes from a row you can see. - Read every reply under the three biggest groups. This is where you catch the merged problems. Split groups that hold two stories, then recount.
A worked check
The numbers below are invented to show the method. They are not from a real survey.
Say 120 people who cancelled an invoicing app answered "What was happening when you decided to cancel?" A one-shot summary comes back with "Pricing concerns (about 35%)". Running the setup above gives this:
| What you see | Replies |
|---|---|
| Summary's estimate for "Pricing concerns" (35% of 120) | 42 |
| IDs the model actually listed under "Pricing" | 31 |
| Of those, the price rose at renewal without warning | 14 |
| Of those, too little use to justify the cost | 11 |
| Of those, about something else (slow exports, missing feature) | 6 |
Worked out from the table: the estimate was 11 replies too high, and 6 of the 31 listed replies weren't about price at all. The real finding is two groups of 14 and 11, and they need different fixes. One is a warning email before renewal. The other is getting people to use the product in the first place. A discount fixes neither.
Where a survey tool fits
If you'd rather not run this by hand, Enform reads every reply, counts what people said, and every count opens to the replies behind it, so step 5 is one click. It can also write the survey from a description and draft a deck from the results. Our roundup of survey tools that summarize open-ended answers compares how the main ones handle this, and pricing is here. Whatever you use, the step-by-step method in how to analyze open-ended survey responses still applies.
FAQ
Can ChatGPT analyze open-ended survey responses accurately?
On a clear question with a fixed list of groups, the best models in published tests sorted answers nearly as well as a human. Accuracy varies by model and by group, though, so the counts can come out different from a human's.
Can I trust the percentages ChatGPT gives me?
Not on their own. Ask it to return each reply's ID and group instead of counts, then count in a spreadsheet so every number comes from rows you can see.
How many survey responses can I paste into ChatGPT at once?
Send about 100 at a time. Language models pay less attention to text in the middle of a long input, so smaller batches keep every reply in view.
What should I check before acting on an AI summary of survey replies?
Read every reply under the three biggest groups. That's where one label usually turns out to hold two different problems that need different fixes.
Sources
- How Pew Research Center is and is not using AI in our work, Pew Research Center (2026)
- Do AIs know what the most important issue is? Using language models to code open-text social survey responses at scale, Mellon et al., Research and Politics (2024)
- AIn't Nothing But a Survey? Using Large Language Models for Coding German Open-Ended Survey Responses on Survey Motivation, von der Heyde et al. (2025)
- Lost in the Middle: How Language Models Use Long Contexts, Liu et al. (2023)
- Very poor results when summarizing questionnaire responses, OpenAI Developer Community (2023)