Use an open-ended question when you don't yet know what people will say, and a closed-ended question when you already know the likely answers and need to count them. Each format skews results in its own way: a list puts answers in people's mouths, and an open box gets skipped and only catches what's on someone's mind.
The rest of this post shows how each format misleads, with numbers from published experiments, then a simple order that uses both: ask open first, then build your list from what people wrote.
What's the difference?
A closed-ended question gives people a fixed set of answers to pick from: multiple choice, yes or no, a 1 to 5 scale. An open-ended question gives them a box and lets them answer in their own words.
Closed answers are quick to give and quick to count. Open answers take longer on both ends, but they can tell you something you didn't think to ask about. Most guides stop there. What matters more is how each one can give you the wrong picture.
How a closed list can mislead you
The list suggests the answer. After the 2008 US election, Pew Research Center asked voters which one issue mattered most to their vote. When the economy was one of the choices, 58% picked it. When people wrote their own answer, 35% named the economy (Pew). Same question, two formats, and a 23-point gap between them.
The list hides what's missing. In that same experiment, fewer than one in ten people on the closed version gave an answer outside the five they were read. On the open version, 43% gave an answer that wasn't on the list at all (Pew). The closed version didn't show that gap. It just looked complete.
Older experiments found the same thing. Howard Schuman and Stanley Presser ran a similar experiment in 1979 about what people want most in a job. Nearly 60% of the open answers didn't fit any of the five choices researchers had written (Bit By Bit). Matthew Salganik, who describes the study, notes that closed questions came to dominate surveys "not because closed questions have been proven to provide better measurement, but rather because they are much easier to use."
Small details in the list change the answers. Pew notes that the order of choices affects what people pick: on the phone people lean toward items they hear last, and on paper or web surveys toward items at the top. So Pew randomizes the order of many answer lists (Pew). The numbers on a scale matter too. In a 1991 study, people rated their success in life on an 11-point scale. When the scale ran from 0 to 10, 34% chose 0 to 5. When it ran from -5 to +5, only 13% chose the matching range, -5 to 0 (Wikipedia, summarising Schwarz and colleagues). The words at each end of the scale were the same. Only the numbers changed.
How an open box can mislead you
People skip it. On Pew's panel, people skip closed questions 1% to 2% of the time on average. For open questions, "double-digit item nonresponse rates are common." Questions that ask for several sentences get skipped more than ones that ask for a word or a phrase (Pew). The people who do answer may not be like the ones who skip.
It only catches what's top of mind. Look at the Pew numbers again from the other side. 58% picked the economy when they saw it, but only 35% thought to write it. An open box undercounts things people care about but don't think of in the moment. A low count in open answers means "few people raised it", not "few people care".
Counting is harder, so people guess. With 300 written replies and no time, it's tempting to skim and remember the vivid ones. That's how one angry reply turns into "everyone hates the new pricing". Open answers are only as honest as the counting you do afterwards. The post on how to analyze open-ended survey responses walks through labelling and counting them.
Which one should you use?
| Situation | Use | Why |
|---|---|---|
| You don't know the likely answers yet | Open | A list you guessed will hide the answers you missed |
| You want the reason behind a score or choice | Open, as a follow-up | The score says how much, the words say why |
| You know the answers and need exact counts | Closed | Faster to answer, fewer skips, easy to compare |
| You'll ask the same thing again next quarter | Closed, built from earlier open answers | Same list each time, so changes are real |
| You're asking several yes-or-no things | A yes or no for each item, not "select all that apply" | Pew found forced-choice questions gave more accurate answers, especially on sensitive topics |
The table is a rule of thumb drawn from the sources in this post, mostly Pew. Your own case may differ.
Start open, then build the list from what people wrote
Pew describes an approach researchers sometimes use: run a pilot with open questions to see which answers are most common, then write closed questions that use those answers as the choices (Pew). You can do a small version of this with any survey.
The numbers below are invented to show the method. They are not from a real survey.
Say you run a budgeting app and want to know why people sign up. You were going to offer four choices: save money, pay off debt, track spending, plan a big purchase. Instead you ask the first 50 people who sign up an open question: "What made you start looking for a budgeting app this week?" 41 of them answer. You label and count the replies:
| What people wrote (labelled) | Replies |
|---|---|
| Moved in with a partner, splitting bills | 12 |
| Track spending | 9 |
| Pay off a card or loan | 8 |
| Irregular income, freelance or gig work | 7 |
| Save for something specific | 5 |
Two of your top four answers weren't on your list. "Moved in with a partner" was the biggest group, and "save money" never came up in those words. Your next sign-up survey gets a closed question with these five choices plus "Something else", and one short open box for people who pick it. Now you can count reasons for thousands of sign-ups, and the list came from people instead of your guesses.
Keep the open box running in a small way. If "Something else" climbs past about one in ten answers, read those replies again. There's a new choice in there.
Where Enform fits
The slow part of this method is reading and counting the open answers before you write the list. Enform writes the survey from a description, you share one link, and it reads every reply and counts what people said. Every count opens to the replies behind it, so you can check a label before it becomes a choice on your next survey. If you're drafting questions now, the survey question checker flags leading wording and answer lists with no way out, and the onboarding feedback survey template is a good starting point for the sign-up example. Plans are on the pricing page.
FAQ
When should I use open-ended questions?
When you don't yet know what people will say, or when you want the reason behind a score or choice. In Pew's experiment, 43% of people on the open version gave an answer that wasn't on the closed list.
When should I use closed-ended questions?
When you already know the likely answers and need to count them or compare them over time. People skip closed questions far less often than open ones, 1% to 2% on Pew's panel.
Do open-ended questions give more honest answers?
They give different answers, not always more honest ones. An open box avoids putting answers in people's mouths, but it only catches what people think of in the moment, and more people skip it.
How many open-ended questions should a survey have?
Keep them short. Pew found that open questions asking for several sentences get skipped more than ones asking for a word or a phrase.
Sources
- Writing Survey Questions, Pew Research Center
- Why do some open-ended survey questions result in higher item nonresponse rates than others?, Pew Research Center (2021)
- Bit By Bit: Social Research in the Digital Age, 3.5.2 Wiki surveys, Matthew Salganik
- Norbert Schwarz, Wikipedia (summarising Schwarz et al., Public Opinion Quarterly, 1991)