How to run a community survey (and count the write-ins)

Members spot gaps you can't. Open source projects have done this for years. Before its 2019 survey, Ember asked the community to "review the list of 2019 survey questions" and to "submit feedback as an issues on that repo," saying "we want your input before we go live" (Ember). In 2010 the Eclipse Foundation did the same on its blog: "Before we field the survey, I would like to get some feedback on the questions" (Eclipse).

Do it in the place members already talk: a pinned message in Discord or Slack, a forum thread, a repo issue. Give it a few days and a clear end date. Ask two things: what is missing, and which question is confusing or doesn't apply to you.

Put the open question first. Pew Research Center found that when closed questions come before an open one, people "are much more likely to mention concepts or considerations raised in those earlier questions" (Pew). If you list ten features and then ask what members want, you'll get back your own list.

Put the questions about the person last. Pew's guidance is that demographic questions "should not be asked near the beginning of a survey." For a community, that means role, region, and how long they've been around go at the end, and stay optional.

Keep every written question short. In Pew's panel, questions that asked for several sentences had a median of 18% left blank, against 13% for one-word and short-phrase questions (Pew). "What is one thing you'd change about this community?" is enough.

Less than you'd like. A community survey is opt-in: members choose to answer. CASRAI describes self-selection bias as "a specific, common form of non-response bias where participants opt themselves into a study," and notes it is harder to detect when there is no list of everyone to compare against (CASRAI).

Planet Argon, which has run the Rails community survey for years, is blunt about this. In 2016 its COO wrote that the data came "from a relatively small sample set (compared to the community as a whole)," that "we don't know what the margin of error is here," and that he wouldn't "present anything as a conclusion" (Planet Argon).

So report the number of people who answered, say how you invited them, and write "of the people who answered" instead of "our members think." The written answers are still useful. They show you what the people who care are saying, and a single clear reply can change what you do.

Free-text answers are where a community tells you what you didn't think to ask. They are also what gets grouped by hand over a weekend. Here is a method that holds up:

1. Read the first 30 or so answers before you decide on any labels.
2. Give each answer a short label for what it says, such as "meetups feel too far away" or "hard to find old answers."
3. Let one answer carry two labels if it says two things.
4. Count the labels, and keep each person's own words next to the count.
5. Reread the replies under your top three counts. A label that looks tidy often hides two different problems.

The numbers below are invented to show the method. They are not from a real survey.

Say a project community of 400 members gets 160 answers to "What is one thing you'd change about this community?"

Counts add to 160 because each answer got one label here (worked out from the table). The top two rows, 67 of 160, are about being able to find and get help. That is two of the biggest counts, and a multiple choice list written in advance might not have offered either one.

For the longer version of this method, see how to analyze open-ended survey responses. This is the step where Enform does the counting. The tour below shows it reading a batch of replies and opening each count to the answers behind it.

Often they have more to say than the questions let them. In Ember's 2020 survey, "about 20% of respondents chose to leave some final remarks." The team reported that 70% of those remarks were encouraging, 25% were feature requests or neutral comments, and 5% were negative (Ember). The page doesn't say how the remarks were sorted.

Leave a last box that says "Anything else?" and read it. It is cheap to add, and some of what you most need to hear lands there.

Three ways to run and read it:

• Google Forms and a spreadsheet. Free to start and members know it. It won't count text answers for you, so the sorting is manual. See Enform vs Google Forms.
• A general AI chat. Paste the answers into ChatGPT and ask for groups and counts. It's quick, but check the counts against the answers before you publish them.
• A survey tool that reads the replies. Enform writes the survey from a description, you share one link, and it reads every reply and counts what people said. Every count opens to the replies behind it. If your survey is all fixed choices, any form tool will do. See pricing.

Quick polls with emoji reactions work for a single yes or no. They don't give you written answers.

Post the results where you posted the invitation. Give the number of people who answered, the top counts, a few replies quoted with permission, and one or two things you'll change. People answer the next survey when they saw the last one used.