# Analyzing 200 Open Responses: A 7-Step Method for Decisions How do you analyse 200 open responses? Frame the corpus, develop themes, count carefully, and revisit counterexamples before deciding. - Canonical URL: https://www.harmate.com/en/blog/analyzing-200-open-responses-without-bias-a-method-for-actionable-decisions - Author: Harmate Team - Published: 2025-12-13 - Updated: 2026-10-18T00:43:53.386917+00:00 - Language: en ## Content # Analyzing 200 Open Responses: A 7-Step Method for Decisions Reading a few open-ended responses is easy. Reading 200 is another story. After a few dozen minutes, reading becomes impressionistic. Striking sentences take up disproportionate space. Weak signals, though frequent, disappear. A summary may seem clear while remaining fragile. The goal is to transform a large volume of text into actionable decisions, without overinterpreting, without twisting the meaning, and without confusing intuition with proof. ## The Main Trap: Striking vs. Frequent The most common trap is confusing "striking" with "frequent." An emotional, well-written, or surprising answer grabs attention. It may represent a rare case. Conversely, a recurring irritant, described in a banal way, may go unnoticed. To reduce this risk, structure the reading and count occurrences, then return to the responses to check their meaning. ## Define the Expected Output Before Reading Analysis benefits from a clear starting question. Without a question or decision to inform, attention can scatter and produce interesting but unusable findings. It can help to state an expected output, such as a list of issues to compare or a decision matrix. This framing influences what counts as a signal; keep it open to revision if the responses raise other topics. ## The 7-Step Method ### 1. Frame the Context Set out a provisional scope before analysis: who responded, to which question, in what situation, and when. “I don’t have enough time” can mean different things in intensive training, onboarding, or a project under pressure. Revisit the scope if the responses reveal a relevant context you had not anticipated. ### 2. Prepare the Corpus Without Distorting It Keep the original corpus. If an empty or off-topic response, or a technical duplicate introduced during import, is excluded from a working copy, record the rule and reason. Do not automatically remove identical responses: they may be separate, legitimate contributions. Avoid changing wording, style, or vocabulary; how people express themselves is part of the information. ### 3. Sample to Surface Theme Hypotheses Rather than coding 200 responses immediately, an initial reading of part of the corpus can suggest provisional themes. A useful label is concise and describes what is reported, not a judgment about the people who wrote it. Check it against other responses and revise it when needed. ### 4. Code by Distinguishing the Topic from Reported Details A response may concern a **topic** (such as an instruction) and include **reported details** about a situation, expectation, or difficulty. Those details do not by themselves reveal the respondent’s intention: interpret them in context and retain the original wording. This distinction can help avoid combining responses that sound similar but describe different situations. ### 5. Count, Then Verify by Targeted Re-reading Once working themes are clear, code the corpus using a consistent rule. Counting provides a reference point, but it does not explain a theme on its own. Re-read examples to check that one theme is not covering different situations. Textual-statistics methods can help identify patterns, but they do not replace back-and-forth with the corpus or knowledge of the topic, as [Bénédicte Garnier and France Guérin-Pace explain in their presentation on open-ended questionnaire analysis](https://mate-shs.cnrs.fr/actions/tutomate/tuto06-garnier-guerin/). ### 6. Add Impact Without Inventing Frequency alone does not establish priority. Assess impact using evidence in the responses, such as described consequences or repetition over time. If information is missing, state the concrete question to investigate—for example, what consequences people observed and over what period—and identify what other observation could help answer it. ### 7. Convert Analysis into Decisions An analysis can inform a next step. For each useful theme, state its frequency under the chosen rule, observed evidence, and a possible operational response (clarify an objective, check a pace, or review a group arrangement). A finding with no immediate action may instead point to a question that still needs investigation. ## 5 Reading Risks to Watch For 1. **Anecdote Bias:** A strong story crushes the rest. -> *Check: compare occurrences and revisit the context.* 2. **Confirmation Bias:** Looking for what was expected. -> *Check: record assumptions before reading, then look for evidence that challenges them.* 3. **Broad Theme Bias:** A theme engulfs everything. -> *Check: revisit excerpts and separate different situations.* 4. **Cause-Symptom Confusion:** The cause is a hypothesis, not a fact. -> *Check: keep a cause as a hypothesis until other evidence supports it.* 5. **Illusion of Unanimity:** Ignoring counter-examples. -> *Check: look for conflicting responses and state who responded.* Fidelity to wording and context continues in the guides to [verbatim quotes](/en/blog/verbatim-how-to-use-exact-quotes-without-distorting-their-meaning) and [word clouds](/en/blog/word-cloud-how-to-create-one-without-distorting-your-responses). For the distinction between coding, themes, and interpretation, see the guide to [thematic analysis](/en/blog/thematic-analysis-codes-themes-and-interpretation). ## Conclusion: From Corpus to a Verifiable Decision ### Fictional example: from a response to a bounded decision Two people write: “I did not have time to finish the exercise” and “We spent half the workshop installing the software.” The provisional theme is “insufficient practice time.” A third person adds: “There was enough time, but I did not understand the instructions.” The counterexample prevents the conclusion that the whole course moves too fast. Separate installation delays from difficulty understanding instructions. For the next session, prepare installation in advance and test the instructions with an example, then check what changes. These three fictional excerpts illustrate the reasoning; they establish neither frequency in a real corpus nor causality. Reading 200 open responses is interpretive work. Framing the question, proposing themes, counting under an explicit rule, and returning to original wording can inform a decision, provided the limits of the corpus remain visible. The approach is useful when its choices remain traceable and open to review. Harmate’s [Open Questions workflow](/en/product/open-questions) can surface themes and keep source excerpts available for review; the team checks counterexamples and decides what to use. For automated assistance, see [AI survey analysis](/en/blog/ai-survey-analysis-dont-confuse-a-summary-with-a-decision).