Analyzing 200 Open Responses Without Bias
Reading a few open-ended responses is easy. Reading 200 is another story. After a few dozen minutes, reading becomes impressionistic. Striking sentences take up disproportionate space. Weak signals, though frequent, disappear. A summary may seem clear while remaining fragile.
The goal is to transform a large volume of text into actionable decisions, without overinterpreting, without twisting the meaning, and without confusing intuition with proof.
The Main Trap: Striking vs. Frequent
The most common trap is confusing "striking" with "frequent." An emotional, well-written, or surprising answer grabs attention. It may represent a rare case. Conversely, a recurring irritant, described in a banal way, may go unnoticed. A reliable method therefore imposes two disciplines: structuring the reading and counting.
Define the Expected Output Before Reading
A successful analysis begins before reading. Without an expected output, attention scatters and the analysis becomes a collection of interesting but unusable findings.
A single output must be chosen. For example: a handful of concrete decisions to take, a short list of priority problems, a typology of response profiles, or a matrix to help arbitrate. This choice changes everything, as it determines what is "signal" and what becomes "noise."
The 7-Step Method
1. Frame the Context
Before analysis, the framework must be set. The same text does not mean the same thing depending on the audience, the moment, and the situation. An answer like "I lack time" does not mean the same thing in an intensive training, an onboarding, or a project under tension.
2. Clean Without Distorting
Cleaning does not mean rephrasing. It consists of removing what prevents correct reading: exact duplicates, empty answers, off-topic answers. Very short answers deserve to be kept but treated separately (they often indicate low engagement or unclear questions).
3. Sample to Build a Theme Dictionary
Rather than coding 200 responses immediately, a first reading of a sample allows identifying the themes that actually exist in the corpus. This builds a dictionary of themes with simple, concrete, and stable labels. A good theme describes a fact, not a judgment.
4. Code by Distinguishing Topic and Signal
This is crucial. An open response rarely says one thing. It speaks of a Topic (what it is about) and transmits a Signal (the intention: a blockage, an expectation, an irritant).
Distinguishing the two prevents mixing elements that look superficially similar but require different actions.
5. Count, Then Verify by Targeted Re-reading
Once the dictionary is stabilized, the full corpus can be coded. Counting provides a map of the terrain. But counting alone is not enough. Targeted re-reading is necessary to ensure the same theme has not been used to cover different situations.
6. Add Impact Without Inventing
Frequency is not synonymous with priority. Some rare subjects have massive impact. The rule: impact must only be evaluated based on indices present in the responses (described consequences, repetition over time). Without indices, impact remains "unknown."
7. Convert Analysis into Decisions
An analysis must end with decisions. Otherwise, it remains reading material. The robust format: for each major theme, indicate its frequency, observed impact indices, and an associated operational decision (clarify an objective, adapt a pace, separate groups).
5 Frequent Biases to Neutralize
-
Anecdote Bias: A strong story crushes the rest. -> Remedy: Counting.
-
Confirmation Bias: Looking for what was expected. -> Remedy: Write hypotheses before reading.
-
Broad Theme Bias: A theme engulfs everything. -> Remedy: Break it down.
-
Cause-Symptom Confusion: The cause is a hypothesis, not a fact. -> Remedy: List it as a hypothesis.
-
Illusion of Unanimity: Ignoring counter-examples. -> Remedy: Actively seek tensions.
Conclusion: Rigor Over Intuition
Reading 200 open-ended responses is not a stylistic exercise. It is an exercise in rigor. A reliable method frames the output, builds a theme dictionary, counts, qualifies impact only when evidence exists, and ends in decisions.
It is less spectacular than a brilliant intuition, but significantly more solid. And for those who handle high volumes, this is exactly the rigorous process that Harmate's AI automates to save you days of coding.
