# Plain Language in Surveys: Pretest a Question Before Launch

Plain language and surveys: test one question with the intended audience, find ambiguities, and revise without leading answers.

- Canonical URL: https://www.harmate.com/en/blog/plain-language-in-surveys-pretest-a-question-before-launch
- Author: Harmate Team
- Published: 2026-09-22
- Updated: 2026-09-22T03:11:34.515988+00:00
- Language: en

## Content

# Plain Language in Surveys: Pretest a Question Before Launch

A question can look obvious to the person who wrote it and still produce answers that are hard to compare. A word may carry several meanings, the reference period may be unclear, or the available choices may not match the path a respondent actually took. Rereading the sentence is not always enough: you need to observe what people close to the intended audience do with it.

This guide offers a short way to test one question before launch. It does not certify comprehension and it is not a substitute for a full pilot. It helps you find an ambiguity, document it, and revise the instrument without manufacturing a correct answer.

## Plain language helps, but it does not prove comprehension

The [French Interministerial Directorate for Public Transformation](https://www.modernisation.gouv.fr/actualites/le-langage-clair-engage-le-fond-comme-la-forme) presents plain language as a method concerned with both substance and form: information should be useful, accessible, and understandable to its users. It is therefore more than a search for shorter words. A short sentence can remain ambiguous if it does not specify the situation, the expected action, or the time it refers to.

In a survey, respondents cannot see the author’s intention. They see a question, possible answer options, and the context created by earlier items. The [Pew Research Center](https://www.pewresearch.org/writing-survey-questions/) describes wording, order, and response format as parts of questionnaire quality. A question that feels clear to a team can therefore lead two people to different answers because they understood the task differently.

Plain language is a design objective. Observed comprehension during a pretest is empirical information. Keep the distinction visible: a checklist can flag jargon, double-barreled sentences, or vague timeframes, but only a session with people close to the intended audience shows the interpretations the question actually produces.

## What a pretest should observe

A question-focused pretest should not ask only: “Is this sentence clear?” A yes does not reveal which interpretation the respondent has chosen. Instead, follow four parts of the response process:

- the meaning assigned to words;
- memories or information used;
- the judgment behind an answer;
- how that judgment fits the available response format.

The [CDC](https://www.cdc.gov/nchs/ccqder/question-evaluation/cognitive-interviewing.html) describes comprehension, recall, judgment, and response probes as ways to examine how respondents answer survey questions. The purpose of a cognitive interview is to identify question problems, not to estimate their frequency in a population.

Hesitation is not automatically misunderstanding. It may result from incomplete memory, a sensitive topic, lack of experience, or answer options that are too narrow. Record what you observed and what remains a hypothesis. A short answer does not prove low comprehension either.

## A 20-minute question pretest

The protocol below is a working proposal for a first pass, not a methodological standard or a sample-size rule. It is designed for one question at a time with one person close to the intended audience. To make a decision, repeat it with several relevant profiles and compare the problems you observe.

**Minutes 0–2: set the frame.** Explain that you are testing the question, not the person. Say that there is no right or wrong answer and that you will not correct their vocabulary. Check the minimum context: role, relevant situation, and when they would answer the survey.

**Minutes 2–6: let them answer naturally.** Present the question exactly as it would appear in the survey. Let the person read or hear it and answer without help. Note their spontaneous words, requests for clarification, time taken, and options considered. Do not interrupt to explain what you meant.

**Minutes 6–10: ask for the meaning they understood.** Use an open probe: “In your own words, what is this question asking?” Then ask what the main term or timeframe covers. Do not offer two interpretations to choose from; that would introduce your hypotheses into the answer.

**Minutes 10–14: follow recall and judgment.** Ask: “What situation did you think of?” and then: “What led you to that answer?” If the respondent compares several episodes, note which ones. If they say they do not know, ask what is missing for them to answer, without turning that gap into an error.

**Minutes 14–17: check the response format.** Ask whether the choices, scale, or text box let them say what they mean. Someone may understand the question but find no suitable option. That is a measurement problem, not only a language problem.

**Minutes 17–20: collect a suggestion without dictating it.** Ask: “What would make this easier to answer?” Let the respondent suggest a context, a timeframe, or a wording change. Record the suggestion as test data, not as the final wording to accept automatically.

Throughout the twenty minutes, keep three columns separate: the person’s exact words, the interviewer’s observation, and the revision hypothesis. “The term is understood as a technical skill” is a reported observation. “We should ask about technical skills” is a decision to test, not a fact.

## Example: a training follow-up question

The example and people below are fictional. A training team wants to know whether its follow-up questionnaire can capture how a course is used at work. It writes:

> To what extent did the training help you develop the skills needed for your current role?

The sentence sounds professional and positive, but it hides several decisions. “To what extent” calls for a scale whose points are not defined. “Skills needed” may mean technical tasks, a posture, or knowledge. “Current role” may not fit someone who recently changed jobs. Finally, “helped you develop” already suggests a relationship between the training and an observed change.

During the session, three people close to the audience produce different signals:

- Sophie thinks of one technical skill and chooses a high score because she liked the training, although she has not used that skill yet;
- Karim answers about his former role because he has just moved teams and does not know whether the question refers to his current job or the one he held during training;
- Lea understands the question but cannot find an option for saying that she has not yet had a chance to apply what was covered.

The issue is not that these people answered “wrong.” The question combines training experience, work context, opportunity to apply, and an assessment of effect. One possible revision is to split those operations:

1. “Since the training ended, have you used a skill covered in it?” with “Yes, several times,” “Yes, once,” “No, not yet,” “I do not know,” and “Not applicable.”
2. If yes: “What situation did you use it in?”
3. If no: “What explains why you have not used it?”
4. If unsure: “What information would help you say whether you used it?”

This version does not prove a transfer of skill. It distinguishes reported use, a described situation, and non-use whose reasons still need exploration. Not using a skill and not having an opportunity to use it are different situations. The pretest must still check whether “skill covered” is understood, whether the options cover the situations people encounter, and whether the question about reasons suggests an expected explanation.

A revision grid keeps the chain visible:

| Observed element | Words or behavior | Problem hypothesis | Revision to test | Remaining limit |
|---|---|---|---|---|
| “Current role” | Karim thinks of his former role | Time reference is ambiguous | specify “since the training ended” | job changes remain varied |
| “To what extent” | Sophie scores based on appreciation | scale is not anchored | ask about concrete use | use does not prove an effect |
| answer options | Lea cannot find “not yet” | categories are incomplete | add an explicit option | non-use still needs exploration |

## Revise without manufacturing the right answer

Start with the problem most likely to change the answer: a misunderstood term, an uncertain timeframe, a double operation, or a missing option. When possible, rewrite one dimension at a time. Otherwise, you will not know what fixed the difficulty.

Prefer an observable context to an abstraction: “since the training ended” is easier to check than “recently.” Ask for a situation, action, or period before asking for a general assessment. If the survey must measure an opinion, keep that purpose, but check whether respondents use the same mental scale.

Do not give an example answer during the test. An example may help reading, but it can also supply a category or interpretation the respondent would not have produced. If an example is essential in the final version, test its effect separately and record what it makes visible or hides.

The [2024 GESIS cognitive pretesting guideline](https://www.gesis.org/fileadmin/admin/Dateikatalog/pdf/guidelines/cognitive_pretesting_lenzer_hadler_neuert_3.0_2024.pdf) distinguishes cognitive pretesting from piloting: the first explores selected questions in depth, while the second checks the questionnaire as a whole under conditions closer to the main fieldwork. Both can be useful, but they answer different questions.

## What Harmate can check, and what the field must decide

Harmate can help create or edit a questionnaire and perform an initial structural review. That review runs structural and syntactic checks, such as empty fields, unusable choices, exact duplicates, or syntactically separate requests within one question. It does not independently establish suitability for the audience or purpose. It cannot see what a person understands before you show them the question. It does not certify comprehension, neutrality, or field validity.

After the pretest, keep the original wording, session notes, disagreements, and revised version. The [open-ended questions guide](/en/blog/open-ended-questions-get-actionable-data-without-bias) explains why free responses provide richer material for exploring interpretations than imposed categories. Closed questions remain useful for comparing established categories; richer exploration does not automatically mean validity or representativeness. If the survey prepares a training course, the [training placement questionnaire guide](/en/blog/training-placement-questionnaire-how-to-keep-it-short-without-losing-quality) explains how declared answers can inform preparation without becoming a diagnosis of a person. For a longer oral session, the [semi-structured interview guide](/en/blog/semi-structured-interview-guide-method-steps-and-15-examples) describes a different setup: it is not reducible to testing one sentence.

## Pre-launch checklist

- Does the question ask for one identifiable operation?
- Do people in the intended audience understand the key words in the same way?
- Is the reference period or situation explicit?
- Can someone answer “not yet,” “do not know,” or “not applicable” when those cases exist?
- Do the options cover the paths actually observed?
- Does the session distinguish meaning, recall, judgment, and response mapping?
- Are probes open and non-corrective?
- Do the notes separate verbatim, observation, hypothesis, and decision?
- Was the revision tested again with people close to the audience?
- Does the questionnaire keep a trace of what remains unknown?

## Frequently asked questions

### How many people do you need for a pretest?

There is no magic number for one question. The choice depends on the audience, the diversity of situations, and the cost of misunderstanding. For a first pass, seek several relevant profiles and stop when the recurring problems are understood well enough to decide on a revision. This is not a population estimate.

### Do you need to test every question?

Prioritize new questions, items central to the decision, sensitive or translated questions, questions aimed at a poorly known audience, and items with difficult response choices. A targeted test is better than a superficial validation of every item. After a substantial change, test the affected items again.

### What is the difference between a pretest, a pilot, and a semi-structured interview?

A pretest examines whether questions work for the people who must answer them. A pilot recreates the fieldwork conditions as closely as possible and checks the questionnaire as a whole. A semi-structured interview explores a topic through a conversation guide. One project can use all three, but their aims and records are not interchangeable.

### Can AI replace this test?

AI can flag a long sentence, suggest variants, or help organize notes. It cannot replace observing a person close to the intended audience, nor decide alone that the question is understood as intended. The final wording must remain linked to field observations and a human decision.