# Sampling: Choose Who to Ask According to What You Need to Understand Sampling guide: decision tree, probability and non-probability matrix, selection bias, nonresponse, and recruitment plan. - Canonical URL: https://www.harmate.com/en/blog/sampling-choose-who-to-ask-according-to-what-you-need-to-understand - Author: Harmate Team - Published: 2026-09-22 - Updated: 2026-09-22T03:12:29.046481+00:00 - Language: en ## Content # Sampling: Choose Who to Ask According to What You Need to Understand Sampling is not primarily a question of size. It is a decision about the people or situations to which your answers can reasonably relate. The choice depends on the question, target population, field access, and intended conclusion. A probability sample can support population inference when the sampling frame, selection probabilities, and nonresponse are handled. A non-probability sample can be appropriate for exploring an experience, reaching hard-to-access cases, or understanding differences between situations. Neither turns a handful of answers into a general truth automatically. Official statistical guidance distinguishes probability and non-probability sampling and stresses that selection and nonresponse affect what can be inferred. The French national statistics office also notes that quotas and voluntary participation can introduce bias, and that increasing sample size does not repair selection bias ([Insee, Statistical sources and methods](https://www.insee.fr/en/information/8325542?sommaire=8325635); see also the [Insee methodological article](https://www.insee.fr/en/statistiques/fichier/8325542/INSEE_CS10_article6_EN.pdf)). For qualitative research, [Alvaro Pires’s essay on qualitative sampling](https://classiques.uqam.ca/contemporains/pires_alvaro/echantillonnage_recherche_qualitative/echantillonnage.html) connects selection to the construction of the research object and the relevance of cases, rather than to automatic statistical representativeness. Start by asking: what do you need to understand, and to whom must the answer apply? ## Start with the decision, not a number Write down: - the decision the study must inform; - the population or situations concerned; - the period and context; - the difference that would change the decision; - the limit you are willing to leave open. “Measure customer opinion” is not specific enough. “Decide whether support should offer a phone session to new users of one offer” defines a population, a situation, and a decision. “Understand why some people stop before first use” probably calls for stories and contrasting cases before a closed measure. An [actionable questionnaire](/en/blog/actionable-questionnaires-start-with-the-decision-not-the-questions) helps you start from the decision. A useful [market-research approach](/en/blog/how-to-do-useful-market-research-listen-before-you-measure) can combine documents, open answers, and measurement once categories are clear. These methods do not produce the same evidence. ## A textual decision tree Use this tree before recruiting: 1. **Do you need to estimate a proportion or compare a defined population?** - Yes: seek a frame that covers that population and examine whether a probability design is possible. - No: continue. 2. **Do you need to understand situations, mechanisms, or exceptions?** - Yes: consider purposive, contrast, or relevant-case selection; define what each case should help you learn. - No: clarify the decision before choosing recruitment. 3. **Does each person in the population have a known or calculable selection probability?** - Yes: document the draw, strata, exclusions, and nonresponse. - No: this is a non-probability design; do not present the result as representative without additional evidence. 4. **Will recruitment rely on volunteers or an easy-to-reach channel?** - Yes: record who the channel excludes, who is likely to respond, and how this may shape the result. - No: describe the frame, criteria, and refusals observed. 5. **Is one experience category enough?** - No: plan useful contrasts, such as new/experienced or successful/abandoned use, without assuming that the contrast itself explains the cause. - Yes: still check absences and edge cases. The tree does not select a method for you. It makes the relationship between purpose, selection, and conclusion visible. ## Compare the main choices | Selection mode | What it can provide | Risk or limit to document | |---|---|---| | Probability | Estimation and comparison when frame and probabilities are defensible | Incomplete coverage, nonresponse, cost, and weighting assumptions | | Quota | Controlled presence of known categories in practical recruitment | Quotas correct only selected variables; unseen differences remain | | Volunteer | Fast access to motivated or affected people | Self-selection, strong experiences, and channel exclusion | | Purposive | Cases chosen for experience, contrast, or qualitative relevance | No statistical generalization by itself | | Snowball | Access to people who are difficult to reach through referrals | Close networks, homogeneity, and dependence between referrals | | Convenience | Low-cost first exploration | Very limited scope; not a mirror of the market | Quotas and volunteers can be useful, but they are not random draws; unmeasured variables can preserve bias regardless of size. ## Probability and non-probability: do not mix promises A probability design should describe its frame, probabilities, exclusions, and nonresponse. A non-probability design may choose contrasting situations or cases that illuminate a mechanism; it is defensible when tied to the question and its limits are explicit. It does not become representative because answers are numerous or consistent. Also distinguish a missing response from a missing person. Someone who never opened the questionnaire, stopped halfway, or refused a question is not a neutral result. Record channels, invitations, drop-offs, and unanswered items when they change interpretation. ## A six-step recruitment plan ### 1. Define the accessible population Write the target population and the part you can actually reach. If you study service users, state whether inactive accounts, former customers, people without a known address, or indirect users are included. ### 2. Choose the unit of selection The unit can be a person, household, team, case, event, or situated answer. A list of people is not enough if the decision concerns events or use situations. ### 3. Describe criteria State inclusion, exclusion, and contrast criteria. Avoid criteria whose only purpose is to make a flattering profile. If the topic is sensitive, limit the data requested and explain how it will be used. ### 4. Choose channels Document where invitations appear: an existing base, partners, an open call, networks, direct outreach, or referrals. For each channel, ask who will not see it and who may be more likely to respond. ### 5. Plan for absence Define a rule for undelivered invitations, refusals, drop-offs, and partial answers. Do not automatically replace an absent person with a “similar” person; substitution can change the question. ### 6. Stop or continue for an explicit reason In qualitative work, continue when a new situation could change the interpretation or reveal an important case; stop when the decision is sufficiently informed and new answers mainly add already-understood variants. In probability work, precision depends on design, variance, and assumptions: do not choose a universal threshold outside context. ## Invented example: support for new users Invented example: a team must decide whether to add an onboarding session for new users of a booking service. A list of recently created accounts is available, but it excludes people who stopped before creating an account. If the goal is to understand obstacles, the team purposively selects three situations: signup completed without a booking, first booking completed, and payment abandonment. It asks for open accounts, keeps the cases, and states that this selection is for understanding journeys, not estimating their share. If the goal changes to estimating the rate of abandonment among all visitors, the team must revisit the frame, invitations, nonresponse, and denominator. The earlier interviews can help phrase response options, but they do not provide that estimate. When a training provider compares two cohorts, keep the invited cohort separate from the respondent cohort. If absentees, refusals, and dropouts are replaced with volunteers, the observed difference may reflect participation rather than experience. For each cohort, retain delivered invitations, refusals, dropouts, and partial responses; do not infer the views of people who did not answer. [Open-ended question guidance](/en/blog/open-ended-questions-get-actionable-data-without-bias) can help explore situations before designing a measure. The plan changes when the decision changes. ## Where Harmate can help Harmate can collect and import answers, retain criteria and sources, compare themes, and prepare a report showing absences, disagreement, and the scope actually observed. It does not provide a panel, automatically select a probability design, or turn a convenience sample into a representative population. A good report should answer four questions: who could participate, who actually answered, which situations are missing, and what conclusion is reasonable under these limits? If the answer is unavailable, reduce the promise before increasing size. Sampling is not a recipe for a reassuring number; it is a way to connect a real selection to an honest decision.