EvalCompassEvalCompass
Data collectionQuantitative researchSamplingScale designOnline surveysBefore-and-after comparison

Survey / Questionnaire

Survey / Questionnaire
Also known as: Questionnaire survey、Survey、Questionnaire study

A method of collecting experiences, behavior and attitudes using common questions. Alongside clarity, record who received the questionnaire and who responded to judge the scope represented by results. [1]

Questionnaire design checklist — openReferences and checked sections
Difficulty
Intermediate
Cost
Medium
Duration
Illustrative planning durations, not common standards: design 2–4 weeks, fieldwork 1–4 weeks, analysis 1–2 weeks, depending on scale
Content updated: 2026-09-13Source checked: 2026-09-13 (includes AI cross-checking)No documented human reviewIncludes fictional examples

How it works at a glance

Questionnaires turn concepts into specific answerable questions. Align options, reference periods and distribution modes, and pilot differences in interpretation and burden. [1]

Distribution to everyone does not guarantee everyone's voice is collected. Report sampling, distribution and valid-response numbers and item missingness, and consider exclusion of people who find responding difficult. [1]

  1. Confirm purpose and measurement concepts
  2. Design questions and select scales
  3. Develop a sampling plan
  4. Pretest
  5. Conduct the survey and improve response
  6. Data cleaning and analysis
  7. Interpret results and state limitations
Sequence of implementation steps. See “Implementation steps” for details of each stage.

When to use it and when it is unsuitable

When to use it

  • Collect comparable quantitative responses using common questions
  • Numerically show attitude, knowledge or behavior changes before and after participation
  • Compare groups by area or characteristics
  • Compare other studies using standardized scales such as life satisfaction or psychological health
  • Understand conditions across a wide area with limited budget

When it is unsuitable

  • Combine interviews or other methods if understanding individual histories and context is the main purpose
  • Groups with substantial differences in questionnaire literacy or language ability, such as preschool children or low-literacy areas
  • Causal inference where response is unlikely: strong selection bias makes interpretation difficult

Participants, roles, and deliverables

Participants and roles

People responsible for design, distribution, collection, management and analysis. Set target numbers from population, sampling, required precision, comparison groups and expected response.

Main deliverables

  • Collected questionnaire dataset
  • Aggregation and analysis report: descriptive statistics and cross-tabulation
  • Methodological records: sampling, response and limitations

Implementation steps

  1. Confirm purpose and measurement concepts
    Organize what to measure, from whom and why against evaluation questions. Clarify concepts such as knowledge, attitudes, behavior and satisfaction and set indicators for each.
  2. Design questions and select scales
    Check existing scales' fit to targets, language and purpose and conditions of use. Check original items for answerability, neutrality and overlapping or missing options. Do not reuse existing scales unconditionally.
  3. Develop a sampling plan
    Define population, sample size calculated from power and effect size, and sampling method: random, systematic, stratified or convenience. Consider oversampling for anticipated non-response.
  4. Pretest
    Pilot with diverse intended respondents to check understanding, time, difficult items and technical faults. Judge numbers needed from purpose and problems found.
  5. Conduct the survey and improve response
    Offer accessible response routes and provide information and reminders with regard to burden and contact consent. Check response patterns and non-response bias. Response rate alone does not establish representativeness.
  6. Data cleaning and analysis
    Check missing values, outliers and consistency, including invalid responses such as minimum-time completion throughout. Use descriptive statistics, cross-tabulation and change tests such as paired t-tests to answer evaluation questions.
  7. Interpret results and state limitations
    Honestly report response- and sampling-related bias. Distinguish statistical significance from substantive importance or effect size and avoid overinterpretation.

Strengths and limitations

Strengths

  • Efficient standardized collection from many people
  • Aggregation, comparison and statistical analysis as quantitative indicators
  • Anonymity is relatively easy, supporting answers on sensitive topics
  • Comparisons using existing scales may be considered when targets, conditions and scoring are comparable

Limitations

  • Cannot capture complex contexts or histories not anticipated in questions
  • High risk of distortion from low response and selection bias
  • Question quality determines credibility and requires specialist skills
  • Researchers may not notice misunderstandings of question intent

Common pitfalls

  • Distribution to everyone does not ensure representation. Report response rates, non-response reasons and known characteristic differences, explaining remaining bias.
  • Do not judge burden from item count alone. Pilot time and difficulty, remove unnecessary items and examine dropout.
  • Retrospective assessment of pre-program conditions is vulnerable to recall bias even if respondents are asked how things were before. Build pre/post collection into the original design.

Conditions of use and evidence

  • Pretest questions and interpret results in light of population reach and non-response. [1]
Fictional example; not actual results

Ask about use after training

A fictional illustration: ask all 40 evaluation trainees about plan use after 3 months.

ItemFictional valueInterpretation
Distribution targets40 peopleAll trainees
Valid responses30 peopleResponse rate 75%
People using plans in meetings18 people60% of respondents; proportion of all trainees remains uncertain

Common errors in judgment

Common mistake

Ask usefulness and behavioral change in one question

How to revise

Ask usefulness and actual use separately

ReasonA single response covering different concepts has no clear meaning.

Common mistake

Count all non-respondents as non-users

How to revise

Classify non-response separately and show denominators

ReasonReasons for non-response differ from whether use occurred.

Organize your thinking by writing

Questionnaire design checklist

A practical template is available to fill in and print on A4 paper.

Open worksheet

Related practical guides

Training effectiveness measurementBaseline surveyHow to design an evaluation questionnaireWhat are mixed methods?

Related methods

InterviewSecondary Data AnalysisIndicator DesignDescriptive Statistics and Cross-tabulationMixed Methods Research

References and checked sections

  1. Questionnaires
    BetterEvaluation
    Section consulted
    Advice for using this method
    How it relates to the text
    Pretest questions and interpret results in light of population reach and non-response.
    Source checked: (includes AI cross-checking)

Editorial and checking information

Content updated: Source checked: (includes AI cross-checking)No documented human reviewIncludes fictional examplesEditorial policy and scope of checkingCite this pageReport a correction

How to cite this page

EvalCompass Editorial Team. “Survey / Questionnaire”. EvalCompass, updated 2026-09-13. https://evalcompass.jun-nakatani.com/en/methods/survey/
Add the date you accessed the page. Text created by this site is available under CC BY 4.0. Quotations, external materials, and bundled fonts have their own terms of use. When reusing content, credit the source, link to the license, and indicate any changes.

Suggest a correction or improvement to this page

Change history
  • — Checked sources for definitions and conditions and added fictional examples, judgment rationales and completed entries for every field. Revised unsupported historical or uptake claims. Reviewed procedures and cautions, adjusting fixed counts, thresholds and institutional generalizations. Preserved field positions, types and counts while revising fixed conditions and thresholds.