Practical evaluation guide

From evaluation questions to a data collection plan

Use a fictional community-activity training example throughout to connect questions, indicators, denominators, measurement times, analysis, ethics and responsibilities to a concrete collection plan.

Summarize the evaluation design on one page

Summarize the evaluation design on one page

Main takeaway

Summarize on one page what change for whom will be checked, when, how and for which decision. Select indicators and methods after questions are decided. [1]

The community watch training here is fictional teaching material. Numbers, proportions, schedules and decision conditions are illustrative settings, not standard effects or durations.

One example connects training offered to 20 applicants to records after 4 weeks and decisions for improving the next session.

Summarize the evaluation design on one page

Connect decisions and questions with records, analysis, ethics, and responsibilities.

1. Derive questions from decisions

All organizations, schedules, numbers, results and judgment criteria below are fictional. A community group holds initial community watch training: 90 minutes offered to 20 applicants, with decisions about the next delivery after 4 weeks. Satisfaction alone does not provide enough decision information.

Check evaluation purpose, program maturity and time until change occurs. Early after launch, improving delivery conditions may need to come first.

Who uses findings: the operations lead and frontline staff. Also give affected people a brief explanation.

What is decided: whether to retain the same content next time or change exercises and post-participation support.

When used: fictional decision date, December 10, 2026. Schedule report completion before this meeting.

Important change: participants can confidently use the agreed community watch procedures in activities.

Practical framework or suggestion

2. Connect questions and records

Use one table for questions counting people, examining change and understanding reasons. Even for KPIs, state whether they measure activity volume or changes for people.

This is an evaluation matrix example. When adding methods, specify which question's gap they address. Record interview selection, missing voices and contrary cases too.

Fictional example: from questions to collection, analysis and decisions
Question / indicator position	What to collect, from whom and when	How to check and use it	Scope of claims
Did it reach intended people? / Outputs	Attendance and absence records for 20 applicants, recorded on the day. Voluntarily check first-time or experienced status too.	Participants ÷ applicants. Organize known absence reasons and adjust timing or information.	Shows participation volume, not participant behavioral change.
Were procedures understood? / Short-term outcomes	Brief situational judgment questions under identical conditions before and after training. Link the same individuals with matching IDs.	Check matched participants' responses and changes. Improve exercises using items where errors remain.	Changes among these respondents. Practice effects and external information may contribute; before-and-after differences alone do not establish causal training effects.
Were procedures used in activities? / Medium-term outcomes	After 4 weeks, voluntarily ask all 20 applicants about opportunities to use agreed procedures, actual use and difficulties.	Distinguish use, no opportunity and non-use. Record denominators and non-response and decide follow-up methods.	Self-reported use. Non-respondents' use and whether training caused it require separate checking.
What helped or hindered use? / Reasons and context	Interview volunteers. Deliberately seek users, non-users, people without opportunities, first-timers and experienced participants.	Organize reasons and exceptions alongside numerical results. Use them as improvement hypotheses for exercises and consultation.	Experiences of interviewees; do not convert a few accounts into proportions of all applicants.

Practical framework or suggestion

3. Decide denominators, missing data and analysis beforehand

Before responses arrive, record whom to count and how to handle non-response. Treating it as no change or silently excluding it alters the results' meaning.

Past training cannot be randomized afterward. Even in an existing program, future additional measures or the next participants may permit assignment design. Comparison groups still require comparability and method-specific assumptions.

Targets: 20 applicants, distinct from 18 participants. Invite all applicants to follow-up.
Behavior definition: self-reported use of agreed watch procedures at least 1 time within 4 weeks. Classify those without opportunities separately.
Aggregation: show response and participation status for all 20 alongside respondent breakdowns. Give counts with differences by experience.
Before/after: compare only ID-matched responses and record unmatched numbers and reasons. Do not treat separate anonymous group means as individual change.
Causal question: to estimate training effects, consult a specialist before implementation on new-participant assignment, comparison conditions and pre-intervention records.

Practical framework or suggestion

4. Check ethics and responsibilities

Distinguish research participation, training attendance and publication. Explain that refusing, skipping uncomfortable questions or stopping carries no disadvantage. Separately confirm use scope for recording and identifiable quotations.

3 months is an agreement in this fictional example, not a common statutory period or recommended standard. Decide according to actual storage policies, contracts and need.

Fictional example: arrangements agreed before implementation
Task	Responsibility and timing	What to check
Information and consent	Frontline staff / recruitment and data collection	Explain purpose, voluntariness, use scope and contacts. Separate service provision from responses.
Matching and storage	Records staff / collection	Keep name-linkage tables separate from responses and restrict staff access. The fictional retention deadline is 3 months after the meeting; record agreed deletion methods.
Aggregation and explanation	Evaluation staff / before the decision meeting	Check denominators, missing data, methods, exceptions and small-group descriptions that could identify people.
Changes and feedback	Operations lead / after the decision meeting	Decide changes, owners, deadlines and checking records, and report back to affected people.

Practical framework or suggestion

5. Apply to your plan

Start with 1–3 important questions and replace each table row with your program's details. Remaining gaps identify information to check next. Review collection plans if questions, users or decision deadlines change.

Record questions, indicators and sources in your own words and retain assumptions needing confirmation.

Practical framework or suggestion

Fictional example; not actual results

Connect collection records to decision use

A fictional example for illustration: community watch training for 20 applicants informs next-session exercises and support.

Question	Information gaps that matter	Next check
Were procedures used after participation?	Respondents' use alone hides non-respondents and people without opportunities	Distinguish denominators such as 20 targets and 15 responses; separate no opportunity from non-use
Why were procedures difficult to use?	Respondents' impressions alone do not reveal missing voices	Seek different voices across use, non-use and no opportunity; record selection and exceptions
Would additional exercises help?	Past participation cannot be randomized retrospectively	Consider next-session comparisons, assignment and main indicators before implementation

What this example supports

Linking records and decision use for each question avoids unnecessary research and specifies explanatory limits beforehand.

Common errors in judgment

Treat participant numbers directly as outcomes

How to revise: Separate participation volume from indicators of understanding or behavioral change

Reason: Participation describes activity provision and has a different role from changes for participants.

Plan an RCT after completion to test past effects

How to revise: Check future assignment possibilities and examine past data through another design fitting its conditions

Reason: Past intervention cannot be converted to random assignment afterward.

Put it into practice next

Record questions, indicators, sources, timing, analysis and claim limitations in an evaluation matrix

View an example and open the worksheet Read the method explanation

Frequently asked questions

Does before-and-after comparison show program effects?

It can establish observed changes, but other factors also contribute, so before-and-after differences alone do not establish causal effects. Design comparison conditions and methodological assumptions separately.

Can non-respondents be excluded from aggregation?

Respondent aggregation is possible, but state the denominator, total targets, non-respondent numbers and bias. Do not present respondent-only proportions as overall proportions.

References and checked sections

Choose methods and processes
BetterEvaluation / Global Evaluation Initiative
Section consulted
2: types of key evaluation questions / 3: situation / 5–6: complementary methods and evaluation matrix / 7–9: feasibility, review, revision
How it relates to the text
Distinguishing descriptive, causal, value-judgment and action questions; organizing complementary methods, feasibility and design review around questions.
Source checked: 2026-10-07 (includes AI cross-checking)
CDC Program Evaluation Framework, 2024
Centers for Disease Control and Prevention (CDC), MMWR
Section consulted
Types of Evaluation / Step 1: evaluability assessment / Step 2: program roadmap / Step 4: Indicators and data sources / Step 6: Act on findings
How it relates to the text
6 stages from context to action, with collaboration, equity, learning and use across stages. Indicators can concern activities or outcomes; outcome checks alone do not establish attribution.
Source checked: 2026-10-07 (includes AI cross-checking)
Quality in qualitative evaluation (QQE)
HM Treasury / Evaluation Task Force
Section consulted
§4.2: guiding principles / Framework questions 4, 7–8, 12: inference, sample coverage, diversity
How it relates to the text
Qualitative evaluation claims require analytical processes, case selection, different voices, and evidence and limits for broader inference.
Source checked: 2026-10-07 (includes AI cross-checking)
Magenta Book: Central Government guidance on evaluation
HM Treasury / Evaluation Task Force
Section consulted
§1.6 pp.23–24: early planning / §2.2: evaluation scoping / §3.2–3.5 pp.71–84: method choice and causal approaches / §6.2 pp.144–145: use and dissemination
How it relates to the text
Integrating evaluation early into program design and selecting methods suited to questions and use. Impact evaluation distinguishes observed change from causal contribution.
Source checked: 2026-10-07 (includes AI cross-checking)
Magenta Book Annex A: Analytical methods for use within an evaluation
HM Treasury / Evaluation Task Force
Section consulted
A2.1 RCT pp.20–21 / A2.7 Difference-in-difference pp.29–30 / A5.4 Output or performance monitoring p.54
How it relates to the text
RCT conditions, pre-intervention trends and comparison conditions for DiD, and implementation and change observable through monitoring.
Source checked: 2026-10-07 (includes AI cross-checking)
UNEG Ethical Guidelines for Evaluation
United Nations Evaluation Group (UNEG); official copy hosted by UNODC
Section consulted
pp.7–10: integrity, accountability, respect, beneficence / pp.20–22: conduct, consent, confidentiality, power / Annex 3 pp.30–35: phase checklists
How it relates to the text
Applying integrity, accountability, respect and beneficence throughout; specifying consent, withdrawal, confidentiality scope and harm reduction.
Source checked: 2026-10-07 (includes AI cross-checking)

Content updated: 2026-10-07Source checked: 2026-10-07 (includes AI cross-checking)No documented human reviewIncludes fictional examplesEditorial policy and scope of checking Cite this page Report a correction

How to cite this page

EvalCompass Editorial Team. “From evaluation questions to a data collection plan”. EvalCompass, updated 2026-10-07. https://evalcompass.jun-nakatani.com/en/guides/evaluation-design-brief/

Add the date you accessed the page. Text created by this site is available under CC BY 4.0. Quotations, external materials, and bundled fonts have their own terms of use. When reusing content, credit the source, link to the license, and indicate any changes.

Suggest a correction or improvement to this page

Change history

2026-10-07 — Added a complete fictional community-training example linked to official primary sources. Specified denominators, missing data, causal scope and responsibility and timing for improvement decisions.