EvalCompassEvalCompass
Practical evaluation guide

Evaluation methods for EBPM: choosing policy evaluation, effect investigation and logic models

Organize methods used in EBPM by purpose, including logic models, indicator design, performance measurement, RCTs, difference-in-differences and contribution analysis.

Choose evidence for policy decisions

Policy question

What do you want to establish?

Inform policy design and improvement

Choose methods that fit the decision and the available data

Distinguish what was delivered, what changes were observed, and what effects were caused by the intervention. Read the full explanation

Choose methods that fit your evaluation purpose

Compare methods according to your purpose and available data.

Compare methods for your purpose

PurposeMethodWhat to examine
Organizing designLogic ModelTheory of ChangeMake policy hypotheses visible
Progress managementPerformance MeasurementIndicator DesignContinuously check implementation
Outcome measurementSurvey / QuestionnaireSecondary Data AnalysisExamine changes for target groups
Causal investigationRandomized Controlled TrialDifference-in-DifferencesRegression Discontinuity DesignInvestigate effects attributable to policy
Explanation and improvementContribution AnalysisCase StudyExplain why change occurred

Logic Model

A diagram that links a program’s resources, activities, direct outputs and changes for participants, setting out the assumptions about how outcomes will occur and what needs to be checked. Completing the diagram does not, by itself, demonstrate a causal effect.

Resources and activitiesChanges experienced by participants

When to use it

Clarifying the links between activities and outcomes with stakeholders when planning a new program

What to check before use
  • A simple one-way diagram may miss branching or cyclical change. Add the necessary relationships while maintaining readability.
  • Can become a document that is completed and then forgotten; without updates, it loses practical value
Organize it in a worksheetRead the explanation and sources

Theory of Change

A method for making explicit the desired change and the pathways and assumptions leading to it. It sets out, as testable hypotheses, why implementing activities is expected to produce outcomes.

When to use it

Designing or evaluating complex social change programs involving multiple agencies and communities

What to check before use
  • Unless the detail of the diagram and its supporting rationale fits the purpose, preparation becomes burdensome. Small programs can begin with their main pathways and assumptions.
  • Too many assumptions can produce a complex diagram that is impractical to use
Organize it in a worksheetRead the explanation and sources

Performance Measurement

Continuously measures defined indicators against baselines and targets to inform operational decisions. Tracks changes in outcomes as well as counts, with consistent numerical definitions and collection responsibilities.

When to use it

When establishing arrangements to regularly manage and report program or policy progress

What to check before use
  • Easily measured indicators may take priority, marginalizing important but difficult-to-measure outcomes such as quality and equity
  • Indicator attainment can become an end in itself, potentially encouraging gaming
Organize it in a worksheetRead the explanation and sources

Indicator Design

Specify the desired change through definitions, sources, collection methods and timing. Check not only indicator values but also what they represent and what they do not.

When to use it

Assigning measurement methods to each outcome element in a logic model or ToC

What to check before use
  • Indicators may favor what is measurable, marginalizing important but difficult changes such as wellbeing and empowerment
  • Target setting can become overly political, with manipulation through easy, low targets
Organize it in a worksheetRead the explanation and sources

Survey / Questionnaire

A method of collecting experiences, behavior and attitudes using common questions. Alongside clarity, record who received the questionnaire and who responded to judge the scope represented by results.

When to use it

Collect comparable quantitative responses using common questions

What to check before use
  • Cannot capture complex contexts or histories not anticipated in questions
  • High risk of distortion from low response and selection bias
Organize it in a worksheetRead the explanation and sources

Secondary Data Analysis

Uses statistics or operational records collected for another purpose in evaluation. Check fit of collection purpose, targets, definitions and timing to evaluation questions.

When to use it

Understand target-area/group backgrounds and trends without new collection

What to check before use
  • Not designed for evaluation, so necessary variables may be absent
  • Time, target and definition differences complicate comparisons
Organize it in a worksheetRead the explanation and sources

Randomized Controlled Trial

Randomly assigns targets to intervention/comparison groups and compares outcomes to estimate effects. Check attrition, non-compliance and between-group influences after assignment.

When to use it

Plan investigation of new social interventions or policies before implementation

What to check before use
  • High internal validity does not establish applicability elsewhere
  • It is often difficult to secure enough participants, conduct long-term follow-up, or withhold the intervention from the control group.
Organize it in a worksheetRead the explanation and sources

Difference-in-Differences

Estimates effects by subtracting before/after change in the comparison group from that in the intervention group. Assumes parallel changes without intervention.

When to use it

Phased or geographically limited policies with before/after data for both groups

What to check before use
  • Counterfactual parallel trends cannot be directly proven; anticipation, spillovers and composition also matter
  • Cannot address time-varying unobserved confounding affecting only intervention groups
Organize it in a worksheetRead the explanation and sources

Regression Discontinuity Design

Uses schemes where assignment changes at a continuous indicator's threshold to compare nearby outcomes. Estimates a local threshold difference, not automatically generalizable to distant targets.

When to use it

Eligibility determined by available numerical score, income or age cutoffs

What to check before use
  • Sharp/fuzzy estimands differ; both usually concern near-cutoff effects, not all-target averages.
  • Deliberate running-variable manipulation undermines identification
Organize it in a worksheetRead the explanation and sources

Contribution Analysis

Examines how programs contribute to outcomes through theory of change, multiple sources and alternative explanations. Explains pathways and remaining uncertainty.

When to use it

Answer causal contribution questions where RCTs/econometric methods are unsuitable

What to check before use
  • Credibility depends heavily on theory quality and thorough evidence; weak stories lack persuasiveness
  • Contribution judgments lack clear objective criteria and rely substantially on evaluators
Organize it in a worksheetRead the explanation and sources

Case Study

A research design examining particular people, organizations or programs in depth and context. Combines sources to examine events and the validity of explanations.

When to use it

Explain processes and contexts behind outcomes or their absence

What to check before use
  • Biased selection weakens analytical generalization
  • Multiple methods and large datasets require time and skill
Organize it in a worksheetRead the explanation and sources
Main takeaway

EBPM clarifies policy objectives and uses rational evidence to plan and improve policy. In evaluation, distinguish implementation monitoring, confirmation of change and investigation of causality, and select methods fitting the evidence needed for decisions and the data obtainable. [1]

EBPM aims to use evidence to design and improve policy. Providing appropriate evidence for questions necessary to policy decisions matters, rather than merely collecting data.

Select evaluation methods according to purpose. Methods differ depending on whether the aim is organizing program design, managing progress, measuring outcomes or investigating causal effects.

Connect policy questions with evidence

Start with policy logic to examine method selection and the limits of judgments.

Start EBPM evaluation design with policy logic

First organize objectives, target groups, intervention content and expected changes. Logic models and theories of change can clarify the pathways by which a policy is expected to produce outcomes.

Starting analysis with unclear logic can allow available data to pull the work away from questions necessary for decisions.

  • Clarify policy objectives and target groups
  • Distinguish activities, outputs and outcomes
  • Check external factors and assumptions
  • Set evaluation questions
  • Link indicators to sources
[1]

Evaluation methods by purpose

EBPM methods extend beyond causal inference. Performance measurement and monitoring suit implementation checks; surveys and interviews suit understanding target groups; and experiments and quasi-experiments suit causal-effect investigation.

With limited budgets and data environments, balance rigor and feasibility.

PurposeExample methodsWhere to use them
Organizing designLogic models and ToCMake policy hypotheses visible
Progress managementPerformance measurement and KPIsContinuously check implementation
Outcome measurementSurveys and administrative data analysisExamine changes for target groups
Causal investigationRCT、DiD、RDDInvestigate effects attributable to policy
Explanation and improvementContribution analysis and case studiesExplain why change occurred
[2]

When rigorous causal inference is difficult

Randomization or comparison groups can be difficult to secure in policy practice. Even then, combining logic models, time-series data, qualitative information, stakeholder interviews and comparisons with similar cases can produce evaluation useful for improvement.

It is important to state honestly how far claims can go. Even where an effect cannot be asserted, wording such as evidence consistent with change or possible other factors remaining conveys the strength of judgment.

[2]

Combining evidence for policy decisions

EBPM requires sufficient evidence for policy decisions rather than the most advanced analysis. Separately checking implementation, target-group reach, outcome changes, costs and practical implementation issues makes findings easier to use for improvement and budget decisions.

For new policies, check reach and implementation barriers early. At design, also secure pre-intervention data and comparison conditions needed for later effect investigation. Waiting for a scheme to stabilize before designing evaluation can lose opportunities for testing.

  • Organize policy logic and indicators first
  • Distinguish implementation data from outcome data
  • Check data quality before investigating effects
  • Clearly report what can and cannot be claimed
Practical framework or suggestion
Fictional example; not actual results

Distinguish decisions for improving municipal consultation information

A fictional example for illustration: information about a new consultation service will be delivered to districts in sequence.

What to learnSources and designScope of judgment
Did the information arrive?Delivery records and confirmation of receiptImplementation and reach
Did consultations increase?Counts before and after information delivery and seasonal variationObserved change
Was the increase due to the information?Design introduction timing and comparison-district conditions beforehandCausal estimates if assumptions hold

What this example supports

If effect investigation is needed, decide delivery conditions and recording methods beforehand rather than merely finding similar districts after distribution. Also judge whether increased consultations are desirable in light of users' circumstances.

Common errors in judgment

Assume EBPM is complete because a logic model was drawn

How to revise: Turn uncertain pathways into questions and connect data to policy decisions

Reason: The diagram is a starting point for testing and improvement.

Wait for implementation records to accumulate before considering comparisons

How to revise: Check necessary comparison conditions and data before starting

Reason: Pre-intervention information may be impossible to recover later.

Put it into practice next

Organize questions needed for policy decisions in an evaluation matrix

View an example and open the worksheetRead the method explanation

Evaluation methods to explore next

Design & theoryLogic Model
A diagram that links a program’s resources, activities, direct outputs and ch…
Design & theoryIndicator Design
Specify the desired change through definitions, sources, collection methods a…
Reporting, use & meta-evaluationPerformance Measurement
Continuously measures defined indicators against baselines and targets to inf…
Causal inference & experimentsRandomized Controlled Trial
Randomly assigns targets to intervention/comparison groups and compares outco…
Causal inference & experimentsDifference-in-Differences
Estimates effects by subtracting before/after change in the comparison group …
Causal inference & experimentsContribution Analysis
Examines how programs contribute to outcomes through theory of change, multip…

Frequently asked questions

Are RCTs mandatory for EBPM?

No. RCTs are powerful, but select appropriate methods according to policy objectives, ethics, practical constraints and data environments.

Can administrative data alone support evaluation?

Sometimes. However, check that the data correspond to evaluation questions and have no missing-data or definition problems.

How does EBPM differ from ordinary evaluation?

EBPM emphasizes using evidence in policy decisions. Evaluation provides design, measurement, analysis and learning to support that use.

References and checked sections

  1. 内閣府におけるEBPMへの取組
    内閣府
    Section consulted
    Opening definition of EBPM (last updated: August, Reiwa 8)
    How it relates to the text
    Clarifying policy objectives and planning policy on rational evidence
    Source checked: (includes AI cross-checking)
  2. Magenta Book Annex A analytical methods for use within an evaluation
    HM Treasury / Evaluation Task Force · 2026
    Section consulted
    A1. Theory-based methods / A2. Experimental and quasi-experimental methods
    How it relates to the text
    Selecting theory-based, experimental and quasi-experimental methods according to evaluation purpose
    Source checked: (includes AI cross-checking)
Content updated: Source checked: (includes AI cross-checking)No documented human reviewIncludes fictional examplesEditorial policy and scope of checkingCite this pageReport a correction

How to cite this page

EvalCompass Editorial Team. “Evaluation methods for EBPM: choosing policy evaluation, effect investigation and logic models”. EvalCompass, updated 2026-09-13. https://evalcompass.jun-nakatani.com/en/guides/ebpm-evaluation-methods/
Add the date you accessed the page. Text created by this site is available under CC BY 4.0. Quotations, external materials, and bundled fonts have their own terms of use. When reusing content, credit the source, link to the license, and indicate any changes.

Suggest a correction or improvement to this page

Change history
  • — Added links to original sources, scope of definitions, and fictional completed and judgment examples. Revised unsupported advice prescribing fixed counts or timing.