EvalCompassEvalCompass
Practical evaluation guide

What is impact evaluation? Effect investigation, main methods and practical selection

An explanation of impact evaluation, its relationship to effect investigation, and selection of methods including RCTs, difference-in-differences, propensity scores and contribution analysis.

Check whether the intervention contributed to change

Do not treat an observed before-and-after change as the intervention’s effect without further evidence.

Choose methods according to the question, data, and feasibility of comparison, and state the limits of what can be claimed. Read the full explanation

Main takeaway

Impact evaluation examines how an intervention contributed to change. Experimental and quasi-experimental methods estimate what would happen without intervention to measure effects; theory-based evaluation examines causal pathways and alternative explanations through evidence. Do not treat simple before-and-after change directly as the intervention's effect. [1]

The term's scope differs between frameworks. This guide focuses on methods for examining intervention-related change. Measuring long-term change alone does not establish a causal relationship with the intervention.

First decide which question to answer, whether an effect-size estimate is needed and whether an explanation of how change occurred is needed.

Distinguish change from causal effects

Choose methods and the scope of claims according to what you need to establish and whether comparison is feasible.

Distinguish observed change from causal effects

Positive target-group change alone does not establish a program effect. The economy, other schemes, personal effort, time or other factors may have produced it.

In its narrow sense, impact evaluation estimates the counterfactual: what would have happened without the program. Comparison groups and time-series data are used for this purpose.

[2]

Main methods

RCTs use random assignment to reduce between-group differences, but attrition, contamination and measurement discrepancies affect estimates. For quasi-experiments too, check whether each method's assumptions can be met.

Even without a comparison group, contribution analysis or process tracing can explain program contribution from multiple sources of evidence.

MethodCharacteristicsSuitable conditions
RCTComparison through random assignmentAssignment, follow-up and implementation can be managed
Difference-in-differencesCompare differences in before-and-after changes between two groupsThe parallel-trends assumption without intervention can be examined
Propensity scoresAlign comparators using observed pre-intervention characteristicsThere is common support and limitations from unobserved confounding can be examined
Regression discontinuityCompare near a thresholdConditions near the threshold and assignment rules can be checked
Contribution analysisExamine pathways and alternative explanations through multiple sourcesRecords on the course of change and other factors can be collected
[1]

Practical method selection

Select according to purpose, data, ethics, budget and decision importance. Advanced causal inference is not necessary for every program. Consider rigorous designs for major policy decisions or large investments; practical evaluations may be preferable for small improvements.

Qualitative research and process evaluation are also needed to understand why, for whom and under which conditions an intervention worked, beyond whether it had an effect.

Practical framework or suggestion

State the limits of claims

Comparison groups or before-and-after data alone do not identify causal effects. Designs require specific assumptions, such as parallel trends without intervention for difference-in-differences and no unobserved confounding for propensity scores. State the estimand, assumptions and uncertainty.

With no comparison group, small samples or no pre-intervention data, distinguish observed change, consistency with the program, possible other factors and future testing needs rather than asserting effects. Avoiding overstatement makes results more usable in policy decisions.

  • Check for comparison groups and pre-intervention data
  • Distinguish causal effects, contribution and outcome measurement
  • Report alternative explanations and limitations
  • Use qualitative research to supplement mechanisms
[1]
Fictional example; not actual results

Distinguish before-and-after differences from differences in change between two groups

A fictional example for illustration: compare identically defined indicators in intervention and comparison districts.

DistrictBefore interventionAfter interventionChange
Intervention district40%60%20 percentage points
Comparison district45%55%10 percentage points
Difference in change——20−10 = 10 percentage points

What this example supports

The 10 percentage points are a calculated difference in change. They can be interpreted causally only after examining assumptions such as parallel trends without intervention. This table alone cannot confirm those assumptions.

Common errors in judgment

Treat the intervention district's entire 20-point increase as an effect

How to revise: Examine other factors and change in the comparison district

Reason: The increase may include change that would have occurred without intervention.

Assume a similar comparison group eliminates confounding

How to revise: Check unobserved differences and method-specific assumptions

Reason: Unobserved participation motives and other factors may affect results.

Put it into practice next

Use the difference-in-differences worksheet to organize pre-intervention trends and alternative explanations

View an example and open the worksheetRead the method explanation

Evaluation methods to explore next

Causal inference & experimentsRandomized Controlled Trial
Randomly assigns targets to intervention/comparison groups and compares outco…
Causal inference & experimentsDifference-in-Differences
Estimates effects by subtracting before/after change in the comparison group …
Causal inference & experimentsPropensity Score Matching
Estimates participation probabilities from observed pre-intervention characte…
Causal inference & experimentsRegression Discontinuity Design
Uses schemes where assignment changes at a continuous indicator's threshold t…
Causal inference & experimentsContribution Analysis
Examines how programs contribute to outcomes through theory of change, multip…
Causal inference & experimentsProcess Tracing
Tests evidence for causal pathways within one case by hypothesis. Considers n…

Frequently asked questions

Are impact evaluation and outcome evaluation the same?

Usage overlaps, but this guide treats outcome evaluation as identifying observed changes and impact evaluation as examining an intervention's causal involvement. Measuring long-term change alone does not establish causal effects. State the meaning within the adopted framework.

Is impact evaluation impossible without a comparison group?

Some theory-based evaluations do not create comparison groups. Even counterfactual estimation varies in design, such as comparisons across time or thresholds. State method-specific assumptions and how far effect size can be estimated.

Do small programs need impact evaluation?

Not necessarily. Select a practical design according to decision importance, cost and data availability.

References and checked sections

  1. Magenta Book Annex A analytical methods for use within an evaluation
    HM Treasury / Evaluation Task Force · 2026
    Section consulted
    A1. Theory-based methods; A2.1 RCT; A2.2 PSM; A2.7 Difference-in-difference; A2.8 Regression discontinuity
    How it relates to the text
    Assumptions of causal estimation methods and the role of theory-based evaluation
    Source checked: (includes AI cross-checking)
  2. Is My NGO Having a Positive Impact?
    David Evans / Bruce Wydick（World Bank Development Impact） · 2016
    Section consulted
    Before and After / Beneficiaries and non-beneficiaries
    How it relates to the text
    Differences between NGO outcome measurement and causal effects, and limitations of before-and-after and participant/non-participant comparisons
    Source checked: (includes AI cross-checking)
Content updated: Source checked: (includes AI cross-checking)No documented human reviewIncludes fictional examplesEditorial policy and scope of checkingCite this pageReport a correction

How to cite this page

EvalCompass Editorial Team. “What is impact evaluation? Effect investigation, main methods and practical selection”. EvalCompass, updated 2026-10-07. https://evalcompass.jun-nakatani.com/en/guides/impact-evaluation/
Add the date you accessed the page. Text created by this site is available under CC BY 4.0. Quotations, external materials, and bundled fonts have their own terms of use. When reusing content, credit the source, link to the license, and indicate any changes.

Suggest a correction or improvement to this page

Change history
  • — A cross-guide review clarified indicator formats, outcome classifications and causal judgments, interpretation of tables, and the scope of SVI2026 where relevant.
  • — Added correspondence between claims and original sources, fictional completed examples, judgment corrections and links to next tasks. Revised unverified generalizations.