EvalCompassEvalCompass
Causal inference & experimentsCausal inferenceQuasi-experimentPolicy evaluationExogeneityCounterfactual

Natural Experiment

Natural Experiment
Also known as: Natural experiment design、Quasi-natural experiment、Exogenous shock from policy change

A research framework evaluating effects through intervention differences produced by policy changes or institutional events whose assignment researchers do not control. Assignment processes and valid comparisons are central. [1]

Natural experiment design checklist — openReferences and checked sections
Difficulty
Advanced
Cost
Medium
Duration
Planning example, not a universal standard duration: 2–6 weeks for design and data checks, and 2–8 weeks for analysis
Content updated: 2026-09-13Source checked: 2026-09-13 (includes AI cross-checking)No documented human reviewIncludes fictional examples

How it works at a glance

Natural experiments use intervention or exposure differences outside researchers' control as research opportunities. Understand policy timing, eligibility rules and area differences, and construct comparisons and data appropriate to the question. [1]

Naturally occurring differences are not automatically equivalent to random assignment. Investigate how rules were determined, confounding, simultaneous changes and spillovers, and examine assumptions required by methods such as difference-in-differences or regression discontinuity. [1]

  1. Identify the intervention and assignment process
  2. Check the basis for comparability
  3. Select an appropriate analytical method
  4. Examine counterfactual validity
  5. Check spillovers
  6. Report design limitations
Sequence of implementation steps. See “Implementation steps” for details of each stage.

When to use it and when it is unsuitable

When to use it

  • When a policy or institutional change applies only to certain areas or groups
  • When implementation timing or eligibility criteria are clear and a comparison group can be defined
  • When examining causal effects of interventions researchers cannot assign, under appropriate comparisons and assumptions
  • When testing effects of past policies or programs using existing data

When it is unsuitable

  • When intervention selection depends strongly on expected outcomes and comparability is poor
  • When multiple concurrent changes prevent isolating the intervention effect

Participants, roles, and deliverables

Participants and roles

Evaluators, policy staff and data analysts. Collaboration between institutional experts and quantitative analysts is important.

Main deliverables

  • Natural experiment design memo
  • Identification assumptions and verification findings
  • Causal effect estimation report

Implementation steps

  1. Identify the intervention and assignment process
    Check documents describing rules generating intervention assignment, such as institutional changes, thresholds, geographical boundaries or differences in timing.
  2. Check the basis for comparability
    Use data and institutional context to assess pre-intervention similarities and differences between intervention and comparison groups.
  3. Select an appropriate analytical method
    Consider DiD with before-and-after data and a comparison group, RDD with a threshold, or ITS or synthetic control with a long time series.
  4. Examine counterfactual validity
    Specify identification assumptions and examine supporting evidence and threats using institutional context, pre-intervention data, placebo analyses and other checks. Tests alone do not prove counterfactual assumptions.
  5. Check spillovers
    Check whether the comparison group is indirectly affected through movement between areas or information spillovers.
  6. Report design limitations
    Clearly explain why the setting qualifies as a natural experiment, which assumptions are weak and the scope of generalization.

Strengths and limitations

Strengths

  • Creates opportunities to evaluate policies researchers cannot randomize, using designs suited to assignment processes and data.
  • Often enables use of existing administrative or open data.
  • Evaluates conditions close to actual policy implementation, sometimes supporting high external validity.
  • Several methodological options enable flexible analysis appropriate to the design.

Limitations

  • Claims of exogeneity are always contestable, making institutional understanding essential.
  • Vulnerable to concurrent policies and external changes.
  • Conclusions are constrained by data timing, granularity and quality.

Common pitfalls

  • Calling something a natural experiment merely because it occurred naturally, without demonstrating comparability
  • Missing selection bias by failing to check intervention selection reasons
  • Reporting only method names without identification assumptions or verification findings

Conditions of use and evidence

  • Researcher non-assignment alone does not guarantee valid causal inference. Examine the assignment process. [1]
Fictional example; not actual results

Consider using a program's eligibility score

A fictional illustrative system providing support to people scoring 60 or above. This is not an actual system or evaluation result.

Item to checkFictional settingEvaluation implication
AssignmentEligibility of scores 60 or above defined before implementationRDD can be considered.
OperationSome discretionary exceptions occur.Investigate the jump in treatment probability.
Other systemsOther support may also start at the same score of 60.Risk of being unable to separate effects

Common errors in judgment

Common mistake

Assuming institutional changes are free of confounding

How to revise

Investigate policy decision reasons and population selection.

ReasonNeed and political judgments can affect both introduction and outcomes.

Common mistake

Calling a setting a natural experiment to fit available data

How to revise

Explain the design through the intervention and assignment process.

ReasonSpecific comparison evidence matters more than the label.

Organize your thinking by writing

Natural experiment design checklist

A practical template is available to fill in and print on A4 paper.

Open worksheet

Related methods

Difference-in-DifferencesRegression Discontinuity DesignInterrupted Time Series AnalysisSynthetic Control MethodRegression Analysis

References and checked sections

  1. Using natural experiments to evaluate population health and health system interventions: new framework for producers and users of evidence
    Peter Craig et al. · 2025
    Section consulted
    Framework for natural experimental evaluations
    How it relates to the text
    Researcher non-assignment alone does not guarantee valid causal inference. Examine the assignment process.
    Source checked: (includes AI cross-checking)

Editorial and checking information

Content updated: Source checked: (includes AI cross-checking)No documented human reviewIncludes fictional examplesEditorial policy and scope of checkingCite this pageReport a correction

How to cite this page

EvalCompass Editorial Team. “Natural Experiment”. EvalCompass, updated 2026-09-13. https://evalcompass.jun-nakatani.com/en/methods/natural-experiment/
Add the date you accessed the page. Text created by this site is available under CC BY 4.0. Quotations, external materials, and bundled fonts have their own terms of use. When reusing content, credit the source, link to the license, and indicate any changes.

Suggest a correction or improvement to this page

Change history
  • — Checked sources for definitions and applicability conditions, and added fictional examples, reasons for judgments and examples for every worksheet field. Revised unverified claims about history and adoption. Also reviewed existing procedures and cautions, and adjusted fixed participant counts and thresholds and generalizations about institutions. Preserved field positions, types and counts while revising fixed conditions and thresholds in the forms.