Natural experiment design checklist

Explain comparability between intervention and comparison groups through institutional context. Checking identification assumptions matters more than the method's name.

When to use this worksheet and what completion involves Fictional completed example

EvalCompass Worksheet

Natural experiment design checklist

Method: Natural Experiment

Explain comparability between intervention and comparison groups through institutional context. Checking identification assumptions matters more than the method's name.

Basic information

Record the policy or event treated as a natural experiment.

Completed example for this section (fictional)

This fictional example is for explanation. Adapt it to your own program. Your entries will not change.

Policy or institutional change: Score-based change in support eligibility (fictional)
Intervention timing: April 2026 (fictional)
Target areas and population: Applicants in the target area
Analyst: Analyst A

Reason for this entry: Record what changed and when.

Policy or institutional change

Intervention timing

Target areas and population

Analyst

Writing tips and explanation for this section

About this section

Record the policy or event treated as a natural experiment.

Record what changed and when.

View the method explanation

Writing tips

Follow each field’s heading to record the program name, focus, period, and other details. Leave undecided matters clearly marked as unresolved.

Read the full fictional example

Basis for exogeneity

Describe the rules producing intervention assignment.

Completed example for this section (fictional)

This fictional example is for explanation. Adapt it to your own program. Your entries will not change.

Scroll horizontally to view the table

Assignment rule or event	Intervention group	Comparison group	Basis for considering assignment exogenous	Concerns
Rule specifying pre-intervention scores of 60 or above	Applicants just above the threshold	Applicants just below the threshold	Other conditions may be continuous near the threshold.	Score manipulation, discretion and other support starting

Reason for this entry: Record exogeneity as a question to investigate, not a conclusion.

Scroll horizontally to view the table

Assignment rule or event	Intervention group	Comparison group	Basis for considering assignment exogenous	Concerns

Writing tips and explanation for this section

About this section

Describe the rules producing intervention assignment.

Record exogeneity as a question to investigate, not a conclusion.

View the method explanation

Writing tips

Keep the information in each row aligned. Scroll wide tables horizontally to see every column.

Read the full fictional example

Select analytical methods

Choose methods suited to the design.

Completed example for this section (fictional)

This fictional example is for explanation. Adapt it to your own program. Your entries will not change.

Scroll horizontally to view the table

Candidate method	Required data	Assumptions to examine	Adopt or reject
RDD	Scores, receipt, outcomes and baseline characteristics	Continuity of potential outcomes, manipulation at the threshold and other systems	Pending checks of operation and other systems

Reason for this entry: Do not decide on the analysis in advance.

Scroll horizontally to view the table

Candidate method	Required data	Assumptions to examine	Adopt or reject

Writing tips and explanation for this section

About this section

Choose methods suited to the design.

Do not decide on the analysis in advance.

View the method explanation

Writing tips

Keep the information in each row aligned. Scroll wide tables horizontally to see every column.

Read the full fictional example

Threats and robustness

Describe responses to concurrent policies, spillovers and selection bias.

Completed example for this section (fictional)

This fictional example is for explanation. Adapt it to your own program. Your entries will not change.

Examine exceptions in operational records as well as formal rules. Check score distributions, baseline characteristics and bandwidth changes. If another system starts at the same threshold, do not interpret findings as the effect of support alone.

Reason for this entry: Record multiple threats and responses.

Writing tips and explanation for this section

About this section

Describe responses to concurrent policies, spillovers and selection bias.

Record multiple threats and responses.

View the method explanation

Writing tips

Use the example as a reference and record what you have established about your own program. Distinguish assumptions and unchecked matters from confirmed facts.

Read the full fictional example

Completion check

Completed example for this section (fictional)

This fictional example is for explanation. Adapt it to your own program. Your entries will not change.

☐ Not checked: Checked intervention assignment rules in documents
☐ Not checked: Examined comparison group validity and threats using pre-intervention data and institutional context
☐ Not checked: Checked concurrent policies and external factors
☑ Checked: Specified analytical assumptions, methods for assessing their basis and limitations

Reason for this entry: This is a plan in which actual rules, pre-intervention data and concurrent policy documents have not yet been obtained.

Checked intervention assignment rules in documents
Examined comparison group validity and threats using pre-intervention data and institutional context
Checked concurrent policies and external factors
Specified analytical assumptions, methods for assessing their basis and limitations

Writing tips and explanation for this section

About this section

This is a plan in which actual rules, pre-intervention data and concurrent policy documents have not yet been obtained.

View the method explanation

Writing tips

Read each item and check those you have confirmed. Do not use the fictional example’s checkbox states as findings about your own program.

Read the full fictional example

Before writing, check when to use the worksheet and what completion involves

Natural experiment design checklist is a worksheet for applying Natural Experiment in meetings or individual work. For the overall duration of the method, use “Planning example, not a universal standard duration: 2–6 weeks for design and data checks, and 2–8 weeks for analysis” as a reference, adjusting for scale and resources. Fill in what you know provisionally and leave unresolved matters as questions to check next.

When this worksheet is useful

When a policy or institutional change applies only to certain areas or groups
When implementation timing or eligibility criteria are clear and a comparison group can be defined
When examining causal effects of interventions researchers cannot assign, under appropriate comparisons and assumptions

Participants and preparation

Evaluators, policy staff and data analysts. Collaboration between institutional experts and quantitative analysts is important.

Bring the materials used for decisions, existing data, and differing stakeholder views to help complete the worksheet.

Suggested order

1. Basic informationRecord the policy or event treated as a natural experiment.
2. Basis for exogeneityDescribe the rules producing intervention assignment.
3. Select analytical methodsChoose methods suited to the design.
4. Threats and robustnessDescribe responses to concurrent policies, spillovers and selection bias.
5. Completion check

Signs the worksheet is ready for review

Natural experiment design memo
Identification assumptions and verification findings
Causal effect estimation report

Cautions for review

Calling something a natural experiment merely because it occurred naturally, without demonstrating comparability
Missing selection bias by failing to check intervention selection reasons
Reporting only method names without identification assumptions or verification findings

Read a completed fictional example (your entries will not change)

Fictional example; not actual results

A fictional illustrative system providing support to people scoring 60 or above. This is not an actual system or evaluation result.

Basic information

Policy or institutional change: Score-based change in support eligibility (fictional)
Intervention timing: April 2026 (fictional)
Target areas and population: Applicants in the target area
Analyst: Analyst A

Reason for this entry: Record what changed and when.

Basis for exogeneity

Scroll horizontally to view the table

Assignment rule or event	Intervention group	Comparison group	Basis for considering assignment exogenous	Concerns
Rule specifying pre-intervention scores of 60 or above	Applicants just above the threshold	Applicants just below the threshold	Other conditions may be continuous near the threshold.	Score manipulation, discretion and other support starting

Reason for this entry: Record exogeneity as a question to investigate, not a conclusion.

Select analytical methods

Scroll horizontally to view the table

Candidate method	Required data	Assumptions to examine	Adopt or reject
RDD	Scores, receipt, outcomes and baseline characteristics	Continuity of potential outcomes, manipulation at the threshold and other systems	Pending checks of operation and other systems

Reason for this entry: Do not decide on the analysis in advance.

Threats and robustness

Reason for this entry: Record multiple threats and responses.

Completion check

☐ Not checked: Checked intervention assignment rules in documents
☐ Not checked: Examined comparison group validity and threats using pre-intervention data and institutional context
☐ Not checked: Checked concurrent policies and external factors
☑ Checked: Specified analytical assumptions, methods for assessing their basis and limitations

Reason for this entry: This is a plan in which actual rules, pre-intervention data and concurrent policy documents have not yet been obtained.

Common errors in judgment

Assuming institutional changes are free of confounding

How to revise: Investigate policy decision reasons and population selection.

Reason: Need and political judgments can affect both introduction and outcomes.

Calling a setting a natural experiment to fit available data

How to revise: Explain the design through the intervention and assignment process.

Reason: Specific comparison evidence matters more than the label.

Editorial and checking information

Content updated: 2026-09-13Source checked: 2026-09-13 (includes AI cross-checking)No documented human reviewIncludes fictional examplesEditorial policy and scope of checking Cite this page Report a correction

References and checked sections

Using natural experiments to evaluate population health and health system interventions: new framework for producers and users of evidence
Peter Craig et al. · 2025
Section consulted
Framework for natural experimental evaluations
How it relates to the text
Researcher non-assignment alone does not guarantee valid causal inference. Examine the assignment process.
Source checked: 2026-09-13 (includes AI cross-checking)

How to cite this page

EvalCompass Editorial Team. “Natural experiment design checklist”. EvalCompass, updated 2026-09-13. https://evalcompass.jun-nakatani.com/en/worksheets/natural-experiment/

Add the date you accessed the page. Text created by this site is available under CC BY 4.0. Quotations, external materials, and bundled fonts have their own terms of use. When reusing content, credit the source, link to the license, and indicate any changes.

Suggest a correction or improvement to this page

Change history

2026-09-13 — Checked sources for definitions and applicability conditions, and added fictional examples, reasons for judgments and examples for every worksheet field. Revised unverified claims about history and adoption. Also reviewed existing procedures and cautions, and adjusted fixed participant counts and thresholds and generalizations about institutions. Preserved field positions, types and counts while revising fixed conditions and thresholds in the forms.