EvalCompassEvalCompass
Analysis & synthesisEvidence synthesisPRISMAEffect sizeCampbell CollaborationEvidence level

Meta-Analysis and Systematic Review

Meta-Analysis and Systematic Review
Also known as: Meta-analysis、Systematic review、PRISMA

Systematic review searches, selects and appraises research against explicit criteria. Meta-analysis statistically combines effect estimates from studies suitable for synthesis. [1]

Systematic review planning and management worksheet — openReferences and checked sections
Difficulty
Advanced
Cost
Medium
Duration
Planning example, not a universal standard duration: 3–12 months for searching, screening, extraction and statistical synthesis
Content updated: 2026-10-07Source checked: 2026-09-13 (includes AI cross-checking)No documented human reviewIncludes fictional examples

How it works at a glance

Design questions, eligibility criteria, searches, extraction and bias assessment first. Even for the same question, differences in interventions, populations or designs may make numerical pooling inappropriate. [1]

When pooling, consider effect measures, weights, models and heterogeneity. Random-effects models do not eliminate heterogeneity or bias; choosing not to pool is also an option. [1]

  1. Define research questions and PICO/PICO-S
  2. Preregister the protocol
  3. Search literature comprehensively
  4. Screen and extract data
  5. Assess risk of bias
  6. Pool effects or synthesize narratively
  7. Report with PRISMA and interpret
Sequence of implementation steps. See “Implementation steps” for details of each stage.

When to use it and when it is unsuitable

When to use it

  • When comprehensively synthesizing evidence on interventions such as employment training or conditional cash transfers
  • When conflicting evaluation studies need statistical organization to clarify overall conclusions
  • When systematically organizing studies and, if pooling is appropriate, showing average effects and uncertainty
  • When using known effect sizes and moderators from previous research to support a new evaluation design

When it is unsuitable

  • New interventions with absent or very limited primary evaluation or research data. A review cannot proceed without studies.
  • When intervention, measurement or contextual heterogeneity is too great. Meta-analysis becomes inappropriate and narrative review remains.
  • When required searching, appraisal and statistical work cannot meet the decision deadline

Participants, roles, and deliverables

Participants and roles

At least 2 independent reviewers for screening and extraction, and a statistician for meta-analysis. Collaboration with external statistical and information specialists for search strategies is recommended.

Main deliverables

  • PRISMA flow diagram from search to inclusion
  • Included-study characteristics
  • Forest plot of pooled and individual effects
  • Systematic review report

Implementation steps

  1. Define research questions and PICO/PICO-S
    Specify Population, Intervention, Comparison and Outcome in PICO format as the basis for all searching and screening decisions.
  2. Preregister the protocol
    Preregistering in PROSPERO, the international systematic review register, or similar systems prevents arbitrary retrospective changes and improves transparency.
  3. Search literature comprehensively
    Search systematically across PubMed, PsycINFO, ERIC, Web of Science, Google Scholar and other databases. Include grey literature, dissertations, institutional reports and proceedings. Record strategies and dates.
  4. Screen and extract data
    2 independent reviewers screen titles and abstracts, then full texts, resolving disagreements through discussion. Extract effect sizes, samples, interventions and measures from included papers.
  5. Assess risk of bias
    Assess internal validity using standardized tools such as Cochrane RoB for RCTs or Newcastle–Ottawa Scale for observational studies. Test whether excluding high-risk studies changes conclusions in sensitivity analysis.
  6. Pool effects or synthesize narratively
    Assess population, intervention, outcome and design comparability before pooling. Choose fixed- or random-effects models according to differences and estimands, not automatic I² thresholds. Random-effects models do not resolve heterogeneity or bias.
  7. Report with PRISMA and interpret
    Follow the PRISMA 2020 checklist and flow diagram tracking search, screening and inclusion counts. Describe substantive significance, limitations, evidence gaps and policy implications.

Strengths and limitations

Strengths

  • Effectively increases sample size beyond individual studies and improves effect estimate precision.
  • Examining heterogeneity informs which contexts may produce stronger effects.
  • International standards such as PRISMA support reproducibility, transparency and persuasive policy evidence.
  • Makes evidence gaps visible and identifies priorities for future evaluation or research.

Limitations

  • Publication bias: significant results are more likely to be published, overestimating effects. Decide whether funnel plots or Egger tests apply based on study counts, standard-error variation and effect scales. Asymmetry may indicate small-study effects and does not uniquely diagnose publication bias. Check unpublished studies and registration information, and explain non-use of tests.
  • Garbage in, garbage out (GIGO): pooling low-quality primary studies produces low-quality conclusions.
  • High heterogeneity makes pooled effects difficult to interpret and averages potentially unrepresentative.
  • Causal mechanisms of complex social interventions, such as collective impact, require alternatives such as realist synthesis.

Common pitfalls

  • Inadequate search strategies. English-only or single-database searches miss important studies. Include multiple languages, databases and grey literature.
  • Pooling despite heterogeneity. Forcing a pooled estimate despite high I² reports a misleading single number.
  • Pooling without bias appraisal may precisely summarize bias. Report appraisal, sensitivity analysis and limitations.

Conditions of use and evidence

  • Meta-analysis is not mandatory in systematic review. Consider the meaning of pooling and differences between studies. [1]
Fictional example; not actual results

Consider conditions for pooling two learning support studies

A fictional illustrative example: studies A and B use the same score scale and time point, with mean differences of 2 and 4 points and standard errors of 1 point each.

StudyMean differenceStandard errorFixed-effect weight
Fictional A2 points1 point1/1²=1
Fictional B4 points1 point1/1²=1
Calculation example(2+4)/2 = 3 pointsPooled SE = 1/√2 ≈ 0.707 pointsFormula assuming a common effect

Common errors in judgment

Common mistake

Collecting only significant studies

How to revise

Set eligibility independently of findings.

ReasonSelecting studies by results biases estimates.

Common mistake

Automatically selecting fixed effects because I² is low

How to revise

Select models from study meaning and the estimand.

ReasonHeterogeneity estimation itself is uncertain with few studies.

Organize your thinking by writing

Systematic review planning and management worksheet

A practical template is available to fill in and print on A4 paper.

Open worksheet

Related methods

Regression AnalysisRandomized Controlled TrialContribution AnalysisMeta-EvaluationEvaluation Standards and EthicsCost-Effectiveness Analysis

References and checked sections

  1. Cochrane Handbook: Chapter 10, Analysing data and undertaking meta-analyses
    Jonathan J. Deeks et al. / Cochrane
    Section consulted
    10.1 / 10.2 / 10.10.3 / 10.10.4
    How it relates to the text
    Meta-analysis is not mandatory in systematic review. Consider the meaning of pooling and differences between studies.
    Source checked: (includes AI cross-checking)

Editorial and checking information

Content updated: Source checked: (includes AI cross-checking)No documented human reviewIncludes fictional examplesEditorial policy and scope of checkingCite this pageReport a correction

How to cite this page

EvalCompass Editorial Team. “Meta-Analysis and Systematic Review”. EvalCompass, updated 2026-10-07. https://evalcompass.jun-nakatani.com/en/methods/meta-analysis/
Add the date you accessed the page. Text created by this site is available under CC BY 4.0. Quotations, external materials, and bundled fonts have their own terms of use. When reusing content, credit the source, link to the license, and indicate any changes.

Suggest a correction or improvement to this page

Change history
  • — Checked sources for definitions and applicability conditions, and added fictional examples, reasons for judgments and examples for every worksheet field. Revised unverified claims about history and adoption. Also reviewed existing procedures and cautions, and adjusted fixed participant counts and thresholds and generalizations about institutions.
  • — Correction based on independent content review (content-check-causal). Specified funnel plot and related applicability, asymmetry interpretation and recording reasons for non-use.