EvalCompassEvalCompass
Participatory & qualitative approachesExpert consensusPredictionPolicy evaluationIndicator developmentRepeated surveys

Delphi Method

Delphi Method
Also known as: Delphi technique、Expert panel method

A panel responds to several surveys and revisits judgments after receiving aggregated opinions and reasons. Report remaining differences as well as agreement levels. [1]

Delphi implementation management worksheet — openReferences and checked sections
Difficulty
Intermediate
Cost
Medium
Duration
Illustrative estimate, adjusted to scale and staffing: 2–4 rounds × 2–4 weeks per round, 1–3 months total
Content updated: 2026-10-07Source checked: 2026-09-13 (includes AI cross-checking)No documented human reviewIncludes fictional examples

How it works at a glance

Delphi repeatedly collects opinions from a knowledgeable or experienced panel and provides statistical summaries and reasons as feedback. Response anonymity, iteration and controlled feedback are important design elements. [1]

Define panel selection, anonymity, response scales, consensus and stopping rules in advance. Do not force agreement; report minority views and attrition. [1]

  1. Design the research question and panel
  2. Round 1: open questions for exploration
  3. Round 2: rating and ranking
  4. Round 3 onward: iterative convergence
  5. Summarize and interpret final results
  6. Provide panel feedback and report
Sequence of implementation steps. See “Implementation steps” for details of each stage.

When to use it and when it is unsuitable

When to use it

  • When seeking consensus from experts dispersed geographically or in time
  • When systematically aggregating expert views on the validity of evaluation indicators or criteria
  • Structured opinion aggregation when face-to-face discussion is difficult
  • When seeking objective expert agreement on policy or program priorities without domination by particular authorities

When it is unsuitable

  • When time for necessary repetition and feedback is unavailable
  • When estimating representative population proportions or actual causal effects solely from panel agreement

Participants, roles, and deliverables

Participants and roles

A panel with expertise or experience relevant to the question and survey administrators. Set numbers according to needed perspectives and conditions, and specify anonymity between participants and toward administrators.

Main deliverables

  • Round-by-round summaries: medians and interquartile ranges
  • Agreed indicators and criteria
  • Convergence process record
  • Minority opinions

Implementation steps

  1. Design the research question and panel
    Select panel members according to the question and required experience or expertise. Set numbers based on diversity, analysis plans and expected attrition; explain anonymity, consensus and stopping conditions in advance.
  2. Round 1: open questions for exploration
    Collect varied ideas and views through open questions. Organize responses and combine similar items into a structured round 2 questionnaire.
  3. Round 2: rating and ranking
    Return round 1 summaries to all panel members and have them rate importance or agreement using Likert or similar scales. Calculate medians and interquartile ranges.
  4. Round 3 onward: iterative convergence
    Return distributions and anonymized reasons for reconsideration. Apply predefined consensus, stability or maximum-round stopping rules, retaining minority views and attrition records.
  5. Summarize and interpret final results
    Aggregate final medians, interquartile ranges and agreement levels, then interpret findings. Record items without consensus and consider their significance.
  6. Provide panel feedback and report
    Share final findings with panel members and acknowledge contributions. Incorporate agreed indicators and criteria into evaluation design and reporting.

Strengths and limitations

Strengths

  • Anonymity helps reduce authority, status and loud voices' influence. Check panel composition and feedback biases separately.
  • Systematically collects views from geographically dispersed experts.
  • Repeated feedback generates higher-quality collective knowledge than isolated individual judgment.
  • Provides an evidence-based account of indicator and criterion validity.

Limitations

  • Expert agreement does not guarantee truth or the best answer. Consider panel composition bias.
  • Multiple rounds take time and are vulnerable to attrition and declining response rates.
  • Overemphasizing convergence risks losing important minority opinions.

Common pitfalls

  • Selecting experts without a clear definition biases the panel toward particular interests or positions. Document selection criteria.
  • Declining response rates between rounds reduce representativeness. Regular contact and compensation arrangements are needed.
  • Returning only medians without minority reasons encourages excessive conformity.

Conditions of use and evidence

  • Define anonymity and stopping rules before recruitment, and return aggregated responses and reasons for the next round. [1]
  • Do not judge Delphi quality solely by response rates. Explain denominators, attrition and final panel adequacy. [1]
Fictional example; not actual results

A panel considers consultation support indicators

A fictional illustrative example with 8 practitioners and people with service experience rating indicator importance from 1–5.

Item numberQuestion or indicator contentMedianInterquartile range (Q1–Q3)Agreement level: high / medium / low
1Percentage connected to support they wanted44～4.5High: meets this example's predefined median≥4 and IQR≤1 criteria

Common errors in judgment

Common mistake

Declaring an indicator professionally correct because its mean is high

How to revise

Report panel composition, agreement scope and dissent together.

ReasonAgreement does not automatically demonstrate validity.

Common mistake

Calling it convergence when dissenters leave and opinions align

How to revise

Check continuing respondents and attrition separately.

ReasonAgreement can rise solely because participants change.

Organize your thinking by writing

Delphi implementation management worksheet

A practical template is available to fill in and print on A4 paper.

Open worksheet

Related methods

Nominal Group TechniqueKey Evaluation QuestionsIndicator DesignRubric-Based EvaluationFocus Group Discussion

References and checked sections

  1. RAND Methodological Guidance for Conducting and Critically Appraising Delphi Panels
    Dmitry Khodyakov, Sean Grant, Jack Kroger, Melissa Bauman / RAND · 2023
    Section consulted
    Chapter 2: What Delphi Is; Chapter 3: Level of Anonymity, Stopping Criteria; Chapter 5: Consensus; p.37 Panelist Flow and Attrition Across Rounds; p.47 DCAT item 8
    How it relates to the text
    Define anonymity and stopping rules before recruitment, and return aggregated responses and reasons for the next round. Explain participation rates, denominators, attrition and final panel adequacy.
    Source checked: (includes AI cross-checking)

Editorial and checking information

Content updated: Source checked: (includes AI cross-checking)No documented human reviewIncludes fictional examplesEditorial policy and scope of checkingCite this pageReport a correction

How to cite this page

EvalCompass Editorial Team. “Delphi Method”. EvalCompass, updated 2026-10-07. https://evalcompass.jun-nakatani.com/en/methods/delphi/
Add the date you accessed the page. Text created by this site is available under CC BY 4.0. Quotations, external materials, and bundled fonts have their own terms of use. When reusing content, credit the source, link to the license, and indicate any changes.

Suggest a correction or improvement to this page

Change history
  • — Updated explanations to correspond with sources, and added method-specific fictional worksheet examples, judgment examples and revision history.
  • — Made minimal corrections to assertions, fixed conditions or inconsistencies between worksheets and examples based on content check FW-08. Preserved field counts, order, labels and section structure. Did not change verification status of unchecked material.
  • — Based on limited follow-up investigation FW-U01–U05, clarified applicability of analysis methods, recording deadlines and participant counts and premises for response checks. Preserved existing fields without declaring numerical guidance itself erroneous.