Reporting, use & meta-evaluationEvaluation qualityEvaluation criteriaThird-party evaluationAccountabilityEvaluation system
Meta-Evaluation
Meta-Evaluation
Also known as: Evaluation of evaluation、Meta-evaluation practice
Examines the quality of an evaluation itself against selected standards and evidence. It supports improvements at the design stage and checks the credibility of completed evaluations. [1]
- Difficulty
- Advanced
- Cost
- Medium
- Duration
- Illustrative estimate (adjust for scale and team): 1–4 weeks (review of a single evaluation) / several months (system-level meta-evaluation)
Content updated: 2026-09-13Source checked: 2026-09-13 (includes AI cross-checking)No documented human reviewIncludes fictional examples
How it works at a glance
Meta-evaluation examines whether an evaluation's design, conduct, evidence and explanations are appropriate, rather than rating the program again. JCSEE standards address five attributes: utility, feasibility, propriety, accuracy and evaluation accountability. [1]
Internal checks by practitioners and independent external checks offer different perspectives. Request questionnaires, analysis records and other materials when the report is insufficient, distinguishing missing documentation from a definitive judgment of poor quality. This site's scoring table is a simplified example for learning about standards. [1]
- Define the purpose and scope of meta-evaluation
- Select review standards and checklists
- Collect evidence and review documents
- Score and judge against standards
- Develop overall judgments and identify strengths
- Provide feedback and recommendations for improvement
When to use it and when it is unsuitable
When to use it
- When checking the credibility and quality of a received evaluation report
- When a third party examines the quality of an organization's or ministry's evaluation system as a whole
- When learning materials on good evaluation practice are needed as part of evaluation capacity building
- When checking individual report quality before synthesizing or comparing multiple evaluations
When it is unsuitable
- When resources or time to conduct evaluation are unavailable. Prioritize conducting basic evaluation first.
- When political distrust of evaluation itself is growing. Meta-evaluation risks being used to deflect responsibility.
Participants, roles, and deliverables
Participants and roles
Meta-evaluation team (1–3 independent evaluation experts) and a commissioner-side coordinator. For a single evaluation, obtain documents from its evaluation team as well.
Main deliverables
- Meta-evaluation report
- Evaluation quality scoring sheet (by standard)
- Recommendations for improvement
Implementation steps
- Define the purpose and scope of meta-evaluationClarify its purpose (quality assurance, system improvement or learning) and agree on the number and scope of evaluations and standards to use, such as Joint Committee standards. Decide whether it is formative or retrospective.
- Select review standards and checklistsUse the Joint Committee's Program Evaluation Standards, third edition (30 standards), or UNEG norms as a basis, adding organization-specific standards. Recognize that selecting standards is itself a value judgment, and disclose the choice transparently.
- Collect evidence and review documentsCollect evaluation reports, design documents, data, questionnaires and stakeholder interviews. Examine actual evaluation processes as well as documents, including stakeholder participation and arrangements for independence.
- Score and judge against standardsRecord evidence and reasons for judgments for each selected standard. Define the scale if using scores, and distinguish an undetermined judgment due to insufficient materials from a low rating. Compare and record differences between reviewers.
- Develop overall judgments and identify strengthsBring together ratings across standards to assess overall strengths and areas for improvement. Prioritize serious weaknesses, such as methodological flaws or conflicts of interest, rather than simply adding scores.
- Provide feedback and recommendations for improvementProvide findings to evaluators and commissioners, and develop recommendations for the next evaluation cycle. Incorporating them into organizational guidelines and training programs increases system-level effects.
Strengths and limitations
Strengths
- Provides evidence enabling users to judge report credibility objectively
- Creates a feedback loop for continuous improvement of evaluation practice across the organization
- Can use existing standardized criteria, such as Joint Committee standards, reducing the burden of designing original standards
- Strengthens evaluator and institutional accountability and public trust in evaluation
Limitations
- Faces the recursive problem of depending on the quality and independence of the meta-evaluators themselves
- Existing standards developed in Western contexts require examination of cultural and institutional fit
- Checklists alone have limits in understanding actual practice when evaluation documentation is poor
Common pitfalls
- Completing checklists mechanically becomes pass/fail labeling. Design the process as an opportunity for improvement and learning.
- Focusing only on methodological standards overlooks utility, including whether evaluation actually informed decisions
- Neglecting to check meta-evaluator independence from the evaluation and conflicts of interest
Conditions of use and evidence
- In addition to evaluation documentation, JCSEE distinguishes internal and external meta-evaluation. [1]
Identify missing records through meta-evaluation
The following is a fictional example for illustration. An external reviewer checks a learning support evaluation report against supporting materials.
Scroll horizontally to view the table
| Standard | Score (1–4) | Evidence and comments |
|---|---|---|
| U3: Clarity of purpose | 3 | States that findings will inform target participant selection next year |
| U5: Relevant information | 2 | No materials on the experiences of absent children |
Common errors in judgment
Common mistake
Rate evaluation quality poorly because its conclusion differs from expectations
How to revise
Check the materials and procedures leading to conclusions against standards
ReasonThis separates evaluation quality from preferences about its findings.
Common mistake
Conclude that consent was not obtained because consent records cannot be found
How to revise
Request supporting materials and treat the judgment as undetermined
ReasonInsufficient evidence differs from a confirmed violation.
Organize your thinking by writing
Meta-evaluation checklist (aligned with Joint Committee standards)
A practical template is available to fill in and print on A4 paper.
Open worksheetRelated practical guides
Related methods
References and checked sections
- Program Evaluation StandardsJoint Committee on Standards for Educational Evaluation · 2010 (third edition)
- Section consulted
- Standards Statements; Evaluation Accountability Standards (E1–E3)
- How it relates to the text
- In addition to evaluation documentation, JCSEE distinguishes internal and external meta-evaluation.
Editorial and checking information
How to cite this page
EvalCompass Editorial Team. “Meta-Evaluation”. EvalCompass, updated 2026-09-13. https://evalcompass.jun-nakatani.com/en/methods/meta-evaluation/
Add the date you accessed the page. Text created by this site is available under CC BY 4.0. Quotations, external materials, and bundled fonts have their own terms of use. When reusing content, credit the source, link to the license, and indicate any changes.
Change history
- — Updated explanations to correspond with sources, and added method-specific fictional worksheet examples, judgment examples and revision history.