Evaluation glossary

Changes in people, organizations or other targets that arise through activities and outputs. This site uses the term to describe changes in behavior or conditions, but some sources classify changes in knowledge and skills as outputs. Check the definitions used by the relevant scheme or source first.

Services or deliverables directly produced by activities. Check their quality as well as counts and participant numbers. Sources classify results differently, so define the distinction between what was provided and what changed for whom.

Assessing the causal relationship between an observed change and an intervention. Examine other influences rather than assuming that all change is the result of a single program. Contribution does not mean the other factors themselves.

Information and grounds that support a question or judgment. Check data quality, collection and analysis methods, and relevance to the question. Do not assess strength by method name alone; specify what the evidence concerns.

Clarifying policy objectives and planning and examining policy on the basis of rational evidence. Connect evaluation findings and statistics to decisions, explaining the scope supported by the sources and the uncertainty involved.

Significant higher-level effects of an intervention. These include positive, negative and unintended effects. Its meaning in the DAC criteria differs from usage in impact evaluation that refers to causal effects, and it does not necessarily refer only to results in the distant future.

A document in which evaluators set out their understanding of the terms of reference and their proposed approach at the start of an evaluation. Share the evaluation framework, methods, constraints and work plan with the commissioner to establish a common understanding before data collection. Adapt its content and submission timing to the contract and evaluation arrangements.

An evaluation that examines causal changes produced by an intervention or progress toward higher-level objectives. Alongside experimental and quasi-experimental approaches, there are non-experimental, theory-based methods. Some usage refers only to estimating effects through an explicit counterfactual comparison, so state the meaning adopted.

The extent to which an observed difference is appropriately estimated as a causal effect of an intervention, distinguished from other factors. Examine the selection of comparators, confounding and changes during implementation. The name of a research method alone does not guarantee validity.

The extent to which evaluation results can be generalized to the target population or other groups and contexts. High internal validity does not necessarily permit generalization. Check differences in conditions at the intended setting of application as well as sample representativeness.

Evaluation by individuals or organizations external to both the funding and implementing organizations. External affiliation alone does not guarantee independence. Also check access to information, autonomy in investigation and reporting, and pressure arising from funding or reporting arrangements.

The state used for comparison that represents what would have happened without the intervention. Because the same target cannot be observed with and without the intervention simultaneously, this state is estimated on the basis of the design and assumptions.

The process of understanding information obtained through evaluation and applying it to individual or organizational decisions and practice. Do not assume that reading a report means its findings were used; record what changed.

Evaluation used to check appropriateness, feasibility and other aspects in preparation for full implementation, and to improve design and operation. Check the specific scope in the framework adopted, including improvements during implementation.

An indicator of the magnitude of a difference or association. Some measures retain the original units, such as a mean difference, while others are standardized, such as Hedges' g. Interpret the number alongside the scale, comparator and uncertainty.

One of the DAC evaluation criteria. It asks whether resources are being used to produce results economically and in a timely way, or are likely to do so. Examine feasible alternatives, implementation timing in light of changing circumstances, and operational efficiency.

Distortion of a causal-effect estimate by another factor associated with both intervention participation and the outcome. For example, if trainees' pre-existing motivation affects both participation and income, the income difference between participants and non-participants cannot be treated as the effect of training alone.

Assigning labels to the meaning of accounts and records and examining patterns relevant to the question. Some approaches emphasize agreement between evaluators, but a uniform agreement-rate criterion should not be imposed on approaches such as reflexive thematic analysis.

The role a program plays in producing change alongside other factors. Contribution analysis compares hypotheses about change with evidence. An explanation of contribution does not automatically yield a standard percentage contribution.

Evaluation of a program by the people responsible for designing and implementing it. Distinguish this from internal evaluation conducted by another department in the organization. Plan for learning and clarify the judgment criteria and treatment of unfavorable evidence.

Evaluation carried out after an intervention is completed. It examines effects after completion, the continuation of benefits and other aspects. Do not impose a common rule on the number of years after completion; decide timing from when outcomes are expected to emerge and the evaluation purpose.

Evaluation carried out before an intervention is implemented. Depending on the evaluation purpose, examine needs, design, feasibility, anticipated risks and other aspects. Do not require a particular diagram or method; set questions needed to decide on implementation or improve the design.

One of the DAC evaluation criteria. It asks whether the net benefits of an intervention continue or are likely to continue. Alongside financial, institutional, social and environmental capacities, examine risks to continuation and trade-offs between benefits.

A quantitative or qualitative item that captures conditions or progress relating to an intervention, its results or its context. Indicators do not necessarily represent every aspect of the subject. In practice, record definitions, sources and measurement timing, as well as units and any necessary denominator for numerical indicators, and distinguish indicators from targets.

People, groups or organizations that benefit directly or indirectly from an intervention. They may include people beyond the intended target group. Not everyone receiving a service necessarily benefits, so distinguish intended beneficiaries from those who actually benefited.

Consistency and stability of data or evaluation judgments. Check whether similar results are obtained when procedures are repeated under similar conditions. This differs from validity, which concerns what is being measured; consistent measurement does not necessarily suit its purpose.

Bias in which factors related to participation or selection are associated with outcomes, making a simple comparison misrepresent the causal effect. Bias can lead to either overestimation or underestimation. Also examine attrition after assignment.

An approach that uses a theory of how a program produces outcomes, such as program theory or a theory of change, as the evaluation framework. It emphasizes explaining why the program works or does not work, and empirically tests the hypotheses represented by boxes and arrows.

Explaining activities, judgments, resource use and other matters to stakeholders and fulfilling the associated responsibility. Consider not only submission of a report but also dialogue and records that allow the evidence and judgment process to be traced.

Evaluation conducted at the end of an intervention or phase to examine the extent to which expected outcomes occurred and to judge value or significance. Distinguish its purpose from formative evaluation used for improvement, while recognizing that its results can also inform the next design.

A general term for evaluation carried out from the position of a third party at a distance from implementers and others. On this site, it does not denote the qualification or independence requirements of any specific scheme. Do not judge independence from the label alone; examine access to information and pressure on investigation or reporting.

The value or level of an indicator to be achieved by a specified time. A baseline describes the current state, while a target describes the intended state. In practice, record the rationale, deadline and available resources.

The OECD-DAC evaluation criteria. They examine value through 6 perspectives: relevance, coherence, effectiveness, efficiency, impact and sustainability. They are not a uniform procedure for scoring all criteria with equal weight.

Change estimated to have occurred even without an intervention. It is estimated in SROI and other approaches to avoid overstating outcomes. Examine others' contributions and displacement separately; subtraction alone does not establish causality.

Meeting a statistical-test criterion, such as a p-value falling below a specified significance level. This differs from effect magnitude or practical importance. A p-value is neither the probability that chance alone produced the result nor the probability that a hypothesis is true. Consider it alongside the design, effect size and interval estimates.

Resources used in a program, including people, funding, equipment, information and time. A logic model records their relationship to activities, but placement in a particular column does not itself define an input.

Combining information sources, methods, analysts, theories or other elements to examine agreement and disagreement in results. Do not collect only agreement; also analyze the targets and conditions associated with discrepancies.

Evaluation by a department or individual within the organization being evaluated. It includes self-evaluation by those responsible for program design and implementation, but may also be undertaken by another internal department. Specify roles, reporting arrangements and management of conflicts of interest.

A term describing the possibility that observation or participation in research affects people's behavior or responses. The magnitude and direction vary by context; do not treat it as a universal law that behavior always improves.

A systematic tendency that shifts estimates or measurements away from the true value in a particular direction. It can arise without intentional misconduct or preconceptions, and increasing the amount of data alone does not necessarily resolve it. In evaluation, examine target selection, question wording, analytical procedures and other aspects.

Conditions or data before an intervention or at the start of a measurement period. They provide the starting point for comparing later change. Do not assign 0 to unmeasured values; when using substitute sources, note differences in timing, targets and definitions. A before-and-after difference alone does not determine the causal effect.

Evaluation of whether a program is implemented as planned, and of the quality, quantity and reach of activities and outputs. It clarifies what was delivered, to whom, how and to what extent, complementing the interpretation of outcome evaluation. It is also used to examine implementation fidelity.

An evaluation approach in which stakeholders, including the target group, jointly engage in planning, data collection and interpretation, conclusions and recommendations, and use of results. In practice, specify who participates in which decisions at each stage, distinguishing this from participation that only involves being consulted.

The procedure for selecting some people, organizations or other units from the whole group of interest, the population. It includes probability and non-probability sampling; purposive and convenience sampling are examples of the latter. Consider the extent of generalization permitted by the purpose and selection method.

Bias arising when respondents choose answers they consider socially desirable or pleasing to the listener. This can occur unconsciously. Examine question wording, interviewer influence, and differences between answering face to face and answering alone.

A person, group or organization with a direct or indirect interest in an intervention or its monitoring and evaluation. This includes beneficiaries, implementers and funders. In practice, also check whether the voices of those affected are missing.

Evaluation carried out approximately halfway through an intervention's implementation period. Set questions according to the purpose, such as checking progress or reviewing design and operation. Check whether and when it is required for each program or scheme.

A document specifying the evaluation purpose, scope, methods, judgment criteria, resources, timeframe, reporting requirements and other elements. It provides a basis for the commissioner and evaluators to share an understanding of the work. In practice, also record methods refined at inception and any changes.

The degree to which the logic and factual basis of an evaluation are appropriate. In measurement, it concerns whether indicators and methods capture what is intended. Reliability, or consistency of results, alone does not establish validity.

One of the DAC evaluation criteria. It asks whether an intervention's objectives and design respond to the needs, policies and priorities of beneficiaries, society, relevant organizations and others. Examine differences between priorities and continued appropriateness after circumstances change.

Norms and criteria for examining evaluation quality. The JCSEE Program Evaluation Standards set out perspectives on utility, feasibility, propriety, accuracy and evaluation accountability. Compare the original text with the evaluation context when deciding how to apply each standard.

A concept used to consider when additional data no longer adds new understanding to concepts or analysis. Distinguish theoretical saturation from code or data saturation and use it according to the qualitative analytical approach adopted. It is not a uniform required participant count.

A method of intentionally selecting targets believed to have information relevant to the research purpose, on the basis of characteristics or conditions. Examples include maximum variation, cases meeting specified criteria and disconfirming cases. Distinguish this from statistical generalization through probability sampling and state the selection rationale and scope of conclusions.

One of the DAC evaluation criteria. It asks to what extent an intervention has achieved or is expected to achieve its objectives and results. Examine the importance of objectives and differences in results between groups. Distinguish this from efficiency, which concerns economical and timely use of resources.

A framework describing levels of quality or achievement and what each level looks like for each judgment dimension. It is used to share how evidence is judged against levels and to reach an overall conclusion. Having a table alone does not automatically ensure agreement in judgments.

Selecting study targets from a population through a probabilistic mechanism. In simple random sampling, every target has an equal probability of selection. Its purpose differs from random assignment to an intervention; also consider non-response and omissions from the sampling frame.

Evaluating the quality, credibility and usefulness of an evaluation itself: evaluation of evaluation. It examines whether an individual evaluation was conducted appropriately against evaluation standards and ethics. Note that this differs conceptually from meta-analysis, which statistically synthesizes multiple evaluations or studies.

A research or evaluation design combining quantitative and qualitative data collection and analysis with integration of their results. Do not stop at running studies in parallel; plan where results will be connected to answer the questions.

The continuous, systematic collection of indicators and related information to understand implementation progress, intended outcomes, unexpected changes, resource use and context. Do not restrict it to activity counts; connect it to management and evaluation use.

Combining monitoring, which involves continuous tracking and recording, with evaluation, which systematically examines value and reasons for change. Coordinate purposes, information sources, roles and timing of use to obtain information needed for daily management and decisions.

A core question that an evaluation should answer, explicitly expressed in forms such as "Did this program improve participants' livelihoods?" or "Why did effects fail to emerge in some areas?" Select evaluation questions on the basis of the evaluation purpose, users' needs and available resources; they determine the design of methods, indicators and analysis.

Efforts to strengthen individuals' and organizations' ability to conduct evaluation and understand and use its findings. This includes not only knowledge and skills but also resources, information-sharing arrangements and an organizational culture that supports evaluation. Participation in evaluation may itself be used as a learning opportunity.

A table linking evaluation questions to information sources and data collection methods that can answer them. Use it to check for gaps in necessary evidence and opportunities to compare multiple sources. In practice, add columns needed for planning, such as indicators, analysis methods, responsibilities and timing.

Conditions or data after an intervention or at the end of a measurement period. They are used to assess change from the baseline. An endline does not necessarily imply a new survey; align the targets, timing and definitions being compared. Do not treat the entire before-and-after difference as the program effect.