Evaluation Management
Introduction
This indicator takes information from the ERC (Evaluation Resource Center) to measure the performance in managing evaluation, including the timely conduct of evaluations, the completion and implementation of management responses, and the quality of decentralised evaluations.
This is both a Global, Regional, and Country Office level indicator.
Organisational Objective
The organisational objective can be broken down into three parts:
100% of planned evaluations were completed on time.
100% of completed evaluations have management responses with no overdue key actions.
100% of evaluations are of acceptable quality.
Notes
We exclude UNV, UNCDF, UNOSSC
Calculation of Scoring
The Evaluation Management Indicator is comprehensively divided into three distinct components, each aligned with a specific organizational objective. These components are as follows:
Evaluation Planning: This part assesses the efficiency and timeliness of the evaluation scheduling process, ensuring that evaluations are completed as planned.
Evaluation Management Responses: This section measures the promptness and effectiveness of management in responding to the evaluations conducted, emphasizing the importance of timely action on feedback and recommendations.
Evaluation Quality: The final component evaluates the overall quality of the evaluations themselves, focusing on the depth, accuracy, and usefulness of the findings and conclusions drawn.
These components provide a holistic view of an organization's evaluation management capabilities, reflecting its commitment to continuous improvement and accountability.
Part 1: Evaluation Planning (30% weight)
This focuses on the punctuality and scheduling discipline of the evaluation process, specifically measuring the percentage of evaluations completed within their designated timeframes. This assesses the effectiveness of evaluation planning and execution at the Country Office level.
Non-Overdue Evaluations / Total Evaluations = % of On-Time Evaluations
Speak with Evaluation team on this, as what is "Completed" precisely?
Example below:
So this is a snapshot of Global, so in this case, it would be 2113/2176
The reason it is 2113, is because there are 2176 total evaluation, and 63 overdue, so that gives us 2113 non-overdue planned evaluation, which is ~97%
Part 2: Evaluation Management Responses (30% weight)
A) % Evaluations without a management response 6+ weeks after evaluation (50% of Part 2)
This component evaluates the proportion of evaluations lacking a management response more than six weeks after completion. It is a key indicator of the efficiency and timeliness with which management acts on evaluation findings.
The formula used is:
1 - (Number of Evaluations Overdue by More Than 6 Weeks / Total Completed Evaluations in the Programme Period) = Percentage of Evaluations with a Management Response
Clarification note: This part of the scoring methodology is intended to highlight evaluations with overdue management response so that corrective action can be taken. Once a management response is submitted, even if overdue, it is no longer counted as overdue.
Example from ERC Dashboard:
There are a total of 1924 Reports, with 89 being overdue for more than 6 weeks.
Total completed evaluation in the programme period = All Reports - Waiting for Management Response.
So the calculation:
1-(89/(1924-122)) = 95.06%
B) Key actions timings (50% of Part 2)
Within Part 2B, the following two metrics each contribute 50% by averaging their scores:
Non-Overdue Key Actions / Total Actions = Percentage of Actions Not Overdue
Not Long-Overdue Key Action / Total Actions = Percentage of Actions Not Long Overdue
Important Note 1: Long Overdue Key Actions are intentionally double-counted in this section to underscore their significance in the evaluation process. This approach highlights the critical importance of addressing long overdue actions promptly, reflecting their heightened impact on overall performance assessment.
Important Note 2: "Total Actions" include ALL key actions, including overdue, completed, and not yet due.
Example for part 1 of 2B:
Total = Total in the screenshot above from ERC
Percentage of Actions Not Overdue = 100-Overdue % in Screenshot Above
Further discussions are required here about cut-off dates, and including and not including "No Longer Applicable" One other consideration: ignoring "Completed" and "No Longer Applicable" because this will keep growing the denominator. Instead, on the above is should be:
Overdue / (Overdue+Not Initiated + Initiated) = % Overdue
This needs to be confirmed with IEO.
Example for part 2 of 2B:
In this case, Long Overdue % would be 1 divided by Open Actions (Overdue+Not Initiated + Initiated)
Part 3: Evaluation Quality (40% Weight)
This section determines the overall quality of evaluations based on a comprehensive scoring system. Evaluation quality is critical, reflecting the evaluation's depth, accuracy, and usefulness.
Six distinct scores are possible, each reflecting a specific level of evaluation quality. These scores are predetermined and assigned based on established criteria that evaluate various aspects of the evaluation process and its outcomes.
This is where it is shown in the ERC Dashboard:
The final quality score is derived by calculating the assigned scores' straight average across all programme period evaluations in the 2022, 2023, and 2024. For each new calendar, we drop the previous calendar year. This average score provides a clear, quantitative measure of the overall quality of evaluations.
Overall Calculation
Part 1: This section's score is derived directly from the Evaluation Planning performance.
Part 2: The score for this section is calculated as the average of two components:
A: The percentage of evaluations that have received a management response.
B: The average of two metrics: the percentage of actions not overdue and the percentage of actions not long overdue.
Part 3: This score comes from the Evaluation Quality assessment.
Final Score Calculation: The final score combines these parts, weighted accordingly:
Part 1 contributes 30%.
Part 2 contributes 30%.
Part 3 contributes 40%.
The formula for the final score calculation is:
Final Score = (Part 1 × 0.3) + (Part 2 × 0.3) + (Part 3 × 0.4).
This calculation method ensures that each aspect of the evaluation process—planning, management response, and quality—is appropriately considered in the overall performance score.
The traffic light indicator methodology for this indicator is:
Green = 90+
Yellow = 75+
Red = <75
Useful Links
Last updated
