ANNEX 1 . 101 There is no separate Annual Evaluation plan but a list of centralised and decentralised evaluations to be undertaken each year is integrated into the DIOS Annual Workplan (the department the evaluation function sits in). DIOS has developed a framework to identify areas to be evaluated which takes into account the UNRWA context: budget size, complexity of the operational environment, closeness of the activity to the core of UNRWA mandate, maturity of the operation, monitoring results, and prior DIOS work with is included as an annex to the Oversight Plan; however, there is limited detail as to how the final selection of evaluations to be undertaken is identified from this framework and it is unclear, in 1, 4, 21, 53 the resource constrained environment, how decisions around prioritisation of evaluations are made in practice. The centralised evaluations are listed in DIOS’s Annual Table of Assignments; these are not prioritised although these is a budgeted number of work days assigned to each evaluation. The decentralised evaluations to be undertaken are listed but it is not clear where these are project funded and they are not prioritised. MI 8.2 Evidence confidence High confidence MI 8.3: Systems are applied to ensure the quality of evaluations Score Overall MI Rating Unsatisfactory Overall MI score 1.6 Element 1: Evaluations are based on design, planning and implementation processes that are inherently quality oriented Element 2: Evaluations use appropriate methodologies for data-collection, analysis and interpretation Element 3: Evaluation reports present in a complete and balanced way the evidence, findings, conclusions, and where relevant, recommendations 2 1 2 Element 4: The methodology presented incudes the methodological limitations and concerns 1 Element 5: A process exists to ensure the quality of all evaluations, including decentralized evaluations 2 MI 8.3 Analysis Source document The UNEG Peer Review of UNRWA’s Evaluation Function in 2015 assessed whether UNRWA had a ‘system in place to ensure the quality of evaluations in their design, methodology, conduct of evaluation and reporting’. This scored 2 (in development). The UNEG Peer Review undertook a quality assessment of a sample of fifteen UNRWA evaluation reports. In terms of design, the average score was 27 per cent (not adequate). The Peer Review score for evaluation design (which would encompass methodologies) was ‘not adequate’ (27 per cent). The Peer Review in its quality assessment rated evaluation limitations as ‘not adequate’ (36 per cent). No meta-synthesis of evaluations has taken place since the UNEG Peer Review due to funding constraints; although undertaking regular meta-synthesis was one of the 34, 53 recommendations in the Peer Review. UNRWA has specific guidance on quality assurance in evaluation developed in 2016, which outline the full evaluation cycle and where quality assurance is necessary at each stage, as well as describing the roles and responsibilities for each of these stages. The document provides quality assurance templates against which evaluations can be assessed. The evaluation division in DIOS backstops decentralized evaluations aiming to insure both independence as well as quality. It is not possible to determine the extent to which these measures have resulted in improvements in evaluation quality. MI 8.3 Evidence confidence High confidence