EVALUATION OF UNHCR-LED INITIATIVES TO END STATELESSNESS – FINAL REPORT – VOLUME I assessment the evaluation team and evaluation manager, in consultation with the ERG, decided interviews with stateless persons could pose them harm and should not be conducted. 13. Phase 2 – Remote global data collection – During this phase the evaluation team conducted an indepth review of global and regional documents, past evaluations and internal assessments, literature and other data sets. A list of documentary sources is included in a bibliography in Annex 7. Remote interviews were conducted with key UNHCR staff at headquarters and in regional offices, and a global survey of UNHCR country offices was undertaken. At the end of this phase, the evaluation team issued an interim desk review report presenting the consolidated and synthesized evidence collected. 14. Phase 3 – Case study data collection – The evaluation collected data for four country case studies (Cote d’Ivoire, Kenya, the Philippines, and Tajikistan), and three regional case studies (Americas, Europe, and the Middle East and North Africa (MENA)).8 The evaluation team also facilitated iterative dialogues with two multi-stakeholder Regional Engagement Groups in Africa and Asia (see Annex 15). 15. Phase 4 – Analysis and sensemaking – Following the close of data collection, the evaluation team completed cleaning and coding of all data and held team analysis workshops to conduct cross case analyses and triangulate findings based on all stages of data collection. Initial case analysis work was discussed and presented in a series of sensemaking analysis meetings with the Evaluation Service and Statelessness Section followed by validation workshops with key stakeholders from each case study. 16. Phase 5 – Reporting – During this phase, the evaluation team drafted findings and developed case study reports (see Annex 4). This global synthesis report was produced with overall findings and recommendations for the evaluation. All final reports were reviewed by the ES, internal UNHCR stakeholders, and the ERG and their comments addressed by the evaluation team to the extent possible. 3.1 Methodological Approach 17. To effectively address the questions identified in the Evaluation Matrix according to available documentation and evidence, this evaluation followed a modified process tracing framework 9. The approach to process tracing enabled the evaluation team to investigate the role of UNHCR’s activities, partnerships, and institutional structure, investments in, and culture around statelessness in contributing to or hindering progress on UNHCR’s statelessness goals. It also helped identify areas of good practice and innovation, while providing concrete recommendations for achieving the aims of the GAP in its remaining years according to country contexts, regional dynamics, and types of statelessness. 18. Table 2 provides an overview of evaluation data collection and validation efforts. Table 2 – Overview of data collection and validation Collection Method Coverage and participation Notes Desk study Over 150 global documents and literature sources reviewed, analysis of financial, RBM and human resources data An evaluation library was compiled by the Evaluation Service covering historical and contemporary background documents on UNHCR and other actors work on statelessness, the campaign and GAP, internal UNHCR systems, past evaluations and assessments, partnerships, UN cooperation, financial data, and performance management data. Documents and data records were collected and reviewed on a rolling basis to fill gaps and on stakeholder recommendations. Global survey to all operations, missions, national offices 483 survey responses received from UNHCR field staff Overall, 53 percent response rate10 by targeted individual, 93 percent by targeted operation or office11 8 Case studies were initially identified by the ES based on criteria related to i) UNHCR financial investment between 2013 and 2019, ii) stateless population size reported by UNHCR or another credible source, iii) significance of progress (positive or limited) against the GAP vis a vis UNHCR financial investments, and iv) factors contributing to statelessness that are representative of the region or new and emerging issues. Following consultations with Regional Bureaux, country offices and the ERG the four country cases were confirmed and the three regional case studies added to the evaluation design. 9 Methodology is adapted from Process-Tracing Methods: Foundations and Guidelines (Beach and Pedersen 2013) according to (1) guidance from the Center for Development Impact (see Number 10 Practice Paper Annex, April 2015, “Applying Process Tracing in Five Steps”) and (2) the tested modifications applied through Oxfam’s evaluation approach as described in the Oxfam Process Tracing Protocol. 10 The survey targeted 918 UNHCR staff in country offices and operations, multi-country offices and representation offices. By stakeholder group the response rates were: 58% for Country Representatives, Heads of National Offices and similar posts, 53% for Protection and Statelessness officers, 48% for Heads of Sub Office / Heads of Operations, and 41% for External Relations Officers. By region the response rates were: 46% Africa, 48% the Americas, 65% Asia and the Pacific, 57% Europe, and 52% MENA. 11 At least one person from each targeted operation or office in Asia and the Pacific and MENA responded to the survey. For other regions the operation/office response rate was Africa – 91%, Americas – 94%, Europe – 88%. 6 UNHCR / May, 2021