UN agencies, government agencies and NGO’s in order to benefit from their field knowledge. In future we aim to extend the disaster verification process to the entire set of annual entries. We have also submitted this methodological annex to external peer reviewers, and elements of our methodology were reviewed in previous years by a different set of independent experts. We will embed the external peer review and internal quality assurance processes into our future work to ensure that the methods we use to produce our figures are robust and that we have presented them accurately. Qualitative assessment of confidence in estimates for displacement associated with conflict Building on lessons from existing assessments There have been several attempts recently to design confidence assessment schemes to evaluate data on internal displacement as part of a broader movement in the field of humanitarian needs assessments.5 The Task Force on Population Movement in Yemen (TFPM), for example, has developed a confidence rating based on disaggregation by sex and age, and the availability of data on districts of origin and displacement.6 IOM Iraq calculates a confidence rating in order to produce an estimate for each location in its displacement tracking matrix, based on the number of informants used, discrepancies between information from different sources, the accessibility of the location and the ability to independently validate the data received.7 The Syria dynamic monitoring report (DYNAMO) gives a confidence rating based on the number of sources, the manner and extent that the data can be independently verified, the amount of convergence among the different sources and the degree to which they correspond with contextual information about the situation.8 Such assessments may seem reassuring, but if poorly conceived or implemented they may provide a false sense of certainty or confidence. They may hide the arbitrariness of the under- lying criteria and the way they are weighted and aggregated. They may also reflect the biases and challenges inherent in the various steps involved in constructing an index and collecting the data. To limit evaluators’ bias and improve objectivity and consistency, clear decision rules are needed that limit the number of dimensions taken into account. There are ways of overcoming the limitations of points-based scores, but their complexity may render them opaque, adding another layer of potential confusion. Using only four indicators with two to five possible values for each, IOM Iraq’s assessment framework yields up to 126 unique possible combinations.10 The challenge of applying nationally specific tools at the global level It is difficult to extrapolate to the global level from confidence ratings designed for national circumstances. The three examples discussed above all refer to situations in which a single organisation or cluster designs the entire national data collection process. At the global level, aggregation and cross-country comparison is made more difficult by the number of data sources and the fact that their motivations for collecting information ranges from rapid needs assessments to victim compensation without any a priori global coordination. Sources’ methodologies also vary widely, from satellite imagery, registration, sampling, key informant interviews and censuses, to name but a few. This diversity stands in stark contrast to the standardisation of data in the three national examples mentioned above. As such, the same set of criteria cannot easily be used to judge reliability, and the diversity in which the results are reported makes it more difficult to make comparisons between countries. IDMC’s confidence assessment We designed a comprehensive framework to assess the confidence we have in the estimates we publish. The methodology and results presented in this report are the initial steps of a process we will continue to develop through several more iterations.4 METHODOLOGICAL ANNEX 109