Appendix F: Factors that contribute to evaluation quality identified in other First DFAT Review of Meta-evaluation of Are SIDA evaluations good NORAD Can we DFID Rapid Review of Operational Evaluations quality and coverage of enough? (2008) demonstrate the difference Embedding Evaluation (2014) USAID evaluations that Norweigan Aid Makes? (2014) (2013) (2014) Summary of Examined the quality of 87 Examined the quality of 340 Examined the quality of 34 Examines why it is difficult A review of DFID’s program operational evaluations evaluations conducted operational evaluation to determine the results (ie to evaluation’ within completed in 2012. between 2009 2012. reports published between outcomes and impacts) of and strengthen the – 2003 and 2005. Norweigan aid. evidence base of its programs. Factors related to evaluation quality Longer evaluations The meta-evaluation The meta-evaluation found Duration of an (including more fieldwork believes that cost and no clear correlation evaluation (total days days) associated with duration influence quality, between evaluation cost and field days) higher evaluation quality (p. but did not have the data to and quality, and disputes 34-5; statistical analysis). test this (p. 27). whether cost is actually a Note that reliable cost data factor that would drive was not available. quality (p. 24) The purpose of an Progress reports are higher Quality of evaluation evaluation (e.g. quality than completion depends on whether it is fit progress or completion reports (p. 33; statistical for intended purposes (p. evaluation) analysis) 40; an assumption and no clear evidence to support it is provided) Evaluation team size The optimal team size for quality evaluation is 3-4 team members (p. 35; statistical analysis) Evaluation team Teams with a strong team The presence of an Evaluator competencies are composition, leader and expertise evaluation specialist on the assumed to influence particularly the include covering evaluation team is linked to better evaluation quality (not presence of specialist expertise particularly evaluation quality (p. 25; tested statistically) evaluator skills evaluation expertise, statistical analysis) leadership, and sectoral knowledge are associated with higher evaluation quality (p. 36; qualitative analysis) Quality of investment The quality of an M&E systems investment’s M&E system is assumed to drive evaluation quality (p. 32; no statistical analysis possible)
Appendix F: Factors that contribute to evaluation quality identified in other First DFAT Review of Operational Evaluations (2014) Meta-evaluation of quality and coverage of USAID evaluations (2013) Are SIDA evaluations good enough? (2008) NORAD Can we demonstrate the difference that Norweigan Aid Makes? (2014) DFID Rapid Review of Embedding Evaluation (2014) Summary of Examined the quality of 87 operational evaluations completed in 2012. Examined the quality of 340 evaluations conducted between 2009 – 2012. Examined the quality of 34 operational evaluation reports published between 2003 and 2005. Examines why it is difficult to determine the results (ie outcomes and impacts) of Norweigan aid. A review of DFID’s program to evaluation’ within and strengthen the evidence base of its programs. Factors related to evaluation quality Duration of an evaluation (total days and field days) Longer evaluations (including more fieldwork days) associated with higher evaluation quality (p. 34-5; statistical analysis). Note that reliable cost data was not available. The meta-evaluation believes that cost and duration influence quality, but did not have the data to test this (p. 27). The meta-evaluation found no clear correlation between evaluation cost and quality, and disputes whether cost is actually a factor that would drive quality (p. 24) The purpose of an evaluation (e.g. progress or completion evaluation) Progress reports are higher quality than completion reports (p. 33; statistical analysis) Quality of evaluation depends on whether it is fit for intended purposes (p. 40; an assumption and no clear evidence to support it is provided) Evaluation team size The optimal team size for quality evaluation is 3-4 team members (p. 35; statistical analysis) Evaluation team composition, particularly the presence of specialist evaluator skills Teams with a strong team leader and expertise include covering evaluation expertise particularly evaluation expertise, leadership, and sectoral knowledge are associated with higher evaluation quality (p. 36; qualitative analysis) The presence of an evaluation specialist on the team is linked to better evaluation quality (p. 25; statistical analysis) Evaluator competencies are assumed to influence evaluation quality (not tested statistically) Quality of investment M&E systems The quality of an investment’s M&E system is assumed to drive evaluation quality (p. 32; no statistical analysis possible)