Analysis of expert interviews Introduction The initial review of the literature on meta-evaluation concluded that are very different understandings of what constitutes meta-evaluation and a wide range of different methods in use. To explore these issues in more detail we undertook a series of semi-structured interviews with experts in the field. This report analyses the views of the experts who we consulted: The next section provides details of the backgrounds of the interviewees; Section 4.3 reports their views on what meta-evaluation is; Section 4.4 describes their assessment of the current state of the art of meta-evaluation and the main challenges which it faces; Section 4.5 presents the experts’ views of how one might evaluate the legacy of the 2012 Olympic and Paralympic Games; and Section 4.6 draws together the key points to emerge from the interviews. Interviewees The interviewees are acknowledged experts in the fields of evaluation and/or sports policy. The initial sample of potential interviews was identified from the literature review (discussed in section 3.0) and the authors' own knowledge of the field. Thereafter a method was ‘snowball’ used which involved asking early interviewees to suggest others who they believed would have useful insights into meta-evaluation approaches. A total of 18 experts were approached. Five declined to participate (some claimed not to know enough about meta-evaluation; one was unwilling to disclose details of the methods which they used). A total of 13 experts drawn from academia and consultancy firms and from across the US, UK and the rest of Europe were interviewed (see Annex 1). All 13 had direct experience of meta-evaluation research (broadly defined) or related activities such as meta-analysis. Interviews were conducted using a topic guide which was adopted by all interviewers (see Annex 2). Results were recorded in contemporaneous notes taken by interviewees and analysed using a standard matrix. Definitions Meta-evaluation in theory The literature review undertaken as part of the methods development study identified three main schools of thought about what constitutes meta-evaluation. Some researchers and commentators see meta-evaluation as being concerned primarily with standard setting. Seen in this light meta-evaluation is a process of establishing criteria for the evaluation of evaluations. The purpose of a meta-evaluation is to examine other studies against an established set of standards and goals in order to determine whether they were conducted in a rigorous and robust fashion.
Analysis of expert interviews Introduction The initial review of the literature on meta-evaluation concluded that are very different understandings of what constitutes meta-evaluation and a wide range of different methods in use. To explore these issues in more detail we undertook a series of semi-structured interviews with experts in the field. This report analyses the views of the experts who we consulted: The next section provides details of the backgrounds of the interviewees; Section 4.3 reports their views on what meta-evaluation is; Section 4.4 describes their assessment of the current state of the art of meta-evaluation and the main challenges which it faces; Section 4.5 presents the experts’ views of how one might evaluate the legacy of the 2012 Olympic and Paralympic Games; and Section 4.6 draws together the key points to emerge from the interviews. Interviewees The interviewees are acknowledged experts in the fields of evaluation and/or sports policy. The initial sample of potential interviews was identified from the literature review (discussed in section 3.0) and the authors' own knowledge of the field. Thereafter a ‘snowball’ method was used which involved asking early interviewees to suggest others who they believed would have useful insights into meta-evaluation approaches. A total of 18 experts were approached. Five declined to participate (some claimed not to know enough about meta-evaluation; one was unwilling to disclose details of the methods which they used). A total of 13 experts drawn from academia and consultancy firms and from across the US, UK and the rest of Europe were interviewed (see Annex 1). All 13 had direct experience of meta-evaluation research (broadly defined) or related activities such as meta-analysis. Interviews were conducted using a topic guide which was adopted by all interviewers (see Annex 2). Results were recorded in contemporaneous notes taken by interviewees and analysed using a standard matrix. Definitions Meta-evaluation in theory The literature review undertaken as part of the methods development study identified three main schools of thought about what constitutes meta-evaluation. Some researchers and commentators see meta-evaluation as being concerned primarily with standard setting. Seen in this light meta-evaluation is a process of establishing criteria for the evaluation of evaluations. The purpose of a meta-evaluation is to examine other studies against an established set of standards and goals in order to determine whether they were conducted in a rigorous and robust fashion.
Interim_Meta-Methods_Report_Nov_2012-1.docx