Qualitative data is more open to differing interpretations, and therefore offers a greater risk of inaccurate interpretation, or of the evaluation being perceived to be subjective or biased. It also means that qualitative data can fail one of the key tests for a good success measure that it should be reliably and consistently interpreted – by different parties. The Evaluation evaluation (Patton, 2003) further notes Center’s ‘Qualitative checklist’ that the of qualitative data depends to a great extent on the methodological ‘quality skill, sensitivity, and integrity of the evaluator ... Generating useful and credible qualitative findings through observation, interviewing, and content analysis requires discipline, knowledge, training, practice, creativity, and hard In this respect, work.’ the data produced through qualitative methods may be regarded as potentially persuasive and useful in influencing others, particularly at reporting stage, but they present substantial challenges in producing data that is able to provide a robust evaluation of change from a baseline for your success measure. Principle 6: We know how we will analyse the data we collect to produce conclusions Another area where self-evaluation processes frequently stall is in the analysis of the data collected. Organisations can quite competently design interventions, set outcomes and success measures, and use appropriate methods to collect the data, and then do very little with it. Analysis can take time, but the key to efficient and effective data analysis is making the process achieved by only looking at data that allows you to manageable. That’s form conclusions. not uncommon to invest time in gathering data you or It’s can’t use in your evaluation. won’t To prevent this, you can ask yourself a few simple questions: will I do with this If the answer is or ‘What information?’ ‘nothing’ ‘I don’t know’, you may be better not collecting it. will I analyse the data I If the answer is or great ‘How collect?’ ‘I don’t know’ ‘with then you are either collecting data that meet the relevance test difficulty’, doesn’t for success measures, or data that is qualitative in nature. Data that allows you to form conclusions about the change from your baseline is what you need for evaluation; that means data that allows you to measure difference, and that means quantitative data.
Qualitative data is more open to differing interpretations, and therefore offers a greater risk of inaccurate interpretation, or of the evaluation being perceived to be subjective or biased. It also means that qualitative data can fail one of the key tests for a good success measure – that it should be reliably and consistently interpreted by different parties. The Evaluation Center’s ‘Qualitative evaluation checklist’ (Patton, 2003) further notes that the ‘quality of qualitative data depends to a great extent on the methodological skill, sensitivity, and integrity of the evaluator ... Generating useful and credible qualitative findings through observation, interviewing, and content analysis requires discipline, knowledge, training, practice, creativity, and hard work.’ In this respect, the data produced through qualitative methods may be regarded as potentially persuasive and useful in influencing others, particularly at reporting stage, but they present substantial challenges in producing data that is able to provide a robust evaluation of change from a baseline for your success measure. Principle 6: We know how we will analyse the data we collect to produce conclusions Another area where self-evaluation processes frequently stall is in the analysis of the data collected. Organisations can quite competently design interventions, set outcomes and success measures, and use appropriate methods to collect the data, and then do very little with it. Analysis can take time, but the key to efficient and effective data analysis is making the process manageable. That’s achieved by only looking at data that allows you to form conclusions. It’s not uncommon to invest time in gathering data you can’t or won’t use in your evaluation. To prevent this, you can ask yourself a few simple questions: ‘What will I do with this information?’ If the answer is ‘nothing’ or ‘I don’t know’, you may be better not collecting it. ‘How will I analyse the data I collect?’ If the answer is ‘I don’t know’ or ‘with great difficulty’, then you are either collecting data that doesn’t meet the relevance test for success measures, or data that is qualitative in nature. Data that allows you to form conclusions about the change from your baseline is what you need for evaluation; that means data that allows you to measure difference, and that means quantitative data.
prejudice-unlawful-behaviour-guide-to-evaluation.docx