Appendix E: Evaluation Framework Evaluation Question Sub-Questions Measures Data source or method Appropriateness 1. To what extent has the 1a. Have the needs identified during the original development analyses. Document review Department documents – program design proven to be of the NTCRS changed over time? Regulatory Impact Statements adequately matched to identified Possible source: Co-regulators may have consumer needs? 1b. What evidence is there that the NTCRS program design Synthesis of evaluation research findings. behaviour surveys/studies. ACIG to check if meets or is likely to meet identified needs? are willing to share or if DoEE can provide. Efficiency 2. How efficiently has the NTCRS 2a. To what extent were available resources used to the best Summative assessment of evaluation findings Evaluation synthesis of costs versus outputs achieved been managed and effect? concerning program delivery. implemented? 3. How efficient has the NTCRS 3a. To what extent are the costs of program delivery Program costs and benefits. Note: the evaluation will 2014-15 Operational Review of the NTCRS been in delivering its intended justifiable against its results? be limited to whatever cost and benefit data is Departmental budget and administrative data outcomes? currently available from the Department. Regulatory Impact Statements Evaluation synthesis of evidence on and benefits 3b. To what degree is the scheme cost-effective compared to Comparison of program costs and benefits with other like programs or other options that address the same programs (to the extent that they are available). needs? Effectiveness and Impact 4. Were communication and 4a. How were the target audiences for education and Degree of reach to audience; website analytics trends; Department documents and interviews education activities adequate? information determined and to what extent were they communications materials created and disseminated. Stakeholder survey reached? Awareness of program objectives, penalties, liabilities Stakeholder interviews (obligations under the Act) by liable parties and Stakeholder survey 5. Were the activities and their 5a. Were the activities that were delivered adequate in Program management’s assessment of program Department documents and interviews resourcing adequate to ensure achieving the planned outputs? implementation. delivery of the planned outputs? 5b. Were the resources provided adequate to provide all the Program management’s assessment of program Department documents and interviews planned outputs? implementation.
Appendix E: Evaluation Framework Evaluation Question Sub-Questions Measures Data source or method Appropriateness 1. To what extent has the program design proven to be adequately matched to identified needs? 1a. Have the needs identified during the original development of the NTCRS changed over time? 1b. What evidence is there that the NTCRS program design meets or is likely to meet identified needs? analyses. Synthesis of evaluation research findings. Document review – Department documents Regulatory Impact Statements Possible source: Co-regulators may have consumer behaviour surveys/studies. ACIG to check if are willing to share or if DoEE can provide. Efficiency 2. How efficiently has the NTCRS been managed and implemented? 2a. To what extent were available resources used to the best effect? Summative assessment of evaluation findings concerning program delivery. Evaluation synthesis of costs versus outputs achieved 3. How efficient has the NTCRS been in delivering its intended outcomes? 3a. To what extent are the costs of program delivery justifiable against its results? 3b. To what degree is the scheme cost-effective compared to like programs or other options that address the same needs? Program costs and benefits. Note: the evaluation will be limited to whatever cost and benefit data is currently available from the Department. Comparison of program costs and benefits with other programs (to the extent that they are available). 2014-15 Operational Review of the NTCRS Departmental budget and administrative data Regulatory Impact Statements Evaluation synthesis of evidence on and benefits Effectiveness and Impact 4. Were communication and education activities adequate? 4a. How were the target audiences for education and information determined and to what extent were they reached? Degree of reach to audience; website analytics trends; communications materials created and disseminated. Awareness of program objectives, penalties, liabilities (obligations under the Act) by liable parties and Department documents and interviews Stakeholder survey Stakeholder interviews Stakeholder survey 5. Were the activities and their resourcing adequate to ensure delivery of the planned outputs? 5a. Were the activities that were delivered adequate in achieving the planned outputs? 5b. Were the resources provided adequate to provide all the planned outputs? Program management’s assessment of program implementation. Program management’s assessment of program implementation. Department documents and interviews Department documents and interviews
ntcrs-evaluation-report.docx