the design and how conclusions from the research will be tempered in light of these threats. Because quasi-experimental designs can meet the WWC’s standards for evidence with reservations only, it is also important to detail how you will ensure that the study will meet these standards (e.g., by establishing baseline equivalence between treatment and comparison groups and preventing high and/or attrition). Describe and justify the counterfactual. In evaluations of education interventions, individuals in the comparison group typically receive some kind of treatment. It may be a well-defined alternative treatment or a less well-defined standard or frequent practice across the district or region. A clear description of the intervention and the counterfactual helps reviewers decide whether the intervention is sufficiently different from what the comparison group receives to produce different student education outcomes. Describe strategies or existing conditions that will reduce potential contamination between treatment and comparison groups. Discuss how your study, if well implemented, will meet WWC evidence standards (with or without reservations). Power Analysis: Discuss the statistical power of the research design to detect a reasonably expected and minimally important effect of the intervention on the focal student education outcomes and consider how the clustering of participants (e.g., students in classrooms and/or schools) will affect statistical power. Identify the minimum effect of the program or policy that you will be able to detect, justify why this level of effect would be expected, and explain why this would be a practically important effect. Detail the procedure used to calculate either the power for detecting the minimum effect or the minimum detectable effect size. Include the following: The statistical formula you used; The parameters with known values used in the formula (e.g., number of clusters, number of participants within the clusters); The parameters whose values are estimated and how those estimates were made (e.g., intraclass correlations, role of covariates); Other aspects of the design and how they may affect power (e.g., stratified sampling/blocking, repeated observations); and Predicted attrition and how it was addressed in the analysis. Provide a similar discussion regarding power for any causal analyses to be done using subgroups of the proposed sample and any tests of mediation or moderation, even if those analyses are considered exploratory/secondary.
the design and how conclusions from the research will be tempered in light of these threats. Because quasi-experimental designs can meet the WWC’s standards for evidence with reservations only, it is also important to detail how you will ensure that the study will meet these standards (e.g., by establishing baseline equivalence between treatment and comparison groups and preventing high and/or attrition). Describe and justify the counterfactual. In evaluations of education interventions, individuals in the comparison group typically receive some kind of treatment. It may be a well-defined alternative treatment or a less well-defined standard or frequent practice across the district or region. A clear description of the intervention and the counterfactual helps reviewers decide whether the intervention is sufficiently different from what the comparison group receives to produce different student education outcomes. Describe strategies or existing conditions that will reduce potential contamination between treatment and comparison groups. Discuss how your study, if well implemented, will meet WWC evidence standards (with or without reservations). Power Analysis: Discuss the statistical power of the research design to detect a reasonably expected and minimally important effect of the intervention on the focal student education outcomes and consider how the clustering of participants (e.g., students in classrooms and/or schools) will affect statistical power. Identify the minimum effect of the program or policy that you will be able to detect, justify why this level of effect would be expected, and explain why this would be a practically important effect. Detail the procedure used to calculate either the power for detecting the minimum effect or the minimum detectable effect size. Include the following: The statistical formula you used; The parameters with known values used in the formula (e.g., number of clusters, number of participants within the clusters); The parameters whose values are estimated and how those estimates were made (e.g., intraclass correlations, role of covariates); Other aspects of the design and how they may affect power (e.g., stratified sampling/blocking, repeated observations); and Predicted attrition and how it was addressed in the analysis. Provide a similar discussion regarding power for any causal analyses to be done using subgroups of the proposed sample and any tests of mediation or moderation, even if those analyses are considered exploratory/secondary.
2018_84305A.docx