Our analysis concerns how estimates would change if observations Violations of SUTVA. were replaced with hypothetical or counterfactual cases. But estimates could also change if observed outcomes changed when were altered. For example, the others’ assignments achievement of the students who were actually retained could change if some promoted students were retained. This could occur if one retention reduced the stigma of another, or child’s competed with the resources, such as the attention of the teacher (Shepard & Smith, 1989). Changes in observed outcomes as a result of to treatment conditions others’ assignments constitute a violation of SUTVA, an assumption made for most inferences. As such, there are some (recent) techniques to inform the implications of violations to SUTVA. For example, one can use agent based models to explore the dynamic implications of changes in treatment assignments (e.g., Maroulis et al., 2010). Error in measurement, especially of the outcomes, could bias estimates. Measurement Error. Generally, measurement error will lead to conservative inferences because it reduces precision of estimates. This might be preferred if one wanted to be cautious in implementing new programs. But measurement error could hide a substantively important negative effect (an example of a type II error, failure to reject the null when in fact the null hypothesis is false). Measurement error is not a large concern for the two focal examples in the article in which the outcomes were measured with high reliability. The reliability of the Terra Nova comprehensive reading test used by Borman et al was very high (.84 to .93 across grades 3-6 2001), and would likely be higher at the classroom level given fairly large sample sizes per classroom (see Brennan, 1995, for a discussion). Similarly, the test used by Hong and Raudenbush had a reliability of .95 (based on the variance of repeated estimates of overall ability --see ECLS-K user guide, section 3.1.6). Furthermore, both Borman et al., and
Violations of SUTVA. Our analysis concerns how estimates would change if observations were replaced with hypothetical or counterfactual cases. But estimates could also change if observed outcomes changed when others’ assignments were altered. For example, the achievement of the students who were actually retained could change if some promoted students were retained. This could occur if one child’s retention reduced the stigma of another, or competed with the resources, such as the attention of the teacher (Shepard & Smith, 1989). Changes in observed outcomes as a result of others’ assignments to treatment conditions constitute a violation of SUTVA, an assumption made for most inferences. As such, there are some (recent) techniques to inform the implications of violations to SUTVA. For example, one can use agent based models to explore the dynamic implications of changes in treatment assignments (e.g., Maroulis et al., 2010). Measurement Error. Error in measurement, especially of the outcomes, could bias estimates. Generally, measurement error will lead to conservative inferences because it reduces precision of estimates. This might be preferred if one wanted to be cautious in implementing new programs. But measurement error could hide a substantively important negative effect (an example of a type II error, failure to reject the null when in fact the null hypothesis is false). Measurement error is not a large concern for the two focal examples in the article in which the outcomes were measured with high reliability. The reliability of the Terra Nova comprehensive reading test used by Borman et al was very high (.84 to .93 across grades 3-6 2001), and would likely be higher at the classroom level given fairly large sample sizes per classroom (see Brennan, 1995, for a discussion). Similarly, the test used by Hong and Raudenbush had a reliability of .95 (based on the variance of repeated estimates of overall ability --see ECLS-K user guide, section 3.1.6). Furthermore, both Borman et al., and
What%20would%20it%20take%20to%20Change%20an%20Inference%20published.docx