Table 2 Quantifying the Robustness of Inferences from Observational Studies Study Predictor of Condition Population Outcome Estimated effect, Effect size % bias to make (author, year) interest on pretest standard error, (correlation) inference invalid source Effects of kindergarten Kindergarten Multiple 7639 ECLS-K 9 .67 85% retention policy on retention versus kindergarteners Reading (.68) (.14) cognitive promotion in 1080 IRT scale Table 11, model children’s growth in reading and retention score based estimate mathematics schools in (Hong & Raudenbush, ECLS-K 2005) Counterfactuals, causal Catholic versus Single 10835 high NELS .99 .23 34% effect heterogeneity, and public school school students Math IRT (.33) (.10) the Catholic school effect nested within scale Table 1, (model on learning, 973 schools in score with pretest + (Morgan, 2001) NELS family background) Effects of Content Gain score 1773 third Terra 2.28 NA 36% teachers’ mathematical knowledge knowledge for graders nested Nova (.75) (.16) for teaching on student teacher within 365 math scale Table 7, model 1, achievement mathematics teachers score (third graders) (Hill et al., 2005)
Table 2 Quantifying the Robustness of Inferences from Observational Studies Study (author, year) Predictor of interest Condition on pretest Population Outcome Estimated effect, standard error, source Effect size (correlation) % bias to make inference invalid Effects of kindergarten retention policy on children’s cognitive growth in reading and mathematics (Hong & Raudenbush, 2005) Kindergarten retention versus promotion Multiple 7639 kindergarteners in 1080 retention schools in ECLS-K ECLS-K Reading IRT scale score 9 (.68) Table 11, model based estimate .67 (.14) 85% Counterfactuals, causal effect heterogeneity, and the Catholic school effect on learning, (Morgan, 2001) Catholic versus public school Single 10835 high school students nested within 973 schools in NELS NELS Math IRT scale score .99 (.33) Table 1, (model with pretest + family background) .23 (.10) 34% Effects of teachers’ mathematical knowledge for teaching on student achievement (Hill et al., 2005) Content knowledge for teacher mathematics Gain score 1773 third graders nested within 365 teachers Terra Nova math scale score 2.28 (.75) Table 7, model 1, (third graders) NA (.16) 36%
What%20would%20it%20take%20to%20Change%20an%20Inference%20published.docx