Aaron Fisher|G. Brooke Anderson|Roger D. Peng|Jeffrey T. Leek
A randomized trial in a massive online open course shows people don’t know what a statistically significant relationship looks like, but they can learn
Johns Hopkins University|Colorado State University|Johns Hopkins University|Johns Hopkins University
scatterplots common way statisticians scientists public visually detect relationships measured variables time despite widely publicized controversy pvalues remain commonly used measure statistically justify relationships identified variables measure ability detect statistically significant relationships scatterplots randomized trial 2039 students statistics massive open online course mooc subject shown random set scatterplots asked visually determine underlying relationships statistically significant p 005 level subjects correctly classified 474 95 ci 451497 statistically significant relationships 746 95 ci 725766 nonsignificant relationships adding visual aids best fit line scatterplot smooth increased probability relationship called significant regardless whether relationship actually significant classification statistically significant relationships improved repeat attempts survey although classification nonsignificant relationships results suggest 1 evidencebased data analysis used identify weaknesses theoretical procedures hands average users 2 data analysts trained improve detection statistically significant results practice 3 data analysts incorrect intuition statistically significant relationships look like particularly small effects built web tool people compare scatterplots corresponding pvalues available httpglimmerrstudiocomafishereda