Bonnie Wintle|Eden T. Smith|Martin Bush|Fallon Mody|David Wilkinson|Anca M. Hanea|Alexandru Marcoci|Hannah Fraser|Victoria Hemming|Felix Singleton Thorn|Marissa McBride|Elliot Gould|Andrew Head|Daniel G. Hamilton|Steven Kambouris|Libby Rumpff|Rink Hoekstra|Mark A. Burgman|Fiona Fidler
Predicting and reasoning about replicability using structured groups
University of Melbourne|University of Melbourne|University of Melbourne|University of Melbourne|University of Melbourne|University of Melbourne|University of Cambridge|University of Melbourne|University of British Columbia|University of Melbourne|Imperial College London|University of Melbourne|University of Melbourne|University of Melbourne|University of Melbourne|University of Melbourne|University of Melbourne|University of Groningen|Imperial College London|University of Melbourne
paper explores judgements replicability social behavioural sciences research drives judgements using mixed methods approach draws qualitative quantitative data elicited groups using structured approach called idea protocol investigate discuss estimate aggregate five groups five people relevant domain expertise evaluated 25 research claims subject least one replication study participants assessed probability 25 research claims would replicate ie replication study would find statistically significant result direction original study described reasoning behind judgements quantitatively analysed possible correlates predictive accuracy including selfrated expertise updating judgements feedback discussion qualitatively analysed reasoning data explore cues heuristics patterns reasoning used participants participants achieved 84 classification accuracy predicting replicability engaged greater breadth reasoning provided accurate replicability judgements reasons commonly invoked accurate participants effect size reputation eg field research also evidence relationship statistical literacy accuracy