Anthea Senior, Colleen Starchuk, Gisele Gaudet-Amigo, Jacqueline Green, Steven Patterson, Arnaldo Perez
A novel model for curriculum design: Preparation, planning, prototyping, and piloting
European journal of dental education : official journal of the Association for Dental Education in Europe
high stakes examinations used credential trainees independent specialist practice evaluated periodically ensure defensible decisions made study aims quantify college intensive care medicine australia new zealand cicm hot case reliability coefficient evaluate contributions variance candidates cases examiners retrospective deidentified analysis cicm examination data used descriptive statistics generalisability theory evaluate reliability hot case examination component decision studies used project generalisability coefficients alternate examination designs examination results 2019 2022 included 592 hot cases totalling 1184 individual examiner scores mean examiner hot case score 517 standard deviation 165 correlation candidates two hot case scores low 030 overall reliability coefficient hot case component consisting two cases observed two separate pairs examiners 042 sources variance included candidate proficiency 25 case difficulty case specificity 634 examiner stringency 35 error 82 achieve reliability coefficient 08 candidate would need perform 11 hot cases observed two examiners reliability coefficient hot case component cicm second part examination generally accepted value high stakes examination modifications case selection introduction clear scoring rubric mitigate effects variation case difficulty may helpful increasing number cases overall assessment time appears best way increase overall reliability research required assess combined reliability hot case viva components