The design of the field trial was led by ETS. Each participating country was asked to recruit pupils and schools using one of the following designs: Design A = Recruit 25 schools and 78 pupils within each Design B = Recruit 39 schools and 52 pupils within each Design C = Recruit 54 schools and 36 pupils within each There was no requirement for these schools to be randomly (or probabilistically) sampled. With respect to our countries, Ireland followed design A, Germany design C and Sweden design C. However, due to pupil exemptions (e.g. for pupils with special educational needs who would not be able to complete the test without assistance and pupils with limited language experience), logistical restrictions and challenges with recruitment, these criteria were not always met. Final achieved sample sizes were therefore: Germany = 62 schools and 2,341 pupils Ireland = 25 schools and 1,503 pupils Sweden = 54 schools and 2,141 pupils All pupils who completed the field trial were randomly assigned to one of three groups: Group 1: Paper-based Assessment (PBA) of trend PISA items (23 percent) 2: Computer-based Assessment (CBA) of trend PISA items (35 percent) Group 3: CBA of new PISA science items (42 percent) Our focus is groups 1 and 2. These pupils sat tests consisting of only PISA identical ‘trend’ items (questions that have been used in previous PISA cycles and are the basis for linking the PISA test scale over time) with the only difference being assessment mode. This leaves the following working samples: Germany = 1,240 pupils (517 paper based and 723 computer based) Ireland = 966 pupils (382 paper based and 583 computer based) Sweden = 1,232 pupils (515 paper based and 717 computer based) Pupils within the paper and computer groups were then randomly assigned one of 18 each containing four different clusters of test questions. Each cluster contained ‘booklets’, test questions from only one subject area. Across the three countries combined, 1,149 pupils completed only science and maths questions (those assigned booklets 1 to 6), 1,156 pupils
The design of the field trial was led by ETS. Each participating country was asked to recruit pupils and schools using one of the following designs: Design A = Recruit 25 schools and 78 pupils within each Design B = Recruit 39 schools and 52 pupils within each Design C = Recruit 54 schools and 36 pupils within each There was no requirement for these schools to be randomly (or probabilistically) sampled. With respect to our countries, Ireland followed design A, Germany design C and Sweden design C. However, due to pupil exemptions (e.g. for pupils with special educational needs who would not be able to complete the test without assistance and pupils with limited language experience), logistical restrictions and challenges with recruitment, these criteria were not always met. Final achieved sample sizes were therefore: Germany = 62 schools and 2,341 pupils Ireland = 25 schools and 1,503 pupils Sweden = 54 schools and 2,141 pupils All pupils who completed the field trial were randomly assigned to one of three groups: Group 1: Paper-based Assessment (PBA) of trend PISA items (23 percent) 2: Computer-based Assessment (CBA) of trend PISA items (35 percent) Group 3: CBA of new PISA science items (42 percent) Our focus is groups 1 and 2. These pupils sat identical tests consisting of only PISA ‘trend’ items (questions that have been used in previous PISA cycles and are the basis for linking the PISA test scale over time) with the only difference being assessment mode. This leaves the following working samples: Germany = 1,240 pupils (517 paper based and 723 computer based) Ireland = 966 pupils (382 paper based and 583 computer based) Sweden = 1,232 pupils (515 paper based and 717 computer based) Pupils within the paper and computer groups were then randomly assigned one of 18 ‘booklets’, each containing four different clusters of test questions. Each cluster contained test questions from only one subject area. Across the three countries combined, 1,149 pupils completed only science and maths questions (those assigned booklets 1 to 6), 1,156 pupils
revised_main_body_27_11_2017_clean.docx