This paper focuses on an application of generalizability theory in evaluating and optimizing the design of pupil attainment surveys. It describes the way in which analysis of response data from one of the mathematics surveys conducted within Scotland’s Assessment of Achievement Programme informed decisions about the design of future surveys in this subject.