Even if many researches deal with validity, especially in psychometrics, this notion hasn’t been significantly discussed in the field of didactics of mathematics. This paper examines how didactics tools, especially a priori analysis of tasks, can be used to bring validity evidence in addition to those provided by psychometrics. A first part of the paper aims to expose a methodology, using a combination of didactic and psychometric approaches, to analyze and design external assessment tests. In a second part, we implement this methodology to study two national large scale assessments, in two mathematical domains at two levels of school (arithmetic at the end of primary school and algebra at the end of grade 9), in France.