The present article analyzes the changes that have been introduced in comparative international large-scale studies (specifically those organized by the OECD) as a consequence of the introduction of computer-assisted testing. We will review the various measurement instruments that have been administered by computer in the different cycles of these large-scale studies and we will discuss whether their introduction implied a real added value, notably by providing information on competencies that are inaccessible through classical paper and pencil tests. We will also analyze the specific relationship that the computer-assisted measures show with background variables, such as gender. Finally, the article will point out the challenges and risks associated with a more and more extensive introduction of computer-assisted measurement instruments, especially regarding two major aspects: the necessity to keep a certain stability in the measured competencies in view of reporting trends, and the lack of adequate theoretical frameworks for the efficient use of the computer-assisted measures.