Students’ evaluations of teaching effectiveness are used for academic and formative purposes, but are also sometimes evoked, in a summative way, for some faculty career decisions. In the present study, two main objectives are pursued. The first objective is to determine the psychometric properties of the instrument in use at the Université du Québec à Rimouski. With the advent of information and communication technologies, the conditions in which the evaluations take place have been changed from paper to online processes. so, the second objective is to verify the impacts of these online vs paper processes on the psychometric properties of the instrument. The data originated from the evaluations gathered during the 2007-2008 (20,245 evaluations) and 2010-2011 (16,432 evaluations) academic years. The results attest that a hierarchical model better describes the data than alternative models, that online vs paper processes have an influence on response rates and, in some cases, on the satisfaction level with teaching, but they have no impact on the fiability and the dimensionnality of the evaluations. The implications of these results are discussed.