This study examines the possibility of assessing with a good level of reliability students’ ability to compute a sequence of operations using the correct order of operations. Based on a research design, two factors linked to the problem statement were manipulated: degree of realism and data completeness. Following a pretest, 12 problems were administered to 82 first grade secondary students from the province of Quebec. In order to assess the edumetric qualities of the evaluation design, the data were analyzed using generalizability theory. The generalizability coefficients were lower than the limit considered to be satisfactory, thus indicating that the evaluation design is not reliable enough to allow the differentiation of students’ ability in computing a sequence of operations. The results demonstrate the difficulty to assess with a good level of reliability students’ ability when two factors linked to the problem statement are manipulated: degree of realism and data completeness.