subsequent administrations of the SFS, there were 4 response categories; as a result was equal m to 3 in the model. The mean difficulties of the step thresholds for each item are allowed to vary in the PCM. By default, in WINSTEPS the item mean summed across the thresholds equals zero; the person and item measures are generated and reported on the logit scale. In the context of SFS, a student with a positive logit value is relatively more affirmative in their responses than a student with a negative logit value, i.e., a student with a positive logit value perceives their teacher to be relatively more effective than a student with a negative logit value. Using the output from WINSTEPS, evidence was built to support the content, substantive and generalizability aspects of validity. 3.2 Quality of Linking Process in Pilot Administration #1 The results shown in this section for Pilot Administration #1 only reflect responses using response option Condition 2 (strongly agree to strongly disagree). Psychometrically, student responses using this rating scale were relatively more reliable when compared to responses from Condition 1 (in no lessons to nearly all lessons). Figure 2 (p. 28) presented the process used to link the pilot forms in Pilot Administration #1 on to the same scale metric using sixteen common items. Table 17 shows the Pearson correlation between forms of the common item deltas for and G3-G5, respectively. These forms were initially calibrated separately to assess the invariance of the common items. The correlations for the G6-G12 pilot were all above 0.90 justifying the use of the concurrent calibration process to place all items from the four forms onto the same scale metric. Similarly, although lower than those of the G6-G12 pilot, the correlations for the G3-G5 are sufficiently robust (≥ 0.85) to justify using a concurrent calibration to place the four forms onto the same scale metric. In this manner, it is possible to assess the ordering and spacing of all items on the item-variable map to determine if the items are appropriately targeted for the person distribution and measuring the teacher effectiveness construct well. Table 17: Pearson Correlations of Common Items in Pilot Administration #11 Based on 16 Common Items Form A2 Form B2 Form C2 Form D2 G6-G12 and G3-G5 Form A2 0.86 0.85 0.90
subsequent administrations of the SFS, there were 4 response categories; as a result m was equal to 3 in the model. The mean difficulties of the step thresholds for each item are allowed to vary in the PCM. By default, in WINSTEPS the item mean summed across the thresholds equals zero; the person and item measures are generated and reported on the logit scale. In the context of SFS, a student with a positive logit value is relatively more affirmative in their responses than a student with a negative logit value, i.e., a student with a positive logit value perceives their teacher to be relatively more effective than a student with a negative logit value. Using the output from WINSTEPS, evidence was built to support the content, substantive and generalizability aspects of validity. 3.2 Quality of Linking Process in Pilot Administration #1 The results shown in this section for Pilot Administration #1 only reflect responses using response option Condition 2 (strongly agree to strongly disagree). Psychometrically, student responses using this rating scale were relatively more reliable when compared to responses from Condition 1 (in no lessons to nearly all lessons). Figure 2 (p. 28) presented the process used to link the pilot forms in Pilot Administration #1 on to the same scale metric using sixteen common items. Table 17 shows the Pearson correlation between forms of the common item deltas for and G3-G5, respectively. These forms were initially calibrated separately to assess the invariance of the common items. The correlations for the G6-G12 pilot were all above 0.90 justifying the use of the concurrent calibration process to place all items from the four forms onto the same scale metric. Similarly, although lower than those of the G6-G12 pilot, the correlations for the G3-G5 are sufficiently robust (≥ 0.85) to justify using a concurrent calibration to place the four forms onto the same scale metric. In this manner, it is possible to assess the ordering and spacing of all items on the item-variable map to determine if the items are appropriately targeted for the person distribution and measuring the teacher effectiveness construct well. Table 17: Pearson Correlations of Common Items in Pilot Administration #11 Based on 16 Common Items G6-G12 and G3-G5 Form A2 Form B2 Form C2 Form D2 Form A2 0.86 0.85 0.90
2014SFS-TechReport.docx