EPSY 5221: Principles of Educational & Psychological Measurement Interpreting Model Fit in IRT (the Rasch Model) Model-data fit is always an important aspect of the data to evaluate whenever doing statistical analysis. This essentially amounts to checking the assumptions of the model. In IRT, there are a number of ways to evaluate model-data fit. We are focusing on two. Item Fit First is the item fit statistic. We do this with a weighted mean squares statistic generated by the model (WMS). If the data fit the model for items, then the data are consistent as the model predicts. Model Prediction: The model predicts the probability of correct response to the item based on person ability. For an item, we expect people with abilities to have lower probabilities of correct responses and people with higher abilities to have higher probabilities of correct responses. If we look at groups of people with various levels of ability across the measure, then smaller proportions with lower scores get the item correct and larger proportions with higher scores get the item correct. If this is what we observe in the data, then "the item fits the model." If an item produces responses where lower-ability people have a higher than predicted probability of correctly responding and/or people with higher abilities have a lower than predicted probability of correctly responding, then the item doesn't fit the model. When this occurs, the WMS value increases (bigger differences between what we observe in the data and what the model predicts). When WMS values go beyond 1.5, we are concerned about the quality of the item. If the item doesn't fit the model, there is a good chance that the item isn't measuring the same thing as the overall test - or that the item has serious flaws. Some high-stakes testing programs use more conservative criteria for making fit decisions, such as when WMS values are larger than 1.2 or 1.3, the item is reviewed for misfit. For item 54 above, the WMS value is 1.17, not bad, just a little over 1.0 which is ideal. We can see that the observed proportion of correct response (y-axis) does not map exactly to the model-predicted curve. At some points around abilities (particularly -2.0 to -1.0), a larger proportion of people answer the item correctly than the model predicts and in areas of moderate ability (particularly –
EPSY 5221: Principles of Educational & Psychological Measurement Interpreting Model Fit in IRT (the Rasch Model) Model-data fit is always an important aspect of the data to evaluate whenever doing statistical analysis. This essentially amounts to checking the assumptions of the model. In IRT, there are a number of ways to evaluate model-data fit. We are focusing on two. Item Fit First is the item fit statistic. We do this with a weighted mean squares statistic generated by the model (WMS). If the data fit the model for items, then the data are consistent as the model predicts. Model Prediction: The model predicts the probability of correct response to the item based on person ability. For an item, we expect people with abilities to have lower probabilities of correct responses and people with higher abilities to have higher probabilities of correct responses. If we look at groups of people with various levels of ability across the measure, then smaller proportions with lower scores get the item correct and larger proportions with higher scores get the item correct. If this is what we observe in the data, then "the item fits the model." If an item produces responses where lower-ability people have a higher than predicted probability of correctly responding and/or people with higher abilities have a lower than predicted probability of correctly responding, then the item doesn't fit the model. When this occurs, the WMS value increases (bigger differences between what we observe in the data and what the model predicts). When WMS values go beyond 1.5, we are concerned about the quality of the item. If the item doesn't fit the model, there is a good chance that the item isn't measuring the same thing as the overall test - or that the item has serious flaws. Some high-stakes testing programs use more conservative criteria for making fit decisions, such as when WMS values are larger than 1.2 or 1.3, the item is reviewed for misfit. For item 54 above, the WMS value is 1.17, not bad, just a little over 1.0 which is ideal. We can see that the observed proportion of correct response (y-axis) does not map exactly to the model-predicted curve. At some points around abilities (particularly -2.0 to -1.0), a larger proportion of people answer the item correctly than the model predicts – and in areas of moderate ability (particularly
Intepreting-fit.docx