Can you correct "bias" in a regression if you can measure/model it? A journey in missing data and reweighting test scores Thank you for joining me on this semi-theoretical journey. Here we will discuss how to account for "predictable" bias in your data.
Let's say we have a test composed on many subtests. A composite score is generated through successive aggregation and weighting of sub-tests.
Let's talk about sub-test A.
You can get a total score of 100 on sub-test A, and each item on the test has a different possible max score. Now you're missing (at-random) some answer for sub-test, rather than try to impute these scores, you can drop the missing items from the subtest and rescale.
How are we rescaling? Let's say you drop q10 and the MAX score you can get on this sub-test in 90. Multiply 90 by the NEW percent correct on this test to scale it back up. Now recalculate your composite score the same way as usual.
Now calculate the difference between your newly calculated/rescaled score and the original score in a sample of subjects who have COMPLETE sub-test sections and figure out how your data will be biased.
Here we are, when you use the method outlined above, we will on average skew the data by ~0.5 points if we drop the missing sub-items. Since I can measure this bias, would it be reasonable to then add back this bias as a constant in a linear regression?


It sounds like your approach assumes that the missing values are missing completely at random (MCAR). That's a very strong assumption, one that wouldn't hold if the probability of missingness to one answer is a function of the responses to other answers on the tests.
Multiple imputation allows you to analyze data efficiently under a much weaker assumption of missing at random (MAR). MAR holds if the probability of missingness is related to data that you have but not to data that you don't have.
See Stef van Buuren's online book for extensive discussion. Chapter 1 of the book explains the different types of missingness and the disadvantages of other approaches, including those that assume MCAR.
