Impute Nulls in Train but remove Nulls in Test I have a continuous variable, call it A, with missing data. In my model, I have created a dummy variable, call it B, where 1 indicates missing in A and 0 otherwise. I impute missing values in A with the mean and include both variables A and B in the model. The test set also has missing values for A. My question: does it make sense to remove the missing values in the test set?
Update: This is regression problem. The response variable follows a gamma distribution.

What would you do at prediction time? Say your model is deployed and it gets data with missing values (is it possible?), what do you do? Can you fall back to something else instead of making predictions? If you need to make a prediction, then you would either need to fill in the missing values or have a model using only the features that are guaranteed to have no missing values--that's the model you need to test. Your testing strategy should reflect how you are going to actually use your model. Your testing data should be similar to the actual data the model would see in the wild. Otherwise, you risk that the performance you measure is not the one you care about.
