How to avoid selection bias while updating lead scoring (predictive) model with new data We developed a standard lead scoring model using logistic regression on couple of months worth data. The model has been working and we have been pushing only top 1/3 leads to sales team basis that. The model is giving around 40% lift.
This model is already 2.5 months old and we are planning to update / retrain the model after adding new data along with conversion results.
I am concerned that since we were only pushing qualified leads to sales team, we do not have the conversion result for low quality leads and hence they will have to be excluded from the model but this in turn would mean that the model will get trained on a dataset which is systematically different from ground reality - how to fix this?

You could combine the models.
Let the refined/upgraded/updated model only make decisions for the type of data for which it has been trained.

*

*Let the old model still select the top 1/3 (because the new model doesn't now much better about the bottom 2/3)

*Let the new model remove some from that 1/3. (Which it should do well and without selection bias)

*Then have the old model add some extra for the amount that the new model removed.

And you can keep updating the model that works on the top 1/3.
To update the model that looks at all cases and selects the top 1/3 you need to make sure that it can train on more data. You do not have to add so much for this. You want anyway the model to fit the top very well and not the bottom. The model isn't gonna become extremely better in the bottom and suddenly place low cases in the top 1/3.
