title: Dealing with missing levels in categorical variables when batch scoring in Python
body: <p>I have encountered a unique problem. My model was trained on DNN framework and the model parameters are saved which I'm using now to score the data. Since my data is quite huge, I'm scoring the data in batches. I'm not hot encoding categorical variables before creating batches as the Onehotencode function runs into memory error when I apply on complete dataset. This led me to explore the option of hot encoding in batches, however this fails as all batches does not contain all levels of values in categorical variables. If anyone has faced a similar issue could you please recommend or suggest a workaround?</p>