Something occurred to me as I was trying to apply my trained models on new datasets, when I tokenised the new dataset, some of the words were obviously assigned a different numeric value than the same word in the original dataset that was used for the training/validation. How can the model make predictions when the datasets have different vocabularies and thus different numeric values for the same words? I am guessing it has something to do with models learning embedding layers, is that correct? Does this apply to non-deep learning models like the ones in sklearn (Forests, SVMs, NB, etc)? I’m just trying to understand the intuition for how it works. Thanks!