How can I use features POS tags and chunk ids to train model when the input test sentence wont have them I have been given a task to train the SVM model on conll2003 dataset for Named Entity "Identification" (That is I have to tag all tokens in "Statue of Liberty" as named entities not as a place, which is the case in named entity recognition.) Conll2003 dataset contains part of speech tags and chunk IDs for each token. We used them to train the SVM model. We can also find models' performance against test and validation datasets as both of them also contain part of speech tags and chunk IDs for each token. But what if someone simply inputs some random sentence (without pos tags and chunk IDs) for predicting (as it is out of test dataset)? How should we handle this? Should we altogether avoid these features while training? Or "somehow" generate these features for input sentence before feeding it to the model for prediction? If yes, then how this generation is usually done? Also what is the standard approach?

When your prediction depends on annotations you don't have, there are three common approaches. You touched on two of them. Let's say that you're trying to model $p(y \mid x, z)$ where $x$ is the text and $z$ is the annotation (e.g. POS tags or chunk IDs).

*

*Use a weaker model that doesn't depend on the annotation: assume that $p(y \mid x, z) = p(y \mid x)$.

*Annotate the text $x$ with some additional model to give you $\hat{z}(x)$. Now assume that $p(y \mid x, z) = p(y \mid \hat{z}(x), x)$.

*Treat the annotation as a latent variable, then marginalize it out without the independence assumption from (1): $p(y \mid x) = \sum_z p(y, z \mid x)$.

Of course, the approach that everyone forgets is to annotate more data: have someone POS-tag and label chunks in your new sentences.
