I actually wasn’t even talking about knowledge destillation but more general about a training set containing senteces where certain entitties are annotated and other’s where they are not. It seems obvious to me, that this woud hurt performance but I guess I wanted to make sure.
I’ve decided that I need to train from scratch and I’ll be trying to annotate the original training set, with the additional entitites I need. Many of them I can cover with regex, so this shouldn’t be a problem.
Since I could cover them with regex, why even put them into the model? I’d like to have one NER model that annotates everything, instead of having to go through multiple steps of annotation. Does that make sense?
I appreciate all your help!