Hello facehugger2020,
Thanks so much for your thorough explanation. But I’d like to ask one follow up question.
As another user posted on AllenNLP github issues, saying that huggingface transformer uses pad_token_label_id to solve problem of mis-matched subtokens, in which the first subtoken will be labeled to the original true label, then the following subtokens are labeled as pad_token_label_id. The model will ignore all these padding labels in training.
I am not sure if the features is on by default, if so, does that mean we don’t need to apply encode_tags and follow-up re-labeling process anymore?
Please find the referred issue 20 here.
Thank again for any clarification.