Hi @lewtun,
Thank you for your reply, it’s so helpful.
I am not sure that I get the trick, why not just setting for the tags [3, 3, 3] for [’@’, ‘hugging’, ‘##face’] if @HuggingFace for example is the labels B-corporation with index 3 ?
Is it necessary to set the -100 as a value for labels to ignore or just use any other value, because for now I am using 0 as an index to tell the model to ignore this label ?
For the problem described in this discussion is something similar to this,
  
    
    
    Converting Word-level labels to WordPiece-level for Token Classification Intermediate
  
  
    Hi all, 
I am building a BertForTokenClassification model but I am having trouble figuring out how to format my dataset. I have already labeled my dataset with span labeling. So for example: 
sequence = “Hugging Face Inc. is a company based in New York City. Its headquarters are in DUMBO, therefore very close to the Manhattan Bridge.” 
In my dataset, I would have this labeled by hand as: 
[(Hugging, B-org), (Face, I-org), (Inc., L-org), (is, O), (a, O), (company, O), … (Bridge, L-org)] 
However,…
  


but in the reverse from WordPiece level to word-level labels.