question about tokenizer changes original sequence length
Hi, I am working on an entity classification task where I know the entity boundary and the context.
when I use tokenizer to encode the entire sequence, some token got split up to word pieces that will change the original length of the sequence.  I want to extract the states only for the entity, but since the seq length changed, does that mean I need to recalcuate the boundary? Is there a way to automatically do this? Thanks!