How do you get token encodings? The method isn’t working in this case.
I need the token offsets in order to translate my labels to a normal list of token tags (my labels are in this format [{start_index: int, end_index: int, tag: str} … ]
Thank you!
Maybe it’s because this is a pre-trained fast tokenizer?