It’s so clear that the solution could be implemented if you already know how the tokenizer work, for me, I am using CamemBERT model where I couldn’t find anything to help to understand how its tokenization works, and I tried to just watch how it deals with some sentences, then I have implemented the solution but I’m not sure if the implementation will not have bugs.

The PreTrainedTokenizer class has a method called decode() 6 that converts token_ids (potentially corresponding to subword pieces) to complete tokens again. There’s a tight loop iterating over each token. I can’t make out the logic myself.