@Sergio Thank you for your reply,
It’s so clear that the solution could be implemented if you already know how the tokenizer work, for me, I am using CamemBERT model where I couldn’t find anything to help to understand how its tokenization works, and I tried to just watch how it deals with some sentences, then I have implemented the solution but I’m not sure if the implementation will not have bugs.