Hi @rgwatwormhill, thank you for your reply,
let’s say that I have the bellow predictions on the sentences “this is a test sentence using transformers model, the model used is CamemBert”,
image729×501 22.7 KB
The problem is I want to go back to string tokens instead of tokens generated by the tokenizer and on the same time preserving the labels, for example using the word “transformers” is mapped to “transfomer” and “s”, let’s say that transfomer is tagged as a label “model”, I am facing a problem to preserve the labels, I tried implementing a simple function to decode tokens and labels at the same time but i found problems because I don’t know very well how the tokenizer works,
I am asking if there’s a way to decode and preserve tokens using the tokenizer class !