There is a word_ids key of the returned BatchEncoding that exactly does this.

A list indicating the word corresponding to each token.


  

      huggingface.co
  

  
    

Tokenizer 2



  

  
    
    
  

  


But it looks like to be available only in PreTrainedTokenizerFast. I didn’t find this word_ids in the returned value when using PreTrainedTokenizer.
The documentation says that this kind of mapping back to the original texts is only available in the fast version:

When the tokenizer is a “Fast” tokenizer (i.e., backed by HuggingFace tokenizers library 1), this class provides in addition several advanced alignment methods which can be used to map between the original string (character and words) and the token space (e.g., getting the index of the token comprising a given character or the span of characters corresponding to a given token).