What is your original data like ?
The “Word level” semantics is usually dealt with the Pretokenizer logic (that basically splits up the data where it’s relevant). In your case, it would depend on your original data.  There is more info in the docs:

  
      huggingface.co
  
  
    

The tokenization pipeline — tokenizers  documentation 5



  
  
    
    
  
  


If you have a specific example of what data comes in, and what you expect as a return, it’s probably going to be easier to give you pointers.
Cheers,
Nicolas