Sorry I meant lossless Tokenization. Please refer to section 3.1 in link below
  
      
      arxiv-vanity.com
  
  
    

SentencePiece: A simple and language independent subword tokenizer and... 10

This paper describes SentencePiece, a language-independent subword
tokenizer and detokenizer designed for Neural-based text processing,
including Neural Machine Translation. It provides open-source C++ and
Python implementations for subword units....


  
  
    
    
  
  


from the paper:
We call this design lossless tokenization, in which all the information to reproduce the normalized text is preserved in the encoder’s output. The basic idea of lossless tokenization is to treat the input text just as a sequence of Unicode characters. Even whitespace is handled as a normal symbol. For the sake of clarity, SentencePiece first escapes the whitespace with a meta symbol _  (U+2581), and tokenizes the input into an arbitrary subword sequence, for example: