For anyone else looking, this can be done, and it’s answered in this question:

  
    
    
    How to add additional custom pre-tokenization processing? 🤗Tokenizers
  
  
    I would like to add a few custom functions for pre-tokenization. For example, I would like to split numerical text from any non-numerical test. 
Eg 
‘1000mg’ would become [‘1000’, ‘mg’]. 
I am trying to figure out the proper way to do this for the python binding; I think it may be a bit tricky since its a binding for the original rust version. 
I am looking at the pretokenizer function 
/huggingface/tokenizers/blob/2ccd16bf5c3dd97759d7bdf5229e2feeba314b4a/bindings/python/py_src/tokenizers/pre_to…