Where can I find raw code for char_to_token function.
I understand the function has been defined in tokenization_utils_base.py but in the return statement there is a recursive call to the same function. I am unable to understand where does the actual offset calculation takes place.

Tapan


I figured it out. Thanks.