token space (e.g., getting the index of the token comprising a given character or the span of characters corresponding
to a given token).
PreTrainedTokenizer
[[autodoc]] PreTrainedTokenizer
    - call
    - batch_decode
    - decode
    - encode
    - push_to_hub
    - all
PreTrainedTokenizerFast
The [PreTrainedTokenizerFast] depend on the tokenizers library. The tokenizers obtained from the 🤗 tokenizers library can be
loaded very simply into 🤗 transformers. Take a look at the Using tokenizers from 🤗 tokenizers page to understand how this is done.
[[autodoc]] PreTrainedTokenizerFast
    - call
    - batch_decode
    - decode
    - encode
    - push_to_hub
    - all
BatchEncoding
[[autodoc]] BatchEncoding
