Tokenizer from tokenizers library cannot be used in Trainer
Hi,

I am trying to train my own model with `Trainer` with a pre-trained `SentencePieceBPETokenizer` from **tokenizers** library. However, it is missing several attributes as well as methods (e.g., `pad`), which makes it incompatible with `transformers.Trainer`. Is there an easy way to convert it to `PretrainedTokenizer` from `transformers`?
Thanks!
You can just do
```
from transformers import PreTrainedTokenizerFast

tokenizer = PreTrainedTokenizerFast(tokenizer_object=your_tokenizer)
```<|||||>> You can just do
> 
> ```
> 
> from transformers import PreTrainedTokenizerFast
> 
> 
> 
> tokenizer = PreTrainedTokenizerFast(tokenizer_object=your_tokenizer)
> 
> ```

Sylvain, million thanks!