This repository doesn't contain a model, but only a tokenizer that can be used with the | |
`tokenizers` library. | |
```python | |
from tokenizers import Tokenizer | |
tokenizer = Tokenizer.from_pretrained("anthony/tokenizers-test") | |
``` | |
This repository doesn't contain a model, but only a tokenizer that can be used with the | |
`tokenizers` library. | |
```python | |
from tokenizers import Tokenizer | |
tokenizer = Tokenizer.from_pretrained("anthony/tokenizers-test") | |
``` | |