Create README.md
Browse files
README.md
ADDED
|
@@ -0,0 +1,2 @@
|
|
|
|
|
|
|
|
|
|
| 1 |
+
It is a sentencepice tokenizer trained using the Oscar Turkish Corpus from scratch. The tokenizer can be used for bigbird models and can be callled either typing
|
| 2 |
+
AutoTokenizer.from_pretrained("FiratIsmailoglu/turkish-bigbird-tokenizer") or BigBirdTokenizerFast.from_pretrained("FiratIsmailoglu/turkish-bigbird-tokenizer").
|