FiratIsmailoglu's picture
Create README.md
9d55e9d verified

It is a sentencepice tokenizer trained using the Oscar Turkish Corpus from scratch. The tokenizer can be used for bigbird models and can be callled either typing AutoTokenizer.from_pretrained("FiratIsmailoglu/turkish-bigbird-tokenizer") or BigBirdTokenizerFast.from_pretrained("FiratIsmailoglu/turkish-bigbird-tokenizer").