SentencePiece Tokenizer

This repository contains a fine-tuned SentencePiece tokenizer on Mongolian text.

Files

  • tokenizer_config.json: The tokenizer configuration file
  • mn_tokenizer.model: The SentencePiece model file
  • mn_tokenizer.vocab: The SentencePiece vocabulary file

Usage

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("Namuun123/mn_sentencepiece_tokenizer")
Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support