Hanbaike
/

kyrgyz_bpe_32000

Model card Files Files and versions Community

kyrgyz_bpe_32000 / tokenizer_config.json

Hanbaike's picture

Upload tokenizer_config.json with huggingface_hub

34c17ab verified 21 days ago

173 Bytes

	{
	"model_type": "bpe",
	"unk_token": "<unk>",
	"bos_token": "<s>",
	"eos_token": "</s>",
	"pad_token": "<pad>",
	"tokenizer_class": "PreTrainedTokenizerFast"
	}