Hugging Face's logo

Bingsu
/
ko_BBPE_tokenizer_bert

Korean
bert
tokenizer only
Model card Files
xet
Community

๋ผ์ด๋ธŒ๋Ÿฌ๋ฆฌ ๋ฒ„์ „

  • transformers: 4.21.1
  • datasets: 2.4.0
  • tokenizers: 0.12.1

Bingsu/ko_BBPE_tokenizer_roberta์—์„œ post-processor๋ฅผ BertProcessing๋กœ ๋ณ€๊ฒฝํ•˜๊ณ  ํ† ํฌ๋‚˜์ด์ € ํด๋ž˜์Šค๋ฅผ BertTokenizerFast๋กœ ๋ณ€๊ฒฝํ•œ ๊ฒƒ์ž…๋‹ˆ๋‹ค.

๋‘ ๋ฌธ์žฅ์„ ํ† ํฌ๋‚˜์ด์ €์— ์ž…๋ ฅํ–ˆ์„ ๋•Œ, sep ํ† ํฐ์ด ๋‹ค๋ฅด๊ฒŒ ์ฒ˜๋ฆฌ๋œ๋‹ค๋Š” ๊ฒƒ์„ ์ œ์™ธํ•˜๊ณ  ๊ฒฐ๊ณผ๋Š” ๋™์ผํ•ฉ๋‹ˆ๋‹ค.

Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support