readme: ---
  language: 
    - ja
    - zh
    - ko
  license: cc-by-sa-4.0
  datasets:
  - wikipedia
  mask_token: "[MASK]"
  widget:
  - text: "早稲田大学で自然言語処理を[MASK]ぶ。"
  - text: "李白是[MASK]朝人。"
  - text: "불고기[MASK] 먹겠습니다."
  ---
  
  ### Model description
  - This model was trained on **ZH, JA, KO**'s Wikipedia (5 epochs).
  
  ### How to use
  ```python
  from transformers import AutoTokenizer, AutoModelForMaskedLM
  tokenizer = AutoTokenizer.from_pretrained("conan1024hao/cjkbert-small")
  model = AutoModelForMaskedLM.from_pretrained("conan1024hao/cjkbert-small")
  ```
  - Before you fine-tune downstream tasks, you don't need any text segmentation.
  - (Though you may obtain better results if you applied morphological analysis to the data before fine-tuning)
  
  ### Morphological analysis tools
  - ZH: For Chinese, we use [LTP](https://github.com/HIT-SCIR/ltp).
  - JA: For Japanese, we use [Juman++](https://github.com/ku-nlp/jumanpp).
  - KO: For Korean, we use [KoNLPy](https://github.com/konlpy/konlpy)(Kkma class).
  
  ### Tokenization
  - We use character-based tokenization with **whole-word-masking** strategy.
  
  ### Model size
  - vocab_size: 15015
  - num_hidden_layers: 4
  - hidden_size: 512
  - num_attention_heads: 8
  - param_num: 25M