readme: ---
  language:
  - ko
  
  ---
  
  ## KoRean based Bert pre-trained (KR-BERT)
  
  This is a release of Korean-specific, small-scale BERT models with comparable or better performances developed by Computational Linguistics Lab at Seoul National University, referenced in [KR-BERT: A Small-Scale Korean-Specific Language Model](https://arxiv.org/abs/2008.03979).
  
  <br>
  
  ### Vocab, Parameters and Data
  
  |                |                              Mulitlingual BERT<br>(Google) |                KorBERT<br>(ETRI) |                              KoBERT<br>(SKT) |                       KR-BERT character |                   KR-BERT sub-character |
  | -------------: | ---------------------------------------------: | ---------------------: | ----------------------------------: | -------------------------------------: | -------------------------------------: |
  |     vocab size |                                        119,547 |                 30,797 |                               8,002 |                                 16,424 |                                 12,367 |
  | parameter size |                                    167,356,416 |            109,973,391 |                          92,186,880 |                             99,265,066 |                             96,145,233 |
  |      data size | -<br>(The Wikipedia data<br>for 104 languages) | 23GB<br>4.7B morphemes | -<br>(25M sentences,<br>233M words) | 2.47GB<br>20M sentences,<br>233M words | 2.47GB<br>20M sentences,<br>233M words |
  
  | Model                                       | Masked LM Accuracy |
  | ------------------------------------------- | ------------------ |
  | KoBERT                                      | 0.750              |
  | KR-BERT character BidirectionalWordPiece     | **0.779**              |
  | KR-BERT sub-character BidirectionalWordPiece | 0.769              |
  
  <br>
  
  ### Sub-character
  
  Korean text is basically represented with Hangul syllable characters, which can be decomposed into sub-characters, or graphemes. To accommodate such characteristics, we trained a new vocabulary and BERT model on two different representations of a corpus: syllable characters and sub-characters.
  
  In case of using our sub-character model, you should preprocess your data with the code below.
  
  ```python
  import torch
  from transformers import BertConfig, BertModel, BertForPreTraining, BertTokenizer
  from unicodedata import normalize
  
  tokenizer_krbert = BertTokenizer.from_pretrained('/path/to/vocab_file.txt', do_lower_case=False)
  
  # convert a string into sub-char
  def to_subchar(string):
      return normalize('NFKD', string)
  
  sentence = '토크나이저 예시입니다.'
  print(tokenizer_krbert.tokenize(to_subchar(sentence)))
  
  ```
  
  ### Tokenization
  
  #### BidirectionalWordPiece Tokenizer
  
  We use the BidirectionalWordPiece model to reduce search costs while maintaining the possibility of choice. This model applies BPE in both forward and backward directions to obtain two candidates and chooses the one that has a higher frequency.
  
  
  |                                         |     Mulitlingual BERT     |   KorBERT<br>character   |          KoBERT           | KR-BERT<br>character<br>WordPiece | KR-BERT<br>character<br>BidirectionalWordPiece | KR-BERT<br>sub-character<br>WordPiece | KR-BERT<br>sub-character<br>BidirectionalWordPiece |
  | :-------------------------------------: | :-----------------------: | :-----------------------: | :-----------------------: | :------------------------------: | :-------------------------------------------: | :----------------------------------: | :-----------------------------------------------: |
  | 냉장고<br>nayngcangko<br>"refrigerator" | 냉#장#고<br>nayng#cang#ko | 냉#장#고<br>nayng#cang#ko | 냉#장#고<br>nayng#cang#ko |      냉장고<br>nayngcangko       |             냉장고<br>nayngcangko             |        냉장고<br>nayngcangko         |               냉장고<br>nayngcangko               |
  |        춥다<br>chwupta<br>"cold"        |           [UNK]           |     춥#다<br>chwup#ta     |     춥#다<br>chwup#ta     |        춥#다<br>chwup#ta         |               춥#다<br>chwup#ta               |         추#ㅂ다<br>chwu#pta          |                추#ㅂ다<br>chwu#pta                |
  |     뱃사람<br>paytsalam<br>"seaman"     |           [UNK]           |   뱃#사람<br>payt#salam   |   뱃#사람<br>payt#salam   |      뱃#사람<br>payt#salam       |             뱃#사람<br>payt#salam             |      배#ㅅ#사람<br>pay#t#salam       |             배#ㅅ#사람<br>pay#t#salam             |
  |    마이크<br>maikhu<br>"microphone"     |   마#이#크<br>ma#i#khu    |    마이#크<br>mai#khu     |   마#이#크<br>ma#i#khu    |         마이크<br>maikhu         |               마이크<br>maikhu                |           마이크<br>maikhu           |                 마이크<br>maikhu                  |
  
  <br>
  
  
  ### Models
  
  |     | TensorFlow | | PyTorch | |
  |:---:|:-------------------------------:|:----------------------------:|:----------------------------:|:----------------------------:|
  |     | character | sub-character | character | sub-character |
  | WordPiece <br> tokenizer | [WP char](