readme: ---
  language: ko
  license: cc-by-sa-4.0
  tags:
    - korean
    - klue
  mask_token: "[MASK]"
  widget:
    - text: 대한민국의 수도는 [MASK] 입니다.
  ---
  
  # KLUE BERT base
  
  ## Table of Contents
  - [Model Details](#model-details)
  - [How to Get Started With the Model](#how-to-get-started-with-the-model)
  - [Uses](#uses)
  - [Risks, Limitations and Biases](#risks-limitations-and-biases)
  - [Training](#training)
  - [Evaluation](#evaluation)
  - [Environmental Impact](#environmental-impact)
  - [Technical Specifications](#technical-specifications)
  - [Citation Information](#citation-information)
  - [Model Card Authors](#model-card-authors)
  
  ## Model Details
  
  **Model Description:** KLUE BERT base is a pre-trained BERT Model on Korean Language. The developers of KLUE BERT base developed the model in the context of the development of the [Korean Language Understanding Evaluation (KLUE) Benchmark](https://arxiv.org/pdf/2105.09680.pdf).
  
  - **Developed by:** See [GitHub Repo](https://github.com/facebookresearch/fairseq/tree/main/examples/roberta) for model developers
  - **Model Type:** Transformer-based language model
  - **Language(s):** Korean
  - **License:** cc-by-sa-4.0
  - **Parent Model:** See the [BERT base uncased model](https://huggingface.co/bert-base-uncased) for more information about the BERT base model.
  - **Resources for more information:**
    - [Research Paper](https://arxiv.org/abs/2105.09680)
    - [GitHub Repo](https://github.com/KLUE-benchmark/KLUE)
  
  ## How to Get Started With the Model
  
  ```python
  from transformers import AutoModel, AutoTokenizer
  
  model = AutoModel.from_pretrained("klue/bert-base")
  tokenizer = AutoTokenizer.from_pretrained("klue/bert-base")
  ```
  
  ## Uses 
  
  #### Direct Use
  
  The model can be used for tasks including topic classification, semantic textual similarity, natural language inference, named entity recognition, and other tasks outlined in the [KLUE Benchmark](https://github.com/KLUE-benchmark/KLUE).
  
  #### Misuse and Out-of-scope Use
  
  The model should not be used to intentionally create hostile or alienating environments for people. In addition, the model was not trained to be factual or true representations of people or events, and therefore using the model to generate such content is out-of-scope for the abilities of this model.
  
  ## Risks, Limitations and Biases
  
  Significant research has explored bias and fairness issues with language models (see, e.g., [Sheng et al. (2021)](https://aclanthology.org/2021.acl-long.330.pdf) and [Bender et al. (2021)](https://dl.acm.org/doi/pdf/10.1145/3442188.3445922)). The model developers discuss several ethical considerations related to the model in the [paper](https://arxiv.org/pdf/2105.09680.pdf), including: 
  
  - Bias issues with the publicly available data used in the pretraining corpora (and considerations related to filtering)
  - PII in the data used in the pretraining corpora (and efforts to pseudonymize the data)
  
  For ethical considerations related to the KLUE Benchmark, also see the [paper](https://arxiv.org/pdf/2105.09680.pdf).
  
  ## Training
  
  #### Training Data
  
  The authors use the following pretraining corpora for the model, described in the [associated paper](https://arxiv.org/pdf/2105.09680.pdf): 
  
  > We gather the following five publicly available Korean corpora from diverse sources to cover a broad set of topics and many different styles. We combine these corpora to build the final pretraining corpus of size approximately 62GB. 
  >
  > - **MODU:** [Modu Corpus](https://corpus.korean.go.kr) is a collection of Korean corpora distributed by [National Institute of Korean Languages](https://corpus.korean.go.kr/). It includes both formal articles (news and books) and colloquial text (dialogues).
  > - **CC-100-Kor:** [CC-100](https://data.statmt.org/cc-100/) is the large-scale multilingual web crawled corpora by using CC-Net ([Wenzek et al., 2020](https://www.aclweb.org/anthology/2020.lrec-1.494)). This is used for training XLM-R ([Conneau et al., 2020](https://aclanthology.org/2020.acl-main.747/)). We use the Korean portion from this corpora.
  > - **NAMUWIKI:** NAMUWIKI is a Korean web-based encyclopedia, similar to Wikipedia, but known to be less formal. Specifically, we download [the dump](http://dump.thewiki.kr) created on March 2nd, 2020.
  > - **NEWSCRAWL:** NEWSCRAWL consists of 12,800,000 news articles published from 2011 to 2020, collected from a news aggregation platform.
  > - **PETITION:** Petition is a collection of public petitions posted to the Blue House asking for administrative actions on social issues. We use the articles in the [Blue House National Petition](https://www1.president.go.kr/petitions) published from [August 2017 to March 2019](https://ko-nlp.github.io/Korpora/en-docs/corpuslist/korean_petitions.html).
  
  The authors also describe ethical considerations related to the pretraining corpora in the [associated paper](https://arxiv.org/pdf/2105.09680.pdf).
  
  #### Training Procedure
  
  ##### Preprocessing
  
  The authors describe their preprocessing procedure in the [associa