readme: ---
  language:
  - ko
  tags:
  - pytorch
  - causal-lm
  license: apache-2.0
  
  ---
  # Polyglot-Ko-5.8B
  
  ## Model Description
  Polyglot-Ko is a series of large-scale Korean autoregressive language models made by the EleutherAI polyglot team.
  
  | Hyperparameter       | Value                                                                                                                                  |
  |----------------------|----------------------------------------------------------------------------------------------------------------------------------------|
  | \\(n_{parameters}\\) | 5,885,059,072                                                                                                                           |
  | \\(n_{layers}\\)     | 28                                                                                                                                     |
  | \\(d_{model}\\)      | 4096                                                                                                                                   |
  | \\(d_{ff}\\)         | 16,384                                                                                                                                   |
  | \\(n_{heads}\\)      | 16                                                                                                                                     |
  | \\(d_{head}\\)       | 256                                                                                                                                    |
  | \\(n_{ctx}\\)        | 2,048                                                                                                                                   |
  | \\(n_{vocab}\\)      | 30,003 / 30,080                                                                                                                        |
  | Positional Encoding  | [Rotary Position Embedding (RoPE)](https://arxiv.org/abs/2104.09864)                                                                   |
  | RoPE Dimensions      | [64](https://github.com/kingoflolz/mesh-transformer-jax/blob/f2aa66e0925de6593dcbb70e72399b97b4130482/mesh_transformer/layers.py#L223) |
  
  The model consists of 28 transformer layers with a model dimension of 4096, and a feedforward dimension of 16384. The model
  dimension is split into 16 heads, each with a dimension of 256. Rotary Position Embedding (RoPE) is applied to 64
  dimensions of each head. The model is trained with a tokenization vocabulary of 30003.
  
  ## Training data
  
  Polyglot-Ko-5.8B was trained on 863 GB of Korean language data (1.2TB before processing), a large-scale dataset curated by [TUNiB](https://tunib.ai/). The data collection process has abided by South Korean laws. This dataset was collected for the purpose of training Polyglot-Ko models, so it will not be released for public use.  
  
  | Source                              |Size (GB) | Link                                  |
  |-------------------------------------|---------|------------------------------------------|
  | Korean blog posts                   | 682.3   | -                                        |
  | Korean news dataset                 | 87.0  | -                                        |
  | Modu corpus                         | 26.4  |corpus.korean.go.kr             |
  | Korean patent dataset               | 19.0  | -                                        |
  | Korean Q & A dataset                | 18.1  | -                                        |
  | KcBert dataset                      | 12.7  | github.com/Beomi/KcBERT          |
  | Korean fiction dataset              | 6.1   | -                                        |
  | Korean online comments              | 4.2   | -                                    |
  | Korean wikipedia                    | 1.4   | ko.wikipedia.org                 |
  | Clova call                          | < 1.0   | github.com/clovaai/ClovaCall     |
  | Naver sentiment movie corpus        | < 1.0   | github.com/e9t/nsmc              |
  | Korean hate speech dataset          | < 1.0   | -                                        |
  | Open subtitles                      | < 1.0  | opus.nlpl.eu/OpenSubtitles.php   |
  | AIHub various tasks datasets        | < 1.0   |aihub.or.kr                 |
  | Standard Korean language dictionary | < 1.0  | stdict.korean.go.kr/main/main.do |
                              
  Furthermore, in order to avoid the model memorizing and generating personally identifiable information (PII) in the training data, we masked out the following sensitive information in the pre-processing stage:
  
  * `<|acc|>` : bank account number
  * `<|rrn|>` : resident registration number
  * `<|tell|>` : phone number
  
  ## Training procedure
  Polyglot-Ko-5.8B was trained for 172 billion tokens over 320,000 steps on 256 A100 GPUs with the [GPT-NeoX framework](https://github.com/EleutherAI/gpt-neox). It was trained as an autoregressive language model, using cross-entropy loss to maximize the likelihood of predicting the next token.