readme: ---
  language: ko
  tags:
  - KakaoBrain
  - KoGPT
  - GPT
  - GPT3
  license: cc-by-nc-4.0
  ---
  
  # KoGPT
  
  KakaoBrain's Pre-Trained Language Models. 
  
  * KoGPT (Korean Generative Pre-trained Transformer)
    * [https://github.com/kakaobrain/kogpt](https://github.com/kakaobrain/kogpt)
    * [https://huggingface.co/kakaobrain/kogpt](https://huggingface.co/kakaobrain/kogpt)
  
  
  ## Model Descriptions
  
  ### KoGPT6B-ryan1.5b
  
  * [\[huggingface\]\[kakaobrain/kogpt\]\[KoGPT6B-ryan1.5b\]](https://huggingface.co/kakaobrain/kogpt/tree/KoGPT6B-ryan1.5b)
  * [\[huggingface\]\[kakaobrain/kogpt\]\[KoGPT6B-ryan1.5b-float16\]](https://huggingface.co/kakaobrain/kogpt/tree/KoGPT6B-ryan1.5b-float16)
  
  | Hyperparameter       | Value         |
  |:---------------------|--------------:|
  | \\(n_{parameters}\\) | 6,166,502,400 |
  | \\(n_{layers}\\)     | 28            |
  | \\(d_{model}\\)      | 4,096         |
  | \\(d_{ff}\\)         | 16,384        |
  | \\(n_{heads}\\)      | 16            |
  | \\(d_{head}\\)       | 256           |
  | \\(n_{ctx}\\)        | 2,048         |
  | \\(n_{vocab}\\)      | 64,512        |
  | Positional Encoding  | [Rotary Position Embedding (RoPE)](https://arxiv.org/abs/2104.09864) |
  | RoPE Dimensions      | 64            |
  
  
  ## Hardware requirements
  
  ### KoGPT6B-ryan1.5b
  
  #### GPU
  The following is the recommended minimum GPU hardware guidance for a handful of example KoGPT.
  * `32GB GPU RAM` in the required minimum memory size
  
  ### KoGPT6B-ryan1.5b-float16
  
  #### GPU
  The following is the recommended minimum GPU hardware guidance for a handful of example KoGPT.
  * half-precision requires NVIDIA GPUS based on Volta, Turing or Ampere
  * `16GB GPU RAM` in the required minimum memory size
  
  
  ## Usage
  
  ### prompt
  ```bash
  python -m kogpt --help
  usage: KoGPT inference [-h] [--model MODEL] [--revision {KoGPT6B-ryan1.5b}]
                         [--device {cpu,cuda}] [-d]
  
  KakaoBrain Korean(hangul) Generative Pre-Training Model
  
  optional arguments:
    -h, --help            show this help message and exit
    --model MODEL         huggingface repo (default:kakaobrain/kogpt)
    --revision {KoGPT6B-ryan1.5b}
    --device {cpu,cuda}   (default:cuda)
    -d, --debug
  ```
  
  ```bash
  python -m kogpt
  prompt> 인간처럼 생각하고, 행동하는 '지능'을 통해 인류가 이제까지 풀지 못했던
  temperature(0.8)> 
  max_length(128)> 64
  인간처럼 생각하고, 행동하는 '지능'을 통해 인류가 이제까지 풀지 못했던 문제의 해답을 찾을 수 있을 것이다. 과학기술이 고도로 발달한 21세기를 살아갈 우리 아이들에게 가장 필요한 것은 사고력 훈련이다. 사고력 훈련을 통해, 세상
  
  prompt>  
  ...
  ```
  
  
  ### python
  ```python
  import torch
  from transformers import AutoTokenizer, AutoModelForCausalLM 
  
  tokenizer = AutoTokenizer.from_pretrained(
    'kakaobrain/kogpt', revision='KoGPT6B-ryan1.5b-float16',  # or float32 version: revision=KoGPT6B-ryan1.5b
    bos_token='[BOS]', eos_token='[EOS]', unk_token='[UNK]', pad_token='[PAD]', mask_token='[MASK]'
  )
  model = AutoModelForCausalLM.from_pretrained(
    'kakaobrain/kogpt', revision='KoGPT6B-ryan1.5b-float16',  # or float32 version: revision=KoGPT6B-ryan1.5b
    pad_token_id=tokenizer.eos_token_id,
    torch_dtype='auto', low_cpu_mem_usage=True
  ).to(device='cuda', non_blocking=True)
  _ = model.eval()
  
  prompt = '인간처럼 생각하고, 행동하는 \'지능\'을 통해 인류가 이제까지 풀지 못했던'
  with torch.no_grad():
    tokens = tokenizer.encode(prompt, return_tensors='pt').to(device='cuda', non_blocking=True)
    gen_tokens = model.generate(tokens, do_sample=True, temperature=0.8, max_length=64)
    generated = tokenizer.batch_decode(gen_tokens)[0]
    
  print(generated)  # print: 인간처럼 생각하고, 행동하는 '지능'을 통해 인류가 이제까지 풀지 못했던 문제의 해답을 찾을 수 있을 것이다. 과학기술이 고도로 발달한 21세기를 살아갈 우리 아이들에게 가장 필요한 것은 사고력 훈련이다. 사고력 훈련을 통해, 세상
  ```
  
  
  ## Experiments
  
  ### In-context Few-Shots
  
  | Models        | #params | NSMC (Acc.) | YNAT (F1) | KLUE-STS (F1) |
  |:--------------|--------:|------------:|----------:|--------------:|
  | HyperCLOVA[1] |    1.3B |        83.9 |      58.7 |          60.9 |
  | HyperCLOVA[1] |    6.9B |        83.8 |      67.5 |          59.3 |
  | HyperCLOVA[1] |   13.0B |        87.9 |      67.9 |          60.0 |
  | HyperCLOVA[1] |   39.0B |        88.0 |      71.4 |          61.6 |
  | HyperCLOVA[1] |   82.0B |    **88.2** |      72.7 |      **65.1** |
  | **Ours**      |    6.0B |        87.8 |  **78.0** |          64.3 |
  
  
  ### Finetuning / P-Tuning
  
  
  We have been reported to have issues(https://github.com/kakaobrain/kogpt/issues/17) with our downstream evaluation.
  
  The previously published performance evaluation table was deleted because it was difficult to see it as a fair comparison because the comparison target algorithm was different and the performance measurement method could not be confirmed.
  
  You can refer to the above issue link for the existing performance evaluation table and troubleshooting results.
  
  
  
  ## Limitations
  
  KakaoBrain `KoGPT` was trained on `ryan dataset`, a dataset known to contain profanity, lewd, political changed, and other harsh language.
  Therefore, `KoGPT` can generate socially unacceptable texts. As with all language models, It is difficult to predict in advance how `KoGPT` will response to particular prompts and offensive content without warn