---
language:
- ko
- en
pipeline_tag: text-generation
inference: false
tags:
- solar
- mistral
- pytorch
- solar-ko
library_name: transformers
license: cc-by-nc-sa-4.0
---

**Update Log**

- 2024.02.19: Initial Test version Release of SOLAR-KOEN-10.8B

# **SOLAR-KOEN-10.8B** ⭐🇰🇷🇺🇸

Solar-KoEn represents an advanced iteration of the upstage/SOLAR-10.7B-v1.0 model, featuring an expanded vocabulary and the inclusion of a Korean+English corpus for enhanced pretraining. 

## Model Details

**Model Developers:** Junbum Lee (Beomi) & Taekyoon Choi (Taekyoon)

**Variations:** Solar-KoEn is available with one parameter sizes — 10.8B with Continual Pretrained version.

**Input:** The model accepts only text input.

**Output:** The model produces text output exclusively.

**Model Architecture:** 

SOLAR-KOEN-10.8B is an auto-regressive language model that leverages an optimized transformer architecture derived from Llama-2.

| |Training Data|Parameters|Content Length|GQA|Tokens|Learning Rate|
|---|---|---|---|---|---|---|
|SOLAR-KOEN-10.8B|*A curated mix of Korean+English Corpora*|10.8B|4k|O|>60B*|5e<sup>-5</sup>|

**Vocab Expansion**

| Model Name | Vocabulary Size | Description | 
| --- | --- | --- |
| Original Solar | 32000 | Sentencepiece BPE |
| **Expanded SOLAR-KOEN-10.8B** | 46336 | Sentencepiece BPE. Added Korean vocab and merges |

**Tokenizing "안녕하세요, 오늘은 날씨가 좋네요."**

- SOLAR-10.7B: 26 tokens
- SOLAR-KO-10.7b: 10 tokens

| Model | Tokens |
| --- | --- |
| SOLAR-10.7B | `['▁', '안', '<0xEB>', '<0x85>', '<0x95>', '하', '세', '요', ',', '▁', '오', '<0xEB>', '<0x8A>', '<0x98>', '은', '▁', '날', '<0xEC>', '<0x94>', '<0xA8>', '가', '▁', '좋', '네', '요', '.']` |
| SOLAR-KOEN-10.8B | `['▁안', '녕', '하세요', ',', '▁오늘', '은', '▁날', '씨가', '▁좋네요', '.']` |

**Tokenizing "Meet 10.7B Solar: Elevating Performance with Upstage Depth UP Scaling!"**

- SOLAR-10.7B: 22 tokens
- SOLAR-KO-10.7b: 22 tokens

| Model | Tokens |
| --- | --- |
| SOLAR-10.7B | `['▁Meet', '▁', '1', '0', '.', '7', 'B', '▁Solar', ':', '▁E', 'lev', 'ating', '▁Performance', '▁with', '▁Up', 'stage', '▁Dep', 'th', '▁UP', '▁Scal', 'ing', '!']` |
| SOLAR-KOEN-10.8B | `['▁Meet', '▁', '1', '0', '.', '7', 'B', '▁Solar', ':', '▁E', 'lev', 'ating', '▁Performance', '▁with', '▁Up', 'stage', '▁Dep', 'th', '▁UP', '▁Scal', 'ing', '!']` |

# LICENSE

Apache 2.0

# **Model Benchmark**

## LM Eval Harness - Korean (polyglot branch)

- Used EleutherAI's lm-evaluation-harness https://github.com/EleutherAI/lm-evaluation-harness/tree/polyglot
- 5-shot scores

|       Task        |Version|   Metric   | Value |   |Stderr|
|-------------------|------:|------------|------:|---|-----:|
|klue_mrc           |      0|exact       |50.2140|   |      |
|                   |       |f1          |54.0330|   |      |
|                   |       |HasAns_exact|73.1786|   |      |
|                   |       |HasAns_f1   |78.7442|   |      |
|                   |       |best_exact  |56.9594|   |      |
|                   |       |best_f1     |60.3743|   |      |
|korquad            |      1|exact_match |81.0530|   |      |
|                   |       |f1          |87.6418|   |      |
|klue_nli           |      0|acc         | 0.4540|±  |0.0091|
|klue_sts           |      0|acc         | 0.3410|±  |0.0208|
|                   |       |f1          | 0.4896|±  |0.0237|
|klue_ynat          |      0|acc         | 0.6308|±  |0.0051|
|                   |       |macro_f1    | 0.6086|±  |0.0057|
|kobest_boolq       |      0|acc         | 0.8711|±  |0.0089|
|                   |       |macro_f1    | 0.8705|±  |0.0090|
|kobest_copa        |      0|acc         | 0.8500|±  |0.0113|
|                   |       |macro_f1    | 0.8498|±  |0.0113|
|kobest_hellaswag   |      0|acc         | 0.5180|±  |0.0224|
|                   |       |acc_norm    | 0.6180|±  |0.0218|
|                   |       |macro_f1    | 0.5138|±  |0.0224|
|kobest_sentineg    |      0|acc         | 0.9723|±  |0.0082|
|                   |       |macro_f1    | 0.9723|±  |0.0083|
|kobest_wic         |      0|acc         | 0.5825|±  |0.0139|
|                   |       |macro_f1    | 0.4952|±  |0.0140|
|kohatespeech_apeach|      0|acc         | 0.7034|±  |0.0074|
|                   |       |macro_f1    | 0.7033|±  |0.0074|
|nsmc               |      0|acc         | 0.8738|±  |0.0015|
|pawsx_ko           |      0|acc         | 0.5510|±  |0.0111|
|kmmlu_direct       |      0|exact_match | 0.4220|±  |0.0909|


## Citation

```
@misc {solar_koen_junbum_taekyoon_2024,
    author       = { {L. Junbum, Taekyoon Choi} },
    title        = { SOLAR-KOEN-10.8B },
    year         = 2024,
    url          = { https://huggingface.co/beomi/SOLAR-KOEN-10.8B },
    publisher    = { Hugging Face }
}

```

## Acknowledgements

- Training support was provided by the [TPU Research Cloud](https://sites.research.google/trc/) program.