|
--- |
|
language: |
|
- ko |
|
- en |
|
pipeline_tag: text-generation |
|
inference: false |
|
tags: |
|
- solar |
|
- mistral |
|
- pytorch |
|
- solar-ko |
|
library_name: transformers |
|
license: cc-by-nc-sa-4.0 |
|
--- |
|
|
|
**Update Log** |
|
|
|
- 2024.02.19: Initial Test version Release of SOLAR-KOEN-10.8B |
|
|
|
# **SOLAR-KOEN-10.8B** ⭐🇰🇷🇺🇸 |
|
|
|
Solar-KoEn represents an advanced iteration of the upstage/SOLAR-10.7B-v1.0 model, featuring an expanded vocabulary and the inclusion of a Korean+English corpus for enhanced pretraining. |
|
|
|
## Model Details |
|
|
|
**Model Developers:** Junbum Lee (Beomi) & Taekyoon Choi (Taekyoon) |
|
|
|
**Variations:** Solar-KoEn is available with one parameter sizes — 10.8B with Continual Pretrained version. |
|
|
|
**Input:** The model accepts only text input. |
|
|
|
**Output:** The model produces text output exclusively. |
|
|
|
**Model Architecture:** |
|
|
|
SOLAR-KOEN-10.8B is an auto-regressive language model that leverages an optimized transformer architecture derived from Llama-2. |
|
|
|
| |Training Data|Parameters|Content Length|GQA|Tokens|Learning Rate| |
|
|---|---|---|---|---|---|---| |
|
|SOLAR-KOEN-10.8B|*A curated mix of Korean+English Corpora*|10.8B|4k|O|>15B*|5e<sup>-5</sup>| |
|
|
|
**Training Corpus** |
|
|
|
The model was trained using selected datasets from AIHub and Modu Corpus. Detailed information about the training datasets is available below: |
|
|
|
- AI Hub: [corpus/AI_HUB](./corpus/AI_HUB) |
|
- Only the `Training` segment of the data was used. |
|
- The `Validation` and `Test` segments were deliberately excluded. |
|
- Modu Corpus: [corpus/MODU_CORPUS](./corpus/MODU_CORPUS) |
|
|
|
The final JSONL dataset used to train this model is approximately 61GB in size. |
|
|
|
Total token count: Approximately 15 billion tokens (*using the expanded tokenizer. With the original SOLAR tokenizer, >60 billion tokens.) |
|
|
|
**Vocab Expansion** |
|
|
|
| Model Name | Vocabulary Size | Description | |
|
| --- | --- | --- | |
|
| Original Solar | 32000 | Sentencepiece BPE | |
|
| **Expanded SOLAR-KOEN-10.8B** | 46336 | Sentencepiece BPE. Added Korean vocab and merges | |
|
|
|
**Tokenizing "안녕하세요, 오늘은 날씨가 좋네요."** |
|
|
|
- SOLAR-10.7B: 26 tokens |
|
- SOLAR-KO-10.7b: 10 tokens |
|
|
|
| Model | Tokens | |
|
| --- | --- | |
|
| SOLAR-10.7B | `['▁', '안', '<0xEB>', '<0x85>', '<0x95>', '하', '세', '요', ',', '▁', '오', '<0xEB>', '<0x8A>', '<0x98>', '은', '▁', '날', '<0xEC>', '<0x94>', '<0xA8>', '가', '▁', '좋', '네', '요', '.']` | |
|
| SOLAR-KOEN-10.8B | `['▁안', '녕', '하세요', ',', '▁오늘', '은', '▁날', '씨가', '▁좋네요', '.']` | |
|
|
|
**Tokenizing "Meet 10.7B Solar: Elevating Performance with Upstage Depth UP Scaling!"** |
|
|
|
- SOLAR-10.7B: 22 tokens |
|
- SOLAR-KO-10.7b: 22 tokens |
|
|
|
| Model | Tokens | |
|
| --- | --- | |
|
| SOLAR-10.7B | `['▁Meet', '▁', '1', '0', '.', '7', 'B', '▁Solar', ':', '▁E', 'lev', 'ating', '▁Performance', '▁with', '▁Up', 'stage', '▁Dep', 'th', '▁UP', '▁Scal', 'ing', '!']` | |
|
| SOLAR-KOEN-10.8B | `['▁Meet', '▁', '1', '0', '.', '7', 'B', '▁Solar', ':', '▁E', 'lev', 'ating', '▁Performance', '▁with', '▁Up', 'stage', '▁Dep', 'th', '▁UP', '▁Scal', 'ing', '!']` | |
|
|
|
# LICENSE |
|
|
|
Apache 2.0 |
|
|
|
# **Model Benchmark** |
|
|
|
## LM Eval Harness - Korean (polyglot branch) |
|
|
|
- Used EleutherAI's lm-evaluation-harness https://github.com/EleutherAI/lm-evaluation-harness/tree/polyglot |
|
- 5-shot scores |
|
|
|
| Task |Version| Metric | Value | |Stderr| |
|
|-------------------|------:|------------|------:|---|-----:| |
|
|klue_mrc | 0|exact |50.2140| | | |
|
| | |f1 |54.0330| | | |
|
| | |HasAns_exact|73.1786| | | |
|
| | |HasAns_f1 |78.7442| | | |
|
| | |best_exact |56.9594| | | |
|
| | |best_f1 |60.3743| | | |
|
|korquad | 1|exact_match |81.0530| | | |
|
| | |f1 |87.6418| | | |
|
|klue_nli | 0|acc | 0.4540|± |0.0091| |
|
|klue_sts | 0|acc | 0.3410|± |0.0208| |
|
| | |f1 | 0.4896|± |0.0237| |
|
|klue_ynat | 0|acc | 0.6308|± |0.0051| |
|
| | |macro_f1 | 0.6086|± |0.0057| |
|
|kobest_boolq | 0|acc | 0.8711|± |0.0089| |
|
| | |macro_f1 | 0.8705|± |0.0090| |
|
|kobest_copa | 0|acc | 0.8500|± |0.0113| |
|
| | |macro_f1 | 0.8498|± |0.0113| |
|
|kobest_hellaswag | 0|acc | 0.5180|± |0.0224| |
|
| | |acc_norm | 0.6180|± |0.0218| |
|
| | |macro_f1 | 0.5138|± |0.0224| |
|
|kobest_sentineg | 0|acc | 0.9723|± |0.0082| |
|
| | |macro_f1 | 0.9723|± |0.0083| |
|
|kobest_wic | 0|acc | 0.5825|± |0.0139| |
|
| | |macro_f1 | 0.4952|± |0.0140| |
|
|kohatespeech_apeach| 0|acc | 0.7034|± |0.0074| |
|
| | |macro_f1 | 0.7033|± |0.0074| |
|
|nsmc | 0|acc | 0.8738|± |0.0015| |
|
|pawsx_ko | 0|acc | 0.5510|± |0.0111| |
|
|kmmlu_direct | 0|exact_match | 0.4220|± |0.0909| |
|
|
|
|
|
## Citation |
|
|
|
``` |
|
@misc {solar_koen_junbum_taekyoon_2024, |
|
author = { {L. Junbum, Taekyoon Choi} }, |
|
title = { SOLAR-KOEN-10.8B }, |
|
year = 2024, |
|
url = { https://huggingface.co/beomi/SOLAR-KOEN-10.8B }, |
|
publisher = { Hugging Face } |
|
} |
|
|
|
``` |
|
|
|
## Acknowledgements |
|
|
|
- Training support was provided by the [TPU Research Cloud](https://sites.research.google/trc/) program. |
|
|
