|
|
--- |
|
|
library_name: transformers |
|
|
language: |
|
|
- ko |
|
|
pipeline_tag: text-generation |
|
|
--- |
|
|
|
|
|
# Hunminai-1.0-12b |
|
|
|
|
|
Hunminai-1.0 is a Korean-aligned language model based on [Google's Gemma-3](https://huggingface.co/collections/google/gemma-3-release-67c6c6f89c4f76621268bb6d) architecture. To improve performance on Korean natural language tasks, the model was fine-tuned on a corpus of 100k instruction examples using Supervised Fine-Tuning (SFT) followed by Direct Preference Optimization (DPO). This approach enables the model to better align with user intents in Korean and enhances its applicability to downstream tasks such as dialogue generation, question answering, and long-form text generation. |
|
|
|
|
|
## Model Details |
|
|
- **Base Model**: [google/gemma-3-12b-it](https://huggingface.co/google/gemma-3-12b-it) |
|
|
- **Base Model Release Date**: March 12, 2025 |
|
|
- **Context Length**: 128k |
|
|
- **License**: [gemma](https://ai.google.dev/gemma/terms) |
|
|
- **Model Type**: Text Generation |
|
|
- **Fine-Tuning Techniques**: Supervised Fine-Tuning (SFT) and Direct Preference Optimization (DPO) |
|
|
|
|
|
## Usage |
|
|
|
|
|
Gemma 3 is supported starting from version 4.50.0 of the Transformers library. |
|
|
|
|
|
To update to the latest version, run the following command: |
|
|
``` |
|
|
$ pip install -U transformers |
|
|
``` |
|
|
|
|
|
Install the required package and run the example code below to load the Hunminai-3-12B model and perform a simple Korean-language chat completion. |
|
|
|
|
|
```python |
|
|
# pip install accelerate |
|
|
|
|
|
from transformers import AutoProcessor, Gemma3ForConditionalGeneration |
|
|
import torch |
|
|
|
|
|
model_id = "davidkim205/Hunminai-1.0-12b" |
|
|
|
|
|
model = Gemma3ForConditionalGeneration.from_pretrained( |
|
|
model_id, device_map="auto" |
|
|
).eval() |
|
|
|
|
|
processor = AutoProcessor.from_pretrained(model_id) |
|
|
|
|
|
messages = [ |
|
|
{ |
|
|
"role": "system", |
|
|
"content": [{"type": "text", "text": "λΉμ μ μ μ©ν AI λΉμμ
λλ€."}] |
|
|
}, |
|
|
{ |
|
|
"role": "user", |
|
|
"content": [ |
|
|
{"type": "text", "text": "λνλ―Όκ΅μ μλλ μ΄λμΈκ°μ?"} |
|
|
] |
|
|
} |
|
|
] |
|
|
|
|
|
inputs = processor.apply_chat_template( |
|
|
messages, add_generation_prompt=True, tokenize=True, |
|
|
return_dict=True, return_tensors="pt" |
|
|
).to(model.device, dtype=torch.bfloat16) |
|
|
|
|
|
input_len = inputs["input_ids"].shape[-1] |
|
|
|
|
|
with torch.inference_mode(): |
|
|
generation = model.generate(**inputs, max_new_tokens=128, do_sample=False) |
|
|
generation = generation[0][input_len:] |
|
|
|
|
|
decoded = processor.decode(generation, skip_special_tokens=True) |
|
|
print(decoded) |
|
|
``` |
|
|
|
|
|
## Training Dataset |
|
|
|
|
|
The model was trained on high-quality Korean instructional data. The dataset was curated to cover a broad range of Korean language contexts and task types, with a focus on aligning model outputs with user intent and natural language generation. It is currently not publicly available. |
|
|
|
|
|
|
|
|
## Evaluation |
|
|
|
|
|
### Benchmarks Datasets |
|
|
|
|
|
The table below contains a description of the Korean LLM evaluation benchmark dataset used for the model evaluation. More information on the benchmarks is available at [Blog](https://davidkim205.github.io/). |
|
|
|
|
|
| Benchmark | Description | Abbreviation | |
|
|
|------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------| |
|
|
| [ko-bench](https://huggingface.co/datasets/davidkim205/ko-bench) | Korean-translated dataset of [MT-Bench](https://github.com/lm-sys/FastChat/blob/main/fastchat/llm_judge/data/mt_bench/question.jsonl) questions | bench | |
|
|
| [ko-ged](https://huggingface.co/datasets/davidkim205/ko-ged) | Korean GED (elementary, middle, high school) open-ended question dataset<br/>Subjects: Korean, English, Mathematics, Science, Social Studies | ged | |
|
|
| [ko-ifeval](https://huggingface.co/datasets/davidkim205/ko-ifeval) | Instruction-following evaluation dataset translated from [IFEval](https://github.com/google-research/google-research/tree/master/instruction_following_eval), adapted for Korean language and culture | ifeval | |
|
|
| [ko-ged-mc-elementary](https://huggingface.co/datasets/davidkim205/ko-ged-mc-elementary) | Korean elementary school GED multiple-choice question dataset | ged\:E | |
|
|
| [ko-ged-mc-middle](https://huggingface.co/datasets/davidkim205/ko-ged-mc-middle) | Korean middle school GED multiple-choice question dataset | ged\:M | |
|
|
| [ko-ged-mc-high](https://huggingface.co/datasets/davidkim205/ko-ged-mc-high) | Korean high school GED multiple-choice question dataset | ged\:H | |
|
|
| [ko-gpqa](https://huggingface.co/datasets/davidkim205/ko-gpqa) | Korean version of GPQA containing challenging physics questions designed to test deep understanding and logical reasoning | gpqa | |
|
|
| [ko-math-500](https://huggingface.co/datasets/davidkim205/ko-math-500) | Korean-translated subset of 500 high school-level math problems from the MATH dataset, including detailed solutions with LaTeX notation | math500 | |
|
|
|
|
|
### Benchmark Results |
|
|
|
|
|
| | **davidkim205<br>Hunminai<br>-1.0-12b** | google<br>gemma-3<br>-12b-it | unsloth<br>gemma-3<br>-12b-it | K-intelligence<br>Midm-2.0<br>-Base-Instruct | LGAI-EXAONE<br>EXAONE-3.5<br>-7.8B-Instruct | |
|
|
|---------|----------------------------------------:|-----------------------------:|------------------------------:|---------------------------------------------:|--------------------------------------------:| |
|
|
| Avg. | **7.80** | 7.75 | 7.71 | 7.54 | 7.31 | |
|
|
| bench | 7.96 | 8.00 | 7.83 | **8.01** | 7.70 | |
|
|
| ged | 8.65 | 8.61 | **8.73** | 8.10 | 8.25 | |
|
|
| ged:E | **9.72** | **9.72** | 9.51 | **9.72** | 9.65 | |
|
|
| ged:M | **9.63** | 9.55 | 9.39 | 9.31 | 9.10 | |
|
|
| ged:H | 9.32 | 9.36 | 9.24 | **9.48** | 9.00 | |
|
|
| gpqa | **3.18** | 2.88 | 2.98 | 2.68 | 3.13 | |
|
|
| math500 | 5.60 | 5.58 | **5.70** | 4.80 | 4.88 | |
|
|
| ifeval | **8.37** | 8.30 | 8.33 | 8.24 | 6.76 | |
|
|
|
|
|
|
