Upload folder using huggingface_hub

.gitattributes

@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+tokenizer.json filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,154 @@
+---
+library_name: transformers
+language:
+- ko
+pipeline_tag: text-generation
+---
+
+# Hunminai-1.0-27b
+
+Hunminai-1.0 is a Korean-aligned language model based on [Google's Gemma-3](https://huggingface.co/collections/google/gemma-3-release-67c6c6f89c4f76621268bb6d) architecture. To improve performance on Korean natural language tasks, the model was fine-tuned on a corpus of 100k instruction examples using Supervised Fine-Tuning (SFT) followed by Direct Preference Optimization (DPO). This approach enables the model to better align with user intents in Korean and enhances its applicability to downstream tasks such as dialogue generation, question answering, and long-form text generation.
+
+## Model Details
+- **Base Model**: [google/gemma-3-27b-it](https://huggingface.co/google/gemma-3-27b-it)
+- **Base Model Release Date**: March 12, 2025
+- **Context Length**: 128k
+- **License**: [gemma](https://ai.google.dev/gemma/terms)
+- **Model Type**: Text Generation
+- **Fine-Tuning Techniques**: Supervised Fine-Tuning (SFT) and Direct Preference Optimization (DPO)
+
+## Usage Application Form
+To use this model, please complete the application form and submit it via email [[email protected]].
+Access will be granted after your application is reviewed and approved.
+We appreciate your cooperation and look forward to assisting you.
+```
+1. **Name:**
+- (e.g., John Doe)
+2. **Date of Birth:**
+- (e.g., January 1, 1990)
+3. **Affiliation:**
+- Are you applying as a company or an individual? [ ] Company [ ] Individual
+- Company Name (if applicable):
+- Department (if applicable):
+4. **Position/Role:**
+- (e.g., Data Scientist, Researcher, etc.)
+5. **Contact Information:**
+- Email:
+- Phone Number:
+
+6. **Purpose of Use:**
+- (e.g., Research and Development, Commercial use, Educational purposes, etc.)
+
+7. **Detailed Reason for Use:**
+- 1. Name and version of the model you wish to use:
+- 2. Reason for selecting this model:
+- 3. Objectives to achieve using this model:
+- 4. Expected use cases (please describe in as much detail as possible):
+
+8. **Data Security and Ethical Use Plan:**
+- (Please describe your plans for data protection and ethical use.)
+```
+
+## Usage
+
+Gemma 3 is supported starting from version 4.50.0 of the Transformers library.
+
+To update to the latest version, run the following command:
+```
+$ pip install -U transformers
+```
+
+Install the required package and run the example code below to load the Hunminai-3-27B model and perform a simple Korean-language chat completion.
+
+```python
+# pip install accelerate
+
+from transformers import AutoProcessor, Gemma3ForConditionalGeneration
+import torch
+
+model_id = "davidkim205/Hunminai-3-27b"
+
+model = Gemma3ForConditionalGeneration.from_pretrained(
+    model_id, device_map="auto"
+).eval()
+
+processor = AutoProcessor.from_pretrained(model_id)
+
+messages = [
+    {
+        "role": "system",
+        "content": [{"type": "text", "text": "당신은 유용한 AI 비서입니다."}]
+    },
+    {
+        "role": "user",
+        "content": [
+            {"type": "text", "text": "대한민국의 수도는 어디인가요?"}
+        ]
+    }
+]
+
+inputs = processor.apply_chat_template(
+    messages, add_generation_prompt=True, tokenize=True,
+    return_dict=True, return_tensors="pt"
+).to(model.device, dtype=torch.bfloat16)
+
+input_len = inputs["input_ids"].shape[-1]
+
+with torch.inference_mode():
+    generation = model.generate(**inputs, max_new_tokens=128, do_sample=False)
+    generation = generation[0][input_len:]
+
+decoded = processor.decode(generation, skip_special_tokens=True)
+print(decoded)
+```
+
+## Training Dataset
+
+The model was trained on approximately 100k high-quality Korean instruction examples. The dataset was curated to cover a wide range of Korean language contexts and tasks, with a focus on aligning model outputs with user intent and natural language generation, and is currently undisclosed.
+
+
+## Evaluation
+
+### Benchmarks Datasets
+
+The table below contains a description of the Korean LLM evaluation benchmark dataset used for the model evaluation. More information on the benchmarks is available at [Blog](https://davidkim205.github.io/).
+
+| Benchmark                                                                          | Description                                                                                                                                                                                           | Abbreviation |
+|------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------|
+| [ko-bench](https://huggingface.co/datasets/davidkim205/ko-bench)                   | Korean-translated dataset of [MT-Bench](https://github.com/lm-sys/FastChat/blob/main/fastchat/llm_judge/data/mt_bench/question.jsonl) questions                                                       | bench        |
+| [ko-bench-v2](https://huggingface.co/datasets/davidkim205/ko-bench-v2)             | Dataset including new questions and answers following the ko-bench format                                                                                                                             | bench2       |
+| [ko-ged](https://huggingface.co/datasets/davidkim205/ko-ged)                       | Korean GED (elementary, middle, high school) open-ended question dataset<br/>Subjects: Korean, English, Mathematics, Science, Social Studies                                                          | ged          |
+| [ko-ged2](https://huggingface.co/datasets/davidkim205/ko-ged2)                     | Korean GED open-ended question dataset for the 2025 1st Korean GED Exam, covering all subjects                                                                                                        | ged2         |
+| [tiny-eval](https://huggingface.co/datasets/davidkim205/tiny-eval)                 | High-quality evaluation dataset designed to assess overall model performance with a small amount of data                                                                                              | tiny         |
+| [ko-ifeval](https://huggingface.co/datasets/davidkim205/ko-ifeval)                 | Instruction-following evaluation dataset translated from [IFEval](https://github.com/google-research/google-research/tree/master/instruction_following_eval), adapted for Korean language and culture | ifeval       |
+| [ko-ged-elementary](https://huggingface.co/datasets/davidkim205/ko-ged-elementary) | Korean elementary school GED multiple-choice question dataset                                                                                                                                         | ged\:E       |
+| [ko-ged-middle](https://huggingface.co/datasets/davidkim205/ko-ged-middle)         | Korean middle school GED multiple-choice question dataset                                                                                                                                             | ged\:M       |
+| [ko-ged-high](https://huggingface.co/datasets/davidkim205/ko-ged-high)             | Korean high school GED multiple-choice question dataset                                                                                                                                               | ged\:H       |
+| [ko-ged2-elementary](https://huggingface.co/datasets/davidkim205/ko-ged2-middle)   | Korean elementary school GED multiple-choice dataset, updated for the 2025 GED Exam                                                                                                                   | ged2\:E      |
+| [ko-ged2-middle](https://huggingface.co/datasets/davidkim205/ko-ged2-elementary)   | Korean middle school GED multiple-choice dataset, updated for the 2025 GED Exam                                                                                                                       | ged2\:M      |
+| [ko-ged2-high](https://huggingface.co/datasets/davidkim205/ko-ged2-high)           | Korean high school GED multiple-choice dataset, updated for the 2025 GED Exam                                                                                                                         | ged2\:H      |
+| [ko-gpqa](https://huggingface.co/datasets/davidkim205/ko-gpqa)                     | Korean version of GPQA containing challenging physics questions designed to test deep understanding and logical reasoning                                                                             | gpqa         |
+| [ko-math-500](https://huggingface.co/datasets/davidkim205/ko-math-500)             | Korean-translated subset of 500 high school-level math problems from the MATH dataset, including detailed solutions with LaTeX notation                                                               | math500      |
+
+### Benchmark Results
+
+
+|         |  **davidkim205<br>ko-gemma-3-27b** |  google<br>gemma-3-27b-it |  unsloth<br>gemma-3-27b-it |  google<br>gemma-2-27b-it |
+|---------|-----------------------------------:|--------------------------:|---------------------------:|--------------------------:|
+| Avg.    |                           **8.83** |                      8.74 |                       8.56 |                      8.08 |
+| bench   |                               8.26 |                      8.06 |                   **8.27** |                      7.59 |
+| bench2  |                               8.74 |                  **8.79** |                       8.73 |                      8.21 |
+| ged     |                           **9.19** |                      9.02 |                       9.03 |                      8.38 |
+| ged2    |                               8.90 |                  **8.91** |                       8.98 |                      8.38 |
+| tiny    |                               8.50 |                      9.08 |                   **9.12** |                           |
+| ifeval  |                                    |                           |                       8.10 |                           |
+| ged:E   |                               9.86 |                      9.86 |                   **9.93** |                      9.51 |
+| ged:M   |                               9.67 |                      9.63 |                   **9.76** |                      9.10 |
+| ged:H   |                           **9.60** |                      9.52 |                       9.52 |                      9.32 |
+| ged2:E  |                               9.77 |                      9.89 |                   **9.94** |                      9.48 |
+| ged2:M  |                           **9.75** |                      9.58 |                       9.46 |                      9.33 |
+| ged2:H  |                           **9.48** |                      9.23 |                       9.40 |                      9.08 |
+| gpqa    |                           **4.55** |                      3.69 |                       3.38 |                      3.54 |
+| math500 |                           **8.56** |                      8.38 |                       6.26 |                      5.00 |
+
+

chat_template.json

@@ -0,0 +1,3 @@
+{
+  "chat_template": "{{ bos_token }}\n{%- if messages[0]['role'] == 'system' -%}\n    {%- if messages[0]['content'] is string -%}\n        {%- set first_user_prefix = messages[0]['content'] + '\n\n' -%}\n    {%- else -%}\n        {%- set first_user_prefix = messages[0]['content'][0]['text'] + '\n\n' -%}\n    {%- endif -%}\n    {%- set loop_messages = messages[1:] -%}\n{%- else -%}\n    {%- set first_user_prefix = \"\" -%}\n    {%- set loop_messages = messages -%}\n{%- endif -%}\n{%- for message in loop_messages -%}\n    {%- if (message['role'] == 'user') != (loop.index0 % 2 == 0) -%}\n        {{ raise_exception(\"Conversation roles must alternate user/assistant/user/assistant/...\") }}\n    {%- endif -%}\n    {%- if (message['role'] == 'assistant') -%}\n        {%- set role = \"model\" -%}\n    {%- else -%}\n        {%- set role = message['role'] -%}\n    {%- endif -%}\n    {{ '<start_of_turn>' + role + '\n' + (first_user_prefix if loop.first else \"\") }}\n    {%- if message['content'] is string -%}\n        {{ message['content'] | trim }}\n    {%- elif message['content'] is iterable -%}\n        {%- for item in message['content'] -%}\n            {%- if item['type'] == 'image' -%}\n                {{ '<start_of_image>' }}\n            {%- elif item['type'] == 'text' -%}\n                {{ item['text'] | trim }}\n            {%- endif -%}\n        {%- endfor -%}\n    {%- else -%}\n        {{ raise_exception(\"Invalid content type\") }}\n    {%- endif -%}\n    {{ '<end_of_turn>\n' }}\n{%- endfor -%}\n{%- if add_generation_prompt -%}\n    {{'<start_of_turn>model\n'}}\n{%- endif -%}\n"
+}

config.json

@@ -0,0 +1,64 @@
+{
+  "architectures": [
+    "Gemma3ForConditionalGeneration"
+  ],
+  "boi_token_index": 255999,
+  "eoi_token_index": 256000,
+  "eos_token_id": [
+    1,
+    106
+  ],
+  "hidden_size": 5376,
+  "image_token_index": 262144,
+  "initializer_range": 0.02,
+  "mm_tokens_per_image": 256,
+  "model_type": "gemma3",
+  "text_config": {
+    "attention_bias": false,
+    "attention_dropout": 0.0,
+    "attn_logit_softcapping": null,
+    "cache_implementation": "hybrid",
+    "final_logit_softcapping": null,
+    "head_dim": 128,
+    "hidden_activation": "gelu_pytorch_tanh",
+    "hidden_size": 5376,
+    "initializer_range": 0.02,
+    "intermediate_size": 21504,
+    "max_position_embeddings": 131072,
+    "model_type": "gemma3_text",
+    "num_attention_heads": 32,
+    "num_hidden_layers": 62,
+    "num_key_value_heads": 16,
+    "query_pre_attn_scalar": 168,
+    "rms_norm_eps": 1e-06,
+    "rope_local_base_freq": 10000.0,
+    "rope_scaling": {
+      "factor": 8.0,
+      "rope_type": "linear"
+    },
+    "rope_theta": 1000000.0,
+    "sliding_window": 1024,
+    "sliding_window_pattern": 6,
+    "torch_dtype": "bfloat16",
+    "use_cache": true,
+    "vocab_size": 262208
+  },
+  "torch_dtype": "bfloat16",
+  "transformers_version": "4.51.3",
+  "use_cache": true,
+  "vision_config": {
+    "attention_dropout": 0.0,
+    "hidden_act": "gelu_pytorch_tanh",
+    "hidden_size": 1152,
+    "image_size": 896,
+    "intermediate_size": 4304,
+    "layer_norm_eps": 1e-06,
+    "model_type": "siglip_vision_model",
+    "num_attention_heads": 16,
+    "num_channels": 3,
+    "num_hidden_layers": 27,
+    "patch_size": 14,
+    "torch_dtype": "bfloat16",
+    "vision_use_head": false
+  }
+}

generation_config.json

@@ -0,0 +1,13 @@
+{
+  "bos_token_id": 2,
+  "cache_implementation": "hybrid",
+  "do_sample": true,
+  "eos_token_id": [
+    1,
+    106
+  ],
+  "pad_token_id": 0,
+  "top_k": 64,
+  "top_p": 0.95,
+  "transformers_version": "4.51.3"
+}

model-00001-of-00012.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7362853ece462bc312ec687be7017321b379d1692608406989c3cd881d44a7ff
+size 4854573696

model-00002-of-00012.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8d2913cd34e87ddfa6d542183ea911ea4d120b65c69bdb6901caca676c2dea25
+size 4954792944

model-00003-of-00012.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b5dce660d7a80292f2acfb1e33ffc9427b5ca33474287a0f9b6dc30e56bbd178
+size 4954792976

model-00004-of-00012.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:dd44980260e7c20bf4d808e46db13fb2344df35c570a0cefbadba70630d2ba8d
+size 4954793016

model-00005-of-00012.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4bab80685b5a3c31e3a4596d5a6bb5b25ef4da6ef405831d395f31f62daf75ad
+size 4954793016

model-00006-of-00012.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7f03b2da4a318451e2520e783aed8a334c7bfca96b35bbbdc0e68e6aff4c0ae0
+size 4954793016

model-00007-of-00012.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ba66a7a7bb63becb9b58d040c2230f70ec0281e6c299c5853b4725684eb90e44
+size 4954793016

model-00008-of-00012.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b321aa48dacc3d6ab511a432687c5b838bc4ee3f1c1ee92c726a8161c09fe9bf
+size 4954793016

model-00009-of-00012.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0b2c7fe1eb69688216170faff8abe3dcedc428031e722510254a6752c02d14c6
+size 4954793016

model-00010-of-00012.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e673a4e2e939349674bf0c9f73d995b42a0c5b0d81318bd4a1a500dcc4a0784a
+size 4954793016

model-00011-of-00012.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:db314f3fe55b785b904da55460bc55ee8119f59abfbe77dc5510b609f2f2c557
+size 4954793016

model-00012-of-00012.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:dc82510643460995e0dc9b180e5ea73d0b4ad65823fe1c729d0833dc060efd0b
+size 462476696

preprocessor_config.json

@@ -0,0 +1,29 @@
+{
+  "do_convert_rgb": null,
+  "do_normalize": true,
+  "do_pan_and_scan": null,
+  "do_rescale": true,
+  "do_resize": true,
+  "image_mean": [
+    0.5,
+    0.5,
+    0.5
+  ],
+  "image_processor_type": "Gemma3ImageProcessor",
+  "image_seq_length": 256,
+  "image_std": [
+    0.5,
+    0.5,
+    0.5
+  ],
+  "pan_and_scan_max_num_crops": null,
+  "pan_and_scan_min_crop_size": null,
+  "pan_and_scan_min_ratio_to_activate": null,
+  "processor_class": "Gemma3Processor",
+  "resample": 2,
+  "rescale_factor": 0.00392156862745098,
+  "size": {
+    "height": 896,
+    "width": 896
+  }
+}

processor_config.json

@@ -0,0 +1,4 @@
+{
+  "image_seq_length": 256,
+  "processor_class": "Gemma3Processor"
+}

special_tokens_map.json

@@ -0,0 +1,36 @@
+{
+  "additional_special_tokens": [
+    "<end_of_turn>"
+  ],
+  "boi_token": "<start_of_image>",
+  "bos_token": {
+    "content": "<bos>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eoi_token": "<end_of_image>",
+  "eos_token": {
+    "content": "<eos>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "image_token": "<image_soft_token>",
+  "pad_token": {
+    "content": "<pad>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "unk_token": {
+    "content": "<unk>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

tokenizer.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4667f2089529e8e7657cfb6d1c19910ae71ff5f28aa7ab2ff2763330affad795
+size 33384568