metadata
license: llama3.2
language:
- en
- zh
base_model:
- meta-llama/Llama-3.2-3B
library_name: transformers
tags:
- Taiwan
- R.O.C
- zhtw
- SLM
- Llama-32
datasets:
- lianghsun/tw-reasoning-instruct
- minyichen/tw-instruct-R1-200k
- minyichen/tw_mm_R1
model-index:
- name: Llama-3.2-3B-F1-Instruct
results:
- task:
type: question-answering
name: Single Choice Question
dataset:
type: ikala/tmmluplus
name: tmmlu+
config: all
split: test
revision: c0e8ae955997300d5dbf0e382bf0ba5115f85e8c
metrics:
- name: single choice
type: accuracy
value: 42.21
Model Card for Llama-3.2-3B-F1-Instruct
Note: The checkpoint for this model will be released soon. Please stay tuned. 🙏
Llama-3.2-3B-F1-Instruct 是由 Twinkle AI 與 APMIC 合作開發，並在國家高速網路與計算中心技術指導之下，針對中華民國台灣語境與任務需求所微調之繁體中文語言模型，涵蓋法律、教育、生活應用等多元場景，並以高指令跟隨能力為目標進行強化。
Model Details
Model Description
- Developed by: Huang Liang Hsun、Min Yi Chen、Wen Bin Lin、Chao Chun Chuang & Dave Sung (All authors have contributed equally to this work.)
- Funded by: APMIC
- Model type: LlamaForCausalLM
- Language(s) (NLP): Tranditional Chinese & English
- License: llama3.2
Model Sources
- Repository: twinkle-ai/Llama-3.2-3B-F1-Instruct
- Paper: (TBA)
Evaluation
Results
下表採用 🌟 Twinkle Eval 評測框架
| 模型 | 評測模式 | TMMLU+(%) | 台灣法律(%) | MMLU(%) | 測試次數 | 選項排序 |
|---|---|---|---|---|---|---|
| mistralai/Mistral-Small-24B-Instruct-2501 | box | 56.15 (±0.0172) | 37.48 (±0.0098) | 74.61 (±0.0154) | 3 | 隨機 |
| meta-llama/Llama-3.2-3B-Instruct | box | 15.49 (±0.0104) | 25.68 (±0.0200) | 6.90 (±0.0096) | 3 | 隨機 |
| meta-llama/Llama-3.2-3B-Instruct | pattern | 35.85 (±0.0174) | 32.22 (±0.0023) | 59.33 (±0.0168) | 3 | 隨機 |
| MediaTek-Research/Llama-Breeze2-3B-Instruct | pattern | 40.32 (±0.0181) | 38.92 (±0.0193) | 55.37 (±0.0180) | 3 | 隨機 |
| twinkle-ai/Llama-3.2-3B-F1-Instruct (ours) | box | 42.18 (±0.0197) | 31.26 (±0.0354) | 52.07 (±0.0189) | 3 | 隨機 |
下表用 lighteval 評測框架
| 模型 | MATH-500 | GPQA Diamond |
|---|---|---|
| meta-llama/Llama-3.2-3B-Instruct | 44.40 | 27.78 |
| twinkle-ai/Llama-3.2-3B-F1-Instruct (ours) | 51.40 | 33.84 |
Citation
@misc{twinkleai2025llama3.2f1,
title = {Llama-3.2-3B-F1-Instruct: A Traditional Chinese Instruction-Tuned Language Model for Taiwan},
author = {Huang, Liang Hsun and Chen, Min Yi and Lin, Wen Bin and Chuang, Chao Chun and Sung, Dave},
year = {2025},
howpublished = {\url{https://huggingface.co/twinkle-ai/Llama-3.2-3B-F1-Instruct}},
note = {Twinkle AI and APMIC. All authors contributed equally.}
}
Acknowledge
特此感謝國家高速網路與計算中心的指導與 APMIC 的算力支援，才得以讓本專案訓利完成。