File size: 4,755 Bytes
3a32b64 c3c6a9c 3a32b64 c3c6a9c 3a32b64 c3c6a9c 3a32b64 c3c6a9c 3a32b64 c3c6a9c 3a32b64 c3c6a9c 3a32b64 c3c6a9c 3a32b64 c3c6a9c 3a32b64 c3c6a9c 3a32b64 3246fe7 848415d 537e3ab 3246fe7 537e3ab 3246fe7 537e3ab 3246fe7 c3c6a9c 3a32b64 8aab157 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 |
---
language:
- ms
- en
- zh
- ta
---
# Malaysian Llama-3.2-3B-Instruct
Continue finetuning https://huggingface.co/meta-llama/Llama-3.2-3B-Instruct on highly curated 1.5B tokens Malaysian instruction dataset.
## Improvement
1. Support respond in Mandarin, Tamil, Jawi, Manglish, Johor, Kedah, Kelantan, Pahang, Perak, Sabah, Sarawak, Selangor, Negeri Sembilan and Terengganu.
2. Able to code in Mandarin, Tamil, Jawi, Manglish, Johor, Kedah, Kelantan, Pahang, Perak, Sabah, Sarawak, Selangor, Negeri Sembilan and Terengganu.
3. Multi-turn Malaysian context such as related to Malaysian Legislation, politics, religions and languages.
## Training session
Finetune on [mesolitica/Malaysian-SFT](https://huggingface.co/datasets/mesolitica/Malaysian-SFT) to make the model understand Malaysian context.
## How we train
1. LoRA on `["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj", "embed_tokens", "lm_head"]`.
2. 128 Rank with alpha 256, or alpha of 2.0
3. Multipacking 8192 context length with proper SDPA causal masking to prevent document contamination and also make sure proper position ids.
4. Chunk CCE loss for LoRA.
5. WanDB at https://wandb.ai/huseinzol05/lora-embedding-128-llama3.2-3b-malaysian-8k?nw=nwuserhuseinzol05
Source code at https://github.com/mesolitica/malaya/tree/master/session/llama3
## Benchmark
### MalayMMLU
#### Probability next tokens
Based on 0-shot official MalayMMLU First token accuracy,
```
Model Accuracy shot by_letter category
0 Malaysian-Llama-3.2-3B-Instruct 57.634056 0shot True STEM
1 Malaysian-Llama-3.2-3B-Instruct 59.351145 0shot True Language
2 Malaysian-Llama-3.2-3B-Instruct 57.559988 0shot True Social science
3 Malaysian-Llama-3.2-3B-Instruct 57.303910 0shot True Others
4 Malaysian-Llama-3.2-3B-Instruct 60.022753 0shot True Humanities
{'Social science': 6918, 'Language': 6288, 'Humanities': 4395, 'Others': 4169, 'STEM': 2443}
Model : Malaysian-Llama-3.2-3B-Instruct
Metric : first
Shot : 0shot
average accuracy 58.43555115020857
accuracy for STEM 57.63405648792468
accuracy for Language 59.35114503816794
accuracy for Social science 57.55998843596415
accuracy for Others 57.30390981050611
accuracy for Humanities 60.02275312855517
```
While the original model,
```
Model Accuracy shot by_letter category
0 Llama-3.2-3B-Instruct 56.733524 0shot True STEM
1 Llama-3.2-3B-Instruct 58.460560 0shot True Language
2 Llama-3.2-3B-Instruct 54.206418 0shot True Social science
3 Llama-3.2-3B-Instruct 52.554569 0shot True Others
4 Llama-3.2-3B-Instruct 60.659841 0shot True Humanities
{'Social science': 6918, 'Language': 6288, 'Humanities': 4395, 'Others': 4169, 'STEM': 2443}
Model : Llama-3.2-3B-Instruct
Metric : first
Shot : 0shot
average accuracy 56.453145004749516
accuracy for STEM 56.73352435530086
accuracy for Language 58.460559796437664
accuracy for Social science 54.20641803989592
accuracy for Others 52.554569441112974
accuracy for Humanities 60.659840728100114
```
#### First token match using vLLM
Based on 0-shot exact first token match using vLLM,
```
Model Accuracy shot category
0 Malaysian-Llama-3.2-3B-Instruct 51.944331 0 STEM
1 Malaysian-Llama-3.2-3B-Instruct 50.795165 0 Language
2 Malaysian-Llama-3.2-3B-Instruct 52.732003 0 Social science
3 Malaysian-Llama-3.2-3B-Instruct 52.026865 0 Others
4 Malaysian-Llama-3.2-3B-Instruct 54.539249 0 Humanities
Model : Malaysian-Llama-3.2-3B-Instruct
Metric : full
Shot : 0
average accuracy 52.35617230413414
accuracy for STEM 51.94433074089234
accuracy for Language 50.795165394402034
accuracy for Social science 52.73200346921075
accuracy for Others 52.02686495562485
accuracy for Humanities 54.53924914675768
```
While the original model,
```
Model Accuracy shot category
0 Llama-3.2-3B-Instruct 50.511666 0 STEM
1 Llama-3.2-3B-Instruct 49.825064 0 Language
2 Llama-3.2-3B-Instruct 48.352125 0 Social science
3 Llama-3.2-3B-Instruct 48.213001 0 Others
4 Llama-3.2-3B-Instruct 51.990899 0 Humanities
Model : Llama-3.2-3B-Instruct
Metric : full
Shot : 0
average accuracy 49.58906372609755
accuracy for STEM 50.51166598444535
accuracy for Language 49.82506361323155
accuracy for Social science 48.35212489158716
accuracy for Others 48.21300071959703
accuracy for Humanities 51.990898748577926
```
## Acknowledgement
Special thanks to https://www.sns.com.my and Nvidia for 8x H100 node! |