mesolitica/Malaysian-Llama-3.1-8B-Instruct-v0.1

Malaysian Llama-3.1 8B-Instruct v0.1

Continue finetuning meta-llama/Llama-3.1-8B-Instruct on highly curated 1.2B tokens Malaysian instruction.

Improvement

1. 128k context length.
2. Support respond in Mandarin, Tamil, Jawi, Manglish, Johor, Kedah, Kelantan, Pahang, Perak, Sabah, Sarawak, Selangor, Negeri Sembilan and Terengganu.
3. Able to code in Mandarin, Tamil, Jawi, Manglish, Johor, Kedah, Kelantan, Pahang, Perak, Sabah, Sarawak, Selangor, Negeri Sembilan and Terengganu.
4. Multi-turn Malaysian context such as related to Malaysian Legislation, politics, religions and languages.
5. Standard RAG.

MalayMMLU

                             Model   Accuracy   shot by_letter        category
0  Malaysian-Llama-3.1-8B-Instruct  61.768318  0shot      True            STEM
1  Malaysian-Llama-3.1-8B-Instruct  62.420483  0shot      True        Language
2  Malaysian-Llama-3.1-8B-Instruct  60.291992  0shot      True  Social science
3  Malaysian-Llama-3.1-8B-Instruct  59.270808  0shot      True          Others
4  Malaysian-Llama-3.1-8B-Instruct  62.366325  0shot      True      Humanities
{'Social science': 6918, 'Language': 6288, 'Humanities': 4395, 'Others': 4169, 'STEM': 2443}
Model : Malaysian-Llama-3.1-8B-Instruct
Metric : first
Shot : 0shot
average accuracy 61.194399702639075
accuracy for STEM 61.76831764224314
accuracy for Language 62.420483460559794
accuracy for Social science 60.2919919051749
accuracy for Others 59.2708083473255
accuracy for Humanities 62.36632536973834

Training session

Finetune on mesolitica/Malaysian-SFT to make the model understand Malaysian context.

How we train

1. LoRA on ["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj", "embed_tokens", "lm_head"].
2. 256 Rank with alpha 512, or alpha of 2.0
3. Multipacking with proper SDPA causal masking to prevent document contamination and also make sure proper position ids.
4. Forked CCE loss for LoRA lm_head to reduce memory consumption.

Source code at https://github.com/malaysia-ai/cooking/tree/main/llama/sft