--- language: - ms - en - zh - ta --- # Malaysian Llama-3.2-3B-Instruct Continue finetuning https://huggingface.co/meta-llama/Llama-3.2-3B-Instruct on highly curated 1.5B tokens Malaysian instruction dataset. ## Improvement 1. Support respond in Mandarin, Tamil, Jawi, Manglish, Johor, Kedah, Kelantan, Pahang, Perak, Sabah, Sarawak, Selangor, Negeri Sembilan and Terengganu. 2. Able to code in Mandarin, Tamil, Jawi, Manglish, Johor, Kedah, Kelantan, Pahang, Perak, Sabah, Sarawak, Selangor, Negeri Sembilan and Terengganu. 3. Multi-turn Malaysian context such as related to Malaysian Legislation, politics, religions and languages. ## Training session Finetune on [mesolitica/Malaysian-SFT](https://huggingface.co/datasets/mesolitica/Malaysian-SFT) to make the model understand Malaysian context. ## How we train 1. LoRA on `["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj", "embed_tokens", "lm_head"]`. 2. 128 Rank with alpha 256, or alpha of 2.0 3. Multipacking 8192 context length with proper SDPA causal masking to prevent document contamination and also make sure proper position ids. 4. Chunk CCE loss for LoRA. 5. WanDB at https://wandb.ai/huseinzol05/lora-embedding-128-llama3.2-3b-malaysian-8k?nw=nwuserhuseinzol05 Source code at https://github.com/mesolitica/malaya/tree/master/session/llama3 ## Benchmark ### MalayMMLU #### Probability next tokens Based on 0-shot official MalayMMLU First token accuracy, ``` Model Accuracy shot by_letter category 0 Malaysian-Llama-3.2-3B-Instruct 57.634056 0shot True STEM 1 Malaysian-Llama-3.2-3B-Instruct 59.351145 0shot True Language 2 Malaysian-Llama-3.2-3B-Instruct 57.559988 0shot True Social science 3 Malaysian-Llama-3.2-3B-Instruct 57.303910 0shot True Others 4 Malaysian-Llama-3.2-3B-Instruct 60.022753 0shot True Humanities {'Social science': 6918, 'Language': 6288, 'Humanities': 4395, 'Others': 4169, 'STEM': 2443} Model : Malaysian-Llama-3.2-3B-Instruct Metric : first Shot : 0shot average accuracy 58.43555115020857 accuracy for STEM 57.63405648792468 accuracy for Language 59.35114503816794 accuracy for Social science 57.55998843596415 accuracy for Others 57.30390981050611 accuracy for Humanities 60.02275312855517 ``` While the original model, ``` Model Accuracy shot by_letter category 0 Llama-3.2-3B-Instruct 56.733524 0shot True STEM 1 Llama-3.2-3B-Instruct 58.460560 0shot True Language 2 Llama-3.2-3B-Instruct 54.206418 0shot True Social science 3 Llama-3.2-3B-Instruct 52.554569 0shot True Others 4 Llama-3.2-3B-Instruct 60.659841 0shot True Humanities {'Social science': 6918, 'Language': 6288, 'Humanities': 4395, 'Others': 4169, 'STEM': 2443} Model : Llama-3.2-3B-Instruct Metric : first Shot : 0shot average accuracy 56.453145004749516 accuracy for STEM 56.73352435530086 accuracy for Language 58.460559796437664 accuracy for Social science 54.20641803989592 accuracy for Others 52.554569441112974 accuracy for Humanities 60.659840728100114 ``` #### First token match using vLLM Based on 0-shot exact first token match using vLLM, ``` Model Accuracy shot category 0 Malaysian-Llama-3.2-3B-Instruct 51.944331 0 STEM 1 Malaysian-Llama-3.2-3B-Instruct 50.795165 0 Language 2 Malaysian-Llama-3.2-3B-Instruct 52.732003 0 Social science 3 Malaysian-Llama-3.2-3B-Instruct 52.026865 0 Others 4 Malaysian-Llama-3.2-3B-Instruct 54.539249 0 Humanities Model : Malaysian-Llama-3.2-3B-Instruct Metric : full Shot : 0 average accuracy 52.35617230413414 accuracy for STEM 51.94433074089234 accuracy for Language 50.795165394402034 accuracy for Social science 52.73200346921075 accuracy for Others 52.02686495562485 accuracy for Humanities 54.53924914675768 ``` While the original model, ``` Model Accuracy shot category 0 Llama-3.2-3B-Instruct 50.511666 0 STEM 1 Llama-3.2-3B-Instruct 49.825064 0 Language 2 Llama-3.2-3B-Instruct 48.352125 0 Social science 3 Llama-3.2-3B-Instruct 48.213001 0 Others 4 Llama-3.2-3B-Instruct 51.990899 0 Humanities Model : Llama-3.2-3B-Instruct Metric : full Shot : 0 average accuracy 49.58906372609755 accuracy for STEM 50.51166598444535 accuracy for Language 49.82506361323155 accuracy for Social science 48.35212489158716 accuracy for Others 48.21300071959703 accuracy for Humanities 51.990898748577926 ``` ## Acknowledgement Special thanks to https://www.sns.com.my and Nvidia for 8x H100 node!