SmolHausaLM-135M

This model is a fine-tuned version of HuggingFaceTB/SmolLM2-135M-Instruct on the None dataset. It achieves the following results on the evaluation set:

• Loss: 9.3567

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0002
• train_batch_size: 10
• eval_batch_size: 10
• seed: 42
• gradient_accumulation_steps: 5
• total_train_batch_size: 50
• optimizer: Use adamw_torch with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: cosine
• lr_scheduler_warmup_ratio: 0.1
• num_epochs: 5

Training results

Training Loss	Epoch	Step	Validation Loss
44.2818	0.3526	100	9.5091
37.501	0.7052	200	8.9090
36.3613	1.0599	300	8.8260
34.3793	1.4126	400	8.9936
34.1721	1.7652	500	8.9672
33.3041	2.1199	600	9.0472
31.0766	2.4725	700	9.0407
30.7626	2.8251	800	9.1113
28.3702	3.1798	900	9.2313
25.6234	3.5324	1000	9.2606
25.4011	3.8850	1100	9.2470
22.6147	4.2398	1200	9.3353
21.252	4.5924	1300	9.3529
21.2066	4.9450	1400	9.3567

Framework versions

• Transformers 4.47.0
• Pytorch 2.5.1+cu121
• Datasets 3.3.1
• Tokenizers 0.21.0