MunicipalPredictionModel-TinyLlama

This model is a fine-tuned version of unsloth/tinyllama-chat-bnb-4bit on the generator dataset. It achieves the following results on the evaluation set:

• Loss: 1.3521

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0001
• train_batch_size: 8
• eval_batch_size: 8
• seed: 42
• gradient_accumulation_steps: 4
• total_train_batch_size: 32
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• num_epochs: 1

Training results

Training Loss	Epoch	Step	Validation Loss
3.8522	0.0377	10	3.0444
2.7085	0.0753	20	2.4069
2.191	0.1130	30	2.0027
1.8996	0.1507	40	1.8018
1.7538	0.1883	50	1.6850
1.6563	0.2260	60	1.6161
1.6275	0.2637	70	1.5722
1.5313	0.3013	80	1.5377
1.5277	0.3390	90	1.5104
1.5039	0.3766	100	1.4889
1.4768	0.4143	110	1.4694
1.4552	0.4520	120	1.4523
1.4481	0.4896	130	1.4385
1.4223	0.5273	140	1.4233
1.4145	0.5650	150	1.4118
1.4207	0.6026	160	1.4012
1.4004	0.6403	170	1.3925
1.4316	0.6780	180	1.3849
1.3841	0.7156	190	1.3784
1.3747	0.7533	200	1.3722
1.388	0.7910	210	1.3665
1.3508	0.8286	220	1.3623
1.386	0.8663	230	1.3586
1.3605	0.9040	240	1.3555
1.369	0.9416	250	1.3534
1.3645	0.9793	260	1.3521

Framework versions

• PEFT 0.12.0
• Transformers 4.43.4
• Pytorch 2.2.0+cu121
• Datasets 2.18.0
• Tokenizers 0.19.0