Momorami/medusa-vicuna_1b_stage1

vicuna_1b_stage1

This model is a fine-tuned version of Jiayi-Pan/Tiny-Vicuna-1B on the Aeala/ShareGPT_Vicuna_unfiltered dataset. It achieves the following results on the evaluation set:

• Loss: 2.9673

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0005
• train_batch_size: 2
• eval_batch_size: 2
• seed: 42
• gradient_accumulation_steps: 4
• total_train_batch_size: 8
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: cosine
• lr_scheduler_warmup_steps: 40
• num_epochs: 1

Training results

Training Loss	Epoch	Step	Validation Loss
3.3461	0.0170	40	3.5421
3.2004	0.0340	80	3.1581
3.0095	0.0510	120	3.0506
2.714	0.0681	160	3.0168
2.9508	0.0851	200	2.9764
2.9774	0.1021	240	2.9598
2.8688	0.1191	280	2.9551
2.8195	0.1361	320	2.9420
2.8471	0.1531	360	2.9328
2.9252	0.1701	400	2.9673

Framework versions

• Transformers 4.43.0
• Pytorch 2.3.1
• Datasets 2.15.0
• Tokenizers 0.19.1