collapse_gemma-2-27b_hs2_replace_iter4_sftsd0

This model is a fine-tuned version of google/gemma-2-27b on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 1.3266
• Num Input Tokens Seen: 4030192

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 8e-06
• train_batch_size: 4
• eval_batch_size: 16
• seed: 0
• gradient_accumulation_steps: 32
• total_train_batch_size: 128
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: constant_with_warmup
• lr_scheduler_warmup_ratio: 0.05
• num_epochs: 1

Training results

Training Loss	Epoch	Step	Validation Loss	Input Tokens Seen
No log	0	0	1.1282	0
3.9014	0.0694	5	1.0629	278000
3.6407	0.1389	10	1.1530	559276
3.5129	0.2083	15	1.2057	839944
3.365	0.2778	20	1.2553	1123192
3.2314	0.3472	25	1.2937	1402000
3.0661	0.4167	30	1.3170	1675956
2.9861	0.4861	35	1.3117	1953884
2.9771	0.5556	40	1.3107	2238624
2.7982	0.625	45	1.3116	2521396
2.8816	0.6944	50	1.3106	2802856
2.8309	0.7639	55	1.3189	3077316
2.7512	0.8333	60	1.3098	3358448
2.7935	0.9028	65	1.3161	3643588
2.8647	0.9722	70	1.3220	3923824

Framework versions

• Transformers 4.44.0
• Pytorch 2.4.0+cu121
• Datasets 2.20.0
• Tokenizers 0.19.1