meng-lab/codellama_34b_instruct_paradec_gsm8k

CodeLlama-34b-Instruct-sft-5e-3-epoch-100-gsm8k

This model is a fine-tuned version of meta-llama/CodeLlama-34b-Instruct-hf on the meng-lab/CodeLlama-34B-Instruct-gsm8k dataset. It achieves the following results on the evaluation set:

• Loss: 4.0230
• Loss Layer 6 Head: 1.2898
• Loss Layer 12 Head: 1.0049
• Loss Layer 18 Head: 0.9093
• Loss Layer 24 Head: 0.4408
• Loss Layer 30 Head: 0.2683
• Loss Layer 36 Head: 0.1391
• Loss Layer 42 Head: 0.0639

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.005
• train_batch_size: 1
• eval_batch_size: 2
• seed: 42
• distributed_type: multi-GPU
• num_devices: 8
• gradient_accumulation_steps: 16
• total_train_batch_size: 128
• total_eval_batch_size: 16
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: cosine
• lr_scheduler_warmup_ratio: 0.1
• num_epochs: 100

Training results

Training Loss	Epoch	Step	Validation Loss	Loss Layer 6 Head	Loss Layer 12 Head	Loss Layer 18 Head	Loss Layer 24 Head	Loss Layer 30 Head	Loss Layer 36 Head	Loss Layer 42 Head
2.6241	25.8065	200	4.3768	1.3707	1.0927	0.9492	0.4907	0.2888	0.1534	0.0899
1.6189	51.6129	400	4.0476	1.3067	0.9916	0.9104	0.4445	0.2716	0.1405	0.0663
1.3737	77.4194	600	4.0230	1.2898	1.0049	0.9093	0.4408	0.2683	0.1391	0.0639

Framework versions

• Transformers 4.43.2
• Pytorch 2.1.2
• Datasets 3.2.0
• Tokenizers 0.19.1