lmind_nq_train5000_eval5000_v1_recite_qa_gpt2-xl

This model is a fine-tuned version of gpt2-xl on the tyzhu/lmind_nq_train5000_eval5000_v1_recite_qa dataset. It achieves the following results on the evaluation set:

• Loss: 0.3682
• Accuracy: 0.8792

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 3e-05
• train_batch_size: 16
• eval_batch_size: 16
• seed: 42
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: constant
• num_epochs: 20.0

Training results

Training Loss	Epoch	Step	Accuracy	Validation Loss
2.1509	1.0	873	0.7132	1.7318
1.4733	2.0	1746	0.7619	1.2526
0.956	3.0	2619	0.8076	0.8630
0.6185	4.0	3492	0.8424	0.5905
0.4012	5.0	4365	0.8635	0.4375
0.2724	6.0	5238	0.8726	0.3726
0.1944	7.0	6111	0.8762	0.3524
0.1556	8.0	6984	0.8777	0.3441
0.1291	9.0	7857	0.8783	0.3419
0.1158	10.0	8730	0.8786	0.3441
0.1085	11.0	9603	0.8789	0.3491
0.1023	12.0	10476	0.8790	0.3513
0.0999	13.0	11349	0.8789	0.3513
0.0972	14.0	12222	0.8792	0.3525
0.0935	15.0	13095	0.8791	0.3588
0.0917	16.0	13968	0.8787	0.3614
0.0894	17.0	14841	0.8793	0.3578
0.0882	18.0	15714	0.8794	0.3630
0.0849	19.0	16587	0.8793	0.3639
0.0845	20.0	17460	0.8792	0.3682

Framework versions

• Transformers 4.34.0
• Pytorch 2.1.0+cu121
• Datasets 2.14.5
• Tokenizers 0.14.1