tinyllama-1.1b-sum-dpo-full_LR5e-8_BS32_2epochs_old

This model is a fine-tuned version of martimfasantos/tinyllama-1.1b-sum-sft-full_old on the openai/summarize_from_feedback dataset. It achieves the following results on the evaluation set:

• Loss: 0.6856
• Rewards/chosen: -0.0618
• Rewards/rejected: -0.0788
• Rewards/accuracies: 0.5955
• Rewards/margins: 0.0169
• Logps/rejected: -71.0584
• Logps/chosen: -64.8961
• Logits/rejected: -3.0381
• Logits/chosen: -3.0439

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 5e-08
• train_batch_size: 8
• eval_batch_size: 8
• seed: 42
• distributed_type: multi-GPU
• gradient_accumulation_steps: 4
• total_train_batch_size: 32
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: cosine
• lr_scheduler_warmup_ratio: 0.1
• num_epochs: 2

Training results

Training Loss	Epoch	Step	Validation Loss	Rewards/chosen	Rewards/rejected	Rewards/accuracies	Rewards/margins	Logps/rejected	Logps/chosen	Logits/rejected	Logits/chosen
0.6932	0.0345	100	0.6932	0.0000	0.0001	0.4805	-0.0001	-63.1716	-58.7091	-3.1575	-3.1632
0.6931	0.0689	200	0.6932	-0.0000	0.0000	0.4863	-0.0000	-63.1768	-58.7119	-3.1575	-3.1632
0.6931	0.1034	300	0.6932	0.0001	0.0002	0.4756	-0.0001	-63.1627	-58.7008	-3.1575	-3.1632
0.693	0.1378	400	0.6931	0.0002	0.0002	0.5007	0.0000	-63.1637	-58.6940	-3.1572	-3.1629
0.6931	0.1723	500	0.6931	0.0003	0.0002	0.4942	0.0001	-63.1590	-58.6825	-3.1569	-3.1625
0.6928	0.2068	600	0.6931	0.0006	0.0005	0.5023	0.0002	-63.1320	-58.6476	-3.1556	-3.1613
0.692	0.2412	700	0.6930	0.0010	0.0006	0.5414	0.0004	-63.1153	-58.6091	-3.1543	-3.1599
0.6923	0.2757	800	0.6928	0.0013	0.0006	0.5588	0.0007	-63.1219	-58.5861	-3.1529	-3.1585
0.6912	0.3101	900	0.6927	0.0017	0.0007	0.5660	0.0010	-63.1103	-58.5464	-3.1501	-3.1558
0.6909	0.3446	1000	0.6925	0.0018	0.0005	0.5646	0.0013	-63.1285	-58.5271	-3.1481	-3.1538
0.6907	0.3790	1100	0.6924	0.0020	0.0003	0.5604	0.0016	-63.1469	-58.5154	-3.1457	-3.1513
0.6898	0.4135	1200	0.6921	0.0018	-0.0003	0.5743	0.0022	-63.2143	-58.5306	-3.1424	-3.1480
0.688	0.4480	1300	0.6919	0.0018	-0.0008	0.5741	0.0026	-63.2606	-58.5351	-3.1392	-3.1448
0.6888	0.4824	1400	0.6917	0.0011	-0.0019	0.5723	0.0030	-63.3749	-58.6054	-3.1364	-3.1420
0.6886	0.5169	1500	0.6915	0.0002	-0.0033	0.5737	0.0035	-63.5057	-58.6878	-3.1325	-3.1382
0.6885	0.5513	1600	0.6912	-0.0003	-0.0043	0.5769	0.0040	-63.6057	-58.7407	-3.1295	-3.1351
0.6861	0.5858	1700	0.6910	-0.0016	-0.0062	0.5746	0.0046	-63.8004	-58.8729	-3.1253	-3.1310
0.6872	0.6203	1800	0.6908	-0.0035	-0.0085	0.5839	0.0050	-64.0325	-59.0604	-3.1214	-3.1270
0.6862	0.6547	1900	0.6905	-0.0054	-0.0110	0.5802	0.0057	-64.2826	-59.2489	-3.1157	-3.1214
0.6859	0.6892	2000	0.6903	-0.0080	-0.0142	0.5869	0.0062	-64.5982	-59.5137	-3.1119	-3.1176
0.6846	0.7236	2100	0.6899	-0.0107	-0.0176	0.5829	0.0069	-64.9428	-59.7842	-3.1059	-3.1116
0.6861	0.7581	2200	0.6897	-0.0133	-0.0207	0.5869	0.0074	-65.2491	-60.0455	-3.1025	-3.1081
0.6836	0.7926	2300	0.6895	-0.0168	-0.0247	0.5922	0.0079	-65.6530	-60.3904	-3.0987	-3.1044
0.6847	0.8270	2400	0.6892	-0.0209	-0.0296	0.5869	0.0087	-66.1402	-60.8069	-3.0949	-3.1007
0.6838	0.8615	2500	0.6889	-0.0250	-0.0343	0.5904	0.0093	-66.6113	-61.2157	-3.0910	-3.0968
0.6841	0.8959	2600	0.6886	-0.0284	-0.0384	0.5955	0.0100	-67.0226	-61.5496	-3.0877	-3.0933
0.6824	0.9304	2700	0.6883	-0.0321	-0.0428	0.5855	0.0107	-67.4593	-61.9186	-3.0839	-3.0897
0.6824	0.9649	2800	0.6880	-0.0334	-0.0447	0.5929	0.0113	-67.6515	-62.0566	-3.0811	-3.0868
0.6812	0.9993	2900	0.6878	-0.0363	-0.0481	0.5906	0.0118	-67.9890	-62.3425	-3.0775	-3.0832
0.6819	1.0338	3000	0.6877	-0.0373	-0.0494	0.5932	0.0120	-68.1166	-62.4440	-3.0740	-3.0797
0.6796	1.0682	3100	0.6874	-0.0392	-0.0518	0.5987	0.0126	-68.3560	-62.6296	-3.0701	-3.0759
0.6776	1.1027	3200	0.6872	-0.0409	-0.0540	0.5906	0.0131	-68.5819	-62.8043	-3.0674	-3.0732
0.6824	1.1371	3300	0.6870	-0.0436	-0.0571	0.5946	0.0135	-68.8899	-63.0750	-3.0643	-3.0701
0.6787	1.1716	3400	0.6869	-0.0458	-0.0596	0.5941	0.0138	-69.1415	-63.2913	-3.0611	-3.0668
0.6801	1.2061	3500	0.6867	-0.0482	-0.0624	0.5929	0.0142	-69.4185	-63.5317	-3.0588	-3.0646
0.6797	1.2405	3600	0.6866	-0.0499	-0.0644	0.5915	0.0145	-69.6206	-63.6998	-3.0559	-3.0616
0.6783	1.2750	3700	0.6864	-0.0511	-0.0659	0.5904	0.0149	-69.7728	-63.8172	-3.0542	-3.0599
0.6771	1.3094	3800	0.6864	-0.0521	-0.0672	0.5920	0.0151	-69.8981	-63.9235	-3.0522	-3.0580
0.6785	1.3439	3900	0.6862	-0.0536	-0.0690	0.5922	0.0154	-70.0814	-64.0693	-3.0499	-3.0556
0.6807	1.3784	4000	0.6861	-0.0551	-0.0708	0.5908	0.0157	-70.2593	-64.2214	-3.0484	-3.0541
0.6769	1.4128	4100	0.6860	-0.0563	-0.0722	0.5929	0.0159	-70.3988	-64.3376	-3.0467	-3.0525
0.6722	1.4473	4200	0.6859	-0.0577	-0.0738	0.5946	0.0161	-70.5629	-64.4845	-3.0456	-3.0513
0.6769	1.4817	4300	0.6858	-0.0582	-0.0745	0.5939	0.0163	-70.6349	-64.5350	-3.0442	-3.0499
0.6785	1.5162	4400	0.6858	-0.0586	-0.0750	0.5955	0.0164	-70.6776	-64.5703	-3.0432	-3.0490
0.6735	1.5507	4500	0.6858	-0.0597	-0.0762	0.5920	0.0164	-70.7972	-64.6853	-3.0421	-3.0479
0.6786	1.5851	4600	0.6857	-0.0603	-0.0769	0.5967	0.0166	-70.8698	-64.7462	-3.0414	-3.0471
0.6803	1.6196	4700	0.6857	-0.0603	-0.0770	0.5978	0.0167	-70.8781	-64.7435	-3.0408	-3.0466
0.6789	1.6540	4800	0.6856	-0.0607	-0.0775	0.5929	0.0168	-70.9263	-64.7804	-3.0399	-3.0457
0.6723	1.6885	4900	0.6856	-0.0611	-0.0779	0.5985	0.0168	-70.9741	-64.8213	-3.0390	-3.0448
0.6767	1.7229	5000	0.6856	-0.0613	-0.0781	0.5960	0.0169	-70.9925	-64.8377	-3.0388	-3.0446
0.6774	1.7574	5100	0.6856	-0.0615	-0.0784	0.5939	0.0168	-71.0176	-64.8661	-3.0387	-3.0445
0.6748	1.7919	5200	0.6855	-0.0616	-0.0786	0.5939	0.0170	-71.0377	-64.8736	-3.0383	-3.0441
0.6761	1.8263	5300	0.6855	-0.0617	-0.0787	0.5950	0.0170	-71.0469	-64.8778	-3.0380	-3.0439
0.6738	1.8608	5400	0.6855	-0.0618	-0.0788	0.5985	0.0171	-71.0633	-64.8885	-3.0380	-3.0438
0.6821	1.8952	5500	0.6855	-0.0618	-0.0788	0.5934	0.0170	-71.0638	-64.8919	-3.0379	-3.0437
0.6724	1.9297	5600	0.6855	-0.0619	-0.0788	0.5955	0.0170	-71.0635	-64.8979	-3.0379	-3.0437
0.6745	1.9642	5700	0.6855	-0.0619	-0.0790	0.5957	0.0171	-71.0788	-64.9037	-3.0380	-3.0438
0.6767	1.9986	5800	0.6856	-0.0618	-0.0788	0.5955	0.0169	-71.0584	-64.8961	-3.0381	-3.0439

Framework versions

• Transformers 4.41.2
• Pytorch 2.1.2
• Datasets 2.19.2
• Tokenizers 0.19.1