btqa/btqa-base

app/bt_qa-out

This model is a fine-tuned version of cerebras/btlm-3b-8k-base on the None dataset. It achieves the following results on the evaluation set:

• Loss: 1.7451

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 8.5e-05
• train_batch_size: 3
• eval_batch_size: 3
• seed: 42
• optimizer: Adam with betas=(0.9,0.95) and epsilon=1e-08
• lr_scheduler_type: cosine
• lr_scheduler_warmup_steps: 32
• num_epochs: 1

Training results

Training Loss	Epoch	Step	Validation Loss
2.7377	0.0	500	2.5700
2.5689	0.01	1000	2.4172
2.6676	0.01	1500	2.3622
2.4234	0.01	2000	2.3293
2.2221	0.01	2500	2.3090
2.3873	0.02	3000	2.2940
2.2688	0.02	3500	2.2644
2.4194	0.02	4000	2.2725
2.4086	0.03	4500	2.2561
2.4476	0.03	5000	2.2477
2.1512	0.03	5500	2.2330
2.1428	0.03	6000	2.2235
2.2834	0.04	6500	2.2141
2.2918	0.04	7000	2.2124
2.4352	0.04	7500	2.2074
1.7196	0.05	8000	2.2038
2.2394	0.05	8500	2.1973
2.1632	0.05	9000	2.1856
2.4313	0.06	9500	2.1820
2.4584	0.06	10000	2.1764
2.3359	0.06	10500	2.1780
2.2105	0.06	11000	2.1671
2.3152	0.07	11500	2.1603
2.3012	0.07	12000	2.1572
2.4636	0.07	12500	2.1553
2.0974	0.08	13000	2.1511
2.298	0.08	13500	2.1481
2.3312	0.08	14000	2.1445
2.5315	0.08	14500	2.1381
2.1854	0.09	15000	2.1364
2.3069	0.09	15500	2.1355
2.0756	0.09	16000	2.1331
2.0094	0.1	16500	2.1306
2.2674	0.1	17000	2.1230
1.8427	0.1	17500	2.1176
2.2277	0.1	18000	2.1168
2.1398	0.11	18500	2.1152
1.9927	0.11	19000	2.1088
2.0119	0.11	19500	2.1105
2.5796	0.12	20000	2.1040
1.3256	0.12	20500	2.0993
2.2051	0.12	21000	2.0992
1.628	0.13	21500	2.0944
2.1926	0.13	22000	2.0927
1.6482	0.13	22500	2.0873
2.1122	0.13	23000	2.0830
1.7405	0.14	23500	2.0828
2.2685	0.14	24000	2.0784
2.1062	0.14	24500	2.0766
2.1308	0.15	25000	2.0714
1.9122	0.15	25500	2.0719
2.3549	0.15	26000	2.0643
2.2159	0.15	26500	2.0655
1.493	0.16	27000	2.0598
1.893	0.16	27500	2.0557
2.1902	0.16	28000	2.0533
2.2353	0.17	28500	2.0524
1.8736	0.17	29000	2.0519
2.0511	0.17	29500	2.0449
1.2872	0.17	30000	2.0453
1.6353	0.18	30500	2.0377
1.992	0.18	31000	2.0419
2.3586	0.18	31500	2.0353
1.9453	0.19	32000	2.0330
2.1322	0.19	32500	2.0305
2.2887	0.19	33000	2.0253
2.0268	0.2	33500	2.0267
1.8397	0.2	34000	2.0207
2.5165	0.2	34500	2.0202
1.9142	0.2	35000	2.0139
1.5993	0.21	35500	2.0179
2.1691	0.21	36000	2.0102
2.4948	0.21	36500	2.0089
1.5422	0.22	37000	2.0039
1.4566	0.22	37500	2.0014
1.852	0.22	38000	2.0043
2.199	0.22	38500	1.9987
1.4852	0.23	39000	1.9976
1.3	0.23	39500	1.9936
2.1237	0.23	40000	1.9917
1.691	0.24	40500	1.9887
2.2169	0.24	41000	1.9870
2.1991	0.24	41500	1.9851
1.9517	0.24	42000	1.9806
1.6369	0.25	42500	1.9762
2.2759	0.25	43000	1.9753
2.2923	0.25	43500	1.9748
2.2552	0.26	44000	1.9702
2.066	0.26	44500	1.9683
2.2703	0.26	45000	1.9686
2.3544	0.27	45500	1.9648
2.255	0.27	46000	1.9635
1.8732	0.27	46500	1.9639
2.1203	0.27	47000	1.9590
2.1314	0.28	47500	1.9573
1.8511	0.28	48000	1.9533
2.1471	0.28	48500	1.9514
1.8417	0.29	49000	1.9509
2.4485	0.29	49500	1.9502
2.0708	0.29	50000	1.9455
1.8272	0.29	50500	1.9416
1.6232	0.3	51000	1.9380
1.6785	0.3	51500	1.9358
1.5734	0.3	52000	1.9313
1.9737	0.31	52500	1.9301
1.8393	0.31	53000	1.9295
1.4789	0.31	53500	1.9281
2.2062	0.31	54000	1.9273
2.3501	0.32	54500	1.9236
2.2756	0.32	55000	1.9218
2.1001	0.32	55500	1.9215
2.0342	0.33	56000	1.9179
1.8066	0.33	56500	1.9143
1.8322	0.33	57000	1.9137
2.0926	0.34	57500	1.9106
2.2106	0.34	58000	1.9083
2.0666	0.34	58500	1.9055
2.2082	0.34	59000	1.9026
2.1768	0.35	59500	1.9007
1.7091	0.35	60000	1.8967
1.7585	0.35	60500	1.8946
1.8968	0.36	61000	1.8936
2.107	0.36	61500	1.8906
1.5162	0.36	62000	1.8870
2.0642	0.36	62500	1.8836
2.0399	0.37	63000	1.8813
2.3971	0.37	63500	1.8785
1.7433	0.37	64000	1.8797
2.0971	0.38	64500	1.8743
1.8212	0.38	65000	1.8726
2.1023	0.38	65500	1.8695
1.9735	0.38	66000	1.8674
1.3196	0.39	66500	1.8657
1.9825	0.39	67000	1.8629
2.0356	0.39	67500	1.8604
1.8522	0.4	68000	1.8581
2.2666	0.4	68500	1.8568
2.3575	0.4	69000	1.8538
2.0086	0.41	69500	1.8537
1.9811	0.41	70000	1.8512
2.0702	0.41	70500	1.8485
1.8554	0.41	71000	1.8456
0.5356	0.42	71500	1.8437
1.4742	0.42	72000	1.8413
2.1901	0.42	72500	1.8420
1.7868	0.43	73000	1.8383
1.3144	0.43	73500	1.8371
2.1158	0.43	74000	1.8347
2.0779	0.43	74500	1.8331
1.9756	0.44	75000	1.8323
2.3395	0.44	75500	1.8309
1.895	0.44	76000	1.8283
2.0369	0.45	76500	1.8274
1.8068	0.45	77000	1.8251
2.2153	0.45	77500	1.8227
2.1389	0.45	78000	1.8212
1.9166	0.46	78500	1.8197
1.711	0.46	79000	1.8187
1.9102	0.46	79500	1.8165
0.8358	0.47	80000	1.8163
1.7278	0.47	80500	1.8148
1.601	0.47	81000	1.8126
1.9794	0.48	81500	1.8107
1.7323	0.48	82000	1.8095
2.2911	0.48	82500	1.8090
1.8962	0.48	83000	1.8065
2.3055	0.49	83500	1.8052
1.6899	0.49	84000	1.8037
1.6409	0.49	84500	1.8031
1.9116	0.5	85000	1.8011
0.6875	0.5	85500	1.8003
2.0829	0.5	86000	1.7983
1.5716	0.5	86500	1.7981
2.4537	0.51	87000	1.7961
1.8236	0.51	87500	1.7942
1.641	0.51	88000	1.7931
1.5533	0.52	88500	1.7916
1.679	0.52	89000	1.7902
2.1463	0.52	89500	1.7893
1.5477	0.52	90000	1.7884
1.2346	0.53	90500	1.7873
1.3352	0.53	91000	1.7859
2.1039	0.53	91500	1.7850
2.0818	0.54	92000	1.7834
1.3987	0.54	92500	1.7830
1.4544	0.54	93000	1.7827
0.4043	0.55	93500	1.7811
2.0149	0.55	94000	1.7794
1.9845	0.55	94500	1.7789
2.1053	0.55	95000	1.7775
2.1572	0.56	95500	1.7768
2.0754	0.56	96000	1.7761
1.7675	0.56	96500	1.7754
2.0023	0.57	97000	1.7743
1.2653	0.57	97500	1.7736
1.5566	0.57	98000	1.7728
1.9408	0.57	98500	1.7724
2.0936	0.58	99000	1.7713
0.5687	0.58	99500	1.7706
2.2833	0.58	100000	1.7702
1.6689	0.59	100500	1.7690
1.5198	0.59	101000	1.7684
1.6968	0.59	101500	1.7679
2.2034	0.59	102000	1.7674
1.7902	0.6	102500	1.7665
2.0557	0.6	103000	1.7658
1.8617	0.6	103500	1.7650
1.8749	0.61	104000	1.7637
1.7674	0.61	104500	1.7632
1.4269	0.61	105000	1.7627
1.989	0.62	105500	1.7621
2.1026	0.62	106000	1.7615
2.0304	0.62	106500	1.7609
1.6286	0.62	107000	1.7603
0.9544	0.63	107500	1.7599
1.6421	0.63	108000	1.7588
1.9841	0.63	108500	1.7586
1.7453	0.64	109000	1.7581
1.2119	0.64	109500	1.7575
2.1092	0.64	110000	1.7568
2.0849	0.64	110500	1.7564
1.9162	0.65	111000	1.7562
1.01	0.65	111500	1.7560
1.301	0.65	112000	1.7556
0.315	0.66	112500	1.7552
1.9964	0.66	113000	1.7548
2.4035	0.66	113500	1.7544
1.3559	0.66	114000	1.7542
2.1874	0.67	114500	1.7538
1.4373	0.67	115000	1.7534
0.0639	0.67	115500	1.7529
1.7667	0.68	116000	1.7526
1.6204	0.68	116500	1.7524
1.9859	0.68	117000	1.7521
0.9717	0.69	117500	1.7516
1.8844	0.69	118000	1.7514
1.3336	0.69	118500	1.7509
1.5781	0.69	119000	1.7506
1.8449	0.7	119500	1.7505
1.5305	0.7	120000	1.7503
2.1904	0.7	120500	1.7500
2.2285	0.71	121000	1.7496
1.8097	0.71	121500	1.7494
2.3631	0.71	122000	1.7493
2.0893	0.71	122500	1.7491
2.1201	0.72	123000	1.7489
1.8334	0.72	123500	1.7488
2.0222	0.72	124000	1.7486
1.6339	0.73	124500	1.7484
1.6754	0.73	125000	1.7482
1.3973	0.73	125500	1.7480
2.0594	0.73	126000	1.7479
1.8674	0.74	126500	1.7478
2.1948	0.74	127000	1.7476
1.4148	0.74	127500	1.7475
1.6734	0.75	128000	1.7473
2.2787	0.75	128500	1.7472
1.8999	0.75	129000	1.7471
1.6945	0.76	129500	1.7470
2.0165	0.76	130000	1.7469
2.2232	0.76	130500	1.7468
1.6201	0.76	131000	1.7466
2.4878	0.77	131500	1.7465
1.5317	0.77	132000	1.7465
1.9361	0.77	132500	1.7464
1.7127	0.78	133000	1.7463
1.7045	0.78	133500	1.7462
2.1827	0.78	134000	1.7461
2.0534	0.78	134500	1.7461
2.0808	0.79	135000	1.7460
1.9572	0.79	135500	1.7459
1.8762	0.79	136000	1.7459
1.4686	0.8	136500	1.7458
1.6241	0.8	137000	1.7458
1.4219	0.8	137500	1.7457
2.1605	0.8	138000	1.7457
2.1298	0.81	138500	1.7456
1.414	0.81	139000	1.7456
1.0115	0.81	139500	1.7455
1.9471	0.82	140000	1.7455
1.8873	0.82	140500	1.7455
1.8286	0.82	141000	1.7454
2.1418	0.83	141500	1.7454
1.9755	0.83	142000	1.7454
1.6908	0.83	142500	1.7454
2.3842	0.83	143000	1.7453
1.7665	0.84	143500	1.7453
1.8266	0.84	144000	1.7453
0.8768	0.84	144500	1.7453
1.2274	0.85	145000	1.7453
1.6647	0.85	145500	1.7453
1.4071	0.85	146000	1.7452
1.6073	0.85	146500	1.7452
2.201	0.86	147000	1.7452
1.5504	0.86	147500	1.7452
1.4377	0.86	148000	1.7452
1.4453	0.87	148500	1.7452
1.6929	0.87	149000	1.7451
1.7631	0.87	149500	1.7451
2.0868	0.87	150000	1.7451
0.6434	0.88	150500	1.7451
1.4851	0.88	151000	1.7451
1.5365	0.88	151500	1.7451
1.8129	0.89	152000	1.7451
1.1623	0.89	152500	1.7451
2.0714	0.89	153000	1.7451
1.9363	0.9	153500	1.7451
1.6408	0.9	154000	1.7451
0.618	0.9	154500	1.7451
1.7957	0.9	155000	1.7451
2.0056	0.91	155500	1.7451
1.3893	0.91	156000	1.7451
2.1426	0.91	156500	1.7451
1.6766	0.92	157000	1.7451
1.4206	0.92	157500	1.7451
1.7285	0.92	158000	1.7451
1.5779	0.92	158500	1.7451
1.8675	0.93	159000	1.7451
2.0217	0.93	159500	1.7451
0.9516	0.93	160000	1.7451
2.219	0.94	160500	1.7450
1.6214	0.94	161000	1.7451
1.7134	0.94	161500	1.7451
1.6128	0.94	162000	1.7451
2.0817	0.95	162500	1.7450
1.8055	0.95	163000	1.7451
1.909	0.95	163500	1.7451
1.7844	0.96	164000	1.7451
2.0719	0.96	164500	1.7451
1.8698	0.96	165000	1.7451
1.6926	0.96	165500	1.7451
2.2161	0.97	166000	1.7451
2.1111	0.97	166500	1.7451
1.8004	0.97	167000	1.7451
2.2364	0.98	167500	1.7451
1.6716	0.98	168000	1.7451
2.1804	0.98	168500	1.7451
1.2691	0.99	169000	1.7451
1.8306	0.99	169500	1.7451
0.5662	0.99	170000	1.7451
1.6516	0.99	170500	1.7451
2.0576	1.0	171000	1.7451
1.3638	1.0	171500	1.7451

Framework versions

• Transformers 4.34.1
• Pytorch 2.0.1+cu117
• Datasets 2.14.6
• Tokenizers 0.14.1