--- license: gemma base_model: google/gemma-2-2b tags: - trl - sft - generated_from_trainer model-index: - name: collapse_gemma-2-2b_hs2_accumulate_iter3_sftsd2 results: [] --- # collapse_gemma-2-2b_hs2_accumulate_iter3_sftsd2 This model is a fine-tuned version of [google/gemma-2-2b](https://huggingface.co/google/gemma-2-2b) on an unknown dataset. It achieves the following results on the evaluation set: - Loss: 1.1022 - Num Input Tokens Seen: 22054048 ## Model description More information needed ## Intended uses & limitations More information needed ## Training and evaluation data More information needed ## Training procedure ### Training hyperparameters The following hyperparameters were used during training: - learning_rate: 8e-06 - train_batch_size: 8 - eval_batch_size: 16 - seed: 2 - gradient_accumulation_steps: 16 - total_train_batch_size: 128 - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08 - lr_scheduler_type: constant_with_warmup - lr_scheduler_warmup_ratio: 0.05 - num_epochs: 1 ### Training results | Training Loss | Epoch | Step | Validation Loss | Input Tokens Seen | |:-------------:|:------:|:----:|:---------------:|:-----------------:| | No log | 0 | 0 | 1.3956 | 0 | | 1.628 | 0.0127 | 5 | 1.3800 | 282000 | | 1.6129 | 0.0254 | 10 | 1.2915 | 565768 | | 1.4755 | 0.0381 | 15 | 1.2119 | 845776 | | 1.2663 | 0.0508 | 20 | 1.1654 | 1119976 | | 1.2503 | 0.0636 | 25 | 1.1530 | 1405752 | | 1.0375 | 0.0763 | 30 | 1.1358 | 1683544 | | 0.9388 | 0.0890 | 35 | 1.1575 | 1962800 | | 0.8887 | 0.1017 | 40 | 1.1613 | 2242448 | | 0.9444 | 0.1144 | 45 | 1.1814 | 2530200 | | 0.8274 | 0.1271 | 50 | 1.1685 | 2813744 | | 0.7725 | 0.1398 | 55 | 1.1846 | 3088392 | | 0.7435 | 0.1525 | 60 | 1.1750 | 3367968 | | 0.8112 | 0.1652 | 65 | 1.1798 | 3653616 | | 0.6116 | 0.1779 | 70 | 1.1803 | 3935936 | | 0.6364 | 0.1907 | 75 | 1.1648 | 4215056 | | 0.6888 | 0.2034 | 80 | 1.1682 | 4498800 | | 0.6489 | 0.2161 | 85 | 1.1755 | 4777456 | | 0.5009 | 0.2288 | 90 | 1.1711 | 5056576 | | 0.6014 | 0.2415 | 95 | 1.1619 | 5333256 | | 0.6265 | 0.2542 | 100 | 1.1702 | 5607960 | | 0.4422 | 0.2669 | 105 | 1.1616 | 5888544 | | 0.5504 | 0.2796 | 110 | 1.1721 | 6157688 | | 0.5325 | 0.2923 | 115 | 1.1638 | 6436816 | | 0.4722 | 0.3051 | 120 | 1.1622 | 6720832 | | 0.3832 | 0.3178 | 125 | 1.1592 | 7010752 | | 0.5639 | 0.3305 | 130 | 1.1548 | 7296936 | | 0.4615 | 0.3432 | 135 | 1.1555 | 7569880 | | 0.5294 | 0.3559 | 140 | 1.1487 | 7848792 | | 0.4983 | 0.3686 | 145 | 1.1543 | 8130552 | | 0.4877 | 0.3813 | 150 | 1.1442 | 8409680 | | 0.419 | 0.3940 | 155 | 1.1497 | 8691616 | | 0.5136 | 0.4067 | 160 | 1.1437 | 8974984 | | 0.4672 | 0.4194 | 165 | 1.1442 | 9258208 | | 0.4665 | 0.4322 | 170 | 1.1359 | 9538392 | | 0.4105 | 0.4449 | 175 | 1.1412 | 9818464 | | 0.5283 | 0.4576 | 180 | 1.1360 | 10102088 | | 0.4097 | 0.4703 | 185 | 1.1388 | 10385664 | | 0.4573 | 0.4830 | 190 | 1.1324 | 10667816 | | 0.4047 | 0.4957 | 195 | 1.1343 | 10947272 | | 0.4657 | 0.5084 | 200 | 1.1281 | 11227664 | | 0.3811 | 0.5211 | 205 | 1.1295 | 11509152 | | 0.43 | 0.5338 | 210 | 1.1294 | 11792720 | | 0.4653 | 0.5466 | 215 | 1.1250 | 12068688 | | 0.3614 | 0.5593 | 220 | 1.1273 | 12350648 | | 0.4405 | 0.5720 | 225 | 1.1234 | 12628784 | | 0.3511 | 0.5847 | 230 | 1.1251 | 12907416 | | 0.4004 | 0.5974 | 235 | 1.1223 | 13192632 | | 0.4819 | 0.6101 | 240 | 1.1201 | 13469328 | | 0.4378 | 0.6228 | 245 | 1.1201 | 13748984 | | 0.3615 | 0.6355 | 250 | 1.1166 | 14033560 | | 0.3767 | 0.6482 | 255 | 1.1185 | 14315712 | | 0.3775 | 0.6609 | 260 | 1.1169 | 14599040 | | 0.4632 | 0.6737 | 265 | 1.1152 | 14883880 | | 0.3246 | 0.6864 | 270 | 1.1148 | 15161064 | | 0.3381 | 0.6991 | 275 | 1.1136 | 15435968 | | 0.3762 | 0.7118 | 280 | 1.1167 | 15715000 | | 0.3853 | 0.7245 | 285 | 1.1128 | 15992552 | | 0.4548 | 0.7372 | 290 | 1.1124 | 16277624 | | 0.3692 | 0.7499 | 295 | 1.1102 | 16554696 | | 0.423 | 0.7626 | 300 | 1.1101 | 16842640 | | 0.3635 | 0.7753 | 305 | 1.1124 | 17126528 | | 0.3939 | 0.7881 | 310 | 1.1096 | 17402024 | | 0.4323 | 0.8008 | 315 | 1.1092 | 17679664 | | 0.3539 | 0.8135 | 320 | 1.1073 | 17959928 | | 0.4876 | 0.8262 | 325 | 1.1077 | 18241888 | | 0.3201 | 0.8389 | 330 | 1.1077 | 18521608 | | 0.3806 | 0.8516 | 335 | 1.1060 | 18805032 | | 0.3601 | 0.8643 | 340 | 1.1062 | 19089648 | | 0.3919 | 0.8770 | 345 | 1.1049 | 19371096 | | 0.3816 | 0.8897 | 350 | 1.1069 | 19650992 | | 0.3584 | 0.9024 | 355 | 1.1051 | 19923856 | | 0.3534 | 0.9152 | 360 | 1.1057 | 20198240 | | 0.4761 | 0.9279 | 365 | 1.1049 | 20480400 | | 0.3723 | 0.9406 | 370 | 1.1053 | 20761832 | | 0.4056 | 0.9533 | 375 | 1.1036 | 21048104 | | 0.3886 | 0.9660 | 380 | 1.1024 | 21323808 | | 0.5005 | 0.9787 | 385 | 1.1028 | 21602888 | | 0.3638 | 0.9914 | 390 | 1.1039 | 21887576 | ### Framework versions - Transformers 4.44.0 - Pytorch 2.4.0+cu121 - Datasets 2.20.0 - Tokenizers 0.19.1