--- license: apache-2.0 library_name: peft tags: - trl - sft - generated_from_trainer base_model: TheBloke/Mistral-7B-v0.1-GPTQ model-index: - name: mistral-augmentation-digikey-rand results: [] --- # mistral-augmentation-digikey-rand This model is a fine-tuned version of [TheBloke/Mistral-7B-v0.1-GPTQ](https://huggingface.co/TheBloke/Mistral-7B-v0.1-GPTQ) on the None dataset. It achieves the following results on the evaluation set: - Loss: 0.4955 ## Model description More information needed ## Intended uses & limitations More information needed ## Training and evaluation data More information needed ## Training procedure ### Training hyperparameters The following hyperparameters were used during training: - learning_rate: 2e-05 - train_batch_size: 4 - eval_batch_size: 8 - seed: 42 - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08 - lr_scheduler_type: cosine - num_epochs: 15 - mixed_precision_training: Native AMP ### Training results | Training Loss | Epoch | Step | Validation Loss | |:-------------:|:-----:|:----:|:---------------:| | 2.0496 | 0.01 | 50 | 1.1570 | | 0.9361 | 0.03 | 100 | 0.8592 | | 0.7691 | 0.04 | 150 | 0.7989 | | 0.7555 | 0.06 | 200 | 0.7768 | | 0.7213 | 0.07 | 250 | 0.7575 | | 0.6993 | 0.09 | 300 | 0.7440 | | 0.6905 | 0.1 | 350 | 0.7291 | | 0.6855 | 0.12 | 400 | 0.7210 | | 0.6732 | 0.13 | 450 | 0.7076 | | 0.6516 | 0.15 | 500 | 0.7005 | | 0.639 | 0.16 | 550 | 0.6920 | | 0.6322 | 0.18 | 600 | 0.6829 | | 0.6164 | 0.19 | 650 | 0.6755 | | 0.6185 | 0.21 | 700 | 0.6704 | | 0.6457 | 0.22 | 750 | 0.6667 | | 0.6238 | 0.24 | 800 | 0.6630 | | 0.6173 | 0.25 | 850 | 0.6570 | | 0.6076 | 0.27 | 900 | 0.6562 | | 0.6097 | 0.28 | 950 | 0.6493 | | 0.5693 | 0.3 | 1000 | 0.6423 | | 0.5887 | 0.31 | 1050 | 0.6404 | | 0.5869 | 0.33 | 1100 | 0.6361 | | 0.5964 | 0.34 | 1150 | 0.6341 | | 0.5373 | 0.36 | 1200 | 0.6281 | | 0.5684 | 0.37 | 1250 | 0.6277 | | 0.5746 | 0.39 | 1300 | 0.6183 | | 0.5703 | 0.4 | 1350 | 0.6221 | | 0.5851 | 0.42 | 1400 | 0.6175 | | 0.5519 | 0.43 | 1450 | 0.6167 | | 0.5716 | 0.45 | 1500 | 0.6115 | | 0.552 | 0.46 | 1550 | 0.6095 | | 0.5885 | 0.47 | 1600 | 0.6100 | | 0.5739 | 0.49 | 1650 | 0.6061 | | 0.5598 | 0.5 | 1700 | 0.6061 | | 0.5729 | 0.52 | 1750 | 0.6011 | | 0.5575 | 0.53 | 1800 | 0.6013 | | 0.5418 | 0.55 | 1850 | 0.6003 | | 0.5365 | 0.56 | 1900 | 0.5940 | | 0.5096 | 0.58 | 1950 | 0.5878 | | 0.5458 | 0.59 | 2000 | 0.5878 | | 0.5603 | 0.61 | 2050 | 0.5863 | | 0.5388 | 0.62 | 2100 | 0.5854 | | 0.5187 | 0.64 | 2150 | 0.5789 | | 0.5402 | 0.65 | 2200 | 0.5809 | | 0.5398 | 0.67 | 2250 | 0.5761 | | 0.5123 | 0.68 | 2300 | 0.5751 | | 0.4936 | 0.7 | 2350 | 0.5712 | | 0.4899 | 0.71 | 2400 | 0.5672 | | 0.5197 | 0.73 | 2450 | 0.5627 | | 0.509 | 0.74 | 2500 | 0.5574 | | 0.4963 | 0.76 | 2550 | 0.5560 | | 0.4989 | 0.77 | 2600 | 0.5544 | | 0.4809 | 0.79 | 2650 | 0.5526 | | 0.49 | 0.8 | 2700 | 0.5473 | | 0.5151 | 0.82 | 2750 | 0.5485 | | 0.5005 | 0.83 | 2800 | 0.5469 | | 0.5072 | 0.85 | 2850 | 0.5466 | | 0.5008 | 0.86 | 2900 | 0.5464 | | 0.4857 | 0.88 | 2950 | 0.5441 | | 0.4889 | 0.89 | 3000 | 0.5429 | | 0.4714 | 0.91 | 3050 | 0.5441 | | 0.4618 | 0.92 | 3100 | 0.5404 | | 0.4623 | 0.93 | 3150 | 0.5418 | | 0.4771 | 0.95 | 3200 | 0.5396 | | 0.4592 | 0.96 | 3250 | 0.5409 | | 0.4783 | 0.98 | 3300 | 0.5373 | | 0.5021 | 0.99 | 3350 | 0.5343 | | 0.4753 | 1.01 | 3400 | 0.5350 | | 0.4369 | 1.02 | 3450 | 0.5338 | | 0.4651 | 1.04 | 3500 | 0.5318 | | 0.4395 | 1.05 | 3550 | 0.5320 | | 0.4771 | 1.07 | 3600 | 0.5311 | | 0.4659 | 1.08 | 3650 | 0.5337 | | 0.4699 | 1.1 | 3700 | 0.5309 | | 0.4717 | 1.11 | 3750 | 0.5301 | | 0.4445 | 1.13 | 3800 | 0.5282 | | 0.4342 | 1.14 | 3850 | 0.5303 | | 0.4599 | 1.16 | 3900 | 0.5266 | | 0.4442 | 1.17 | 3950 | 0.5275 | | 0.4628 | 1.19 | 4000 | 0.5260 | | 0.4339 | 1.2 | 4050 | 0.5243 | | 0.4577 | 1.22 | 4100 | 0.5283 | | 0.463 | 1.23 | 4150 | 0.5253 | | 0.4602 | 1.25 | 4200 | 0.5243 | | 0.4411 | 1.26 | 4250 | 0.5255 | | 0.4542 | 1.28 | 4300 | 0.5263 | | 0.4379 | 1.29 | 4350 | 0.5213 | | 0.4471 | 1.31 | 4400 | 0.5189 | | 0.4372 | 1.32 | 4450 | 0.5236 | | 0.4526 | 1.34 | 4500 | 0.5203 | | 0.4504 | 1.35 | 4550 | 0.5198 | | 0.4708 | 1.36 | 4600 | 0.5171 | | 0.4748 | 1.38 | 4650 | 0.5177 | | 0.4511 | 1.39 | 4700 | 0.5152 | | 0.4758 | 1.41 | 4750 | 0.5179 | | 0.4543 | 1.42 | 4800 | 0.5165 | | 0.4506 | 1.44 | 4850 | 0.5167 | | 0.44 | 1.45 | 4900 | 0.5152 | | 0.4443 | 1.47 | 4950 | 0.5135 | | 0.4538 | 1.48 | 5000 | 0.5140 | | 0.435 | 1.5 | 5050 | 0.5142 | | 0.439 | 1.51 | 5100 | 0.5135 | | 0.4408 | 1.53 | 5150 | 0.5121 | | 0.4532 | 1.54 | 5200 | 0.5137 | | 0.4177 | 1.56 | 5250 | 0.5143 | | 0.4434 | 1.57 | 5300 | 0.5139 | | 0.4395 | 1.59 | 5350 | 0.5117 | | 0.4327 | 1.6 | 5400 | 0.5124 | | 0.4257 | 1.62 | 5450 | 0.5128 | | 0.4225 | 1.63 | 5500 | 0.5106 | | 0.4517 | 1.65 | 5550 | 0.5119 | | 0.4632 | 1.66 | 5600 | 0.5076 | | 0.4371 | 1.68 | 5650 | 0.5110 | | 0.4209 | 1.69 | 5700 | 0.5082 | | 0.4336 | 1.71 | 5750 | 0.5072 | | 0.4269 | 1.72 | 5800 | 0.5125 | | 0.4208 | 1.74 | 5850 | 0.5105 | | 0.4334 | 1.75 | 5900 | 0.5074 | | 0.4306 | 1.77 | 5950 | 0.5052 | | 0.4454 | 1.78 | 6000 | 0.5073 | | 0.4227 | 1.8 | 6050 | 0.5068 | | 0.4467 | 1.81 | 6100 | 0.5041 | | 0.4279 | 1.82 | 6150 | 0.5034 | | 0.4368 | 1.84 | 6200 | 0.5021 | | 0.4205 | 1.85 | 6250 | 0.5025 | | 0.415 | 1.87 | 6300 | 0.5029 | | 0.4213 | 1.88 | 6350 | 0.5019 | | 0.4316 | 1.9 | 6400 | 0.5053 | | 0.4065 | 1.91 | 6450 | 0.5004 | | 0.4578 | 1.93 | 6500 | 0.5045 | | 0.4479 | 1.94 | 6550 | 0.4998 | | 0.43 | 1.96 | 6600 | 0.4947 | | 0.4192 | 1.97 | 6650 | 0.4967 | | 0.4061 | 1.99 | 6700 | 0.4961 | | 0.4309 | 2.0 | 6750 | 0.4960 | | 0.4118 | 2.02 | 6800 | 0.4979 | | 0.4149 | 2.03 | 6850 | 0.4955 | ### Framework versions - PEFT 0.7.1 - Transformers 4.36.2 - Pytorch 2.1.2+cu121 - Datasets 2.16.1 - Tokenizers 0.15.0