readme: ---
  license: mit
  tags:
  - generated_from_trainer
  model-index:
  - name: gpt-expt-sp-v3-K-300-9-mixed-with-tv
    results: []
  ---
  
  <!-- This model card has been generated automatically according to the information the Trainer had access to. You
  should probably proofread and complete it, then remove this comment. -->
  
  # gpt-expt-sp-v3-K-300-9-mixed-with-tv
  
  This model is a fine-tuned version of [gpt2](https://huggingface.co/gpt2) on an unknown dataset.
  It achieves the following results on the evaluation set:
  - Loss: 0.0349
  
  ## Model description
  
  More information needed
  
  ## Intended uses & limitations
  
  More information needed
  
  ## Training and evaluation data
  
  More information needed
  
  ## Training procedure
  
  ### Training hyperparameters
  
  The following hyperparameters were used during training:
  - learning_rate: 0.0005
  - train_batch_size: 32
  - eval_batch_size: 32
  - seed: 42
  - gradient_accumulation_steps: 8
  - total_train_batch_size: 256
  - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
  - lr_scheduler_type: cosine
  - lr_scheduler_warmup_steps: 1000
  - num_epochs: 500
  - mixed_precision_training: Native AMP
  
  ### Training results
  
  | Training Loss | Epoch  | Step   | Validation Loss |
  |:-------------:|:------:|:------:|:---------------:|
  | 0.3825        | 4.28   | 5000   | 0.1047          |
  | 0.0883        | 8.56   | 10000  | 0.0668          |
  | 0.0633        | 12.84  | 15000  | 0.0505          |
  | 0.0545        | 17.12  | 20000  | 0.0472          |
  | 0.0503        | 21.4   | 25000  | 0.0441          |
  | 0.0477        | 25.68  | 30000  | 0.0420          |
  | 0.0457        | 29.97  | 35000  | 0.0408          |
  | 0.0444        | 34.25  | 40000  | 0.0401          |
  | 0.0435        | 38.53  | 45000  | 0.0399          |
  | 0.0428        | 42.81  | 50000  | 0.0391          |
  | 0.0423        | 47.09  | 55000  | 0.0388          |
  | 0.0418        | 51.37  | 60000  | 0.0386          |
  | 0.0414        | 55.65  | 65000  | 0.0384          |
  | 0.0411        | 59.93  | 70000  | 0.0381          |
  | 0.0408        | 64.21  | 75000  | 0.0381          |
  | 0.0405        | 68.49  | 80000  | 0.0379          |
  | 0.0403        | 72.77  | 85000  | 0.0376          |
  | 0.0401        | 77.05  | 90000  | 0.0375          |
  | 0.0399        | 81.34  | 95000  | 0.0375          |
  | 0.0398        | 85.62  | 100000 | 0.0374          |
  | 0.0396        | 89.9   | 105000 | 0.0373          |
  | 0.0395        | 94.18  | 110000 | 0.0373          |
  | 0.0393        | 98.46  | 115000 | 0.0372          |
  | 0.0392        | 102.74 | 120000 | 0.0370          |
  | 0.0391        | 107.02 | 125000 | 0.0369          |
  | 0.039         | 111.3  | 130000 | 0.0370          |
  | 0.0389        | 115.58 | 135000 | 0.0369          |
  | 0.0388        | 119.86 | 140000 | 0.0369          |
  | 0.0387        | 124.14 | 145000 | 0.0368          |
  | 0.0386        | 128.42 | 150000 | 0.0367          |
  | 0.0385        | 132.71 | 155000 | 0.0367          |
  | 0.0388        | 136.99 | 160000 | 0.0511          |
  | 0.0384        | 141.27 | 165000 | 0.0367          |
  | 0.0383        | 145.55 | 170000 | 0.0366          |
  | 0.0382        | 149.83 | 175000 | 0.0365          |
  | 0.0381        | 154.11 | 180000 | 0.0365          |
  | 0.0381        | 158.39 | 185000 | 0.0364          |
  | 0.038         | 162.67 | 190000 | 0.0364          |
  | 0.0379        | 166.95 | 195000 | 0.0364          |
  | 0.0378        | 171.23 | 200000 | 0.0365          |
  | 0.0378        | 175.51 | 205000 | 0.0363          |
  | 0.0377        | 179.79 | 210000 | 0.0362          |
  | 0.0377        | 184.08 | 215000 | 0.0362          |
  | 0.0376        | 188.36 | 220000 | 0.0362          |
  | 0.0375        | 192.64 | 225000 | 0.0361          |
  | 0.0375        | 196.92 | 230000 | 0.0361          |
  | 0.0374        | 201.2  | 235000 | 0.0361          |
  | 0.0373        | 205.48 | 240000 | 0.0360          |
  | 0.0373        | 209.76 | 245000 | 0.0360          |
  | 0.0372        | 214.04 | 250000 | 0.0360          |
  | 0.0372        | 218.32 | 255000 | 0.0359          |
  | 0.0371        | 222.6  | 260000 | 0.0359          |
  | 0.0371        | 226.88 | 265000 | 0.0359          |
  | 0.037         | 231.16 | 270000 | 0.0359          |
  | 0.037         | 235.45 | 275000 | 0.0358          |
  | 0.0375        | 239.73 | 280000 | 0.0358          |
  | 0.0369        | 244.01 | 285000 | 0.0357          |
  | 0.0368        | 248.29 | 290000 | 0.0357          |
  | 0.0367        | 252.57 | 295000 | 0.0358          |
  | 0.0367        | 256.85 | 300000 | 0.0357          |
  | 0.0368        | 261.13 | 305000 | 0.0356          |
  | 0.0366        | 265.41 | 310000 | 0.0357          |
  | 0.0366        | 269.69 | 315000 | 0.0356          |
  | 0.0365        | 273.97 | 320000 | 0.0355          |
  | 0.0364        | 278.25 | 325000 | 0.0356          |
  | 0.0364        | 282.53 | 330000 | 0.0356          |
  | 0.0363        | 286.81 | 335000 | 0.0355          |
  | 0.0363        | 291.1  | 340000 | 0.0355          |
  | 0.0362        | 295.38 | 345000 | 0.0354          |
  | 0.0362        | 299.66 | 