readme: ---
  license: mit
  tags:
  - generated_from_trainer
  model-index:
  - name: gpt-expt-sp
    results: []
  ---
  
  <!-- This model card has been generated automatically according to the information the Trainer had access to. You
  should probably proofread and complete it, then remove this comment. -->
  
  # gpt-expt-sp
  
  This model is a fine-tuned version of [gpt2](https://huggingface.co/gpt2) on an unknown dataset.
  It achieves the following results on the evaluation set:
  - Loss: 0.2561
  
  ## Model description
  
  More information needed
  
  ## Intended uses & limitations
  
  More information needed
  
  ## Training and evaluation data
  
  More information needed
  
  ## Training procedure
  
  ### Training hyperparameters
  
  The following hyperparameters were used during training:
  - learning_rate: 0.0005
  - train_batch_size: 32
  - eval_batch_size: 32
  - seed: 42
  - gradient_accumulation_steps: 8
  - total_train_batch_size: 256
  - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
  - lr_scheduler_type: cosine
  - lr_scheduler_warmup_steps: 1000
  - num_epochs: 300
  - mixed_precision_training: Native AMP
  
  ### Training results
  
  | Training Loss | Epoch  | Step | Validation Loss |
  |:-------------:|:------:|:----:|:---------------:|
  | 3.8623        | 3.12   | 100  | 1.7653          |
  | 1.6403        | 6.24   | 200  | 1.5635          |
  | 1.5806        | 9.37   | 300  | 1.5326          |
  | 1.5433        | 12.49  | 400  | 1.4568          |
  | 1.362         | 15.61  | 500  | 0.9368          |
  | 0.8739        | 18.73  | 600  | 0.5006          |
  | 0.5905        | 21.85  | 700  | 0.3875          |
  | 0.4755        | 24.98  | 800  | 0.3440          |
  | 0.4252        | 28.12  | 900  | 0.3238          |
  | 0.3904        | 31.24  | 1000 | 0.3093          |
  | 0.366         | 34.37  | 1100 | 0.3004          |
  | 0.3492        | 37.49  | 1200 | 0.2922          |
  | 0.3345        | 40.61  | 1300 | 0.2860          |
  | 0.3277        | 43.73  | 1400 | 0.2819          |
  | 0.324         | 46.85  | 1500 | 0.2800          |
  | 0.318         | 49.98  | 1600 | 0.2766          |
  | 0.314         | 53.12  | 1700 | 0.2736          |
  | 0.308         | 56.24  | 1800 | 0.2740          |
  | 0.306         | 59.37  | 1900 | 0.2716          |
  | 0.3037        | 62.49  | 2000 | 0.2708          |
  | 0.2993        | 65.61  | 2100 | 0.2685          |
  | 0.2991        | 68.73  | 2200 | 0.2680          |
  | 0.297         | 71.85  | 2300 | 0.2670          |
  | 0.2964        | 74.98  | 2400 | 0.2662          |
  | 0.2964        | 78.12  | 2500 | 0.2653          |
  | 0.2942        | 81.24  | 2600 | 0.2664          |
  | 0.2937        | 84.37  | 2700 | 0.2655          |
  | 0.2886        | 87.49  | 2800 | 0.2631          |
  | 0.2877        | 90.61  | 2900 | 0.2634          |
  | 0.2859        | 93.73  | 3000 | 0.2628          |
  | 0.2852        | 96.85  | 3100 | 0.2629          |
  | 0.2841        | 99.98  | 3200 | 0.2629          |
  | 0.2848        | 103.12 | 3300 | 0.2625          |
  | 0.2811        | 106.24 | 3400 | 0.2611          |
  | 0.281         | 109.37 | 3500 | 0.2608          |
  | 0.2794        | 112.49 | 3600 | 0.2599          |
  | 0.2787        | 115.61 | 3700 | 0.2604          |
  | 0.2781        | 118.73 | 3800 | 0.2601          |
  | 0.2777        | 121.85 | 3900 | 0.2604          |
  | 0.2776        | 124.98 | 4000 | 0.2600          |
  | 0.2786        | 128.12 | 4100 | 0.2597          |
  | 0.2757        | 131.24 | 4200 | 0.2597          |
  | 0.2754        | 134.37 | 4300 | 0.2590          |
  | 0.2758        | 137.49 | 4400 | 0.2596          |
  | 0.2742        | 140.61 | 4500 | 0.2598          |
  | 0.2731        | 143.73 | 4600 | 0.2581          |
  | 0.2738        | 146.85 | 4700 | 0.2587          |
  | 0.273         | 149.98 | 4800 | 0.2583          |
  | 0.2736        | 153.12 | 4900 | 0.2579          |
  | 0.271         | 156.24 | 5000 | 0.2580          |
  | 0.2709        | 159.37 | 5100 | 0.2578          |
  | 0.2708        | 162.49 | 5200 | 0.2582          |
  | 0.2697        | 165.61 | 5300 | 0.2578          |
  | 0.2695        | 168.73 | 5400 | 0.2578          |
  | 0.269         | 171.85 | 5500 | 0.2582          |
  | 0.2691        | 174.98 | 5600 | 0.2574          |
  | 0.2705        | 178.12 | 5700 | 0.2574          |
  | 0.2678        | 181.24 | 5800 | 0.2572          |
  | 0.2692        | 184.37 | 5900 | 0.2582          |
  | 0.2687        | 187.49 | 6000 | 0.2572          |
  | 0.2673        | 190.61 | 6100 | 0.2571          |
  | 0.2666        | 193.73 | 6200 | 0.2568          |
  | 0.2662        | 196.85 | 6300 | 0.2573          |
  | 0.2662        | 199.98 | 6400 | 0.2568          |
  | 0.2688        | 203.12 | 6500 | 0.2567          |
  | 0.2658        | 206.24 | 6600 | 0.2570          |
  | 0.2666        | 209.37 | 6700 | 0.2567          |
  | 0.2652        | 212.49 | 6800 | 0.2565          |
  | 0.2651        | 215.61 | 6900 | 0.2568          |
  | 0.2649        | 218.73 | 7000 | 0.2566          |
  | 0.2648        | 221.85 | 7100 | 0.2564          |
  | 0.2645        | 224.98 | 7200 | 0.2564          |
  | 0.2662        | 228.12 | 7300 | 0.2564          |
  | 0.2641   