readme: ---
  license: apache-2.0
  tags:
  - generated_from_trainer
  datasets:
  - wikitext
  metrics:
  - accuracy
  model-index:
  - name: distilbert_add_pre-training-complete
    results:
    - task:
        name: Masked Language Modeling
        type: fill-mask
      dataset:
        name: wikitext wikitext-103-raw-v1
        type: wikitext
        config: wikitext-103-raw-v1
        split: validation
        args: wikitext-103-raw-v1
      metrics:
      - name: Accuracy
        type: accuracy
        value: 0.23073914743840437
  ---
  
  <!-- This model card has been generated automatically according to the information the Trainer had access to. You
  should probably proofread and complete it, then remove this comment. -->
  
  # distilbert_add_pre-training-complete
  
  This model is a fine-tuned version of [distilbert-base-uncased](https://huggingface.co/distilbert-base-uncased) on the wikitext wikitext-103-raw-v1 dataset.
  It achieves the following results on the evaluation set:
  - Loss: 5.0239
  - Accuracy: 0.2307
  
  ## Model description
  
  More information needed
  
  ## Intended uses & limitations
  
  More information needed
  
  ## Training and evaluation data
  
  More information needed
  
  ## Training procedure
  
  ### Training hyperparameters
  
  The following hyperparameters were used during training:
  - learning_rate: 5e-05
  - train_batch_size: 64
  - eval_batch_size: 64
  - seed: 10
  - distributed_type: multi-GPU
  - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
  - lr_scheduler_type: linear
  - lr_scheduler_warmup_steps: 100
  - training_steps: 300000
  - mixed_precision_training: Native AMP
  
  ### Training results
  
  | Training Loss | Epoch | Step   | Validation Loss | Accuracy |
  |:-------------:|:-----:|:------:|:---------------:|:--------:|
  | 6.295         | 1.0   | 3573   | 6.0701          | 0.1522   |
  | 6.0482        | 2.0   | 7146   | 5.9533          | 0.1565   |
  | 5.9799        | 3.0   | 10719  | 5.9008          | 0.1584   |
  | 5.9378        | 4.0   | 14292  | 5.8997          | 0.1545   |
  | 5.9057        | 5.0   | 17865  | 5.8905          | 0.1536   |
  | 5.8811        | 6.0   | 21438  | 5.8646          | 0.1550   |
  | 5.8617        | 7.0   | 25011  | 5.8322          | 0.1534   |
  | 5.844         | 8.0   | 28584  | 5.8563          | 0.1523   |
  | 5.8297        | 9.0   | 32157  | 5.8352          | 0.1548   |
  | 5.8175        | 10.0  | 35730  | 5.8136          | 0.1558   |
  | 5.8056        | 11.0  | 39303  | 5.8147          | 0.1526   |
  | 5.7921        | 12.0  | 42876  | 5.8020          | 0.1548   |
  | 5.7777        | 13.0  | 46449  | 5.7891          | 0.1545   |
  | 5.7596        | 14.0  | 50022  | 5.7370          | 0.1587   |
  | 5.7414        | 15.0  | 53595  | 5.7396          | 0.1604   |
  | 5.7243        | 16.0  | 57168  | 5.7490          | 0.1564   |
  | 5.6997        | 17.0  | 60741  | 5.7135          | 0.1561   |
  | 5.6698        | 18.0  | 64314  | 5.6858          | 0.1620   |
  | 5.6398        | 19.0  | 67887  | 5.6735          | 0.1644   |
  | 5.6135        | 20.0  | 71460  | 5.6174          | 0.1681   |
  | 5.5899        | 21.0  | 75033  | 5.6191          | 0.1684   |
  | 5.5699        | 22.0  | 78606  | 5.5977          | 0.1669   |
  | 5.5487        | 23.0  | 82179  | 5.6139          | 0.1669   |
  | 5.529         | 24.0  | 85752  | 5.5272          | 0.1741   |
  | 5.512         | 25.0  | 89325  | 5.5271          | 0.1727   |
  | 5.4939        | 26.0  | 92898  | 5.5190          | 0.1721   |
  | 5.4765        | 27.0  | 96471  | 5.4824          | 0.1770   |
  | 5.4604        | 28.0  | 100044 | 5.5159          | 0.1747   |
  | 5.4422        | 29.0  | 103617 | 5.4577          | 0.1807   |
  | 5.4243        | 30.0  | 107190 | 5.4546          | 0.1772   |
  | 5.408         | 31.0  | 110763 | 5.4297          | 0.1837   |
  | 5.3915        | 32.0  | 114336 | 5.4089          | 0.1866   |
  | 5.3766        | 33.0  | 117909 | 5.3996          | 0.1848   |
  | 5.3594        | 34.0  | 121482 | 5.3974          | 0.1841   |
  | 5.3451        | 35.0  | 125055 | 5.3718          | 0.1908   |
  | 5.3294        | 36.0  | 128628 | 5.3706          | 0.1878   |
  | 5.3155        | 37.0  | 132201 | 5.3677          | 0.1903   |
  | 5.2996        | 38.0  | 135774 | 5.2970          | 0.1994   |
  | 5.287         | 39.0  | 139347 | 5.3127          | 0.1977   |
  | 5.2735        | 40.0  | 142920 | 5.3145          | 0.1955   |
  | 5.26          | 41.0  | 146493 | 5.2985          | 0.2017   |
  | 5.2487        | 42.0  | 150066 | 5.2661          | 0.2025   |
  | 5.2362        | 43.0  | 153639 | 5.2712          | 0.2031   |
  | 5.2248        | 44.0  | 157212 | 5.2452          | 0.2049   |
  | 5.2115        | 45.0  | 160785 | 5.2325          | 0.2054   |
  | 5.1998        | 46.0  | 164358 | 5.2233          | 0.2075   |
  | 5.188         | 47.0  | 167931 | 5.1994          | 0.2118   |
  | 5.1779        | 48.0  | 171504 | 5.2436          | 0.2069   |
  | 5.1664        | 49.0  | 175077 | 5.2203          | 0.2129   |
  | 5.1546        | 50.0  | 178650 | 5.1820          | 0.2134   |
  | 5.1431        | 51.0  | 182223 | 5.2029          | 0.2122   |
  | 5.133         | 52.0  | 185796 | 5.1458          