readme: ---
  license: apache-2.0
  tags:
  - generated_from_trainer
  datasets:
  - wikitext
  metrics:
  - accuracy
  model-index:
  - name: distilbert_sa_pre-training-complete
    results:
    - task:
        name: Masked Language Modeling
        type: fill-mask
      dataset:
        name: wikitext wikitext-103-raw-v1
        type: wikitext
        config: wikitext-103-raw-v1
        split: validation
        args: wikitext-103-raw-v1
      metrics:
      - name: Accuracy
        type: accuracy
        value: 0.7119499337524314
  ---
  
  <!-- This model card has been generated automatically according to the information the Trainer had access to. You
  should probably proofread and complete it, then remove this comment. -->
  
  # distilbert_sa_pre-training-complete
  
  This model is a fine-tuned version of [distilbert-base-uncased](https://huggingface.co/distilbert-base-uncased) on the wikitext wikitext-103-raw-v1 dataset.
  It achieves the following results on the evaluation set:
  - Loss: 1.3367
  - Accuracy: 0.7119
  
  ## Model description
  
  More information needed
  
  ## Intended uses & limitations
  
  More information needed
  
  ## Training and evaluation data
  
  More information needed
  
  ## Training procedure
  
  ### Training hyperparameters
  
  The following hyperparameters were used during training:
  - learning_rate: 5e-05
  - train_batch_size: 64
  - eval_batch_size: 64
  - seed: 10
  - distributed_type: multi-GPU
  - num_devices: 2
  - total_train_batch_size: 128
  - total_eval_batch_size: 128
  - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
  - lr_scheduler_type: linear
  - lr_scheduler_warmup_steps: 100
  - training_steps: 300000
  - mixed_precision_training: Native AMP
  
  ### Training results
  
  | Training Loss | Epoch  | Step   | Validation Loss | Accuracy |
  |:-------------:|:------:|:------:|:---------------:|:--------:|
  | 1.8819        | 1.0    | 1787   | 1.6350          | 0.6570   |
  | 1.7699        | 2.0    | 3574   | 1.5847          | 0.6664   |
  | 1.7308        | 3.0    | 5361   | 1.5857          | 0.6678   |
  | 1.7062        | 4.0    | 7148   | 1.5679          | 0.6684   |
  | 1.6858        | 5.0    | 8935   | 1.5573          | 0.6689   |
  | 1.6684        | 6.0    | 10722  | 1.5591          | 0.6687   |
  | 1.6545        | 7.0    | 12509  | 1.5253          | 0.6756   |
  | 1.6438        | 8.0    | 14296  | 1.5304          | 0.6748   |
  | 1.631         | 9.0    | 16083  | 1.4976          | 0.6805   |
  | 1.6236        | 10.0   | 17870  | 1.5153          | 0.6773   |
  | 1.613         | 11.0   | 19657  | 1.5010          | 0.6786   |
  | 1.6046        | 12.0   | 21444  | 1.5069          | 0.6818   |
  | 1.5963        | 13.0   | 23231  | 1.4796          | 0.6805   |
  | 1.5906        | 14.0   | 25018  | 1.4858          | 0.6809   |
  | 1.5833        | 15.0   | 26805  | 1.4962          | 0.6806   |
  | 1.5771        | 16.0   | 28592  | 1.4749          | 0.6826   |
  | 1.5703        | 17.0   | 30379  | 1.4724          | 0.6874   |
  | 1.5663        | 18.0   | 32166  | 1.4946          | 0.6827   |
  | 1.5614        | 19.0   | 33953  | 1.4934          | 0.6798   |
  | 1.5558        | 20.0   | 35740  | 1.4512          | 0.6893   |
  | 1.5508        | 21.0   | 37527  | 1.4417          | 0.6921   |
  | 1.5466        | 22.0   | 39314  | 1.4744          | 0.6837   |
  | 1.5427        | 23.0   | 41101  | 1.4530          | 0.6904   |
  | 1.5382        | 24.0   | 42888  | 1.4378          | 0.6900   |
  | 1.5342        | 25.0   | 44675  | 1.4732          | 0.6820   |
  | 1.5293        | 26.0   | 46462  | 1.4609          | 0.6865   |
  | 1.5259        | 27.0   | 48249  | 1.4400          | 0.6942   |
  | 1.5223        | 28.0   | 50036  | 1.4586          | 0.6879   |
  | 1.5197        | 29.0   | 51823  | 1.4371          | 0.6906   |
  | 1.5161        | 30.0   | 53610  | 1.4335          | 0.6908   |
  | 1.5126        | 31.0   | 55397  | 1.4159          | 0.6978   |
  | 1.5087        | 32.0   | 57184  | 1.4412          | 0.6858   |
  | 1.5061        | 33.0   | 58971  | 1.4181          | 0.6958   |
  | 1.5035        | 34.0   | 60758  | 1.4357          | 0.6903   |
  | 1.5011        | 35.0   | 62545  | 1.4147          | 0.6914   |
  | 1.4967        | 36.0   | 64332  | 1.4251          | 0.6931   |
  | 1.4954        | 37.0   | 66119  | 1.4366          | 0.6910   |
  | 1.4925        | 38.0   | 67906  | 1.4488          | 0.6896   |
  | 1.4889        | 39.0   | 69693  | 1.4336          | 0.6907   |
  | 1.4866        | 40.0   | 71480  | 1.4226          | 0.6945   |
  | 1.4838        | 41.0   | 73267  | 1.4210          | 0.6951   |
  | 1.4821        | 42.0   | 75054  | 1.4081          | 0.6944   |
  | 1.4792        | 43.0   | 76841  | 1.4496          | 0.6895   |
  | 1.4778        | 44.0   | 78628  | 1.4263          | 0.6925   |
  | 1.4754        | 45.0   | 80415  | 1.3995          | 0.6982   |
  | 1.4736        | 46.0   | 82202  | 1.3958          | 0.6992   |
  | 1.4702        | 47.0   | 83989  | 1.4073          | 0.6989   |
  | 1.4683        | 48.0   | 85776  | 1.3917          | 0.6996   |
  | 1.4663        | 49.0   | 87563  | 1.4109          | 0.6956   |
  | 1.4648        | 50.0   | 89350  | 1.4003          | 0