readme: ---
  license: apache-2.0
  tags:
  - generated_from_trainer
  datasets:
  - wikitext
  metrics:
  - accuracy
  model-index:
  - name: mobilebert_add_pre-training-complete
    results:
    - task:
        name: Masked Language Modeling
        type: fill-mask
      dataset:
        name: wikitext wikitext-103-raw-v1
        type: wikitext
        config: wikitext-103-raw-v1
        split: validation
        args: wikitext-103-raw-v1
      metrics:
      - name: Accuracy
        type: accuracy
        value: 0.46066022044935584
  ---
  
  <!-- This model card has been generated automatically according to the information the Trainer had access to. You
  should probably proofread and complete it, then remove this comment. -->
  
  # mobilebert_add_pre-training-complete
  
  This model is a fine-tuned version of [google/mobilebert-uncased](https://huggingface.co/google/mobilebert-uncased) on the wikitext wikitext-103-raw-v1 dataset.
  It achieves the following results on the evaluation set:
  - Loss: 2.9849
  - Accuracy: 0.4607
  
  ## Model description
  
  More information needed
  
  ## Intended uses & limitations
  
  More information needed
  
  ## Training and evaluation data
  
  More information needed
  
  ## Training procedure
  
  ### Training hyperparameters
  
  The following hyperparameters were used during training:
  - learning_rate: 5e-05
  - train_batch_size: 64
  - eval_batch_size: 64
  - seed: 10
  - distributed_type: multi-GPU
  - num_devices: 2
  - total_train_batch_size: 128
  - total_eval_batch_size: 128
  - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
  - lr_scheduler_type: linear
  - lr_scheduler_warmup_steps: 100
  - training_steps: 300000
  
  ### Training results
  
  | Training Loss | Epoch  | Step   | Validation Loss | Accuracy |
  |:-------------:|:------:|:------:|:---------------:|:--------:|
  | 4.8119        | 1.0    | 1787   | 4.3700          | 0.3199   |
  | 4.2649        | 2.0    | 3574   | 4.0930          | 0.3445   |
  | 4.0457        | 3.0    | 5361   | 3.9375          | 0.3545   |
  | 3.9099        | 4.0    | 7148   | 3.8534          | 0.3644   |
  | 3.8193        | 5.0    | 8935   | 3.7993          | 0.3669   |
  | 3.7517        | 6.0    | 10722  | 3.7414          | 0.3730   |
  | 3.6983        | 7.0    | 12509  | 3.6737          | 0.3818   |
  | 3.6565        | 8.0    | 14296  | 3.6657          | 0.3794   |
  | 3.619         | 9.0    | 16083  | 3.6129          | 0.3869   |
  | 3.5899        | 10.0   | 17870  | 3.5804          | 0.3910   |
  | 3.5597        | 11.0   | 19657  | 3.5432          | 0.3964   |
  | 3.5329        | 12.0   | 21444  | 3.5397          | 0.3958   |
  | 3.5088        | 13.0   | 23231  | 3.4896          | 0.4011   |
  | 3.4904        | 14.0   | 25018  | 3.4731          | 0.4000   |
  | 3.4703        | 15.0   | 26805  | 3.4971          | 0.3994   |
  | 3.4533        | 16.0   | 28592  | 3.4609          | 0.4049   |
  | 3.4369        | 17.0   | 30379  | 3.4411          | 0.4067   |
  | 3.423         | 18.0   | 32166  | 3.4219          | 0.4066   |
  | 3.4084        | 19.0   | 33953  | 3.4477          | 0.4014   |
  | 3.3949        | 20.0   | 35740  | 3.4013          | 0.4087   |
  | 3.3811        | 21.0   | 37527  | 3.3642          | 0.4130   |
  | 3.3688        | 22.0   | 39314  | 3.4173          | 0.4031   |
  | 3.3598        | 23.0   | 41101  | 3.4018          | 0.4101   |
  | 3.3484        | 24.0   | 42888  | 3.3499          | 0.4143   |
  | 3.3363        | 25.0   | 44675  | 3.3675          | 0.4119   |
  | 3.3274        | 26.0   | 46462  | 3.3562          | 0.4154   |
  | 3.3161        | 27.0   | 48249  | 3.3487          | 0.4159   |
  | 3.3073        | 28.0   | 50036  | 3.3293          | 0.4159   |
  | 3.2991        | 29.0   | 51823  | 3.3317          | 0.4160   |
  | 3.2899        | 30.0   | 53610  | 3.3058          | 0.4183   |
  | 3.2814        | 31.0   | 55397  | 3.2795          | 0.4235   |
  | 3.2734        | 32.0   | 57184  | 3.3185          | 0.4143   |
  | 3.266         | 33.0   | 58971  | 3.2682          | 0.4268   |
  | 3.2578        | 34.0   | 60758  | 3.3145          | 0.4181   |
  | 3.2506        | 35.0   | 62545  | 3.2726          | 0.4230   |
  | 3.2423        | 36.0   | 64332  | 3.2735          | 0.4218   |
  | 3.2359        | 37.0   | 66119  | 3.2845          | 0.4175   |
  | 3.2293        | 38.0   | 67906  | 3.3067          | 0.4193   |
  | 3.2207        | 39.0   | 69693  | 3.2586          | 0.4257   |
  | 3.2138        | 40.0   | 71480  | 3.2543          | 0.4250   |
  | 3.2077        | 41.0   | 73267  | 3.2395          | 0.4226   |
  | 3.202         | 42.0   | 75054  | 3.2224          | 0.4270   |
  | 3.1964        | 43.0   | 76841  | 3.2562          | 0.4234   |
  | 3.1925        | 44.0   | 78628  | 3.2544          | 0.4251   |
  | 3.1865        | 45.0   | 80415  | 3.2043          | 0.4353   |
  | 3.1812        | 46.0   | 82202  | 3.2280          | 0.4286   |
  | 3.1744        | 47.0   | 83989  | 3.2174          | 0.4276   |
  | 3.1699        | 48.0   | 85776  | 3.1972          | 0.4317   |
  | 3.1652        | 49.0   | 87563  | 3.2016          | 0.4302   |
  | 3.1609        | 50.0   | 89350  | 3.2018          | 0.4338   |
  | 3.1548        | 51.0