readme: ---
  tags:
  - masked-auto-encoding
  - generated_from_trainer
  model-index:
  - name: contrastive-mpixel22-small-ps-nocl
    results: []
  ---
  
  <!-- This model card has been generated automatically according to the information the Trainer had access to. You
  should probably proofread and complete it, then remove this comment. -->
  
  # contrastive-mpixel22-small-ps-nocl
  
  This model was trained from scratch on the wikipedia dataset.
  It achieves the following results on the evaluation set:
  - Loss: 0.4802
  
  ## Model description
  
  More information needed
  
  ## Intended uses & limitations
  
  More information needed
  
  ## Training and evaluation data
  
  More information needed
  
  ## Training procedure
  
  ### Training hyperparameters
  
  The following hyperparameters were used during training:
  - learning_rate: 0.0009
  - train_batch_size: 48
  - eval_batch_size: 8
  - seed: 1234
  - distributed_type: multi-GPU
  - num_devices: 32
  - total_train_batch_size: 1536
  - total_eval_batch_size: 256
  - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
  - lr_scheduler_type: cosine
  - lr_scheduler_warmup_ratio: 0.05
  - training_steps: 125000
  - mixed_precision_training: Native AMP
  
  ### Training results
  
  | Training Loss | Epoch | Step   | Validation Loss |
  |:-------------:|:-----:|:------:|:---------------:|
  | 0.7699        | 0.03  | 1000   | 0.7692          |
  | 0.7723        | 0.06  | 2000   | 0.7698          |
  | 0.767         | 0.08  | 3000   | 0.7673          |
  | 0.8095        | 0.11  | 4000   | 0.8122          |
  | 0.8092        | 0.14  | 5000   | 0.8116          |
  | 0.8088        | 0.17  | 6000   | 0.8114          |
  | 0.8087        | 0.19  | 7000   | 0.8114          |
  | 0.8084        | 0.22  | 8000   | 0.8116          |
  | 0.8096        | 0.25  | 9000   | 0.8122          |
  | 0.809         | 0.28  | 10000  | 0.8114          |
  | 0.8084        | 0.3   | 11000  | 0.8114          |
  | 0.8083        | 0.33  | 12000  | 0.8112          |
  | 0.8083        | 0.36  | 13000  | 0.8112          |
  | 0.8085        | 0.39  | 14000  | 0.8114          |
  | 0.8084        | 0.41  | 15000  | 0.8113          |
  | 0.8085        | 0.44  | 16000  | 0.8115          |
  | 0.7819        | 0.47  | 17000  | 0.9056          |
  | 0.7672        | 0.5   | 18000  | 0.8277          |
  | 0.7655        | 0.52  | 19000  | 0.7939          |
  | 0.7671        | 0.55  | 20000  | 0.7727          |
  | 0.7666        | 0.58  | 21000  | 0.7746          |
  | 0.7651        | 0.61  | 22000  | 0.7772          |
  | 0.7648        | 0.63  | 23000  | 0.7661          |
  | 0.7638        | 0.66  | 24000  | 0.7649          |
  | 0.7604        | 0.69  | 25000  | 0.7609          |
  | 0.7459        | 0.72  | 26000  | 0.7385          |
  | 0.7334        | 0.74  | 27000  | 0.7260          |
  | 0.7245        | 0.77  | 28000  | 0.7502          |
  | 0.7104        | 0.8   | 29000  | 0.7024          |
  | 0.6969        | 0.83  | 30000  | 0.6896          |
  | 0.6853        | 0.85  | 31000  | 0.6782          |
  | 0.6725        | 0.88  | 32000  | 0.6603          |
  | 0.6593        | 0.91  | 33000  | 0.6487          |
  | 0.653         | 0.94  | 34000  | 0.6433          |
  | 0.6474        | 0.96  | 35000  | 0.6366          |
  | 0.6432        | 0.99  | 36000  | 0.6325          |
  | 0.6396        | 1.02  | 37000  | 0.6290          |
  | 0.6347        | 1.05  | 38000  | 0.6241          |
  | 0.6334        | 1.07  | 39000  | 0.6205          |
  | 0.6302        | 1.1   | 40000  | 0.6185          |
  | 0.626         | 1.13  | 41000  | 0.6142          |
  | 0.6215        | 1.16  | 42000  | 0.6110          |
  | 0.6209        | 1.19  | 43000  | 0.6086          |
  | 0.6277        | 1.21  | 44000  | 0.6070          |
  | 0.6159        | 1.24  | 45000  | 0.6028          |
  | 0.6177        | 1.27  | 46000  | 0.5969          |
  | 0.6016        | 1.3   | 47000  | 0.5831          |
  | 0.6407        | 1.32  | 48000  | 0.5942          |
  | 0.5815        | 1.35  | 49000  | 0.5629          |
  | 0.6953        | 1.38  | 50000  | 0.8118          |
  | 0.8084        | 1.41  | 51000  | 0.8116          |
  | 0.8083        | 1.43  | 52000  | 0.8109          |
  | 0.7944        | 1.46  | 53000  | 0.7977          |
  | 0.7675        | 1.49  | 54000  | 0.7755          |
  | 0.7648        | 1.52  | 55000  | 0.7670          |
  | 0.7635        | 1.54  | 56000  | 0.7688          |
  | 0.7397        | 1.57  | 57000  | 0.7170          |
  | 0.6495        | 1.6   | 58000  | 0.6313          |
  | 0.6133        | 1.63  | 59000  | 0.5924          |
  | 0.5916        | 1.65  | 60000  | 0.5752          |
  | 0.5768        | 1.68  | 61000  | 0.5675          |
  | 0.5677        | 1.71  | 62000  | 0.5579          |
  | 0.5606        | 1.74  | 63000  | 0.5466          |
  | 0.5548        | 1.76  | 64000  | 0.5420          |
  | 0.5513        | 1.79  | 65000  | 0.5374          |
  | 0.5473        | 1.82  | 66000  | 0.5328          |
  | 0.5514        | 1.85  | 67000  | 0.5311          |
  | 0.5408        | 1.87  | 68000  | 0.5264          |
  | 0.538         | 1.9   | 69000  | 0.5242          |
  | 0.5358        | 1.93  | 70000  | 0.5235          |
  | 0.533         | 1.9