readme: ---
  tags:
  - masked-auto-encoding
  - generated_from_trainer
  model-index:
  - name: contrastive-mpixel22-small-ps
    results: []
  ---
  
  <!-- This model card has been generated automatically according to the information the Trainer had access to. You
  should probably proofread and complete it, then remove this comment. -->
  
  # contrastive-mpixel22-small-ps
  
  This model was trained from scratch on the wikipedia dataset.
  It achieves the following results on the evaluation set:
  - Loss: 0.4315
  
  ## Model description
  
  More information needed
  
  ## Intended uses & limitations
  
  More information needed
  
  ## Training and evaluation data
  
  More information needed
  
  ## Training procedure
  
  ### Training hyperparameters
  
  The following hyperparameters were used during training:
  - learning_rate: 0.0009
  - train_batch_size: 48
  - eval_batch_size: 8
  - seed: 1234
  - distributed_type: multi-GPU
  - num_devices: 32
  - total_train_batch_size: 1536
  - total_eval_batch_size: 256
  - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
  - lr_scheduler_type: cosine
  - lr_scheduler_warmup_ratio: 0.05
  - training_steps: 125000
  - mixed_precision_training: Native AMP
  
  ### Training results
  
  | Training Loss | Epoch | Step   | Validation Loss |
  |:-------------:|:-----:|:------:|:---------------:|
  | 1.3776        | 0.03  | 1000   | 0.8455          |
  | 0.9706        | 0.06  | 2000   | 0.7995          |
  | 1.0757        | 0.08  | 3000   | 0.7652          |
  | 1.0133        | 0.11  | 4000   | 0.7115          |
  | 0.7865        | 0.14  | 5000   | 0.6501          |
  | 1.7624        | 0.17  | 6000   | 0.9375          |
  | 1.2427        | 0.19  | 7000   | 0.8730          |
  | 0.9536        | 0.22  | 8000   | 0.6316          |
  | 0.7871        | 0.25  | 9000   | 0.6082          |
  | 0.8993        | 0.28  | 10000  | 0.6123          |
  | 0.6846        | 0.3   | 11000  | 0.5897          |
  | 0.792         | 0.33  | 12000  | 0.5869          |
  | 1.1064        | 0.36  | 13000  | 0.6096          |
  | 0.6449        | 0.39  | 14000  | 0.5692          |
  | 0.6374        | 0.41  | 15000  | 0.5677          |
  | 0.6385        | 0.44  | 16000  | 0.5662          |
  | 0.6469        | 0.47  | 17000  | 0.6052          |
  | 0.6093        | 0.5   | 18000  | 0.5499          |
  | 0.6058        | 0.52  | 19000  | 0.5468          |
  | 0.7922        | 0.55  | 20000  | 0.5492          |
  | 0.5918        | 0.58  | 21000  | 0.5395          |
  | 0.5852        | 0.61  | 22000  | 0.5352          |
  | 0.5925        | 0.63  | 23000  | 0.5340          |
  | 0.7584        | 0.66  | 24000  | 0.5463          |
  | 0.5728        | 0.69  | 25000  | 0.5274          |
  | 0.5668        | 0.72  | 26000  | 0.5220          |
  | 0.5632        | 0.74  | 27000  | 0.5184          |
  | 0.7635        | 0.77  | 28000  | 1.0526          |
  | 0.5682        | 0.8   | 29000  | 0.5209          |
  | 0.5672        | 0.83  | 30000  | 0.5167          |
  | 0.5535        | 0.85  | 31000  | 0.5126          |
  | 0.5505        | 0.88  | 32000  | 0.5101          |
  | 0.5467        | 0.91  | 33000  | 0.5085          |
  | 0.8298        | 0.94  | 34000  | 0.5129          |
  | 0.5421        | 0.96  | 35000  | 0.5054          |
  | 0.539         | 0.99  | 36000  | 0.5017          |
  | 0.5368        | 1.02  | 37000  | 0.5025          |
  | 0.5343        | 1.05  | 38000  | 0.4984          |
  | 0.5324        | 1.07  | 39000  | 0.4965          |
  | 0.6033        | 1.1   | 40000  | 0.5020          |
  | 0.5307        | 1.13  | 41000  | 0.4965          |
  | 0.5266        | 1.16  | 42000  | 0.4952          |
  | 0.532         | 1.19  | 43000  | 0.4913          |
  | 0.5227        | 1.21  | 44000  | 0.4899          |
  | 0.5209        | 1.24  | 45000  | 0.4878          |
  | 0.5193        | 1.27  | 46000  | 0.4873          |
  | 0.5216        | 1.3   | 47000  | 0.4885          |
  | 0.5149        | 1.32  | 48000  | 0.4819          |
  | 0.516         | 1.35  | 49000  | 0.4830          |
  | 0.5124        | 1.38  | 50000  | 0.4792          |
  | 0.5111        | 1.41  | 51000  | 0.4770          |
  | 0.5084        | 1.43  | 52000  | 0.4780          |
  | 0.5071        | 1.46  | 53000  | 0.4753          |
  | 0.5051        | 1.49  | 54000  | 0.4745          |
  | 0.5045        | 1.52  | 55000  | 0.4745          |
  | 0.5011        | 1.54  | 56000  | 0.4706          |
  | 0.4997        | 1.57  | 57000  | 0.4703          |
  | 0.4989        | 1.6   | 58000  | 0.4679          |
  | 0.4976        | 1.63  | 59000  | 0.4687          |
  | 0.6589        | 1.65  | 60000  | 0.4748          |
  | 0.4956        | 1.68  | 61000  | 0.4648          |
  | 0.4933        | 1.71  | 62000  | 0.4640          |
  | 0.4925        | 1.74  | 63000  | 0.4616          |
  | 0.4916        | 1.76  | 64000  | 0.4630          |
  | 0.4907        | 1.79  | 65000  | 0.4612          |
  | 0.4888        | 1.82  | 66000  | 0.4600          |
  | 0.4882        | 1.85  | 67000  | 0.4586          |
  | 0.4866        | 1.87  | 68000  | 0.4563          |
  | 0.486         | 1.9   | 69000  | 0.4564          |
  | 0.485         | 1.93  | 70000  | 0.4573          |
  | 0.4837        | 1.96  | 71000