readme: ---
  tags:
  - masked-auto-encoding
  - generated_from_trainer
  model-index:
  - name: zh_wiki_small
    results: []
  ---
  
  <!-- This model card has been generated automatically according to the information the Trainer had access to. You
  should probably proofread and complete it, then remove this comment. -->
  
  # zh_wiki_small
  
  This model is a fine-tuned version of [](https://huggingface.co/) on the wikipedia dataset.
  It achieves the following results on the evaluation set:
  - Loss: 0.4159
  
  ## Model description
  
  More information needed
  
  ## Intended uses & limitations
  
  More information needed
  
  ## Training and evaluation data
  
  More information needed
  
  ## Training procedure
  
  ### Training hyperparameters
  
  The following hyperparameters were used during training:
  - learning_rate: 0.00015
  - train_batch_size: 32
  - eval_batch_size: 4
  - seed: 42
  - distributed_type: multi-GPU
  - num_devices: 4
  - gradient_accumulation_steps: 2
  - total_train_batch_size: 256
  - total_eval_batch_size: 16
  - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
  - lr_scheduler_type: cosine
  - lr_scheduler_warmup_ratio: 0.05
  - training_steps: 500000
  - mixed_precision_training: Native AMP
  
  ### Training results
  
  | Training Loss | Epoch | Step   | Validation Loss |
  |:-------------:|:-----:|:------:|:---------------:|
  | 0.4793        | 0.14  | 1000   | 0.4540          |
  | 0.4701        | 0.28  | 2000   | 0.4467          |
  | 0.4673        | 0.42  | 3000   | 0.4304          |
  | 0.4669        | 0.55  | 4000   | 0.4413          |
  | 0.4668        | 0.69  | 5000   | 0.4368          |
  | 0.4676        | 0.83  | 6000   | 0.4358          |
  | 0.4691        | 0.97  | 7000   | 0.4367          |
  | 0.4693        | 1.11  | 8000   | 0.4429          |
  | 0.4709        | 1.25  | 9000   | 0.4388          |
  | 0.4722        | 1.39  | 10000  | 0.4453          |
  | 0.4729        | 1.53  | 11000  | 0.4415          |
  | 0.4732        | 1.67  | 12000  | 0.4510          |
  | 0.4751        | 1.8   | 13000  | 0.4461          |
  | 0.4765        | 1.94  | 14000  | 0.4448          |
  | 0.477         | 2.08  | 15000  | 0.4498          |
  | 0.4779        | 2.22  | 16000  | 0.4447          |
  | 0.4795        | 2.36  | 17000  | 0.4430          |
  | 0.481         | 2.5   | 18000  | 0.4499          |
  | 0.4821        | 2.64  | 19000  | 0.4551          |
  | 0.4829        | 2.78  | 20000  | 0.4519          |
  | 0.4838        | 2.91  | 21000  | 0.4520          |
  | 0.4856        | 3.05  | 22000  | 0.4633          |
  | 0.4857        | 3.19  | 23000  | 0.4576          |
  | 0.4869        | 3.33  | 24000  | 0.4485          |
  | 0.4882        | 3.47  | 25000  | 0.4591          |
  | 0.4883        | 3.61  | 26000  | 0.4645          |
  | 0.4889        | 3.75  | 27000  | 0.4570          |
  | 0.4884        | 3.89  | 28000  | 0.4572          |
  | 0.4897        | 4.02  | 29000  | 0.4553          |
  | 0.4883        | 4.16  | 30000  | 0.4534          |
  | 0.4881        | 4.3   | 31000  | 0.4587          |
  | 0.4889        | 4.44  | 32000  | 0.4632          |
  | 0.4886        | 4.58  | 33000  | 0.4587          |
  | 0.4883        | 4.72  | 34000  | 0.4621          |
  | 0.4876        | 4.86  | 35000  | 0.4522          |
  | 0.4878        | 5.0   | 36000  | 0.4560          |
  | 0.4883        | 5.13  | 37000  | 0.4579          |
  | 0.4882        | 5.27  | 38000  | 0.4554          |
  | 0.4883        | 5.41  | 39000  | 0.4588          |
  | 0.4872        | 5.55  | 40000  | 0.4561          |
  | 0.4868        | 5.69  | 41000  | 0.4614          |
  | 0.4875        | 5.83  | 42000  | 0.4584          |
  | 0.4868        | 5.97  | 43000  | 0.4619          |
  | 0.4874        | 6.11  | 44000  | 0.4519          |
  | 0.4874        | 6.24  | 45000  | 0.4625          |
  | 0.487         | 6.38  | 46000  | 0.4579          |
  | 0.4872        | 6.52  | 47000  | 0.4534          |
  | 0.4872        | 6.66  | 48000  | 0.4516          |
  | 0.4865        | 6.8   | 49000  | 0.4635          |
  | 0.4865        | 6.94  | 50000  | 0.4610          |
  | 0.4863        | 7.08  | 51000  | 0.4515          |
  | 0.4861        | 7.22  | 52000  | 0.4584          |
  | 0.4866        | 7.35  | 53000  | 0.4541          |
  | 0.4862        | 7.49  | 54000  | 0.4508          |
  | 0.4863        | 7.63  | 55000  | 0.4565          |
  | 0.486         | 7.77  | 56000  | 0.4665          |
  | 0.486         | 7.91  | 57000  | 0.4565          |
  | 0.4861        | 8.05  | 58000  | 0.4580          |
  | 0.4852        | 8.19  | 59000  | 0.4596          |
  | 0.4846        | 8.33  | 60000  | 0.4527          |
  | 0.4848        | 8.46  | 61000  | 0.4505          |
  | 0.4849        | 8.6   | 62000  | 0.4407          |
  | 0.4851        | 8.74  | 63000  | 0.4579          |
  | 0.4848        | 8.88  | 64000  | 0.4559          |
  | 0.4851        | 9.02  | 65000  | 0.4505          |
  | 0.4846        | 9.16  | 66000  | 0.4615          |
  | 0.4842        | 9.3   | 67000  | 0.4618          |
  | 0.484         | 9.44  | 68000  | 0.4559          |
  | 0.4841        | 9.57  | 69000  | 0.4613          |
  | 0.484         | 9.71  | 70000  | 0.4527          |
  | 0.