readme: ---
  tags:
  - generated_from_trainer
  model-index:
  - name: airberta_ccnet_airbnb_dat
    results: []
  ---
  
  <!-- This model card has been generated automatically according to the information the Trainer had access to. You
  should probably proofread and complete it, then remove this comment. -->
  
  # airberta_ccnet_airbnb_dat
  
  This model is a pretained model based on deberta architecture on Airbnb (100G) + ccnet (500G).
  It achieves the following results on the evaluation set:
  - Loss: 0.9213
  
  ## Model description
  
  More information needed
  
  ## Intended uses & limitations
  
  More information needed
  
  ## Training and evaluation data
  
  More information needed
  
  ## Training procedure
  
  ### Training hyperparameters
  
  The following hyperparameters were used during training:
  - learning_rate: 0.0002
  - train_batch_size: 32
  - eval_batch_size: 32
  - seed: 42
  - distributed_type: sagemaker_data_parallel
  - num_devices: 64
  - total_train_batch_size: 2048
  - total_eval_batch_size: 2048
  - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-06
  - lr_scheduler_type: linear
  - lr_scheduler_warmup_steps: 10000
  - training_steps: 1000000
  - mixed_precision_training: Native AMP
  
  ### Training results
  
  | Training Loss | Epoch | Step    | Validation Loss |
  |:-------------:|:-----:|:-------:|:---------------:|
  | 6.4807        | 0.06  | 5000    | 6.2557          |
  | 2.3529        | 0.12  | 10000   | 2.1745          |
  | 1.8769        | 0.19  | 15000   | 1.7572          |
  | 1.6818        | 0.25  | 20000   | 1.5789          |
  | 1.5754        | 0.31  | 25000   | 1.4746          |
  | 1.5058        | 0.37  | 30000   | 1.4060          |
  | 1.4561        | 0.43  | 35000   | 1.3562          |
  | 1.4169        | 0.5   | 40000   | 1.3214          |
  | 1.3871        | 0.56  | 45000   | 1.2913          |
  | 1.3628        | 0.62  | 50000   | 1.2665          |
  | 1.341         | 0.68  | 55000   | 1.2464          |
  | 1.3222        | 0.75  | 60000   | 1.2288          |
  | 1.307         | 0.81  | 65000   | 1.2146          |
  | 1.2942        | 0.87  | 70000   | 1.2003          |
  | 1.2824        | 0.93  | 75000   | 1.1874          |
  | 1.2717        | 0.99  | 80000   | 1.1784          |
  | 1.2618        | 1.06  | 85000   | 1.1696          |
  | 1.2533        | 1.12  | 90000   | 1.1604          |
  | 1.2445        | 1.18  | 95000   | 1.1537          |
  | 1.2371        | 1.24  | 100000  | 1.1457          |
  | 1.2319        | 1.3   | 105000  | 1.1386          |
  | 1.2256        | 1.37  | 110000  | 1.1309          |
  | 1.2202        | 1.43  | 115000  | 1.1267          |
  | 1.2132        | 1.49  | 120000  | 1.1226          |
  | 1.2077        | 1.55  | 125000  | 1.1169          |
  | 1.2033        | 1.62  | 130000  | 1.1127          |
  | 1.1982        | 1.68  | 135000  | 1.1061          |
  | 1.1929        | 1.74  | 140000  | 1.1032          |
  | 1.1892        | 1.8   | 145000  | 1.0993          |
  | 1.1861        | 1.86  | 150000  | 1.0938          |
  | 1.1798        | 1.93  | 155000  | 1.0901          |
  | 1.179         | 1.99  | 160000  | 1.0886          |
  | 1.1744        | 2.05  | 165000  | 1.0838          |
  | 1.1717        | 2.11  | 170000  | 1.0809          |
  | 1.168         | 2.17  | 175000  | 1.0772          |
  | 1.164         | 2.24  | 180000  | 1.0752          |
  | 1.1615        | 2.3   | 185000  | 1.0719          |
  | 1.1586        | 2.36  | 190000  | 1.0717          |
  | 1.1564        | 2.42  | 195000  | 1.0674          |
  | 1.1539        | 2.48  | 200000  | 1.0626          |
  | 1.15          | 2.55  | 205000  | 1.0623          |
  | 1.1473        | 2.61  | 210000  | 1.0591          |
  | 1.1453        | 2.67  | 215000  | 1.0582          |
  | 1.1436        | 2.73  | 220000  | 1.0558          |
  | 1.1425        | 2.8   | 225000  | 1.0531          |
  | 1.1401        | 2.86  | 230000  | 1.0501          |
  | 1.1374        | 2.92  | 235000  | 1.0483          |
  | 1.1354        | 2.98  | 240000  | 1.0458          |
  | 1.1337        | 3.04  | 245000  | 1.0455          |
  | 1.1308        | 3.11  | 250000  | 1.0434          |
  | 1.1296        | 3.17  | 255000  | 1.0406          |
  | 1.1282        | 3.23  | 260000  | 1.0393          |
  | 1.126         | 3.29  | 265000  | 1.0381          |
  | 1.1244        | 3.35  | 270000  | 1.0356          |
  | 1.1209        | 3.42  | 275000  | 1.0348          |
  | 1.1192        | 3.48  | 280000  | 1.0311          |
  | 1.1186        | 3.54  | 285000  | 1.0319          |
  | 1.1173        | 3.6   | 290000  | 1.0301          |
  | 1.1144        | 3.67  | 295000  | 1.0270          |
  | 1.1139        | 3.73  | 300000  | 1.0254          |
  | 1.1115        | 3.79  | 305000  | 1.0260          |
  | 1.1115        | 3.85  | 310000  | 1.0217          |
  | 1.1088        | 3.91  | 315000  | 1.0219          |
  | 1.1064        | 3.98  | 320000  | 1.0190          |
  | 1.1058        | 4.04  | 325000  | 1.0184          |
  | 1.1056        | 4.1   | 330000  | 1.0167          |
  | 1.1039        | 4.16  | 335000  | 1.0154          |
  | 1.1027        | 4.22  | 340000  | 1.0159          |
  | 1.1002        | 4.29  | 345000  | 1.0126       