readme: ---
  license: mit
  tags:
  - generated_from_trainer
  model-index:
  - name: airberta_airbnb_dat_lang8_xml_roberta_base
    results: []
  ---
  
  <!-- This model card has been generated automatically according to the information the Trainer had access to. You
  should probably proofread and complete it, then remove this comment. -->
  
  # airberta_airbnb_dat_lang8_xml_roberta_base
  
  This model is a fine-tuned version of [xlm-roberta-base](https://huggingface.co/xlm-roberta-base) on the Airbnb 1T dataset.
  It achieves the following results on the evaluation set:
  - Loss: 0.8560
  
  ## Model description
  
  AirBert is trained on a combination airbnb in-house data including reviews, listing descriptions, customer messages, message between guest and host, agent message and notes, house manual descriptions and customer emails, which contains a total size of 1T. 
  
  
  ## Training procedure
  
  ### Training hyperparameters
  
  The following hyperparameters were used during training:
  - learning_rate: 1e-05
  - train_batch_size: 8
  - eval_batch_size: 8
  - seed: 42
  - distributed_type: sagemaker_data_parallel
  - num_devices: 64
  - total_train_batch_size: 512
  - total_eval_batch_size: 512
  - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-06
  - lr_scheduler_type: linear
  - lr_scheduler_warmup_steps: 10000
  - training_steps: 1000000
  - mixed_precision_training: Native AMP
  
  ### Training results
  
  | Training Loss | Epoch | Step    | Validation Loss |
  |:-------------:|:-----:|:-------:|:---------------:|
  | 1.5943        | 0.02  | 5000    | 1.3627          |
  | 1.4333        | 0.04  | 10000   | 1.2376          |
  | 1.3562        | 0.06  | 15000   | 1.1754          |
  | 1.3105        | 0.08  | 20000   | 1.1387          |
  | 1.2822        | 0.1   | 25000   | 1.1142          |
  | 1.2615        | 0.12  | 30000   | 1.0941          |
  | 1.2418        | 0.14  | 35000   | 1.0804          |
  | 1.2273        | 0.16  | 40000   | 1.0663          |
  | 1.2156        | 0.18  | 45000   | 1.0552          |
  | 1.2036        | 0.2   | 50000   | 1.0466          |
  | 1.192         | 0.22  | 55000   | 1.0373          |
  | 1.1854        | 0.24  | 60000   | 1.0299          |
  | 1.1762        | 0.26  | 65000   | 1.0223          |
  | 1.1676        | 0.28  | 70000   | 1.0156          |
  | 1.1663        | 0.3   | 75000   | 1.0109          |
  | 1.1569        | 0.32  | 80000   | 1.0049          |
  | 1.1515        | 0.34  | 85000   | 0.9994          |
  | 1.145         | 0.36  | 90000   | 0.9961          |
  | 1.1427        | 0.38  | 95000   | 0.9912          |
  | 1.1369        | 0.4   | 100000  | 0.9865          |
  | 1.1304        | 0.42  | 105000  | 0.9834          |
  | 1.1273        | 0.44  | 110000  | 0.9783          |
  | 1.1256        | 0.46  | 115000  | 0.9771          |
  | 1.1209        | 0.48  | 120000  | 0.9728          |
  | 1.1153        | 0.5   | 125000  | 0.9695          |
  | 1.1126        | 0.52  | 130000  | 0.9661          |
  | 1.1088        | 0.54  | 135000  | 0.9645          |
  | 1.1087        | 0.56  | 140000  | 0.9605          |
  | 1.1049        | 0.58  | 145000  | 0.9577          |
  | 1.1           | 0.6   | 150000  | 0.9559          |
  | 1.0977        | 0.62  | 155000  | 0.9533          |
  | 1.0952        | 0.64  | 160000  | 0.9511          |
  | 1.0947        | 0.66  | 165000  | 0.9477          |
  | 1.0916        | 0.68  | 170000  | 0.9456          |
  | 1.0883        | 0.7   | 175000  | 0.9432          |
  | 1.086         | 0.72  | 180000  | 0.9423          |
  | 1.0842        | 0.74  | 185000  | 0.9392          |
  | 1.082         | 0.76  | 190000  | 0.9380          |
  | 1.0774        | 0.78  | 195000  | 0.9365          |
  | 1.0776        | 0.8   | 200000  | 0.9340          |
  | 1.0756        | 0.82  | 205000  | 0.9331          |
  | 1.0728        | 0.84  | 210000  | 0.9299          |
  | 1.0705        | 0.86  | 215000  | 0.9294          |
  | 1.0685        | 0.88  | 220000  | 0.9281          |
  | 1.0675        | 0.9   | 225000  | 0.9265          |
  | 1.0664        | 0.92  | 230000  | 0.9246          |
  | 1.0667        | 0.94  | 235000  | 0.9226          |
  | 1.0633        | 0.96  | 240000  | 0.9207          |
  | 1.0636        | 0.98  | 245000  | 0.9203          |
  | 1.0608        | 1.0   | 250000  | 0.9186          |
  | 1.06          | 1.02  | 255000  | 0.9179          |
  | 1.0582        | 1.04  | 260000  | 0.9167          |
  | 1.0552        | 1.06  | 265000  | 0.9156          |
  | 1.0548        | 1.08  | 270000  | 0.9143          |
  | 1.0537        | 1.1   | 275000  | 0.9134          |
  | 1.0537        | 1.12  | 280000  | 0.9117          |
  | 1.0525        | 1.14  | 285000  | 0.9106          |
  | 1.048         | 1.16  | 290000  | 0.9104          |
  | 1.05          | 1.18  | 295000  | 0.9086          |
  | 1.0459        | 1.2   | 300000  | 0.9075          |
  | 1.0443        | 1.22  | 305000  | 0.9063          |
  | 1.0445        | 1.24  | 310000  | 0.9057          |
  | 1.0454        | 1.26  | 315000  | 0.9042          |
  | 1.046         | 1.28  | 320000  | 0.9037          |
  | 1.0415        | 1.3   | 325000  | 0.9022          |
  | 1.041         | 1.3