readme: ---
  license: mit
  tags:
  - generated_from_trainer
  model-index:
  - name: BERiT_2000_2_layers_300_epochs
    results: []
  ---
  
  <!-- This model card has been generated automatically according to the information the Trainer had access to. You
  should probably proofread and complete it, then remove this comment. -->
  
  # BERiT_2000_2_layers_300_epochs
  
  This model is a fine-tuned version of [roberta-base](https://huggingface.co/roberta-base) on an unknown dataset.
  It achieves the following results on the evaluation set:
  - Loss: 5.0648
  
  ## Model description
  
  More information needed
  
  ## Intended uses & limitations
  
  More information needed
  
  ## Training and evaluation data
  
  More information needed
  
  ## Training procedure
  
  ### Training hyperparameters
  
  The following hyperparameters were used during training:
  - learning_rate: 0.0005
  - train_batch_size: 8
  - eval_batch_size: 8
  - seed: 42
  - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
  - lr_scheduler_type: linear
  - num_epochs: 300
  - label_smoothing_factor: 0.2
  
  ### Training results
  
  | Training Loss | Epoch  | Step   | Validation Loss |
  |:-------------:|:------:|:------:|:---------------:|
  | 14.9639       | 0.19   | 500    | 8.4999          |
  | 7.8976        | 0.39   | 1000   | 7.3944          |
  | 7.3281        | 0.58   | 1500   | 7.1320          |
  | 7.1202        | 0.77   | 2000   | 7.0376          |
  | 7.0738        | 0.97   | 2500   | 7.0277          |
  | 7.0327        | 1.16   | 3000   | 6.9313          |
  | 6.9775        | 1.36   | 3500   | 6.8580          |
  | 6.9568        | 1.55   | 4000   | 6.7909          |
  | 6.9242        | 1.74   | 4500   | 6.7869          |
  | 6.8842        | 1.94   | 5000   | 6.7403          |
  | 6.8904        | 2.13   | 5500   | 6.7860          |
  | 6.8757        | 2.32   | 6000   | 6.7235          |
  | 6.8164        | 2.52   | 6500   | 6.7383          |
  | 6.8439        | 2.71   | 7000   | 6.6904          |
  | 6.8074        | 2.9    | 7500   | 6.7116          |
  | 6.79          | 3.1    | 8000   | 6.6995          |
  | 6.7915        | 3.29   | 8500   | 6.6930          |
  | 6.7664        | 3.49   | 9000   | 6.6794          |
  | 6.7822        | 3.68   | 9500   | 6.6467          |
  | 6.7585        | 3.87   | 10000  | 6.6787          |
  | 6.7784        | 4.07   | 10500  | 6.6596          |
  | 6.7344        | 4.26   | 11000  | 6.6315          |
  | 6.7374        | 4.45   | 11500  | 6.7104          |
  | 6.7309        | 4.65   | 12000  | 6.6566          |
  | 6.728         | 4.84   | 12500  | 6.6726          |
  | 6.7154        | 5.03   | 13000  | 6.6502          |
  | 6.7159        | 5.23   | 13500  | 6.6477          |
  | 6.7114        | 5.42   | 14000  | 6.6440          |
  | 6.7111        | 5.62   | 14500  | 6.6685          |
  | 6.7038        | 5.81   | 15000  | 6.6363          |
  | 6.7037        | 6.0    | 15500  | 6.6036          |
  | 6.7           | 6.2    | 16000  | 6.6199          |
  | 6.6864        | 6.39   | 16500  | 6.5995          |
  | 6.6944        | 6.58   | 17000  | 6.6211          |
  | 6.6743        | 6.78   | 17500  | 6.6274          |
  | 6.6519        | 6.97   | 18000  | 6.5919          |
  | 6.6707        | 7.16   | 18500  | 6.6141          |
  | 6.6722        | 7.36   | 19000  | 6.5356          |
  | 6.6695        | 7.55   | 19500  | 6.5895          |
  | 6.6699        | 7.75   | 20000  | 6.5913          |
  | 6.6783        | 7.94   | 20500  | 6.6037          |
  | 6.651         | 8.13   | 21000  | 6.6032          |
  | 6.6415        | 8.33   | 21500  | 6.5818          |
  | 6.6485        | 8.52   | 22000  | 6.5829          |
  | 6.6232        | 8.71   | 22500  | 6.6029          |
  | 6.6407        | 8.91   | 23000  | 6.5676          |
  | 6.6265        | 9.1    | 23500  | 6.6313          |
  | 6.6436        | 9.3    | 24000  | 6.5415          |
  | 6.6196        | 9.49   | 24500  | 6.5655          |
  | 6.6093        | 9.68   | 25000  | 6.5663          |
  | 6.6354        | 9.88   | 25500  | 6.5946          |
  | 6.6202        | 10.07  | 26000  | 6.5805          |
  | 6.5849        | 10.26  | 26500  | 6.5799          |
  | 6.6035        | 10.46  | 27000  | 6.5763          |
  | 6.5922        | 10.65  | 27500  | 6.5716          |
  | 6.5924        | 10.84  | 28000  | 6.5744          |
  | 6.6083        | 11.04  | 28500  | 6.5326          |
  | 6.5896        | 11.23  | 29000  | 6.5797          |
  | 6.607         | 11.43  | 29500  | 6.5312          |
  | 6.5942        | 11.62  | 30000  | 6.5917          |
  | 6.5863        | 11.81  | 30500  | 6.5619          |
  | 6.5841        | 12.01  | 31000  | 6.5590          |
  | 6.548         | 12.2   | 31500  | 6.4872          |
  | 6.5831        | 12.39  | 32000  | 6.5914          |
  | 6.5577        | 12.59  | 32500  | 6.5784          |
  | 6.5585        | 12.78  | 33000  | 6.5267          |
  | 6.5722        | 12.97  | 33500  | 6.5539          |
  | 6.5832        | 13.17  | 34000  | 6.5535          |
  | 6.5704        | 13.36  | 34500  | 6.5624          |
  | 6.5437        | 13.56  | 35000  | 6.5531          |
  | 6.5492        | 13.75  | 35500  | 6.5616          |
  | 6.5437        | 13.94