readme: ---
  license: mit
  tags:
  - generated_from_trainer
  model-index:
  - name: BERiT_2000_2_layers_40_epochs
    results: []
  ---
  
  <!-- This model card has been generated automatically according to the information the Trainer had access to. You
  should probably proofread and complete it, then remove this comment. -->
  
  # BERiT_2000_2_layers_40_epochs
  
  This model is a fine-tuned version of [roberta-base](https://huggingface.co/roberta-base) on an unknown dataset.
  It achieves the following results on the evaluation set:
  - Loss: 6.8375
  
  ## Model description
  
  More information needed
  
  ## Intended uses & limitations
  
  More information needed
  
  ## Training and evaluation data
  
  More information needed
  
  ## Training procedure
  
  ### Training hyperparameters
  
  The following hyperparameters were used during training:
  - learning_rate: 0.0005
  - train_batch_size: 8
  - eval_batch_size: 8
  - seed: 42
  - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
  - lr_scheduler_type: linear
  - num_epochs: 40
  - label_smoothing_factor: 0.2
  
  ### Training results
  
  | Training Loss | Epoch | Step   | Validation Loss |
  |:-------------:|:-----:|:------:|:---------------:|
  | 15.0851       | 0.19  | 500    | 8.5468          |
  | 7.8971        | 0.39  | 1000   | 7.3376          |
  | 7.3108        | 0.58  | 1500   | 7.1632          |
  | 7.134         | 0.77  | 2000   | 7.0700          |
  | 7.0956        | 0.97  | 2500   | 7.0723          |
  | 7.0511        | 1.16  | 3000   | 6.9560          |
  | 7.0313        | 1.36  | 3500   | 6.9492          |
  | 7.0028        | 1.55  | 4000   | 6.9048          |
  | 6.9563        | 1.74  | 4500   | 6.8456          |
  | 6.9214        | 1.94  | 5000   | 6.8019          |
  | 11.1596       | 2.13  | 5500   | 7.5882          |
  | 7.5824        | 2.32  | 6000   | 7.1291          |
  | 7.2581        | 2.52  | 6500   | 7.1123          |
  | 7.2232        | 2.71  | 7000   | 7.1059          |
  | 7.1734        | 2.9   | 7500   | 7.1120          |
  | 7.1504        | 3.1   | 8000   | 7.0946          |
  | 7.1314        | 3.29  | 8500   | 7.0799          |
  | 7.1236        | 3.49  | 9000   | 7.1175          |
  | 7.1275        | 3.68  | 9500   | 7.0905          |
  | 7.1087        | 3.87  | 10000  | 7.0839          |
  | 7.1212        | 4.07  | 10500  | 7.0822          |
  | 7.1136        | 4.26  | 11000  | 7.0703          |
  | 7.1025        | 4.45  | 11500  | 7.1035          |
  | 7.0931        | 4.65  | 12000  | 7.0759          |
  | 7.0899        | 4.84  | 12500  | 7.0883          |
  | 7.0834        | 5.03  | 13000  | 7.1307          |
  | 7.0761        | 5.23  | 13500  | 7.0642          |
  | 7.0706        | 5.42  | 14000  | 7.0324          |
  | 7.0678        | 5.62  | 14500  | 7.0704          |
  | 7.0614        | 5.81  | 15000  | 7.0317          |
  | 7.0569        | 6.0   | 15500  | 7.0421          |
  | 7.057         | 6.2   | 16000  | 7.0250          |
  | 7.0503        | 6.39  | 16500  | 7.0129          |
  | 7.0529        | 6.58  | 17000  | 7.0316          |
  | 7.0453        | 6.78  | 17500  | 7.0436          |
  | 7.0218        | 6.97  | 18000  | 7.0064          |
  | 7.0415        | 7.16  | 18500  | 7.0385          |
  | 7.0338        | 7.36  | 19000  | 6.9756          |
  | 7.0488        | 7.55  | 19500  | 7.0054          |
  | 7.0347        | 7.75  | 20000  | 6.9946          |
  | 7.0464        | 7.94  | 20500  | 7.0055          |
  | 7.017         | 8.13  | 21000  | 7.0158          |
  | 7.0159        | 8.33  | 21500  | 7.0052          |
  | 7.0223        | 8.52  | 22000  | 6.9925          |
  | 6.9989        | 8.71  | 22500  | 7.0307          |
  | 7.0218        | 8.91  | 23000  | 6.9767          |
  | 6.9998        | 9.1   | 23500  | 7.0096          |
  | 7.01          | 9.3   | 24000  | 6.9599          |
  | 6.9964        | 9.49  | 24500  | 6.9896          |
  | 6.9906        | 9.68  | 25000  | 6.9903          |
  | 7.0336        | 9.88  | 25500  | 6.9807          |
  | 7.0053        | 10.07 | 26000  | 6.9776          |
  | 6.9826        | 10.26 | 26500  | 6.9836          |
  | 6.9897        | 10.46 | 27000  | 6.9886          |
  | 6.9829        | 10.65 | 27500  | 6.9991          |
  | 6.9849        | 10.84 | 28000  | 6.9651          |
  | 6.9901        | 11.04 | 28500  | 6.9822          |
  | 6.9852        | 11.23 | 29000  | 6.9921          |
  | 6.9757        | 11.43 | 29500  | 6.9636          |
  | 6.991         | 11.62 | 30000  | 6.9952          |
  | 6.9818        | 11.81 | 30500  | 6.9799          |
  | 6.9911        | 12.01 | 31000  | 6.9725          |
  | 6.9423        | 12.2  | 31500  | 6.9540          |
  | 6.9885        | 12.39 | 32000  | 6.9771          |
  | 6.9636        | 12.59 | 32500  | 6.9475          |
  | 6.9567        | 12.78 | 33000  | 6.9653          |
  | 6.9749        | 12.97 | 33500  | 6.9711          |
  | 6.9739        | 13.17 | 34000  | 6.9691          |
  | 6.9651        | 13.36 | 34500  | 6.9569          |
  | 6.9599        | 13.56 | 35000  | 6.9608          |
  | 6.957         | 13.75 | 35500  | 6.9531          |
  | 6.9539        | 13.94 | 36000  | 6.9704          |
  | 6.958         | 14.14 | 36500  | 6.9478     