readme: ---
  license: mit
  tags:
  - generated_from_trainer
  model-index:
  - name: BERiT_2000_custom_architecture_40_epochs_ls_.1
    results: []
  ---
  
  <!-- This model card has been generated automatically according to the information the Trainer had access to. You
  should probably proofread and complete it, then remove this comment. -->
  
  # BERiT_2000_custom_architecture_40_epochs_ls_.1
  
  This model is a fine-tuned version of [roberta-base](https://huggingface.co/roberta-base) on an unknown dataset.
  It achieves the following results on the evaluation set:
  - Loss: 6.0606
  
  ## Model description
  
  More information needed
  
  ## Intended uses & limitations
  
  More information needed
  
  ## Training and evaluation data
  
  More information needed
  
  ## Training procedure
  
  ### Training hyperparameters
  
  The following hyperparameters were used during training:
  - learning_rate: 0.0005
  - train_batch_size: 8
  - eval_batch_size: 8
  - seed: 42
  - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
  - lr_scheduler_type: linear
  - num_epochs: 40
  - label_smoothing_factor: 0.1
  
  ### Training results
  
  | Training Loss | Epoch | Step   | Validation Loss |
  |:-------------:|:-----:|:------:|:---------------:|
  | 15.8251       | 0.19  | 500    | 8.3567          |
  | 7.8217        | 0.39  | 1000   | 7.2693          |
  | 7.2486        | 0.58  | 1500   | 7.0533          |
  | 7.0209        | 0.77  | 2000   | 6.9330          |
  | 6.9572        | 0.97  | 2500   | 6.9266          |
  | 6.897         | 1.16  | 3000   | 6.7520          |
  | 6.8273        | 1.36  | 3500   | 6.6686          |
  | 6.7692        | 1.55  | 4000   | 6.5613          |
  | 6.7269        | 1.74  | 4500   | 6.5789          |
  | 6.6783        | 1.94  | 5000   | 6.5036          |
  | 6.6792        | 2.13  | 5500   | 6.5714          |
  | 6.6638        | 2.32  | 6000   | 6.4733          |
  | 6.5913        | 2.52  | 6500   | 6.4967          |
  | 6.634         | 2.71  | 7000   | 6.4530          |
  | 6.5831        | 2.9   | 7500   | 6.4763          |
  | 6.556         | 3.1   | 8000   | 6.4616          |
  | 6.5676        | 3.29  | 8500   | 6.4449          |
  | 6.5447        | 3.49  | 9000   | 6.4795          |
  | 6.5531        | 3.68  | 9500   | 6.4253          |
  | 6.5356        | 3.87  | 10000  | 6.4508          |
  | 6.5677        | 4.07  | 10500  | 6.4002          |
  | 6.5035        | 4.26  | 11000  | 6.3985          |
  | 6.5112        | 4.45  | 11500  | 6.4798          |
  | 6.5038        | 4.65  | 12000  | 6.4138          |
  | 6.504         | 4.84  | 12500  | 6.4381          |
  | 6.4993        | 5.03  | 13000  | 6.4241          |
  | 6.4853        | 5.23  | 13500  | 6.4275          |
  | 6.4881        | 5.42  | 14000  | 6.3979          |
  | 6.4882        | 5.62  | 14500  | 6.4468          |
  | 6.4728        | 5.81  | 15000  | 6.4319          |
  | 6.4853        | 6.0   | 15500  | 6.4124          |
  | 6.4764        | 6.2   | 16000  | 6.4013          |
  | 6.4638        | 6.39  | 16500  | 6.3955          |
  | 6.4727        | 6.58  | 17000  | 6.4140          |
  | 6.4595        | 6.78  | 17500  | 6.4229          |
  | 6.4273        | 6.97  | 18000  | 6.3758          |
  | 6.4594        | 7.16  | 18500  | 6.3889          |
  | 6.4457        | 7.36  | 19000  | 6.3175          |
  | 6.4538        | 7.55  | 19500  | 6.3748          |
  | 6.453         | 7.75  | 20000  | 6.3782          |
  | 6.4662        | 7.94  | 20500  | 6.3953          |
  | 6.437         | 8.13  | 21000  | 6.4125          |
  | 6.4342        | 8.33  | 21500  | 6.3641          |
  | 6.4424        | 8.52  | 22000  | 6.3911          |
  | 6.4035        | 8.71  | 22500  | 6.4061          |
  | 6.44          | 8.91  | 23000  | 6.3751          |
  | 6.4206        | 9.1   | 23500  | 6.4066          |
  | 6.4297        | 9.3   | 24000  | 6.3342          |
  | 6.408         | 9.49  | 24500  | 6.3508          |
  | 6.4026        | 9.68  | 25000  | 6.3609          |
  | 6.4262        | 9.88  | 25500  | 6.4123          |
  | 6.4311        | 10.07 | 26000  | 6.3867          |
  | 6.3788        | 10.26 | 26500  | 6.3724          |
  | 6.3943        | 10.46 | 27000  | 6.4175          |
  | 6.3867        | 10.65 | 27500  | 6.3834          |
  | 6.3868        | 10.84 | 28000  | 6.4073          |
  | 6.4119        | 11.04 | 28500  | 6.3267          |
  | 6.387         | 11.23 | 29000  | 6.4038          |
  | 6.4092        | 11.43 | 29500  | 6.3426          |
  | 6.3994        | 11.62 | 30000  | 6.3827          |
  | 6.3868        | 11.81 | 30500  | 6.3681          |
  | 6.3779        | 12.01 | 31000  | 6.3419          |
  | 6.3427        | 12.2  | 31500  | 6.2775          |
  | 6.3763        | 12.39 | 32000  | 6.4021          |
  | 6.349         | 12.59 | 32500  | 6.3800          |
  | 6.3538        | 12.78 | 33000  | 6.3369          |
  | 6.3734        | 12.97 | 33500  | 6.3495          |
  | 6.3857        | 13.17 | 34000  | 6.3715          |
  | 6.3678        | 13.36 | 34500  | 6.3540          |
  | 6.3345        | 13.56 | 35000  | 6.3504          |
  | 6.3399        | 13.75 | 35500  | 6.3641          |
  | 6.3383        | 13.94 | 36000  | 6.3556          |
  | 6.3637    