readme: ---
  tags:
  - generated_from_trainer
  metrics:
  - accuracy
  model-index:
  - name: roberta-tiny-10M
    results: []
  ---
  
  <!-- This model card has been generated automatically according to the information the Trainer had access to. You
  should probably proofread and complete it, then remove this comment. -->
  
  # roberta-tiny-10M
  
  This model was trained from scratch on an unknown dataset.
  It achieves the following results on the evaluation set:
  - Loss: 6.4867
  - Accuracy: 0.1377
  
  ## Model description
  
  More information needed
  
  ## Intended uses & limitations
  
  More information needed
  
  ## Training and evaluation data
  
  More information needed
  
  ## Training procedure
  
  ### Training hyperparameters
  
  The following hyperparameters were used during training:
  - learning_rate: 2e-05
  - train_batch_size: 32
  - eval_batch_size: 64
  - seed: 42
  - gradient_accumulation_steps: 4
  - total_train_batch_size: 128
  - optimizer: Adam with betas=(0.9,0.98) and epsilon=1e-08
  - lr_scheduler_type: constant_with_warmup
  - lr_scheduler_warmup_steps: 50
  - num_epochs: 40.0
  
  ### Training results
  
  | Training Loss | Epoch | Step | Validation Loss | Accuracy |
  |:-------------:|:-----:|:----:|:---------------:|:--------:|
  | 10.5287       | 0.26  | 50   | 10.4478         | 0.0481   |
  | 10.0073       | 0.51  | 100  | 9.9550          | 0.0488   |
  | 9.6268        | 0.77  | 150  | 9.5865          | 0.0488   |
  | 9.9837        | 1.03  | 200  | 9.2502          | 0.0471   |
  | 8.9701        | 1.29  | 250  | 8.9370          | 0.0466   |
  | 8.6689        | 1.54  | 300  | 8.6447          | 0.0473   |
  | 8.3893        | 1.8   | 350  | 8.3794          | 0.0473   |
  | 8.1697        | 2.06  | 400  | 8.1342          | 0.0506   |
  | 7.926         | 2.32  | 450  | 7.9221          | 0.0617   |
  | 7.7329        | 2.58  | 500  | 7.7398          | 0.0627   |
  | 7.582         | 2.83  | 550  | 7.5844          | 0.0691   |
  | 7.4419        | 3.09  | 600  | 7.4620          | 0.0729   |
  | 7.3658        | 3.35  | 650  | 7.3735          | 0.0781   |
  | 7.2857        | 3.61  | 700  | 7.3049          | 0.0801   |
  | 7.224         | 3.86  | 750  | 7.2554          | 0.0831   |
  | 7.1851        | 4.12  | 800  | 7.2082          | 0.0853   |
  | 7.1327        | 4.38  | 850  | 7.1678          | 0.0878   |
  | 7.0947        | 4.64  | 900  | 7.1326          | 0.0909   |
  | 7.0761        | 4.89  | 950  | 7.1069          | 0.0919   |
  | 7.0551        | 5.15  | 1000 | 7.0806          | 0.0943   |
  | 7.0389        | 5.41  | 1050 | 7.0588          | 0.0952   |
  | 7.0226        | 5.67  | 1100 | 7.0379          | 0.0964   |
  | 6.9992        | 5.92  | 1150 | 7.0142          | 0.0975   |
  | 6.9382        | 6.18  | 1200 | 6.9979          | 0.0986   |
  | 6.956         | 6.44  | 1250 | 6.9828          | 0.0987   |
  | 6.9425        | 6.7   | 1300 | 6.9619          | 0.1008   |
  | 6.8872        | 6.96  | 1350 | 6.9468          | 0.1014   |
  | 6.8848        | 7.22  | 1400 | 6.9320          | 0.1024   |
  | 6.8578        | 7.47  | 1450 | 6.9190          | 0.1039   |
  | 6.8699        | 7.73  | 1500 | 6.9022          | 0.1050   |
  | 6.8402        | 7.99  | 1550 | 6.8910          | 0.1057   |
  | 6.8172        | 8.25  | 1600 | 6.8730          | 0.1069   |
  | 6.823         | 8.5   | 1650 | 6.8662          | 0.1073   |
  | 6.8028        | 8.76  | 1700 | 6.8487          | 0.1082   |
  | 7.3146        | 9.02  | 1750 | 6.8400          | 0.1083   |
  | 6.8014        | 9.28  | 1800 | 6.8303          | 0.1092   |
  | 6.8028        | 9.53  | 1850 | 6.8226          | 0.1088   |
  | 6.7817        | 9.79  | 1900 | 6.8079          | 0.1107   |
  | 7.28          | 10.05 | 1950 | 6.8021          | 0.1115   |
  | 6.7624        | 10.31 | 2000 | 6.7930          | 0.1118   |
  | 6.7416        | 10.56 | 2050 | 6.7868          | 0.1124   |
  | 6.7288        | 10.82 | 2100 | 6.7805          | 0.1133   |
  | 6.7468        | 11.08 | 2150 | 6.7720          | 0.1123   |
  | 6.7387        | 11.34 | 2200 | 6.7636          | 0.1135   |
  | 6.7242        | 11.6  | 2250 | 6.7557          | 0.1134   |
  | 6.702         | 11.85 | 2300 | 6.7496          | 0.1141   |
  | 6.6662        | 12.11 | 2350 | 6.7433          | 0.1150   |
  | 6.6781        | 12.37 | 2400 | 6.7362          | 0.1148   |
  | 6.6743        | 12.63 | 2450 | 6.7275          | 0.1161   |
  | 6.6843        | 12.88 | 2500 | 6.7247          | 0.1165   |
  | 6.6726        | 13.14 | 2550 | 6.7127          | 0.1173   |
  | 6.6656        | 13.4  | 2600 | 6.7098          | 0.1170   |
  | 6.6428        | 13.66 | 2650 | 6.7019          | 0.1185   |
  | 6.6355        | 13.91 | 2700 | 6.6979          | 0.1175   |
  | 6.6521        | 14.17 | 2750 | 6.6923          | 0.1188   |
  | 6.6735        | 14.43 | 2800 | 6.6842          | 0.1186   |
  | 6.6151        | 14.69 | 2850 | 6.6791          | 0.1195   |
  | 6.6248        | 14.94 | 2900 | 6.6752          | 0.1198   |
  | 6.6427        | 15.21 | 2950 | 6.6665          | 0.1207   |
  | 6.5947        | 15.46 | 3000 | 6.6639          | 0.1207   |
  | 6.6199        | 15.72 | 3050 | 6.6598          | 0.1217   |
  | 6.6127        | 15.98 | 3100 | 6.