readme: ---
  tags:
  - generated_from_trainer
  metrics:
  - accuracy
  model-index:
  - name: nl_electra
    results: []
  ---
  
  <!-- This model card has been generated automatically according to the information the Trainer had access to. You
  should probably proofread and complete it, then remove this comment. -->
  
  # nl_electra
  
  This model is a pretrained version of [ELECTRA](https://huggingface.co/docs/transformers/model_doc/electra) on the Dutch subset of the [CC100](https://huggingface.co/datasets/cc100) dataset.
  It achieves the following results on the evaluation set:
  - Loss: 2.4650
  - Accuracy: 0.5392
  
  ## Training procedure
  
  ### Training hyperparameters
  
  The following hyperparameters were used during training:
  - learning_rate: 5e-05
  - train_batch_size: 16
  - eval_batch_size: 16
  - seed: 703
  - gradient_accumulation_steps: 32
  - total_train_batch_size: 512
  - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
  - lr_scheduler_type: linear
  - lr_scheduler_warmup_steps: 8000
  - num_epochs: 400.0
  
  ### Training results
  
  | Training Loss | Epoch  | Step   | Validation Loss | Accuracy |
  |:-------------:|:------:|:------:|:---------------:|:--------:|
  | No log        | 0.67   | 500    | 9.9977          | 0.0486   |
  | No log        | 1.35   | 1000   | 9.5620          | 0.0543   |
  | No log        | 2.02   | 1500   | 8.9306          | 0.0741   |
  | No log        | 2.69   | 2000   | 8.2617          | 0.0826   |
  | No log        | 3.36   | 2500   | 7.6880          | 0.0792   |
  | No log        | 4.04   | 3000   | 7.3316          | 0.0757   |
  | No log        | 4.71   | 3500   | 7.1944          | 0.0747   |
  | No log        | 5.38   | 4000   | 7.1349          | 0.0802   |
  | No log        | 6.06   | 4500   | 7.0752          | 0.0887   |
  | 8.201         | 6.73   | 5000   | 7.0046          | 0.1021   |
  | 8.201         | 7.4    | 5500   | 6.9295          | 0.1090   |
  | 8.201         | 8.08   | 6000   | 6.8483          | 0.1132   |
  | 8.201         | 8.75   | 6500   | 6.7750          | 0.1171   |
  | 8.201         | 9.42   | 7000   | 6.7116          | 0.1187   |
  | 8.201         | 10.09  | 7500   | 6.6560          | 0.1218   |
  | 8.201         | 10.77  | 8000   | 6.6178          | 0.1239   |
  | 8.201         | 11.44  | 8500   | 6.5824          | 0.1255   |
  | 8.201         | 12.11  | 9000   | 6.5521          | 0.1273   |
  | 8.201         | 12.79  | 9500   | 6.5203          | 0.1292   |
  | 6.7257        | 13.46  | 10000  | 6.5027          | 0.1303   |
  | 6.7257        | 14.13  | 10500  | 6.4809          | 0.1314   |
  | 6.7257        | 14.8   | 11000  | 6.4631          | 0.1322   |
  | 6.7257        | 15.48  | 11500  | 6.4483          | 0.1329   |
  | 6.7257        | 16.15  | 12000  | 6.4320          | 0.1338   |
  | 6.7257        | 16.82  | 12500  | 6.4169          | 0.1348   |
  | 6.7257        | 17.5   | 13000  | 6.4067          | 0.1359   |
  | 6.7257        | 18.17  | 13500  | 6.3994          | 0.1359   |
  | 6.7257        | 18.84  | 14000  | 6.3823          | 0.1368   |
  | 6.7257        | 19.52  | 14500  | 6.3759          | 0.1373   |
  | 6.4502        | 20.19  | 15000  | 6.3629          | 0.1374   |
  | 6.4502        | 20.86  | 15500  | 6.3638          | 0.1373   |
  | 6.4502        | 21.53  | 16000  | 6.3505          | 0.1382   |
  | 6.4502        | 22.21  | 16500  | 6.3416          | 0.1387   |
  | 6.4502        | 22.88  | 17000  | 6.3420          | 0.1383   |
  | 6.4502        | 23.55  | 17500  | 6.3330          | 0.1389   |
  | 6.4502        | 24.23  | 18000  | 6.3289          | 0.1388   |
  | 6.4502        | 24.9   | 18500  | 6.3184          | 0.1389   |
  | 6.4502        | 25.57  | 19000  | 6.3099          | 0.1396   |
  | 6.4502        | 26.24  | 19500  | 6.2789          | 0.1405   |
  | 6.3474        | 26.92  | 20000  | 6.2398          | 0.1404   |
  | 6.3474        | 27.59  | 20500  | 6.2012          | 0.1412   |
  | 6.3474        | 28.26  | 21000  | 6.1803          | 0.1414   |
  | 6.3474        | 28.94  | 21500  | 6.1579          | 0.1414   |
  | 6.3474        | 29.61  | 22000  | 6.1403          | 0.1431   |
  | 6.3474        | 30.28  | 22500  | 6.1316          | 0.1423   |
  | 6.3474        | 30.96  | 23000  | 6.1102          | 0.1435   |
  | 6.3474        | 31.63  | 23500  | 6.0998          | 0.1439   |
  | 6.3474        | 32.3   | 24000  | 6.0867          | 0.1446   |
  | 6.3474        | 32.97  | 24500  | 6.0700          | 0.1451   |
  | 6.1758        | 33.65  | 25000  | 6.0554          | 0.1457   |
  | 6.1758        | 34.32  | 25500  | 6.0487          | 0.1457   |
  | 6.1758        | 34.99  | 26000  | 6.0328          | 0.1469   |
  | 6.1758        | 35.67  | 26500  | 6.0265          | 0.1469   |
  | 6.1758        | 36.34  | 27000  | 5.9992          | 0.1486   |
  | 6.1758        | 37.01  | 27500  | 5.9934          | 0.1485   |
  | 6.1758        | 37.68  | 28000  | 5.9702          | 0.1501   |
  | 6.1758        | 38.36  | 28500  | 5.9661          | 0.1503   |
  | 6.1758        | 39.03  | 29000  | 5.9558          | 0.1512   |
  | 6.1758        | 39.7   | 29500  | 5.9321          | 0.1528   |
  | 6.052         | 40.38  |