readme: ---
  license: cc-by-4.0
  tags:
  - generated_from_trainer
  datasets:
  - dcc_v1.1.0
  metrics:
  - accuracy
  model-index:
  - name: dfm-bert-large-v1-2048bsz-1Msteps
    results:
    - task:
        name: Masked Language Modeling
        type: fill-mask
      dataset:
        name: dcc_v1.1.0
        type: dcc_v1.1.0
      metrics:
      - name: Accuracy
        type: accuracy
        value: 0.7328012831797821
  ---
  
  <!-- This model card has been generated automatically according to the information the Trainer had access to. You
  should probably proofread and complete it, then remove this comment. -->
  
  # dfm-bert-large-v1-2048bsz-1Msteps
  
  This model is a fine-tuned version of [NbAiLab/nb-bert-large](https://huggingface.co/NbAiLab/nb-bert-large) on the dcc_v1.1.0 dataset.
  It achieves the following results on the evaluation set:
  - Loss: 1.3175
  - Accuracy: 0.7328
  
  ## Model description
  
  More information needed
  
  ## Intended uses & limitations
  
  More information needed
  
  ## Training and evaluation data
  
  More information needed
  
  ## Training procedure
  
  ### Training hyperparameters
  
  The following hyperparameters were used during training:
  - learning_rate: 0.0002
  - train_batch_size: 256
  - eval_batch_size: 256
  - seed: 42
  - gradient_accumulation_steps: 8
  - total_train_batch_size: 2048
  - optimizer: Adam with betas=(0.9,0.98) and epsilon=1e-06
  - lr_scheduler_type: linear
  - lr_scheduler_warmup_steps: 10000
  - training_steps: 100000
  - mixed_precision_training: Native AMP
  
  ### Training results
  
  | Training Loss | Epoch | Step   | Accuracy | Validation Loss |
  |:-------------:|:-----:|:------:|:--------:|:---------------:|
  | 1.4239        | 0.02  | 2000   | 0.6481   | 1.9361          |
  | 1.299         | 0.04  | 4000   | 0.6646   | 1.8073          |
  | 1.2008        | 0.06  | 6000   | 0.6766   | 1.7281          |
  | 1.193         | 0.08  | 8000   | 0.6770   | 1.6885          |
  | 1.138         | 0.1   | 10000  | 0.6803   | 1.6729          |
  | 1.1401        | 0.12  | 12000  | 0.6729   | 1.7227          |
  | 4.1932        | 0.14  | 14000  | 0.3016   | 4.5455          |
  | 2.3732        | 0.16  | 16000  | 0.5607   | 2.3964          |
  | 1.2114        | 0.18  | 18000  | 0.6667   | 1.7638          |
  | 1.1482        | 0.2   | 20000  | 0.6576   | 1.7839          |
  | 1.0815        | 0.22  | 22000  | 0.6862   | 1.6308          |
  | 1.085         | 0.24  | 24000  | 0.6837   | 1.6383          |
  | 1.0788        | 0.26  | 26000  | 0.6812   | 1.6585          |
  | 1.0389        | 0.28  | 28000  | 0.6861   | 1.5927          |
  | 1.0283        | 0.3   | 30000  | 0.6937   | 1.5779          |
  | 1.0145        | 0.32  | 32000  | 0.6967   | 1.5439          |
  | 1.0023        | 0.34  | 34000  | 0.6980   | 1.5237          |
  | 0.9976        | 0.36  | 36000  | 0.6962   | 1.5692          |
  | 1.019         | 0.38  | 38000  | 0.6970   | 1.5460          |
  | 1.0137        | 0.4   | 40000  | 0.6967   | 1.5564          |
  | 1.0067        | 0.42  | 42000  | 0.7008   | 1.5176          |
  | 0.992         | 0.44  | 44000  | 0.7060   | 1.4806          |
  | 0.9796        | 0.46  | 46000  | 0.7026   | 1.5085          |
  | 0.976         | 0.48  | 48000  | 0.7092   | 1.4705          |
  | 0.9571        | 0.5   | 50000  | 0.7052   | 1.4895          |
  | 0.9723        | 0.52  | 52000  | 0.7135   | 1.4516          |
  | 0.9581        | 0.54  | 54000  | 0.7145   | 1.4343          |
  | 0.9511        | 0.56  | 56000  | 0.7124   | 1.4334          |
  | 0.9608        | 0.58  | 58000  | 0.7151   | 1.4268          |
  | 0.9588        | 0.6   | 60000  | 0.7127   | 1.4471          |
  | 0.9473        | 0.62  | 62000  | 0.7202   | 1.4037          |
  | 0.9266        | 0.64  | 64000  | 0.7158   | 1.4225          |
  | 0.925         | 0.66  | 66000  | 0.7208   | 1.3940          |
  | 0.9242        | 0.68  | 68000  | 0.7189   | 1.4090          |
  | 0.9141        | 0.7   | 70000  | 0.7229   | 1.3831          |
  | 0.8884        | 0.72  | 72000  | 1.3738   | 0.7233          |
  | 0.9145        | 0.74  | 74000  | 1.3478   | 0.7275          |
  | 0.8741        | 0.76  | 76000  | 1.3691   | 0.7255          |
  | 0.8752        | 0.78  | 78000  | 1.3530   | 0.7276          |
  | 0.8634        | 0.8   | 80000  | 1.3428   | 0.7272          |
  | 0.8882        | 0.82  | 82000  | 1.3490   | 0.7270          |
  | 0.8872        | 0.84  | 84000  | 1.3458   | 0.7296          |
  | 0.892         | 0.86  | 86000  | 1.3382   | 0.7279          |
  | 0.9002        | 0.88  | 88000  | 1.3091   | 0.7341          |
  | 0.8805        | 0.9   | 90000  | 1.3209   | 0.7310          |
  | 0.8944        | 0.92  | 92000  | 1.3133   | 0.7332          |
  | 0.8605        | 0.94  | 94000  | 1.3404   | 0.7311          |
  | 0.879         | 0.96  | 96000  | 1.2890   | 0.7356          |
  | 0.871         | 0.98  | 98000  | 1.2954   | 0.7352          |
  | 0.8676        | 1.0   | 100000 | 1.2935   | 0.7369          |
  
  
  ### Framework versions
  
  - Transformers 4.20.1
  - Pytorch 1.11.0+cu102
  - Datasets 2.5.3.dev0
  - Tokenizers 0.12.1
  