readme: ---
  license: mit
  language: en
  tags:
  - generated_from_trainer
  model-index:
  - name: verdict-classifier-en
    results:
    - task:
        type: text-classification
        name: Verdict Classification
  widget:
  - "Even though it might look true, it has been taken out of context."
  ---
  
  # English Verdict Classifier
  This model is a fine-tuned version of [roberta-base](https://huggingface.co/roberta-base) on 2,500 deduplicated verdicts from [Google Fact Check Tools API](https://developers.google.com/fact-check/tools/api/reference/rest/v1alpha1/claims/search), translated into English with the [Google Cloud Translation API](https://cloud.google.com/translate/docs/reference/rest/).
  It achieves the following results on the evaluation set, being 1,000 such verdicts translated into English, but here including duplicates to represent the true distribution:
  - Loss: 0.1290
  - F1 Macro: 0.9171
  - F1 Misinformation: 0.9896
  - F1 Factual: 0.9890
  - F1 Other: 0.7727
  - Precision Macro: 0.8940
  - Precision Misinformation: 0.9954
  - Precision Factual: 0.9783
  - Precision Other: 0.7083
  
  ## Training procedure
  
  ### Training hyperparameters
  
  The following hyperparameters were used during training:
  - learning_rate: 2e-05
  - train_batch_size: 4
  - eval_batch_size: 4
  - seed: 42
  - gradient_accumulation_steps: 8
  - total_train_batch_size: 32
  - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
  - lr_scheduler_type: linear
  - lr_scheduler_warmup_steps: 2500
  - num_epochs: 1000
  
  ### Training results
  
  | Training Loss | Epoch | Step | Validation Loss | F1 Macro | F1 Misinformation | F1 Factual | F1 Other | Precision Macro | Precision Misinformation | Precision Factual | Precision Other |
  |:-------------:|:-----:|:----:|:---------------:|:--------:|:-----------------:|:----------:|:--------:|:----------:|:-------------------:|:------------:|:----------:|
  | 1.1493        | 0.16  | 50   | 1.1040          | 0.0550   | 0.0               | 0.1650     | 0.0      | 0.0300     | 0.0                 | 0.0899       | 0.0        |
  | 1.0899        | 0.32  | 100  | 1.0765          | 0.0619   | 0.0203            | 0.1654     | 0.0      | 0.2301     | 0.6                 | 0.0903       | 0.0        |
  | 1.0136        | 0.48  | 150  | 1.0487          | 0.3102   | 0.9306            | 0.0        | 0.0      | 0.2900     | 0.8701              | 0.0          | 0.0        |
  | 0.9868        | 0.64  | 200  | 1.0221          | 0.3102   | 0.9306            | 0.0        | 0.0      | 0.2900     | 0.8701              | 0.0          | 0.0        |
  | 0.9599        | 0.8   | 250  | 0.9801          | 0.3102   | 0.9306            | 0.0        | 0.0      | 0.2900     | 0.8701              | 0.0          | 0.0        |
  | 0.9554        | 0.96  | 300  | 0.9500          | 0.3102   | 0.9306            | 0.0        | 0.0      | 0.2900     | 0.8701              | 0.0          | 0.0        |
  | 0.935         | 1.12  | 350  | 0.9071          | 0.3102   | 0.9306            | 0.0        | 0.0      | 0.2900     | 0.8701              | 0.0          | 0.0        |
  | 0.948         | 1.28  | 400  | 0.8809          | 0.3102   | 0.9306            | 0.0        | 0.0      | 0.2900     | 0.8701              | 0.0          | 0.0        |
  | 0.9344        | 1.44  | 450  | 0.8258          | 0.3102   | 0.9306            | 0.0        | 0.0      | 0.2900     | 0.8701              | 0.0          | 0.0        |
  | 0.9182        | 1.6   | 500  | 0.7687          | 0.3102   | 0.9306            | 0.0        | 0.0      | 0.2900     | 0.8701              | 0.0          | 0.0        |
  | 0.8942        | 1.76  | 550  | 0.5787          | 0.3102   | 0.9306            | 0.0        | 0.0      | 0.2900     | 0.8701              | 0.0          | 0.0        |
  | 0.8932        | 1.92  | 600  | 0.4506          | 0.4043   | 0.9628            | 0.0        | 0.25     | 0.3777     | 0.9753              | 0.0          | 0.1579     |
  | 0.7448        | 2.08  | 650  | 0.2884          | 0.5323   | 0.9650            | 0.3303     | 0.3017   | 0.7075     | 0.9810              | 0.9474       | 0.1942     |
  | 0.6616        | 2.24  | 700  | 0.2162          | 0.8161   | 0.9710            | 0.9724     | 0.5051   | 0.7910     | 0.9824              | 0.9670       | 0.4237     |
  | 0.575         | 2.4   | 750  | 0.1754          | 0.8305   | 0.9714            | 0.9780     | 0.5421   | 0.7961     | 0.9881              | 0.9674       | 0.4328     |
  | 0.5246        | 2.56  | 800  | 0.1641          | 0.8102   | 0.9659            | 0.9175     | 0.5472   | 0.7614     | 0.9892              | 0.8558       | 0.4394     |
  | 0.481         | 2.72  | 850  | 0.1399          | 0.8407   | 0.9756            | 0.9780     | 0.5686   | 0.8082     | 0.9894              | 0.9674       | 0.4677     |
  | 0.4588        | 2.88  | 900  | 0.1212          | 0.8501   | 0.9786            | 0.9783     | 0.5934   | 0.8247     | 0.9871              | 0.9574       | 0.5294     |
  | 0.4512        | 3.04  | 950  | 0.1388          | 0.8270   | 0.9702            | 0.9836     | 0.5273   | 0.7904     | 0.98