readme: ---
  license: mit
  language: 
  - en
  metrics:
  - accuracy
  - precision
  - recall
  model-index:
  - name: PolicyBERTa-7d
    results: []
  widget: 
  - text: "Russia must end the war."
  - text: "Democratic institutions must be supported."
  - text: "The state must fight political corruption."
  - text: "Our energy economy must be nationalised."
  - text: "We must increase social spending."
  
  ---
  
  # PolicyBERTa-7d
  
  This model is a fine-tuned version of [roberta-base](https://huggingface.co/roberta-base) on data from the [Manifesto Project](https://manifesto-project.wzb.eu/). It was inspired by the model from [Laurer (2020)](https://huggingface.co/MoritzLaurer/policy-distilbert-7d).
  
  It achieves the following results on the evaluation set:
  - Loss: 0.8549
  - Accuracy: 0.7059
  - F1-micro: 0.7059
  - F1-macro: 0.6683
  - F1-weighted: 0.7033
  - Precision: 0.7059
  - Recall: 0.7059
  
  ## Model description
  
  This model was trained on 115,943 manually annotated sentences to classify text into one of seven political categories: "external relations", "freedom and democracy", "political system", "economy", "welfare and quality of life", "fabric of society" and "social groups".
  
  
  ## Intended uses & limitations
  
  The model output reproduces the limitations of the dataset in terms of country coverage, time span, domain definitions and potential biases of the annotators - as any supervised machine learning model would. Applying the model to other types of data (other types of texts, countries etc.) will reduce performance.
  
  ```python
  from transformers import pipeline
  import pandas as pd
  
  classifier = pipeline(
      task="text-classification",
      model="niksmer/PolicyBERTa-7d")
  
  # Load text data you want to classify
  text = pd.read_csv("example.csv")["text_you_want_to_classify"].to_list()
  
  # Inference
  output = classifier(text)
  
  # Print output
  pd.DataFrame(output).head()
  ```
  
  ## Training and evaluation data
  
  PolicyBERTa-7d was trained on the English-speaking subset of the [Manifesto Project Dataset (MPDS2021a)](https://manifesto-project.wzb.eu/datasets). The model was trained on 115,943 sentences from 163 political manifestos in 7 English-speaking countries (Australia, Canada, Ireland, New Zealand, South Africa, United Kingdom, United States). The manifestos were published between 1992 - 2020. 
  
  | Country        | Count manifestos | Count sentences | Time span        |
  |----------------|------------------|-----------------|--------------------|
  | Australia      | 18               | 14,887          | 2010-2016          |
  | Ireland        | 23               | 24,966          | 2007-2016          |
  | Canada         | 14               | 12,344          | 2004-2008 & 2015   |
  | New Zealand    | 46               | 35,079          | 1993-2017          |
  | South Africa   | 29               | 13,334          | 1994-2019          |
  | USA            | 9                | 13,188          | 1992   & 2004-2020 |
  | United Kingdom | 34               | 30,936          | 1997-2019          |
  
  Canadian manifestos between 2004 and 2008 are used as test data.
  
  
  The Manifesto Project mannually annotates individual sentences from political party manifestos in 7 main political domains: 'Economy', 'External Relations', 'Fabric of Society', 'Freedom and Democracy', 'Political System', 'Welfare and Quality of Life' or 'Social Groups' - see the [codebook](https://manifesto-project.wzb.eu/down/papers/handbook_2021_version_5.pdf) for the exact definitions of each domain. 
  
  ### Tain data
  
  Train data was higly imbalanced.
  
  | Label | Description | Count |
  |------------|--------------|--------|
  | 0          | external relations       | 7,640 |
  | 1          | freedom and democracy       | 5,880 |
  | 2          | political system       | 11,234 |
  | 3          | economy | 29,218 |
  | 4          | welfare and quality of life       | 37,200 |
  | 5          | fabric of society | 13,594 |
  | 6          | social groups       | 11,177 |
  
  Overall count: 115,943
  
  ### Validation data
  
  The validation was created by chance.
  
  | Label | Description | Count |
  |------------|--------------|--------|
  | 0          | external relations       | 1,345 |
  | 1          | freedom and democracy       | 1,043 |
  | 2          | political system       | 2,038 |
  | 3          | economy | 5,140 |
  | 4          | welfare and quality of life       | 6,554 |
  | 5          | fabric of society | 2,384 |
  | 6          | social groups       | 1,957 |
  
  Overall count: 20,461
  
  ## Test data
  
  The test dataset contains ten canadian manifestos between 2004 and 2008.
  
  | Label | Description | Count |
  |------------|--------------|--------|
  | 0          | external relations       | 824 |
  | 1          | freedom and democracy       | 296 |
  | 2          | political system       | 1,041 |
  | 3          | economy | 2,188 |
  | 4          | welfare and quality of life       | 2,654 |
  | 5          | fabric of society | 940 |
  | 6          | social groups       | 387 |
  
  Overall count: 8,330
  
  ## Training procedure
  
  ### Training hyperparameters
  
  The following hyperparame