readme: ---
  license: mit
  metrics:
  - accuracy
  - precision
  - recall
  model-index:
  - name: RoBERTa-RILE
    results: []
  widget: 
  - text: "Russia must end the war."
  - text: "Democratic institutions must be supported."
  - text: "The state must fight political corruption."
  - text: "Our energy economy must be nationalised."
  - text: "We must increase social spending."
  
  ---
  
  
  # RoBERTa-RILE
  
  This model is a fine-tuned version of [roberta-base](https://huggingface.co/roberta-base) on data from the [Manifesto Project](https://manifesto-project.wzb.eu/).
  
  
  ## Model description
  
  This model was trained on 115,943 manually annotated sentences to classify text into one of three political categories: "neutral", "left", "right".
  
  
  ## Intended uses & limitations
  
  The model output reproduces the limitations of the dataset in terms of country coverage, time span, domain definitions and potential biases of the annotators - as any supervised machine learning model would. Applying the model to other types of data (other types of texts, countries etc.) will reduce performance.
  
  ```python
  from transformers import pipeline
  import pandas as pd
  classifier = pipeline(
      task="text-classification",
      model="niksmer/RoBERTa-RILE")
  # Load text data you want to classify
  text = pd.read_csv("example.csv")["text_you_want_to_classify"].to_list()
  # Inference
  output = classifier(text)
  # Print output
  pd.DataFrame(output).head()
  ```
  
  ## Training and evaluation data
  
  ## Training and evaluation data
  
  RoBERTa-RILE was trained on the English-speaking subset of the [Manifesto Project Dataset (MPDS2021a)](https://manifesto-project.wzb.eu/datasets). The model was trained on 115,943 sentences from 163 political manifestos in 7 English-speaking countries (Australia, Canada, Ireland, New Zealand, South Africa, United Kingdom, United States). The manifestos were published between 1992 - 2020. 
  
  | Country        | Count manifestos | Count sentences | Time span        |
  |----------------|------------------|-----------------|--------------------|
  | Australia      | 18               | 14,887          | 2010-2016          |
  | Ireland        | 23               | 24,966          | 2007-2016          |
  | Canada         | 14               | 12,344          | 2004-2008 & 2015   |
  | New Zealand    | 46               | 35,079          | 1993-2017          |
  | South Africa   | 29               | 13,334          | 1994-2019          |
  | USA            | 9                | 13,188          | 1992   & 2004-2020 |
  | United Kingdom | 34               | 30,936          | 1997-2019          |
  
  Canadian manifestos between 2004 and 2008 are used as test data.
  
  The Manifesto Project mannually annotates individual sentences from political party manifestos in over 50 main categories - see the [codebook](https://manifesto-project.wzb.eu/down/papers/handbook_2021_version_5.pdf) for the exact definitions of each categorie. It has created a valid left-right-scale, the rile-index, to aaggregate manifesto in a standardized, onde-dimensional political space from left to right based on saliency-theory.
  RoBERTa-RILE classifies texts based on the rile index.
  
  ### Tain data
  
  Train data was slightly imbalanced.
  
  | Label | Description | Count |
  |------------|--------------|--------|
  | 0          | neutral       | 52,277 |
  | 1          | left       | 37,106 |
  | 2          | right       | 26,560 |
  
  Overall count: 115,943
  
  ### Validation data
  
  The validation was created by chance.
  
  | Label | Description | Count |
  |------------|--------------|--------|
  | 0          | neutral       | 9,198 |
  | 1          | left       | 6,637 |
  | 2          | right       | 4,626 |
  
  Overall count: 20,461
  
  ### Test data
  
  The test dataset contains ten canadian manifestos between 2004 and 2008.
  
  | Label | Description | Count |
  |------------|--------------|--------|
  | 0          | neutral       | 3,881 |
  | 1          | left       | 2,611 |
  | 2          | right       | 1,838 |
  
  Overall count: 8,330
  
  ## Training procedure
  
  ### Training hyperparameters
  
  The following hyperparameters were used during training:
  ```
  training_args = TrainingArguments(
      warmup_ratio=0.05,
      weight_decay=0.1, 
      learning_rate=1e-05,
      fp16 = True,
      evaluation_strategy="epoch",
      num_train_epochs=5,
      per_device_train_batch_size=16,
      per_device_eval_batch_size=16,
      save_strategy="no",
      logging_dir='logs',   
      logging_strategy= 'steps',     
      logging_steps=10,
      push_to_hub=True,
      hub_strategy="end")
  ```
  
  ### Training results
  
  | Training Loss | Epoch | Step | Validation Loss | Accuracy | F1-micro | F1-macro | F1-weighted | Precision | Recall |
  |:-------------:|:-----:|:----:|:---------------:|:--------:|:--------:|:--------:|:-----------:|:---------:|:------:|
  | 0.7442        | 1.0   | 1812 | 0.6827          | 0.7120   | 0.7120   | 0.7007   | 0.7126      | 0.7120    | 0.7120 |
  | 0.6447        | 2.0   | 3624 | 0.6618          | 0.7281   | 0.7281   | 0.7169   | 0.7281      | 0.7281    | 0.7281 |
  | 0.5467        | 3.0   | 5436 