transformers pytorch xlm-roberta text-classification es autotrain_compatible endpoints_compatible region:us
---
language:
- es
metrics:
- accuracy
- f1-score
tags:
- text-classification
- pytorch
extra_gated_prompt: 'Our models are intended for academic use only. If you are not
  affiliated with an academic institution, please provide a rationale for using our
  models. Please allow us a few business days to manually review subscriptions.

  If you use our models for your work or research, please cite this paper: Sebők,
  M., Máté, Á., Ring, O., Kovács, V., & Lehoczki, R. (2024). Leveraging Open Large
  Language Models for Multilingual Policy Topic Classification: The Babel Machine
  Approach. Social Science Computer Review, 0(0). https://doi.org/10.1177/08944393241259434'
extra_gated_fields:
  Name: text
  Country: country
  Institution: text
  Institution Email: text
  Please specify your academic use case: text
---
# xlm-roberta-large-spanish-execspeech-cap-v3
## Model description
An `xlm-roberta-large` model fine-tuned on spanish training data containing executive speeches labeled with [major topic codes](https://www.comparativeagendas.net/pages/master-codebook) from the [Comparative Agendas Project](https://www.comparativeagendas.net/).

We follow the master codebook of the Comparative Agendas Project, and all of our models use the same major topic codes.

## How to use the model

```python
from transformers import AutoTokenizer, pipeline

tokenizer = AutoTokenizer.from_pretrained("xlm-roberta-large")
pipe = pipeline(
    model="poltextlab/xlm-roberta-large-spanish-execspeech-cap-v3",
    task="text-classification",
    tokenizer=tokenizer,
    use_fast=False,
    token="<your_hf_read_only_token>"
)

text = "We will place an immediate 6-month halt on the finance driven closure of beds and wards, and set up an independent audit of needs and facilities."
pipe(text)
```

The translation table from the model results to CAP codes is the following:
```python
CAP_NUM_DICT = {
    0: 1,
    1: 2,
    2: 3,
    3: 4,
    4: 5,
    5: 6,
    6: 7,
    7: 8,
    8: 9,
    9: 10,
    10: 12,
    11: 13,
    12: 14,
    13: 15,
    14: 16,
    15: 17,
    16: 18,
    17: 19,
    18: 20,
    19: 21,
    20: 23,
    21: 999,
}
```

We have included a 999 label because our models are fine-tuned on training data containing the label 'None' in addition to the 21 CAP major policy topic codes, indicating that the given text contains no relevant policy content. We use the label 999 for these cases.

### Gated access
Due to the gated access, you must pass the `token` parameter when loading the model. In earlier versions of the Transformers package, you may need to use the `use_auth_token` parameter instead.

## Model performance
The model was evaluated on a test set of 4794 examples.<br>
Model accuracy is **0.71**.
| label        |   precision |   recall |   f1-score |   support |
|:-------------|------------:|---------:|-----------:|----------:|
| 0            |        0.72 |     0.73 |       0.72 |       418 |
| 1            |        0.62 |     0.63 |       0.62 |        62 |
| 2            |        0.64 |     0.55 |       0.59 |        49 |
| 3            |        0.72 |     0.71 |       0.72 |       275 |
| 4            |        0.76 |     0.75 |       0.75 |       342 |
| 5            |        0.74 |     0.73 |       0.74 |       118 |
| 6            |        0.83 |     0.49 |       0.62 |        49 |
| 7            |        0.68 |     0.64 |       0.66 |        42 |
| 8            |        0.79 |     0.67 |       0.72 |        66 |
| 9            |        0.79 |     0.72 |       0.75 |       231 |
| 10           |        0.81 |     0.68 |       0.74 |       230 |
| 11           |        0.72 |     0.62 |       0.66 |       206 |
| 12           |        0.81 |     0.72 |       0.76 |       101 |
| 13           |        0.64 |     0.74 |       0.68 |        95 |
| 14           |        0.72 |     0.55 |       0.62 |       110 |
| 15           |        0.69 |     0.79 |       0.74 |       280 |
| 16           |        0.69 |     0.73 |       0.71 |       554 |
| 17           |        0.77 |     0.72 |       0.74 |       233 |
| 18           |        0.63 |     0.85 |       0.72 |       343 |
| 19           |        0.66 |     0.59 |       0.62 |       503 |
| 20           |        0.75 |     0.8  |       0.77 |        15 |
| 21           |        0.68 |     0.72 |       0.7  |       472 |
| macro avg    |        0.72 |     0.69 |       0.7  |      4794 |
| weighted avg |        0.71 |     0.71 |       0.71 |      4794 |

### Fine-tuning procedure
This model was fine-tuned with the following key hyperparameters:

- **Number of Training Epochs**: 10
- **Batch Size**: 8
- **Learning Rate**: 5e-06
- **Early Stopping**: enabled with a patience of 2 epochs

## Inference platform
This model is used by the [CAP Babel Machine](https://babel.poltextlab.com), an open-source and free natural language processing tool, designed to simplify and speed up projects for comparative research.  

## Cooperation
Model performance can be significantly improved by extending our training sets. We appreciate every submission of CAP-coded corpora (of any domain and language) at poltextlab{at}poltextlab{dot}com or by using the [CAP Babel Machine](https://babel.poltextlab.com).

## Reference
Sebők, M., Máté, Á., Ring, O., Kovács, V., & Lehoczki, R. (2024). Leveraging Open Large Language Models for Multilingual Policy Topic Classification: The Babel Machine Approach. Social Science Computer Review, 0(0). https://doi.org/10.1177/08944393241259434

## Debugging and issues
This architecture uses the `sentencepiece` tokenizer. In order to use the model before `transformers==4.27` you need to install it manually.

If you encounter a `RuntimeError` when loading the model using the `from_pretrained()` method, adding `ignore_mismatched_sizes=True` should solve the issue.