xlm-roberta-large-pooled-cap-media2

Model description

An xlm-roberta-large model finetuned on multilingual (english, german, hungarian, spanish, slovakian) training data labelled with major topic codes from the Comparative Agendas Project. Furthermore we used the follwoing 18 media codes:

State and Local Government Administration (24)
Weather (25)
Fires, emergencies and natural disasters (26)
Crime and trials (27)
Arts, culture, entertainment and history (28)
Style and fashion (29)
Food (30)
Travel (31)
Wellbeing and learning (32)
Personal finance and real estate (33)
Personal technology and popular science (34)
Churches and Religion (35)
Celebrities and human interest (36)
Obituaries and death notices (37)
Sports (38)
Crosswords, puzzles, comics (39)
Media production/internal, letters (40)
Advertisements (41)

How to use the model

from transformers import AutoTokenizer, pipeline

tokenizer = AutoTokenizer.from_pretrained("xlm-roberta-large")
pipe = pipeline(
    model="poltextlab/xlm-roberta-large-pooled-cap-media2",
    task="text-classification",
    tokenizer=tokenizer,
    use_fast=False,
    token="<your_hf_read_only_token>"
)

text = "We will place an immediate 6-month halt on the finance driven closure of beds and wards, and set up an independent audit of needs and facilities."
pipe(text)

Gated access

Due to the gated access, you must pass the token parameter when loading the model. In earlier versions of the Transformers package, you may need to use the use_auth_token parameter instead.

Model performance

The model was evaluated on a test set of 74322 english examples.

Accuracy: 0.79.
Precision: 0.77.
Recall: 0.77
Weighted Average F1-score: 0.79

Heatmap

Classification Report

Class	precision	recall	f1-score	support
Macroeconomics (1)	0.71	0.75	0.73	2471
Civil Rights (2)	0.71	0.66	0.69	1886
Health (3)	0.81	0.83	0.82	2471
Agriculture (4)	0.77	0.76	0.76	811
Labor (5)	0.72	0.7	0.71	1277
Education (6)	0.84	0.87	0.86	2080
Environment (7)	0.76	0.79	0.78	1283
Energy (8)	0.79	0.83	0.81	1370
Immigration (9)	0.71	0.78	0.74	514
Transportation (10)	0.8	0.82	0.81	2375
Law and Crime (12)	0.68	0.67	0.67	2471
Social Welfare (13)	0.67	0.69	0.68	683
Housing (14)	0.72	0.71	0.71	1023
Banking, Finance, and Domestic Commerce (15)	0.72	0.68	0.7	2471
Defense (16)	0.74	0.77	0.75	2471
Technology (17)	0.73	0.73	0.73	1375
Foreign Trade (18)	0.71	0.64	0.67	533
International Affairs (19)	0.69	0.62	0.66	2471
Government Operations (20)	0.72	0.65	0.68	2471
Public Lands (21)	0.64	0.64	0.64	554
Culture (23)	0.73	0.75	0.74	2142
State and Local Government Administration (24)	0.79	0.73	0.76	2471
Weather (25)	0.98	0.98	0.98	2471
Fires, emergencies and natural disasters (26)	0.96	0.98	0.97	2471
Crime and trials (27)	0.77	0.84	0.8	2467
Arts, culture, entertainment and history (28)	0.78	0.72	0.75	2423
Style and fashion (29)	0.8	0.69	0.74	2407
Food (30)	0.79	0.83	0.81	2210
Travel (31)	0.8	0.86	0.83	2095
Wellbeing and learning (32)	0.77	0.81	0.79	2376
Personal finance and real estate (33)	0.84	0.85	0.85	2222
Personal technology and popular science (34)	0.82	0.83	0.82	2388
Churches and Religion (35)	0.92	0.94	0.93	2469
Celebrities and human interest (36)	0.84	0.87	0.86	2454
Obituaries and death notices (37)	0.88	0.92	0.9	2407
Sports (38)	0.89	0.89	0.89	2423
Crosswords, puzzles, comics (39)	0.96	0.95	0.96	126
Media production/internal, letters (40)	0.9	0.9	0.9	763
Advertisements (41)	0	0	0	5
No Policy and No Media Content (998)	0.82	0.8	0.81	2471
accuracy	0.79	0.79	0.79	0.79
macro avg	0.77	0.77	0.77	74322
weighted avg	0.79	0.79	0.79	74322

Inference platform

This model is used by the CAP Babel Machine, an open-source and free natural language processing tool, designed to simplify and speed up projects for comparative research.

Cooperation

Model performance can be significantly improved by extending our training sets. We appreciate every submission of CAP-coded corpora (of any domain and language) at poltextlab{at}poltextlab{dot}com or by using the CAP Babel Machine.

Debugging and issues

This architecture uses the sentencepiece tokenizer. In order to run the model before transformers==4.27 you need to install it manually.

If you encounter a RuntimeError when loading the model using the from_pretrained() method, adding ignore_mismatched_sizes=True should solve the issue.

poltextlab
/

xlm-roberta-large-pooled-cap-media2-v1

You need to agree to share your contact information to access this model