bertweet-base_ordinal_7_seed42_EN-NL

This model is a fine-tuned version of vinai/bertweet-base on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 3.9585
• Mse: 6.1125
• Rmse: 2.4723
• Mae: 1.5109
• R2: 0.1525
• F1: 0.7447
• Precision: 0.7446
• Recall: 0.7479
• Accuracy: 0.7479

Model description

This is the best-performing model for Dutch irony detection. The model was fine-tuned both a mix of English and Dutch tweets. The model predicts one of 7 labels indicating for irony likelihood, where 0 is not ironic and 6 is ironic. When merging for binary classification, we advise mapping labels 0,1,2,3 as not-ironic and labels 4,5,6 as ironic.

Intended uses & limitations

More information needed

Training and evaluation data

The model was trained and evaluated on the TRIC dataset.

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 5e-06
• train_batch_size: 16
• eval_batch_size: 16
• seed: 42
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• lr_scheduler_warmup_steps: 200
• num_epochs: 10

Training results

Training Loss	Epoch	Step	Validation Loss	Mse	Rmse	Mae	R2	F1	Precision	Recall	Accuracy
5.7557	0.2141	100	5.6476	7.6297	2.7622	2.6574	-0.0806	0.4761	0.3859	0.6212	0.6212
5.2388	0.4283	200	5.2492	7.1086	2.6662	2.4741	-0.0068	0.4761	0.3859	0.6212	0.6212
4.9773	0.6424	300	5.0558	6.8733	2.6217	2.3016	0.0266	0.4761	0.3859	0.6212	0.6212
4.7427	0.8565	400	4.8666	6.6212	2.5732	2.1990	0.0623	0.4761	0.3859	0.6212	0.6212
4.6378	1.0707	500	4.6806	6.0772	2.4652	2.0941	0.1393	0.6773	0.6795	0.6888	0.6888
4.3851	1.2848	600	4.6153	6.2799	2.5060	1.9928	0.1106	0.6915	0.6964	0.6888	0.6888
4.3077	1.4989	700	4.5016	6.2147	2.4929	1.9276	0.1198	0.6882	0.6928	0.7008	0.7008
4.2337	1.7131	800	4.2877	5.5862	2.3635	1.8854	0.2088	0.7183	0.7218	0.7274	0.7274
4.2273	1.9272	900	4.3769	5.9397	2.4371	1.8601	0.1588	0.6994	0.6991	0.6996	0.6996
4.0563	2.1413	1000	4.2168	6.1013	2.4701	1.7033	0.1359	0.7088	0.7203	0.7238	0.7238
3.7778	2.3555	1100	4.1356	6.1098	2.4718	1.6562	0.1347	0.7260	0.7269	0.7322	0.7322
3.7206	2.5696	1200	4.2222	6.1062	2.4711	1.7394	0.1352	0.7245	0.7326	0.7214	0.7214
3.7175	2.7837	1300	4.0073	5.7021	2.3879	1.6224	0.1924	0.7277	0.7345	0.7382	0.7382
3.8003	2.9979	1400	4.1116	5.8166	2.4118	1.7346	0.1762	0.7258	0.7268	0.7250	0.7250
3.6247	3.2120	1500	4.1286	6.0663	2.4630	1.6876	0.1409	0.7309	0.7355	0.7286	0.7286
3.4364	3.4261	1600	4.2100	6.3353	2.5170	1.7467	0.1028	0.7235	0.7329	0.7201	0.7201
3.3301	3.6403	1700	4.0403	6.0483	2.4593	1.6357	0.1434	0.7436	0.7442	0.7431	0.7431
3.3634	3.8544	1800	3.9496	5.5790	2.3620	1.6297	0.2099	0.7259	0.7282	0.7334	0.7334
3.4602	4.0685	1900	3.8729	5.7334	2.3945	1.5597	0.1880	0.7402	0.7410	0.7455	0.7455
3.1223	4.2827	2000	4.0417	6.3812	2.5261	1.5875	0.0963	0.7144	0.7394	0.7346	0.7346
3.1337	4.4968	2100	4.0039	5.9493	2.4391	1.6285	0.1574	0.7389	0.7421	0.7370	0.7370
3.1321	4.7109	2200	3.9092	5.8926	2.4275	1.5742	0.1655	0.7347	0.7339	0.7358	0.7358
3.1927	4.9251	2300	4.0312	5.9928	2.4480	1.6140	0.1513	0.7459	0.7540	0.7431	0.7431
2.9806	5.1392	2400	3.9638	6.0145	2.4524	1.5633	0.1482	0.7524	0.7536	0.7515	0.7515
2.9582	5.3533	2500	3.9413	5.9409	2.4374	1.5549	0.1586	0.7539	0.7539	0.7539	0.7539
2.7418	5.5675	2600	3.9578	5.9843	2.4463	1.5525	0.1525	0.7456	0.7476	0.7443	0.7443
2.9866	5.7816	2700	3.8793	5.8070	2.4098	1.5416	0.1776	0.7426	0.7425	0.7467	0.7467
2.8627	5.9957	2800	3.8625	5.7805	2.4043	1.5103	0.1813	0.7615	0.7609	0.7624	0.7624
2.8191	6.2099	2900	3.9342	5.9964	2.4488	1.5211	0.1508	0.7628	0.7622	0.7636	0.7636
2.6259	6.4240	3000	3.9203	6.0893	2.4676	1.5006	0.1376	0.7487	0.7478	0.7503	0.7503
2.8785	6.6381	3100	3.8633	5.8444	2.4175	1.4946	0.1723	0.7600	0.7601	0.7600	0.7600
2.6016	6.8522	3200	4.0736	6.2654	2.5031	1.5923	0.1127	0.7456	0.7518	0.7431	0.7431
2.5155	7.0664	3300	3.9459	6.0688	2.4635	1.5211	0.1405	0.7584	0.7597	0.7575	0.7575
2.6918	7.2805	3400	3.9312	6.0072	2.4510	1.5271	0.1492	0.7541	0.7534	0.7551	0.7551

Framework versions

• Transformers 4.40.2
• Pytorch 2.1.2
• Datasets 2.18.0
• Tokenizers 0.19.1