RoBERTa-Bio-ES

This model is a fine-tuned version of PlanTL-GOB-ES/roberta-base-biomedical-clinical-es on the breast cancer dataset in Spanish: anvorja/breast_cancer_es_dataset. It achieves the following results on the evaluation set:

• Loss: 0.1023
• Precision: 0.9405
• Recall: 0.9577
• F1: 0.9490
• Accuracy: 0.9831

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 1e-05
• train_batch_size: 8
• eval_batch_size: 8
• seed: 42
• gradient_accumulation_steps: 8
• total_train_batch_size: 64
• optimizer: Use OptimizerNames.ADAMW_TORCH with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: cosine_with_restarts
• lr_scheduler_warmup_ratio: 0.2
• num_epochs: 20
• mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss	Precision	Recall	F1	Accuracy
3.3536	1.0	113	3.1266	0.0	0.0	0.0	0.5493
1.7045	2.0	226	1.5127	0.5690	0.2992	0.3922	0.7015
1.0293	3.0	339	0.9145	0.5449	0.4747	0.5074	0.7965
0.6538	4.0	452	0.5374	0.7423	0.7620	0.7520	0.9012
0.4369	5.0	565	0.3537	0.8216	0.8605	0.8406	0.9371
0.3371	6.0	678	0.2680	0.8505	0.8838	0.8669	0.9525
0.2646	7.0	791	0.2140	0.8742	0.9129	0.8931	0.9636
0.2229	8.0	904	0.1800	0.8974	0.9217	0.9094	0.9684
0.1664	9.0	1017	0.1590	0.9044	0.9312	0.9176	0.9723
0.1412	10.0	1130	0.1462	0.9103	0.9413	0.9255	0.9754
0.1364	11.0	1243	0.1321	0.9217	0.9432	0.9323	0.9783
0.14	12.0	1356	0.1194	0.9306	0.9482	0.9393	0.9798
0.1189	13.0	1469	0.1125	0.9354	0.9508	0.9430	0.9810
0.092	14.0	1582	0.1101	0.9326	0.9527	0.9425	0.9813
0.0989	15.0	1695	0.1084	0.9362	0.9539	0.9450	0.9823
0.0997	16.0	1808	0.1036	0.9375	0.9571	0.9472	0.9832
0.0927	17.0	1921	0.1031	0.9370	0.9571	0.9469	0.9829
0.1028	18.0	2034	0.1023	0.9405	0.9577	0.9490	0.9831
0.0999	19.0	2147	0.1023	0.9399	0.9577	0.9487	0.9831
0.0921	19.8294	2240	0.1022	0.9399	0.9577	0.9487	0.9831

Framework versions

• Transformers 4.51.3
• Pytorch 2.6.0+cu124
• Datasets 3.6.0
• Tokenizers 0.21.1