flan-t5-neutralisation

This model is a fine-tuned version of google/flan-t5-small on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 0.0412
• Bleu: 54.4994
• Gen Len: 18.5938

Model description

This model is a fine-tuned version of T5 for transforming Spanish sentences containing gendered language into gender-neutral Spanish.

The model was trained on the Spanish Gender Neutralization dataset, which contains aligned sentence pairs:

• a gendered Spanish sentence

• its corresponding gender-neutral version.

The goal of the model is to automatically generate inclusive alternatives that avoid gender-specific forms.

Intended uses & limitations

The model can be used for:

• experimentation with inclusive language generation

• research on bias mitigation in language technologies

• educational purposes in natural language processing and machine translation

It is particularly suitable for tasks such as:

• rewriting Spanish sentences in a gender-neutral form

• studying automatic gender neutralization in Spanish

• evaluating inclusive language strategies in NLP systems

Training and evaluation data

The model was fine-tuned on the dataset:

hackathon-pln-es/neutral-es

Dataset characteristics:

• parallel dataset

• Spanish → gender-neutral Spanish

• sentence-level transformations

Each example contains two fields:

• gendered: original sentence

• neutral: neutralized version

Example pair:

gendered: Los alumnos deben entregar el trabajo mañana. neutral: El alumnado debe entregar el trabajo mañana.

Training procedure

The model was fine-tuned using the HuggingFace Transformers Trainer API.

Hyperparameters

model checkpoint: flan-t5-small

learning rate: 5.6e-5

weight decay: 0.01

batch size: 8

epochs: 6

maximum sequence length: 128

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 5.6e-05
• train_batch_size: 8
• eval_batch_size: 8
• seed: 42
• optimizer: Use OptimizerNames.ADAMW_TORCH with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: linear
• num_epochs: 6

Training results

Training Loss	Epoch	Step	Validation Loss	Bleu	Gen Len
No log	1.0	440	0.0773	52.243	18.5417
0.2124	2.0	880	0.0626	53.6111	18.5104
0.1009	3.0	1320	0.0501	54.3864	18.5625
0.0742	4.0	1760	0.0424	54.4994	18.5729
0.0655	5.0	2200	0.0415	54.4994	18.5938
0.0593	6.0	2640	0.0412	54.4994	18.5938

Framework versions

• Transformers 4.51.2
• Pytorch 2.10.0+cu128
• Datasets 4.0.0
• Tokenizers 0.21.4