๐บ๐ฆ uk-gender-swapper-aya-101
This model is a PEFT-fine-tuned version of the aya-101 large language model, adapted for Ukrainian gender swapping. It is designed to generate gender-swapped versions of input sentences, making it useful for tasks involving gender bias mitigation, inclusive language generation, or synthetic dataset creation.
Model Type: Aya-101 (12.9B) + PEFT (LoRA)
Language: Ukrainian ๐บ๐ฆ
Task: Gender Swapping (Text-to-Text Generation)
๐ก Use Cases
- Bias analysis and mitigation in NLP models
- Synthetic data generation for gender-balanced corpora
- Exploration of linguistic gender structures in Ukrainian
- Educational and research purposes related to gender and language
๐ง Model Details
- Base Model: CohereForAI/aya-101
- Fine-tuning Method: Parameter-Efficient Fine-Tuning (PEFT) with LoRA
- Training Data: A custom parallel dataset of Ukrainian sentences with gender-swapped equivalents (e.g., male-to-female and female-to-male subject transformations)
- Tokenizer: Same as Aya-101 (LLaMA tokenizer)
๐ง How to Use
You can use this model with the peft
library and transformers
. Hereโs an example using the merged adapter:
from transformers import AutoModelForSeq2SeqLM, AutoTokenizer
from peft import PeftModel, PeftConfig
adapter_repo_id = "linndfors/uk-sent-gender-swapper_aya-101"
config = PeftConfig.from_pretrained(adapter_repo_id)
base_model_id = config.base_model_name_or_path
tokenizer = AutoTokenizer.from_pretrained(base_model_id)
base_model = AutoModelForSeq2SeqLM.from_pretrained(base_model_id)
base_model.resize_token_embeddings(len(tokenizer))
inference_model = PeftModel.from_pretrained(base_model, adapter_repo_id)
prefix = "ะะตัะตััะฐะทัะน ัะต ัะตัะตะฝะฝั, ะทะผัะฝะธะฒัะธ ะณะตะฝะดะตัะฝั ัััะฝะพััั ะฝะฐ ะฟัะพัะธะปะตะถะฝั (ัะพะปะพะฒััะธะน <-> ะถัะฝะพัะธะน): "
def generate_swapped_response(input_text, inference_model, tokenizer):
inputs = tokenizer(prefix + input_text, return_tensors="pt")
with torch.no_grad():
outputs = inference_model.generate(**inputs, max_new_tokens=100, temperature=0.1)
generated = tokenizer.decode(outputs[0], skip_special_tokens=True)
return generated
input = "ะัะฝ ะฟัะฐััั ะฒะธะบะปะฐะดะฐัะตะผ ะฒ ัะฝัะฒะตััะธัะตัั."
gender-swapped_output = generate_swapped_response(input, inference_model, tokenizer)
๐ Example
Input:ะัะฝ ะฟัะฐััั ะฒะธะบะปะฐะดะฐัะตะผ ะฒ ัะฝัะฒะตััะธัะตัั.
Output:ะะพะฝะฐ ะฟัะฐััั ะฒะธะบะปะฐะดะฐัะบะพั ะฒ ัะฝัะฒะตััะธัะตัั.
๐๏ธ Training Details
- Adapter Type: LoRA (rank=32)
- PEFT Library:
peft
- Hardware: Trained on Colab Pro+ (A100)
- Epochs: 15
- Batch Size: 4
- Optimizer: adamw_torch
- Learning Rate: 5e-5
๐ Files and Artifacts
adapter_config.json
โ PEFT configurationadapter_model.bin
โ Fine-tuned LoRA weightsREADME.md
โ Descriptiontokenizer_config.json
,tokenizer.model
, etc. โ Tokenizer files from the base model
โ ๏ธ Limitations & Warnings
- May not generalize well to complex or idiomatic language.
- Designed specifically for binary gender swaps in Ukrainian.
- Not suitable for tasks requiring preservation of named entities in all cases.
๐ Citation
If you use this model in your work, please cite the original Aya-101 paper and consider crediting this repository:
@misc{linndfors2024ukgenderswapper,
title={uk-gender-swapper-aya-101},
author={Olha Nahurna},
year={2024},
howpublished={\url{https://huggingface.co/linndfors/uk-gender-swapper-aya-101}}
}
๐ค Questions or Feedback?
Feel free to open an issue or reach out via the Hugging Face discussion tab.
Model tree for linndfors/uk-gender-swapper-aya-101
Base model
CohereLabs/aya-101