transformers safetensors mt5 text2text-generation nepali roman english translation transliteration ne en dataset:syubraj/roman2nepali-transliteration base_model:google/mt5-small base_model:finetune:google/mt5-small doi:10.57967/hf/3247 license:apache-2.0 autotrain_compatible endpoints_compatible region:us
---
base_model:
- google/mt5-small
datasets:
- syubraj/roman2nepali-transliteration
language:
- ne
- en
library_name: transformers
license: apache-2.0
pipeline_tag: translation
tags:
- nepali
- roman english
- translation
- transliteration
new_version: syubraj/romaneng2nep_v2
---

# Model Card for Model ID

Due to compute issues, The model has been trained on multiple iterations:

1. Model Trained for 8500 steps on [0 : 5%] of the dataset.
2. Model continued from 8500 upto 16500 steps on [5% : 20%] of the dataset
3. Model continued from 16500 upto 22000 steps on [20% : 40%] of the dataset





### Model Description

<!-- Provide a longer summary of what this model is. -->

This is the model card of a 🤗 transformers model that has been pushed on the Hub. This model card has been automatically generated.

- **Model type:** (Translation)
- **Language(s) (NLP):**  Nepali, English
- **License:** [Apache license 2.0]
- **Finetuned from model :** [google/mt5-small]


## How to Get Started with the Model

Use the code below to get started with the model.

```Python

from transformers import AutoTokenizer, MT5ForConditionalGeneration

checkpoint = "syubraj/RomanEng2Nep-v2"
tokenizer = AutoTokenizer.from_pretrained(checkpoint)
model = MT5ForConditionalGeneration.from_pretrained(checkpoint)

# Set max sequence length
max_seq_len = 20

def translate(text):
    # Tokenize the input text with a max length of 20
    inputs = tokenizer(text, return_tensors="pt", truncation=True, max_length=max_seq_len)

    # Generate translation
    translated = model.generate(**inputs)

    # Decode the translated tokens back to text
    translated_text = tokenizer.decode(translated[0], skip_special_tokens=True)
    return translated_text

# Example usage
source_text = "muskuraudai"  # Example Romanized Nepali text
translated_text = translate(source_text)
print(f"Translated Text: {translated_text}")
```



### Training Data

[syubraj/roman2nepali-transliteration](https://huggingface.co/datasets/syubraj/roman2nepali-transliteration)


#### Training Hyperparameters

- **Training regime:**
```Python
training_args = Seq2SeqTrainingArguments(
    output_dir="/content/drive/MyDrive/romaneng2nep_v2/",
    eval_strategy="steps",
    learning_rate=2e-5,
    per_device_train_batch_size=16,
    per_device_eval_batch_size=8,
    weight_decay=0.01,
    save_total_limit=3,
    num_train_epochs=2,
    predict_with_generate=True,
)
```

## Training and Validation Metrics

Step    | Training Loss | Validation Loss | Gen Len
--------|---------------|-----------------|---------
500     | 21.636200     | 9.776628        | 2.001900
1000    | 10.103400     | 6.105016        | 2.077900
1500    | 6.830800      | 5.081259        | 3.811600
2000    | 6.003100      | 4.702793        | 4.237300
2500    | 5.690200      | 4.469123        | 4.700000
3000    | 5.443100      | 4.274406        | 4.808300
3500    | 5.265300      | 4.121417        | 4.749400
4000    | 5.128500      | 3.989708        | 4.782300
4500    | 5.007200      | 3.885391        | 4.805100
5000    | 4.909600      | 3.787640        | 4.874800
5500    | 4.836000      | 3.715750        | 4.855500
6000    | 4.733000      | 3.640963        | 4.962000
6500    | 4.673500      | 3.587330        | 5.011600
7000    | 4.623800      | 3.531883        | 5.068300
7500    | 4.567400      | 3.481622        | 5.108500
8000    | 4.523200      | 3.445404        | 5.092700
8500    | 4.464000      | 3.413630        | 5.132700
9000    | 4.423100      | 3.326201        | 5.211700
9500    | 4.315700      | 3.238422        | 5.200600
10000   | 4.218200      | 3.143774        | 5.288100
10500   | 4.133600      | 3.080613        | 5.202300
11000   | 4.087700      | 3.011713        | 5.271800
11500   | 4.004300      | 2.957386        | 5.178700
12000   | 3.956700      | 2.898953        | 5.209600
12500   | 3.922800      | 2.850440        | 5.210100
13000   | 3.853400      | 2.796974        | 5.171700
13500   | 3.807900      | 2.745325        | 5.281200
14000   | 3.755700      | 2.708517        | 5.223000
14500   | 3.729300      | 2.678200        | 5.210700
15000   | 3.673600      | 2.637842        | 5.230200
15500   | 3.625400      | 2.607649        | 5.264100
16000   | 3.601100      | 2.592188        | 5.129800
16500   | 3.608200      | 2.556329        | 5.215800
17000   | 3.557900      | 2.536781        | 5.162900
17500   | 3.533500      | 2.504695        | 5.206000
18000   | 3.500000      | 2.477887        | 5.211600
18500   | 3.463600      | 2.456758        | 5.201000
19000   | 3.457100      | 2.433362        | 5.210000
19500   | 3.435400      | 2.411479        | 5.197600
20000   | 3.413300      | 2.392534        | 5.221100
20500   | 3.366100      | 2.378421        | 5.165200
21000   | 3.363500      | 2.357117        | 5.187300
21500   | 3.346500      | 2.343485        | 5.193600
22000   | 3.328300      | 2.331021        | 5.183300