region:us
---
{}
---
{
# Neural Machine Translation: English to Divehi using LSTM and Seq2Seq Models

## Introduction
This project focuses on building Neural Machine Translation (NMT) models to translate text from English to Divehi. The models implemented include an LSTM-based model and a Seq2Seq model without attention. The primary goal is to evaluate and compare their performances on the given dataset.

## Problem Statement
Divehi is a low-resource language, and there are limited datasets available for natural language processing tasks like translation. This project aims to develop translation models to improve the quality of translation between English and Divehi using deep learning approaches.

## Approach
Two models were implemented for this project:
1. **LSTM-based Model**: Utilizes an encoder-decoder architecture using LSTMs.
2. **Seq2Seq Model (without Attention)**: A simpler sequence-to-sequence model with an encoder-decoder structure but without attention mechanisms.

## Dataset
The dataset used consists of parallel English and Divehi text pairs. The data was preprocessed using a GPT-2 tokenizer, followed by splitting into training and validation sets with an 80-20 ratio.

## Model Architectures

### LSTM Model
The LSTM model uses an encoder-decoder architecture where:
- The encoder processes the input English sentence and encodes it into hidden states.
- The decoder generates the Divehi translation token-by-token using the hidden states.

### Seq2Seq Model (without Attention)
The Seq2Seq model uses a simple encoder-decoder structure:
- The encoder reads the input English sentence and produces a context vector.
- The decoder utilizes the context vector to predict the output sequence in Divehi.

## Training Setup
- Optimizer: AdamW
- Loss Function: Cross-Entropy Loss
- Learning Rate: 0.0001 with early stopping
- Epochs: 30 (early stopping applied if validation loss plateaued)
- Batch Size: 32

## Results

### **LSTM Model**
- **Average Sentence-level BLEU Score**: `0.9823`
- **Average Sentence-level CHRF Score**: `96.0884`
- The LSTM model demonstrated exceptional performance with high BLEU and CHRF scores, indicating strong translation accuracy.

#### **Training Loss Graph**



### **Seq2Seq Model (without Attention)**
- **Average BLEU Score**: `0.0000`
- **Average CHRF Score**: `0.1318`
- The Seq2Seq model struggled to generate meaningful translations, resulting in very low BLEU and CHRF scores.

#### **Training Loss Graph**



## Analysis
From the results, it is evident that the LSTM model significantly outperformed the Seq2Seq model without attention. The Seq2Seq model's poor performance can be attributed to the lack of an attention mechanism, which is crucial for long sentences and complex translations. Additionally, Divehi being a low-resource language made it challenging for the Seq2Seq model to learn effectively.

### Why did the Seq2Seq model perform poorly?
- **Lack of Attention**: The Seq2Seq model without attention struggles to focus on relevant parts of the input, especially for longer sentences.
- **Limited Data**: The dataset might not have been sufficient for the model to generalize well, especially without attention.

## Future Work
To improve the performance of the Seq2Seq model, future work could include:
- Adding an attention mechanism to the Seq2Seq model to enhance translation accuracy.
- Increasing the dataset size to provide more examples for training.
- Fine-tuning the models with hyperparameter optimization and using more advanced architectures like Transformers.

## How to Run

### Prerequisites
- Python 3.8+
- Required packages listed in `requirements.txt`

### Steps to Reproduce
1. Clone this repository:
    ```bash
    git clone https://github.com/yourusername/english-divehi-translation.git
    cd english-divehi-translation
    ```

2. Install the dependencies:
    ```bash
    pip install -r requirements.txt
    ```

3. Download and place the datasets (`Divehi.json` and `alpaca_data_cleaned.json`) in the `data/` directory.

4. Run the training scripts:
    - **LSTM Model**:
      ```bash
      python train_lstm.py
      ```
    - **Seq2Seq Model**:
      ```bash
      python train_seq2seq.py
      ```

5. Evaluate the models:
    ```bash
    python evaluate.py
    ```

## Conclusion
The project demonstrated the effectiveness of LSTM-based models for low-resource language translation. However, a simple Seq2Seq model without attention was not sufficient to produce meaningful translations. Future work will explore using attention mechanisms and transformers to improve translation quality.

## Acknowledgements
Special thanks to my professors, mentors, and peers who provided valuable feedback throughout this project.
}
