safetensors region:us
---
{}
---
# **Transformer-Based Fine-Grained Text Emotion Recognition (FGTER)**

### **Authors**

Kshitij Goyal, Ishwak Sharda, Divnoor Nagra  
Emails:  
[kshitij.goyal@student.ufv.ca](mailto:kshitij.goyal@student.ufv.ca), [ishwak.sharda@student.ufv.ca](mailto:ishwak.sharda@student.ufv.ca), [divnoor.nagra@student.ufv.ca](mailto:divnoor.nagra@student.ufv.ca).

## **Project Overview**

Human-computer interaction (HCI) systems, particularly social companion robots, require robust mechanisms to detect and adapt to nuanced human emotional states. Existing emotion recognition systems are often limited by their inability to capture fine-grained distinctions in emotions and intensity, crucial for empathetic interactions. To address this, we introduce a Fine-Grained Text Emotion Recognition (FGTER) framework, which classifies five emotions—anger, sadness, happiness, disgust, and fear—across three intensity levels: low, medium, and high.

Our FGTER system integrates a transformer-based emotion classifier with a feedback loop mechanism that iteratively refines predictions using context extracted from multimodal observations. To enhance computational efficiency for real-time deployment on Miro-e, a social companion robot, the system utilizes LLaMA 3 (8B parameters) for lightweight sentence generation during the feedback process. This ensures that emotion predictions remain accurate and context-aware even in resource-constrained edge-computing environments.

By combining fine-grained emotion recognition with a dynamic feedback loop, this work advances the development of emotionally intelligent robots capable of nuanced and empathetic interactions. The FGTER framework lays the groundwork for deploying real-time emotion recognition systems in practical applications, ensuring adaptability and precision in human-robot interactions.

## **Key Contributions**

1. **Fine-Grained Emotion Recognition**:  
   Developed a transformer-based model (fine-tuned RoBERTa) capable of classifying five key emotions—**anger, fear, disgust, happiness, and sadness**—each across three distinct intensity levels (**low, medium, high**). This model bridges the gap between coarse-grained and fine-grained emotion detection, enabling nuanced emotional insights for advanced HCI applications.

2. **Dynamic Feedback Loop**:  
   Introduced an iterative feedback mechanism that refines emotion predictions in real-time by incorporating additional contextual observations. This approach ensures higher prediction accuracy and adaptability in dynamic human-computer interaction scenarios.

3. **Augmented Fine-Grained Dataset**:  
   Enhanced datasets, including **GoEmotions**, **MELD**, and **EMOTION**, through data augmentation techniques such as **paraphrasing, word-swapping, and word-deletion**. These methods improved class balance and enabled better generalization across nuanced emotion-intensity combinations.

4. **Contextual Keyword Extraction**:  
   Simulated multimodal keyword extraction from diverse sources such as **facial expressions, speech tone, gestures**, and **IoT wearables**. These extracted keywords were integrated into the feedback loop for generating contextually enriched prompts, ensuring context-aware emotion recognition.

5. **Optimized for Edge-Computing Applications**:  
   Designed the system to support real-time deployment in **social companion robots (SCRs)**, such as **Miro-e**. By leveraging **LLaMA 3 (8B parameters)** for lightweight and efficient sentence generation, the framework is optimized for resource-constrained edge-computing environments while maintaining robust performance.

6. **Real-Time Emotional Adaptability**:  
   Enabled the FGTER framework to detect emotions dynamically and adapt responses in real time. This capability is expected to enhance the **empathy** and **engagement** of SCRs, making them better suited for personalized and meaningful human-robot interactions.

---

## **Development Environment**

- **Operating System**: Windows 10, Linux, or macOS
- **Python Version**: 3.9 or higher
- **Virtual Environment**: Python `venv` or `conda`

## **Project Folder Hierarchy**

```plaintext
project-root/
│
├── creating-datasets/                     # Scripts for creating fine-grained datasets
│   ├── creating-fine-grained-dataset-emotions.py
│   ├── creating-fine-grained-dataset-meld.py
│   ├── creating-fine-grained-dataset-goemotions.py
│   ├── meld/                              # MELD dataset and splits
│
├── fine_grained_datasets/                 # Final processed datasets
│   ├── emotion_fine_grained.csv
│   ├── meld_fine_grained.csv
│   ├── goemotions_fine_grained.csv
│
├── fine_grained_emotion_model/            # Folder to store the fine-tuned model
│
├── venv/                                  # Virtual environment
│
├── augmentation.py                        # Preprocessing and augmentation
├── finetuning-fgsam.py                    # Fine-tuning script
├── testing-model.py                       # Model evaluation script
├── llama-8b.py                            # Feedback loop with LLaMA 8B model
├── final_combined_augmented_data.csv      # Augmented dataset
├── graphs.py                              # Scripts for data visualization
├── requirements.txt                       # Python dependencies
```

## **Installation Steps**

### **1. Clone the Repository**

```bash
git clone https://github.com/yourusername/FGTER.git
cd FGTER
```

### **2. Set Up Virtual Environment**

- **Using `venv`:**
  ```bash
  python3 -m venv venv
  source venv/bin/activate   # Linux/macOS
  venv\Scripts\activate      # Windows
  ```
- **Using `conda`:**
  ```bash
  conda create --name fgter python=3.9
  conda activate fgter
  ```

### **3. Install Dependencies**

```bash
pip install -r requirements.txt
```

### **4. Install LLaMA 3 (8B Parameters)**

To enable the feedback loop:

- **Install Ollama CLI**:
  ```bash
  brew install ollama  # For macOS
  ```
- **Pull the LLaMA 3 Model**:
  ```bash
  ollama pull llama3
  ```
- **Verify Installation**:
  ```bash
  ollama chat llama3
  ```

## **Execution Steps**

### **Preprocess and Augment Data**

```bash
python augmentation.py
```

### **Fine-Tune the FGTER Model**

```bash
python finetuning-fgsam.py
```

### **Evaluate the FGTER Model**

```bash
python testing-model.py
```

### **Feedback Loop**

```bash
python llama-8b.py
```

## **Feedback Loop Evaluation**

The feedback loop was evaluated using simulated keywords derived from **10 samples per class** across 15 fine-grained emotions (anger, sadness, fear, happiness, disgust) × (low, medium, high). Observations included:

1. **Initial Observations**: Baseline keywords from text descriptions.
2. **Additional Observations**: Contextual cues to refine predictions.
3. **More Observations**: Final refinements for confidence improvement.

**Example Process**:

- **Keywords**: "eyes wide open, trembling hands, shallow breathing"
- **Generated Sentence**: "The individual has wide-open eyes and trembling hands while breathing shallowly."
- **Emotion Prediction**: `fear_high` with 95% confidence.

## **Results**

| Metric                   | Before Feedback Loop | After Feedback Loop | Improvement (%) |
| ------------------------ | -------------------- | ------------------- | --------------- |
| **Accuracy**             | 38.67                | 41.33               | 2.67            |
| **Precision (Weighted)** | 42.59                | 46.97               | 4.38            |
| **Recall (Weighted)**    | 38.67                | 41.33               | 2.67            |

Accuracy improved by 2.67%. The feedback loop successfully refined the model's predictions.
Precision improved by 4.38%. The model's predictions are more specific after feedback.
Recall improved by 2.67%. The model is better at identifying relevant instances after feedback.

### **Log of Progress**

1. **Dataset Compilation**  
   **Assigned To**: Kshitij Goyal, Ishwak Sharda, Divnoor Singh

   - Curated datasets (**GoEmotions**, **Emotions**, **MELD**) to focus on five primary emotions with three intensity levels (low, medium, high).
   - Used GPT for coarse-level predictions and manual validation to assign fine-grained emotion labels.  
     **Completion Date**: 27/09/2024

2. **Preprocessing & Augmentation**  
   **Assigned To**: Kshitij Goyal, Ishwak Sharda

   - Balanced the dataset using **paraphrasing**, **word swapping**, and **word deletion** augmentation techniques.  
     **Completion Date**: 15/10/2024

3. **Model Fine-Tuning**  
   **Assigned To**: Kshitij Goyal

   - Fine-tuned a RoBERTa model for nuanced emotion intensity detection.  
     **Completion Date**: 24/10/2024

4. **System Integration**  
   **Assigned To**: Kshitij Goyal, Divnoor Singh

   - Developed a keyword dataset simulating multimodal inputs (speech, gestures, text).  
     **Completion Date**: 27/10/2024

5. **Sentence Generation**  
   **Assigned To**: Ishwak Sharda

   - Optimized **LLaMA 3 (8B)** for sentence generation on resource-constrained platforms.  
     **Completion Date**: 20/10/2024

6. **Model Integration**  
   **Assigned To**: Ishwak Sharda, Kshitij Goyal

   - Integrated sentence generation with emotion recognition, introducing a confidence-based feedback mechanism.  
     **Completion Date**: 30/10/2024

7. **Feedback Loop Development**  
   **Assigned To**: Kshitij Goyal

   - Implemented an iterative feedback mechanism to refine low-confidence predictions.  
     **Completion Date**: 07/11/2024

8. **System Testing**  
   **Assigned To**: Divnoor Singh

   - Validated system performance in simulated real-world scenarios.  
     **Completion Date**: 15/01/2024

9. **Final Optimization**  
   **Assigned To**: Kshitij Goyal, Ishwak Sharda

   - Enhanced real-time responsiveness for edge-computing deployment.  
     **Completion Date**: 19/11/2024

10. **Documentation & Reporting**  
    **Assigned To**: Kshitij Goyal, Ishwak Sharda, Divnoor Singh
    - Preparing comprehensive documentation and final project report.  
      **Completion Date**: In Progress

## **Future Work**

1. **Enhance Model Accuracy**: Improve the accuracy of the FGTER model by acquiring or constructing higher-quality datasets with better annotations, ensuring a richer and more diverse representation of fine-grained emotions and their intensity levels.

2. **Deploy on Miro-E**: Finalize the integration and deployment of the framework on **Miro-E**, our social companion robot, enabling real-time fine-grained emotion recognition in dynamic human-robot interaction scenarios.

3. **Develop a Multimodal Emotion Recognition System**: Combine the **FGTER**, **FGFER** (Fine-Grained Facial Emotion Recognition), and **FGSER** (Fine-Grained Speech Emotion Recognition) models to create a unified **Fine-Grained Multimodal Emotion Recognition Framework**, aiming to enhance emotion detection accuracy by leveraging complementary data from text, speech, and facial expressions.

## **Tools and Licenses**

1. **Transformers Library**: _(Apache 2.0 License)_
2. **PEGASUS**: _(Apache 2.0 License)_
3. **LLaMA 3**: _(Meta-specific license)_
4. **Scikit-learn**: _(BSD License)_
5. **OpenPose/OpenFace**: _(MIT License)_
