laxsvips/minilm-finetuned-emotion

This model is a fine-tuned version of microsoft/MiniLM-L12-H384-uncased on the Hugging Face emotion (https://huggingface.co/datasets/emotion) dataset. It achieves the following results on the evaluation set:

Train Loss: 0.1168
Train Accuracy: 0.9446
Validation Loss: 0.1709
Validation Accuracy: 0.9350
Epoch: 4

Model description

MiniLM: Small and Fast Pre-trained Models for Language Understanding and Generation

MiniLM is a distilled model from the paper "MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers".

Intended uses & limitations

This model has been created as a learning guide on:

How to implement a text classification model using Hugging Face Transformers in TensorFlow
How to handle imbalanced class distribution

How to use the model

from transformers import pipeline

model_cpt = "laxsvips/minilm-finetuned-emotion"
pipe = pipeline("text-classification", model=model_cpt)
predicted_scores = pipe("I am so glad you could help me")
print(predicted_scores)

The results:

[[{'label': 'sadness', 'score': 0.003758953418582678},
  {'label': 'joy', 'score': 0.9874302744865417},
  {'label': 'love', 'score': 0.00610917154699564},
  {'label': 'anger', 'score': 9.696640336187556e-05},
  {'label': 'fear', 'score': 0.0006420552381314337},
  {'label': 'surprise', 'score': 0.00196251692250371}]]

Training and evaluation data

Emotion

Emotion is a dataset of English Twitter messages with six basic emotions: anger, fear, joy, love, sadness, and surprise.

Training procedure

Refer to the Colab notebook

Training hyperparameters

The following hyperparameters were used during training:

optimizer: 'Adam',
learning_rate': 5e-05,
batch_size : 64
num_epochs - 5

Training results

Train Loss	Train Accuracy	Validation Loss	Validation Accuracy	Epoch
0.9485	0.5543	0.8404	0.6870	0
0.4192	0.8347	0.3450	0.9040	1
0.2132	0.9178	0.2288	0.9240	2
0.1465	0.9364	0.1838	0.9295	3
0.1168	0.9446	0.1709	0.9350	4

Evaluation Metrics

{'accuracy': 0.935,
 'precision': 0.937365614416424,
 'recall': 0.935,
 'f1_score': 0.9355424419858925}

Framework versions

Transformers 4.26.1
TensorFlow 2.11.0
Datasets 2.9.0
Tokenizers 0.13.2

laxsvips
/

minilm-finetuned-emotion