Sentiment Analysis with Bi-LSTM and Luong Attention

Model: sentiment_model.h5
Demo: Try it Live!
Dataset: Sentiment Analysis Dataset (Kaggle)
Author: logasanjeev

Overview

This model performs binary sentiment classification on text inputs, predicting Negative or Positive sentiment using a Bidirectional LSTM (Bi-LSTM) architecture enhanced with Luong Attention.

Embeddings: Pretrained GloVe (300D)
Threshold: Optimized at 0.5173
Accuracy: 86.58%
ROC-AUC: 0.9032
Strength: Effectively handles negations (e.g., "i wouldn't recommend it to anyone" → Negative)

Model Architecture

Embedding Layer: GloVe 300D, max_words=20000, frozen
Bi-LSTM: 2 layers, 128 units, return_sequences=True, dropout=0.3
Luong Attention: Custom attention for context weighting
Dense Layers:
- Dense (64 units, ReLU) + Dropout (0.5)
- Output: Dense (1 unit, sigmoid)
Loss: Focal loss (γ=2.0, α=0.25)
Optimizer: Adam (lr=0.001)
Input Length: max_len=60
Trainable Parameters: ~9 million

Performance

Tested on a set of ~156,935 samples:

Metric	Score
Accuracy	86.58%
Precision	0.7600
Recall	0.6500
F1-Score	0.7000
ROC-AUC	0.9032

Classification Report

Class	Precision	Recall	F1-Score	Support
Negative	0.89	0.93	0.91	114,935
Positive	0.76	0.65	0.70	42,000
Macro Avg	0.83	0.79	0.81	156,935

Confusion Matrix Highlights

True Negatives: ~106,889
True Positives: ~27,300
False Positives: ~8,046
False Negatives: ~14,700

Sample Predictions

Try them at the Live Demo:

Input Text	Prediction	Confidence
"i wouldn't recommend it to anyone"	Negative 😣	0.4977
"the food service is not good at all"	Negative 😣	~0.85
"this place is awesome!"	Positive 😊	~0.92
"i'm so happy with this product!"	Positive 😊	~0.95
"why does everything go wrong today?"	Negative 😣	~0.90

Usage

1. Install Dependencies

pip install tensorflow==2.18.0 nltk==3.9.1 numpy==1.26.4 huggingface_hub==0.25.2 contractions==0.1.73 scikit-learn==1.5.2

2. Download Model and Assets

from huggingface_hub import hf_hub_download

model_path = hf_hub_download(repo_id="logasanjeev/sentiment-analysis-bilstm-luong", filename="sentiment_model.h5")
tokenizer_path = hf_hub_download(repo_id="logasanjeev/sentiment-analysis-bilstm-luong", filename="tokenizer.pkl")
encoder_path = hf_hub_download(repo_id="logasanjeev/sentiment-analysis-bilstm-luong", filename="label_encoder.pkl")
config_path = hf_hub_download(repo_id="logasanjeev/sentiment-analysis-bilstm-luong", filename="config.json")

3. Run Inference

import tensorflow as tf
from tensorflow.keras.models import load_model
from tensorflow.keras.preprocessing.sequence import pad_sequences
import pickle, re, numpy as np
import nltk
from nltk.tokenize import word_tokenize
from nltk.stem import WordNetLemmatizer
import contractions

# Setup
nltk.download('punkt')
nltk.download('wordnet')
nltk.download('omw-1.4')
lemmatizer = WordNetLemmatizer()

# Custom Attention Layer
class LuongAttention(tf.keras.layers.Layer):
    def __init__(self, **kwargs):
        super().__init__(**kwargs)
    def build(self, input_shape):
        self.W = self.add_weight("attention_weight", (input_shape[-1], input_shape[-1]), initializer="glorot_normal", trainable=True)
        self.b = self.add_weight("attention_bias", (input_shape[-1],), initializer="zeros", trainable=True)
    def call(self, inputs):
        e = tf.keras.backend.tanh(tf.keras.backend.dot(inputs, self.W) + self.b)
        alpha = tf.keras.backend.softmax(e, axis=1)
        context = tf.keras.backend.sum(inputs * alpha, axis=1)
        return context

# Load Model
model = load_model(model_path, custom_objects={"LuongAttention": LuongAttention, "focal_loss_fn": lambda y_true, y_pred: y_true})
with open(tokenizer_path, "rb") as f: tokenizer = pickle.load(f)
with open(encoder_path, "rb") as f: label_encoder = pickle.load(f)

# Preprocess
def clean_text(text):
    text = contractions.fix(text).lower()
    text = re.sub(r'http\S+|www\S+|https\S+|@\w+|#\w+|<.*?>+|
|\w*\d\w*|[^\w\s]', '', text)
    tokens = word_tokenize(' '.join(text.split()))
    tokens = [lemmatizer.lemmatize(token, pos='v') for token in tokens]
    return ' '.join(tokens).strip()

# Predict
def predict_sentiment(text):
    cleaned = clean_text(text)
    seq = tokenizer.texts_to_sequences([cleaned])
    padded = pad_sequences(seq, maxlen=60, padding='post', truncating='post')
    prob = model.predict(padded, verbose=0)[0][0]
    threshold = 0.5173
    label = (prob >= threshold).astype(int)
    sentiment = label_encoder.inverse_transform([label])[0]
    confidence = prob if sentiment.lower() == "positive" else 1 - prob
    return sentiment, {"Negative": 1 - prob, "Positive": prob}, confidence

# Example
sentiment, probs, confidence = predict_sentiment("i wouldn't recommend it to anyone")
print(f"Sentiment: {sentiment}, Confidence: {confidence:.4f}, Probabilities: {probs}")

Repository Contents

sentiment_model.h5: Trained Bi-LSTM model (~75MB)
tokenizer.pkl: Keras tokenizer (~13MB)
label_encoder.pkl: Scikit-learn label encoder (~264B)
embedding_matrix.npy: GloVe embeddings (~22MB)
config.json: Model configuration

Dataset

Source: Kaggle Sentiment Dataset
Total Samples: ~1,045,567

Training: ~888,632 (85%)
Test: ~156,935 (15%)

Class Distribution:

Class	Count	Percentage
Negative	~762,064	73%
Positive	~283,503	27%

Preprocessing Steps:

Expand contractions (e.g., "wouldn't" → "would not")
Lowercasing, remove URLs/usernames/HTML/numbers/special characters
Tokenization + Lemmatization
Fixed input length: max_len=60

Live Demo

Explore the model in real-time: Gradio Space

Input text box + sentiment output
Confidence bars for Positive and Negative
Preprocessed cleaned text preview
Fast response (1-2s, CPU)

Limitations

Class Imbalance: Lower recall for Positive class (~0.65)
Short Inputs: Less reliable when input has <3 tokens
Edge Cases: Rare expressions like "not bad" can confuse the model
Confidence Scores: Borderline cases (prob ~0.5) have low certainty

Future Improvements

Retrain with Positive class oversampling
Tune focal loss (γ=3.0) to boost recall
Enhance preprocessing (e.g., negation handling)

logasanjeev
/

sentiment-analysis-bilstm-luong