Qwen3-0.6B-IMDB-LoRA

A fine-tuned version of Qwen3-0.6B for sentiment analysis on movie reviews using LoRA (Low-Rank Adaptation).

Model Details

Model Description

This model is a fine-tuned version of Qwen3-0.6B specifically optimized for binary sentiment classification (positive/negative) on movie reviews. It uses LoRA adaptation to efficiently fine-tune the model while maintaining a small memory footprint.

• Developed by: [Yiwen]
• Model type: Causal Language Model with LoRA adaptation for sequence classification
• Language(s) (NLP): English
• License: Same as base model (Apache 2.0)
• Finetuned from model: Qwen/Qwen3-0.6B

Model Sources

• Repository: https://huggingface.co/yiwenX/Qwen3-0.6B-imdb
• Base Model: https://huggingface.co/Qwen/Qwen3-0.6B

Uses

Direct Use

This model can be used directly for sentiment analysis of English movie reviews or similar text. It classifies text into two categories:

• Label 0: Negative sentiment
• Label 1: Positive sentiment

Downstream Use

The model can be further fine-tuned for:

• Other binary text classification tasks
• Domain-specific sentiment analysis (product reviews, social media, etc.)
• Multi-class sentiment analysis with additional training

Out-of-Scope Use

• The model is trained on English movie reviews and may not perform well on:
  • Non-English text
  • Highly technical or domain-specific content
  • Text significantly different from movie reviews
• Should not be used for critical decision-making without human oversight

Bias, Risks, and Limitations

• Dataset Bias: IMDB dataset may contain biases related to movie genres, time periods, and reviewer demographics
• Language Limitation: English-only model
• Domain Specificity: Optimized for movie reviews, performance may degrade on other text types
• Binary Classification: Only distinguishes positive/negative, cannot detect neutral sentiment

How to Get Started with the Model

from transformers import AutoModelForSequenceClassification, AutoTokenizer
import torch

# 直接从HuggingFace加载你的模型
model_name = "yiwenX/Qwen3-0.6B-imdb"
model = AutoModelForSequenceClassification.from_pretrained(
    model_name,
    torch_dtype=torch.float16,
    device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained(model_name)

# 如果tokenizer没有pad_token，设置一下
if tokenizer.pad_token is None:
    tokenizer.pad_token = tokenizer.eos_token

# 推理示例
def predict_sentiment(text):
    inputs = tokenizer(text, return_tensors="pt", truncation=True, max_length=512, padding=True)
    
    # 移动到正确的设备
    inputs = {k: v.to(model.device) for k, v in inputs.items()}
    
    with torch.no_grad():
        outputs = model(**inputs)
        prediction = torch.argmax(outputs.logits, dim=-1)
    
    return "Positive" if prediction.item() == 1 else "Negative"

# 测试示例
test_texts = [
    "This movie was absolutely fantastic! Great acting and storyline.",
    "Terrible movie, waste of time. Bad acting and boring plot.",
    "One of the best films I've ever seen. Highly recommend!",
    "I fell asleep halfway through. Very disappointing."
]

for text in test_texts:
    sentiment = predict_sentiment(text)
    print(f"Text: {text[:50]}...")
    print(f"Sentiment: {sentiment}\n")

Training Details

Training Data

• Dataset: IMDB Movie Review Dataset
• Training samples: 25,000
• Validation samples: 25,000
• Data source: https://huggingface.co/datasets/imdb

Training Procedure

Preprocessing

• Tokenization using Qwen tokenizer
• Maximum sequence length: 512 tokens
• Padding to max length
• Truncation of longer sequences

Training Hyperparameters

• Training regime: fp16 mixed precision
• LoRA rank (r): 16
• LoRA alpha: 32
• LoRA dropout: 0.1
• Target modules: ["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj"]
• Batch size: 8 per device
• Gradient accumulation steps: 4
• Learning rate: 2e-4
• Number of epochs: 3
• Warmup steps: 100
• Weight decay: 0.01
• Optimizer: AdamW

Speeds, Sizes, Times

• Training time: ~50 minutes on single GPU
• Model size: ~10MB (LoRA weights only)
• Base model size: ~1.2GB
• Training speed: ~0.72 iterations/second

Evaluation

Testing Data, Factors & Metrics

Testing Data

• IMDB test set (25,000 samples)
• Balanced dataset (50% positive, 50% negative)

Metrics

• Accuracy: Primary metric for balanced binary classification
• F1 Score: Harmonic mean of precision and recall
• Precision/Recall: For detailed performance analysis

Results

Metric	Value
Test Accuracy	~92-94% (expected)
F1 Score	~0.92-0.94
Training Loss (final)	~0.1-0.2

Note: Exact values depend on random seed and training conditions

Environmental Impact

• Hardware Type: NVIDIA GPU (13GB VRAM)
• Hours used: ~1 hour
• Cloud Provider: [Local]
• Carbon Efficiency: Minimal due to LoRA efficiency

Technical Specifications

Model Architecture and Objective

• Base Architecture: Qwen3 (Transformer-based)
• Parameters: 606M total, 10M trainable (1.66%)
• Objective: Cross-entropy loss for binary classification
• Adaptation Method: LoRA (Low-Rank Adaptation)

Compute Infrastructure

Hardware

• Single GPU with 13GB+ VRAM
• CUDA-capable device

Software

• PyTorch 2.0+
• Transformers 4.30+
• PEFT 0.15.2
• CUDA 11.7+

Citation

@misc{qwen3-imdb-lora,
  author = {yiwenX},
  title = {Qwen3-0.6B-IMDB: Fine-tuned Sentiment Analysis Model},
  year = {2024},
  publisher = {HuggingFace},
  url = {https://huggingface.co/yiwenX/Qwen3-0.6B-imdb}
}

Model Card Contact

[email protected]

Framework versions

• PEFT 0.15.2
• Transformers 4.30+
• PyTorch 2.0+