jengyang/trained-llama3-sentences_allagree-financial-sentiment

Trained Llama3 Sentences_Allagree

Model Description

Llama3-8B fine-tuned on financial sentiment (100% agreement threshold). This model was fine-tuned using LoRA (Low-Rank Adaptation) on the Financial PhraseBank dataset with 100% annotator agreement threshold.

Model Details

• Base Model: meta-llama/Meta-Llama-3-8B
• Fine-tuning Method: LoRA (Low-Rank Adaptation)
• Dataset: Financial PhraseBank (sentences with 100% annotator agreement)
• Task: Financial Sentiment Analysis (3-class: positive, negative, neutral)
• Language: English

Performance

Metric	Value
Accuracy	83.0%
F1 Score	82.5%
Precision	83.5%
Recall	82.0%

Training Details

This model was fine-tuned as part of a Final Year Project on Financial Sentiment Analysis and Stock Prediction. The training used:

• Training Framework: Transformers + PEFT
• Quantization: 4-bit quantization using BitsAndBytes
• Hardware: CUDA-enabled GPU
• Hyperparameter Optimization: Extensive Optuna-based tuning

Usage

from transformers import AutoTokenizer, AutoModelForCausalLM
from peft import PeftModel
import torch

# Load base model and tokenizer
base_model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Meta-Llama-3-8B",
    torch_dtype=torch.float16,
    device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained("meta-llama/Meta-Llama-3-8B")

# Load fine-tuned model
model = PeftModel.from_pretrained(base_model, "jengyang/trained-llama3-sentences_allagree-financial-sentiment")

# Prepare input
text = "The company reported strong quarterly earnings, exceeding analyst expectations."
prompt = f"Classify the sentiment of this financial text as positive, negative, or neutral: {text}\n\nSentiment:"

# Tokenize and generate
inputs = tokenizer(prompt, return_tensors="pt")
with torch.no_grad():
    outputs = model.generate(
        **inputs,
        max_new_tokens=10,
        do_sample=False,
        pad_token_id=tokenizer.eos_token_id
    )

response = tokenizer.decode(outputs[0], skip_special_tokens=True)
print(response)

Training Data

The model was trained on the Financial PhraseBank dataset, specifically using sentences where 100% of annotators agreed on the sentiment label. This ensures higher quality and consistency in the training data.

The Financial PhraseBank contains financial news headlines categorized into:

• Positive: Favorable financial news
• Negative: Unfavorable financial news
• Neutral: Factual financial information without clear sentiment

Evaluation

The model was evaluated on a held-out test set from the Financial PhraseBank dataset. The evaluation metrics reflect performance on financial sentiment classification with the 100% agreement threshold.

Note: This Llama3 model was fine-tuned with optimized hyperparameters determined through extensive Optuna studies.

Limitations and Bias

• The model is specifically designed for financial text sentiment analysis
• Performance may vary on non-financial text or different domains
• The model reflects the biases present in the Financial PhraseBank dataset
• Results should be interpreted within the context of financial sentiment analysis
• The model may not capture nuanced sentiment in complex financial scenarios

Intended Use

Intended Use Cases:

• Financial news sentiment analysis
• Investment research and analysis
• Automated financial content classification
• Academic research in financial NLP

Out-of-Scope Use Cases:

• General-purpose sentiment analysis
• Medical or legal text analysis
• Real-time trading decisions without human oversight

Citation

If you use this model, please cite:

@misc{trained_llama3_sentences_allagree,
  title={Trained Llama3 Sentences_Allagree: Fine-tuned Meta-Llama-3-8B for Financial Sentiment Analysis},
  author={Final Year Project},
  year={2024},
  howpublished={\url{https://huggingface.co/jengyang/trained-llama3-sentences_allagree-financial-sentiment}}
}

Model Card Authors

This model card was generated as part of a Final Year Project on Financial Sentiment Analysis and Stock Prediction.