title: FinMTEB: Finance Massive Text Embedding Benchmark
summary: Embedding models play a crucial role in representing and retrieving
  information across various NLP applications. Recent advances in large language
  models (LLMs) have further enhanced the performance of embedding models. While
  these models are often benchmarked on general-purpose datasets, real-world
  applications demand domain-specific evaluation. In this work, we introduce the
  Finance Massive Text Embedding Benchmark (FinMTEB), a specialized counterpart
  to MTEB designed for the financial domain. FinMTEB comprises 64 financial
  domain-specific embedding datasets across 7 tasks that cover diverse textual
  types in both Chinese and English, such as financial news articles, corporate
  annual reports, ESG reports, regulatory filings, and earnings call transcripts.
  We also develop a finance-adapted model, FinPersona-E5, using a persona-based
  data synthetic method to cover diverse financial embedding tasks for training.
  Through extensive evaluation of 15 embedding models, including FinPersona-E5,
  we show three key findings: (1) performance on general-purpose benchmarks shows
  limited correlation with financial domain tasks; (2) domain-adapted models
  consistently outperform their general-purpose counterparts; and (3)
  surprisingly, a simple Bag-of-Words (BoW) approach outperforms sophisticated
  dense embeddings in financial Semantic Textual Similarity (STS) tasks,
  underscoring current limitations in dense embedding techniques. Our work
  establishes a robust evaluation framework for financial NLP applications and
  provides crucial insights for developing domain-specific embedding models.
paper_authors._id: 67b3ee6c1e80a69e79c3155a
paper_authors.name: Yixuan Tang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-19T09:04:50.969Z
paper_authors._id: 647d834618274bce03013cc2
paper_authors.avatarUrl: /avatars/a95c7df96dc4fb6a96193f6dd5068227.svg
paper_authors.fullname: yixuan
paper_authors.type: user
paper_authors.user: yixuantt
paper_authors._id: 67b3ee6c1e80a69e79c3155b
paper_authors.name: Yi Yang
paper_title: FinMTEB: Finance Massive Text Embedding Benchmark
paper_summary: Embedding models play a crucial role in representing and retrieving
  information across various NLP applications. Recent advances in large language
  models (LLMs) have further enhanced the performance of embedding models. While
  these models are often benchmarked on general-purpose datasets, real-world
  applications demand domain-specific evaluation. In this work, we introduce the
  Finance Massive Text Embedding Benchmark (FinMTEB), a specialized counterpart
  to MTEB designed for the financial domain. FinMTEB comprises 64 financial
  domain-specific embedding datasets across 7 tasks that cover diverse textual
  types in both Chinese and English, such as financial news articles, corporate
  annual reports, ESG reports, regulatory filings, and earnings call transcripts.
  We also develop a finance-adapted model, FinPersona-E5, using a persona-based
  data synthetic method to cover diverse financial embedding tasks for training.
  Through extensive evaluation of 15 embedding models, including FinPersona-E5,
  we show three key findings: (1) performance on general-purpose benchmarks shows
  limited correlation with financial domain tasks; (2) domain-adapted models
  consistently outperform their general-purpose counterparts; and (3)
  surprisingly, a simple Bag-of-Words (BoW) approach outperforms sophisticated
  dense embeddings in financial Semantic Textual Similarity (STS) tasks,
  underscoring current limitations in dense embedding techniques. Our work
  establishes a robust evaluation framework for financial NLP applications and
  provides crucial insights for developing domain-specific embedding models.
paper_ai_keywords: embedding models large language models (LLMs) real-world applications Finance Massive Text Embedding Benchmark (FinMTEB) financial domain-specific embedding datasets financial news articles corporate annual reports ESG reports regulatory filings earnings call transcripts persona-based data synthetic method finance-adapted model FinPersona-E5 financial Semantic Textual Similarity (STS) tasks Bag-of-Words (BoW) approach dense embeddings domain-adapted models general-purpose counterparts