FinMTEB: Finance Massive Text Embedding Benchmark
Embedding models play a crucial role in representing and retrieving
information across various NLP applications. Recent advances in large language
models (LLMs) have further enhanced the performance of embedding models. While
these models are often benchmarked on general-purpose datasets, real-world
applications demand domain-specific evaluation. In this work, we introduce the
Finance Massive Text Embedding Benchmark (FinMTEB), a specialized counterpart
to MTEB designed for the financial domain. FinMTEB comprises 64 financial
domain-specific embedding datasets across 7 tasks that cover diverse textual
types in both Chinese and English, such as financial news articles, corporate
annual reports, ESG reports, regulatory filings, and earnings call transcripts.
We also develop a finance-adapted model, FinPersona-E5, using a persona-based
data synthetic method to cover diverse financial embedding tasks for training.
Through extensive evaluation of 15 embedding models, including FinPersona-E5,
we show three key findings: (1) performance on general-purpose benchmarks shows
limited correlation with financial domain tasks; (2) domain-adapted models
consistently outperform their general-purpose counterparts; and (3)
surprisingly, a simple Bag-of-Words (BoW) approach outperforms sophisticated
dense embeddings in financial Semantic Textual Similarity (STS) tasks,
underscoring current limitations in dense embedding techniques. Our work
establishes a robust evaluation framework for financial NLP applications and
provides crucial insights for developing domain-specific embedding models.
67b3ee6c1e80a69e79c3155a
Yixuan Tang
claimed_verified
2025-02-19T09:04:50.969Z
647d834618274bce03013cc2
/avatars/a95c7df96dc4fb6a96193f6dd5068227.svg
yixuan
user
yixuantt
67b3ee6c1e80a69e79c3155b
Yi Yang
FinMTEB: Finance Massive Text Embedding Benchmark
Embedding models play a crucial role in representing and retrieving
information across various NLP applications. Recent advances in large language
models (LLMs) have further enhanced the performance of embedding models. While
these models are often benchmarked on general-purpose datasets, real-world
applications demand domain-specific evaluation. In this work, we introduce the
Finance Massive Text Embedding Benchmark (FinMTEB), a specialized counterpart
to MTEB designed for the financial domain. FinMTEB comprises 64 financial
domain-specific embedding datasets across 7 tasks that cover diverse textual
types in both Chinese and English, such as financial news articles, corporate
annual reports, ESG reports, regulatory filings, and earnings call transcripts.
We also develop a finance-adapted model, FinPersona-E5, using a persona-based
data synthetic method to cover diverse financial embedding tasks for training.
Through extensive evaluation of 15 embedding models, including FinPersona-E5,
we show three key findings: (1) performance on general-purpose benchmarks shows
limited correlation with financial domain tasks; (2) domain-adapted models
consistently outperform their general-purpose counterparts; and (3)
surprisingly, a simple Bag-of-Words (BoW) approach outperforms sophisticated
dense embeddings in financial Semantic Textual Similarity (STS) tasks,
underscoring current limitations in dense embedding techniques. Our work
establishes a robust evaluation framework for financial NLP applications and
provides crucial insights for developing domain-specific embedding models.
The Finance Massive Text Embedding Benchmark (FinMTEB) evaluates embedding models in the financial domain, showing that domain-adapted models outperform general-purpose ones and that simple BoW methods can excel in financial STS tasks.
embedding models large language models (LLMs) domain-specific evaluation Finance Massive Text Embedding Benchmark (FinMTEB) MTEB financial domain-specific embedding datasets persona-based data synthetic method FinPersona-E5 Bag-of-Words (BoW) dense embeddings Semantic Textual Similarity (STS)