Improving Text Embeddings with Large Language Models
In this paper, we introduce a novel and simple method for obtaining
high-quality text embeddings using only synthetic data and less than 1k
training steps. Unlike existing methods that often depend on multi-stage
intermediate pre-training with billions of weakly-supervised text pairs,
followed by fine-tuning with a few labeled datasets, our method does not
require building complex training pipelines or relying on manually collected
datasets that are often constrained by task diversity and language coverage. We
leverage proprietary LLMs to generate diverse synthetic data for hundreds of
thousands of text embedding tasks across nearly 100 languages. We then
fine-tune open-source decoder-only LLMs on the synthetic data using standard
contrastive loss. Experiments demonstrate that our method achieves strong
performance on highly competitive text embedding benchmarks without using any
labeled data. Furthermore, when fine-tuned with a mixture of synthetic and
labeled data, our model sets new state-of-the-art results on the BEIR and MTEB
benchmarks.
65936c78a33889b27ffac470
Liang Wang
extracted_confirmed
2024-01-02T01:53:26.435Z
62343594c63d91cec1ca4a8d
/avatars/5a1ee74c2dbe349a6ec9843a1599d281.svg
Liang Wang
user
intfloat
65936c78a33889b27ffac471
Nan Yang
claimed_verified
2024-05-13T07:50:05.041Z
6341342a9948f573f36d77da
/avatars/813717cb180b2ce566421b43dadabb43.svg
nyanyanya
user
nyanyanya
65936c78a33889b27ffac472
Xiaolong Huang
admin_assigned
2024-02-09T14:01:41.433Z
64149ba8e682c73521d46ecb
/avatars/c1859aa2d295077e798c8bdc83664a07.svg
Xiaolong Huang
user
DiamondH
65936c78a33889b27ffac473
Linjun Yang
admin_assigned
2024-02-09T14:01:34.657Z
641db563c9ce8fdbce4eaa86
/avatars/ec4012d12a56cba5e27c9eaf58fc7691.svg
Linjun Yang
user
Wadeyang
65936c78a33889b27ffac474
Rangan Majumder
admin_assigned
2024-02-09T14:01:25.226Z
64851c3610e6b874559ef7ae
/avatars/2847eb17a5fa45c93280d4224c33ddf0.svg
Rangan Majumder
user
Rangosaur
65936c78a33889b27ffac475
Furu Wei
admin_assigned
2024-02-09T14:01:16.092Z
6368c512fbfe97c16a40baba
/avatars/1c23bc7c0b6d9225699ce27647623d7a.svg
Furu Wei
user
thegenerality
Improving Text Embeddings with Large Language Models
In this paper, we introduce a novel and simple method for obtaining
high-quality text embeddings using only synthetic data and less than 1k
training steps. Unlike existing methods that often depend on multi-stage
intermediate pre-training with billions of weakly-supervised text pairs,
followed by fine-tuning with a few labeled datasets, our method does not
require building complex training pipelines or relying on manually collected
datasets that are often constrained by task diversity and language coverage. We
leverage proprietary LLMs to generate diverse synthetic data for hundreds of
thousands of text embedding tasks across nearly 100 languages. We then
fine-tune open-source decoder-only LLMs on the synthetic data using standard
contrastive loss. Experiments demonstrate that our method achieves strong
performance on highly competitive text embedding benchmarks without using any
labeled data. Furthermore, when fine-tuned with a mixture of synthetic and
labeled data, our model sets new state-of-the-art results on the BEIR and MTEB
benchmarks.
A method using synthetic data and contrastive loss achieves high-quality text embeddings without labeled data, and sets state-of-the-art results when combined with labeled data.
LLMs synthetic data contrastive loss text embeddings BEIR MTEB