title: Improving Text Embeddings with Large Language Models
summary: In this paper, we introduce a novel and simple method for obtaining
  high-quality text embeddings using only synthetic data and less than 1k
  training steps. Unlike existing methods that often depend on multi-stage
  intermediate pre-training with billions of weakly-supervised text pairs,
  followed by fine-tuning with a few labeled datasets, our method does not
  require building complex training pipelines or relying on manually collected
  datasets that are often constrained by task diversity and language coverage. We
  leverage proprietary LLMs to generate diverse synthetic data for hundreds of
  thousands of text embedding tasks across nearly 100 languages. We then
  fine-tune open-source decoder-only LLMs on the synthetic data using standard
  contrastive loss. Experiments demonstrate that our method achieves strong
  performance on highly competitive text embedding benchmarks without using any
  labeled data. Furthermore, when fine-tuned with a mixture of synthetic and
  labeled data, our model sets new state-of-the-art results on the BEIR and MTEB
  benchmarks.
paper_authors._id: 65936c78a33889b27ffac470
paper_authors.name: Liang Wang
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2024-01-02T01:53:26.435Z
paper_authors._id: 62343594c63d91cec1ca4a8d
paper_authors.avatarUrl: /avatars/5a1ee74c2dbe349a6ec9843a1599d281.svg
paper_authors.fullname: Liang Wang
paper_authors.type: user
paper_authors.user: intfloat
paper_authors._id: 65936c78a33889b27ffac471
paper_authors.name: Nan Yang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-05-13T07:50:05.041Z
paper_authors._id: 6341342a9948f573f36d77da
paper_authors.avatarUrl: /avatars/813717cb180b2ce566421b43dadabb43.svg
paper_authors.fullname: nyanyanya
paper_authors.type: user
paper_authors.user: nyanyanya
paper_authors._id: 65936c78a33889b27ffac472
paper_authors.name: Xiaolong Huang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-09T14:01:41.433Z
paper_authors._id: 64149ba8e682c73521d46ecb
paper_authors.avatarUrl: /avatars/c1859aa2d295077e798c8bdc83664a07.svg
paper_authors.fullname: Xiaolong Huang
paper_authors.type: user
paper_authors.user: DiamondH
paper_authors._id: 65936c78a33889b27ffac473
paper_authors.name: Linjun Yang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-09T14:01:34.657Z
paper_authors._id: 641db563c9ce8fdbce4eaa86
paper_authors.avatarUrl: /avatars/ec4012d12a56cba5e27c9eaf58fc7691.svg
paper_authors.fullname: Linjun Yang
paper_authors.type: user
paper_authors.user: Wadeyang
paper_authors._id: 65936c78a33889b27ffac474
paper_authors.name: Rangan Majumder
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-09T14:01:25.226Z
paper_authors._id: 64851c3610e6b874559ef7ae
paper_authors.avatarUrl: /avatars/2847eb17a5fa45c93280d4224c33ddf0.svg
paper_authors.fullname: Rangan Majumder
paper_authors.type: user
paper_authors.user: Rangosaur
paper_authors._id: 65936c78a33889b27ffac475
paper_authors.name: Furu Wei
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-09T14:01:16.092Z
paper_authors._id: 6368c512fbfe97c16a40baba
paper_authors.avatarUrl: /avatars/1c23bc7c0b6d9225699ce27647623d7a.svg
paper_authors.fullname: Furu Wei
paper_authors.type: user
paper_authors.user: thegenerality
paper_title: Improving Text Embeddings with Large Language Models
paper_summary: In this paper, we introduce a novel and simple method for obtaining
  high-quality text embeddings using only synthetic data and less than 1k
  training steps. Unlike existing methods that often depend on multi-stage
  intermediate pre-training with billions of weakly-supervised text pairs,
  followed by fine-tuning with a few labeled datasets, our method does not
  require building complex training pipelines or relying on manually collected
  datasets that are often constrained by task diversity and language coverage. We
  leverage proprietary LLMs to generate diverse synthetic data for hundreds of
  thousands of text embedding tasks across nearly 100 languages. We then
  fine-tune open-source decoder-only LLMs on the synthetic data using standard
  contrastive loss. Experiments demonstrate that our method achieves strong
  performance on highly competitive text embedding benchmarks without using any
  labeled data. Furthermore, when fine-tuned with a mixture of synthetic and
  labeled data, our model sets new state-of-the-art results on the BEIR and MTEB
  benchmarks.
paper_ai_summary: A method using synthetic data and contrastive loss achieves high-quality text embeddings without labeled data, and sets state-of-the-art results when combined with labeled data.
paper_ai_keywords: LLMs synthetic data contrastive loss text embeddings BEIR MTEB