title: Gecko: Versatile Text Embeddings Distilled from Large Language Models
summary: We present Gecko, a compact and versatile text embedding model. Gecko
  achieves strong retrieval performance by leveraging a key idea: distilling
  knowledge from large language models (LLMs) into a retriever. Our two-step
  distillation process begins with generating diverse, synthetic paired data
  using an LLM. Next, we further refine the data quality by retrieving a set of
  candidate passages for each query, and relabeling the positive and hard
  negative passages using the same LLM. The effectiveness of our approach is
  demonstrated by the compactness of the Gecko. On the Massive Text Embedding
  Benchmark (MTEB), Gecko with 256 embedding dimensions outperforms all existing
  entries with 768 embedding size. Gecko with 768 embedding dimensions achieves
  an average score of 66.31, competing with 7x larger models and 5x higher
  dimensional embeddings.
paper_authors._id: 660a3edbdf88d3dbac12b9e2
paper_authors.name: Jinhyuk Lee
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2024-04-09T18:33:23.443Z
paper_authors._id: 63fd3edc3c880680af44aa78
paper_authors.avatarUrl: /avatars/98759e23e89f9da3ce13266d030e611b.svg
paper_authors.fullname: Jinhyuk Lee
paper_authors.type: user
paper_authors.user: jinhyuklee
paper_authors._id: 660a3edbdf88d3dbac12b9e3
paper_authors.name: Zhuyun Dai
paper_authors._id: 660a3edbdf88d3dbac12b9e4
paper_authors.name: Xiaoqi Ren
paper_authors._id: 660a3edbdf88d3dbac12b9e5
paper_authors.name: Blair Chen
paper_authors._id: 660a3edbdf88d3dbac12b9e6
paper_authors.name: Daniel Cer
paper_authors._id: 660a3edbdf88d3dbac12b9e7
paper_authors.name: Jeremy R. Cole
paper_authors._id: 660a3edbdf88d3dbac12b9e8
paper_authors.name: Kai Hui
paper_authors._id: 660a3edbdf88d3dbac12b9e9
paper_authors.name: Michael Boratko
paper_authors._id: 660a3edbdf88d3dbac12b9ea
paper_authors.name: Rajvi Kapadia
paper_authors._id: 660a3edbdf88d3dbac12b9eb
paper_authors.name: Wen Ding
paper_authors._id: 660a3edbdf88d3dbac12b9ec
paper_authors.name: Yi Luan
paper_authors._id: 660a3edbdf88d3dbac12b9ed
paper_authors.name: Sai Meher Karthik Duddu
paper_authors._id: 660a3edbdf88d3dbac12b9ee
paper_authors.name: Gustavo Hernandez Abrego
paper_authors._id: 660a3edbdf88d3dbac12b9ef
paper_authors.name: Weiqiang Shi
paper_authors._id: 660a3edbdf88d3dbac12b9f0
paper_authors.name: Nithi Gupta
paper_authors._id: 660a3edbdf88d3dbac12b9f1
paper_authors.name: Aditya Kusupati
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-16T09:45:24.961Z
paper_authors._id: 65dd3b03f0f26b277be1e91e
paper_authors.avatarUrl: /avatars/67160c2d5241aea5c5d9b211f21f9caa.svg
paper_authors.fullname: Aditya Kusupati
paper_authors.type: user
paper_authors.user: adityakusupati
paper_authors._id: 660a3edbdf88d3dbac12b9f2
paper_authors.name: Prateek Jain
paper_authors._id: 660a3edbdf88d3dbac12b9f3
paper_authors.name: Siddhartha Reddy Jonnalagadda
paper_authors._id: 660a3edbdf88d3dbac12b9f4
paper_authors.name: Ming-Wei Chang
paper_authors._id: 660a3edbdf88d3dbac12b9f5
paper_authors.name: Iftekhar Naim
paper_title: Gecko: Versatile Text Embeddings Distilled from Large Language Models
paper_summary: We present Gecko, a compact and versatile text embedding model. Gecko
  achieves strong retrieval performance by leveraging a key idea: distilling
  knowledge from large language models (LLMs) into a retriever. Our two-step
  distillation process begins with generating diverse, synthetic paired data
  using an LLM. Next, we further refine the data quality by retrieving a set of
  candidate passages for each query, and relabeling the positive and hard
  negative passages using the same LLM. The effectiveness of our approach is
  demonstrated by the compactness of the Gecko. On the Massive Text Embedding
  Benchmark (MTEB), Gecko with 256 embedding dimensions outperforms all existing
  entries with 768 embedding size. Gecko with 768 embedding dimensions achieves
  an average score of 66.31, competing with 7x larger models and 5x higher
  dimensional embeddings.
paper_ai_summary: Gecko is a compact and versatile text embedding model that distills knowledge from large language models to achieve strong retrieval performance on benchmarks.
paper_ai_keywords: text embedding model distillation large language models synthetic paired data candidate passages retriever hard negative passages Massive Text Embedding Benchmark