title: Generative Representational Instruction Tuning
summary: All text-based language problems can be reduced to either generation or
  embedding. Current models only perform well at one or the other. We introduce
  generative representational instruction tuning (GRIT) whereby a large language
  model is trained to handle both generative and embedding tasks by
  distinguishing between them through instructions. Compared to other open
  models, our resulting GritLM 7B sets a new state of the art on the Massive Text
  Embedding Benchmark (MTEB) and outperforms all models up to its size on a range
  of generative tasks. By scaling up further, GritLM 8x7B outperforms all open
  generative language models that we tried while still being among the best
  embedding models. Notably, we find that GRIT matches training on only
  generative or embedding data, thus we can unify both at no performance loss.
  Among other benefits, the unification via GRIT speeds up Retrieval-Augmented
  Generation (RAG) by > 60% for long documents, by no longer requiring separate
  retrieval and generation models. Models, code, etc. are freely available at
  https://github.com/ContextualAI/gritlm.
paper_authors._id: 65cec094e1c8a3d33cf694d7
paper_authors.name: Niklas Muennighoff
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-16T10:30:18.105Z
paper_authors._id: 5f1eb362eec0ad2a071ad6e2
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/5f1eb362eec0ad2a071ad6e2/IXMYkYKuTwn6kBdWnQeeY.png
paper_authors.fullname: Niklas Muennighoff
paper_authors.type: user
paper_authors.user: Muennighoff
paper_authors._id: 65cec094e1c8a3d33cf694d8
paper_authors.name: Hongjin Su
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-19T11:25:09.019Z
paper_authors._id: 632d53951538d4798a73c849
paper_authors.avatarUrl: /avatars/b7d0a895e669bcd1303c4716b5401c36.svg
paper_authors.fullname: Hongjin SU
paper_authors.type: user
paper_authors.user: multi-train
paper_authors._id: 65cec094e1c8a3d33cf694d9
paper_authors.name: Liang Wang
paper_authors._id: 65cec094e1c8a3d33cf694da
paper_authors.name: Nan Yang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-05-13T07:49:59.502Z
paper_authors._id: 6341342a9948f573f36d77da
paper_authors.avatarUrl: /avatars/813717cb180b2ce566421b43dadabb43.svg
paper_authors.fullname: nyanyanya
paper_authors.type: user
paper_authors.user: nyanyanya
paper_authors._id: 65cec094e1c8a3d33cf694db
paper_authors.name: Furu Wei
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-16T10:34:15.306Z
paper_authors._id: 6368c512fbfe97c16a40baba
paper_authors.avatarUrl: /avatars/1c23bc7c0b6d9225699ce27647623d7a.svg
paper_authors.fullname: Furu Wei
paper_authors.type: user
paper_authors.user: thegenerality
paper_authors._id: 65cec094e1c8a3d33cf694dc
paper_authors.name: Tao Yu
paper_authors._id: 65cec094e1c8a3d33cf694dd
paper_authors.name: Amanpreet Singh
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-02-19T09:34:42.650Z
paper_authors._id: 6230c6ecfd8b720a5648f6c4
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1652253065548-6230c6ecfd8b720a5648f6c4.jpeg
paper_authors.fullname: Amanpreet Singh
paper_authors.type: user
paper_authors.user: aps
paper_authors._id: 65cec094e1c8a3d33cf694de
paper_authors.name: Douwe Kiela
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-16T10:34:07.576Z
paper_authors._id: 61dc997715b47073db1620dc
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1641847245435-61dc997715b47073db1620dc.jpeg
paper_authors.fullname: Douwe Kiela
paper_authors.type: user
paper_authors.user: douwekiela
paper_title: Generative Representational Instruction Tuning
paper_summary: All text-based language problems can be reduced to either generation or
  embedding. Current models only perform well at one or the other. We introduce
  generative representational instruction tuning (GRIT) whereby a large language
  model is trained to handle both generative and embedding tasks by
  distinguishing between them through instructions. Compared to other open
  models, our resulting GritLM 7B sets a new state of the art on the Massive Text
  Embedding Benchmark (MTEB) and outperforms all models up to its size on a range
  of generative tasks. By scaling up further, GritLM 8x7B outperforms all open
  generative language models that we tried while still being among the best
  embedding models. Notably, we find that GRIT matches training on only
  generative or embedding data, thus we can unify both at no performance loss.
  Among other benefits, the unification via GRIT speeds up Retrieval-Augmented
  Generation (RAG) by > 60% for long documents, by no longer requiring separate
  retrieval and generation models. Models, code, etc. are freely available at
  https://github.com/ContextualAI/gritlm.
paper_ai_keywords: generative representational instruction tuning (GRIT) GritLM 7B Massive Text Embedding Benchmark (MTEB) generative tasks GritLM 8x7B Retrieval-Augmented Generation (RAG)