title: Your Mixture-of-Experts LLM Is Secretly an Embedding Model For Free
summary: While large language models (LLMs) excel on generation tasks, their
  decoder-only architecture often limits their potential as embedding models if
  no further representation finetuning is applied. Does this contradict their
  claim of generalists? To answer the question, we take a closer look at
  Mixture-of-Experts (MoE) LLMs. Our study shows that the expert routers in MoE
  LLMs can serve as an off-the-shelf embedding model with promising performance
  on a diverse class of embedding-focused tasks, without requiring any
  finetuning. Moreover, our extensive analysis shows that the MoE routing weights
  (RW) is complementary to the hidden state (HS) of LLMs, a widely-used
  embedding. Compared to HS, we find that RW is more robust to the choice of
  prompts and focuses on high-level semantics. Motivated by the analysis, we
  propose MoEE combining RW and HS, which achieves better performance than using
  either separately. Our exploration of their combination and prompting strategy
  shed several novel insights, e.g., a weighted sum of RW and HS similarities
  outperforms the similarity on their concatenation. Our experiments are
  conducted on 6 embedding tasks with 20 datasets from the Massive Text Embedding
  Benchmark (MTEB). The results demonstrate the significant improvement brought
  by MoEE to LLM-based embedding without further finetuning.
paper_authors._id: 670e2ff1ffc411cfa85d2f1f
paper_authors.name: Ziyue Li
paper_authors._id: 670e2ff1ffc411cfa85d2f20
paper_authors.name: Tianyi Zhou
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-16T09:44:17.073Z
paper_authors._id: 647f5af5b0e96764589f3b2a
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/VJ4cDyjp5M3V5WmI5gPIU.jpeg
paper_authors.fullname: Tianyi Zhou
paper_authors.type: user
paper_authors.user: zhoutianyi
paper_title: Your Mixture-of-Experts LLM Is Secretly an Embedding Model For Free
paper_summary: While large language models (LLMs) excel on generation tasks, their
  decoder-only architecture often limits their potential as embedding models if
  no further representation finetuning is applied. Does this contradict their
  claim of generalists? To answer the question, we take a closer look at
  Mixture-of-Experts (MoE) LLMs. Our study shows that the expert routers in MoE
  LLMs can serve as an off-the-shelf embedding model with promising performance
  on a diverse class of embedding-focused tasks, without requiring any
  finetuning. Moreover, our extensive analysis shows that the MoE routing weights
  (RW) is complementary to the hidden state (HS) of LLMs, a widely-used
  embedding. Compared to HS, we find that RW is more robust to the choice of
  prompts and focuses on high-level semantics. Motivated by the analysis, we
  propose MoEE combining RW and HS, which achieves better performance than using
  either separately. Our exploration of their combination and prompting strategy
  shed several novel insights, e.g., a weighted sum of RW and HS similarities
  outperforms the similarity on their concatenation. Our experiments are
  conducted on 6 embedding tasks with 20 datasets from the Massive Text Embedding
  Benchmark (MTEB). The results demonstrate the significant improvement brought
  by MoEE to LLM-based embedding without further finetuning.
paper_ai_keywords: Mixture-of-Experts (MoE) LLMs (Large Language Models) decoder-only architecture embedding models representation finetuning expert routers off-the-shelf embedding model embedding tasks MoE routing weights (RW) hidden state (HS) MoEE weighted sum of RW and HS similarities Massive Text Embedding Benchmark (MTEB)