title: R2-T2: Re-Routing in Test-Time for Multimodal Mixture-of-Experts
summary: In large multimodal models (LMMs), the perception of non-language modalities
  (e.g., visual representations) is usually not on par with the large language
  models (LLMs)' powerful reasoning capabilities, deterring LMMs' performance on
  challenging downstream tasks. This weakness has been recently mitigated by
  replacing the vision encoder with a mixture-of-experts (MoE), which provides
  rich, multi-granularity, and diverse representations required by diverse
  downstream tasks. The performance of multimodal MoE largely depends on its
  router, which reweights and mixes the representations of different experts for
  each input. However, we find that the end-to-end trained router does not always
  produce the optimal routing weights for every test sample. To bridge the gap,
  we propose a novel and efficient method "Re-Routing in Test-Time(R2-T2) that
  locally optimizes the vector of routing weights in test-time by moving it
  toward those vectors of the correctly predicted samples in a neighborhood of
  the test sample. We propose three R2-T2 strategies with different optimization
  objectives and neighbor-search spaces. R2-T2 consistently and greatly improves
  state-of-the-art LMMs' performance on challenging benchmarks of diverse tasks,
  without training any base-model parameters.
paper_authors._id: 67c12b5def9af74902537b98
paper_authors.name: Zhongyang Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-28T12:14:22.809Z
paper_authors._id: 671002fd13203512e7b8f9e3
paper_authors.avatarUrl: /avatars/313d8ea313ed300750cfdaaca44fdb6e.svg
paper_authors.fullname: Zhongyang Li
paper_authors.type: user
paper_authors.user: Lzy01241010
paper_authors._id: 67c12b5def9af74902537b99
paper_authors.name: Ziyue Li
paper_authors._id: 67c12b5def9af74902537b9a
paper_authors.name: Tianyi Zhou
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-28T12:14:16.482Z
paper_authors._id: 647f5af5b0e96764589f3b2a
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/VJ4cDyjp5M3V5WmI5gPIU.jpeg
paper_authors.fullname: Tianyi Zhou
paper_authors.type: user
paper_authors.user: zhoutianyi
paper_title: R2-T2: Re-Routing in Test-Time for Multimodal Mixture-of-Experts
paper_summary: In large multimodal models (LMMs), the perception of non-language modalities
  (e.g., visual representations) is usually not on par with the large language
  models (LLMs)' powerful reasoning capabilities, deterring LMMs' performance on
  challenging downstream tasks. This weakness has been recently mitigated by
  replacing the vision encoder with a mixture-of-experts (MoE), which provides
  rich, multi-granularity, and diverse representations required by diverse
  downstream tasks. The performance of multimodal MoE largely depends on its
  router, which reweights and mixes the representations of different experts for
  each input. However, we find that the end-to-end trained router does not always
  produce the optimal routing weights for every test sample. To bridge the gap,
  we propose a novel and efficient method "Re-Routing in Test-Time(R2-T2) that
  locally optimizes the vector of routing weights in test-time by moving it
  toward those vectors of the correctly predicted samples in a neighborhood of
  the test sample. We propose three R2-T2 strategies with different optimization
  objectives and neighbor-search spaces. R2-T2 consistently and greatly improves
  state-of-the-art LMMs' performance on challenging benchmarks of diverse tasks,
  without training any base-model parameters.
paper_ai_summary: A test-time re-routing method improves the performance of multimodal models by optimizing routing weights for non-language modalities, enhancing accuracy in challenging tasks without training.
paper_ai_keywords: multimodal models large language models vision encoder mixture-of-experts router test-time re-routing R2-T2 routing weights end-to-end training optimization strategies benchmark tasks