title: Symbolic Mixture-of-Experts: Adaptive Skill-based Routing for
    Heterogeneous Reasoning
summary: Combining existing pre-trained expert LLMs is a promising avenue for scalably
  tackling large-scale and diverse tasks. However, selecting experts at the task
  level is often too coarse-grained, as heterogeneous tasks may require different
  expertise for each instance. To enable adaptive instance-level mixing of
  pre-trained LLM experts, we propose Symbolic-MoE, a symbolic, text-based, and
  gradient-free Mixture-of-Experts framework. Symbolic-MoE takes a fine-grained
  approach to selection by emphasizing skills, e.g., algebra in math or molecular
  biology in biomedical reasoning. We propose a skill-based recruiting strategy
  that dynamically selects the most relevant set of expert LLMs for diverse
  reasoning tasks based on their strengths. Each selected expert then generates
  its own reasoning, resulting in k outputs from k experts, which are then
  synthesized into a final high-quality response by an aggregator chosen based on
  its ability to integrate diverse reasoning outputs. We show that Symbolic-MoE's
  instance-level expert selection improves performance by a large margin but --
  when implemented naively -- can introduce a high computational overhead due to
  the need for constant model loading and offloading. To address this, we
  implement a batch inference strategy that groups instances based on their
  assigned experts, loading each model only once. This allows us to integrate 16
  expert models on 1 GPU with a time cost comparable to or better than prior
  multi-agent baselines using 4 GPUs. Through extensive evaluations on diverse
  benchmarks (MMLU-Pro, GPQA, AIME, and MedMCQA), we demonstrate that
  Symbolic-MoE outperforms strong LLMs like GPT4o-mini, as well as multi-agent
  approaches, with an absolute average improvement of 8.15% over the best
  multi-agent baseline. Moreover, Symbolic-MoE removes the need for expensive
  multi-round discussions, outperforming discussion baselines with less
  computation.
paper_authors._id: 67d0446aa9ae7682cd1c62b4
paper_authors.name: Justin Chih-Yao Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-11T14:15:18.833Z
paper_authors._id: 62ce26129f723d34cf1f595a
paper_authors.avatarUrl: /avatars/8e305ac7c170d70fbf83c109789b40d9.svg
paper_authors.fullname: Justin Chen
paper_authors.type: user
paper_authors.user: dinobby
paper_authors._id: 67d0446aa9ae7682cd1c62b5
paper_authors.name: Sukwon Yun
paper_authors._id: 67d0446aa9ae7682cd1c62b6
paper_authors.name: Elias Stengel-Eskin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-11T14:14:40.165Z
paper_authors._id: 61781c4caf41befe8ff060e8
paper_authors.avatarUrl: /avatars/8871d7b046fc28cbc8638228da8e9737.svg
paper_authors.fullname: Elias Stengel-Eskin
paper_authors.type: user
paper_authors.user: esteng
paper_authors._id: 67d0446aa9ae7682cd1c62b7
paper_authors.name: Tianlong Chen
paper_authors._id: 67d0446aa9ae7682cd1c62b8
paper_authors.name: Mohit Bansal
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-11T14:14:22.014Z
paper_authors._id: 665d9d3a057f7c508f98c625
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/665d9d3a057f7c508f98c625/u1R9P9sJoAl4zEIcetbPy.jpeg
paper_authors.fullname: Mohit Bansal
paper_authors.type: user
paper_authors.user: mohitbansal
paper_title: Symbolic Mixture-of-Experts: Adaptive Skill-based Routing for
    Heterogeneous Reasoning
paper_summary: Combining existing pre-trained expert LLMs is a promising avenue for scalably
  tackling large-scale and diverse tasks. However, selecting experts at the task
  level is often too coarse-grained, as heterogeneous tasks may require different
  expertise for each instance. To enable adaptive instance-level mixing of
  pre-trained LLM experts, we propose Symbolic-MoE, a symbolic, text-based, and
  gradient-free Mixture-of-Experts framework. Symbolic-MoE takes a fine-grained
  approach to selection by emphasizing skills, e.g., algebra in math or molecular
  biology in biomedical reasoning. We propose a skill-based recruiting strategy
  that dynamically selects the most relevant set of expert LLMs for diverse
  reasoning tasks based on their strengths. Each selected expert then generates
  its own reasoning, resulting in k outputs from k experts, which are then
  synthesized into a final high-quality response by an aggregator chosen based on
  its ability to integrate diverse reasoning outputs. We show that Symbolic-MoE's
  instance-level expert selection improves performance by a large margin but --
  when implemented naively -- can introduce a high computational overhead due to
  the need for constant model loading and offloading. To address this, we
  implement a batch inference strategy that groups instances based on their
  assigned experts, loading each model only once. This allows us to integrate 16
  expert models on 1 GPU with a time cost comparable to or better than prior
  multi-agent baselines using 4 GPUs. Through extensive evaluations on diverse
  benchmarks (MMLU-Pro, GPQA, AIME, and MedMCQA), we demonstrate that
  Symbolic-MoE outperforms strong LLMs like GPT4o-mini, as well as multi-agent
  approaches, with an absolute average improvement of 8.15% over the best
  multi-agent baseline. Moreover, Symbolic-MoE removes the need for expensive
  multi-round discussions, outperforming discussion baselines with less
  computation.
paper_ai_keywords: Symbolic-MoE Mixture-of-Experts skill-based recruiting strategy instance-level expert selection aggregator batch inference strategy MMLU-Pro GPQA AIME MedMCQA GPT4o-mini