title: C3PO: Critical-Layer, Core-Expert, Collaborative Pathway Optimization
    for Test-Time Expert Re-Mixing
summary: Mixture-of-Experts (MoE) Large Language Models (LLMs) suffer from severely
  sub-optimal expert pathways-our study reveals that naive expert selection
  learned from pretraining leaves a surprising 10-20% accuracy gap for
  improvement. Motivated by this observation, we develop a novel class of
  test-time optimization methods to re-weight or "re-mixing" the experts in
  different layers jointly for each test sample. Since the test sample's ground
  truth is unknown, we propose to optimize a surrogate objective defined by the
  sample's "successful neighbors" from a reference set of samples. We introduce
  three surrogates and algorithms based on mode-finding, kernel regression, and
  the average loss of similar reference samples/tasks. To reduce the cost of
  optimizing whole pathways, we apply our algorithms merely to the core experts'
  mixing weights in critical layers, which enjoy similar performance but save
  significant computation. This leads to "Critical-Layer, Core-Expert,
  Collaborative Pathway Optimization (C3PO)". We apply C3PO to two recent MoE
  LLMs and examine it on six widely-used benchmarks. It consistently improves the
  base model by 7-15% in accuracy and outperforms widely used test-time learning
  baselines, e.g., in-context learning and prompt/prefix tuning, by a large
  margin. Moreover, C3PO enables MoE LLMs with 1-3B active parameters to
  outperform LLMs of 7-9B parameters, hence improving MoE's advantages on
  efficiency. Our thorough ablation study further sheds novel insights on
  achieving test-time improvement on MoE.
paper_authors._id: 67f88c848178ca61d74e001d
paper_authors.name: Zhongyang Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-11T07:25:08.340Z
paper_authors._id: 671002fd13203512e7b8f9e3
paper_authors.avatarUrl: /avatars/313d8ea313ed300750cfdaaca44fdb6e.svg
paper_authors.fullname: Zhongyang Li
paper_authors.type: user
paper_authors.user: Lzy01241010
paper_authors._id: 67f88c848178ca61d74e001e
paper_authors.name: Ziyue Li
paper_authors._id: 67f88c848178ca61d74e001f
paper_authors.name: Tianyi Zhou
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-11T07:25:10.558Z
paper_authors._id: 647f5af5b0e96764589f3b2a
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/VJ4cDyjp5M3V5WmI5gPIU.jpeg
paper_authors.fullname: Tianyi Zhou
paper_authors.type: user
paper_authors.user: zhoutianyi
paper_title: C3PO: Critical-Layer, Core-Expert, Collaborative Pathway Optimization
    for Test-Time Expert Re-Mixing
paper_summary: Mixture-of-Experts (MoE) Large Language Models (LLMs) suffer from severely
  sub-optimal expert pathways-our study reveals that naive expert selection
  learned from pretraining leaves a surprising 10-20% accuracy gap for
  improvement. Motivated by this observation, we develop a novel class of
  test-time optimization methods to re-weight or "re-mixing" the experts in
  different layers jointly for each test sample. Since the test sample's ground
  truth is unknown, we propose to optimize a surrogate objective defined by the
  sample's "successful neighbors" from a reference set of samples. We introduce
  three surrogates and algorithms based on mode-finding, kernel regression, and
  the average loss of similar reference samples/tasks. To reduce the cost of
  optimizing whole pathways, we apply our algorithms merely to the core experts'
  mixing weights in critical layers, which enjoy similar performance but save
  significant computation. This leads to "Critical-Layer, Core-Expert,
  Collaborative Pathway Optimization (C3PO)". We apply C3PO to two recent MoE
  LLMs and examine it on six widely-used benchmarks. It consistently improves the
  base model by 7-15% in accuracy and outperforms widely used test-time learning
  baselines, e.g., in-context learning and prompt/prefix tuning, by a large
  margin. Moreover, C3PO enables MoE LLMs with 1-3B active parameters to
  outperform LLMs of 7-9B parameters, hence improving MoE's advantages on
  efficiency. Our thorough ablation study further sheds novel insights on
  achieving test-time improvement on MoE.
paper_ai_summary: A novel test-time optimization method, C3PO, improves the accuracy and efficiency of Mixture-of-Experts Large Language Models by re-weighting experts based on surrogate objectives from neighboring samples.
paper_ai_keywords: Mixture-of-Experts (MoE) Large Language Models (LLMs) expert selection test-time optimization surrogate objective successful neighbors mode-finding kernel regression average loss core experts critical layers in-context learning prompt/prefix tuning ablation study