title: Mixture-of-Supernets: Improving Weight-Sharing Supernet Training with
    Architecture-Routed Mixture-of-Experts
summary: Weight-sharing supernet has become a vital component for performance
  estimation in the state-of-the-art (SOTA) neural architecture search (NAS)
  frameworks. Although supernet can directly generate different subnetworks
  without retraining, there is no guarantee for the quality of these subnetworks
  because of weight sharing. In NLP tasks such as machine translation and
  pre-trained language modeling, we observe that given the same model
  architecture, there is a large performance gap between supernet and training
  from scratch. Hence, supernet cannot be directly used and retraining is
  necessary after finding the optimal architectures.
    In this work, we propose mixture-of-supernets, a generalized supernet
  formulation where mixture-of-experts (MoE) is adopted to enhance the expressive
  power of the supernet model, with negligible training overhead. In this way,
  different subnetworks do not share the model weights directly, but through an
  architecture-based routing mechanism. As a result, model weights of different
  subnetworks are customized towards their specific architectures and the weight
  generation is learned by gradient descent. Compared to existing weight-sharing
  supernet for NLP, our method can minimize the retraining time, greatly
  improving training efficiency. In addition, the proposed method achieves the
  SOTA performance in NAS for building fast machine translation models, yielding
  better latency-BLEU tradeoff compared to HAT, state-of-the-art NAS for MT. We
  also achieve the SOTA performance in NAS for building memory-efficient
  task-agnostic BERT models, outperforming NAS-BERT and AutoDistil in various
  model sizes.
paper_authors._id: 64828d90709d2cdcac47bdc8
paper_authors.name: Ganesh Jawahar
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-06-09T17:06:17.883Z
paper_authors._id: 64835bc7c826244a1fddd0a4
paper_authors.avatarUrl: /avatars/20ca0523fa980da7063a0b83e20ef7b2.svg
paper_authors.fullname: Ganesh Jawahar
paper_authors.type: user
paper_authors.user: ganeshjwhr
paper_authors._id: 64828d90709d2cdcac47bdc9
paper_authors.name: Haichuan Yang
paper_authors._id: 64828d90709d2cdcac47bdca
paper_authors.name: Yunyang Xiong
paper_authors._id: 64828d90709d2cdcac47bdcb
paper_authors.name: Zechun Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-12T13:42:14.734Z
paper_authors._id: 620ae55fe190de4b8f27ba9d
paper_authors.avatarUrl: /avatars/aaa2d0eadbcf96c2eb9059e3d73c2760.svg
paper_authors.fullname: Liu
paper_authors.type: user
paper_authors.user: Zechun
paper_authors._id: 64828d90709d2cdcac47bdcc
paper_authors.name: Dilin Wang
paper_authors._id: 64828d90709d2cdcac47bdcd
paper_authors.name: Fei Sun
paper_authors._id: 64828d90709d2cdcac47bdce
paper_authors.name: Meng Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-10T07:57:51.809Z
paper_authors._id: 65211f76c709aaca9a8bc5c7
paper_authors.avatarUrl: /avatars/4fa7619cb18b7ad6869e8fe1547c02cf.svg
paper_authors.fullname: Meng Li
paper_authors.type: user
paper_authors.user: ml-pku
paper_authors._id: 64828d90709d2cdcac47bdcf
paper_authors.name: Aasish Pappu
paper_authors._id: 64828d90709d2cdcac47bdd0
paper_authors.name: Barlas Oguz
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-12T13:44:23.294Z
paper_authors._id: 6317cb5b212fce5a3cd97227
paper_authors.avatarUrl: /avatars/c2b4eefa7c17cdf2299c07466c361ab0.svg
paper_authors.fullname: Barlas Oguz
paper_authors.type: user
paper_authors.user: barlaso
paper_authors._id: 64828d90709d2cdcac47bdd1
paper_authors.name: Muhammad Abdul-Mageed
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-12T13:44:43.727Z
paper_authors._id: 5feeae8dd8e0b2d24ed258cc
paper_authors.avatarUrl: /avatars/1d66840db4ed6dc12bdee7ba8168ea9f.svg
paper_authors.fullname: Muhammad Abdul-Mageed
paper_authors.type: user
paper_authors.user: mageed
paper_authors._id: 64828d90709d2cdcac47bdd2
paper_authors.name: Laks V. S. Lakshmanan
paper_authors._id: 64828d90709d2cdcac47bdd3
paper_authors.name: Raghuraman Krishnamoorthi
paper_authors._id: 64828d90709d2cdcac47bdd4
paper_authors.name: Vikas Chandra
paper_title: Mixture-of-Supernets: Improving Weight-Sharing Supernet Training with
    Architecture-Routed Mixture-of-Experts
paper_summary: Weight-sharing supernet has become a vital component for performance
  estimation in the state-of-the-art (SOTA) neural architecture search (NAS)
  frameworks. Although supernet can directly generate different subnetworks
  without retraining, there is no guarantee for the quality of these subnetworks
  because of weight sharing. In NLP tasks such as machine translation and
  pre-trained language modeling, we observe that given the same model
  architecture, there is a large performance gap between supernet and training
  from scratch. Hence, supernet cannot be directly used and retraining is
  necessary after finding the optimal architectures.
    In this work, we propose mixture-of-supernets, a generalized supernet
  formulation where mixture-of-experts (MoE) is adopted to enhance the expressive
  power of the supernet model, with negligible training overhead. In this way,
  different subnetworks do not share the model weights directly, but through an
  architecture-based routing mechanism. As a result, model weights of different
  subnetworks are customized towards their specific architectures and the weight
  generation is learned by gradient descent. Compared to existing weight-sharing
  supernet for NLP, our method can minimize the retraining time, greatly
  improving training efficiency. In addition, the proposed method achieves the
  SOTA performance in NAS for building fast machine translation models, yielding
  better latency-BLEU tradeoff compared to HAT, state-of-the-art NAS for MT. We
  also achieve the SOTA performance in NAS for building memory-efficient
  task-agnostic BERT models, outperforming NAS-BERT and AutoDistil in various
  model sizes.
paper_ai_summary: Mixture-of-supernets with an architecture-based routing mechanism improves NAS efficiency and performance in NLP tasks by minimizing retraining time.
paper_ai_keywords: mixture-of-supernets mixture-of-experts (MoE) architecture-based routing mechanism neural architecture search (NAS) machine translation pre-trained language modeling weight-sharing supernet gradient descent latency-BLEU tradeoff BERT models NAS-BERT AutoDistil