MoTE: Mixture of Ternary Experts for Memory-efficient Large Multimodal
  Models
Large multimodal Mixture-of-Experts (MoEs) effectively scale the model size
to boost performance while maintaining fixed active parameters. However,
previous works primarily utilized full-precision experts during sparse
up-cycling. Despite they show superior performance on end tasks, the large
amount of experts introduces higher memory footprint, which poses significant
challenges for the deployment on edge devices. In this work, we propose MoTE, a
scalable and memory-efficient approach to train Mixture-of-Ternary-Experts
models from dense checkpoint. Instead of training fewer high-precision experts,
we propose to train more low-precision experts during up-cycling. Specifically,
we use the pre-trained FFN as a shared expert and train ternary routed experts
with parameters in {-1, 0, 1}. Extensive experiments show that our approach has
promising scaling trend along model size. MoTE achieves comparable performance
to full-precision baseline MoE-LLaVA while offering lower memory footprint.
Furthermore, our approach is compatible with post-training quantization methods
and the advantage further amplifies when memory-constraint goes lower. Given
the same amount of expert memory footprint of 3.4GB and combined with
post-training quantization, MoTE outperforms MoE-LLaVA by a gain of 4.3%
average accuracy on end tasks, demonstrating its effectiveness and potential
for memory-constrained devices.
68537b2a99bf39f9665c7990
Hongyu Wang
claimed_verified
2025-07-03T07:41:50.384Z
63f71771d36951307fcb4dcd
https://cdn-avatars.huggingface.co/v1/production/uploads/63f71771d36951307fcb4dcd/1c-GSQmSSuDXyVWjxZFy_.jpeg
Hongyu Wang
user
hongyuw
68537b2a99bf39f9665c7991
Jiayu Xu
68537b2a99bf39f9665c7992
Ruiping Wang
68537b2a99bf39f9665c7993
Yan Feng
68537b2a99bf39f9665c7994
Yitao Zhai
68537b2a99bf39f9665c7995
Peng Pei
68537b2a99bf39f9665c7996
Xunliang Cai
68537b2a99bf39f9665c7997
Xilin Chen
MoTE: Mixture of Ternary Experts for Memory-efficient Large Multimodal
  Models
Large multimodal Mixture-of-Experts (MoEs) effectively scale the model size
to boost performance while maintaining fixed active parameters. However,
previous works primarily utilized full-precision experts during sparse
up-cycling. Despite they show superior performance on end tasks, the large
amount of experts introduces higher memory footprint, which poses significant
challenges for the deployment on edge devices. In this work, we propose MoTE, a
scalable and memory-efficient approach to train Mixture-of-Ternary-Experts
models from dense checkpoint. Instead of training fewer high-precision experts,
we propose to train more low-precision experts during up-cycling. Specifically,
we use the pre-trained FFN as a shared expert and train ternary routed experts
with parameters in {-1, 0, 1}. Extensive experiments show that our approach has
promising scaling trend along model size. MoTE achieves comparable performance
to full-precision baseline MoE-LLaVA while offering lower memory footprint.
Furthermore, our approach is compatible with post-training quantization methods
and the advantage further amplifies when memory-constraint goes lower. Given
the same amount of expert memory footprint of 3.4GB and combined with
post-training quantization, MoTE outperforms MoE-LLaVA by a gain of 4.3%
average accuracy on end tasks, demonstrating its effectiveness and potential
for memory-constrained devices.
MoTE, a scalable and memory-efficient method, improves Mixture-of-Experts models using low-precision ternary experts, enhancing performance and reducing memory footprint for deployment on edge devices.
Mixture-of-Experts MoEs sparse up-cycling low-precision ternary experts shared expert FFN pre-trained post-training quantization memory-constrained end tasks