VeOmni: Scaling Any Modality Model Training with Model-Centric
  Distributed Recipe Zoo
Recent advances in large language models (LLMs) have driven impressive
progress in omni-modal understanding and generation. However, training
omni-modal LLMs remains a significant challenge due to the heterogeneous model
architectures required to process diverse modalities, necessitating
sophisticated system design for efficient large-scale training. Existing
frameworks typically entangle model definition with parallel logic, incurring
limited scalability and substantial engineering overhead for end-to-end
omni-modal training. % We present \veomni, a modular and efficient training
framework to accelerate the development of omni-modal LLMs. \veomni introduces
model-centric distributed recipes that decouples communication from
computation, enabling efficient 3D parallelism on omni-modal LLMs. \veomni also
features a flexible configuration interface supporting seamless integration of
new modalities with minimal code change. % Using \veomni, a omni-modal
mixture-of-experts (MoE) model with 30B parameters can be trained with over
2,800 tokens/sec/GPU throughput and scale to 160K context lengths via 3D
parallelism on 128 GPUs, showcasing its superior efficiency and scalability for
training large omni-modal LLMs.
68917ae7f01a094725f834cd
Qianli Ma
68917ae7f01a094725f834ce
Yaowei Zheng
68917ae7f01a094725f834cf
Zhelun Shi
68917ae7f01a094725f834d0
Zhongkai Zhao
68917ae7f01a094725f834d1
Bin Jia
68917ae7f01a094725f834d2
Ziyue Huang
68917ae7f01a094725f834d3
Zhiqi Lin
68917ae7f01a094725f834d4
Youjie Li
68917ae7f01a094725f834d5
Jiacheng Yang
68917ae7f01a094725f834d6
Yanghua Peng
68917ae7f01a094725f834d7
Zhi Zhang
68917ae7f01a094725f834d8
Xin Liu
VeOmni: Scaling Any Modality Model Training with Model-Centric
  Distributed Recipe Zoo
Recent advances in large language models (LLMs) have driven impressive
progress in omni-modal understanding and generation. However, training
omni-modal LLMs remains a significant challenge due to the heterogeneous model
architectures required to process diverse modalities, necessitating
sophisticated system design for efficient large-scale training. Existing
frameworks typically entangle model definition with parallel logic, incurring
limited scalability and substantial engineering overhead for end-to-end
omni-modal training. % We present \veomni, a modular and efficient training
framework to accelerate the development of omni-modal LLMs. \veomni introduces
model-centric distributed recipes that decouples communication from
computation, enabling efficient 3D parallelism on omni-modal LLMs. \veomni also
features a flexible configuration interface supporting seamless integration of
new modalities with minimal code change. % Using \veomni, a omni-modal
mixture-of-experts (MoE) model with 30B parameters can be trained with over
2,800 tokens/sec/GPU throughput and scale to 160K context lengths via 3D
parallelism on 128 GPUs, showcasing its superior efficiency and scalability for
training large omni-modal LLMs.
A modular training framework accelerates the development of omni-modal LLMs through efficient 3D parallelism and flexible configuration.
large language models omni-modal understanding omni-modal generation heterogeneous model architectures parallel logic model-centric distributed recipes communication computation 3D parallelism flexible configuration interface mixture-of-experts tokens/sec/GPU throughput context lengths