Disaggregated Multi-Tower: Topology-aware Modeling Technique for Efficient Large-Scale Recommendation
<|reference_start|>Disaggregated Multi-Tower: Topology-aware Modeling Technique for Efficient Large-Scale Recommendation: We study a mismatch between the deep learning recommendation models' flat architecture, common distributed training paradigm and hierarchical data center topology. To address the associated inefficiencies, we propose Disaggregated Multi-Tower (DMT), a modeling technique that consists of (1) Semantic-preserving Tower Transform (SPTT), a novel training paradigm that decomposes the monolithic global embedding lookup process into disjoint towers to exploit data center locality; (2) Tower Module (TM), a synergistic dense component attached to each tower to reduce model complexity and communication volume through hierarchical feature interaction; and (3) Tower Partitioner (TP), a feature partitioner to systematically create towers with meaningful feature interactions and load balanced assignments to preserve model quality and training throughput via learned embeddings. We show that DMT can achieve up to 1.9x speedup compared to the state-of-the-art baselines without losing accuracy across multiple generations of hardware at large data center scales.<|reference_end|>
@article{luo2024disaggregated,
    title={Disaggregated Multi-Tower: Topology-aware Modeling Technique for
  Efficient Large-Scale Recommendation},
    author={Liang Luo, Buyun Zhang, Michael Tsang, Yinbin Ma, Ching-Hsiang Chu,
  Yuxin Chen, Shen Li, Yuchen Hao, Yanli Zhao, Guna Lakshminarayanan, Ellie
  Dingqiao Wen, Jongsoo Park, Dheevatsa Mudigere, Maxim Naumov},
    journal={arXiv preprint arXiv:2403.00877},
    year={2024},
    archivePrefix={arXiv},
    eprint={2403.00877},
    primaryClass={cs.LG cs.DC cs.IR}
}