Improved Quantization Strategies for Managing Heavy-tailed Gradients in Distributed Learning
<|reference_start|>Improved Quantization Strategies for Managing Heavy-tailed Gradients in Distributed Learning: Gradient compression has surfaced as a key technique to address the challenge of communication efficiency in distributed learning. In distributed deep learning, however, it is observed that gradient distributions are heavy-tailed, with outliers significantly influencing the design of compression strategies. Existing parameter quantization methods experience performance degradation when this heavy-tailed feature is ignored. In this paper, we introduce a novel compression scheme specifically engineered for heavy-tailed gradients, which effectively combines gradient truncation with quantization. This scheme is adeptly implemented within a communication-limited distributed Stochastic Gradient Descent (SGD) framework. We consider a general family of heavy-tail gradients that follow a power-law distribution, we aim to minimize the error resulting from quantization, thereby determining optimal values for two critical parameters: the truncation threshold and the quantization density. We provide a theoretical analysis on the convergence error bound under both uniform and non-uniform quantization scenarios. Comparative experiments with other benchmarks demonstrate the effectiveness of our proposed method in managing the heavy-tailed gradients in a distributed learning environment.<|reference_end|>
@article{yan2024improved,
    title={Improved Quantization Strategies for Managing Heavy-tailed Gradients in
  Distributed Learning},
    author={Guangfeng Yan, Tan Li, Yuanzhang Xiao, Hanxu Hou and Linqi Song},
    journal={arXiv preprint arXiv:2402.01798},
    year={2024},
    archivePrefix={arXiv},
    eprint={2402.01798},
    primaryClass={cs.LG cs.DC}
}