Log-Scale Quantization in Distributed First-Order Methods: Gradient-based Learning from Distributed Data
<|reference_start|>Log-Scale Quantization in Distributed First-Order Methods: Gradient-based Learning from Distributed Data: Decentralized strategies are of interest for learning from large-scale data over networks. This paper studies learning over a network of geographically distributed nodes/agents subject to quantization. Each node possesses a private local cost function, collectively contributing to a global cost function, which the proposed methodology aims to minimize. In contrast to many existing literature, the information exchange among nodes is quantized. We adopt a first-order computationally-efficient distributed optimization algorithm (with no extra inner consensus loop) that leverages node-level gradient correction based on local data and network-level gradient aggregation only over nearby nodes. This method only requires balanced networks with no need for stochastic weight design. It can handle log-scale quantized data exchange over possibly time-varying and switching network setups. We analyze convergence over both structured networks (for example, training over data-centers) and ad-hoc multi-agent networks (for example, training over dynamic robotic networks). Through analysis and experimental validation, we show that (i) structured networks generally result in a smaller optimality gap, and (ii) logarithmic quantization leads to smaller optimality gap compared to uniform quantization.<|reference_end|>
@article{doostmohammadian2024log-scale,
    title={Log-Scale Quantization in Distributed First-Order Methods:
  Gradient-based Learning from Distributed Data},
    author={Mohammadreza Doostmohammadian, Muhammad I. Qureshi, Mohammad Hossein
  Khalesi, Hamid R. Rabiee, Usman A. Khan},
    journal={arXiv preprint arXiv:2406.00621},
    year={2024},
    archivePrefix={arXiv},
    eprint={2406.00621},
    primaryClass={eess.SY cs.SY eess.SP math.OC}
}