Minimizing Energy Costs in Deep Learning Model Training: The Gaussian Sampling Approach
<|reference_start|>Minimizing Energy Costs in Deep Learning Model Training: The Gaussian Sampling Approach: Computing the loss gradient via backpropagation consumes considerable energy during deep learning (DL) model training. In this paper, we propose a novel approach to efficiently compute DL models' gradients to mitigate the substantial energy overhead associated with backpropagation. Exploiting the over-parameterized nature of DL models and the smoothness of their loss landscapes, we propose a method called {\em GradSamp} for sampling gradient updates from a Gaussian distribution. Specifically, we update model parameters at a given epoch (chosen periodically or randomly) by perturbing the parameters (element-wise) from the previous epoch with Gaussian ``noise''. The parameters of the Gaussian distribution are estimated using the error between the model parameter values from the two previous epochs. {\em GradSamp} not only streamlines gradient computation but also enables skipping entire epochs, thereby enhancing overall efficiency. We rigorously validate our hypothesis across a diverse set of standard and non-standard CNN and transformer-based models, spanning various computer vision tasks such as image classification, object detection, and image segmentation. Additionally, we explore its efficacy in out-of-distribution scenarios such as Domain Adaptation (DA), Domain Generalization (DG), and decentralized settings like Federated Learning (FL). Our experimental results affirm the effectiveness of {\em GradSamp} in achieving notable energy savings without compromising performance, underscoring its versatility and potential impact in practical DL applications.<|reference_end|>
@article{revanth2024minimizing,
    title={Minimizing Energy Costs in Deep Learning Model Training: The Gaussian
  Sampling Approach},
    author={Challapalli Phanindra Revanth, Sumohana S. Channappayya and C Krishna
  Mohan},
    journal={arXiv preprint arXiv:2406.07332},
    year={2024},
    archivePrefix={arXiv},
    eprint={2406.07332},
    primaryClass={cs.CV}
}