New logarithmic step size for stochastic gradient descent
<|reference_start|>New logarithmic step size for stochastic gradient descent: In this paper, we propose a novel warm restart technique using a new logarithmic step size for the stochastic gradient descent (SGD) approach. For smooth and non-convex functions, we establish an $O(\frac{1}{\sqrt{T}})$ convergence rate for the SGD. We conduct a comprehensive implementation to demonstrate the efficiency of the newly proposed step size on the ~FashionMinst,~ CIFAR10, and CIFAR100 datasets. Moreover, we compare our results with nine other existing approaches and demonstrate that the new logarithmic step size improves test accuracy by $0.9\%$ for the CIFAR100 dataset when we utilize a convolutional neural network (CNN) model.<|reference_end|>
@article{shamaee2024new,
    title={New logarithmic step size for stochastic gradient descent},
    author={M. Soheil Shamaee, S. Fathi Hafshejani, Z. Saeidian},
    journal={Frontiers of Computer Science, 2025},
    year={2024},
    doi={10.1007/s11704-023-3245-z},
    archivePrefix={arXiv},
    eprint={2404.01257},
    primaryClass={cs.LG math.OC}
}