Projected Forward Gradient-Guided Frank-Wolfe Algorithm via Variance Reduction
<|reference_start|>Projected Forward Gradient-Guided Frank-Wolfe Algorithm via Variance Reduction: This paper aims to enhance the use of the Frank-Wolfe (FW) algorithm for training deep neural networks. Similar to any gradient-based optimization algorithm, FW suffers from high computational and memory costs when computing gradients for DNNs. This paper introduces the application of the recently proposed projected forward gradient (Projected-FG) method to the FW framework, offering reduced computational cost similar to backpropagation and low memory utilization akin to forward propagation. Our results show that trivial application of the Projected-FG introduces non-vanishing convergence error due to the stochastic noise that the Projected-FG method introduces in the process. This noise results in an non-vanishing variance in the Projected-FG estimated gradient. To address this, we propose a variance reduction approach by aggregating historical Projected-FG directions. We demonstrate rigorously that this approach ensures convergence to the optimal solution for convex functions and to a stationary point for non-convex functions. These convergence properties are validated through a numerical example, showcasing the approach's effectiveness and efficiency.<|reference_end|>
@article{rostami2024projected,
    title={Projected Forward Gradient-Guided Frank-Wolfe Algorithm via Variance
  Reduction},
    author={M. Rostami, S. S. Kia},
    journal={arXiv preprint arXiv:2403.12511},
    year={2024},
    archivePrefix={arXiv},
    eprint={2403.12511},
    primaryClass={cs.LG math.OC}
}