How Does Gradient Descent Learn Features -- A Local Analysis for Regularized Two-Layer Neural Networks
<|reference_start|>How Does Gradient Descent Learn Features -- A Local Analysis for Regularized Two-Layer Neural Networks: The ability of learning useful features is one of the major advantages of neural networks. Although recent works show that neural network can operate in a neural tangent kernel (NTK) regime that does not allow feature learning, many works also demonstrate the potential for neural networks to go beyond NTK regime and perform feature learning. Recently, a line of work highlighted the feature learning capabilities of the early stages of gradient-based training. In this paper we consider another mechanism for feature learning via gradient descent through a local convergence analysis. We show that once the loss is below a certain threshold, gradient descent with a carefully regularized objective will capture ground-truth directions. Our results demonstrate that feature learning not only happens at the initial gradient steps, but can also occur towards the end of training.<|reference_end|>
@article{zhou2024how,
    title={How Does Gradient Descent Learn Features -- A Local Analysis for
  Regularized Two-Layer Neural Networks},
    author={Mo Zhou, Rong Ge},
    journal={arXiv preprint arXiv:2406.01766},
    year={2024},
    archivePrefix={arXiv},
    eprint={2406.01766},
    primaryClass={cs.LG stat.ML}
}