On the weight dynamics of learning networks
<|reference_start|>On the weight dynamics of learning networks: Neural networks have become a widely adopted tool for tackling a variety of problems in machine learning and artificial intelligence. In this contribution we use the mathematical framework of local stability analysis to gain a deeper understanding of the learning dynamics of feed forward neural networks. Therefore, we derive equations for the tangent operator of the learning dynamics of three-layer networks learning regression tasks. The results are valid for an arbitrary numbers of nodes and arbitrary choices of activation functions. Applying the results to a network learning a regression task, we investigate numerically, how stability indicators relate to the final training-loss. Although the specific results vary with different choices of initial conditions and activation functions, we demonstrate that it is possible to predict the final training loss, by monitoring finite-time Lyapunov exponents or covariant Lyapunov vectors during the training process.<|reference_end|>
@article{sharafi2024on,
    title={On the weight dynamics of learning networks},
    author={Nahal Sharafi, Christoph Martin, and Sarah Hallerberg},
    journal={arXiv preprint arXiv:2405.00743},
    year={2024},
    archivePrefix={arXiv},
    eprint={2405.00743},
    primaryClass={cs.LG nlin.CD}
}