A spring-block theory of feature learning in deep neural networks
<|reference_start|>A spring-block theory of feature learning in deep neural networks: A central question in deep learning is how deep neural networks (DNNs) learn features. DNN layers progressively collapse data into a regular low-dimensional geometry. This collective effect of non-linearity, noise, learning rate, width, depth, and numerous other parameters, has eluded first-principles theories which are built from microscopic neuronal dynamics. Here we present a noise-non-linearity phase diagram that highlights where shallow or deep layers learn features more effectively. We then propose a macroscopic mechanical theory of feature learning that accurately reproduces this phase diagram, offering a clear intuition for why and how some DNNs are ``lazy'' and some are ``active'', and relating the distribution of feature learning over layers with test accuracy.<|reference_end|>
@article{shi2024a,
    title={A spring-block theory of feature learning in deep neural networks},
    author={Cheng Shi, Liming Pan and Ivan Dokmani'c},
    journal={arXiv preprint arXiv:2407.19353},
    year={2024},
    archivePrefix={arXiv},
    eprint={2407.19353},
    primaryClass={cond-mat.dis-nn cond-mat.stat-mech cs.LG stat.ML}
}