Statistical signatures of abstraction in deep neural networks
<|reference_start|>Statistical signatures of abstraction in deep neural networks: We study how abstract representations emerge in a Deep Belief Network (DBN) trained on benchmark datasets. Our analysis targets the principles of learning in the early stages of information processing, starting from the "primordial soup" of the under-sampling regime. As the data is processed by deeper and deeper layers, features are detected and removed, transferring more and more "context-invariant" information to deeper layers. We show that the representation approaches an universal model -- the Hierarchical Feature Model (HFM) -- determined by the principle of maximal relevance. Relevance quantifies the uncertainty on the model of the data, thus suggesting that "meaning" -- i.e. syntactic information -- is that part of the data which is not yet captured by a model. Our analysis shows that shallow layers are well described by pairwise Ising models, which provide a representation of the data in terms of generic, low order features. We also show that plasticity increases with depth, in a similar way as it does in the brain. These findings suggest that DBNs are capable of extracting a hierarchy of features from the data which is consistent with the principle of maximal relevance.<|reference_end|>
@article{caputo2024statistical,
    title={Statistical signatures of abstraction in deep neural networks},
    author={Carlo Orientale Caputo and Matteo Marsili},
    journal={arXiv preprint arXiv:2407.01656},
    year={2024},
    archivePrefix={arXiv},
    eprint={2407.01656},
    primaryClass={cs.LG cond-mat.dis-nn physics.data-an stat.ML}
}