Sparse and Structured Hopfield Networks
<|reference_start|>Sparse and Structured Hopfield Networks: Modern Hopfield networks have enjoyed recent interest due to their connection to attention in transformers. Our paper provides a unified framework for sparse Hopfield networks by establishing a link with Fenchel-Young losses. The result is a new family of Hopfield-Fenchel-Young energies whose update rules are end-to-end differentiable sparse transformations. We reveal a connection between loss margins, sparsity, and exact memory retrieval. We further extend this framework to structured Hopfield networks via the SparseMAP transformation, which can retrieve pattern associations instead of a single pattern. Experiments on multiple instance learning and text rationalization demonstrate the usefulness of our approach.<|reference_end|>
@article{santos2024sparse,
    title={Sparse and Structured Hopfield Networks},
    author={Saul Santos, Vlad Niculae, Daniel McNamee, Andre F. T. Martins},
    journal={arXiv preprint arXiv:2402.13725},
    year={2024},
    archivePrefix={arXiv},
    eprint={2402.13725},
    primaryClass={cs.LG}
}