Neural Message Passing Induced by Energy-Constrained Diffusion
<|reference_start|>Neural Message Passing Induced by Energy-Constrained Diffusion: Learning representations for structured data with certain geometries (observed or unobserved) is a fundamental challenge, wherein message passing neural networks (MPNNs) have become a de facto class of model solutions. In this paper, we propose an energy-constrained diffusion model as a principled interpretable framework for understanding the mechanism of MPNNs and navigating novel architectural designs. The model, inspired by physical systems, combines the inductive bias of diffusion on manifolds with layer-wise constraints of energy minimization. As shown by our analysis, the diffusion operators have a one-to-one correspondence with the energy functions implicitly descended by the diffusion process, and the finite-difference iteration for solving the energy-constrained diffusion system induces the propagation layers of various types of MPNNs operated on observed or latent structures. On top of these findings, we devise a new class of neural message passing models, dubbed as diffusion-inspired Transformers, whose global attention layers are induced by the principled energy-constrained diffusion. Across diverse datasets ranging from real-world networks to images and physical particles, we show that the new model can yield promising performance for cases where the data structures are observed (as a graph), partially observed or completely unobserved.<|reference_end|>
@article{wu2024neural,
    title={Neural Message Passing Induced by Energy-Constrained Diffusion},
    author={Qitian Wu, David Wipf, Junchi Yan},
    journal={arXiv preprint arXiv:2409.09111},
    year={2024},
    archivePrefix={arXiv},
    eprint={2409.09111},
    primaryClass={cs.LG cs.AI}
}