Unveiling the Hidden Structure of Self-Attention via Kernel Principal Component Analysis
<|reference_start|>Unveiling the Hidden Structure of Self-Attention via Kernel Principal Component Analysis: The remarkable success of transformers in sequence modeling tasks, spanning various applications in natural language processing and computer vision, is attributed to the critical role of self-attention. Similar to the development of most deep learning models, the construction of these attention mechanisms rely on heuristics and experience. In our work, we derive self-attention from kernel principal component analysis (kernel PCA) and show that self-attention projects its query vectors onto the principal component axes of its key matrix in a feature space. We then formulate the exact formula for the value matrix in self-attention, theoretically and empirically demonstrating that this value matrix captures the eigenvectors of the Gram matrix of the key vectors in self-attention. Leveraging our kernel PCA framework, we propose Attention with Robust Principal Components (RPC-Attention), a novel class of robust attention that is resilient to data contamination. We empirically demonstrate the advantages of RPC-Attention over softmax attention on the ImageNet-1K object classification, WikiText-103 language modeling, and ADE20K image segmentation task.<|reference_end|>
@article{teo2024unveiling,
    title={Unveiling the Hidden Structure of Self-Attention via Kernel Principal
  Component Analysis},
    author={Rachel S.Y. Teo, Tan M. Nguyen},
    journal={arXiv preprint arXiv:2406.13762},
    year={2024},
    archivePrefix={arXiv},
    eprint={2406.13762},
    primaryClass={cs.LG cs.AI cs.CL cs.CV stat.ML}
}