Learning with SASQuaTCh: a Novel Variational Quantum Transformer Architecture with Kernel-Based Self-Attention
<|reference_start|>Learning with SASQuaTCh: a Novel Variational Quantum Transformer Architecture with Kernel-Based Self-Attention: The widely popular transformer network popularized by the generative pre-trained transformer (GPT) has a large field of applicability, including predicting text and images, classification, and even predicting solutions to the dynamics of physical systems. In the latter context, the continuous analog of the self-attention mechanism at the heart of transformer networks has been applied to learning the solutions of partial differential equations and reveals a convolution kernel nature that can be exploited by the Fourier transform. It is well known that many quantum algorithms that have provably demonstrated a speedup over classical algorithms utilize the quantum Fourier transform. In this work, we explore quantum circuits that can efficiently express a self-attention mechanism through the perspective of kernel-based operator learning. In this perspective, we are able to represent deep layers of a vision transformer network using simple gate operations and a set of multi-dimensional quantum Fourier transforms. We analyze the computational and parameter complexity of our novel variational quantum circuit, which we call Self-Attention Sequential Quantum Transformer Channel (SASQuaTCh), and demonstrate its utility on simplified classification problems.<|reference_end|>
@article{evans2024learning,
    title={Learning with SASQuaTCh: a Novel Variational Quantum Transformer
  Architecture with Kernel-Based Self-Attention},
    author={Ethan N. Evans, Matthew Cook, Zachary P. Bradshaw, and Margarite L.
  LaBorde},
    journal={arXiv preprint arXiv:2403.14753},
    year={2024},
    archivePrefix={arXiv},
    eprint={2403.14753},
    primaryClass={quant-ph cs.LG}
}