A One-Layer Decoder-Only Transformer is a Two-Layer RNN: With an Application to Certified Robustness
<|reference_start|>A One-Layer Decoder-Only Transformer is a Two-Layer RNN: With an Application to Certified Robustness: This paper reveals a key insight that a one-layer decoder-only Transformer is equivalent to a two-layer Recurrent Neural Network (RNN). Building on this insight, we propose ARC-Tran, a novel approach for verifying the robustness of decoder-only Transformers against arbitrary perturbation spaces. Compared to ARC-Tran, current robustness verification techniques are limited either to specific and length-preserving perturbations like word substitutions or to recursive models like LSTMs. ARC-Tran addresses these limitations by meticulously managing position encoding to prevent mismatches and by utilizing our key insight to achieve precise and scalable verification. Our evaluation shows that ARC-Tran (1) trains models more robust to arbitrary perturbation spaces than those produced by existing techniques and (2) shows high certification accuracy of the resulting models.<|reference_end|>
@article{zhang2024a,
    title={A One-Layer Decoder-Only Transformer is a Two-Layer RNN: With an
  Application to Certified Robustness},
    author={Yuhao Zhang and Aws Albarghouthi and Loris D'Antoni},
    journal={arXiv preprint arXiv:2405.17361},
    year={2024},
    archivePrefix={arXiv},
    eprint={2405.17361},
    primaryClass={cs.CL}
}