A Transformer with Stack Attention
<|reference_start|>A Transformer with Stack Attention: Natural languages are believed to be (mildly) context-sensitive. Despite underpinning remarkably capable large language models, transformers are unable to model many context-free language tasks. In an attempt to address this limitation in the modeling power of transformer-based language models, we propose augmenting them with a differentiable, stack-based attention mechanism. Our stack-based attention mechanism can be incorporated into any transformer-based language model and adds a level of interpretability to the model. We show that the addition of our stack-based attention mechanism enables the transformer to model some, but not all, deterministic context-free languages.<|reference_end|>
@article{li2024a,
    title={A Transformer with Stack Attention},
    author={Jiaoda Li, Jennifer C. White, Mrinmaya Sachan, Ryan Cotterell},
    journal={arXiv preprint arXiv:2405.04515},
    year={2024},
    archivePrefix={arXiv},
    eprint={2405.04515},
    primaryClass={cs.CL}
}