PIDformer: Transformer Meets Control Theory
<|reference_start|>PIDformer: Transformer Meets Control Theory: In this work, we address two main shortcomings of transformer architectures: input corruption and rank collapse in their output representation. We unveil self-attention as an autonomous state-space model that inherently promotes smoothness in its solutions, leading to lower-rank outputs and diminished representation capacity. Moreover, the steady-state solution of the model is sensitive to input perturbations. We incorporate a Proportional-Integral-Derivative (PID) closed-loop feedback control system with a reference point into the model to improve robustness and representation capacity. This integration aims to preserve high-frequency details while bolstering model stability, rendering it more noise-resilient. The resulting controlled state-space model is theoretically proven robust and adept at addressing the rank collapse. Motivated by this control framework, we derive a novel class of transformers, PID-controlled Transformer (PIDformer), aimed at improving robustness and mitigating the rank-collapse issue inherent in softmax transformers. We empirically evaluate the model for advantages and robustness against baseline transformers across various practical tasks, including object classification, image segmentation, and language modeling.<|reference_end|>
@article{nguyen2024pidformer:,
    title={PIDformer: Transformer Meets Control Theory},
    author={Tam Nguyen, C'esar A. Uribe, Tan M. Nguyen, Richard G. Baraniuk},
    journal={arXiv preprint arXiv:2402.15989},
    year={2024},
    archivePrefix={arXiv},
    eprint={2402.15989},
    primaryClass={cs.AI cs.SY eess.SY}
}