Mansformer: Efficient Transformer of Mixed Attention for Image Deblurring and Beyond
<|reference_start|>Mansformer: Efficient Transformer of Mixed Attention for Image Deblurring and Beyond: Transformer has made an enormous success in natural language processing and high-level vision over the past few years. However, the complexity of self-attention is quadratic to the image size, which makes it infeasible for high-resolution vision tasks. In this paper, we propose the Mansformer, a Transformer of mixed attention that combines multiple self-attentions, gate, and multi-layer perceptions (MLPs), to explore and employ more possibilities of self-attention. Taking efficiency into account, we design four kinds of self-attention, whose complexities are all linear. By elaborate adjustment of the tensor shapes and dimensions for the dot product, we split the typical self-attention of quadratic complexity into four operations of linear complexity. To adaptively merge these different kinds of self-attention, we take advantage of an architecture similar to Squeeze-and-Excitation Networks. Furthermore, we make it to merge the two-staged Transformer design into one stage by the proposed gated-dconv MLP. Image deblurring is our main target, while extensive quantitative and qualitative evaluations show that this method performs favorably against the state-of-the-art methods far more than simply deblurring. The source codes and trained models will be made available to the public.<|reference_end|>
@article{kuo2024mansformer:,
    title={Mansformer: Efficient Transformer of Mixed Attention for Image
  Deblurring and Beyond},
    author={Pin-Hung Kuo, Jinshan Pan, Shao-Yi Chien, Ming-Hsuan Yang},
    journal={arXiv preprint arXiv:2404.06135},
    year={2024},
    archivePrefix={arXiv},
    eprint={2404.06135},
    primaryClass={cs.CV}
}