  Although transformer has achieved great progress on computer vision tasks,
the scale variation in dense image prediction is still the key challenge. Few
effective multi-scale techniques are applied in transformer and there are two
main limitations in the current methods. On one hand, self-attention module in
vanilla transformer fails to sufficiently exploit the diversity of semantic
information because of its rigid mechanism. On the other hand, it is hard to
build attention and interaction among different levels due to the heavy
computational burden. To alleviate this problem, we first revisit multi-scale
problem in dense prediction, verifying the significance of diverse semantic
representation and multi-scale interaction, and exploring the adaptation of
transformer to pyramidal structure. Inspired by these findings, we propose a
novel Semantic-aware Decoupled Transformer Pyramid (SDTP) for dense image
prediction, consisting of Intra-level Semantic Promotion (ISP), Cross-level
Decoupled Interaction (CDI) and Attention Refinement Function (ARF). ISP
explores the semantic diversity in different receptive space. CDI builds the
global attention and interaction among different levels in decoupled space
which also solves the problem of heavy computation. Besides, ARF is further
added to refine the attention in transformer. Experimental results demonstrate
the validity and generality of the proposed method, which outperforms the
state-of-the-art by a significant margin in dense image prediction tasks.
Furthermore, the proposed components are all plug-and-play, which can be
embedded in other methods.

Computer Vision and Pattern Recognition