  Existing multi-scale solutions lead to a risk of just increasing the
receptive field sizes while neglecting small receptive fields. Thus, it is a
challenging problem to effectively construct adaptive neural networks for
recognizing various spatial-scale objects. To tackle this issue, we first
introduce a new attention dimension, i.e., depth, in addition to existing
attention dimensions such as channel, spatial, and branch, and present a novel
selective depth attention network to symmetrically handle multi-scale objects
in various vision tasks. Specifically, the blocks within each stage of a given
neural network, i.e., ResNet, output hierarchical feature maps sharing the same
resolution but with different receptive field sizes. Based on this structural
property, we design a stage-wise building module, namely SDA, which includes a
trunk branch and a SE-like attention branch. The block outputs of the trunk
branch are fused to globally guide their depth attention allocation through the
attention branch. According to the proposed attention mechanism, we can
dynamically select different depth features, which contributes to adaptively
adjusting the receptive field sizes for the variable-sized input objects. In
this way, the cross-block information interaction leads to a long-range
dependency along the depth direction. Compared with other multi-scale
approaches, our SDA method combines multiple receptive fields from previous
blocks into the stage output, thus offering a wider and richer range of
effective receptive fields. Moreover, our method can be served as a pluggable
module to other multi-scale networks as well as attention networks, coined as
SDA-$x$Net. Their combination further extends the range of the effective
receptive fields towards small receptive fields, enabling interpretable neural
networks. Our source code is available at
\url{https://github.com/QingbeiGuo/SDA-xNet.git}.

Computer Vision and Pattern Recognition