  The attention mechanism is one of the most important priori knowledge to
enhance convolutional neural networks. Most attention mechanisms are bound to
the convolutional layer and use local or global contextual information to
recalibrate the input. This is a popular attention strategy design method.
Global contextual information helps the network to consider the overall
distribution, while local contextual information is more general. The
contextual information makes the network pay attention to the mean or maximum
value of a particular receptive field. Different from the most attention
mechanism, this article proposes a novel attention mechanism with the heuristic
difference attention module, HDAM. HDAM's input recalibration is based on the
difference between the local and global contextual information instead of the
mean and maximum values. At the same time, to make different layers have a more
suitable local receptive field size and increase the exibility of the local
receptive field design, we use genetic algorithm to heuristically produce local
receptive fields. First, HDAM extracts the mean value of the global and local
receptive fields as the corresponding contextual information. Then the
difference between the global and local contextual information is calculated.
Finally HDAM uses this difference to recalibrate the input. In addition, we use
the heuristic ability of genetic algorithm to search for the local receptive
field size of each layer. Our experiments on CIFAR-10 and CIFAR-100 show that
HDAM can use fewer parameters than other attention mechanisms to achieve higher
accuracy. We implement HDAM with the Python library, Pytorch, and the code and
models will be publicly available.

Computer Vision and Pattern Recognition