  Multispectral pedestrian detection has shown great advantages under poor
illumination conditions, since the thermal modality provides complementary
information for the color image. However, real multispectral data suffers from
the position shift problem, i.e. the color-thermal image pairs are not strictly
aligned, making one object has different positions in different modalities. In
deep learning based methods, this problem makes it difficult to fuse the
feature maps from both modalities and puzzles the CNN training. In this paper,
we propose a novel Aligned Region CNN (AR-CNN) to handle the weakly aligned
multispectral data in an end-to-end way. Firstly, we design a Region Feature
Alignment (RFA) module to capture the position shift and adaptively align the
region features of the two modalities. Secondly, we present a new multimodal
fusion method, which performs feature re-weighting to select more reliable
features and suppress the useless ones. Besides, we propose a novel RoI jitter
strategy to improve the robustness to unexpected shift patterns of different
devices and system settings. Finally, since our method depends on a new kind of
labelling: bounding boxes that match each modality, we manually relabel the
KAIST dataset by locating bounding boxes in both modalities and building their
relationships, providing a new KAIST-Paired Annotation. Extensive experimental
validations on existing datasets are performed, demonstrating the effectiveness
and robustness of the proposed method. Code and data are available at
https://github.com/luzhang16/AR-CNN.

Computer Vision and Pattern Recognition