  In this paper, we seek reasons for the two major failure cases in Semantic
Segmentation (SS): 1) missing small objects or minor object parts, and 2)
mislabeling minor parts of large objects as wrong classes. We have an
interesting finding that Failure-1 is due to the underuse of detailed features
and Failure-2 is due to the underuse of visual contexts. To help the model
learn a better trade-off, we introduce several Self-Regulation (SR) losses for
training SS neural networks. By "self", we mean that the losses are from the
model per se without using any additional data or supervision. By applying the
SR losses, the deep layer features are regulated by the shallow ones to
preserve more details; meanwhile, shallow layer classification logits are
regulated by the deep ones to capture more semantics. We conduct extensive
experiments on both weakly and fully supervised SS tasks, and the results show
that our approach consistently surpasses the baselines. We also validate that
SR losses are easy to implement in various state-of-the-art SS models, e.g.,
SPGNet and OCRNet, incurring little computational overhead during training and
none for testing.

Computer Vision and Pattern Recognition