  Image enhancement from degradation of rainy artifacts plays a critical role
in outdoor visual computing systems. In this paper, we tackle the notion of
scale that deals with visual changes in appearance of rain steaks with respect
to the camera. Specifically, we revisit multi-scale representation by
scale-space theory, and propose to represent the multi-scale correlation in
convolutional feature domain, which is more compact and robust than that in
pixel domain. Moreover, to improve the modeling ability of the network, we do
not treat the extracted multi-scale features equally, but design a novel
scale-space invariant attention mechanism to help the network focus on parts of
the features. In this way, we summarize the most activated presence of feature
maps as the salient features. Extensive experiments results on synthetic and
real rainy scenes demonstrate the superior performance of our scheme over the
state-of-the-arts.

Computer Vision and Pattern Recognition