Deep Neural Networks with Box Convolutions
Box filters computed using integral images have been part of the computer vision toolset for a long time. Here, we show that a convolutional layer that computes box filter responses in a sliding manner can be used within deep architectures, whereas the dimensions and the offsets of the sliding boxes in such a layer can be learned as a part of an end-to-end loss minimization. Crucially, the training process can make the size of the boxes in such a layer arbitrarily large without incurring extra computational cost and without the need to increase the number of learnable parameters. Due to its ability to integrate information over large boxes, the new layer facilitates long-range propagation of information and leads to the efficient increase of the receptive fields of network units. By incorporating the new layer into existing architectures for semantic segmentation, we are able to achieve both the increase in segmentation accuracy as well as the decrease in the computational cost and the number of learnable parameters.
This paper present a method to compute convolutions with filters of any size with a fixed computational cost by using the integral image trick as in Viola-Jones detector [30]. This method is used to create a box convolution layer, that is a layer in which every filter is a set of boxes (one per channel) whose position and size are learnable parameters. This layer is used in a new block (Fig. 1) that is then used to replace standard dilated convolution blocks in ENet and ERFNet architectures. Due to the larger receptive field and fewer parameters of the proposed layer, the modified architectures obtain better performance than the original ones.  Pros: - The idea of using integral image for convolutions and learn the filter size and position is new in my knowledge and seems to make sense when we want to read a broad proportion of the image without relying on several convutional layers. - The reported experiments seem to show that for the specific problem this box convolution makes sense and performs better than other approaches.  Cons: - It seems that the authors did not mention (or not fully explain) certain important aspects of the approach such as:  i) the computational cost of reading a value is 4 times more than what it would be in a normal convolution due to the integral image trick (plus the time needed to pre-compute the integrals)  ii) the computational time of the full approach is then reduced because instead of using 3x3 filters, they read just average boxes and they reduce the number for channels by a factor 4 with 1x1 convolution  iii) at test time, when the computational cost is actually evaluated, the authors avoid to compute the float value of the boxes (which is needed in training for learning the parameters); thus training is expected to be computationally more expensive. - I would have liked to see the idea implemented for a full comparison with a learnable convolutional filter, i.e. instead of single boxes, learn 3x3 learnable size filters and use them as a basic convolutional block. Of course, this would incur in a much higher computational cost (compared to the standard 3x3 filter), but probably also much better results. - In the experiments the places in which some blocks have been substituted by the proposed block seem a bit ad-hoc and one might consider that it is optimized to obtain the best performance gap.   Additional comments: - l.179: it is not clear to me why the 1x1 convolution has a complexity of O(N^2) while the box convolution of only O(N). In my understanding the box convolution has N/4 inputs and N outputs, thus O(N^2) operations per spatial location. - The block in Fig.1 in my understanding is composed of a 1x1 convolution and the box convolution, which can be also considered 1x1 in the sense that there is a box for each channel (in my understanding). Thus, the complete block lack of spatial evaluations within channel (in the sense of 3x3 convolutions that can evaluate nearby pixels in the same channel). This might not be a problem for the given task, but it might be problematic for more difficult problems. This problem can also be compensated by the fact that the position of the box can be learned. Authors should consider to present this kind of reasoning about the approach and propose some experiments to better understand it.  - The symmetry of the boxes in Fig. 2 can be due to the use of the horizontal flip of the training images for data augmentation?   After rebuttal: I read the authors rebuttal and I think they managed to give an explanation to most of my concerns. In this sense I think the paper deserves publication. Still, I think it should be important to clarify that using the integral image trick makes the computation of the convolution much more expensive, but allows any size filter with the same computational cost. In this paper the proposed models are computationally cheaper because the proposed layers (based on integral image) are reduced convolutions, with spatial filters of WxHxC of 1x1x1.