We propose a novel framework, ICNN, which combines the input-conditioned filter generation module and a decoder based network to incorporate contextual information present in images into Convolutional Neural Networks (CNNs). In contrast to traditional CNNs, we do not employ the same set of learned convolution filters for all input image instances. And our proposed decoder network serves the purpose of reducing the transformation present in the input image by learning to construct a representative image of the input image class. Our proposed joint supervision of input-aware framework when combined with techniques inspired by Multi-instance learning and max-pooling, results in a transformation-invariant neural network. We investigated the performance of our proposed framework on three MNIST variations, which covers both rotation and scaling variance, and achieved 0.98% error on MNIST-rot-12k, 1.12% error on Half-rotated MNIST and 0.68% error on Scaling MNIST, which is significantly better than the state-ofthe-art results. Our proposed model also showcased consistent improvement on the CIFAR dataset. We make use of visualization to further prove the effectiveness of our input-aware convolution filters. Our proposed convolution filter generation framework can also serve as a plugin for any CNN based architecture and enhance its modeling capacity.
*Paper summary*
The authors build an input transformation invariant CNN using the TI-Pooling architecture of (Laptev et al., 2016). They make some modifications to this, namely 1) they replace the convolutional filters, with input-dependent convolutional filters, and 2) they add a decoder network from the final representation, which reconstructs an input transformation rectified image, to encourage the final representation to be fully transformation invariant.
*Paper decision*
I have decided to assign this paper a reject because of two main reasons. 
*Supporting arguments*
One reason is that the base architecture is not novel. This in itself is not a key issue, but I would expect the authors to have done some in depth analysis or experimentation otherwise to compensate for this. I regret, the authors may just not have known that the ideas were already explored in the literature. The second reason is that the work is not well placed in context with prior works. This is both evident in the lack of referenced works (see below for a list) and the lack of sufficient baselines, against which they compare. For instance, if the authors had considered “Learning Steerable Filters for Rotation Equivariant CNNs” by Weiler et al. (2018) they would have known that their MNIST-rot-12k results are not state of the art as they state. In Weiler et al., the authors report 0.714 test set error on MNIST-rot-12k compared to ICNN’s 0.98.
This all said, I think the paper is well-written and very clear. The structure is straightforward and the experiments seem repeatable from the descriptions made. The stated aims of the paper are also clear: to learn input transformation invariant CNNs using input-conditioned filters.
Unfortunately a lot of supporting material and prior work has been missed. I list a lot of them here. 
Works on input-conditioned filters and invariance. These are the most important
-Dynamic Steerable Frame Networks, Jacobsen et al., 2017
-Dynamic Steerable Blocks in Deep Residual Networks., Jacobsen et al., 2017
Works on input-conditioned filters:
-HyperNetworks, Ha et al., 2016
-Dynamic Filter Networks, de Brabandere et al., 2016
Works on invariance:
-Invariance and neural nets, Barnard and Casasent, 1991
-Group Equivariant Convolutional Networks Cohen and Welling (2015)
-Harmonic Networks: Deep Translation and Rotation Equivariance: Worrall et al. (2017)
-Steerable CNNs, Cohen and Welling (2017)
-Spherical CNNs, Cohen et al. (2018)
-CubeNet: Equivariance to 3D Rotation and Translation, Worrall and Brostow (2018)
-Learning steerable filters for rotation equivariant CNNs, Weiler et al. (2018)
-Gauge Equivariant Convolutional Networks and the Icosahedral CNN, Cohen et al. (2019)
*Questions/notes for the authors*
- Please address the missing references
- Are the input-conditioned filters conditional on position in the activations, or are they shared across all spatial locations of the image? This is not clear from the text.
- The image reconstruction reminds me of Transforming Auto-encoders (Hinton et al., 2016) and Interpretable Transformations with Encoder-Decoder Networks (Worrall et al., 2017). How is your setup different?