Deep Symmetry Networks
The chief difficulty in object recognition is that objects' classes are obscured by a large number of extraneous sources of variability, such as pose and part deformation. These sources of variation can be represented by symmetry groups, sets of composable transformations that preserve object identity. Convolutional neural networks (convnets) achieve a degree of translational invariance by computing feature maps over the translation group, but cannot handle other groups. As a result, these groups' effects have to be approximated by small translations, which often requires augmenting datasets and leads to high sample complexity. In this paper, we introduce deep symmetry networks (symnets), a generalization of convnets that forms feature maps over arbitrary symmetry groups. Symnets use kernel-based interpolation to tractably tie parameters and pool over symmetry spaces of any dimension. Like convnets, they are trained with backpropagation. The composition of feature transformations through the layers of a symnet provides a new approach to deep learning. Experiments on NORB and MNIST-rot show that symnets over the affine group greatly reduce sample complexity relative to convnets by better capturing the symmetries in the data.
This paper proposes to extend usual convolutional networks from thetranslation symmetry group to the affine group (symnets). This entailscomputation that may grow as k^6 instead of k^2 when each feature iscomputed on a window of width k in each of the group dimensions.  To dealwith this computational complexity, the paper proposes to optimize thelocation (in R^6) of N control points located in the space of the group,for each feature map.  The output of the convolution+subsampling can thenbe approximated by a form of weighted averaging between the control points,and the whole thing can still be optimized by backprop + gradient-basedoptimization.This is a really interesting idea and it is also new (the part about usingoptimized control points, not the part about extending convnets totransformations other than translation) as well as attacking an age-oldproblem, but the weak point of the paper are the experiments, along withthe selling tone used in many places. The results suggest on the surface that symnets can generalize better onvery small datasets (a few hundred examples), but the appropriate controlexperiments are not performed, and both the conclusions (and generally thelanguage of the paper) seem not in line with the standards of scientificreporting, in an attempt to make the proposal and the results look better.The experiments should have compared with a convnet that uses the sameprior information, i.e., with an augmented dataset, randomly transformingalong the affine group (which is the standard practice anyways). Theexperiments should have compared networks of arbitrary depth and size(optimized on the validation set), and not be restricted to networks of thesame size and depth, because it is likely that in order to emulate a 6-Dsymnet, a regular convnet needs more resources (both in depth and number offeatures and parameters), but these can be compensated by using anaugmented training set. Finally, a fair comparison should includecomputation time (especially for training). One might suspect that theproposed algorithm (which is not 100% clear, although the big picture isclear) requires a lot more computation per training example than a regularconvnet. The reader needs to know how much more, in the specificexperiments reported. A number of elements of that paper are not acceptable in the scientificreporting standards: hiding the above computational cost information (ofwhich surely the authors are aware - not a single word is spoken abouttraining and test time), not doing the above proper and fair comparisons,and then claiming that symnets generalize better without stating that theadvantage is mostly for tiny datasets with a few hundred examples. It isunfortunate that many authors have a similar behavior, but still, it shouldnot be acceptable at NIPS.Additional detailed pointsLine 39: The authors write that convnets require enormous datasets. But thereare such datasets, so the question is whether one can do better with the givendatasets. It is true that we would like to generalize with few examples fornew classes for which there are not such large number of labeled examples.The use of the word 'kernel' initially threw me off. Of course it is used herein a sense different from the more common use in the ML community. I suggestto clarify that early on.Line 69: "we do not completely pool out a symmetry", but neither do convnets.Similarly for discriminative training, same line. Clarify that you refer tosome other approach, maybe.Line 241: "polynomial dependence on D", actually should be exponential in D,and polynomial in k, i.e. simply k^D.Line 290: why use the minimization of squared error rather than maximizationof the dot product? Is the norm of the feature vectors kept constant? how? Line 303: "penalizing relative locations [10]". Not having read [10], the readerwould like to know briefly what this is about.Figures 5, 6: it is crazy to compare only with a single-layer or a 2-layer convnet.See main discussion above.Section 6.2: what was N, at the end of the day?Line 427: I found this conclusion really misleading "affine symnets achievedhigher accuracy than convnets while using significantly less data". The truthis that when *only* a few hundred examples are available, and when the convnetcannot exploit transformed augmented datasets and is highly limited in capacity(all of which are not realistic), then the symnets generalize better. Clearlythis is a very different kind of message...Line 431: what about the other hyper-parameters?=========== POST-REBUTTAL NOTESThe authors have argued that "data augmentation is not feasible for many datasets" to justify the fact that they did not want to compare symmetry nets with convnets + data augmentation. The truth is that data augmentation has been for many years the standard way of training convnets in practice, it is cheap, simple, and effective, and helps to embody the same kind of prior information that symmetry nets try to exploit. Data augmentation does not need to visit the cross-product of all possible deformations: that's the beauty of it. A random sample of deformations along all the desired directions of invariance is sufficient to give a huge boost to convnets ESPECIALLY ON SMALL DATASETS like the ones the authors used.  If symmnets can exploit that same prior information better than random deformations, then this is a very important piece of information! The authors have also refused to change their misleading conclusion. That is sad. Most careful readers will not be fooled but will instead come out with a bad impression about the authors.Nonetheless, I have raised my rating above the acceptance threshold because I believe I like the underlying idea, which is original and should be a good contribution to NIPS.I strongly encourage the authors to reconsider the above two points in order to make their paper a more credible scientific contribution that will be more likely to have impact in the field.