Group Equivariant Capsule Networks
We present group equivariant capsule networks, a framework to introduce guaranteed equivariance and invariance properties to the capsule network idea. Our work can be divided into two contributions. First, we present a generic routing by agreement algorithm defined on elements of a group and prove that equivariance of output pose vectors, as well as invariance of output activations, hold under certain conditions. Second, we connect the resulting equivariant capsule networks with work from the field of group convolutional networks. Through this connection, we provide intuitions of how both methods relate and are able to combine the strengths of both approaches in one deep neural network architecture. The resulting framework allows sparse evaluation of the group convolution operator, provides control over specific equivariance and invariance properties, and can use routing by agreement instead of pooling operations. In addition, it is able to provide interpretable and equivariant representation vectors as output capsules, which disentangle evidence of object existence from its pose.
While previous work on capsules [1,2,3] identifies equivariance of the routing process as a desirable property, these methods do not guarantee equivariance. The goal if this paper is to create capsule networks with built-in equivariance.  The paper contains several interesting ideas, but lacks sufficient theoretical motivation and experimental support for the proposed architectural innovations. Several more or less independent changes are proposed (equivariant routing, pose normalization layers, spatial aggregation, combination with group convolution), and although each addition is motivated by a plausible sounding line of reasoning, ultimately the proposed innovations don't seem innevitable, and no rigorous ablation study is performed.  Although section 2 is very clearly written, I found it hard to understand section 3. It is not clear to me why simply summing the vectors in Fig 2b would be a problem (the result would be equivariant). Figure 2c seems like a very unnatural way to represent the data.   It seems like a lot of complexity in this paper can be removed by working with representations of G instead of G itself. Specifically, if the pose vectors (which is a misnomer if they are group elements) are taken to be vectors in a vector space V in which a (unitary) representation of G is defined, then M and delta can be simply defined as the vector mean and inner product, and the required properties will follow automatically (and this works for any group, not just SO(2)). The issue identified in section 3, where each vector is seen to rotate and move to a new place, can then be understood very simply as in [4] via the notion of induced representation.  I find it strange that the authors write "Since we take expressiveness from the trainable pose transformations to ensure equivariance and invariance, it is not to be expected that the proposed capsule networks using the proposed routing algorithms will achieve the same classification accuracies as their original counterparts" If these networks are not expected to work better, then what is the point? In general we would expect equivariant networks to work better on data with the appropriate invariances, as demonstrated in many previous works. A more likely explanation for why the network is not working well yet is that it has become a bit too complicated, using too many components that deviate far from the existing engineering best practices (i.e. CNNs with relus, batchnorm and skip connections, etc.).  For comparison with related work, results in MNIST-rot would be nice to see. This dataset has only 12k training samples. The reported number of 98.42% (obtained from 60k training samples, if I understand correctly) is not really competitive here (SOTA is 0.7% error [5]). Results on non-MNIST datasets would also be very welcome.  [1] Hinton et al. 2011, Transforming Auto-encoders [2] Sabour et al. 2017, Dynamic Routing Between Capsules [3] Hinton et al. 2018, Matrix Capsules with EM Routing [4] Cohen et al. 2016, Steerable CNNs [5] Weiler et al., 2017, Learning Steerable Filters for Rotation Equivariant CNNs   -------------------- Post author response edit:  Thank you for the clear headed author response. Although there is a lot to like about this paper, I will maintain my score because I think that the current evidence does not support the effectiveness of the method.   I still do not really understand the point about "pose normalization". I look forward to a future version of the paper where this is clarified.  Regarding representations of G, I should have been clear that the concept I had in mind is that of a "linear group representation" (https://en.wikipedia.org/wiki/Group_representation), not a parameterization of G. Indeed, the fact that a Lie group G is a manifold (whose points we cannot easily add / average) was the motivation for mentioning group representations. If instead of viewing poses as elements g in G, you view them as vectors v in V (where V is the vector space of a group representation), then one can average pose vectors v without any difficulty. Several papers in the equivariant networks literature now take this approach.