Published as a conference paper at ICLR 2018 MATRIX CAPSULES WITH EM ROUTING
A capsule is a group of neurons whose outputs represent different properties of the same entity. Each layer in a capsule network contains many capsules. We describe a version of capsules in which each capsule has a logistic unit to represent the presence of an entity and a 4x4 matrix which could learn to represent the relationship between that entity and the viewer (the pose). A capsule in one layer votes for the pose matrix of many different capsules in the layer above by multiplying its own pose matrix by trainable viewpoint-invariant transformation matrices that could learn to represent part-whole relationships. Each of these votes is weighted by an assignment coefficient. These coefficients are iteratively updated for each image using the Expectation-Maximization algorithm such that the output of each capsule is routed to a capsule in the layer above that receives a cluster of similar votes. The transformation matrices are trained discriminatively by backpropagating through the unrolled iterations of EM between each pair of adjacent capsule layers. On the smallNORB benchmark, capsules reduce the number of test errors by 45% compared to the state-of-the-art. Capsules also show far more resistance to white box adversarial attacks than our baseline convolutional neural network.
The paper proposes a novel architecture for capsule networks. Each capsule has a logistic unit representing the presence of an entity plus a 4x4 pose matrix representing the entity/viewer relationship. This new representation comes with a novel iterative routing scheme, based on the EM algorithm.
Evaluated on the SmallNORB dataset, the approach proves to be more accurate than previous work (beating also the recently proposed "routing-by-agreement" approach for capsule networks by Sabour et al.). It also generalizes well to new, unseen viewpoints and proves to be more robust to adversarial examples than traditional CNNs.
Capsule networks have recently gained attention from the community. The paper addresses important shortcomings exhibited by previous work (Sabour et al.), introducing a series of valuable technical novelties.
There are, however, some weaknesses. The proposed routing scheme is quite complex (involving an EM-based step at each layer); it's not fully clear how efficiently it can be performed / how scalable it is. Evaluation is performed on a small dataset for shape recognition; as noted in Sec. 6, the approach will need to be tested on larger, more challenging datasets. Clarity could be improved in some parts of the paper (e.g.: Sec. 1.1 may not be fully clear if the reader is not already familiar with (Sabour et al., 2017); the authors could give a better intuition about what is kept and what is discarded, and why, from that approach. Sec. 2: the sentence "this is incorrect because the transformation matrix..." could be elaborated more. V_{ih} in eq. 1 is defined only a few lines below; perhaps, defining the variables before the equations could improve clarity. Sec. 2.1 could be accompanied by mathematical formulation).
All in all, the paper brings an original contribution and will encourage further research / discussion on an important research question (how to effectively leverage knowledge about the part-whole relationships).
Other notes:
- There are a few typos (e.g. Sec. 1.2 "(Jaderberg et al. (2015)",  Sec. 2 "the the transformation", Sec. 4 "cetral crop" etc.).
- The authors could discuss in more detail why the approach does not show significant improvement on NORB with respect to the state of the art.
- The authors could provide more insights about why capsule gradients are smaller than CNN ones.
- It would be interesting to discuss how the network could potentially be adapted, in the future, to: 1. be more efficient 2. take into account other changes produced by viewpoint changes (pixel intensities, as noted in Sec. 1).
- In Sec, 4, the authors could provide more details about the network training.
- In Procedure 1, for indexing tensors and matrices it might be better to use a comma to separate dimensions (e.g. V_{:,c,:} instead of V_{:c:}).