Dynamic Routing Between Capsules
A capsule is a group of neurons whose activity vector represents the instantiation parameters of a specific type of entity such as an object or an object part. We use the length of the activity vector to represent the probability that the entity exists and its orientation to represent the instantiation parameters. Active capsules at one level make predictions, via transformation matrices, for the instantiation parameters of higher-level capsules. When multiple predictions agree, a higher level capsule becomes active. We show that a discrimininatively trained, multi-layer capsule system achieves state-of-the-art performance on MNIST and is considerably better than a convolutional net at recognizing highly overlapping digits. To achieve these results we use an iterative routing-by-agreement mechanism: A lower-level capsule prefers to send its output to higher level capsules whose activity vectors have a big scalar product with the prediction coming from the lower-level capsule.
Overview :this paper introduces a dynamic routing process for connecting layers in a feedforward neural net, as described in Procedure 1 on p 3. The key idea here is that the coupling coeff c_ij between unit i and unit j is computed dynamically (layerwise), taking into account the agreement between the output v_j of unit j, and the prediction from unit i \hat{u}_{{j|i}. This process is iterates between each layer l and l+1, but does not (as far as I can tell) spread further back.
Another innovation used in the paper is a form of nonlinearity as in eq 1 for units which uses the length of the capsule output v_j to encode strength of activity, and the direction of v_j to encode the values of the capsule parameters.
A shallow CapsNet model is trained on MNIST, and obtains very good performance (a check of the MNIST leaderboard shows best performance of 0.23 obtained with a committee of deep conv nets), cf performance in Table 1.
I regard this paper as very interesting, as it has successfully married the capsules idea with conv nets, and makes use of the dynamic routing capabilities. The results on highly overlapping digits (sec 6) are also impressive. This idea could rally take off and be heavily used.
One major question about the paper is the convergence of the iteration in Proc 1. It would be highly desirable to prove that this converges (if it does so). This issue is as far as I can tell not discussed in the paper.
It would also be very nice to see an example of the dynamic routing process in action, especially to see how a unit whose \hat{u}_{j|i} is out of agreement with the others gets suppressed.
This process of assessing agreement of bottom-up contributions is reminiscent of the work in Rao and Ballard (Predictive coding in the visual cortex: a functional interpretation of some extra-classical receptive-field effects, Nature Neurosci. 2(1) 79-87, 1999), esp the example where local bar detectors all predict the same longer bar. One could e.g. look at how activations of 3 units in a line --- compare to one that is disoriented, e.g. |-- in your network.
I would encourage the authors to publish their code (this can be following acceptance). This would greatly facilitate uptake by the community.
Quality: As far as I can tell this is technically sound.
Clarity: The writing is clear but my detailed comments give a number of suggestions to make the presentation less idiosyncratic.
Originality: The idea of capsules is not new, but this paper seems to have achieved a step change in their performance and moved beyond the "transforming autoencoder" framework in which they were trained.
Significance: The level of performance that can be achieved with a relatively shallow network is impressive. Given the interest in object recognition etc it is possible that this dynamic routing idea could really take off.
Other points:
p 1 Need to ref Hinton et al (20111) when you first mention "capsules" in the text.
p2 and Procedure 1:I strongly recommend "squashing" to "squidging" as it is a term that has been used in the NN literature (and sounds more mathematical).
p 2 The constraint that the sum_j c_{ij} = 1 is reminiscent of the "single parent constraint" in Hinton et al (2000). It would be good to discuss this more.
p 3. The 0.5 in eq (4) seems like a hack, although it would look better if you introduced a weighting parameter alpha and set it to 0.5. Also I don't understand why you are not just using softmax loss (for a single digit classifier) on the ||v_j||'s (as they lie in 0,1). (For the multiclass case you can use multiple independent sigmoids.)
p 3 I recommend "PrimaryCapsules" rather than "PrimaryCaps" as Caps sounds like CAPS.
p 5 Lee et al [2016] baseline -- justify why this is an appropriate baseline.
p 5 sec 5.1 Tab. 4 -> Fig. 4
p 6 Figure 4 caption: "Each column" -> "Each row".
p 6 Sec 6 first para -- it is confusing to talk about your performance level before you have introduced the task -- leave this for later.
p 6 sec 6: note a heavily overlapping digits task was used in Hinton et al (2000).
p 8. It would be better to put the related work earlier, e.g. after sec 4.
p 9. The ordering of the references (as they appear in the paper) is unusual and makes them hard to find. Suggest alphabetical or numbered in order of use. These are standard bibtex options.