Capsules with Inverted Dot-Product Attention Routing
We introduce a new routing algorithm for capsule networks, in which a child capsule is routed to a parent based only on agreement between the parent*s state and the child*s vote. Unlike previously proposed routing algorithms, the parent*s ability to reconstruct the child is not explicitly taken into account to update the routing probabilities. This simplifies the routing procedure and improves performance on benchmark datasets such as CIFAR-10 and CIFAR-100. The new mechanism 1) designs routing via inverted dot-product attention; 2) imposes Layer Normalization as normalization; and 3) replaces sequential iterative routing with concurrent iterative routing. Besides outperforming existing capsule networks, our model performs at-par with a powerful CNN (ResNet-18), using less than 25% of the parameters. On a different task of recognizing digits from overlayed digit images, the proposed capsule model performs favorably against CNNs given the same number of layers and neurons per layer. We believe that our work raises the possibility of applying capsule networks to complex real-world tasks.
This paper presents a new simpler routing mechanism for capsule networks and achieves good performance on real world data sets making use of this new capsule structure along with a restnet backbone. Strong performance on the cifar10 and cifar100 datasets are presented and the network outperforms earlier versions of capsule networks. This new structure also performs well on an augmented MNIST dataset of overlapping digits (similar to the one used by Sabour et al 2017). Overall the paper is well written and presents solid results. The paper also presents a thorough comparison of two earlier versions of capsules which is a worthwhile contribution in its own right. The paper could be improved by clearing up a few ambiguities: - is the learning rate schedule the same for all three models? in figure 4 it looks like the learning rate is decayed at two distinct points for your model, but only one distinct point for both the EM and Dynamic routing models. -*Notably, the prediction becomes random guess when the iteration number increases to 5.* this sentence is a little confusing. Do you mean when the iteration number the performance is equivalent to not random assignments? - This new algorithm requires that the capsules in L+1 have initialized poses with which to compare agreement between the poses in L. This is initial value seems like it may greatly effect the performance of the model. In the paper it is set to 0 and not expanded upon. It would be interesting to see if randomizing this value, or learning a bias for it would effect performance. -unlike the two previous versions of capsules, the inverted dot product capsules show in figure 4 sudden huge decreases in test accuracy while training. These moments seem to be overcome quite quickly and the model ends up outperforming the other two. But it would be worth mentioning this behavior and perhaps attempting to explain it.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Capsules with Inverted Dot-Product Attention Routing
We introduce a new routing algorithm for capsule networks, in which a child capsule is routed to a parent based only on agreement between the parent*s state and the child*s vote. Unlike previously proposed routing algorithms, the parent*s ability to reconstruct the child is not explicitly taken into account to update the routing probabilities. This simplifies the routing procedure and improves performance on benchmark datasets such as CIFAR-10 and CIFAR-100. The new mechanism 1) designs routing via inverted dot-product attention; 2) imposes Layer Normalization as normalization; and 3) replaces sequential iterative routing with concurrent iterative routing. Besides outperforming existing capsule networks, our model performs at-par with a powerful CNN (ResNet-18), using less than 25% of the parameters. On a different task of recognizing digits from overlayed digit images, the proposed capsule model performs favorably against CNNs given the same number of layers and neurons per layer. We believe that our work raises the possibility of applying capsule networks to complex real-world tasks.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



This paper presents a new simpler routing mechanism for capsule networks and achieves good performance on real world data sets making use of this new capsule structure along with a restnet backbone. Strong performance on the cifar10 and cifar100 datasets are presented and the network outperforms earlier versions of capsule networks. This new structure also performs well on an augmented MNIST dataset of overlapping digits (similar to the one used by Sabour et al 2017). Overall the paper is well written and presents solid results. The paper also presents a thorough comparison of two earlier versions of capsules which is a worthwhile contribution in its own right. The paper could be improved by clearing up a few ambiguities: - is the learning rate schedule the same for all three models? in figure 4 it looks like the learning rate is decayed at two distinct points for your model, but only one distinct point for both the EM and Dynamic routing models. -*Notably, the prediction becomes random guess when the iteration number increases to 5.* this sentence is a little confusing. Do you mean when the iteration number the performance is equivalent to not random assignments? - This new algorithm requires that the capsules in L+1 have initialized poses with which to compare agreement between the poses in L. This is initial value seems like it may greatly effect the performance of the model. In the paper it is set to 0 and not expanded upon. It would be interesting to see if randomizing this value, or learning a bias for it would effect performance. -unlike the two previous versions of capsules, the inverted dot product capsules show in figure 4 sudden huge decreases in test accuracy while training. These moments seem to be overcome quite quickly and the model ends up outperforming the other two. But it would be worth mentioning this behavior and perhaps attempting to explain it.
assistant
This paper presents a new simpler routing mechanism for capsule networks and achieves good performance on real world data sets making use of this new capsule structure along with a restnet backbone. Strong performance on the cifar10 and cifar100 datasets are presented and the network outperforms earlier versions of capsule networks. This new structure also performs well on an augmented MNIST dataset of overlapping digits (similar to the one used by Sabour et al 2017). Overall the paper is well written and presents solid results. The paper also presents a thorough comparison of two earlier versions of capsules which is a worthwhile contribution in its own right. The paper could be improved by clearing up a few ambiguities: - is the learning rate schedule the same for all three models? in figure 4 it looks like the learning rate is decayed at two distinct points for your model, but only one distinct point for both the EM and Dynamic routing models. -*Notably, the prediction becomes random guess when the iteration number increases to 5. * this sentence is a little confusing. Do you mean when the iteration number the performance is equivalent to not random assignments? - This new algorithm requires that the capsules in L+1 have initialized poses with which to compare agreement between the poses in L. This is initial value seems like it may greatly effect the performance of the model. In the paper it is set to 0 and not expanded upon. It would be interesting to see if randomizing this value, or learning a bias for it would effect performance. -unlike the two previous versions of capsules, the inverted dot product capsules show in figure 4 sudden huge decreases in test accuracy while training. These moments seem to be overcome quite quickly and the model ends up outperforming the other two. But it would be worth mentioning this behavior and perhaps attempting to explain it.