DIVERSITY AND DEPTH IN PER-EXAMPLE ROUTING MODELS
Routing models, a form of conditional computation where examples are routed through a subset of components in a larger network, have shown promising results in recent works. Surprisingly, routing models to date have lacked important properties, such as architectural diversity and large numbers of routing decisions. Both architectural diversity and routing depth can increase the representational power of a routing network. In this work, we address both of these deficiencies. We discuss the significance of architectural diversity in routing models, and explain the tradeoffs between capacity and optimization when increasing routing depth. In our experiments, we find that adding architectural diversity to routing models significantly improves performance, cutting the error rates of a strong baseline by 35% on an Omniglot setup. However, when scaling up routing depth, we find that modern routing techniques struggle with optimization. We conclude by discussing both the positive and negative results, and suggest directions for future research.
The paper "Diversity and Depth in Per-Example Routing Models" extends previous work on routing networks by adding diversity to the type of architectural unit available for the router at each decision and by scaling to deeper networks. They evaluate their approach on Omniglot, where they achieve state of the art performance.
Overall, the paper is very well written and every aspect can be easily understood. The overview over related work given in the paper is thorough, and the authors explain very well how their approach relates to previous approaches.
The architecture presented is a natural and important extension of previous work. Adding diversity in routing units has indeed not been investigated well and is an important contribution to the community. Additionally, the authors do a good job of identifying problems with existing approaches (overfitting, routing depth) and offer a empirically convincing solutions. 
The result section given in the paper is its weakness and requires a more in-depth analysis:
+ the results given for Omniglot are impressive
+ the experiments analyzing the impact of diversity and routing depth are interesting and offer interesting insight into the architecture
- the results do not show learning behavior over epochs; this is not necessary, but would give an additional insight into the learning behavior of the architecture
- the experimental settings are confusing: why are the different experiments performed with different datasets? This makes it seem as if the authors cherry-picked the best results for the different experiments (this might not be the case, but the results on Omniglot alone are good enough that negative results and a detailed discussion of them would not have hurt the paper, but enriched the discourse)
- additional experiments that offer a transition from larger datasets to smaller ones would be interesting; seeing how the performance of the architecture behaves e.g. on CIFAR10 for 1k, 5k, 10k, 25k and 50k would have illustrated how well the architecture is able to generalize from different numbers of samples
In summary, I think the paper analyzes a very important problem and has a lot of potential. However, it needs more extensive experiments that illustrate how the proposed architecture behaves over a wider variety of datasets.
UPDATE AFTER REBUTTAL:
I am still torn about this paper. On one hand, I still think that the topic and discourse provided by this paper is extremely important. On the other, the results - even after the revision - do not completely convince me. I might update my score after some discussion with the other reviewers.
2ND UPDATE:
After giving it some more thought, I find myself convinced that this paper has a contribution important enough to be accepted. I increase my score to 7.