Deep neural networks and decision trees operate on largely separate paradigms; typically, the former performs representation learning with pre-specified architectures, while the latter is characterised by learning hierarchies over pre-specified features with data-driven architectures. We unite the two via adaptive neural trees (ANTs), a model that incorporates representation learning into edges, routing functions and leaf nodes of a decision tree, along with a backpropagation-based training algorithm that adaptively grows the architecture from primitive modules (e.g., convolutional layers). ANTs allow increased interpretability via hierarchical clustering, e.g., learning meaningful class associations, such as separating natural vs. man-made objects. We demonstrate this on classification and regression tasks, achieving over 99% and 90% accuracy on the MNIST and CIFAR-10 datasets, and outperforming standard neural networks, random forests and gradient boosted trees on the SARCOS dataset. Furthermore, ANT optimisation naturally adapts the architecture to the size and complexity of the training data.
The paper is written rather well, however I find the experiments incomplete and have some reservations about the
method. My main points of critique are:
1.  Combining DT & NN 
I have doubts that the way you combine DT &NN  you get the "Best of both world". In some ways your architecture also 
shares disadvantages of both:
1.1 Interpretability
Because each node in the tree can a neural network (with arbritrary complexity), this approach looses one central advantage of DT, that is the interpretability of the result.    Each node in the tree can perform arbritrary complex (and hierarchical) 
computations. The authors only show one particular example (Fig. 2a), where the model has learned is a reasonable 
structure.
1.2 Complexity:
The whole architecture is much more complex than either a neural network or a decision tree. I expect that therefore training these is not easy, and expert knowledge in either DT  or NN may not be enough to use this model.
2. Limited experiments
2.1 The authors only consider 2 experiments from vision (MNIST & CIFAR 10) while proposing a universal method.  To show universality the authors should use data sets from different domains (e.g UCI data sets)
2.2 The authors argue that a  strength of the method   is  that it uses a low number of parameters on average for a forward path (compared to the total parameter size).  I don't find this argument to be convincing. In the limit this would imply a high memorization of the  data.  Also, a similar case can be made for standard CNN, when a particular filter is mostly inactive for some data points.
2.3 The interpretability of DT compared to NN I mentioned earlier.  To make the argument that their method learns the
hierarchical structure of the data , the authors should have added experiments to support this, where  such a hierarchical structure is clearly present and can be evaluated empirically.
--
In light of the extended experiments w.r.t. to 2.1 I increased my score from 5 to 6.  Overall, I still have doubts about the interpretability and complexity of the proposed method.  
Complexity:  "but all the intuitions needed would come solely from training NN".    I disagree with this response.   The architecture is a mix between a tree (hard, decision-tree like error surface,  non-local) and neural network (smooth, mostly convex error surface). This also implies that the training process and its behavior will possess patterns and challenges of both approaches. 
Interpretability:  I think the method misses "priors" that enforce credit assignment.  Partitioning the problem in subp-roblems should be done via the tree components, whereas processing (such as image filtering) should be done in the network nodes. However,  the method does not enforce, or encourage this behavior, for instance
via constraints:   also nodes can do partitioning (because neural networks can approximate decision trees)  and edges can do processing (e.g. decisions-trees can be used for mnist).
So I still believe this to be a borderline paper, however, the experiments support a more general applicability.