I think there are a few additional important details.<p>It&#x27;s not so much that they stack a random forest on top of a neural network, as that they are trying to train neural networks that have tree-like properties. Deep neural networks can be expensive to evaluate because every layer is needed to compute the resulting label. On the other hand, in a decision tree you only need to evaluate one of a nodes children (i.e., evaluation is linear in the depth of the tree).<p>They combine these ideas by allowing some nodes in the network to branch (so for a given image, you choose one of two neural networks to evaluate). These nodes don&#x27;t have to occur at the top level, but can occur in the middle of the network as well.<p>The difficulty is how you train these nodes; that&#x27;s where the differential technique for optimizing trees come in. Because you have a unified training algorithm, you can mix and match tree-like nodes and neural network like nodes and train the whole thing.<p>All in all, the hope is that getting a label for a given node is much faster, so you can run these on smaller devices such as phones without sacrificing accuracy.