Convolutional neural networks (convnets) have achieved impressive results on recent computer vision benchmarks. While they benefit from multiple layers that encode nonlinear decision boundaries and a degree of translation invariance, training convnets is a lengthy procedure fraught with local optima. Alternatively, a kernel method that incorporates the compositionality and symmetry of convnets could learn similar nonlinear concepts yet with easier training and architecture selection. We propose compositional kernel machines (CKMs), which effectively create an exponential number of virtual training instances by composing transformed sub-regions of the original ones. Despite this, CKM discriminant functions can be computed efficiently using ideas from sum-product networks. The ability to compose virtual instances in this way gives CKMs invariance to translations and other symmetries, and combats the curse of dimensionality. Just as support vector machines (SVMs) provided a compelling alternative to multilayer perceptrons when they were introduced, CKMs could become an attractive approach for object recognition and other vision problems. In this paper we define CKMs, explore their properties, and present promising results on NORB datasets. Experiments show that CKMs can outperform SVMs and be competitive with convnets in a number of dimensions, by learning symmetries and compositional concepts from fewer samples without data augmentation.
1 INTRODUCTION
The depth of state-of-the-art convnets is a double-edged sword: it yields both nonlinearity for sophisticated discrimination and nonconvexity for frustrating optimization. The established training procedure for ILSVRC classification cycles through the million-image training set more than fifty times, requiring substantial stochasticity, data augmentation, and hand-tuned learning rates. On today’s consumer hardware, the process takes several days. However, performance depends heavily on hyperparameters, which include the number and connections of neurons as well as optimization details. Unfortunately, the space of hyperparameters is unbounded, and each configuration of hyperparameters requires the aforementioned training procedure. It is no surprise that large organizations with enough computational power to conduct this search dominate this task.
Yet mastery of object recognition on a static dataset is not enough to propel robotics and internetscale applications with ever-growing instances and categories. Each time the training set is modified, the convnet must be retrained (“fine-tuned”) for optimum performance. If the training set grows linearly with time, the total training computation grows quadratically.
We propose the Compositional Kernel Machine (CKM), a kernel-based visual classifier that has the symmetry and compositionality of convnets but with the training benefits of instance-based learning (IBL). CKMs branch from the original instance-based methods with virtual instances, an exponential set of plausible compositions of training instances. The first steps in this direction are promising compared to IBL and deep methods, and future work will benefit from over fifty years of research into nearest neighbor algorithms, kernel methods, and neural networks.
In this paper we first define CKMs, explore their formal and computational properties, and compare them to existing kernel methods. We then propose a key contribution of this work: a sum-product function (SPF) that efficiently sums over an exponential number of virtual instances. We then de-
scribe how to train the CKM with and without parameter optimization. Finally, we present results on NORB and variants that show a CKM trained on a CPU can be competitive with convnets trained for much longer on a GPU and can outperform them on tests of composition and symmetry, as well as markedly improving over previous IBL methods.
2 COMPOSITIONAL KERNEL MACHINES
The key issue in using an instance-based learner on large images is the curse of dimensionality. Even millions of training images are not enough to construct a meaningful neighborhood for a 256×256 pixel image. The compositional kernel machine (CKM) addresses this issue by constructing an exponential number of virtual instances. The core hypothesis is that a variation of the visual world can be understood as a rearrangement of low-dimensional pieces that have been seen before. For example, an image of a house could be recognized by matching many pieces from other images of houses from different viewpoints. The virtual instances represent this set of all possible transformations and recombinations of the training images. The arrangement of these pieces cannot be arbitrary, so CKMs learn how to compose virtual instances with weights on compositions. A major contribution of this work is the ability to efficiently sum over this set with a sum-product function.
The set of virtual instances is related to the nonlinear image manifolds described by Simard et al. (1992) but with key differences. Whereas the tangent distance accounts for transformations applied to the whole image, virtual instances can depict local transformations that are applied differently across an image. Secondly, the tangent plane approximation of the image manifold is only accurate near the training images. Virtual instances can easily represent distant transformations. Unlike the explicit augmentation of virtual support vectors in Schölkopf et al. (1996), the set of virtual instances in a CKM is implicit and exponentially larger. Platt & Allen (1996) demonstrated an early version of virtual instances to expand the set of negative examples for a linear classifier.
2.1 DEFINITION
We define CKMs using notation common to other IBL techniques. The two prototypical instancebased learners are k-nearest neighbors and support vector machines. The foundation for both algorithms is a similarity or kernel function K(x, x′) between two instances. Given a training set of m labeled instances of the form 〈xi, yi〉 and query xq , the k-NN algorithm outputs the most common label of the k nearest instances:
ykNN(xq) = arg max c m∑ i=1 1 [ c = yi ∧K(xi, xq) ≥ K(xk, xq) ] where 1 [·] equals one if its argument is true and zero otherwise, and xk is the kth nearest training instance to query xq assuming unique distances. The multiclass support vector machine (Crammer & Singer, 2001) in its dual form can be seen as a weighted nearest neighbor that outputs the class with the highest weighted sum of kernel values with the query:
ySVM(xq) = arg max c m∑ i=1 αi,cK(xi, xq) (1)
where αi,c is the weight on training instance xi that contributes to the score of class c.
The CKM performs the same classification as these instance-based methods but it sums over an exponentially larger set of virtual instances to mitigate the curse of dimensionality. Virtual instances are composed of rearranged elements from one or more training instances. Depending on the design of the CKM, elements can be subsets of instance variables (e.g., overlapping pixel patches) or features thereof (e.g., ORB features or a 2D grid of convnet feature vectors). We assume there is a deterministic procedure that processes each training or test instance xi into a fixed tuple of indexed elements Exi = (ei,1, . . . , ei,|Exi |), where instances may have different numbers of elements. The query instance xq (with tuple of elements Exq ) is the example that is being classified by the CKM; it is a training instance during training and a test instance during testing. A virtual instance z is represented by a tuple of elements from training instances, e.g. Ez = (e10,5, e71,2, . . . , e46,17). Given a query instance xq , the CKM represents a set of virtual instances each with the same number of elements asExq . We define a leaf kernelKL(ei,j , ei′,j′) that measures the similarity between any two elements. Using kernel composition (Aronszajn, 1950), we define the kernel between the query instance xq and a virtual instance z as the product of leaf kernels over their corresponding elements:
K(z, xq) = ∏|Exq | j KL(ez,j , eq,j).
We combine leaf kernels with weighted sums and products to compactly represent a sum over kernels with an exponential number of virtual instances. Just as a sum-product network can compactly represent a mixture model that is a weighted sum over an exponential number of mixture components, the same algebraic decomposition can compactly encode a weighted sum over an exponential number of kernels. For example, if the query instance is represented by two elements Exq = (eq,1, eq,2) and the training set contains elements {e1, e2, e3, e4, e5, e6}, then
[w1KL(eq,1, e1) + w2KL(eq,1, e2) + w3KL(eq,1, e3)]× [w4KL(eq,2, e4) + w5KL(eq,2, e5) + w6KL(eq,2, e6)]
expresses a weighted sum over nine virtual instances using eleven additions/multiplications instead of twenty-six for an expanded flat sum w1KL(eq,1, e1)KL(eq,2, e4) + . . . + w9KL(eq,1, e3) KL(eq,2, e6). If the query instance and training set contained 100 and 10000 elements, respectively, then a similar factorization would useO(106) operations compared to a naı̈ve sum over 10500 virtual instances. Leveraging the Sum-Product Theorem (Friesen & Domingos, 2016), we define CKMs to allow for more expressive architectures with this exponential computational savings. Definition 1. A compositional kernel machine (CKM) is defined recursively.
1. A leaf kernel over a query element and a training set element is a CKM. 2. A product of CKMs with disjoint scopes is a CKM. 3. A weighted sum of CKMs with the same scope is a CKM.
The scope of an operator is the set of query elements it takes as inputs; it is analogous to the receptive field of a unit in a neural network, but with CKMs the query elements are not restricted to being pixels on the image grid (e.g., they may be defined as a set of extracted image features). A leaf kernel has singleton scope, internal nodes have scope over some subset of the query elements, and the root node of the CKM has full scope of all query elements Exq . This definition allows for rich CKM architectures with many layers to represent elaborate compositions. The value of each sum node child is multiplied by a weight wk,c and optionally a constant cost function φ(ei,j , ei′,j′) that rewards certain compositions of elements. Analogous to a multiclass SVM, the CKM has a separate set of weights for each class c in the dataset. The CKM classifies a query instance as yCKM(xq) = arg maxc Sc(xq), where Sc(xq) is the value of the root node of the CKM evaluating query instance xq using weights for class c. Definition 2 (Friesen & Domingos (2016)). A product node is decomposable iff the scopes of its children are disjoint. An SPF is decomposable iff all of its product nodes are decomposable. Theorem 1 (Sum-Product Theorem, Friesen & Domingos (2016)). Every decomposable SPF can be summed over its domain in time linear in its size. Corollary 1. Sc(xq) can sum over the set of virtual instances in time linear in the size of the SPF. Proof. For each query instance element eq,j we define a discrete variable Zj with a state for each training element ei′,j′ found in a leaf kernel KL(eq,j , ei′,j′) in the CKM. The Cartesian product of the domains of the variables Z defines the set of virtual instances represented by the CKM. Sc(xq) is a SPF over semiring (R,⊕,⊗, 0, 1), variables Z, constant functions w and φ, and univariate functions KL(eq,j , Zj). With the appropriate definition of leaf kernels, any semiring can be used. The definition above provides that the children of every product node have disjoint scopes. Constant functions have empty scope so there is no intersection with scopes of other children. With all product nodes decomposable, Sc(xq) is a decomposable SPF and can therefore sum over all states of Z, the virtual instances, in time linear to the size of the CKM.
Special cases of CKMs include multiclass SVMs (flat sum-of-products) and naive Bayes nearest neighbor (Boiman et al., 2008) (flat product-of-sums). A CKM can be seen as a generalization of an image grammar (Fu, 1974) where terminal symbols corresponding to pieces of training images are scored with kernels and non-terminal symbols are sum nodes with a production for each child product node.
The weights and cost functions of the CKM control the weights on the virtual instances. Each virtual instance represented by the CKM defines a tree that connects the root to the leaf kernels over its unique composition of training set elements. If we were to expand the CKM into a flat sum (cf. Equation 1), the weight on a virtual instance would be the product of the weights and cost functions along the branches of its corresponding tree. These weights are important as they can prevent implausible virtual instances. For example, if we use image patches as the elements and allow all compositions, the set of virtual instances would largely contain nonsense noise patterns. If
the elements were pixels, the virtual instances could even contain arbitrary images from classes not present in the training set. There are many aspects of composition that can be encoded by the CKM. For example, we can penalize virtual instances that compose training set elements using different symmetry group transformations. We could also penalize compositions that juxtapose elements that disagree on the contents of their borders. Weights can be learned to establish clusters of elements and reward certain arrangements. In Section 3 we demonstrate one choice of weights and cost functions in a CKM architecture built from extracted image features.
2.2 LEARNING
The training procedure for a CKM builds an SPF that encodes the virtual instances. There are then two options for how to set weights in the model. As with k-NN, the weights in the CKM could be set to uniform. Alternatively, as with SVMs, the weights could be optimized to improve generalization and reduce model size.
For weight learning, we use block-coordinate gradient descent to optimize leave-one-out loss over the training set. The leave-one-out loss on a training instance xi is the loss on that instance made by the learner trained on all data except xi. Though it is an almost unbiased estimate of generalization error (Luntz & Brailovsky, 1969), it is typically too expensive to compute or optimize with non-IBL methods (Chapelle et al., 2002). With CKMs, caching the SPFs and efficient data structures make it feasible to compute exact partial derivatives of the leave-one-out loss over the whole training set. We use a multiclass squared-hinge loss
L(xi, yi) = max 1 + Sy′(xi)︸ ︷︷ ︸ Best incorrect class −Syi(xi)︸ ︷︷ ︸ True class , 0  2
for the loss on training instance xi with true label yi and highest-scoring incorrect class y′. We use the squared version of the hinge loss as it performs better empirically and prioritizes updates to element weights that led to larger margin violations. In general, this objective is not convex as it involves the difference of the two discriminant functions which are strictly convex (due to the choice of semiring and the product of weights on each virtual instance). In the special case of the sum-product semiring and unique weights on virtual instances the objective is convex as is true for L2-SVMs. Convnets also have a non-convex objective, but they require lengthy optimization to perform well. As we show in Section 3, CKMs can achieve high accuracy with uniform weights, which further serves as good initialization for gradient descent.
For each epoch, we iterate through the training set, for each training instance xi optimizing the block of weights on those branches with Exi as descendants. We take gradient steps to lower the leaveone-out loss over the rest of the training set ∑ i′∈([1,m]\i) L(xi′ , yi′). We iterate until convergence or an early stopping condition. A component of the gradient of the squared-hinge loss on an instance takes the form
∂
∂wk,c L(xi, yi) =  2∆(xi, yi) ∂Sy′ (xi) ∂wk,c if ∆(xi, yi) > 0 ∧ c = y′
−2∆(xi, yi)∂Syi (xi)∂wk,c if ∆(xi, yi) > 0 ∧ c = yi 0 otherwise
where ∆(xi, yi) = 1 + Sy′(xi) − Syi(xi). We compute partial derivatives ∂Sc(xi)∂wk,c with backpropagation through the SPF. For efficiency, terms of the gradient can be set to zero and excluded from backpropagation if the values of corresponding leaf kernels are small enough. This is either exact (e.g., if ⊕ is maximization) or an approximation (e.g., if ⊕ is normal addition).
2.3 SCALABILITY
CKMs have several scalability advantages over convnets. As mentioned previously, they do not require a lengthy training procedure. This makes it much easier to add new instances and categories. Whereas most of the computation to evaluate a single setting of convnet hyperparameters is sunk in training, CKMs can efficiently race hyperparameters on hold-out data (Lee & Moore, 1994).
The evaluation of the CKM depends on the structure of the SPF, the size of the training set, and the computer architecture. A basic building block of these SPFs is a sum node with a number of children on the order of magnitude of the training set elements |E|. On a sufficiently parallel
computer, assuming the size of the training set elements greatly exceeds the dimensionality of the leaf kernel, this sum node will require O(log(|E|)) time (the depth of a parallel ⊕ reduction circuit) and O(|E|) space. Duda et al. (2000) describe a constant time nearest neighbor circuit that relies on precomputed Voronoi partitions, but this has impractical space requirements in high dimensions. As with SVMs, optimization of sparse element weights can greatly reduce model size.
On a modest multicore computer, we must resort to using specialized data structures. Hash codes can be used to index raw features or to measure Hamming distance as a proxy to more expensive distance functions. While they are perhaps the fastest method to accelerate a nearest neighbor search, the most accurate hashing methods involve a training period yet do not necessarily result in high recall (Torralba et al., 2008; Heo et al., 2012). There are many space-partitioning data structure trees in the literature, however in practice none are able to offer exact search of nearest neighbors in high dimensions in logarithmic time. In our experiments we use hierarchical k-means trees (Muja & Lowe, 2009), which are a good compromise between speed and accuracy.
3 EXPERIMENTS
We test CKMs on three image classification scenarios that feature images from either the small NORB dataset or the NORB jittered-cluttered dataset (LeCun et al., 2004). Both NORB datasets contain greyscale images of five categories of plastic toys photographed with varied altitudes, azimuths, and lighting conditions. Table 1 summarizes the datasets. We first describe the SPN architecture and then detail each of the three scenarios.
3.1 EXPERIMENTAL ARCHITECTURE
In our experiments the architecture of the SPF Sc(xq) for each query image is based on its unique set of extracted ORB features. Like SIFT features, ORB features are rotation-invariant and produce a descriptor from intensity differences, but ORB is much faster to compute and thus suitable for real time applications (Rublee et al., 2011). The elements Exi = (ei,1, . . . , ei,|Ei|) of each image xi are its extracted keypoints, where an element’s feature vector and image position are denoted by ~f(ei,j) and ~p(ei,j) respectively. We use the max-sum semiring (⊕ = max, ⊗ = +) because it is more robust to noisy virtual instances, yields sparser gradients, is more efficient to compute, and performs better empirically compared with the sum-product semiring.
The SPF Sc(xq) maximizes over variables Z = (Z1, . . . , Z|Exq |) corresponding to query elements Exq with states for all possible virtual instances. The SPF contains a unary scope max node for every variable {Zj} that maximizes over the weighted kernels of all possible training elements E : ⊕(Zj) = ⊕ zj∈E wzj ,c ⊗KL(zj , eq,j). The SPF contains a binary scope max node for all pairs of variables {Zj , Zj′} for which at least one corresponding query element is within the k-nearest spatial neighbors of the other. These nodes maximize over the weighted kernels of all possible combinations of training set elements.
⊕(Zj , Zj′) = ⊕ zj∈E ⊕ zj′∈E wzj ,c ⊗ wzj′ ,c ⊗ φ(zj , zj′)⊗KL(zj , eq,j)⊗KL(zj′ , eq,j′) (2)
This maximizes over all possible pairs of training set elements, weighting the two leaf kernels by two corresponding element weights and a cost function. We use a leaf kernel for image elements that incorporates both the Hamming distance between their features and the Euclidean distance between their image positions: KL(ei,j , ei′,j′) = max(β0 − β1dHam(~f(ei,j), ~f(ei′,j′)), 0) + max(β2||(~p(ei,j), ~p(ei′,j′)||, β3). This rewards training set elements that look like a query instance element and appear in a similar location, with thresholds for efficiency. This can represent, for example, the photographic bias to center foreground objects or a discriminative cue from seeing sky at the top of the image. We use the pairwise cost function φ(ei,j , ei′,j′) = 1[i = i′]β4 that rewards combinations of elements from the same source training image. This captures the intuition that
compositions sourced from more images are less coherent and more likely to contain nonsense than those using fewer. The image is represented as a sum of these unary and binary max nodes. The scopes of children of the sum are restricted to be disjoint, so the children {⊕(Z1, Z2),⊕(Z2, Z3)} would be disallowed, for example. This restriction is what allows the SPF to be tractable, and with multiple sums the SPF has high-treewidth. By comparison, a Markov random field expressing these dependencies would be intractable. The root max node of the SPF has P sums as children, each of which has its random set of unary and binary scope max node children that cover full scope Z. We illustrate a simplified version of the SPF architecture in Figure 1. Though this SPF models limited image structure, the definition of CKMs allows for more expressive architectures as with SPNs.
In the following sections, we refer to two variants CKM and CKMW . The CKM version uses uniform weights wk,c, similar to the basic k-nearest neighbor algorithm. The CKMW method optimizes weights wk,c as described in Section 2.2. Both versions restrict weights for class c to be −∞ (⊕ identity) for those training elements not in class c. This constraint ensures that method CKM is discriminative (as is true with k-NN) and reduces the number of parameters optimized by CKMW . The hyperparameters of ORB feature extraction, leaf kernels, cost function, and optimization were chosen using grid search on a validation set.
With our CPU implementation, CKM trains in a single pass of feature extraction and storage at ∼5ms/image, CKMW trains in under ten epochs at ∼90ms/image, and both versions test at ∼80ms/image. The GPU-optimized convnets train at ∼2ms/image for many epochs and test at ∼1ms/image. Remarkably, CKM on a CPU trains faster than the convnet on a GPU.
3.2 SMALL NORB
We use the original train-test separation which measures generalization to new instances of a category (i.e. tested on toy truck that is different from the toys it was trained on). We show promising results in Table 2 comparing CKMs to deep and IBL methods. With improvement over k-NN and SVM, the CKM and CKMW results show the benefit of using virtual instances to combat the curse of dimensionality. We note that the CKM variant that does not optimize weights performs nearly as well as the CKMW version that does. Since the test set uses a different set of toys, the use of untrained ORB features hurts the performance of the CKM. Convnets have an advantage here because they discriminatively train their lowest level of features and represent richer image structure in their architecture. To become competitive, future work should improve upon this preliminary CKM
architecture. We demonstrate the advantage of CKMs for representing composition and symmetry in the following experiments.
3.3 NORB COMPOSITIONS
A general goal of representation learning is to disentangle the factors of variation of a signal without having to see those factors in all combinations. To evaluate progress towards this, we created images containing three toys each, sourced from the small NORB training set. Small NORB contains ten types of each toy category (e.g., ten different airplanes), which we divided into two collections. Each image is generated by choosing one of the collections uniformly and for each of three categories (person, airplane, animal) randomly sampling a toy from that collection with higher probability (P = 56 ) than from the other collection (i.e., there are two children with disjoint toy collections but they sometimes borrow). The task is to determine which of the two collections generated the image. This dataset measures whether a method can distinguish different compositions without having seen all possible permutations of those objects through symmetries and noisy intra-class variation. Analogous tasks include identifying people by their clothing, recognizing social groups by their members, and classifying cuisines by their ingredients.
We compare CKMs to other methods in Table 3. Convnets and their features are computed using the TensorFlow library (Abadi et al., 2015). Training convnets from few images is very difficult without resorting to other datasets; we augment the training set with random crops, which still yields test accuracy near chance. In such situations it is common to train an SVM with features extracted by a convnet trained on a different, larger dataset. We use 2048-dimensional features extracted from the penultimate layer of the pre-trained Inception network (Szegedy et al., 2015) and a linear kernel SVM with squared-hinge loss (Pedregosa et al., 2011). Notably, the CKM is much more accurate than the deep methods, and it is about as fast as the SVM despite not taking advantage of the GPU.
3.4 NORB SYMMETRIES
Composition is a useful tool for modeling the symmetries of objects. When we see an image of an object in a new pose, parts of the image may look similar to parts of images of the object in poses we have seen before. In this experiment, we partition the training set of NORB jittered-cluttered into a
new dataset with 10% withheld for each of validation and testing. Training and testing on the same group of toy instances, this measures the ability to generalize to new angles, lighting conditions, backgrounds, and distortions.
We vary the amount of training data to plot learning curves in Figure 3. We observe that CKMs are better able to generalize to these distortions than other methods, especially with less data. Importantly, the performance of CKM improves with more data, without requiring costly optimization as data is added. We note that the benefit of CKMW using weight learning becomes apparent with 200 training instances. This learning curve suggests that CKMs would be well suited for applications in cluttered environments with many 3D transformations (e.g., loop closure).
4 CONCLUSION
This paper proposed compositional kernel machines, an instance-based method for object recognition that addresses some of the weaknesses of deep architectures and other kernel methods. We showed how using a sum-product function to represent a discriminant function leads to tractable summation over the weighted kernels to an exponential set of virtual instances, which can mitigate the curse of dimensionality and improve sample complexity. We proposed a method to discriminatively learn weights on individual instance elements and showed that this improves upon uniform weighting. Finally, we presented results in several scenarios showing that CKMs are a significant improvement for IBL and show promise compared with deep methods.
Future research directions include developing other architectures and learning procedures for CKMs, integrating symmetry transformations into the architecture through kernels and cost functions, and applying CKMs to structured prediction, regression, and reinforcement learning problems. CKMs exhibit a reversed trade-off of fast learning speed and large model size compared to neural networks. Given that animals can benefit from both trade-offs, these results may inspire computational theories of different brain structures, especially the neocortex versus the cerebellum (Ito, 2012).
ACKNOWLEDGMENTS
The authors are grateful to John Platt for helpful discussions and feedback. This research was partly supported by ONR grant N00014-16-1-2697, AFRL contract FA8750-13-2-0019, a Google PhD Fellowship, an AWS in Education Grant, and an NVIDIA academic hardware grant. The views and conclusions contained in this document are those of the authors and should not be interpreted as necessarily representing the official policies, either expressed or implied, of ONR, AFRL, or the United States Government.

The authors propose a method to efficiently augment an SVM variant with many virtual instances, and show promising preliminary results. The paper was an interesting read, with thoughtful methodology, but has partially unsupported and potentially misleading claims.
Pros:
- Thoughtful methodology with sensible design choices
- Potentially useful for smaller (n < 10000) datasets with a lot of statistical structure
- Nice connections with sum-product literature
Cons:
- Claims about scalability are very unclear
- Generally the paper does not succeed in telling a complete story about the properties and applicability of the proposed method.
- Experiments are very preliminary
The scalability claims are particularly unclear. The paper repeatedly mentions lack of scalability as a drawback for convnets, but it appears the proposed CKM is less scalable than a standard SVM, yet SVMs often handle much fewer training instances than deep neural networks. It appears the scalability advantages are mostly for training sets with roughly fewer than 10,000 instances -- and even if the method could scale to >> 10,000 training instances, it's unclear whether the predictive accuracy would be competitive with convnets in that domain. Moreover, the idea of doing 10^6 operations simply for creating virtual instances on 10^4 training points and 100 test points is still somewhat daunting. What if we had 10^6 training instances and 10^5 testing instances?  Because scalability (in the number of training instances) is one of the biggest drawbacks of using SVMs (e.g. with Gaussian kernels) on modern datasets, the scalability claims in this paper need to be significantly expanded and clarified. On a related note, the suggestion that convnets grow quadratically in computation with additional training instances in the introduction needs to be augmented with more detail, and is potentially misleading. Convnets typically scale linearly with additional training data.
In general, the paper suffers greatly from a lack of clarity and issues of presentation. As above, the full story is not presented, with critical details often missing. Moreover, it would strengthen the paper to remove broad claims such as "Just as support vector machines (SVMs) eclipsed multilayer perceptrons in the 1990s, CKMs could become a compelling alternative to convnets with reduced training time and sample complexity", suggesting that CKMs could eclipse convolutional neural networks, and instead provide more helpful and precise information. Convnets are multilayer perceptrons used in the 1990s (as well as now) and they are not eclipsed by SVMs -- they have different relative advantages. And based on the information presented, broadly advertising scalability over convnets is misleading. Can CKMs scale to datasets with millions of training and test instances?  It seems as if the scalability advantages are limited to smaller datasets, and asymptotic scalability could be much worse in general. And even if CKMs could scale to such datasets would they have as good predictive accuracy as convnets on those applications? Being specific and with full disclosure about the precise strengths and limitations of the work would greatly improve this paper.
CKMs may be more robust to adversarial examples than standard convnets, due to the virtual instances. But there are many approaches to make deep nets more robust to adversarial examples. It would be useful to consider and compare to these. The ideas behind CKMs also are not inherently specific to kernel methods. Have you considered looking at using virtual instances in a similar way with deep networks? A full exploration might be its own paper, but the idea is worth at least brief discussion in the text.
A big advantage of SVMs (with Gaussian kernels) over deep neural nets is that one can achieve quite good performance with very little human intervention (design choices). However, CKMs seem to require extensive intervention, in terms of architecture (as with a neural network), and in insuring that the virtual instances are created in a plausible manner for the particular application at hand. It's very unclear in general how one would want to create sensible virtual instances and this topic deserves further consideration. Moreover, unlike SVMs (with for example Gaussian or linear kernels) or standard convolutional networks, which are quite general models, CKMs as applied in this paper seem more like SVMs (or kernel methods) which have been highly tailored to a particular application -- in this case, the NORB dataset. There is certainly nothing wrong with the tailored approach, but it would help to be clear and detailed about where the presented ideas can be applied out of the box, or how one would go about making the relevant design choices for a range of different problems. And indeed, it would be good to avoid the potentially misleading suggestions early in the paper that the proposed method is a general alternative to convnets.
The experiments give some insights into the advantages of the proposed approach, but are very limited. To get a sense of the properties --the strengths and limitations -- of the proposed method, one needs a greater range of datasets with a much larger range of training and test sizes. The comparisons are also quite limited: why not an SVM with a Gaussian kernel?  What about an SVM using convnet features from the dataset at hand (light blue curve in figure 3) -- it should do at least as well as the light blue curve. There are also other works that could be considered which combine some of the advantages of kernel methods with deep networks. Also the claim that the approach helps with the curse of dimensionality is sensible but not particularly explored. It also seems the curse of dimensionality could affect the scalability of creating a useful set of virtual instances. And it's unclear how CKM would work without any ORB features.
Even if the method can (be adapted to) scale to n >> 10000, it's unclear whether it will be more useful than convnets in that domain. Indeed, in the experiments here, convnets essentially match CKMs in performance after 12,000 examples, and would probably perform better than CKMs on larger datasets.  We can only speculate because the experiments don't consider larger problems.
The methodology largely takes inspiration from sum product networks, but its application in the context of a kernel approach is reasonably original, and worthy of exploration. It's reasonable to expect the approach to be significant, but its significance is not demonstrated.
The quality is high in the sense that the methods and insights are thoughtful, but suffers from broad claims and a lack of full and precise detail.
In short: I like the paper, but it needs more specific details, and a full disclosure of where the method should be most applicable, and its precise advantages and limitations.  Code would be helpful for reproducibility.


A DIFFERENTIABLE PHYSICS ENGINE
One of the most important fields in robotics is the optimization of controllers. Currently, robots are often treated as a black box in this optimization process, which is the reason why derivative-free optimization methods such as evolutionary algorithms or reinforcement learning are omnipresent. When gradient-based methods are used, models are kept small or rely on finite difference approximations for the Jacobian. This method quickly grows expensive with increasing numbers of parameters, such as found in deep learning. We propose an implementation of a modern physics engine, which can differentiate control parameters. This engine is implemented for both CPU and GPU. Firstly, this paper shows how such an engine speeds up the optimization process, even for small problems. Furthermore, it explains why this is an alternative approach to deep Q-learning, for using deep learning in robotics. Finally, we argue that this is a big step for deep learning in robotics, as it opens up new possibilities to optimize robots, both in hardware and software.
1 INTRODUCTION
To solve tasks efficiently, robots require an optimization of their control system. This optimization process can be done in automated testbeds (Degrave et al., 2015), but typically these controllers are optimized in simulation. Standard methods to optimize these controllers include particle swarms, reinforcement learning, genetic algorithms and evolutionary strategies. These are all derivative-free methods.
A recently popular alternative approach is to use deep Q-learning, a reinforcement learning algorithm. This method requires a lot of evaluations in order to train the many parameters (Levine et al., 2016). However, deep learning experience has taught us that optimizing with a gradient is often faster and more efficient. This fact is especially true when there are a lot of parameters, as is common in deep learning. However, in the optimization processes for control systems, the robot is almost exclusively treated as a non-differentiable black box. The reason for this is that the robot in hardware is not differentiable, nor are current physics engines able to provide the gradient of the robot models. The resulting need for derivative-free optimization approaches limits both the optimization speed and the number of parameters in the controllers.
Recent physics engines, such as mujoco (Todorov et al., 2012), can derive gradients through the model of a robot but rely on a finite difference method to approximate the gradient. Evaluating finite difference approximations, however, requires the same number of model evaluations as the number of states with respect to which is differentiated. Additionally, the gradient is an estimation.
In this paper, we suggest an alternative approach, by introducing a differentiable physics engine with analytical gradients. This idea is not novel. It has been done before with spring-damper models in 2D and 3D (Hermans et al., 2014). This technique is also similar to adjoint optimization, a method widely used in various applications such as thermodynamics (Jarny et al., 1991) and fluid
∗Former member, currently unaffiliated
dynamics (Iollo et al., 2001). However, modern engines to model robotics are not based on springdamper systems. The most commonly used ones are 3D rigid body engines, which rely on impulsebased velocity stepping methods (Erez et al., 2015). In this paper, we test whether these engines are also differentiable and whether this gradient is computationally tractable. We will show how this method does speed up the optimization process tremendously, and give some examples where we optimize deep learned neural network controllers with millions of parameters.
2 A 3D RIGID BODY ENGINE
The goal is to implement a modern 3D Rigid body engine, in which parameters can be differentiated with respect to the fitness a robot achieves in a simulation, such that these parameters can be optimized with methods based on gradient descent.
The most frequently used simulation tools for model-based robotics, such as PhysX, Bullet, Havok and ODE, go back to MathEngine (Erez et al., 2015). These tools are all 3D rigid body engines, where bodies have 6 degrees of freedom, and the relations between them are defined as constraints. These bodies exert impulses on each other, but their positions are constrained, e.g. to prevent the bodies from penetrating each other. The velocities, positions and constraints of the rigid bodies define a linear complementarity problem (LCP) (Chappuis, 2013), which is then solved using a Gauss-Seidel projection (GSP) method (Jourdan et al., 1998). The solution of this problem are the new velocities of the bodies, which are then integrated by semi-implicit Euler integration to get the new positions (Stewart and Trinkle, 2000). This system is not always numerically stable. Therefore the constraints are usually softened (Catto, 2009).
The recent growth of automatic differentiation libraries, such as Theano (Al-Rfou et al., 2016), Caffe (Jia et al., 2014) and Tensorflow (Abadi et al., 2015), has allowed for efficient differentiation of remarkably complex functions before (Degrave et al., 2016). Therefore, we implemented such a physics engine as a mathematical expression in Theano, a software library which does automatic evaluation and differentiation of expressions with a focus on deep learning. The resulting computational graph to evaluate this expression is then compiled for both CPU and GPU. To be able to compile for GPU however, we had to limit our implementation to a restricted set of elementary operations. The range of implementable functions is therefore severely capped. However, since the analytic gradient is determined automatically, the complexity of correctly implementing the differentiation is removed entirely.
One of these limitations with this restricted set of operations, is the limited support for conditionals. Therefore we needed to implement our physics engine without branching, as this is not yet available in Theano for GPU. Therefore some sacrifices had to be made. For instance, our system only allows for contact constraints between different spheres or between spheres and the ground plane. Collision detection algorithms for cubes typically have a lot of branching (Mirtich, 1998). However, this sphere based approach can in principle be extended to any other shape (Hubbard, 1996). On the other hand, we did implement a rather accurate model of servo motors, with gain, maximal torque, and maximal velocity parameters.
Another design choice was to use rotation matrices rather than the more common quaternions for representing rotations. Consequently, the states of the bodies are larger, but the operations required are matrix multiplications. This design reduced the complexity of the graph. However, cumulative operations on a rotation matrix might move the rotation matrix away from orthogonality. To correct for this, we renormalize our matrix with the update equation (Premerlani and Bizard, 2009):
A′ = 3A−A ◦ (A ·A)
2 (1)
where A′ is the renormalized version of the rotation matrix A. ‘◦’ denotes the elementwise multiplication, and ‘·’ the matrix multiplication. These design decisions are the most important aspects of difference with the frequently used simulation tools. In the following section, we will evaluate our physics simulator on some different problems. We take a look at the speed of computation and the number of evaluations required before the parameters of are optimized.
2.1 THROWING A BALL
To test our engine, we implemented the model of a giant soccer ball in the physics engine, as shown in Fig. 2a. The ball has a 1m diameter, a friction of µ = 1.0 and restitution e = 0.5. The ball starts off at position (0, 0). After 5 s it should be at position (10, 0) with zero velocity v and zero angular velocity ω. We optimized the initial velocity v0 and angular velocity ω0 at time t = 0 s until the errors at t = 5 s are less than 0.01m and 0.01m/s respectively.
Since the quantity we optimize is only know at the end of the simulation, but we need to optimize the parameters at the beginning of the simulation, we need to backpropagate our error through time (BPTT) (Sutskever, 2013). This approach is similar to the backpropagation through time method used for optimizing recurrent neural networks (RNN). In our case, every time step in the simulation can be seen as one pass through a neural network, which transforms the inputs from this timestep to inputs for the next time step. For finding the gradient, this RNN is unfolded completely, and the gradient can be obtained by differentiating this unfolded structure. This analytic differentiation is done automatically by the Theano library.
Optimizing the six parameters in v0 and ω0 took only 88 iterations with gradient descent and backpropagation through time. Optimizing this problem with CMA-ES (Hansen, 2006), a state of the art derivative-free optimization method, took 2422 iterations. Even when taking the time to compute the gradient into account, the optimization with gradient descent takes 16.3 s, compared to 59.9 s with CMA-ES. This result shows that gradient-based optimization of kinematic systems can in some cases already outperform gradient-free optimization algorithms from as little as six parameters.
3 POLICY SEARCH
To evaluate the relevance of our differentiable physics engine, we use a neural network as a general controller for a robot, as shown in Figure 1. We consider a general robot model in a discrete-time dynamical system xt+1 = fph(xt,ut) with a task cost function of l(xt,p), where xt is the state of the system at time t and ut is the input of the system at time t. p provides some freedom in parameterizing the loss. If Xt is the trajectory of the state up to time t − 1, the goal is to find a policy ut = π(Xt) such that we minimize the loss Lπ .
Lπ = T∑ t=0 l(xt,p)
s.t. xt+1 = fph(xt, π(Xt)) and x0 = xinit (2)
In previous research, finding a gradient for this objective has been described as presenting challenges (Mordatch and Todorov, 2014). An approximation to tackle these issues has been discussed in Levine and Koltun (2013).
We implement this equation into an automatic differentiation library, ignoring these challenges in finding the analytic gradient altogether. The automatic differentiation library, Theano in our case, analytically derives this equation and compiles code to evaluate both the equation and its gradient.
We define our controller as a deep neural network gdeep with weights W. We do not pass all information Xt to this neural network, but only a vector of values st observed by the modeled sensors s(xt). We also provide our network with (some of the) task-specific parameters p′. Finally, we add a recurrent connection to the controller in the previous timestep ht. Therefore, our policy is the following:
π(Xt) = gdeep(s(x t),ht,p′ |W)
s.t. ht = hdeep(s(xt−1),ht−1,p′ |W) and h0 = 0 (3)
Notice the similarity between equations 2 and 3. Indeed, the equations for recurrent neural networks (RNN) in equation 3 are very similar to the ones of the loss of a physical model in equation 2. Therefore, we optimize this entire system as an RNN unfolded over time, as illustrated in Figure 4. The weights W are optimized with stochastic gradient descent. The gradient required for that is the Jacobian dL/dW, which is found with automatic differentiation software.
We have now reduced the problem to a standard deep learning problem. We need to train our network gdeep on a sufficient amount of samples xinit and for a sufficient amount of sampled tasks p in order to get adequate generalization. Standard RNN regularization approaches could also improve this generalization. We reckon that generalization of gdeep to more models fph, in order to ease the transfer of the controller from the model to the real system, is also possible (Hermans et al., 2014), but it is outside the scope of this paper.
3.1 QUADRUPEDAL ROBOT – COMPUTING SPEED
To verify the speed of our engine, we also implemented a small quadrupedal robot model, as illustrated in Fig. 2b. This model has a total of 81 sensors, e.g. encoders and an inertial measurement unit (IMU). The servo motors are controlled in a closed loop by a small neural network gdeep with a number of parameters, as shown previously in Fig. 4. The gradient is the Jacobian of L, the total traveled distance of the robot in 10 s , differentiated with respect to all the parameters of the controller W. This Jacobian is found by using BPTT and propagating all 10 s back. The time it takes to compute this traveled distance and the accompanying Jacobian is shown in Table 1. We include both the computation time with and without the gradient, i.e. both the forward and backward pass and the forward pass alone. This way, the numbers can be compared to other physics engines, as those only calculate without gradient. Our implementation and our model can probably be made more efficient, and evaluating the gradient can probably be made faster a similar factor.
When only a single controller is optimized, our engine runs more slowly on GPU than on CPU. To tackle this issue, we implemented batch gradient descent, which is commonly used in complex optimization problems. In this case, by batching our robot models, we achieve significant acceleration on GPU. Although backpropagating the gradient through physics slows down the computations by roughly a factor 10, this factor only barely increases with the number of parameters in our controller.
Combining this with our previous observation that fewer iterations are needed when using gradient descent, our approach can enable the use of gradient descent through physics for highly complex
deep neural network controllers with millions of parameters. Also note that by using a batch method, a single GPU can simulate about 864 000 model seconds per day, or 86 400 000 model states. This should be plenty for deep learning. It also means that a single simulation step of a single robot, which includes collision detection, solving the LCP problem, integrating the velocities and backpropagating the gradient through it all, takes about 1ms on average. Without the backpropagation, this process is only about seven times faster.
3.2 4 DEGREE OF FREEDOM ROBOT ARM
As a first test of optimizing robot controllers, we implemented a four degree of freedom robotic arm, as depicted in Fig. 2c. The bottom of the robot has a 2 degrees of freedom actuated universal joint; the elbow has a 2 degree of freedom actuated joint as well. The arm is 1m long, and has a total mass of 32 kg. The servos have a gain of 30 s−1, a torque of 30Nm and a velocity of 45◦ s−1.
For this robot arm, we train controllers for a task with a gradually increasing amount of difficulty. To be able to train our parameters, we have to use a couple of tricks often used in the training of recurrent neural networks.
• We choose an objective which is evaluated at every time step and then averaged, rather than at specific points of the simulation. This approach vastly increases the number of samples over which the gradient is averaged, which in turn makes the gradient direction more reliable (Sjöberg et al., 1995).
• The value of the gradient is decreased by a factor α < 1 at every time step. This trick has the effect of a prior. Namely, events further in the past are less important for influencing current events, because intermediate events might diminish their influence altogether. It also improves robustness against exploding gradients (Hermans et al., 2014).
• We initialize the controller intelligently. We do not want the controller to shake the actuators violently and explore outside the accurate domain of our simulation model. Therefore our controllers are initialized such that they only output zeros at the start of the simulation. The initial policy is the zero policy.
• We constraint the size of the gradient to an L2-norm of 1. This makes sure that gradients close to discontinuities in the fitness landscape do not push the parameter values too far away, such that everything which was learned is forgotten (Sutskever, 2013).
3.2.1 REACHING A FIXED POINT
A first simple task, is to have a small neural net controller learn to move the controller to a certain fixed point in space, at coordinates (0.5m; 0.5m; 0.5m). The objective we minimize for this task, is the distance between the end effector and the target point, averaged over the 8 seconds we simulate our model.
We provide the controller with a single sensor input, namely the current distance between the end effector and the target point. Input is not required for this task, as there are solutions for which the motor signals are constant in time. However, this would not necessarily be the optimal approach for minimizing the average distance over time, it only solves the distance at the end of the simulation, but does not minimize the distance during the trajectory to get at the final position.
As a controller, we use a dense neural network with 1 input, 2 hidden layers of 128 units with a rectifier activation function, and 4 outputs with an identity activation function. This controller has 17 284 parameters in total. We disabled the recurrent connections ht.
We use gradient descent with a batch size of 1 robot for optimization, as the problem is not stochastic in nature. The parameters are optimized with Adam’s rule (Kingma and Ba, 2014) with a learning rate of 0.001. Every update step with this method takes about 5 seconds on CPU. We find that the controller comes within 4 cm of the target in 100 model evaluations, and within 1 cm in 150 model evaluations, which is small compared to the 1m arm of the robot. Moreover, the controller does find a more optimal trajectory which takes into account the sensor information.
Solving problems like these in fewer iteration steps than the number of parameters, is unfeasible with derivative free methods (Sjöberg et al., 1995). Despite that, we did try to optimize the same
problem with CMA-ES. After a week of computing and 60 000 model evaluations, CMA-ES did not show any sign of convergence, as it cannot handle the sheer amount of parameters.
3.2.2 REACHING A RANDOM POINT
As a second task, we sample a random target point in the reachable space of the end effector. We give this point as input v′ to the controller, and the task is to again minimize the average distance between the end effector and the target point v. Our objective L is this distance averaged over all timesteps.
As a controller, we use a dense neural network comparable to the previous section, but this time with 3 inputs. We used 3 hidden layers with 1024 units each, so the controller has 2 107 396 parameters in total. This is not necessary for this task, but we do it like this to demonstrate the power of this approach. In order to train for this task, we use a batch size of 128 robots, such that every update step takes 58 s on GPU. Each simulation takes 8 s with a simulation step of 0.01 s. Therefore, the gradient on the parameters of the controllers has been averaged over 51 200 timesteps at every update step. We update the parameters with Adam’s rule, where we scale the learning rate with the average error achieved in the previous step.
We find that it takes 576 update steps before the millions of parameters are optimized, such that the end effector of the robot is on average less than 10 cm of target, 2 563 update steps before the error is less than 5 cm.
3.3 A QUADRUPEDAL ROBOT – REVISITED
Optimizing a gait for a quadrupedal robot is a problem of a different order, something the authors have extensive experience with (Sproewitz et al., 2013; Degrave et al., 2013; 2015). The problem is way more challenging and allows for a broad range of possible solutions. In nature, we find a wide variety of gaits, from hopping over trotting, walking and galloping. With hand tuning on the robot model shown in Figure 2b, we were able to obtain a trotting motion with an average forward speed of 0.7m/s. We found it tricky to find a gait where the robot did not end up like an upside down turtle, as 75% of the mass of the robot is located in its torso.
As a controller for our quadrupedal robot, we use a neural network with 2 input signals st, namely a sine and a cosine signal with a frequency of 1.5Hz. On top of this, we added 2 hidden layers of 128 units and a rectifier activation function. As output layer, we have a dense layer with 8 units and a linear activation function, which has as input both the input layer and the top layer of the hidden layers. In total, this controller has 17 952 parameters. Since the problem is not stochastic in nature, we use a batch size of 1 robot. We initialize the output layer with zero weights, so the robot starts the optimization in a stand still position.
We optimize these parameters to maximize the average velocity of the spine over the course of 10 s of time in simulation. This way, the gradient used in the update step is effectively an average of the 1 000 time steps after unrolling the recurrent connections. This objective does not take into account energy use, or other metrics typically employed in robotic problems.
In only 500 model evaluations or about 1 hour of optimizing on CPU, the optimization with BPTT comes up with a solution with a speed of 1.17m/s. This solution is a hopping gait, with a summersault every 3 steps1, despite limiting the torque of the servos to 4Nm on this 28.7 kg robot. For more life-like gaits, energy efficiency could be use as a regularization method. Evaluating these improvements are however outside the scope of this paper.
3.3.1 THE INVERTED PENDULUM WITH A CAMERA AS SENSOR
As a fourth example, we implemented a model of the pendulum-cart system we have in our laboratorium. This pendulum-cart system is used for the classic control task of the underactuated inverted pendulum (Vaccaro, 1995). In this example however, a camera which is set up in front of the system is the only available information for the controller. It therefore has to observe the system it controls using vision. A frame captured by this camera is shown in Figure 3.
1A video is available at https://goo.gl/5ykZZe
In order to build this model, we implemented a differentiable camera in our physics engine. This camera uses a ray-tracing approach to find where it needs to sample from the textures, and uses bilinear interpolation to sample from these textures, similar to the one used for the spatial transform layer (Jaderberg et al., 2015). This interpolation is necessary for making the frame captured by the camera differentiable to the state of the robot with non-zero derivatives.
We minimize the distance from the end of the pendulum to the desired point and regularize the speed of the pendulum. The memoryless deep controller receives the current image of the camera, in addition to two images from the past such that it can estimate velocity and acceleration. We observe that a controller with 1,065,888 parameters is able to learn to swing up and keep the pendulum stable after only 2420 episodes of 3 model seconds. The complete optimization process took 15 hours. The resulting controller keeps the pendulum stable for more than one minute2. In order to do this, the controller has learned to interpret the frames it receives from the camera and found a suitable control strategy.
4 DISCUSSION
Our results show the first prototype of a differentiable physics engine based on similar algorithms as those that are commonly used in modern robotics simulators. When initially addressing the problem, we did not know whether finding the gradient would be computationally tractable, let alone whether evaluating it would be fast enough to be beneficial for optimization. In this paper, we have demonstrated that evaluating the gradient is tractable enough to speed up optimization on problems with as little as six parameters. The speed of this evaluation mainly depends on the complexity of the physics model and only slightly on the number of parameters to optimize. Therefore, our results suggest that this cost is dominated by the gain achieved from the combination of using batch gradient descent and GPU acceleration.
Optimizing the controller of a robot model with gradient-based optimization is equivalent to optimizing an RNN. After all, the gradient passes through each parameter at every time step. The parameter space is therefore very noisy. Consequently, training the parameters of this controller is a highly non-trivial problem, as it corresponds to training the parameters of an RNN. On top of that, exploding and vanishing signals and gradients cause far more challenging problems compared to feed forward networks.
In section 3.2, we already discussed some of the tricks used for optimizing RNNs. Earlier research shows that these methods can be extended to more complicated tasks than the ones discussed here (Hermans et al., 2014; Sutskever, 2013). Hence, we believe that this approach towards learning controllers for robotics applies to more complex problems than the illustrative examples in this paper.
All of the results in this paper will largely depend on showing how these controllers will work on the physical counterparts of our models. Nonetheless, we would like to conjecture that to a certain extent, this gradient of a model is close to the gradient of the physical system. The gradient of the model is more susceptible to high-frequency noise introduced by modeling the system, than the imaginary gradient of the system itself. Nonetheless, it contains information which might be indicative, even if it is not perfect. We would theorize that using this noisy gradient is still better than
2https://twitter.com/317070/status/821062814798331905
optimizing in the blind and that the transferability to real robots can be improved by evaluating the gradients on batches of (slightly) different robots in (slightly) different situations and averaging the results. This technique has already been applied in (Hermans et al., 2014) as a regularization method to avoid bifurcations during online learning. If the previous proves to be correct, our approach can offer an addition or possibly even an alternative to deep Q-learning for deep neural network controllers in robotics.
We can see the use of this extended approach for a broad range of applications in robotics. Not only do we think there are multiple ways where recent advances in deep learning could be applied to robotics more efficiently with a differentiable physics engine, we also see various ways in which this engine could improve existing angles at which robotics are currently approached:
• In this paper, we added memory by introducing recurrent connections in the neural network controller. We reckon that advanced, recurrent connections such as ones with a memory made out of LSTM cells (Hochreiter and Schmidhuber, 1997) can allow for more powerful controllers than the controllers described in this paper.
• Using a differentiable physics engine, we reckon that knowledge of a model can be transferred more efficiently into a forward or backward model in the form of a neural network, similar to methods such as used in Johnson et al. (2016) and Dumoulin et al. (2016). By differentiating through an exact model and defining a relevant error on this model, it should be possible to transfer knowledge from a forward or backward model in the differentiable physics engine to a forward or backward neural network model. Neural network models trained this way might be more robust than the ones learned from generated trajectories (Christiano et al., 2016). In turn, this neural model could then be used for faster but approximate evaluation of the model.
• Although we did not address this in this paper, there is no reason why only control parameters could be differentiated. Hardware parameters of the robot have been optimized the same way before (Jarny et al., 1991; Iollo et al., 2001; Hermans et al., 2014). The authors reckon that the reverse process is also true. A physics engine can provide a strong prior, which can be used for robots to learn (or adjust) their robot models based on their hardware measurements faster than today. You could optimize the model parameters with gradient descent through physics, to have the model better mimic the actual observations.
• Where adversarial networks are already showing their use in generating image models, we believe adversarial robotics training (ART) will create some inventive ways to design and control robots. Like in generative adversarial nets (GAN) (Goodfellow et al., 2014), where the gradient is pulled through two competing neural networks, the gradient could be pulled through multiple competing robots as well. It would form an interesting approach for swarm robotics, similar to previous results in evolutionary robotics (Sims, 1994; Pfeifer and Bongard, 2006; Cheney et al., 2014), but possibly faster.
5 CONCLUSION
In this paper, we show it is possible to build a differentiable physics engine. We implemented a modern engine which can run a 3D rigid body model, using the same algorithm as other engines commonly used to simulate robots, but we can additionally differentiate control parameters with BPTT. Our implementation also runs on GPU, and we show that using GPUs to simulate the physics can speed up the process for large batches of robots. We show that even complex sensors such as cameras, can be implemented and differentiated through, allowing for computer vision to be learned together with a control policy.
We find that these gradients can be computed fast enough for use in applications. We also show that using gradient descent with BPTT speeds up optimization processes often found in robotics, even for rather small problems, due to the reduced number of model evaluations required. We show that this improvement in speed scales to problems with a lot of parameters. We also show that using this engine, finding policies for robot models can be done faster and in a more straightforward way. This method should allow for a new approach to apply deep learning techniques in robotics.
ACKNOWLEDGMENTS
Special thanks to David Pfau for pointing out relevant prior art we were previously unaware of, and Iryna Korshunova for proofreading the paper. The research leading to these results has received funding from the Agency for Innovation by Science and Technology in Flanders (IWT). The NVIDIA Corporation donated the GTX 1080 used for this research.
6.2 TABLES

This paper creates a physics simulator using theano, and uses it to learn a neural network policy by back propagating gradients through the simulation. The approach is novel, and is motivated by being able to learn policies for robotics.
My two key reservations with the paper are as follows:
1. The method is motivated by learning policies for robotics. However, the proposed method is *only* useful for robotics if the learned policy can transfer the real world. Transferring policies from simulation to real-world is an open research problem, and is particularly challenging with less realistic simulators.
2. They key novelty/benefit of this approach over other model-based approaches is that the simulator is differentiable. However, the only empirical comparison in the paper is to a model-free approach (CMA-ES). To appropriately demonstrate the approach, it should be compared to other model-based approaches, which do not require analytic derivatives of the model.
For the reader to fully understand the pros and cons of the approach, the paper should also include quantitative comparison between the speed of the proposed simulator, and that of standard simulation platforms.
Because the idea is interesting and novel, I think it lies above the acceptance threshold. However, it would be significantly improved with the aforementioned comparisons.
Lastly, the writing of the paper could be improved, as it is rather informal and/or imprecise in a number of places. Here are some examples:
-- “we model the use of a neural network as a general controller for a robot” - can be more concisely phrased as something like “we model the robot controller using a neural network” or “the robot controller is modeled using a neural network"
-- “In previous research, finding a gradient…” - This is a run-on sentence.
-- “We basically jam this entire equation into” - This sentence is informal and imprecise.
-- “deep learning neural network” - the word “learning” should be omitted
-- “one big RNN, where we unfold over time” - should be “…RNN, which we unfold over time” or “…RNN, unfolded over time”
The writing would also be improved by making it more concise and fitting the paper into 8 pages.