EXTRAPOLATION AND LEARNING EQUATIONS
In classical machine learning, regression is treated as a black box process of identifying a suitable function from a hypothesis set without attempting to gain insight into the mechanism connecting inputs and outputs. In the natural sciences, however, finding an interpretable function for a phenomenon is the prime goal as it allows to understand and generalize results. This paper proposes a novel type of function learning network, called equation learner (EQL), that can learn analytical expressions and is able to extrapolate to unseen domains. It is implemented as an end-to-end differentiable feed-forward network and allows for efficient gradient based training. Due to sparsity regularization concise interpretable expressions can be obtained. Often the true underlying source expression is identified. INTRODUCTION The quality of a model is typically measured by its ability to generalize from a training set to previously unseen data from the same distribution. In regression tasks generalization essentially boils down to interpolation if the training data is sufficiently dense. As long as models are selected correctly, i. e. in a way to not overfit the data, the regression problem is well understood and can – at least conceptually – be considered solved. However, when working with data from real-world devices, e. g. controlling a robotic arm, interpolation might not be sufficient. It could happen that future data lies outside of the training domain, e. g. when the arm is temporarily operated outside of its specifications. For the sake of robustness and safety it is desirable in such a case to have a regression model that continues to make good predictions, or at least does not fail catastrophically. This setting, which we call extrapolation generalization, is the topic of the present paper. We are particularly interested in regression tasks for systems that can be described by real-valued analytic expression, e. g. mechanical systems such as a pendulum or a robotic arm. These are typically governed by a highly nonlinear function but it is nevertheless possible, in principle, to infer their behavior on an extrapolation domain from their behavior elsewhere. We make two main contributions: 1) a new type of network that can learn analytical expressions and is able to extrapolate to unseen domains and 2) a model selection strategy tailored to the extrapolation setting. The following section describes the setting of regression and extrapolation. Afterwards we introduce our method and discuss the architecture, its training, and its relation to prior art. We present our results in the Section Experimental evaluation and close with conclusions. REGRESSION AND EXTRAPOLATION We consider a multivariate regression problem with a training set {(x1, y1), . . . , (xN , yN )} with x ∈ R, y ∈ R. Because our main interest lies on extrapolation in the context of learning the dynamics of physical systems we assume the data originates from an unknown analytical function (or system of functions), φ : R → R with additive zero-mean noise, ξ, i. e. y = φ(x) + ξ and Eξ = 0. The function φ may, for instance, reflect a system of ordinary differential equations that govern the movements of a robot arm or the like. The general task is to learn a function ψ : R → R that approximates the true functional relation as well as possible in the squared loss sense, i. e. achieves minimal expected error E‖ψ(x) − φ(x)‖2. In practice, we only have particular examples of the function values available and measure the quality of predicting in terms of the empirical error on
INTRODUCTION
The quality of a model is typically measured by its ability to generalize from a training set to previously unseen data from the same distribution. In regression tasks generalization essentially boils down to interpolation if the training data is sufficiently dense. As long as models are selected correctly, i. e. in a way to not overfit the data, the regression problem is well understood and can – at least conceptually – be considered solved. However, when working with data from real-world devices, e. g. controlling a robotic arm, interpolation might not be sufficient. It could happen that future data lies outside of the training domain, e. g. when the arm is temporarily operated outside of its specifications. For the sake of robustness and safety it is desirable in such a case to have a regression model that continues to make good predictions, or at least does not fail catastrophically. This setting, which we call extrapolation generalization, is the topic of the present paper.
We are particularly interested in regression tasks for systems that can be described by real-valued analytic expression, e. g. mechanical systems such as a pendulum or a robotic arm. These are typically governed by a highly nonlinear function but it is nevertheless possible, in principle, to infer their behavior on an extrapolation domain from their behavior elsewhere. We make two main contributions: 1) a new type of network that can learn analytical expressions and is able to extrapolate to unseen domains and 2) a model selection strategy tailored to the extrapolation setting.
The following section describes the setting of regression and extrapolation. Afterwards we introduce our method and discuss the architecture, its training, and its relation to prior art. We present our results in the Section Experimental evaluation and close with conclusions.
REGRESSION AND EXTRAPOLATION
We consider a multivariate regression problem with a training set {(x1, y1), . . . , (xN , yN )} with x ∈ Rn, y ∈ Rm. Because our main interest lies on extrapolation in the context of learning the dynamics of physical systems we assume the data originates from an unknown analytical function (or system of functions), φ : Rn → Rm with additive zero-mean noise, ξ, i. e. y = φ(x) + ξ and Eξ = 0. The function φ may, for instance, reflect a system of ordinary differential equations that govern the movements of a robot arm or the like. The general task is to learn a function ψ : Rn → Rm that approximates the true functional relation as well as possible in the squared loss sense, i. e. achieves minimal expected error E‖ψ(x) − φ(x)‖2. In practice, we only have particular examples of the function values available and measure the quality of predicting in terms of the empirical error on
training or test data D,
E(D) = 1
N N∑ i=1 ‖ψ(xi)− yi‖2 . (1)
If training and test data are sampled from the same distribution then we speak about an interpolation problem. In the extrapolation setting the training data is assumed to cover only a limited range of the data domain. In the example of the robot arm, for instance, the training may be restricted to a certain joint angle range or maximal velocity. For testing we want to make predictions about the unseen domains, e. g. for higher velocities. To succeed in this task, it is essential to identify the underlying functional relationship instead of just minimizing the empirical error, as detailed below. As usual, we split the data that is available at training time into a part for model training and a part for validation or model selection.
LEARNING A NETWORK FOR FUNCTION EXTRAPOLATION
The main model we propose is a multi-layered feed-forward network with computational units specifically designed for the extrapolation regression tasks. For an L-layer network, there are L− 1 hidden layers, each consisting of a linear mapping followed by non-linear transformations. For simplicity of notation, we explain the network as if each hidden layer had the same structure (k′ inputs, k outputs). In practice, each layer can be designed independently of the others, of course, as long as input/output dimensions match.
The linear mapping at level l maps the k′-dimensional input y(l−1) to the d-dimensional intermediate representation z given by
z(l) = W (l)y(l−1) + b(l), (2)
where y(l−1) is the output of the previous layer, with the convention y(0) = x. The weight matrix W (l) ∈ Rd×k′ and the bias vector b(l) ∈ Rd are free parameters that are learned during training. The non-linear transformation contains u unary units, fi : R→ R, for i = 1, . . . , u, and v binary units, gj : R× R→ R for j = 1, . . . , v. Their outputs are concatenated to form the layer output
y(l) := ( f1(z (l) 1 ), f2(z (l) 2 ), . . . , fu(z (l) u ), g1(z (l) u+1, z (l) u+2), . . . , gv(z (l) u+2v−1, z (l) u+2v) ) . (3)
In total, the nonlinear stage has k = u+v outputs and d = u+2v inputs. The unary units, f1, . . . , fu receive the respective component, z1, . . . , zu as inputs, and each unit may be one of the following base functions as specified in a fixed type parameter Ii ∈ {0, 1, 2, 3}
fi(zi) :=  zi if Ii = 0, sin(zi) if Ii = 1, cos(zi) if Ii = 2, sigm(zi) if Ii = 3,
for i = 1, . . . , u, (4)
where sigm(z) = 11+e−z is the standard sigmoid function. The binary units, g1, . . . , gv receive the remaining component, zu+1, . . . , zu+2v, as input in pairs of two. They are multiplication units that compute the product of their two input values:
gj(zu+2j−1, zu+2j) := zu+2j−1 · zu+2j for j = 1, . . . , v. (5)
Finally, the L-th and last layer computes the regression values by a linear read-out
y(L) := W (L)y(L−1) + b(L). (6)
The architecture is depicted in Fig. 1. We call the new architecture Equation Learner (EQL) and denote the function it defines by ψ.
DISCUSSION OF THE ARCHITECTURE
The proposed network architecture differs in two main aspects from typical feed-forward networks: the existence of multiplication units and the possibility of sine and cosine as nonlinearities for the unary units. Both design choices are motivated by our objective of learning a system of equations that govern a physical system and can extrapolate to new parts of the input space.
Sigmoid nonlinearities are the canonical choice of activation function for artificial neural networks (ANN) and proved to be successful. In fact, we include sigmoids in our architecture, making it a super class of ANNs. However, they were typically disabled by the training procedure corresponding to their absence in the considered physical equations. Other, predominantly local nonlinearities, in particular radial basis functions Broomhead & Lowe (1988) we do not include, since one cannot expect them to extrapolate at all. Further nonlinearities, such as (square) roots and logarithms, could in principle be useful for learning physical equations, but they pose problems because their domains of definition is restricted to positive inputs. We leave the task of incorporating them in a principled way to future work.
The ability to multiply two values is a second crucial component of our network architecture. Again, it is inspired by the typical form of physical equations, where multiplication of components is arguably second common basic operation after addition (which the linear layers can perform). Multiplication was introduced into neural networks long ago as product-units Durbin & Rumelhart (1989) and Pi-Sigma-unit Shin & Ghosh (1991). The product-units have large fan-in that compute products over all their inputs, potentiated by the respective weights. The result is typically the behavior of a high order polynomial, which are powerful function approximators, but rarely occur in physical equations. Polynomials are also known to require careful fine-tuning in order not to overfit, which makes them a risky choice for the purpose of extrapolation. The Pi-Sigma units are multiplication units with a fixed number of factors and our multiplication units are a special for 2 factors. We find that multiplying just two values at a time is well adjusted to the task we aim at, as it allows to control the maximal degree of the learned polynomial by the depth of the network.
Finally, each layer of the network contains unary units that act as identity maps, which in particular gives the network the option to learn functions with smaller number of nonlinearities than the total network depths.
NETWORK TRAINING
The EQL is fully differentiable in its free parameters θ = {W (1), . . . ,W (L), b(1), . . . , b(L)}, which allows us to train it in an end-to-end fashion using back-propagation. We adopt a Lasso-like objective Tibshirani (1996),
L(D) = 1 N |D|∑ i=1 ‖ψ(xi)− yi‖2 + λ L∑ l=1 ∣∣W (l)∣∣ 1 , (7)
that is, a linear combination of L2 loss and L1 regularization, and apply a stochastic gradient descent algorithm with mini-batches and Adam Kingma & Ba (2015) for calculating the updates:
θt+1 = θt + Adam
( ∂L ( D(t) ) ∂θ , α ) , (8)
where D(t) denotes the current mini-batch and α is the stepsize parameter. The choice of Adam is not critical and standard stochastic gradient descent also works. In all numerical experiments we use α = 0.001 and a mini-batch size of 20.
The role of the L1 regularization is to encourage networks with sparse connections, matching the intuition that a typical formula describing a physical system contains only a small number of terms, each operating only on a few variables. However, in a non-convex setting where local minima are likely to occur, this type of regularization can have an undesirable side-effect: during the course of the optimization the weights hardly ever change their sign. The reason is that the regularization leads to a constant rate of weight decay whereas the counteracting derivative with respect to the square loss is proportional to the backpropagated error signal and the input to the unit. The latter contributions are often smaller along paths with small weights, such that many weights go to zero and stay there. Additionally, any non-zero regularization term causes the learned weights to reflect a trade-off between minimizing the loss and the regularizer. Although, this can lead to improved generalization, it also results in a systematic underestimation of the function values.
Therefore, we follow a hybrid regularization strategy: at the beginning of the training procedure (t < t1) we use no regularization (λ = 0), such that parameters can vary freely and reach reasonable starting points. Afterwards, we switch on the regularization by setting λ to a nonzero value, which has the effect that a sparse network structure emerges. Finally, for the last steps of the training (t > t2) we disable L1 regularization (λ = 0) but enforce the same L0 norm of the weights. This is achieved by keeping all weights w ∈ W 1...L that are close to 0 at 0, i. e. if |w| < 0.001 then w = 0 during the remaining epochs. This ensures that the learned model finds not only a function of the right parametric form, but also fits the observed values as closely as possible. We observed that the exact choice of breakpoints t1 and t2 is not critical. In practice, we use t1 = 14T and t2 = 19 20T , where T is total number of update steps. T was selected large enough to ensure convergence. Note, that convergence to a sparse structure is important here, so early stopping will be disadvantageous.
MODEL SELECTION FOR EXTRAPOLATION
EQL networks have a number of hyper-parameters, e. g. the number of layers, the number of units and the regularization constant. Unfortunately, standard techniques for model selection, such as evaluation on a hold-out set or cross-validation, will not be optimal for our purpose, since they rely on interpolation quality. In order to extrapolate the network has to find the “right” formula. But how can we tell? Using Occams razor principle: the simplest formula is most likely the right one. Intuitively, if we have the choice between cos(x) and its truncated power series approximation 1− x2/2 + x4/24, the first one is preferred. We use the number of active hidden units in the network as a proxy for the complexity of the formula, see Appendix A1 for details. One could also think of differentiating between the unit types. In any case, this argumentation is only correct if the model explains the data well, i. e. it has a low validation error. So we have a dual objective to minimize, which we solve by ranking the instances w. r. t. validation error and sparsity and select the one with the smallest L2 norm (in rank-space), see Eq. (15).
Furthermore, the optimization process may only find a local optimum of the training objective, which depends on the initialization of the parameters. We use independent runs to quantify expected performance deviations.
RELATED WORK
In the field of machine learning, regression is often treated as a black box process of identifying a suitable real-valued function from a hypothesis set, e. g. a reproducing kernel Hilbert space for Gaussian Processes Regression (GPR) Williams & Rasmussen (2006) or Support Vector Regression (SVR) Smola & Schölkopf (2004), or a multi-layer network of suitable expressive power Specht (1991). The goal is to find a prediction function that leads to a small expected error on future data, not
necessarily to gain insight into the mechanism of how the output values derive from the inputs. The goal of finding an interpretable function is rather common in the natural sciences, such as biology, where high noise levels and strong inter-system variability often make it important to rely on external prior knowledge, and finding a “biologically plausible” model is often preferable over finding one that makes the highest prediction accuracy. As a consequence, model classes are often highly constrained, e. g. allowing only for sparse linear models.
The task of learning a true, nonlinear, functional dependence from observing a physical system, has received little attention in the machine learning literature so far, but forms the basis of the field of system identification. There, typically the functional form of the system is known and only the parameters have to be identified. Another approach is to model the time evolution with autoregressive models or higher order convolution integrals (Volterra series) but learning analytic formulas is not common.
Causal learning is an area of recent research that aims at identifying a causal relation between multiple observables, which are typically the result of a physical process. Classically, this tasks reduces to finding a minimal graphical model based only on tests of conditional independence Pearl (2000). Although very successful in some fields, this classical approach only provides a factorization of the problem, separating causes and effects, but it leaves the exact functional dependency unexplained. Recent extensions of causal learning can take a functional view, but typically do not constrain the regression functions to physically plausible ones, but rather constrain the noise distributions Peters et al. (2014). The topic of learning a regression function with emphasis on extrapolation performance has not been studied much in the literature so far. Existing work on time series prediction deals with extrapolation in the temporal domain, i. e. predict the next value(s) Wiener (1949). By our nomenclature, this is typically rather an interpolation task, when the prediction is based on the behaviour of the series at earlier time steps but with similar value distribution Müller et al. (1997); Györfi et al. (2013). Extrapolating in the data domain implies that the data distribution at prediction time will differ from the data distribution at training time. This is traditionally called the domain adaptation setting. In particular, since we assume a common labeling function, our setting would fall under the covariate shift setting Quionero-Candela et al. (2009). Unfortunately, this connection is not particularly useful for our problem. As domain adaptation typically does not make additional assumptions about how the data distribution may change, existing methods need access to some unlabeled data from the test distribution already at training time Ben-David et al. (2010). In our setting this is not possible to obtain.
On the technical level, EQL networks are an instance of general feed-forward networks for function approximation Bishop (1995). In contrast to recent trends towards deep learning Bengio (2009); Bengio et al. (2013), our goal is not to learn any data representation, but to learn a function which compactly represents the input-output relation and generalizes between different regions of the data space, like a physical formula. Structurally, EQL networks resemble sum-product networks (SPNs) Poon & Domingos (2012) and Pi-Sigma networks (PSNs) Shin & Ghosh (1991), in the sense that both are based on directed acyclic graphs with computational units that allows for summation and multiplication. Otherwise, SPNs are different as they act as efficient alternative to probabilistic graphical models for representing probability distributions, whereas EQL networks are meant for the classical task of function approximation. In PSNs each output needs to be passed through multiplicative units, whereas in EQL multiplication is optional.
Finding equations for observations is also known as symbolic regression where a search is performed in a certain function space, typically done with evolutionary computation. With these techniques it is possible to discover physical laws such as invariants and conserved quantities Schmidt & Lipson (2009). Unfortunately, the computational complexity/search time explodes for larger expressions and high-dimensional problems. We attempt to circumvent this by modeling it as a gradient based optimization problem. Related to symbolic regression is finding mathematical identities for instance to find computationally more efficient expressions. In Zaremba et al. (2014) this was done using machine learning to overcome the potentially exponential search space.
EXPERIMENTAL EVALUATION
We demonstrate the ability of EQL to learn physically inspired models with good extrapolation quality by experiments on synthetic and real data. For this, we implemented the network training and
evaluation procedure in python based on the theano framework Theano Development Team (2016). We will make the code for training and evaluation public after acceptance of the manuscript.
Pendulum. We first present the results of learning the equations of motion for a very simple physical system: a pendulum. The state space of a pendulum is X = R× R where the first value is the angle of the pole in radians and the second value is the angular velocity. In the physics literature, these are usually denoted as (θ, ω), but for our purposes, we call them (x1, x2) in order to keep the notation consistent between experiments. The pendulum’s dynamic behavior is governed by the following two ordinary differential equations:
ẋ1 = x2 and ẋ2 = −g sin(x1) , (9) where g = 9.81 is the gravitation constant.
We divide each equation by g in order to balance the output scales and form a regression problem with two output values, y1 = 1gx2 and y2 = − sin(x1). As training data, we sample 1000 points uniformly in the hypercube [−h, h] × [−h, h] for h = 2. Note that this domain contains more than half of a sine period, so it should be sufficient to identify the analytic expression. The target values are disturbed by Gaussian noise with standard derivation σ = 0.01. We also define three test sets, each with 1000 points. The interpolation test set is sampled from the same data distribution as the training set. The extrapolation (near) test set contains data sampled uniformly from the data domain [− 3
2 h, 3 2 h] × [− 3 2 h, 3 2 h] \ [−h, h] × [−h, h], which is
relatively near the training region and the extrapolation (far) test set extends the region to further outside: [−2h, 2h]× [−2h, 2h]\ [−h, h]× [−h, h]. We train a 2-layer EQL and perform model selection among the hyper-parameters: the regularization strength λ ∈ 10{−7,−6.3,−6,−5.3,−5,−4.3,−4,−3.3,−3} and the number of nodes 1
4 u = v ∈ {1, 3, 5}. All weights are randomly initialized from a normal distribution with σ = √
1/(k′ + d). The unit selection I is set such that all unit types are equally often. To ensure convergence we chose T = 10000 epochs. We compare our algorithm to a standard multilayer perceptron (MLP) with tanh activation functions and possible hyperparameters: λ as for EQL, number of layers L ∈ {2, 3}, and number of neurons k ∈ {5, 10, 20}. A second baseline is given by epsilon support vector regression (SVR) Basak et al. (2007) with two hyperparameters C ∈ 10{−3,−2,−1,0,1,2,3,3.5} and ∈ 10{−3,−2,−1,0} using radial basis function kernel with width γ ∈ {0.05, 0.1, 0.2, 0.5, 1.0}.
Numeric results are reported in Tab. 1. As expected all models are able to interpolate well with a test error on the order of the noise level (σ = 0.01). For extrapolation however, the performance differ between the approaches. For MLP the prediction quality decreases quickly when leaving the training domain. SVR remains a bit better in the near extrapolation but also fails catastrophically on the far extrapolation data. EQL, on the other hand, extrapolates well, both near and far away from the training domain. The reasons can be seen in Figure 2: while the MLP and SVR simply learns a function that interpolates the training values, EQL finds the correct functional expression and therefore predicts the correct values for any input data.
Double pendulum kinematics. The second system we consider real double pendulum where the forward kinematics should be learned. For that we use recorded trajectories of a real double pendulum Schmidt & Lipson (2009). The task here is to learn the position of the tips of the double pendulum segments from the given joint angles (x1, x2). These positions where not measured such that we supply them by the following formula: y1 = cos(x1), y2 = cos(x1) + cos(x1 + x2), y3 = sin(x1), y4 = sin(x1) + sin(x1 + x2) where (y1, y3) and (y2, y4) correspond to x-y-coordinates of the first and second end-point respectively. The dataset contains two short trajectories. The first
covers only part of the domain (input as well as output) and consists of 819 samples where 10% was used as validation set (randomly sampled), see Fig. 3(a). The second trajectory corresponds to a behavior with several spins of both pendulum segments such that a much larger domain is covered. Nevertheless the angle values are confined to [−π, π]. We use this trajectory as extrapolation test set. The trajectory and the outputs of our method are shown in Fig. 3(b). The prediction for unseen domains is perfect, which is also illustrated in a systematic sweep, see Fig. 3(c). The performance of MLP is off already near the training domain. SVR is a bit better, but still does not give usable predictions for the test data, see also the root means square error in Fig. 3(d).
Model selection is performed to determine λ as above, u = v ∈ {3, 5}, (MLP: k ∈ {5, 10, 20}) and layer number L ∈ {2, 3}.
Robotic arms. A more complicated task is to learn the forward kinematics of multi-segment robotic arms. We consider planar arms with 3, 4, and 5 joints, where each segment is 0.5 units long. For training the arm is controlled by sinusoidal joint target angles with amplitude in [−π/2, π/2], each joint with a different frequency. The number of data points are: 3000, 6000, and 18000 for the 3, 4, and 5 segment arms respectively, with added noise as above. For testing extrapolation performance the amplitude [−π, π] was used. Note that the extrapolation space is much larger than the training space. The task is to predict the coordinates of the end-effector of the arms (kin-3-end, kin-4-end) and the coordinates of all segment positions kin-5-all. The numerical results, see Tab. 2, shows that our method is able to extrapolate in these cases. Model selection as above with u = v ∈ {10, 20}, (MLP: k ∈ {10, 50}) and layer number L ∈ {2, 3, 4}. To illustrate the dependence on the amount of
noise and the number of available training points we provide a quantification in Appendix A2. In short, increasing noise can be compensated by increasing amount of data to keep the performance.
Learning complex formula. In order to find out whether EQL can also learn more complicated formulas, we consider three examples with four-dimensional input and one-dimensional output:
y = 1/3 (sin(πx1) + sin (2πx2 + π/8) + x2 − x3x4) F-1 (10) y = 1/3 ( sin(πx1) + x2 cos(2πx1 + π/4) + x3 − x24 ) F-2 (11)
y = 1/3 ((1 + x2) sin(πx1) + x2x3x4) F-3 (12)
The first equation requires only one hidden layer to be represented. The second equation and third equation should requires two hidden layers. In particular, F-2 contains a product of x2 and cos and F-3 contains a product of three terms, and we use it to test if our restriction to only pairwise product units causes problems for more complex target functions. We follow the same procedure as in the pendulum case for building training and test sets, though with h = 1 as input data range. We use 10000 points for training set and validation set (90%-10% split) and 5000 points for each of the test sets. Model selection for EQL is performed as above using the number of layers L ∈ 2, 3, 4. The number of units is set to 14u = v = 10. For the MLP, we select L and λ from the same set as above as well as k ∈ {10, 30}.
Table 3 shows the numerical results. Again, all methods are able to interpolate, but only EQL achieves good extrapolation results, except for equation F-3. There it settles in 9 out of 10 cases into a local minimum and finds only an approximating equation that deviates outside the training domain. Interestingly, if we restrict the base functions to not contain cosine, the algorithm finds the right formula. Note, the sparsity of the correct formula is lower than those of the approximation, so it should be selected if found. Figure Fig. 4 illustrates the performance and the learned networks visually. It shows one of the model-selected instances for each case. For F-1 the correct formula was identified, so correct predictions can be made even far outside the training region (much further than illustrated). For F-2 the network provided us with a surprise, because it yields good extrapolation performance with only one hidden layer! How can it implement x2 cos(2πx1 + π/4)? Apparently it uses 1.21(cos(−2πx1+π+π/4+0.41x2)+sin(2πx1+π/4+0.41x2)) which is a good approximation for x2 ∈ [−2, 2]. The sparsity of this solution is 5 whereas the true solution needs at least 6, which explains its selection. For F-3 the suboptimal local minima uses some strange way of approximating (1 + x2) sin(x1) using (x1 + x1x2) cos(βx1), which deviates fast, however the true solution would be sparser but was not found. Only if we remove cosine from the base functions we get always the correct formula, see Fig. 4(c).
X-Ray transition energies. As a further example we consider data measured in atomic physics. When shooting electron beams onto atoms one can excite them and they consequently emit x-ray radiation with characteristic peak energies. For each element/isotope these energies are different as they correspond to the potential difference between the electron shells, such that one can identify elements in a probe this way. The data is taken from Deslattes et al. (2003), where we consider one specific transition, called the K α2 line, because it was measured for all elements. The true relationship between atomic number Z and transition energies is complicated, as it involves many body interactions and no closed-form solution exists. Nevertheless we can find out which relationships our system proposes. It is known that the main relationship isK α2 ∝ Z2 according to Moseley’s law. Further correction terms for elements with larger Z are potentially of higher order. We have data for elements with 10 ≤ Z ≤ 100, which is split into training/validation sets in the range [10, 91] (70/10 data points) and extrapolation test set in the interval [92, 100] (14 data points because of isotops). Since we have so little data we evaluate the performance for 10 independent training/validation
splits. The data is scaled to lie in [0, 1], i. e. x = Z/100 and y = Kα2/100000. Model selection is here based on validation error only. The selection for sparsity and validation error only yields the Z2 relationship. Mini-batch size is 2 here and T = 50000 was used. Figure 5 presents the data, the predictions, the learned formulas and the numerical results. EQL and SVR achieve similar performance and MLP is significantly worse. However, EQL also yields interpretable formulas, see Fig. 5(e) that can be used to gain insights into the potential relationship.
POOR EXTRAPOLATION OUT OF MODEL CLASS — CART-PENDULUM SYSTEM
Let us now go beyond our assumptions and consider cases where the true target function is not an element of the hypothesis set.
Consider a pendulum attached to a cart that can move horizontally along a rail but that is attached to a spring damper system, see Fig. 6(a). The system is parametrized by 4 unknowns: the position of the cart, the velocity of the cart, the angle of the pendulum and the angular velocity of the pendulum. We combine these into a four-dimensional vector x = (x1, . . . , x4).
We set up a regression problem with four outputs from the corresponding system of ordinary differential equations where y1 = ẋ1 = x3, y2 = ẋ2 = x4 and
y3 = −x1 − 0.01x3 + x24 sin (x2) + 0.1x4 cos (x2) + 9.81 sin (x2) cos (x2)
sin2 (x2) + 1 , (13)
y4 = −0.2x4 − 19.62 sin (x2) + x1 cos (x2) + 0.01x3 cos (x2)− x24 sin (x2) cos (x2)
sin2 (x2) + 1 .
The formulas contain divisions which are not included in our architecture due to their singularities. To incorporate them in a principled manner is left for future work. Thus, the cart-pendulum dynamics is outside the hypothesis class. In this case we cannot expect great extrapolation performance and this is confirmed by the experiments. In Fig. 6(b,c) the extrapolation performance is illustrated by slicing through the input space. The near extrapolation performance is still acceptable for both EQL and MLP, but as soon as the training region is left further even the best instances differ considerably from the true values, see also the numeric results in Tab. 4. The SVR is performing poorly also for near extrapolation range. Inspecting the learned expressions we find that the sigmoid functions are rarely used.
CONCLUSIONS
We presented a new network architecture called EQL that can learn analytic expressions that typically occur in equations governing physical, in particular mechanical, systems. The network is fully differentiable, which allows end-to-end training using backpropagation. By sequencing L1 regularization and fixing L0 norm we achieve sparse representations with unbiased estimation of factors within the learned equations. We also introduce a model selection procedure specifically designed to select for good extrapolation quality by a multiobjective criterion based on validation error and sparsity. The proposed method is able to learn functional relations and extrapolate them to unseen parts of the data space, as we demonstrate by experiments on synthetic as well as real data. The approach learns concise functional forms that may provide insights into the relationships within the data, as we show on physical measurements of x-ray transition energies.
The optimization problem is nontrivial and has many local minima. We have shown cases where the algorithm is not reliably finding the right equation but instead finds an approximation only, in which case extrapolation may be poor.
If the origin of the data is not in the hypothesis class, i. e. the underlying expression cannot be represented by the network and good extrapolation performance cannot be achieved. Thus it is important to increase the model class by incorporating more base functions which we will address in future work alongside the application to even larger examples. We expect good scaling capabilities to larger systems due to the gradient based optimization. Apart from the extrapolation we also expect improved interpolation results in high-dimensional spaces, where data is less dense.
ACKNOWLEDGMENTS
This work was in parts funded by the European Research Council under the European Union’s Seventh Framework Programme (FP7/2007-2013)/ERC grant agreement no. 308036: "Life-long learning of visual scene understanding" (L3ViSU). GM received funding from the People Programme (Marie Curie Actions) in FP7/2007-2013 under REA grant agreement no. 291734.
APPENDIX
A1: MODEL SELECTION DETAILS
QUANTIFYING SPARSITY
We actually want a measure of complexity of the formula, however, since it is not clear what is the right choice of a measure, we use the sparsity instead, by counting the number of active/used hidden units denoted by s. For a given network phi we get
s(φ) = L∑ l=1 k∑ i=1 Θ(|W (l)i,· | ∗ |W (l+1) ·,i | − 0.01) , (14)
where Θ is the heavyside function and 0.01 is an arbitrary threshold. For the multiplication units the norm of the incoming weights for both inputs are added (omitted to avoid clutter in the formula).
SELECTION CRITERIA
As stated in the main text, we strive to choose the model that is both simple and has good performance in terms of the validation set. Since both quantities have different scales, we proposed to choose them based on their ranking. Let rv(φ) and rs(φ) be the ranks of the network φ w. r. t. the validation error and sparsity s(φ)respectively, then the network with minimal squared rank norm is selected:
arg min φ
[ rv(φ)2 + rs(φ)2 ] (15)
In Fig. 7 the extrapolation performance of all considered networks for the kin2D-4-end dataset is visualized in dependence of validation error and the sparsity. It becomes evident that the best performing networks are both sparse and have a low validation error.
A2: DEPENDENCE ON NOISE AND NUMBER OF DATA POINTS
In order to understand how the method depends on the amount of noise and the number of datapoints we scan through the two parameters and present the empirical results in Fig. 8. In general the method is robust to noise and as expected, more noise can be compensated by more data.

Thank you for an interesting perspective on the neural approaches to approximate physical phenomenon. This paper describes a method to extrapolate a given dataset and predict formulae with naturally occurring functions like sine, cosine, multiplication etc.

Pros
- The approach is rather simple and hence can be applied to existing methods. The major difference is incorporating functions with 2 or more inputs which was done successfully in the paper.

- It seems that MLP, even though it is good for interpolation, it fails to extrapolate data to model the correct function. It was a great idea to use basis functions like sine, cosine to make the approach more explicit.

Cons
- Page 8, the claim that x2 cos(ax1 + b) ~ 1.21(cos(-ax1 + π + b + 0.41x2) + sin(ax1 + b + 0.41x2)) for y in [-2,2] is not entirely correct. There should be some restrictions on 'a' and 'b' as well as the approximate equality doesn't hold for all real values of 'a' and 'b'. Although, for a=2*pi and b=pi/4, the claim is correct so the model is predicting a correct solution within certain limits.

- Most of the experiments involve up to 4 variables. It would be interesting to see how the neural approach models hundreds of variables.

- Another way of looking at the model is that the non-linearities like sine, cosine, multiplication act as basis functions. If the data is a linear combination of such functions, the model will be able to learn the weights. As division is not one of the non-linearities, predicting expressions in Equation 13 seems unlikely. Hence, I was wondering, is it possible to make sure that this architecture is a universal approximator.

Suggested Edits
- Page 8, It seems that there is a typographical error in the expression 1.21(cos(ax1 + π + b + 0.41x2) + sin(ax1 + b + 0.41x2)). When compared with the predicted formula in Figure 4(b), it should be 1.21(cos(-ax1 + π + b + 0.41x2) + sin(ax1 + b + 0.41x2)).


TRANSFORMATIONAL SPARSE CODING
A fundamental problem faced by object recognition systems is that objects and their features can appear in different locations, scales and orientations. Current deep learning methods attempt to achieve invariance to local translations via pooling, discarding the locations of features in the process. Other approaches explicitly learn transformed versions of the same feature, leading to representations that quickly explode in size. Instead of discarding the rich and useful information about feature transformations to achieve invariance, we argue that models should learn object features conjointly with their transformations to achieve equivariance. We propose a new model of unsupervised learning based on sparse coding that can learn object features jointly with their affine transformations directly from images. Results based on learning from natural images indicate that our approach matches the reconstruction quality of traditional sparse coding but with significantly fewer degrees of freedom while simultaneously learning transformations from data. These results open the door to scaling up unsupervised learning to allow deep feature+transformation learning in a manner consistent with the ventral+dorsal stream architecture of the primate visual cortex.
1 INTRODUCTION
A challenging problem in computer vision is the reliable recognition of objects under a wide range of transformations. Approaches such as deep learning that have achieved success in recent years usually require large amounts of labeled data, whereas the human brain has evolved to solve the problem using an almost unsupervised approach to learning object representations. During early development, the brain builds an internal representation of objects from unlabeled images that can be used in a wide range of tasks.
Much of the complexity in learning efficient and general-purpose representations comes from the fact that objects can appear in different poses, at different scales, locations, orientations and lighting conditions. Models have to account for these transformed versions of objects and their features. Current successful approaches to recognition use pooling to allow limited invariance to two-dimensional translations (Ranzato et al. (2007)). At the same time pooling discards information about the location of the detected features. This can be problematic because scaling to large numbers of objects requires modeling objects in terms of parts and their relative pose, requiring the pose information to be retained.
Previous unsupervised learning techniques such as sparse coding (Olshausen & Field (1997)) can learn features similar to the ones in the visual cortex but these models have to explicitly learn large numbers of transformed versions of the same feature and as such, quickly succumb to combinatorial explosion, preventing hierarchical learning. Other approaches focus on computing invariant object signatures (Anselmi et al. (2013; 2016)), but are completely oblivious to pose information.
Ideally, we want a model that allows object features and their relative transformations to be simultaneously learned, endowing itself with a combinatorial explanatory capacity by being able to apply learned object features with object-specific transformations across large numbers of objects. The goal of modeling transformations in images is two-fold: (a) to facilitate the learning of
pose-invariant sparse feature representations, and (b) to allow the use of pose information of object features in object representation and recognition.
We propose a new model of sparse coding called transformational sparse coding that exploits a tree structure to account for large affine transformations. We apply our model to natural images. We show that our model can extract pose information from the data while matching the reconstruction quality of traditional sparse coding with significantly fewer degrees of freedom. Our approach to unsupervised learning is consistent with the concept of “capsules” first introduced by Hinton et al. (2011), and more generally, with the dorsal-ventral (features+transformations) architecture observed in the primate visual cortex.
2 TRANSFORMATIONAL SPARSE CODING
2.1 TRANSFORMATION MODEL
Sparse coding (Olshausen & Field (1997)) models each image I as a sparse combination of features:
I ' Fw s.t. w is sparse
Sparsity is usually enforced by the appropriate penalty. A typical choice is S1(w) = ‖w‖1. We can enhance sparse coding with affine transformations by transforming features before combining them. The vectorized input image I is then modeled as:
I = K∑ k=1 wkT (xk)Fk
where wk, Fk denote the k-th weight specific to the image and the k-th feature respectively and T (xk) is a feature and image specific transformation.
In modeling image transformations we follow the approach of Rao & Ruderman (1999) and Miao & Rao (2007). We consider the 2D general affine transformations. These include rigid motions such as vertical and horizontal translations and rotations, as well as scaling, parallel hyperbolic deformations along the X/Y axis and hyperbolic deformations along the diagonals. A discussion on why these are good candidates for inclusion in a model of visual perception can be found in Dodwell (1983). Figure 5 in Appendix A shows the effects of each transformation.
Any subset of these transformations forms a Lie group with the corresponding number of dimensions (6 for the full set). Any transformation in this group can be expressed as the matrix exponential of a weighted combination of matrices (the group generators) that describe the behaviour of infinitesimal transformations around the identity:
T (x) = e ∑ j xjGj
For images of M pixels, T (x) is a matrix of size M ×M . Note that the generator matrices and the features used are common across all images. The feature weights and transformation parameters can be inferred (and the features learned) by gradient descent on the regularized MSE objective:
L(w, x, F ) = 1
N N∑ i=1 ∥∥∥∥∥Ii − K∑ k=1 wikT (xik)Fk ∥∥∥∥∥ 2
2
+ λwS1(w) + λF ‖F‖22
Although this model ties sparse coding with transformations elegantly, learning large transformations with it is intractable. The error surface of the loss function is highly non-convex with many shallow local minima. Figures 1(a),1(b),1(c) show the surface of L as a function of horizontal and vertical translation, horizontal translation and rotation and vertical translation and rotation parameters. The model tends to settle for small transformations around the identity. Due to the size of the parameters that we need to maintain, a random restart approach would be infeasible.
2.2 TRANSFORMATION TREES
We introduce Transformational Sparse Coding Trees to circumvent this problem using hierarchies of transformed features. The main idea is to gradually marginalize over an increasing range of
transformations. Each node in the tree represents a feature derived as a transformed version of its parent, with the root being the template of the feature. The leaves are equivalent to a set of sparse basis features and are combined to reconstruct the input as described above. A version of the model using a forest of trees of depth one (flat trees), is given by:
I ' V∑ v=1 ∑ b∼ch(v) wbUb
where Ub = T (xv→b)Fv and ch(v) the children of root v. The feature Ub is a leaf, derived from the root feature Fv via the fixed (across all data-points) transformation T (xv→b). Deeper trees can be built accordingly (Section 3.3). A small example of a tree learned from natural image patches is shown in Figure 2.
There are multiple advantages to such a hierarchical organization of sparse features. Some transformations are more common in data than others. Each path in the tree corresponds to a transformation that is common across images. Such a path can be viewed as a “transformation feature” learned from the data. Each additional node in the tree “costs” a fixed set of new parameters equal in size to the dimensions of the underlying Lie group (six in our case). At the same time the node contributes a whole new feature to the sparse code. Averaging over many data points, smoothens the surface of the error function and makes larger transformations more accessible to optimization. Figures 1(d),1(e),1(f) show the error surface averaged over a batch of 2000 patches.
For every leaf that is activated, the root template represents the identity of the feature and the transformation associated with the path to the root, the pose. In other words the tree is an equivariant representation of the feature over the parameter region defined by the set of paths to the leaves, very similar to the concept of a capsule introduced by Hinton et al. (2011). In fact, every increasing subtree corresponds to a capsule of increasing size.
2.3 LEARNING
The reconstruction mean squared-error (MSE) for a forest of flat trees is given by:
LMSE(w, x, F ) = 1
N N∑ i=1 ∥∥∥∥∥∥Ii − V∑ v=1 ∑ b∼ch(v) wibT (xv→b)Fv ∥∥∥∥∥∥ 2
2
Increasing the feature magnitudes and decreasing the weights will result in a decrease in loss. We constraint the root feature magnitudes to be of unit `2 norm. Consider different, transformed, versions of the same root template. For every such version there is a set of tree parameters that compensates for the intrinsic transformation of the root and results in the same leaves. To make the solution unique we directly penalize the transformation parameter magnitudes. Since scaling and parallel deformation can also change the magnitude of the filter, we penalize them more to keep features/leaves close to unit norm. The full loss function of the model is:
L(w, x, F ) = LMSE(w, x, F ) + λwS1(w) + 6∑ j=1 λjX[j]
s.t. ∀v, ‖Fv‖2 = 1
where X[j] is the vector of the collective parameters for generator Gj .
Lee et al. (2007) use an alternating optimization approach to sparse coding. First the weights are inferred using the feature sign algorithm and then the features are learned using a Lagrange dual approach. We use the same approach for the weights. Then we optimize the transformation parameters using gradient descent. The root features can be optimized using the analytical solution and projecting to unit norm.
The matrix exponential gradient ∂L∂x can be computed using the following formula (Ortiz et al. (2001)):
∂eA(t)
∂t = ∫ 1 0 eαA(t) ∂A(t) ∂t e(1−α)A(t)dα
The formula can be interpreted as: Eα∼U(0,1) [D(α)]
where D(α) = eαA(t) ∂A(t)∂t e (1−α)A(t). For our experiments we approximated the gradient by drawing a few samples 1 {α̃s}Ss=1 and computingEα̃ [D(α̃)]. This can be regarded as a stochastic version of the approach used by Culpepper & Olshausen (2009).
Some features might get initialized near shallow local optima (i.e. close to the borders or outside the receptive field). These features eventually become under-used by the model. We periodically check
1In practice even a single sample works well. The computation over samples is easily parallelizable.
for under-used features and re-initialize their transformation parameters 2. For re-initialization we select another feature in the same tree at random with probability proportional to the fraction of data points that used it in that batch. We then reset the transformation parameters at random, with small variance and centered around the chosen filter’s parameters.
3 EXPERIMENTS
3.1 LEARNING REPRESENTATIONS
We apply transformational sparse coding (TSC) with forests of flat trees to natural image patches. Our approach allows us to learn features resembling those of traditional sparse coding. Apart from reconstructing the input, the model also extracts transformation parameters from the data. Figure 3 shows a reconstruction example. Figure 4 shows the root features learned from 10 × 10 natural image patches using a forest of size 8 with branching factor 8, equipped with the full six-dimensional group. The forest has a total of 64 features. Figure 4(a) shows the features corresponding to the roots. Figure 4(b) shows the corresponding leaves. Each row contains features derived from the same root. More examples of learned features are shown in Figures 7, 8, 9 and 10 in Appendix ??.
3.2 COMPARISON WITH SPARSE CODING
Even though derivative features have to be explicitly constructed for inference, the degrees of freedom of our model are significantly lower than that of traditional sparse coding. Specifically:
dfTSC = (# of roots)× (# of pixels− 1 + branching factor× group dimension) whereas: dfSC = # of features× (# of pixels− 1) Note that the group dimension is equal to 3 for rigid motions and 6 for general 2D affine transformations.
We compare transformational sparse coding forests of various layouts and choices for λw with traditional sparse coding on 10 × 10 natural image patches. Some transformations change the feature magnitudes and therefore the sparsity pattern of the weights. To make the comparison clearer, for each choice of layout and penalty coefficient, we run sparse coding, constraining the feature magnitudes to be equal to the average feature magnitude of our model. The results are shown in Table 1. The reconstruction error of our model is close to that of sparse coding, albeit with slightly less sparse solutions, even though it has significantly fewer degrees of freedom. Our model extracts pose information in the form of group parameters.
2A feature is under-used when the total number of data-points using it in a batch drops close to zero.
(a)
3.3 DEEPER TREES
We can define deeper trees by associating a set of transformation parameters with each branch. These correspond to additive contributions to the complete transformation that yields the leaf when applied to the root:
Ii ' V∑ v=1 ∑ b∼ch(v) wibT (xP )Fv
where xP = ∑ e∈path(b,v) xe. Optimizing deeper trees is more demanding due to the increased number of parameters. Their advantage is that they lend structure to model. The parameters corresponding to the subtree of an internal node tend to explore the parameter subspace close to the transformation defined by that internal node. In tasks where it is disadvantageous to marginalize completely over transformations, equivariant representations corresponding to intermediate tree layers can be used. An example of such structure is shown in Figure 6 in Appendix B.
4 RELATED WORK
Sohl-Dickstein et al. (2010) present a model for fitting Lie groups to video data. Their approach only works for estimating a global transformation between consecutive video frames. They only support transformations of a single kind (ie only rotations). Different such single-parameter transformations have to be chained together to produce the global one. The corresponding transformation parameters also have to be inferred and stored in memory and cannot be directly converted to parameters of a single transformation. Kokiopoulou & Frossard (2009) present an approach to optimally estimating transformations between pairs of images. They support rigid motions and isotropic scaling. Culpepper & Olshausen (2009) focus on learning the group operators and transformation parameters from pairs of images, but do not learn features from data. Our model supports all six transformations and learns object parts and their individual transformations. In contrast with those approaches, our model learns object parts jointly with their transformations within the same image. Our model utilizes the full, six-dimensional, general affine Lie group and captures the pose of each object part in the form of a single set of six transformation parameters.
Grimes & Rao (2005) propose a bilinear model that combines sparse coding with transformations. The model accounts for global transformations that apply to the entire image region. Our model accounts for individual transformations of image parts. Rao & Ballard (1998) propose a model that captures small image transformations with Lie groups using a first-order Taylor approximation. Our model estimates larger transformations of image parts using the full exponential model. Rao & Ruderman (1999) and Miao & Rao (2007) use a first-order Taylor approximation to learn the group operators and the transformation parameters for small transformations.
The work closest to ours is that of Hinton et al. (2011) on capsules. A capsule learns to recognize its template (feature) over a wide range of poses. The pose is computed by a neural network (encoder). The decoder, resembling a computer graphics engine combines the capsule templates in different poses to reconstruct the image. Each transformational sparse coding tree can be thought of as capsule. The template corresponds to the root. The tree learns to “recognize” transformed versions of that template. Our work arrives at the concept of a capsule from a sparse coding perspective. A major difference is that our approach allows us to reuse each feature multiple times in different, transformed versions for each data point.
Gens & Domingos (2014) propose a convolutional network that captures symmetries in the data by modeling symmetry groups. Experiments with rigid motions or various affine transformations show reduced sample complexity. Cohen & Welling (2016) propose a convolutional network that can handle translations, reflections and rotations of 90 degrees. Dieleman et al. (2016) propose a network that handles translations and rotations. All the above are supervised learning models and apart from the first can handle a limited set of transformations. Our model is completely unsupervised, extends sparse coding and can handle all transformations given by the first order differential equation:
∂I(θ)
∂θ = AI(θ)
as described by Rao & Ruderman (1999).
5 CONCLUSION
In this paper, we proposed a sparse coding based model that learns object features jointly with their transformations, from data. Naively extending sparse coding for data-point specific transformations makes inference intractable. We introduce a new technique that circumvents this issue by using a tree structure that represents common transformations in data. We show that our approach can learn interesting features from natural image patches with performance comparable to that of traditional sparse coding.
Investigating the properties of deeper trees, learning the tree structure dynamically from the data and extending our model into a hierarchy are subjects of ongoing research.
A GENERATOR EFFECTS
Figure 5 presents the effects of each individual transformation of the six that are supported by our model. The template is a square 5(a).
B DEEPER TREES AND STRUCTURE
Figure 6 presents an example of structure learned by deeper trees. This example consists of vertical and horizontal lines. Each image patch is either blank, contains one vertical or one horizontal line or both. A patch is blank with probability 19 , contains exactly one line with probability 2 3 or two lines with probability 29 . Each line is then generated at one of eight positions at random. Fitting two binary trees results in some continuity in the features, whereas flat trees provide no such structure.
C LEARNED FEATURES
(a)

A new sparse coding model is introduced that learns features jointly with their transformations. It is found that inference over per-image transformation variables is hard, so the authors suggest tying these variables across all data points, turning them into global parameters, and using multiple transformations for each feature. Furthermore, it is suggested to use a tree of transformations, where each path down the tree generates a feature by multiplying the root feature by the transformations associated with the edges. The one-layer tree model achieves similar reconstruction error as traditional sparse coding, while using fewer parameters.
This is a nice addition to the literature on sparse coding and the literature on learning transformation models. The authors identify and deal with a difficult inference problem that can occur in transformation models. That said, I am skeptical about the usefulness of the general approach. The authors take it as a given that “learning sparse features and transformations jointly” is an important goal in itself, but this is never really argued or demonstrated with experiments. It doesn’t seem like this method enables new applications, extends our understanding of learning what/where pathways in the brain, or improve our ability to model natural images.
The authors claim that the model extracts pose information, but although the model explicitly captures the transformation that relates different features in a tree, at test time, inference is only performed over the (sparse) coefficient associated with each (feature, transformation) combination, just like in sparse coding. It is not clear what we gain by knowing that each coefficient is associated with a transformation, especially since there are many models that do this general “what / where” split.
It would be good to check that the x_{v->b} actually change significantly from their initialization values. The loss surface still looks pretty bad even for tied transformations, so they may actually not move much. Does the proposed model work better according to some measure, compared to a model where x_{v->b} are fixed and chosen from some reasonable range of parameter values (either randomly or spaced evenly)?
One of the conceptually interesting aspects of the paper is the idea of a tree of transformations, but the advantage of deeper trees is never demonstrated convincingly. It looks like the authors have only just gotten this approach to work on toy data with vertical and horizontal bars.
Finally, it is not clear how the method could be extended to have multiple layers. The transformation operators T can be defined in the first layer because they act on the input space, but the same cannot be done in the learned feature space. It is also not clear how the pose information should be further processed in a hierarchical manner, or how learning in a deep version should work.
In summary, I do not recommend this paper for publication, because it is not clear what problem is being solved, the method is only moderately novel and the novel aspects are not convincingly shown to be beneficial.