The loss surface of deep neural networks has recently attracted interest in the optimization and machine learning communities as a prime example of highdimensional non-convex problem. Some insights were recently gained using spin glass models and mean-field approximations, but at the expense of strongly simplifying the nonlinear nature of the model. In this work, we do not make any such assumption and study conditions on the data distribution and model architecture that prevent the existence of bad local minima. Our theoretical work quantifies and formalizes two important folklore facts: (i) the landscape of deep linear networks has a radically different topology from that of deep half-rectified ones, and (ii) that the energy landscape in the non-linear case is fundamentally controlled by the interplay between the smoothness of the data distribution and model over-parametrization. Our main theoretical contribution is to prove that half-rectified single layer networks are asymptotically connected, and we provide explicit bounds that reveal the aforementioned interplay. The conditioning of gradient descent is the next challenge we address. We study this question through the geometry of the level sets, and we introduce an algorithm to efficiently estimate the regularity of such sets on large-scale networks. Our empirical results show that these level sets remain connected throughout all the learning phase, suggesting a near convex behavior, but they become exponentially more curvy as the energy level decays, in accordance to what is observed in practice with very low curvature attractors.
1 INTRODUCTION
Optimization is a critical component in deep learning, governing its success in different areas of computer vision, speech processing and natural language processing. The prevalent optimization strategy is Stochastic Gradient Descent, invented by Robbins and Munro in the 50s. The empirical performance of SGD on these models is better than one could expect in generic, arbitrary non-convex loss surfaces, often aided by modifications yielding significant speedups Duchi et al. (2011); Hinton et al. (2012); Ioffe & Szegedy (2015); Kingma & Ba (2014). This raises a number of theoretical questions as to why neural network optimization does not suffer in practice from poor local minima.
The loss surface of deep neural networks has recently attracted interest in the optimization and machine learning communities as a paradigmatic example of a hard, high-dimensional, non-convex problem. Recent work has explored models from statistical physics such as spin glasses Choromanska et al. (2015), in order to understand the macroscopic properties of the system, but at the expense of strongly simplifying the nonlinear nature of the model. Other authors have advocated that the real danger in high-dimensional setups are saddle points rather than poor local minima Dauphin et al. (2014), although recent results rigorously establish that gradient descent does not get stuck on saddle points Lee et al. (2016) but merely slowed down. Other notable recent contributions are Kawaguchi (2016), which further develops the spin-glass connection from Choromanska et al. (2015) and resolves the linear case by showing that no poor local minima exist; Sagun et al. (2014) which also
∗Currently on leave from UC Berkeley.
discusses the impact of stochastic vs plain gradient, Soudry & Carmon (2016), that studies Empirical Risk Minimization for piecewise multilayer neural networks under overparametrization (which needs to grow with the amount of available data), and Goodfellow et al. (2014), which provided insightful intuitions on the loss surface of large deep learning models and partly motivated our work. Additionally, the work Safran & Shamir (2015) studies some topological properties of homogeneous nonlinear networks and shows how overparametrization acts upon these properties, and the pioneering Shamir (2016) studied the distribution-specific hardness of optimizing non-convex objectives. Lastly, several papers submitted concurrently and independently of this one deserve note, particularly Swirszcz et al. (2016) which analyzes the explicit criteria under which sigmoid-based neural networks become trapped by poor local minima, as well as Tian (2017), which offers a complementary study of two layer ReLU based networks, and their learning dynamics.
In this work, we do not make any linearity assumption and study conditions on the data distribution and model architecture that prevent the existence of bad local minima. The loss surface F (θ) of a given model can be expressed in terms of its level sets Ωλ, which contain for each energy level λ all parameters θ yielding a loss smaller or equal than λ. A first question we address concerns the topology of these level sets, i.e. under which conditions they are connected. Connected level sets imply that one can always find a descent direction at each energy level, and therefore that no poor local minima can exist. In absence of nonlinearities, deep (linear) networks have connected level sets Kawaguchi (2016). We first generalize this result to include ridge regression (in the two layer case) and provide an alternative, more direct proof of the general case. We then move to the half-rectified case and show that the topology is intrinsically different and clearly dependent on the interplay between data distribution and model architecture. Our main theoretical contribution is to prove that half-rectified single layer networks are asymptotically connected, and we provide explicit bounds that reveal the aforementioned interplay.
Beyond the question of whether the loss contains poor local minima or not, the immediate follow-up question that determines the convergence of algorithms in practice is the local conditioning of the loss surface. It is thus related not to the topology but to the shape or geometry of the level sets. As the energy level decays, one expects the level sets to exhibit more complex irregular structures, which correspond to regions where F (θ) has small curvature. In order to verify this intuition, we introduce an efficient algorithm to estimate the geometric regularity of these level sets by approximating geodesics of each level set starting at two random boundary points. Our algorithm uses dynamic programming and can be efficiently deployed to study mid-scale CNN architectures on MNIST, CIFAR-10 and RNN models on Penn Treebank next word prediction. Our empirical results show that these models have a nearly convex behavior up until their lowest test errors, with a single connected component that becomes more elongated as the energy decays. The rest of the paper is structured as follows. Section 2 presents our theoretical results on the topological connectedness of multilayer networks. Section 3 presents our path discovery algorithm and Section 4 covers the numerical experiments.
2 TOPOLOGY OF LEVEL SETS
Let P be a probability measure on a product space X ×Y , where we assume X and Y are Euclidean vector spaces for simplicity. Let {(xi, yi)}i be an iid sample of size L drawn from P defining the training set. We consider the classic empirical risk minimization of the form
Fe(θ) = 1
L L∑ l=1 ‖Φ(xi; θ)− yi‖2 + κR(θ) , (1)
where Φ(x; θ) encapsulates the feature representation that uses parameters θ ∈ RS and R(θ) is a regularization term. In a deep neural network, θ contains the weights and biases used in all layers. For convenience, in our analysis we will also use the oracle risk minimization:
Fo(θ) = E(X,Y )∼P ‖Φ(X; θ)− Y ‖2 + κR(θ) . (2)
Our setup considers the case whereR consists on either `1 or `2 norms, as we shall describe below. They correspond to well-known sparse and ridge regularization respectively.
2.1 POOR LOCAL MINIMA CHARACTERIZATION FROM TOPOLOGICAL CONNECTEDNESS
We define the level set of F (θ) as
ΩF (λ) = {θ ∈ RS ; F (θ) ≤ λ} . (3)
The first question we study is the structure of critical points of Fe(θ) and Fo(θ) when Φ is a multilayer neural network. For simplicity, we consider first a strict notion of local minima: θ ∈ RS is a strict local minima of F if there is > 0 with F (θ′) > F (θ) for all θ′ ∈ B(θ, ) and θ′ 6= θ. In particular, we are interested to know whether Fe has local minima which are not global minima. This question is answered by knowing whether ΩF (λ) is connected at each energy level λ: Proposition 2.1. If ΩF (λ) is connected for all λ then every local minima of F (θ) is a global minima.
Strict local minima implies that ∇F (θ) = 0 and HF (θ) 0, but avoids degenerate cases where F is constant along a manifold intersecting θ. In that scenario, if Uθ denotes that manifold, our reasoning immediately implies that if ΩF (λ) are connected, then for all > 0 there exists θ′ with dist(θ′,Uθ) ≤ and F (θ′) < F (θ). In other words, some element at the boundary of Uθ must be a saddle point. A stronger property that eliminates the risk of gradient descent getting stuck at Uθ is that all elements at the boundary of Uθ are saddle points. This can be guaranteed if one can show that there exists a path connecting any θ to the lowest energy level such that F is strictly decreasing along it.
Such degenerate cases arise in deep linear networks in absence of regularization. If θ = (W1, . . . ,WK) denotes any parameter value, with N1, . . . NK denoting the hidden layer sizes, and Fk ∈ GL+Nk(R) are arbitrary elements of the general linear group of invertible Nk × Nk matrices with positive determinant, then
Uθ = {W1F−11 , F1W2F −1 2 , . . . , FKWK ; Fk ∈ GL + Nk (R)} .
In particular, Uθ has a Lie Group structure. In the half-rectified nonlinear case, the general linear group is replaced by the Lie group of homogeneous invertible matrices Fk = diag(α1, . . . , αNk) with αj > 0.
This proposition shows that a sufficient condition to prevent the existence of poor local minima is having connected level sets, but this condition is not necessary: one can have isolated local minima lying at the same energy level. This can be the case in systems that are defined up to a discrete symmetry group, such as multilayer neural networks. However, as we shall see next, this case puts the system in a brittle position, since one needs to be able to account for all the local minima (and there can be exponentially many of them as the parameter dimensionality increases) and verify that their energy is indeed equal.
2.2 THE LINEAR CASE
We first consider the particularly simple case where F is a multilayer network defined by
Φ(x; θ) = WK . . .W1x , θ = (W1, . . . ,WK) . (4)
and the ridge regression R(θ) = ‖θ‖2. This model defines a non-convex (and non-concave) loss Fe(θ). When κ = 0, it has been shown in Saxe et al. (2013) and Kawaguchi (2016) that in this case, every local minima is a global minima. We provide here an alternative proof of that result that uses a somewhat simpler argument and allows for κ > 0 in the case K = 2. Proposition 2.2. Let W1,W2, . . . ,WK be weight matrices of sizes nk × nk+1, k < K, and let Fe(θ), Fo(θ) denote the risk minimizations using Φ as in (4). Assume that nj ≥ min(n1, nK) for j = 2 . . .K − 1. Then ΩFe(λ) (and ΩFo ) is connected for all λ and all K when κ = 0, and for κ > 0 when K = 2; and therefore there are no poor local minima in these cases. Moreover, any θ can be connected to the lowest energy level with a strictly decreasing path.
Let us highlight that this result is slightly complementary than that of Kawaguchi (2016), Theorem 2.3. Whereas we require nj ≥ min(n1, nK) for j = 2 . . .K − 1 and our analysis does not inform about the order of the saddle points, we do not need full rank assumptions on ΣX nor the weights Wk.
This result does also highlight a certain mismatch between the picture of having no poor local minima and generalization error. Incorporating regularization drastically changes the topology, and the fact that we are able to show connectedness only in the two-layer case with ridge regression is profound; we conjecture that extending it to deeper models requires a different regularization, perhaps using more general atomic norms Bach (2013). But we now move our interest to the nonlinear case, which is more relevant to our purposes.
2.3 HALF-RECTIFIED NONLINEAR CASE
We now study the setting given by
Φ(x; θ) = WKρWK−1ρ . . . ρW1x , θ = (W1, . . . ,WK) , (5)
where ρ(z) = max(0, z). The biases can be implemented by replacing the input vector x with x = (x, 1) and by rebranding each parameter matrix as
W i = ( Wi bi 0 1 ) ,
where bi contains the biases for each layer. For simplicity, we continue to use Wi and x in the following.
2.3.1 NONLINEAR MODELS ARE GENERALLY DISCONNECTED
One may wonder whether the same phenomena of global connectedness also holds in the halfrectified case. A simple motivating counterexample shows that this is not the case in general. Consider a simple setup with X ∈ R2 drawn from a mixture of two Gaussians N−1 and N1, and let Y = (X − µZ) · Z , where Z is the (hidden) mixture component taking {1,−1} values. Let Ŷ = Φ(X; {W1,W2}) be a single-hidden layer ReLU network, with two hidden units. Let θA be a configuration that bisects the two mixture components, and let θB the same configuration, but swapping the bisectrices. One can verify that they can both achieve arbitrarily small risk by letting the covariance of the mixture components go to 0. However, any path that connects θA to θB must necessarily pass through a point in which W1 has rank 1, which leads to an estimator with risk at least 1/2.
In fact, it is easy to see that this counter-example can be extended to any generic half-rectified architecture, if one is allowed to adversarially design a data distribution. For any given Φ(X; θ) with arbitrary architecture and current parameters θ = (Wi), let Pθ = {A1, . . . ,AS} be the underlying tessellation of the input space given by our current choice of parameters; that is, Φ(X; θ) is piece-wise linear and Pθ contains those pieces. Now let X be any arbitrary distribution with density p(x) > 0 for all x ∈ Rn, for example a Gaussian, and let Y | X d= Φ(X; θ) . Since Φ is invariant under a subgroup of permutations θσ of its hidden layers, it is easy to see that one can find two parameter values θA = θ and θB = θσ such that Fo(θA) = Fo(θB) = 0, but any continuous path γ(t) from θA to θB will have a different tessellation and therefore won’t satisfy Fo(γ(t)) = 0. Moreover, one can build on this counter-example to show that not only the level sets are disconnected, but also that there exist poor local minima. Let θ′ be a different set of parameters, and Y ′ | X d= Φ(X; θ′) be a different target distribution. Now consider the data distribution given by the mixture
X | p(x) , z ∼ Bernoulli(π) , Y | X, z d= zΦ(X; θ) + (1− z)Φ(X; θ′) . By adjusting the mixture component π we can clearly change the risk at θ and θ′ and make them different, but we conjecture that this preserves the status of local minima of θ and θ′. Appendix E constructs a counter-example numerically.
This illustrates an intrinsic difficulty in the optimization landscape if one is after universal guarantees that do not depend upon the data distribution. This difficulty is non-existent in the linear case and not easy to exploit in mean-field approaches such as Choromanska et al. (2015), and shows that in general we should not expect to obtain connected level sets. However, connectedness can be recovered if one is willing to accept a small increase of energy and make some assumptions on the complexity of the regression task. Our main result shows that the amount by which the energy is allowed to increase is upper bounded by a quantity that trades-off model overparametrization and smoothness in the data distribution.
For that purpose, we start with a characterization of the oracle loss, and for simplicity let us assume Y ∈ R and let us first consider the case with a single hidden layer and `1 regularization: R(θ) = ‖θ‖1.
2.3.2 PRELIMINARIES
Before proving our main result, we need to introduce preliminary notation and results. We first describe the case with a single hidden layer of size m.
We define
e(m) = min W1∈Rm×n,‖W1(i)‖2≤1,W2∈Rm
E{|Φ(X; θ)− Y |2}+ κ‖W2‖1 . (6)
to be the oracle risk using m hidden units with norm ≤ 1 and using sparse regression. It is a well known result by Hornik and Cybenko that a single hidden layer is a universal approximator under very mild assumptions, i.e. limm→∞ e(m) = 0. This result merely states that our statistical setup is consistent, and it should not be surprising to the reader familiar with classic approximation theory. A more interesting question is the rate at which e(m) decays, which depends on the smoothness of the joint density (X,Y ) ∼ P relative to the nonlinear activation family we have chosen. For convenience, we redefine W = W1 and β = W2 and Z(W ) = max(0,WX). We also write z(w) = max(0, 〈w,X〉) where (X,Y ) ∼ P and w ∈ RN is any deterministic vector. Let ΣX = EPXXT ∈ RN×N be the covariance operator of the random input X . We assume ‖ΣX‖ <∞. A fundamental property that will be essential to our analysis is that, despite the fact that Z is nonlinear, the quantity [w1, w2]Z := EP {z(w1)z(w2)} is locally equivalent to the linear metric 〈w1, w2〉X = EP {wT1 XXTw2} = 〈w1,ΣXw2〉, and that the linearization error decreases with the angle between w1 and w2. Without loss of generality, we assume here that ‖w1‖ = ‖w2‖ = 1, and we write ‖w‖2Z = E{|z(w)|2}. Proposition 2.3. Let α = cos−1(〈w1, w2〉) be the angle between unitary vectors w1 and w2 and let wm =
w1+w2 ‖w1+w2‖ be their unitary bisector. Then
1 + cosα
2 ‖wm‖2Z − 2‖ΣX‖
( 1− cosα
2 + sin2 α
) ≤ [w1, w2]Z ≤ 1 + cosα
2 ‖wm‖2Z . (7)
The term ‖ΣX‖ is overly pessimistic: we can replace it by the energy of X projected into the subspace spanned byw1 andw2 (which is bounded by 2‖ΣX‖). When α is small, a Taylor expansion of the trigonometric terms reveals that
2
3‖ΣX‖ 〈w1, w2〉 =
2
3‖ΣX‖ cosα =
2 3‖ΣX‖ (1− α
2
2 +O(α4))
≤ (1− α2/4)‖wm‖2Z − ‖ΣX‖(α2/4 + α2) +O(α4) ≤ [w1, w2]Z +O(α4) ,
and similarly [w1, w2]Z ≤ 〈w1, w2〉‖wm‖2Z ≤ ‖ΣX‖〈w1, w2〉 .
The local behavior of parameters w1, w2 on our regression problem is thus equivalent to that of having a linear layer, provided w1 and w2 are sufficiently close to each other. This result can be seen as a spoiler of what is coming: increasing the hidden layer dimensionality m will increase the chances to encounter pairs of vectors w1, w2 with small angle; and with it some hope of approximating the previous linear behavior thanks to the small linearization error.
In order to control the connectedness, we need a last definition. Given a hidden layer of size m with current parameters W ∈ Rn×m, we define a “robust compressibility” factor as
δW (l, α;m) = min ‖γ‖0≤l,supi |∠(w̃i,wi)|≤α
E{|Y − γZ(W̃ )|2 + κ‖γ‖1} , (l ≤ m) . (8)
This quantity thus measures how easily one can compress the current hidden layer representation, by keeping only a subset of l its units, but allowing these units to move by a small amount controlled by α. It is a form of n-width similar to Kolmogorov width Donoho (2006) and is also related to robust sparse coding from Tang et al. (2013); Ekanadham et al. (2011).
2.3.3 MAIN RESULT
Our main result considers now a non-asymptotic scenario given by some fixed size m of the hidden layer. Given two parameter values θA = (WA1 ,W A 2 ) ∈ W and θB = (WB1 ,WB2 ) with Fo(θ {A,B}) ≤ λ, we show that there exists a continuous path γ : [0, 1] → W connecting θA and θB such that its oracle risk is uniformly bounded by max(λ, ), where decreases with model overparametrization.
Theorem 2.4. For any θA, θB ∈ W and λ ∈ R satisfying Fo(θ{A,B}) ≤ λ, there exists a continuous path γ : [0, 1]→W such that γ(0) = θA, γ(1) = θB and
Fo(γ(t)) ≤ max(λ, ) , with (9)
= inf l,α
( max { e(l),δWA1 (m, 0;m), δWA1 (m− l, α;m), (10)
δWB1 (m, 0;m), δWB1 (m− l, α;m) } + C1α+O(α 2) ) , (11)
where C1 is an absolute constant depending only on κ and P .
Some remarks are in order. First, our regularization term is currently a mix between `2 norm constraints on the first layer and `1 norm constraints on the second layer. We believe this is an artifact of our proof technique, and we conjecture that more general regularizations yield similar results. Next, this result uses the data distribution through the oracle bound e(m) and the covariance term. The extension to empirical risk is accomplished by replacing the probability measure P by the empirical measure P̂ = 1L ∑ l δ ((x, y)− (xl, yl)). However, our asymptotic analysis has to be carefully reexamined to take into account and avoid the trivial regime when M outgrows L. A consequence of Theorem 2.4 is that as m increases, the model becomes asymptotically connected, as proven in the following corollary.
Corollary 2.5. As m increases, the energy gap satisfies = O(m− 1n ) and therefore the level sets become connected at all energy levels.
This is consistent with the overparametrization results from Safran & Shamir (2015); Shamir (2016) and the general common knowledge amongst deep learning practitioners. Our next sections explore this question, and refine it by considering not only topological properties but also some rough geometrical measure of the level sets.
3 GEOMETRY OF LEVEL SETS
3.1 THE GREEDY ALGORITHM
The intuition behind our main result is that, for smooth enough loss functions and for sufficient overparameterization, it should be “easy” to connect two equally powerful models—i.e., two models with FoθA,B ≤ λ. A sensible measure of this ease-of-connectedness is the normalized length of the geodesic connecting one model to the other: |γA,B(t)|/|θA − θB |. This length represents approximately how far of an excursion one must make in the space of models relative to the euclidean distance between a pair of models. Thus, convex models have a geodesic length of 1, because the geodesic is simply linear interpolation between models, while more non-convex models have geodesic lengths strictly larger than 1.
Because calculating the exact geodesic is difficult, we approximate the geodesic paths via a dynamic programming approach we call Dynamic String Sampling. We comment on alternative algorithms in Appendix A.
For a pair of models with network parameters θi, θj , each with Fe(θ) below a threshold L0, we aim to efficienly generate paths in the space of weights where the empirical loss along the path remains below L0. These paths are continuous curves belonging to ΩF (λ)–that is, the level sets of the loss function of interest.
Algorithm 1 Greedy Dynamic String Sampling 1: L0 ← Threshold below which path will be found 2: Φ1 ← randomly initialize θ1, train Φ(xi θ1) to L0 3: Φ2 ← randomly initialize θ2, train Φ(xi θ2) to L0 4: BeadList←(Φ1,Φ2) 5: Depth← 0 6: procedure FINDCONNECTION(Φ1,Φ2) 7: t∗ ← t such that dγ(θ1,θ2,t)dt ∣∣∣∣ t = 0 OR t = 0.5
8: Φ3 ← train Φ(xi; t∗θ1 + (1− t∗)θ2) to L0 9: BeadList← insert(Φ3, after Φ1, BeadList)
10: MaxError1 ← maxt(Fe(tθ3 + (1− t)θ1)) 11: MaxError2 ← maxt(Fe(tθ2 + (1− t)θ3)) 12: ifMaxError1 > L0 then return FindConnection(Φ1,Φ3) 13: ifMaxError2 > L0 then return FindConnection(Φ3,Φ2) 14: Depth← Depth+1
The algorithm recursively builds a string of models in the space of weights which continuously connect θi to θj . Models are added and trained until the pairwise linearly interpolated loss, i.e. maxtFe(tθi + (1 − t)θj) for t ∈ (0, 1), is below the threshold, L0, for every pair of neighboring models on the string. We provide a cartoon of the algorithm in Appendix C.
3.2 FAILURE CONDITIONS AND PRACTICALITIES
While the algorithm presented will faithfully certify two models are connected if the algorithm converges, it is worth emphasizing that the algorithm does not guarantee that two models are disconnected if the algorithm fails to converge. In general, the problem of determining if two models are connected can be made arbitrarily difficult by choice of a particularly pathological geometry for the loss function, so we are constrained to heuristic arguments for determining when to stop running the algorithm. Thankfully, in practice, loss function geometries for problems of interest are not intractably difficult to explore. We comment more on diagnosing disconnections more carefully in Appendix E.
Further, if the MaxError exceeds L0 for every new recursive branch as the algorithm progresses, the worst case runtime scales as O(exp(Depth)). Empirically, we find that the number of new models added at each depth does grow, but eventually saturates, and falls for a wide variety of models and architectures, so that the typical runtime is closer to O(poly(Depth))—at least up until a critical value of L0.
To aid convergence, either of the choices in line 7 of the algorithm works in practice—choosing t∗ at a local maximum can provide a modest increase in algorithm runtime, but can be unstable if the the calculated interpolated loss is particularly flat or noisy. t∗ = .5 is more stable, but slower. Finally, we find that training Φ3 to αL0 for α < 1 in line 8 of the algorithm tends to aid convergence without noticeably impacting our numerics. We provide further implementation details in 4.
4 NUMERICAL EXPERIMENTS
For our numerical experiments, we calculated normalized geodesic lengths for a variety of regression and classification tasks. In practice, this involved training a pair of randomly initialized models to the desired test loss value/accuracy/perplexity, and then attempting to connect that pair of models via the Dynamic String Sampling algorithm. We also tabulated the average number of “beads”, or the number intermediate models needed by the algorithm to connect two initial models. For all of the below experiments, the reported losses and accuracies are on a restricted test set. For more complete architecture and implementation details, see our GitHub page.
The results are broadly organized by increasing model complexity and task difficulty, from easiest to hardest. Throughout, and remarkably, we were able to easily connect models for every dataset and architecture investigated except the one explicitly constructed counterexample discussed in Appendix E.1. Qualitatively, all of the models exhibit a transition from a highly convex regime at high loss to a non-convex regime at low loss, as demonstrated by the growth of the normalized length as well as the monotonic increase in the number of required “beads” to form a low-loss connection.
4.1 POLYNOMIAL REGRESSION
We studied a 1-4-4-1 fully connected multilayer perceptron style architecture with sigmoid nonlinearities and RMSProp/ADAM optimization. For ease-of-analysis, we restricted the training and test data to be strictly contained in the interval x ∈ [0, 1] and f(x) ∈ [0, 1]. The number of required beads, and thus the runtime of the algorithm, grew approximately as a power-law, as demonstrated in Table 1 Fig. 1. We also provide a visualization of a representative connecting path between two models of equivalent power in Appendix D.
The cubic regression task exhibits an interesting feature around L0 = .15 in Table 1 Fig. 2, where the normalized length spikes, but the number of required beads remains low. Up until this point, the
cubic model is strongly convex, so this first spike seems to indicate the onset of non-convex behavior and a concomitant radical change in the geometry of the loss surface for lower loss.
4.2 CONVOLUTIONAL NEURAL NETWORKS
To test the algorithm on larger architectures, we ran it on the MNIST hand written digit recognition task as well as the CIFAR10 image recognition task, indicated in Table 1, Figs. 3 and 4. Again, the data exhibits strong qualitative similarity with the previous models: normalized length remains low until a threshold loss value, after which it grows approximately as a power law. Interestingly, the MNIST dataset exhibits very low normalized length, even for models nearly at the state of the art in classification power, in agreement with the folk-understanding that MNIST is highly convex and/or “easy”. The CIFAR10 dataset, however, exhibits large non-convexity, even at the modest test accuracy of 80%.
4.3 RECURRENT NEURAL NETWORKS
To gauge the generalizability of our algorithm, we also applied it to an LSTM architecture for solving the next word prediction task on the PTB dataset, depicted in Table 1 Fig. 5. Noteably, even for a radically different architecture, loss function, and data set, the normalized lengths produced by the DSS algorithm recapitulate the same qualitative features seen in the above datasets—i.e., models can be easily connected at high perplexity, and the normalized length grows at lower and lower perplexity after a threshold value, indicating an onset of increased non-convexity of the loss surface.
5 DISCUSSION
We have addressed the problem of characterizing the loss surface of neural networks from the perspective of gradient descent algorithms. We explored two angles – topological and geometrical aspects – that build on top of each other.
On the one hand, we have presented new theoretical results that quantify the amount of uphill climbing that is required in order to progress to lower energy configurations in single hidden-layer ReLU networks, and proved that this amount converges to zero with overparametrization under mild conditions. On the other hand, we have introduced a dynamic programming algorithm that efficiently approximates geodesics within each level set, providing a tool that not only verifies the connectedness of level sets, but also estimates the geometric regularity of these sets. Thanks to this information, we can quantify how ‘non-convex’ an optimization problem is, and verify that the optimization of quintessential deep learning tasks – CIFAR-10 and MNIST classification using CNNs, and next word prediction using LSTMs – behaves in a nearly convex fashion up until they reach high accuracy levels.
That said, there are some limitations to our framework. In particular, we do not address saddle-point issues that can greatly affect the actual convergence of gradient descent methods. There are also a number of open questions; amongst those, in the near future we shall concentrate on:
• Extending Theorem 2.4 to the multilayer case. We believe this is within reach, since the main analytic tool we use is that small changes in the parameters result in small changes in the covariance structure of the features. That remains the case in the multilayer case.
• Empirical versus Oracle Risk. A big limitation of our theory is that right now it does not inform us on the differences between optimizing the empirical risk versus the oracle risk. Understanding the impact of generalization error and stochastic gradient in the ability to do small uphill climbs is an open line of research.
• Influence of symmetry groups. Under appropriate conditions, the presence of discrete symmetry groups does not prevent the loss from being connected, but at the expense of increasing the capacity. An important open question is whether one can improve the asymptotic properties by relaxing connectedness to being connected up to discrete symmetry.
• Improving numerics with Hyperplane method. Our current numerical experiments employ a greedy (albeit faster) algorithm to discover connected components and estimate geodesics. We plan to perform experiments using the less greedy algorithm described in Appendix A.
ACKNOWLEDGMENTS
We would like to thank Mark Tygert for pointing out the reference to the -nets and Kolmogorov capacity, and Martin Arjovsky for spotting several bugs in early version of the results. We would also like to thank Maithra Raghu and Jascha Sohl-Dickstein for enlightening discussions, as well as Yasaman Bahri for helpful feedback on an early version of the manuscript. CDF was supported by the NSF Graduate Research Fellowship under Grant DGE-1106400.
A CONSTRAINED DYNAMIC STRING SAMPLING
While the algorithm presented in Sec. 3.1 is fast for sufficiently smooth families of loss surfaces with few saddle points, here we present a slightly modified version which, while slower, provides more control over the convergence of the string. We did not use the algorithm presented in this section for our numerical studies.
Instead of training intermediate models via full SGD to a desired accuracy as in step 8 of the algorithm, intermediate models are be subject to a constraint that ensures they are “close” to the neighboring models on the string. Specifically, intermediate models are constrained to the unique hyperplane in weightspace equidistant from its two neighbors. This can be further modified by additional regularization terms to control the “springy-ness” of the string. These heuristics could be chosen to try to more faithfully sample the geodesic between two models.
In practice, for a given model on the string, θi, these two regularizations augment the standard loss by: F̃ (θ) = F (θ) + ζ(‖θi−1− θi‖+ ‖θi+1− θi‖) +κ‖ (θi−1−θi+1)/2‖(θi−1−θi+1)/2‖ · (θi−(θi−1−θi+1)/2) ‖(θi−(θi−1−θi+1)/2)‖‖. The ζ regularization term controls the “springy-ness” of the weightstring, and the κ regularization term controls how far off the hyperplane a new model can deviate.
Because adapting DSS to use this constraint is straightforward, here we will describe an alternative “breadth-first” approach wherein models are trained in parallel until convergence. This alternative approach has the advantage that it will indicate a disconnection between two models “sooner” in training. The precise geometry of the loss surface will dictate which approach to use in practice.
Given two random models σi and σj where |σi − σj | < L0, we aim to follow the evolution of the family of models connecting σi to σj . Intuitively, almost every continuous path in the space of random models connecting σi to σj has, on average, the same (high) loss. For simplicity, we choose to initialize the string to the linear segment interpolating between these two models. If this entire segment is evolved via gradient descent, the segment will either evolve into a string which is entirely contained in a basin of the loss surface, or some number of points will become fixed at a higher loss. These fixed points are difficult to detect directly, but will be indirectly detected by the persistence of a large interpolated loss between two adjacent models on the string.
The algorithm proceeds as follows:
(0.) Initialize model string to have two models, σi and σj .
1. Begin training all models to the desired loss, keeping the instantaneous loss, L0(t), of all models being trained approximately constant.
2. If the pairwise interpolated loss between σn and σn+1 exceeds L0(t), insert a new model at the maximum of the interpolated loss (or halfway) between these two models.
3. Repeat steps (1) and (2) until all models (and interpolated errors) are below a threshold loss L0(tfinal) := L0, or until a chosen failure condition (see 3.2).
B PROOFS
B.1 PROOF OF PROPOSITION 2.1
Suppose that θ1 is a local minima and θ2 is a global minima, but F (θ1) > F (θ2). If λ = F (θ1), then clearly θ1 and θ2 both belong to ΩF (λ). Suppose now that ΩF (λ) is connected. Then we
could find a smooth (i.e. continuous and differentiable) path γ(t) with γ(0) = θ1, γ(1) = θ2 and F (γ(t)) ≤ λ = F (θ1). But this contradicts the strict local minima status of θ1, and therefore ΩF (λ) cannot be connected .
B.2 PROOF OF PROPOSITION 2.2
Let us first consider the case with κ = 0. We proceed by induction over the number of layers K. For K = 1, the loss F (θ) is convex. Let θA, θB be two arbitrary points in a level set Ωλ. Thus F (θA) ≤ λ and F (θB) ≤ λ. By definition of convexity, a linear path is sufficient in that case to connect θA and θB :
F ((1− t)θA + tθB) ≤ (1− t)F (θA) + tF (θB) ≤ λ . Suppose the result is true for K − 1. Let θA = (WA1 , . . . ,WAK ) and θB = (WB1 , . . . ,WBK ) with F (θA) ≤ λ, F (θB) ≤ λ. Since nj ≥ min(n1, nK) for j = 2 . . .K − 1, we can find k∗ = {1,K − 1} such that nk∗ ≥ min(nk∗−1, nk∗+1). For each W1, . . . ,WK , we denote W̃j = Wj for j 6= k∗, k∗ − 1 and W̃k∗ = Wk∗−1Wk∗ . By induction hypothesis, the loss expressed in terms of θ̃ = (W̃1, . . . , W̃K−1) is connected between θ̃A and θ̃B . Let W̃k∗(t) the corresponding linear path projected in the layer k∗. We need to produce a path in the variables Wk∗−1(t), Wk∗(t) such that:
i Wk∗−1(0) = WAk∗−1, Wk∗−1(1) = W B k∗−1,
ii Wk∗(0) = WAk∗ , Wk∗(1) = W B k∗ ,
iii Wk∗(t)Wk∗−1(t) = W̃k∗−1(t) for t ∈ (0, 1).
We construct it as follows. Let
Wk∗(t) = tW B k∗ + (1− t)WAk∗ + t(1− t)V ,
Wk∗−1(t) = Wk∗(t) †W̃k∗−1(t) ,
where Wk∗(t)† = (Wk∗(t)TWk∗(t))−1Wk∗(t)T denotes the pseudoinverse and V is a nk∗−1×nk∗ matrix drawn from a iid distribution. Conditions (i) and (ii) are immediate from the definition, and condition (iii) results from the fact that
Wk∗(t)Wk∗(t) † = IN∗k ,
since W ∗k (t) has full rank for all t ∈ (0, 1). Finally, let us prove that the result is also true when K = 2 and κ > 0. We construct the path using the variational properties of atomic norms Bach (2013). When we pick the ridge regression regularization, the corresponding atomic norm is the nuclear norm:
‖X‖∗ = min UV T=X
1 2 (‖U‖2 + ‖V ‖2) .
The path is constructed by exploiting the convexity of the variational norm ‖X‖∗. Let θA = (WA1 ,W A 2 ) and θ B = (WB1 ,W B 2 ), and we define W̃ = W1W2. Since W̃ {A,B} = W {A,B} 1 W {A,B} 2 , it results that
‖W̃ {A,B}‖∗ ≤ 1 2 (‖W {A,B}1 ‖2 + ‖W {A,B} 2 ‖2) . (12)
From (12) it results that the loss Fo(W1,W2) can be minored by another loss expressed in terms of W̃ of the form E{|Y − W̃X|2}+ 2κ‖W̃‖∗ , which is convex with respect to W̃ . Thus a linear path in W̃ from W̃A to W̃B is guaranteed to be below Fo(θ{A,B}). Let us define
∀ t , W1(t),W2(t) = arg min UV T=W̃ (t) (‖U‖2 + ‖V ‖2) .
One can verify that we can first consider a path (βA1 (s), β A 2 (s)) from (W A 1 ,W A 2 ) to (W1(0),W2(0) such that ∀ s β1(s)β2(s) = W̃A and ‖β1(s)‖2 + ‖β2(s)‖2 decreases ,
and similarly for (WB1 ,W B 2 ) to (W1(1),W2(1). The path (β A {1,2}(s),W{1,2}(t), β B {1,2}(s)) satisfies (i-iii) by definition. We also verify that
‖W1(t)‖2 + ‖W2(t)‖2 = 2‖W̃ (t)‖∗ ≤ 2(1− t)‖W̃ (0)‖∗ + 2t‖W̃ (1)‖∗ ≤ (1− t)(‖W‖21(0) + ‖W‖22(0)) + t(‖W‖21(1) + ‖W‖22(1)) .
Finally, we verify that the paths we have just created, when applied to θA arbitrary and θB = θ∗ a global minimum, are strictly decreasing, again by induction. For K = 1, this is again an immediate consequence of convexity. ForK > 1, our inductive construction guarantees that for any 0 < t < 1, the path θ(t) = (Wk(t))k≤K satisfies Fo(θ(t)) < Fo(θA). This concludes the proof .
B.3 PROOF OF PROPOSITION 2.3
Let A(w1, w2) = {x ∈ Rn; 〈x,w1〉 ≥ 0 , 〈x,w2〉 ≥ 0} .
By definition, we have
〈w1, w2〉Z = E{max(0, 〈X,w1〉) max(0, 〈X,w2〉)} (13)
= ∫ A(w1,w2) 〈x,w1〉〈x,w2〉dP (x) , (14)
= ∫ Q(A(w1,w2)) 〈Q(x), w1〉〈Q(x), w2〉(dP̄ (Q(x))) , (15)
whereQ is the orthogonal projection onto the space spanned byw1 andw2 and dP̄ (x) = dP̄ (x1, x2) is the marginal density on that subspace. Since this projection does not interfere with the rest of the proof, we abuse notation by dropping the Q and still referring to dP (x) as the probability density.
Now, let r = 12‖w1 + w2‖ = 1+cos(α) 2 and d = w2−w1 2 . By construction we have
w1 = rwm − d , w2 = rwm + d , and thus 〈x,w1〉〈x,w2〉 = r2|〈x,wm〉|2 − |〈x, d〉|2 . (16) By denoting C(wm) = {x ∈ Rn; 〈x,wm〉 ≥ 0}, observe that A(w1, w2) ⊆ C(wm). Let us denote by B = C(wm) \A(w1, w2) the disjoint complement. It results that
〈w1, w2〉Z = ∫ A(w1,w2) 〈x,w1〉〈x,w2〉dP (x)
= ∫ C(wm) [r2|〈x,wm〉|2 − |〈x, d〉|2]dP (x)−
r2 ∫ B |〈x,wm〉|2dP (x) + ∫ B |〈x, d〉|2dP (x)
= r2‖wm‖2Z − r2 ∫ B
|〈x,wm〉|2dP (x)︸ ︷︷ ︸ E1
− ∫ A(w1,w2)
|〈x, d〉|2dP (x)︸ ︷︷ ︸ E2 . (17)
We conclude by bounding each error term E1 and E2 separately: 0 ≤ E1 ≤ r2| sin(α)|2 ∫ B ‖x‖2dP (x) ≤ r2| sin(α)|22‖ΣX‖ , (18)
since every point in B by definition has angle greater than π/2− α from wm. Also, 0 ≤ E2 ≤ ‖d‖2 ∫ A(w1,w2) ‖x‖2dP (x) ≤ 1− cos(α) 2 2‖ΣX‖ (19)
by direct application of Cauchy-Schwartz. The proof is completed by plugging the bounds from (18) and (19) into (17) .
B.4 PROOF OF THEOREM 2.4
Consider a generic α and l ≤ m. A path from θA to θB will be constructed by concatenating the following paths:
1. from θA to θlA, the best linear predictor using the same first layer as θA,
2. from θlA to θsA, the best (m− l)-term approximation using perturbed atoms from θA, 3. from θsA to θ∗ the oracle l term approximation,
4. from θ∗ to θsB , the best (m− l)-term approximation using perturbed atoms from θB , 5. from θsB to θlB , the best linear predictor using the same first layer as θB ,
6. from θlB to θB .
The proof will study the increase in the loss along each subpath and aggregate the resulting increase into a common bound.
Subpaths (1) and (6) only involve changing the parameters of the second layer while leaving the firstlayer weights fixed, which define a convex loss. Therefore a linear path is sufficient to guarantee that the loss along that path will be upper bounded by λ on the first end and δWA1 (m, 0,m) on the other end.
Concerning subpaths (3) and (4), we notice that they can also be constructed using only parameters of the second layer, by observing that one can fit into a single n × m parameter matrix both the (m − l)-term approximation and the oracle l-term approximation. Indeed, let us describe subpath (3) in detail ( subpath (4) is constructed analogously by replacing the role of θsA with θsB). Let W̃A the first-layer parameter matrix associated with the m − l-sparse solution θsA, and let γA denote its second layer coefficients, which is a m-dimensional vector with at most m − l non-zero coefficients. LetW∗ be the first-layer matrix of the l-term oracle approximation, and γ∗ the corresponding second-layer coefficients. Since there are only m − l columns of W̃A that are used, corresponding to the support of γA, we can consider a path θ̄ that replaces the remaining l columns with those from W∗ while keeping the second-layer vector γA fixed. Since the modified columns correspond to zeros in γA, such paths have constant loss. Call W̄ the resulting first-layer matrix, containing both the active m− l active columns of W̃A and the l columns of W∗ in the positions determined by the zeros of γA. Now we can consider the linear subpath that interpolates between γA and γ∗ while keeping the first layer fixed at W̄ . Since again this is a linear subpath that only moves second-layer coefficients, it is non-increasing thanks to the convexity of the loss while fixing the first layer. We easily verify that at the end of this linear subpath we are using the oracle l-term approximation, which has loss e(l), and therefore subpath (3) incurs in a loss that is bounded by its extremal values δWA1 (m− l, α,m) and e(l).
Finally, we need to show how to construct the subpaths (2) and (5), which are the most delicate step since they cannot be bounded using convexity arguments as above. Let W̃A be the resulting perturbed first-layer parameter matrix withm− l sparse coefficients γA. Let us consider an auxiliary regression of the form W = [WA; W̃A] ∈ Rn×2m . and regression parameters
β1 = [β1; 0] , β2 = [0; γA] .
Clearly E{|Y − β1W |2}+ κ‖β1‖1 = E{|Y − β1WA|2}+ κ‖β1‖1
and similarly for β2. By convexity, the augmented linear path η(t) = (1− t)β1 + tβ2 thus satisfies
∀ t , L(t) = E{|Y − η(t)W |2}+ κ‖η(t)‖1 ≤ max(L(0), L(1)) .
Let us now approximate this augmented linear path with a path in terms of first and second layer weights. We consider
η1(t) = (1− t)WA + tW̃A , and η2(t) = (1− t)β1 + tγA .
We have that
Fo({η1(t), η2(t)}) = E{|Y − η2(t)Z(η1(t))|2}+ κ‖η2(t)‖1 (20) ≤ E{|Y − η2(t)Z(η1(t))|2}+ κ((1− t)‖β1‖1 + t‖γA‖1) = L(t) + E{|Y − η2(t)Z(η1(t))|2} − E{|Y − (1− t)β1Z(WA)− tγAZ(W̃A)|2} . (21)
Finally, we verify that∣∣∣E{|Y − η2(t)Z(η1(t))|2} − E{|Y − (1− t)β1Z(WA)− tγAZ(W̃A)|2}∣∣∣ ≤ (22) ≤ 4αmax(E|Y |2, √ E|Y 2|)‖ΣX‖(κ−1/2 + α √ E|Y 2|κ−1) + o(α2) .
Indeed, from Proposition 2.3, and using the fact that ∀ i ≤M, t ∈ [0, 1] , ∣∣∠((1− t)wAi + tw̃Ai ;wAi )∣∣ ≤ α , ∣∣∠((1− t)wAi + tw̃Ai ; w̃Ai )∣∣ ≤ α
we can write (1− t)β1,iz(wAi ) + tγA,iz(w̃Ai ) d = η2(t)iz(η1(t)i) + ni ,
with E{|ni|2} ≤ 4|η2(t)i|2‖ΣX‖α2 + O(α4) and E|ni| ≤ 2|η2(t)i|α √ ‖ΣX‖ using concavity of the moments. Thus∣∣∣E{|Y − η2(t)Z(η1(t))|2} − E{|Y − (1− t)β1Z(WA)− tγAZ(W̃A)|2}∣∣∣ ≤ 2E
{∑ i (Y − η2(t)Z(η1(t)))ni } + E { | ∑ i ni|2 }
≤ 4 ( α √ E|Y 2|‖ΣX‖‖η2‖+ α2(‖η2‖1)2‖ΣX‖ ) ≤ 4αmax(1, √ E|Y 2|)‖ΣX‖(‖η2‖1 + α‖η2‖21) + o(α2)
≤ 4αmax( √ E|Y 2|,E|Y 2|)‖ΣX‖(κ−1 + α √ E|Y 2|κ−2) + o(α2) ,
which proves (22).
We have just constructed a path from θA to θB , in which all subpaths except (2) and (5) have energy maximized at the extrema due to convexity, given respectively by λ, δW 1A(m, 0,m), δW 1A(m − l, α,m), e(l), δW 1B (m − l, α,m), and δW 1B (m, 0,m). For the two subpaths (2) and (5), (22) shows that it is sufficient to add the corresponding upper bound to the linear subpath, which is of the form Cα + o(α2) where C is an explicit constant independent of θ. Since l and α are arbitrary, we are free to pick the infimum, which concludes the proof.
B.5 PROOF OF COROLLARY 2.5
Let us consider a generic first layer weight matrix W ∈ Rn×m. Without loss of generality, we can assume that ‖wk‖ = 1 for all k, since increasing the norm of ‖wk‖within the unit ball has no penalty in the loss, and we can compensate this scaling in the second layer thanks to the homogeneity of the half-rectification. Since this results in an attenuation of these second layer weights, they too are guaranteed not to increase the loss.
From Vershynin (2010) [Lemma 5.2] we verify that the covering number N (Sn−1, ) of the Euclidean unit sphere Sn−1 satisfies
N (Sn−1, ) ≤ ( 1 + 2 )n ,
which means that we can cover the unit sphere with an -net of size N (Sn−1, ).
Let 0 < η < n−1(1 + n−1)−1, and let us pick, for each m, m = m η−1 n . Let us consider its corresponding -net of size
um = N (Sn−1, m) ' ( 1 + 2
m
)n ' m1−η .
Since we have m vectors in the unit sphere, it results from the pigeonhole principle that at least one element of the net will be associated with at least vm = mu−1m ' mη vectors; in other words, we are guaranteed to find amongst our weight vector W a collection Qm of vm ' mη vectors that are all at an angle at most 2 m apart. Let us now apply Theorem 2.4 by picking n = vm and α = m. We need to see that the terms involved in the bound all converge to 0 as m→∞. The contribution of the oracle error e(vm) − e(m) goes to zero as m → ∞ by the fact that limm→∞ e(m) exists (it is a decreasing, positive sequence) and that vm →∞. Let us now verify that δ(m − vm, m,m) also converges to zero. We are going to prune the first layer by removing one by one the vectors in Qm. Removing one of these vectors at a time incurs in an error of the order of m. Indeed, let wk be one of such vectors and let β′ be the solution of
min β′ E(β′) = min β′=(βf ;βk)∈Rk E{|Y − βTf Z(W−k)− βkz(wk)|2}+ κ(‖βf‖1 + |βk|) ,
where W−k is a shorthand for the matrix containing the rest of the vectors that have not been discarded yet. Removing the vector wk from the first layer increases the loss by a factor that is upper bounded by E(βp)− E(β), where
(βp)j =
{ β′j for j < k − 1 ,
β′k−1 + β ′ k otherwise.
,
since now βp is a feasible solution for the pruned first layer.
Let us finally bound E(βp)− E(β). Since ∠(wk, wk−1) ≤ m, it results from Proposition 2.3 that
z(wk) d = z(wk−1) + n ,
with E{|n|2} ≤ Cα2 for some constantC independent ofm. By redefining p1 = Y −βTp Z(W−k)− 1 2n and p2 = 1 2n, we have
E{|Y − βTp Z(W−k)|2} − E{|Y − β′ T Z(W−k)− βkz(wk)|2}
= E{|p1 + p2|2} − E{|p1 − p2|2} = 4E{|p1p2|}
≤ √√√√E{∣∣∣∣Y − βTp Z(W−k)− 12n ∣∣∣∣2 }√ E{|n|2}
≤ (C + α)α ' m ,
where C only depends on E{|Y |2}. We also verify that ‖βp‖1 ≤ ‖β′‖1. It results that removing |Qm| of such vectors incurs an increase of the loss at most |Qm| m ' mηm η−1 n = mη+
η−1 n . Since we picked η such that η + η−1n < 0, this term converges to zero. The
proof is finished.
C CARTOON OF ALGORITHM
Refer to Fig. 2.
D VISUALIZATION OF CONNECTION
Because the weight matrices are anywhere from high to extremely high dimensional, for the purposes of visualization we projected the models on the connecting path into a three dimensionsal subspace. Snapshots of the algorithm in progress for the quadratic regression task are indicated in Fig. 3. This was done by vectorizing all of the weight matrices for all the beads for a given connecting path, and then performing principal component analysis to find the three highest weight projections for the collection of models that define the endpoints of segments for a connecting path—i.e., the
θi discussed in the algorithm. We then projected the connecting string of models onto these three directions.
The color of the strings was chosen to be representative of the test loss under a log mapping, so that extremely high test loss mapped to red, whereas test loss near the threshold mapped to blue. An animation of the connecting path can be seen on our Github page.
Finally, projections onto pairs of principal components are indicated by the black curves.
E A DISCONNECTION
E.1 A DISCONNECTION
As a sanity check for the algorithm, we also applied it to a problem for which we know that it is not possible to connect models of equivalent power by the arguments of section 2.3.1. The input data is 3 points in R2, and the task is to permute the datapoints, i.e. map {x1, x2, x3} → {x2, x3, x1}. This map requires at least 12 parameters in general for the three linear maps which take xi → xj for i, j ∈ {{1, 2}, {2, 3}, {3, 1}}. Our archticture was a 2-3-2 fully connected neural network with a single relu nonlinearity after the hidden layer—a model which clearly has 12 free parameters by construction. The two models we tried to connect were a single model, θ, and a copy of θ with the first two neurons in the hidden layer permuted, θ̃σ . The algorithm fails to converge when initialized with these two models. We provide a visualization of the string of models produced by the algorithm in Fig. 4.
In general, a persistent high interpolated loss between two neighboring beads on the string of models could arise from either a slowly converging, connected pair of models or from a truly disconnected pair of models. “Proving” a disconnection at the level of numerical experiments is intractable in general, but a collection of negative results—i.e., failures to converge—are highly suggestive of a true disconnection.

This work contributes to understanding the landscape of deep networks in terms of its topology and geometry. The paper analyzes the former theoretically, and studies the latter empirically. Although the provided contributions are very specific (ReLU nets with single hidden layer, and a heuristic to calculate the normalized geodesic), the results are original and of interest. Thus, they could potentially be used as stepping stones for deeper developments in this area.
Pros:
1. Providing new theory about existence of "poor" local minima for ReLU networks with a hidden unit that relies on input distribution properties as well as the size of the hidden layer.
2. Coming up with a heuristic algorithm to compute the normalized geodesic between two solution points. The latter reflects how curved the path between the two is.
Cons:
The results are very specific in both topology and geometry analysis.
1. The analysis is performed only over a "single" hidden layer ReLU network. Given the importance of depth in deep architectures, this result cannot really explain the kinds of architectures we are interested in practically.
2. The normalized geodesic criterion is somewhat limited in representing how easy it is to connect two equally good points. For example, there might exist a straight line between the two (which is considered as easy by the geodesic criterion), but this line might be going through a very narrow valley, challenging gradient based optimization algorithms (and thus extremely difficult to navigate in practice). In addition, the proposed algorithm for computing the normalized geodesic is a greedy heuristic, which as far as I can tell, makes it difficult to know how we can trust in the estimated geodesics obtained by this algorithm.
With all cons said, I stress that I understand both problems tackled in the paper are challenging, and thus I find the contributions valuable and interesting.


Relational embeddings have emerged as an excellent tool for inferring novel facts from partially observed knowledge bases. Recently, it was shown that some classes of embeddings can also be exploited to perform a simplified form of rule mining. By interpreting logical conjunction as a form of composition between relation embeddings, simplified logical theories can be mined directly in the space of latent representations. In this paper, we present a method to mine full-fledged logical theories, which are significantly more expressive, by casting the semantics of the logical operators to the space of the embeddings. In order to extract relevant rules in the space of relation compositions we borrow sparse reconstruction procedures from the field of compressed sensing. Our empirical analysis showcases the advantages of our approach.
Relational embeddings have emerged as an excellent tool for inferring novel facts from partially observed knowledge bases. Recently, it was shown that some classes of embeddings can also be exploited to perform a simplified form of rule mining. By interpreting logical conjunction as a form of composition between relation embeddings, simplified logical theories can be mined directly in the space of latent representations. In this paper, we present a method to mine full-fledged logical theories, which are significantly more expressive, by casting the semantics of the logical operators to the space of the embeddings. In order to extract relevant rules in the space of relation compositions we borrow sparse reconstruction procedures from the field of compressed sensing. Our empirical analysis showcases the advantages of our approach.
1 INTRODUCTION
Knowledge Bases (KB) capture relational knowledge about a domain of choice by modelling entities and facts relating them. In so doing, KBs allow for rich answers to user queries, as happens with the knowledge panels powered by the Google Knowledge Graph. Furthermore, KBs can be mined for rules, i.e. patterns of relations which are frequently found to hold in the KB. Mining theories from data is the task of Rule Mining (Dzeroski & Lavrac, 2000) and Inductive Logic Programming (Dzeroski & Lavrac, 1994; Muggleton et al., 1992).
Classical ILP methods mine theories by searching over the (exponentially large) space of logical theories, resorting to language biases and heuristics to simplify the learning problem. While powerful, pure ILP methods do not scale to large relational datasets, preventing them from mining Web-scale KBs such as YAGO (Hoffart et al., 2013) and DBpedia (Auer et al., 2007). Further, purely logical methods can not gracefully deal with noise. Next-generation miners that specialize on large KBs, such as AMIE (Galárraga et al., 2015), work around these issues by trading off theory expressiveness for runtime efficiency.
A general strategy for processing huge datasets is dimensionality reduction: instead of working on the original KB directly, one first squeezes it to a summary of manageable size, and then performs the required operations on the summary itself. Common summarization techniques for relational data include relational factorization (Nickel et al., 2011; London et al., 2013; Riedel et al., 2013) and representation learning (Bordes et al., 2011; Socher et al., 2013). The core idea is to learn compressed latent representations, or embeddings, of entities and relations able to reconstruct the original KB by minimizing a suitable reconstruction loss. Until recently, relational embeddings have been mostly employed for link prediction and knowledge base completion (Nickel et al., 2016).
However, Yang et al. (2015) have shown that low-dimensional representations can also be exploited to perform a simplified form of theory learning. Their paper shows that, under reasonable assumptions, a simple nearest neighbor algorithm can recover logical rules directly from the fixed size embeddings of a KB, with potential runtime benefits. Furthermore, since the embeddings generalize beyond the observed facts, the rules are implicitly mined over a completion of the KB. Despite the novelty of their insight, their proposed method has several major downsides. First, their simple approach is limited to extracting rules as conjunctions of relations, with no support for logical dis-
junction and negation. Second, the rules are mined independently of one another, which can lead to redundant theories and compromise generalization ability and interpretability.
Building on the insights of Yang et al. (2015), we propose a novel approach to theory learning from low-dimensional representations. We view theory learning as a special sparse recovery problem. In this setting, a logical theory is merely an algebraic combination of embedded relations that best reconstructs the original KB, in a sense that will be made clear later. The recovery problem can be solved with specialized compressed sensing algorithms, such as Orthogonal Matching Pursuit (Pati et al., 1993) or variants thereof. Our approach offers two key advantages: it automatically models the inter-dependency between different rules, discouraging redundancy in the learned theory, and it supports all propositional logic connectives, i.e. conjunction, disjunction, and negation. Our empirical analysis indicates that our method can mine satisfactory theories in realistic KBs, demonstrating its ability to discover diverse and interpretable sets of rules. Additionally, our method can in principle be applied to “deeper” embeddings, that is, embeddings produced by deep models that take into consideration both relational and feature-level aspects of the data.
The paper is structured as follows. In the next section we introduce the required background material. We proceed by detailing our approach in Section 3 and evaluating it empirically in Section 4. We discuss relevant related work in Section 5, and conclude with some final remarks in Section 6.
2 BACKGROUND
In this section we briefly overview the required background. Let us start with the notation we will use. We write column vectors x in bold-face, matricesX in upper-case, and third-order tensors X in calligraphic upper-case. Xk is the kth frontal slice of the tensor X , and vec(X) is the vectorization (flattening) of X . We denote the usual Frobenius matrix norm as ‖X‖F := √∑ ij x 2 ij , the number of non-zero entries as ‖X‖0, the set {1, . . . , n} as [n] and the Cartesian product of ` sets {1, . . . , n} as [n]`. We reserve typewriter fonts for logical entities Ann and relations motherOf.
Knowledge Bases and Theories. A knowledge base (KB) is a collection of known true facts about a domain of interest. As an example, a KB about kinship relations may include facts such as (Ann,motherOf,Bob), which states that Ann is known to be the mother of Bob. In the following we will use n and m to denote the number of distinct entities and relations in the KB, respectively. With a slight abuse of notation, we will refer to logical constants and relations (e.g. Ann and motherOf) by their index in the KB (e ∈ [n] or r ∈ [m], respectively). Triples not occurring in the KB are unobserved, i.e. neither true nor false.
Given an input KB, the goal of theory learning, also known as Inductive Logic Programming (Muggleton et al., 1992), is to induce a compact logical theory that both explains the observed facts and generalizes to the unobserved ones. Most ILP methods extract theories in definite clausal form, which offers a good compromise between expressiveness and efficiency. A theory in this form is an implicitly conjoined set of Horn rules, i.e. rules like:
∀ e, e′ ∈ [n] (e,uncleOf, e′)⇐ ∃ e′′ ∈ [n] (e,brotherOf, e′′) ∧ (e′′,parentOf, e′)
Here ⇐ represents logical entailment. The left-hand side is called the head of the rule, while the right-hand side is the body. The semantics of Horn rules are simple: whenever the body is satisfied by a given set of entities and relations, so is the head. The length of a rule is the number of relations appearing in its body; the above is a length 2 rule.
Classical ILP approaches cast theory learning as a search problem over the (exponentially large) space of candidate theories. When there are no negative facts, as in our case, the quality of a theory is given by the number of true facts it entails. In practice, learning is regularized by the size of the theory (number and length of the rules) to encourage compression, generalization and interpretability. Due to the combinatorial nature of the problem, the search task is solved heuristically, e.g. by searching individual Horn rules either independently or sequentially, or by optimizing surrogate objective functions. A language bias, provided by a domain expert, is often employed to guide the search toward more promising theories. Please see (Dzeroski & Lavrac, 1994; Muggleton et al., 1992) for more details.
Relational embeddings. Relational embedding techniques learn a low-dimensional latent representation of a KB. In order to ground the discussion, we focus on a prototypical factorization method, RESCAL (Nickel et al., 2011; 2012); many alternative formulations can be seen as variations or generalizations thereof. We stress, however, that our method can be applied to other kinds of relational embeddings, as sketched in Section 6. For a general treatment of the subject, see Nickel et al. (2016).
In RESCAL, each entity e ∈ [n] in the KB is mapped to a vector xe ∈ Rd, and each binary relation r ∈ [m] to a matrix W r ∈ Rd×d. These parameters are learned from data. Here d ∈ [n] is a user-specified constant (the rank) controlling the amount of compression. The key idea is to model the plausibility, or score, of each fact as a function of its embedding. In particular, in RESCAL the score of a fact (e, r, e′) is given by the bilinear product:
score(e, r, e′) := (xe)>W rxe ′ = d∑ i=1 d∑ j=1 xeiW r ijx e′ j
The bilinear product measures how similar xe andW rxe ′
are: the higher the dot product, the higher the score.
The embeddings can be expressed compactly in tensor form by grouping the entity vectors sideby-side into a matrix X ∈ Rd×n, and stacking the relation matrices into a tensor W ∈ Rd×d×m. The embeddings (X,W) are learned so as to reconstruct the original KB as accurately as possible, modulo regularization. More formally, let Y ∈ {0, 1}n×n×m be a tensor such that Y ree′ evaluates to 1 if the fact (e, r, e′) appears in the KB, and to 0 otherwise. The learned embeddings should satisfy Y ree′ ≈ score(e, r, e′) for all possible triples (e, r, e′). Learning equates to solving the optimization problem:
min W,X m∑ r=1 ‖Y r −X>W rX‖2F + λ ( ‖X‖2F + m∑ r=1 ‖W r‖2F ) (1)
The second summand is a quadratic regularization term, whose impact is modulated by the λ > 0 hyperparameter. Note that the entity embeddings X are shared between relations. Choosing d n forces RESCAL to learn more compressed latent features, that hopefully better generalize over distinct facts, at the cost of a potentially larger reconstruction error. While the optimization problem is non-convex and can not be solved exactly in general, RESCAL pairs clever initialization with an alternating least squares procedure to obtain good quality solutions (Nickel et al., 2011).
In the next section we will see how theory learning can be generalized to work directly on the embeddings produced by RESCAL and analogous models.
3 RULE MINING IN FEATURE SPACE
In this section we detail our take on rule mining. Given a knowledge base in tensor form Y , our goal is to learn a theory T that (1) entails many of the observed facts and few of the unobserved ones, and (2) is composed of few, diverse rules, for improved generalization.
The theory T includes rules for all possible relations h ∈ [m], where the relation is the head of the rule and the body is an “explanation” of the relation as a (logical) combination of relations. Let Th be the set of rules for head h. In our setting, Th is a conjunction of Horn rules, where each rule is at most ` long 1; ` is provided by the user. Following Yang et al. (2015), we require the rules to be closed paths, i.e. to be in the following form:
(e1, h, e`+1)⇐ (e1, b1, e2) ∧ (e2, b2, e3) ∧ . . . ∧ (e`, b`, e`+1) (2)
Here h is the head relation, and b1, . . . , bl are the body relations; quantifiers have been left implicit. Formally, a Horn rule is a closed path if (i) consecutive relations share the middle argument, and (ii) the left argument of the head appears as the first argument of the body (and conversely for the right argument). This special form enables us to cast theory learning in terms of Boolean matrix operations, as follows.
1For the sake of exposition, in the following we only consider rules exactly ` long; as a matter of fact, the miners we consider can return rules of length ` or shorter.
Let Y be a knowledge base and h ∈ [m] the target head relation. Note that the conjunction of Horn rules with the same head relation h amounts to the disjunction of their bodies. Due to requirement (1), the set of rules targeting h should approximate the truth values of h, i.e.
Y h ≈ ∨ B∈Th ∧ b∈B Y b
Here B is the body of a rule, and the logical connectives operate element-wise. In order to learn T from Y , we define a loss function that encourages the above condition. We define the loss ∆(Y h, Th) as the accuracy of reconstruction of Y h w.r.t. Th, written as:
∆(Y h, Th) := ∥∥Y h ⊕∨B∈Th ∧b∈B Y b∥∥0 (3)
where ⊕ is the element-wise exclusive OR operator and ‖ · ‖0 computes the misclassification error of Th over Y h. Minimizing Eq. (3) unfortunately is a hard combinatorial problem. We will next show how to approximate the latter as a continuous sparse reconstruction problem.
The relaxed reconstruction problem. Our goal is to approximate Eq. (3) in terms of algebraic matrix operations over the relation embeddings W . First, we replace conjunctions with products between the embeddings of the relations along the path in the body of the rule, i.e.∧
b∈B Y b ≈ X> (∏ b∈BW b ) X
The idea is that a linear operator W b maps the embedding of the left argument of relation b to vectors similar to the embedding of the right one, as per Eq. 1. For instance, W motherOf will map the embedding of Ann to a vector with high dot product w.r.t. the embedding of Bob. The closed path represented by the conjunction of the relations in the body B is emulated by composition of embeddings and obtained by repeated applications of this mapping (Yang et al., 2015).
Second, we replace disjunctions with sums: Y h ≈ X>WhX ≈ X> [∑ B∈Th ∏ b∈BW b ] X
Intuitively, each path should represent an alternative explanation for the head relation, so that two entities are in relation h if at least one path (approximately) maps the left entity to the right one. Diversity between these alternatives will be enforced by imposing orthogonality between the mappings of the corresponding paths during the mining procedure, as explained later on in the section.
Clearly, the set of rules Th is unknown and needs to be learned in solving the reconstruction problem. We thus let the summation run over all possible paths of length `, i.e. [m]`, adding a coefficient αB for each candidate path. The problem boils down to learning these alphas:
minα ∥∥∥X>WhX −∑B∈[m]` αBX>∏b∈BW bX∥∥∥ F
(4)
In principle, the coefficients αB should be zero-one; however, we relax them to be real-valued to obtain a tractable optimization problem. This choice has another beneficial side effect: the relaxed formulation gives us a straightforward way to introduce negations in formulas, thus augmenting the expressiveness of our approach beyond purely Horn clauses. The idea builds on the concept of set difference from set theory. A relation like brotherOf can be explained by the rule “a sibling who is not a sister”. This could be represented in the space of the embeddings as the difference between the siblingOf mapping (accounting for both brothers and sisters) and the sisterOf one. More specifically, siblingOf ∧¬ sisterOf would be encoded as W siblingOf − W sisterOf. We thus allow α to also take negative values, with the interpretation that negative bodies are negated and conjoint (rather than disjoint) with the rest of the formula.
The last step is to get rid of the instances X , and mine rules for head h only in terms of their ability to reconstruct its embedding Wh. This is justified by the observation (Yang et al., 2015; Gu et al., 2015; Neelakantan et al., 2015; Garcı́a-Durán et al., 2015) that the embeddings are learned so that their composition is close to that of the embedding of the head.
Putting everything together, we obtain an optimization problem of the form:
minα ∥∥∥Wh −∑B∈[m]` αB∏b∈BW b∥∥∥ F
(5)
for each target head h. Upon finding the coefficients α, we convert them into a logic theory based on their sign and magnitude. First, only bodies with absolute coefficients larger than a threshold τ > 0 are retained. Each body is then converted to the conjuction of the relations it contains. Bodies with positive coefficients are disjunctively combined with the rest of the formula, while bodies with negative coefficients are added as conjunctions of their negations. The final theory for the mined rule can be written as:
Y h ≈ (∨
B:αB>τ ∧ b∈B Y b ) ∧ ¬ (∨ B:αB<−τ ∧ b∈B Y b )
(6)
Solving the reconstruction problem. Equation 5 is a matrix recovery problem in Frobenius norm. Instead of solving it directly, we leverage the norm equivalence ‖A−B‖F = ‖vec(A)− vec(B)‖2 to reinterpret it as a simpler vector recovery problem. Most importantly, since most of the candidate paths B can not explain the head h, the recovery problem is typically sparse. Sparse recovery problems are a main subject of study in compressed sensing (Candès et al., 2006), and a multitude of algorithms can be employed to solve them, including Orthogonal Matching Pursuit (OMP) (Pati et al., 1993), Basis Pursuit (Chen et al., 1998), and many recent alternatives. In Appendix A we show how minimizing the sparse recovery problem in Eq. 5 equates to minimizing an upper bound of the total loss.
Two features of the above problem stand out. First, if the target theory is sparse enough, existing recovery algorithms can solve the reconstruction to global optimality with high probability (Candès et al., 2006). We do not explicitly leverage this perk; we leave finding conditions guaranteeing perfect theory recovery to future work. Second and most importantly, reconstruction algorithms choose the non-zero coefficients αB so that the corresponding path embeddings ∏ b∈BW
b are mutually orthogonal. This means that similar paths will not be mined together, thus encouraging rule diversity, as per requirement (2).
4 EMPIRICAL EVALUATION
We compare our method, dubbed Feature Rule Miner (FRM for short), against two variants of the kNN-based theory miner of Yang et al. (2015) on four publicly available knowledge bases: Nations, Kinship and UMLS from Kemp et al. (2006), and Family from Fang et al. (2013). The KB statistics can be found in Table 1. Given that FRM requires the relational embeddings W to be normalized (with respect to the Frobenius norm), we compare it against both the original kNN-based miner, which mines the unnormalized embeddings, and a variant that uses the normalized embeddings instead, for the sake of fairness.
The miners were tested in a 10-fold cross-validation setting. We computed the relational embeddings over the training sets using non-negative RESCAL (Krompaß et al., 2013) 2 variant with the default parameters (500 maximum iterations, convergence threshold 10−5). The size of the embeddings d was set to a reasonable value for each KB: 100 for Family, 25 for Kinship and UMLS, and 5 for Nations. We configured all competitors to mine at most 100 rules for each head relation. The kNN distance threshold was set to 100 (although the actual value used is chosen dynamically, as done by Yang et al. (2015)). The desired reconstruction threshold of OMP was set ot 10−3. Finally, the coefficient threshold τ was set to 0.2.
We evaluate both the F-score and the per-rule recall of all the methods. The F-score measures how well the mined rules reconstruct the test facts in terms of both precision and recall. The per-rule recall is simply the recall over the number of rules mined for the target head; it favors methods that
2Standard RESCAL tends to penalize the kNN-based competitors.
focus on few rules with high coverage, and penalizes those that mine many irrelevant rules. The results on the four KBs (averaged over all target relations) are reported in Figures 1 and 2, and an example of mined rules in Figure 3. More detailed per-head results can be found in Appendix B. (Unfortunately, the normalized kNN method failed to work with the UMLS dataset; we left a blank in the plots.)
The plots show a clear trend: FRM performs better than the kNN-based methods in all four knowledge bases, both in terms of F-score and in terms of per-rule recall. Further, the normalized kNN variant tends to outperform the original, unnormalized version, providing support for our use of normalized relation embeddings.
Notably, the three methods mine similar amounts of rules. While OMP stops automatically when the mined body reconstructs the target head sufficiently well, the kNN methods compensate for the lack of a proper termination criterion by employing a distance-based pruning heuristict (as discussed by Yang et al. (2015)). Rather, the poor per-rule recall performance of the kNN methods can be imputed to insufficient rule diversity. The kNN miners discover the rules independently of each other, leading to theory redundancy. This is a well known problem in rule mining. On the contrary, OMP avoids this issue by enforcing orthogonality between the mined bodies. The resulting theories performs much better especially in terms of per-rule recall.
The phenomenon is also visible in Figure 3. The theory found by FRM contains many diverse bodies, while the one found by kNN does not. The two rules also show the power of negation: the FRM theory includes the “perfect” definition of a brother, i.e. siblingOf ∧ ¬sisterOf (as well as an obvious error, i.e. that a brother can not be a sibling of a sibling). In contrast the theory found by kNN completely ignores the complementarity of brotherOf and sisterOf, and includes the rule brotherOf⇐ sisterOf.
5 RELATED WORK
There is a huge body of work on theory learning, historically studied in Inductive Logic Programming (Dzeroski & Lavrac, 1994; Muggleton et al., 1992). For the sake of brevity, we focus on techniques that are more closely related to our proposal.
The core of most ILP methods, e.g. FOIL (Quinlan, 1990), Progol (Muggleton, 1995), and Aleph3, is a search loop over the space of candidate theories. Bottom-up methods start from an initially empty theory, and add one Horn rule at a time. Individual rules are constructed by conjoining firstorder relations so as to maximize the number of covered positive facts, while trying to keep covered negative facts to a minimum. After each rule is constructed, all covered facts are removed from the KB. These methods are extremely expressive, and can handle general nary relations. Instead, FRM focuses on binary relations only, which are more common in today’s Web-centric knowledge bases. ILP methods are designed to operate on the original KB only; this fact, paired with the sheer magnitude of the search space, makes standard ILP methods highly non-scalable. More recent extensions (e.g. kFOIL (Landwehr et al., 2006)) adopt a feature-space view of relational facts, but are still based on the classical search loop and can not be trivially adapted to working on the relational embeddings directly. Finally, rule elongation can be hindered by the presence of plateaus in the cost function.
Our path-based learning procedure is closely related to Relational Pathfinding (RP) (Richards & Mooney, 1992). RP is based on the observation that ground relation paths (that is, conjunctions of true relation instances) do act as support for arbitrary-length rules. It follows that mining these paths directly allows to detect longer rules with high support, avoiding the rule elongation problem entirely. There are many commonalities between RP and FRM. Both approaches are centered around relation paths, although in different representations (original versus compressed), and focus on pathbased theories. The major drawback of RP is that it requires exhaustive enumeration of relation paths (up to a maximum length), which can be impractical depending on the size of the KB. FRM sidesteps this issue by leveraging efficient online decoding techniques, namely Online Search OMP (Weinstein & Wakin, 2012).
To alleviate its computational requirements, a lifting procedure for RP was presented in Kok & Domingos (2009). Similarly to FRM, lifted RP is composed of separate compression and learning stages. In the first stage, the original KB is “lifted” by clustering functionally identical relation paths together, producing a smaller KB as output. In the second stage, standard RP is applied to the compressed KB. A major difference with FRM is that lifting is exact, while RESCAL is typically lossy. Consequently, lifted RP guarantees equivalence of the original and compressed learning problems, but it also ignores the potential generalization benefit provided by the embeddings. Additionally, the first step of lifted RP relies on a (rather complex) agglomerative clustering procedure, while FRM can make use of state-of-the-art representation learning methods. Note that, just like lifted RP, FRM can be straightforwardly employed for structure learning of statistical relational models.
The work of Malioutov & Varshney (2013) is concerned with mining one-level rules from binary data. Like in FRM, rule learning is viewed as a recovery problem, and solved using compressed sensing techniques. Two major differences with FRM exist. In Malioutov & Varshney (2013) the truth value matrix is recovered with an extension of Basis Pursuit that handles 0-1 coefficients through
3http://www.cs.ox.ac.uk/activities/machinelearning/Aleph/
a mixed-integer linear programming (MILP) formulation, which is however solved approximately using linear relaxations. BP however requires the dictionary to be explicitly grounded, which is not the case for FRM. Additionally, their method is limited to one-level rules, i.e. either conjunctions or disjunctions of relations, but not both. An extension to two-level rules has been presented by Su et al. (2015), where BP is combined with heuristics to aggregate individual rules into two-level theories. In contrast, FRM natively supports mining two-level rules via efficient online search.
The only other theory learning method that is explicitly designed for working on embeddings is the one of Yang et al. (2015). It is based on the observation (also made by Gu et al. (2015)) that closed path Horn rules can be converted to path queries, which can be answered approximately by searching the space of (type-compatible) compositions of relation embeddings. They propose to perform a simple nearest neighbor search around the embedding of the head relation, Wh, while avoiding type-incompatible relation compositions. Unfortunately, rules are searched for independently of one another, which seriously affects both quality and interpretability of the results as shown by our experimental evaluation.
6 CONCLUSION
We presented a novel approach for performing rule mining directly over a compressed summary of a KB. A major advantage over purely logical alternatives is that the relational embeddings automatically generalize beyond the observed facts; as consequence, our method implicitly mines a completion of the knowledge base. The key idea is that theory learning can be approximated by a recovery problem in the space of relation embeddings, which can be solved efficiently using well-known sparse recovery algorithms. This novel formulation enables our method to deal with all propositional logic connectives (conjunction, disjunction, and negation), unlike previous techniques. We presented experimental results highlighting the ability of our miner to discover relevant and, most importantly, diverse rules.
One difficulty in applying our methods is that classical sparse recovery algorithm require the complete enumeration of the candidate rule bodies, which is exponential in rule length. In order to solve this issue, we plan to apply recent online recovery algorithms, like Online Search OMP (Weinstein & Wakin, 2012), which can explore the space of alternative bodies on-the-fly.
As the quality of relational embedding techniques improves, for instance thanks to path-based Gu et al. (2015); Neelakantan et al. (2015); Garcı́a-Durán et al. (2015) and logic-based Rocktäschel et al. (2015) training techniques, we expect the reliability and performance of theory learning in feature space to substantially improve as well.
APPENDIX A: ERROR DERIVATION
Fix a target head relation h. Let Eh denote the RESCAL error matrix Eh := Y h − X>WhX , and Ẽh denote the error matrix of Wh due to FRM, namely Ẽh := Wh − ∑ B∈Th α
BWB where WB = ∏ b∈BW b. Putting the two definitions together, we obtain:
Eh = Y h −X> [∑
B∈Th W B + Ẽh
] X = Y h −X> [∑ B∈Th W B ] X −X>ẼhX
Then, the Frobenius norm of the reconstruction error of head h is: ‖Y h−X> ∑ B∈Th αBW BX‖ = ‖X>ẼhX+Eh‖ ≤ ‖X>ẼhX‖+‖Eh‖ ≤ ‖X‖2‖Ẽh‖+‖Eh‖
where the last step follows from the sub-multiplicativity of the Frobenius norm. Now, FRM minimizes ‖Ẽh‖, and therefore minimizes an upper bound of the misclassification error of Th over Y h.
We note in passing that the bound can be tightened by reducing the norm of the entity embeddings X , for instance by choosing the proper embedding method. The question of how to find an optimal choice, however, is left as future work.
APPENDIX B: EXTENDED RESULTS

The paper presents a nice idea of directly finding rules such as brother(father) => uncle in knowledge bases, by directly searching in embedding space. The idea is to interpret the successive application of relationships as the multiplication of the relation-dependent matrices in non-negative RESCAL.
The experimental section provides an evaluation of the rules that are found by the algorithm. Nonetheless, the work seems only at its first stages for now, and many questions are left open:
1) while the approach to find rules seems very general, the reason why it should work is unclear. What properties of the embedding space or of the initial algorithm are required for this approach to find meaningful rules? Can we apply the same principles to other algorithms than non-negative RESCAL?
2) there is no real evaluation in terms of link prediction. How can we use these rules in conjunction with the original algorithm to improve link prediction? What performance gains can be expected? Can these rules find links that would not be found be the original algorithm in the first place?
3) scaling: for now the number of parameters of the rule miner is (#relationships)^(max. path length + 1). How does this method scale on standard benchmarks such as FB15k where there is more than a 1000 of relationships?