Sum-product networks have recently emerged as an attractive representation due to their dual view as a special type of deep neural network with clear semantics and a special type of probabilistic graphical model for which inference is always tractable. Those properties follow from some conditions (i.e., completeness and decomposability) that must be respected by the structure of the network. As a result, it is not easy to specify a valid sum-product network by hand and therefore structure learning techniques are typically used in practice. This paper describes the first online structure learning technique for continuous SPNs with Gaussian leaves. We also introduce an accompanying new parameter learning technique.
1 INTRODUCTION
Sum-product networks (SPNs) were first introduced by Poon & Domingos (2011) as a new type of deep representation. They distinguish themselves from other types of neural networks by several desirable properties:
1. The quantities computed by each node can be clearly interpreted as (un-normalized) probabilities.
2. SPNs are equivalent to Bayesian and Markov networks (Zhao et al., 2015) while ensuring that exact inference has linear complexity with respect to the size of the network.
3. They represent generative models that naturally handle arbitrary queries with missing data while changing which variables are treated as inputs and outputs.
There is a catch: these nice properties arise only when the structure of the network satisfies certain conditions (i.e., decomposability and completeness) (Poon & Domingos, 2011). Hence, it is not easy to specify sum-product networks by hand. In particular, fully connected networks typically violate those conditions. Similarly, most sparse structures that are handcrafted by practitioners to compute specific types of features or embeddings also violate those conditions. While this may seem like a major drawback, the benefit is that researchers have been forced to develop structure learning techniques to obtain valid SPNs that satisfy those conditions (Dennis & Ventura, 2012; Gens & Domingos, 2013; Peharz et al., 2013; Lee et al., 2013; Rooshenas & Lowd, 2014; Adel et al., 2015; Vergari et al., 2015; Rahman & Gogate, 2016; Mazen Melibari, 2016). At the moment, the search for good network structures in other types of neural networks is typically done by hand based on intuitions as well as trial and error. However the expectation is that automated structure learning techniques will eventually dominate. For this to happen, we need structure learning techniques that can scale easily to large amounts of data.
To that effect, we propose the first online structure learning technique for SPNs with Gaussian leaves. The approach starts with a network structure that assumes that all variables are independent. This network structure is then updated as a stream of data points is processed. Whenever a statistically significant correlation is detected between some variables, a correlation is introduced in the network in the form of a multivariate Gaussian or a mixture distribution. This is done while ensuring that the resulting network structure is necessarily valid. The approach is evaluated on several large benchmark datasets.
The paper is structured as follows. Section 2 provides some background about sum-product networks. Section 3 describes our online structure learning technique for SPNs with Gaussian leaves. Section 4 evaluates the performance of our structure learning technique on several large benchmark datasets. Finally, Section 5 concludes the paper and discusses possible directions for future work.
2 BACKGROUND
Sum-product networks (SPNs) were first proposed by Poon & Domingos (2011) as a new type of deep architecture consisting of a rooted acyclic directed graph with interior nodes that are sums and products while the leaves are tractable distributions, including Bernoulli distributions for discrete SPNs and Gaussian distributions for continuous SPNs. The edges emanating from sum nodes are labeled with non-negative weights w. An SPN encodes a function f(X = x) that takes as input a variable assignment X = x and produces an output at its root. This function is defined recursively at each node n as follows:
fn(X = x) =  Pr(Xn = xn) if isLeaf(n)∑i wifchildi(n)(x) if isSum(n)∏ i fchildi(n)(x) if isProduct(n)
(1)
Here, Xn = xn denotes the variable assignment restricted to the variables contained in the leaf n. If none of the variables in leaf n are instantiated by X = x then Pr(Xn = xn) = Pr(∅) = 1. Note also that if leaf n contains continuous variables, then Pr(Xn = xn) should be interpreted as pdf(Xn = xn).
An SPN is a neural network in the sense that each interior node can be interpreted as computing a linear combination of its children followed by a potentially non-linear activation function. Without loss of generality, assume that the SPN is organized in alternating layers of sums and product nodes.1 It is easy to see that sum-nodes compute a linear combination of their children. Product nodes can be interpreted as the sum of its children in the log domain. Hence sum-product networks can be viewed as neural networks with logarithmic and exponential activation functions.
An SPN can also be viewed as encoding a joint distribution over the random variables in its leaves when the network structure satisfies certain conditions. These conditions are often defined in terms of the notion of scope. Definition 1 (Scope). The scope(n) of a node n is the set of variables that are descendants of n.
A sufficient set of conditions to ensure a valid joint distribution includes: Definition 2 (Completeness (Poon & Domingos, 2011)). An SPN is complete if all children of the same sum node have the same scope. Definition 3 (Decomposability (Poon & Domingos, 2011)). An SPN is decomposable if all children of the same product node have disjoint scopes.
Here decomposability allows us to interpret product nodes as computing factored distributions with respect to disjoint sets of variables, which ensures that the product is a valid distribution over the union of the scopes of the children. Similarly, completeness allows us to interpret sum nodes as computing a mixture of the distributions encoded by the children since they all have the same scope. Each child is a mixture component with mixture probability proportional to its weight. Hence, in complete and decomposable SPNs, the sub-SPN rooted at each node can be interpreted as encoding an (un-normalized) joint distribution over its scope. We can use the function f to answer inference queries with respect to the joint distribution encoded by the entire SPN as follows:
• Marginal queries: Pr(X = x) = froot(X=x)froot(∅) • Conditional queries: Pr(X = x|Y = y) = froot(X=x,Y=y)froot(Y=y)
Unlike most neural networks that can answer only queries with fixed inputs and outputs, SPNs can answer conditional inference queries with varying inputs and outputs simply by changing the set of
1Consecutive sum nodes can always be merged into a single sum node. Similarly, consecutive product nodes can always be merged into a single product node.
variables that are queried (outputs) and conditioned on (inputs). Furthermore, SPNs can be used to generate data by sampling from the joint distributions they encode. This is achieved by a top-down pass through the network. Starting at the root, each child of a product node is followed, a single child of a sum node is sampled according to the unnormalized distribution encoded by the weights of the sum node and a variable assignment is sampled in each leaf that is reached. This is particularly useful in natural language generation tasks and image completion tasks (Poon & Domingos, 2011).
Note also that inference queries can be answered exactly in linear time with respect to the size of the network since each query requires two evaluations of the network function f and each evaluation is performed in a bottom-up pass through the network. This means that SPNs can also be viewed as a special type of tractable probabilistic graphical model, in contrast to Bayesian and Markov networks for which inference is #P-hard (Roth, 1996). Any SPN can be converted into an equivalent bipartite Bayesian network without any exponential blow up, while Bayesian and Markov networks can be converted into equivalent SPNs at the risk of an exponential blow up (Zhao et al., 2015).
2.1 PARAMETER LEARNING
The weights of an SPN are its parameters. They can be estimated by maximizing the likelihood of a dataset (generative training) (Poon & Domingos, 2011) or the conditional likelihood of some output features given some input features (discriminative training) by Stochastic Gradient Descent (SGD) (Gens & Domingos, 2012). Since SPNs are generative probabilistic models where the sum nodes can be interpreted as hidden variables that induce a mixture, the parameters can also be estimated by Expectation Maximization (EM) (Poon & Domingos, 2011; Peharz, 2015). Zhao & Poupart (2016) provides a unifying framework that explains how likelihood maximization in SPNs corresponds to a signomial optimization problem where SGD is a first order procedure, one can also consider a sequential monomial approximation and EM corresponds to a concave-convex procedure that converges faster than the other techniques. Since SPNs are deep architectures, SGD and EM suffer from vanishing updates and therefore ”hard” variants have been proposed to remedy to this problem (Poon & Domingos, 2011; Gens & Domingos, 2012). By replacing all sum nodes by max nodes in an SPN, we obtain a max-product network where the gradient is constant (hard SGD) and latent variables become deterministic (hard EM). It is also possible to train SPNs in an online fashion based on streaming data (Lee et al., 2013; Rashwan et al., 2016; Zhao et al., 2016; Jaini et al., 2016). In particular, it was shown that online Bayesian moment matching (Rashwan et al., 2016; Jaini et al., 2016) and online collapsed variational Bayes (Zhao et al., 2016) perform much better than SGD and online EM.
2.2 STRUCTURE LEARNING
Since it is difficult to specify network structures for SPNs that satisfy the decomposability and completeness properties, several automated structure learning techniques have been proposed (Dennis & Ventura, 2012; Gens & Domingos, 2013; Peharz et al., 2013; Lee et al., 2013; Rooshenas & Lowd, 2014; Adel et al., 2015; Vergari et al., 2015; Rahman & Gogate, 2016; Mazen Melibari, 2016). The first two structure learning techniques (Dennis & Ventura, 2012; Gens & Domingos, 2013) are top down approaches that alternate between instance clustering to construct sum nodes and variable partitioning to construct product nodes. We can also combine instance clustering and variable partitioning in one step with a rank-one submatrix extraction by performing a singular value decomposition (Adel et al., 2015). Alternatively, we can learn the structure of SPNs in a bottom-up fashion by incrementally clustering correlated variables (Peharz et al., 2013). These algorithms all learn SPNs with a tree structure and univariate leaves. It is possible to learn SPNs with multivariate leaves by using a hybrid technique that learns an SPN in a top down fashion, but stops early and constructs multivariate leaves by fitting a tractable probabilistic graphical model over the variables in each leaf (Rooshenas & Lowd, 2014; Vergari et al., 2015). It is also possible to merge similar subtrees into directed acyclic graphs in a post-processing step to reduce the size of the resulting SPN (Rahman & Gogate, 2016). Furthermore, Mazen Melibari (2016) proposed dynamic SPNs for variable length data and described a search-and-score structure learning technique that does a local search over the space of network structures.
So far, all these structure learning algorithms are batch techniques that assume that the full dataset is available and can be scanned multiple times. Lee et al. (2013) describes an online structure
learning technique that gradually grows a network structure based on mini-batches. The algorithm is a variant of LearnSPN (Gens & Domingos, 2013) where the clustering step is modified to use online clustering. As a result, sum nodes can be extended with more children when the algorithm encounters a mini-batch that is better clustered with additional clusters. Product nodes are never modified after their creation.
Since existing structure learning techniques have all been designed for discrete SPNs and have yet to be extended to continuous SPNs such as Gaussian SPNs, the state of the art for continuous (and large scale) datasets is to generate a random network structure that satisfies decomposability and completeness after which the weights are learned by a scalable online learning technique (Jaini et al., 2016). We advance the state of the art by proposing a first online structure learning technique for Gaussian SPNs.
3 PROPOSED ALGORITHM
In this work, we assume that the leaf nodes all have Gaussian distributions. A leaf node may have more than one variable in its scope, in which case it follows a multivariate Gaussian distribution.
Suppose we want to model a probability distribution over a d-dimensional space. The algorithm starts with a fully factorized joint probability distribution over all variables, p(x) = p(x1, x2, . . . , xd) = p1(x1)p2(x2) · · · pd(xd). This distribution is represented by a product node with d children, the ith of which is a univariate distribution over the variable xi. Therefore, initially we assume that the variables are independent, and the algorithm will update this probability distribution as new data points are processed.
Given a mini-batch of data points, the algorithm passes the points through the network from the root to the leaf nodes and updates each node along the way. This update includes two parts:
• updating the parameters of the SPN, and • updating the structure of the network.
3.1 PARAMETER UPDATE
The parameters are updated by keeping track of running sufficient statistics. There are two types of parameters in the model: weights on the branches under a sum node, and parameters for the Gaussian distribution in a leaf node.
We propose a new online algorithm for parameter learning that is simple while ensuring that after each update, the likelihood of the last processed data point is increased (similar to stochastic gradient ascent). Algorithm 1 describes the pseudocode of this procedure. Every node in the network has a count, nc, initialized to 1. When a data point is received, the likelihood of this data point is computed at each node. Then the parameters of the network are updated in a recursive top-down fashion by starting at the root node. When a sum node is traversed, its count is increased by 1 and the count of the child with the highest likelihood is increased by 1. This effectively increases the weight of the child with the highest likelihood while decreasing the weights of the remaining children. As a result, the overall likelihood at the sum node will increase. The weight ws,c of a branch between a sum node s and one of its children c can then be estimated as
ws,c = nc ns
(2)
where ns is the count of the sum node and nc is the count of the child node. We also recursively update the subtree of the child with the highest likelihood. In the case of ties, we simply choose one of the children with highest likelihood at random to be updated.
Since there are no parameters associated with a product node, the only way to increase its likelihood is to increase the likelihood at each of its children. We increment the count at each child of a product node and recursively update the subtrees rooted at each child.
Since each leaf node represents a Gaussian distribution, it keeps track of the empirical mean vector µ and empirical covariance matrix Σ for the variables in its scope. When a leaf node with a current
Algorithm 1 parameterUpdate(root(SPN),data) Input: SPN and m data points Output: SPN with updated parameters nroot ← nroot +m if isProduct(root) then
for each child of root do parameterUpdate(child, data)
end for else if isSum(root) then
for each child of root do subset← {x ∈ data | likelihood(child, x) ≥ likelihood(child′, x) ∀child′ of root} parameterUpdate(child, subset) wroot,child ← nchild+1nroot+#children
end for else if isLeaf(root) then
update mean µ(root) based on Eq. 3 update covariance matrix Σ(root) based on Eq. 4
end if
count of n receives a batch of m data points x(1), x(2), . . . , x(m), the empirical mean and empirical covariance are updated according to the equations:
µ′i = 1
n+m
( nµi +
m∑ k=1 x (k) i
) (3)
and
Σ′i,j = 1
n+m
[ nΣi,j +
m∑ k=1 ( x (k) i − µi )( x (k) j − µj )] − (µ′i − µi)(µ′j − µj) (4)
where i and j index the variables in the leaf node’s scope, and µ′ and Σ′ are the new mean and covariance after the update.
This parameter update technique is related to, but different from hard SGD and hard EM used in (Poon & Domingos, 2011; Gens & Domingos, 2012; Lee et al., 2013). Hard SGD and hard EM also keep track of a count for the child of each sum node and increment those counts each time a data point reaches this child. However, to decide when a child is reached by a data point, they replace all descendant sum nodes by max nodes and evaluate the resulting max-product network. In contrast, we retain the descendant sum nodes and evaluate the original sum-product network as it is. This evaluates more faithfully the probability that a data point is generated by a child.
Alg. 1 does a single pass through the data. The complexity of updating the parameters after each data point is linear in the size of the network (i.e., # of edges) since it takes one bottom up pass to compute the likelihood of the data point at each node and one top-down pass to update the sufficient statistics and the weights. The update of the sufficient statistics can be seen as locally maximizing the likelihood of the data. The empirical mean and covariance of the Gaussian leaves locally increase the likelihood of the data that reach that leaf. Similarly, the count ratios used to set the weights under a sum node locally increase the likelihood of the data that reach each child. We prove this result below.
Theorem 1. Let θs be the set of parameters of an SPN s, and let fs(·|θs) be the probability density function of the SPN. Given an observation x, suppose the parameters are updated to θ′s based on the running average update procedure, then we have fs(x|θ′s) ≥ fs(x|θs).
Proof. We will prove the theorem by induction. First suppose the SPN is just one leaf node. In this case, the parameters are the empirical mean and covariance, which is the maximum likelihood estimator for Gaussian distribution. Suppose θ consists of the parameters learned using n data points x(1), . . . , x(n), and θ′ consists of the parameters learned using the same n data points and an
additional observation x. Then we have
fs(x|θ′s) n∏
i=1
f(x(i)|θ′s) ≥ fs(x|θs) n∏
i=1
fs(x (i)|θs) ≥ fs(x|θs) n∏ i=1 fs(x (i)|θ′s) (5)
Thus we get fs(x|θ′s) ≥ fs(x|θs). Now suppose we have an SPN s where each child SPN t satisfies the property ft(x|θ′t) ≥ ft(x|θt). If the root of s is a product node, then fs(x|θ′s) = ∏ t ft(x|θ′t) ≥ ∏ t ft(x|θt) = fs(x|θs).
Now suppose the root of s is a sum node. Let nt be the count of child t, and let u = arg maxt ft(x|θt). Then we have
fs(x|θ′s) = 1
n+ 1
( fu(x|θ′u) + ∑ t ntft(x|θ′t) )
≥ 1 n+ 1
( fu(x|θu) +
∑ t ntft(x|θt)
) by inductive hypothesis
≥ 1 n+ 1 (∑ t nt n ft(x|θt) + ∑ t ntft(x|θt) )
= 1
n ∑ t ntft(x|θt)
= fs(x|θs)
3.2 STRUCTURE UPDATE
The simple online parameter learning described above can be easily extended to enable online structure learning. Algorithm 2 describes the pseudocode of the resulting procedure called oSLRAU (online Structure Learning with Running Average Update). Similar to leaf nodes, each product node also keeps track of the empirical mean vector and empirical covariance matrix of the variables in its scope. These are updated in the same way as the leaf nodes.
Initially, when a product node is created, all variables in the scope are assumed independent (see Algorithm 5). As new data points arrive at a product node, the covariance matrix is updated, and if the absolute value of the Pearson correlation coefficient between two variables are above a certain threshold, the algorithm updates the structure so that the two variables become correlated in the model.
We correlate two variables in the model by combining the child nodes whose scopes contain the two variables. The algorithm employs two approaches to combine the two child nodes:
• create a multivariate leaf node (Algorithm 4), or • create a mixture of two components over the variables (Algorithm 3).
These two processes are depicted in Figure 1. On the left, a product node with scope x1, . . . , x5 originally has three children. The product node keeps track of the empirical mean and empirical covariance for these five variables. Suppose it receives a mini-batch of data and updates the statistics. As a result of this update, x1 and x3 now have a correlation above the threshold.
Figure 1 illustrates the two approaches to model this correlation. In the middle of Figure 1, the algorithm combines the two child nodes that have x1 and x3 in their scope, and turns them into a multivariate leaf node. Since the product node already keeps track of the mean and covariance of these variables, we can simply use those statistics as the parameters for the new leaf node.
Another way to correlate x1 and x3 is to create a mixture, as shown in the right part of Figure 1. The mixture has two components. The first component contains the original children of the product node that contain x1 and x3. The second component is a new product node, which is again initialized to have a fully factorized distribution over its scope (Alg. 5). The mini-batch of data points are then passed down the new mixture to update its parameters.
Note that although the children are drawn like leaf nodes in the diagrams, they can in fact be entire subtrees. Since the process does not involve the parameters in a child, it works the same way if some of the children are trees instead of single nodes.
The technique chosen to induce a correlation depends on the number of variables in the scope. The algorithm creates a multivariate leaf node when the combined scope of the two child nodes has a number of variables that does not exceed some threshold and if the total number of variables in the problem is greater than this threshold, otherwise it creates a mixture. Since the number of parameters in multivariate Gaussian leaves grows at a quadratic rate with respect to the number of variables, it is not advised to consider multivariate leaves with too many variables. In contrast, the mixture construction increases the number of parameters at a linear rate, which is less prone to overfitting when many variables are correlated.
To simplify the structure, if a product node ends up with only one child, it is removed from the network, and its only child is joined with its parent. Similarly, if a sum node ends up being a child of another sum node, then the child sum node can be removed, and all its children are promoted one layer up.
Note that the this structure learning technique does a single pass through the data and therefore is entirely online. The time and space complexity of updating the structure after each data point is linear in the size of the network (i.e., # of edges) and quadratic in the number of features (since product nodes store a covariance matrix that is quadratic in the size of their scope). The algorithm also ensures that the decomposability and completeness properties are preserved after each update.
Our algorithm (oSLRAU) is related to, but different from the online structure learning technique proposed by Lee et al. (2013). Lee et al.’s technique was applied to discrete datasets while oSLRAU learns SPNs with Gaussian leaves based on real-valued data. Furthermore, Lee et al.’s technique incrementally constructs a network in a top down fashion by adding children to sum nodes by online clustering. Once a product node is constructed, it is never modified. In contrast, oSLRAU incrementally constructs a network in a bottom up fashion by detecting correlations and modifying product nodes to represent these correlations. Finally, Lee et al.’s technique updates the parameters by hard EM (which implicitly works with a max-product network) while oSLRAU updates the parameters by Alg. 1 (which retains the original sum-product network) as explained in the previous section.
4 EXPERIMENTS
The source code for our new online structure learning algorithm is available at github.com/whsu/spn.
Algorithm 2 oSLRAU(root(SPN), data) Input: SPN and m data points Output: SPN with updated parameters nroot ← nroot +m if isProduct(root) then
update covariance matrix Σ(root) based on Eq. 4 highestCorrelation← 0 for each c, c′ ∈ children(root) where c 6= c′ do
correlationc,c′ ← maxi∈scope(c),j∈scope(c′) |Σ(root)ij |√
Σ (root) ii Σ (root) jj
if correlationc,c′ > highestCorrelation then highestCorrelation← correlationc,c′ child1 ← c child2 ← c′
end if end for if highest ≥ threshold then
if |scope(child1) ∪ scope(child2)| ≥ nV ars then createMixture(root, child1, child2) else createMultivariateGaussian(root, child1, child2)
end if end if for each child of root do oSLRAU(child, data)
end for else if isSum(root) then
for each child of root do subset← {x ∈ data | likelihood(child, x) ≥ likelihood(child′, x) ∀child′ of root} oSLRAU(child, subset) wroot,child ← nchild+1nroot+#children
end for else if isLeaf(root) then
update mean µ(root) based on Eq. 3 update covariance matrix Σ(root) based on Eq. 4
end if
4.1 TOY DATASET
As a proof of concept, we first test the algorithm on a toy synthetic dataset. We generate data from the 3-dimensional distribution
p(x1, x2, x3) = [0.25N(x1|1, 1)N(x2|2, 2) + 0.25N(x1|11, 1)N(x2|12, 2) + 0.25N(x1|21, 1)N(x2|22, 2) + 0.25N(x1|31, 1)N(x2|32, 2)]N(x3|3, 3),
where N(·|µ, σ2) is the normal distribution with mean µ and variance σ2. Therefore, the first two dimensions x1 and x2 are generated from a Gaussian mixture with four components, and x3 is independent from the other two variables.
Starting from a fully factorized distribution, we would expect x3 to remain factorized after learning from data. Furthermore, the algorithm should generate new components along the first two dimensions as more data points are received since x1 and x2 are correlated.
This is indeed what happens. Figure 2 shows the structure learned after 200 and 500 data points. The variable x3 remains factorized regardless of the number of data points seen, whereas more components are created for x1 and x2 as more data points are processed.
Algorithm 3 createMixture(root, child1, child2) Input: SPN and two children to be merged Output: new mixture model
remove child1 and child2 from root component1 ← create product node add child1 and child2 as children of component1 ncomponent1 ← nroot jointScope← scope(child1) ∪ scope(child2) Σ(component1) ← Σ(root)jointScope,jointScope component2 ← createFactoredModel(jointScope) ncomponent2 ← 0 mixture← create sum node add component1 and component2 as children of mixture nmixture ← nroot wmixture,component1 ← ncomponent1+1
nmixture+2
wmixture,component2 ← ncomponent2+1
nmixture+2
add mixture as child of root return root
Algorithm 4 createMultiV arGaussian(root, child1, child2) Input: SPN, two children to be merged and data Output: new multivariate Gaussian
create multiV arGaussian jointScope← {scope(child1) ∪ scope(child2)} µ(multiV arGaussian) ← µ(root)jointScope Σ(multiV arGaussian) ← Σ(root)jointScope,jointScope nmultiV arGaussian ← nroot return multiV arGaussian
Algorithm 5 createFactoredModel(scope) Input: scope (set of variables) Output: fully factored SPN factoredModel← create product node for each i ∈ scope do
add Ni(µ=0, σ=Σ (root) i,i ) as child of factoredModel
end for Σ(factoredModel) ← 0 nfactoredModel ← 0 return factoredModel
Figure 3 shows the data points along the first two dimensions and the Gaussian components learned. We can see that the algorithm generates new components to model the correlation between x1 and x2 as it processes more data.
4.2 COMPARISON TO OTHER ALGORITHMS
In a second experiment, we compare our algorithm to several alternatives on the same datasets used by Jaini et al. (2016). We use 0.1 as the correlation threshold in all experiments, and we use minibatch sizes of 1 for the three datasets with fewest instances (Quake, Banknote, Abalone), 8 for the two slightly larger ones (Kinematics, CA), and 256 for the two datasets with most instances (Flow Size, Sensorless).
The experimental results for our algorithm called online structure learning with running average update (oSLRAU) are listed in Table 1 along with results reproduced from Jaini et al. (2016). The table reports the average test log likelihoods with standard error on 10-fold cross validation. oSLRAU achieved better log likelihoods than online Bayesian moment matching (oBMM) (Jaini et al., 2016) and online expectation maximization (oEM) (Cappé & Moulines, 2009) with network structures generated at random or corresponding to Gaussian mixture models (GMMs). This highlights the main advantage of oSLRAU: learning a structure that models the data. Stacked Restricted Boltzmann Machines (SRBMs) (Salakhutdinov & Hinton, 2009) and Generative Moment Matching Networks (GenMMNs) (Li et al., 2015) are other types of deep generative models. Since it is not possible to compute the likelihood of data points with GenMMNs, the model is augmented with Parzen windows. More specifically, 10,000 samples are generated using the resulting GenMMNs and a Gaussian kernel is estimated for each sample by adjusting its parameters to maximize the likelihood of a validation set. However, as pointed out by Theis et al. (2015) this method only provides an approximate estimate of the log-likelihood and therefore the log-likelihood reported for GenMMNs in Table 1 may not be directly comparable to the log-likelihood of other models.
The network structures for GenMMNs and SRBMs are fully connected while ensuring that the number of parameters is comparable to those of the SPNs. oSLRAU outperforms these models on 5 datasets while SRBMs and GenMMNs each outperform oSLRAU on one dataset. Although SRBMs and GenMMNs are more expressive than SPNs since they allow other types of nodes beyond sums and products, training GenMMNs and SRBMs is notoriously difficult. In contrast, oSLRAU provides a simple and effective way of optimizing the structure and parameters of SPNs that captures well the correlations between variables and therefore yields good results.
4.3 LARGE DATASETS
We also tested oSLRAU on larger datasets to evaluate its scaling properties. Table 2 shows the number of attributes and data points in each dataset. Table 3 compares the average log-likelihood of oSLRAU to that of randomly generated networks (which are the state of the art for obtain a valid continuous SPNs) for those large datasets. For a fair comparison we generated random networks that are at least as large as the networks obtained by oSLRAU. oSLRAU achieves higher log-likelihood than random networks since it effectively discovers empirical correlations and generates a structure that captures those correlations.
We also compare oSLRAU to a publicly available implementation of RealNVP2. Since the benchmarks include a variety of problems from different domains and it is not clear what network architecture would work best, we used a default 2-hidden-layer fully connected network. The two
2https://github.com/taesung89/real-nvp
layers have the same size. For a fair comparison, we used a number of nodes per layer that yields approximately the same number of parameters as the sum product networks. Training was done by stochastic gradient descent in TensorFlow with a step size of 0.01 and mini-batch sizes that vary from 100 to 1500 depending on the size of the dataset. We report the results for online learning (single iteration) and offline learning (validation loss stops decreasing). In this experiment, the correlation threshold was kept constant at 0.1. To determine the maximum number of variables in multivariate leaves, we followed the following rule: at most one variable per leaf if the problem has 3 features or less and then increase the maximum number of variables per leaf up to 4 depending on the number of features. Further analysis on the effects of varying the maximum number of variables per leaf are available below. We do this to balance the size and the expressiveness of the resulting SPN. oSLRAU outperformed RealNVP on 5 of the 6 datasets. This can be explained by the fact that oSLRAU learns a structure that is suitable for each problem while RealNVP does not learn any structure. Note that it should be possible for RealNVP to obtain better results by using a better architecture than a default 2-hidden-layer network, however in the absence of domain knowledge this is difficult. Furthermore, in online learning with streaming data, it is not possible to do an offline search over some hyperparameters such as the number of layers and nodes in order to fine tune the architecture. Hence, the results presented in Table 3 highlight the importance of an online structure learning technique such as oSLRAU to obtain a suitable network structure with streaming data in the absence of domain knowledge.
Table 4 reports the training time (seconds) and the size (# of nodes) of the resulting SPNs for each dataset when running oSLRAU and a variant that stops structure learning early. The experiments were carried out on an Amazon c4.xlarge machine with 4 vCPUs (high frequency Intel Xeon E52666 v3 Haswell processors) and 7.5 Gb of RAM. The times are relatively short since oSLRAU is an online algorithm and therefore does a single pass through the data. Since it gradually constructs the structure of the SPN as it processes the data, we can also stop the updates to the structure early (while still updating the parameters). This helps to mitigate overfitting while producing much smaller SPNs and reducing the running time. In the columns labeled ”early stop” we report the results achieved when structure learning is stopped after processing one ninth of the data. The resulting SPNs are significantly smaller, while achieving a log-likelihood that is close to that of oSLRAU without early stopping.
The size of the resulting SPNs and their log-likelihood also depend on the correlation threshold used to determine when the structure should be updated to account for a detected correlation, and the maximum size of a leaf node used to determine when to branch off into a new subtree.
To understand the impact that the maximum number of variables per leaf node has on the resulting SPN, we performed experiments where the minibatch size and correlation threshold were held constant for a given dataset while the maximum number of variables per leaf node varies. We report the log likelihood with standard error after ten-fold cross validation, as well as average size and average time in Tables 5, 6 and 7. As expected, the number of nodes in an SPN decreases as the leaf node cap increases, since there will be less branching. What’s interesting is that depending on the type of correlations in the datasets, different sizes perform better or worse. For example in Power, we notice that univariate leaf nodes are the best, but in GasSenH, slightly larger leaf nodes tend to do well. We show that too many variables in a leaf node leads to worse performance and underfitting, and in some cases too few variables per leaf node leads to overfitting. These results show that in general, the largest decrease in size and time while maintaining good performance occurs with a maximum of 3 variables per leaf node. Therefore in practice, 3 variables per leaf node works well, except when there are only a few variables in the dataset, then 1 is a good choice.
Tables 8, 9 and 10 show respectively how the log-likelihood, time and size changes as we vary the correlation threshold from 0.05 to 0.7. A very small correlation threshold tends to detect spurious correlations and lead to overfitting while a large correlation threshold tends to miss some correlations and lead to underfitting. The results in Table 8 generally support this tendency subject to noise due to sample effects. Since the highest log-likelihood was achieved in three of the datasets with a correlation threshold of 0.1, this explains why we used 0.1 as the threshold in the previous experiments. Tables 9 and 10 also show that the average time and size of the resulting SPNs generally decrease (subject to noise) as the correlation threshold increases since fewer correlations tend to be detected.
5 CONCLUSION AND FUTURE WORK
This paper describes a first online structure learning technique for Gaussian SPNs that does a single pass through the data. This allowed us to learn the structure of Gaussian SPNs in domains for which the state of the art was previously to generate a random network structure. This algorithm can also scale to large datasets efficiently.
In the future, this work could be extended in several directions. We are investigating the combination of our structure learning technique with other parameter learning methods. Currently, we are simply learning the parameters by keeping running statistics for the weights, mean vectors, and covariance matrices. It might be possible to improve the performance by using more sophisticated parameter learning algorithms. We would also like to extend the structure learning algorithm to discrete variables. Finally, we would like to look into ways to automatically control the complexity of the networks. For example, it would be useful to add a regularization mechanism to avoid possible overfitting.

The authors present an online learning method for learning the structure of sum-product networks. The algorithm assumes Gaussian coordinate-wise marginal distributions, and learns both parameters and structure online. The parameters are updated by a recursive procedure that reweights nodes in the network that most contribute to the likelihood of the current data point. The structure learning is done by either merging independent product Gaussian nodes into multivariate leaf nodes, or creating a mixture over the two nodes when the multivariate would be too large.
The fact that the dataset is scaled to some larger datasets (in terms of the number of datapoints) is promising, although the number of variables is still quite small. Current benchmarks for tractable continuous density modeling with neural networks include the NICE and Real-NVP families of models, which can be scaled to both large numbers of datapoints and variables. Intractable methods like GAN, GenMMN, VAE have the same property.
The main issue with this work for the ICLR audience is the use of mainly a set of SPN-specific datasets that are not used in the deep learning generative modeling literature. The use of GenMMN as a baseline is also not a good choice to bridge the gap to the neural community, as its Parzen-window based likelihood evaluation is not really meaningful. Better ways to evaluate the likelihood through annealed importance sampling are discussed in "On the Quantitative Analysis of Decoder-Based Generative Models" by Wu et al. I would recommend the use of a simple VAE type model to get a lower bound on the likelihood, or something like Real-NVP.
Most neural network density models are scalable to large numbers of observations as well as instances, and it is not clear that this method scales well "horizontally" like this. Evaluating the feasibility of modeling something like MNIST would be interesting.
SPNs have the strength that not only marginal but also various type of conditional queries are tractable, but performance on this is not evaluated or compared. One interesting application could be in imputation of unknown pixels or color channels in images, for which there is not currently a high-performing tractable model.
Despite the disconnect from other ICLR generative modeling literature, the algorithm here seems simple and intuitive and convincingly works better than the previous state of the art for online SPN structure learning. I think VAE is a much better baseline for continuous data than GenMMN when attempting to compare to neural network approaches. Further, the sum-product network could actually be combined with such deep latent variable models as an observation model or posterior, which could be a very powerful combination.
I would like it if these SPN models were better known by the ICLR probabilistic modeling community, but I do not know if this paper does enough to make them relevant. As with the other reviewers, I am not an expert on SPNs. However, this seems to be a simple and effective algorithm for online structure induction, and the scalability aspect is something that is important in much recent work in the learning of representations. I think it is good enough for publication, although I would prefer to see many of the above additions to more clearly bridge the gap with other literature in deep generative modeling.


The authors present empirical distributions for the halting time (measured by the number of iterations to reach a given accuracy) of optimization algorithms applied to two random systems: spin glasses and deep learning. Given an algorithm, which we take to be both the optimization routine and the form of the random landscape, the fluctuations of the halting time follow a distribution that, after centering and scaling, remains unchanged even when the distribution on the landscape is changed. We observe two main classes, a Gumbel-like distribution that appears in Google searches, human decision times, QR factorization and spin glasses, and a Gaussian-like distribution that appears in conjugate gradient method, deep network with MNIST input data and deep network with random input data. This empirical evidence suggests presence of a class of distributions for which the halting time is independent of the underlying distribution under some conditions.
1 INTRODUCTION
In this paper we discuss both the presence and application of universality in optimization algorithms. More precisely, in order to optimize an energy functional when the functional itself and the initial guess are random, we consider the following iterative algorithms: conjugate gradient for solving a linear system, gradient descent for spin glasses, and stochastic gradient descent for deep learning.
A bounded, piecewise differentiable random field (See Adler & Taylor (2009) for an account on the connection of random fields and geometry), where the randomness is non-degenerate, yields a landscape with many saddle points and local minima. Given such a landscape and a moving particle that takes steps to reach a low-energy level, an essential quantity is the time the particle takes until it stops which we call the halting time. Many useful bounds on the halting time are known for convex cases, where the stopping condition produces a halting time that is, essentially, the time to find the minimum. In non-convex cases, however, the particle knows only the information that can be calculated locally. And a locally measurable stopping condition, such as the norm of the gradient at the present point, or the difference in altitude with respect to the previous step, can lead the algorithm to locate a local minimum. This feature allows the halting time to be calculated in a broad range of non-convex, high-dimensional problems. A prototypical example of such a random field is the class of polynomials with random coefficients. Spin glasses and deep learning cost functions are then special cases of such fields that yield different landscapes. Polynomials with random coefficients are not only a broad class of functions, but also they are hard to study mathematically in any generality. Therefore, in order to capture essential features of such problems, we focus on their subclasses that are well studied (spin glasses) and practically relevant (deep learning cost functions).
The halting time in such landscapes, when normalized to mean zero and variance one (subtracting the mean and dividing by the standard deviation), appears to follow a distribution that is independent of the input data, in other words it follows a universal distribution: the fluctuations are universal. In statistical mechanics, the term “universality” is used to refer to a class of systems which, on a certain macroscopic scale, behave statistically the same while having different statistics on a microscopic scale. An example of such a law is the central limit theorem, which states that the sums of observations tend to follow the same distribution independent of the distribution of the individual observations, as long as contribution from individual observations is reasonably small. It may fail to hold, if the microscopic behavior is not independent, does not have a finite second-moment, or if we consider something different than the sum. This work’s focus is an attempt to put forward the cases
where we see universality. But in this spirit, we show a degenerate case in which halting time fails to follow a universal law.
A rather surprising example of halting time universality is in the cases of observed human decision times and GoogleTM query times. In Bakhtin & Correll (2012) the time it takes a person make a decision in the presence of visual stimulus is shown to have universal fluctuations. The theoretically predicted curve in this experiment follows a Gumbel-like distribution. In addition, we randomly sampled words from two different dictionaries and submitted search queries. The time it takes Google to present the results are recorded. The normalized search times closely follow the same Gumbel-like curve.
In the cases we observe, we find two main universality classes: (1) A Gumbel-like distribution that appears in Google searches, human decision times, QR factorization and spin glasses, and (2) a Gaussian-like distribution that appears in conjugate gradient algorithm and deep learning. To the best of our knowledge, our work along with the accompanying references in this introduction are the first ones to address the question of observing and classifying the distribution of the halting time.
1.1 DEFINITION OF UNIVERSALITY
Definition 1.1. An algorithm A consists of both a random cost function F (x, w) where x is a given random input and an optimization routine that seeks to minimize F with respect to w.
To each algorithm we attach a precise -dependent halting criteria for the algorithm. The halting time, which is a random variable, is the time it takes to meet this criteria. Within each algorithm there must be an intrinsic notion of dimension which we denote by N . The halting time T ,N,A,E depends on , N , the choice of algorithm A, and the ensemble E (or probability distribution). We use the empirical distribution of T ,N,A,E to provide heuristics for understanding the qualitative performance of the algorithms.
The presence of universality in an algorithm is the observation that for sufficiently large N and = (N), the halting time random variable satisfies
τ ,N,A,E := T ,N,A,E − E[T ,N,A,E ]√
Var(T ,N,A,E) ≈ τ∗A, (1)
where τ∗A is a continuous random variable that depends only on the algorithm. The random variable τ ,N,A,E is referred to as the fluctuations and when such an approximation appears to be valid we
say that N and (and any other external parameters) are in the scaling region. For example, in Section 1.2, A is the QR eigenvalue algorithm, N is the size of the matrix, is a small tolerance and E is given by a distribution on complex Hermitian (or real symmetric) matrices.
Some remarks must be made:
• A statement like (1) is known to hold rigorously for a few algorithms (see Deift & Trogdon (2016; 2017)) but in practice, it is verified experimentally. This was first done in Pfrang et al. (2014) and expanded in Deift et al. (2014) for a total of 8 different algorithms.
• The random variable τ∗A depends fundamentally on the functional form of F . And we only expect (1) to hold for a restricted class of ensembles E.
• T ,N,A,E is an integer-valued random variable. For it to become a continuous distribution limit must be taken. This is the only reason N must be large — in practice, the approximation in (1) is seen even for small to moderate N .
Universality in this sense is a measure of stability in an algorithm. For example, it is known from the work of Kostlan (1988) that halting time for the power method to compute the largest eigenvalue (in modulus) of symmetric Gaussian matrices has infinite expectation and hence this type of universality is not believed to be present. One could use this to conclude that the power method is a naı̈ve method for these matrices. Yet, it is known that the power method is much more efficient on positive-definite matrices where universality can be shown Deift & Trogdon (2017). Therefore, we have evidence that the presence of universality is a desirable feature of a numerical method.
1.2 EXAMPLE: DEMONSTRATION OF UNIVERSALITY IN THE QR ALGORITHM
To give some context, we discuss the universality in the solution of the eigenvalue problem with the classical QR algorithm. Historically, this was first noticed in Pfrang et al. (2014). In this example the fundamental object is the QR factorization (Q,R) = QR(A) where A = QR, Q is orthogonal (or unitary) and R is upper-triangular with positive diagonal entries. The QR algorithm applied to a Hermitian N ×N matrix A is given by the iteration
A0 := A,
(Qj , Rj) := QR(Aj),
Aj+1 := RjQj .
Generically, Aj → D as j → ∞ where D is a diagonal matrix whose diagonal entries are the eigenvalues of A. The halting time in Pfrang et al. (2014) was set to be the time of first deflation, T ,N,A,E(A), as:
min{j : √ N(N − k)‖Aj(k + 1 : N, 1 : k)‖∞ <
for some 1 ≤ k ≤ N − 1}.
Here ‖A‖∞ refers to the maximum entry of a matrix A in absolute value and the notation A(i : j, k : l) refers to the submatrix of A consisting of entries only in rows i, i+1, . . . , j and in columns k, k + 1, . . . , l. Thus the halting time for the QR algorithm is the time at which at least one offdiagonal block is appropriately small. Next, we have to discuss choices for the randomness, or ensembleE, by choosing different distributions on the entries ofA. Four such choices for ensembles are, Bernoulli ensemble (BE), Gaussian orthogonal ensemble (GOE), Gaussian unitary ensemble (GUE), Quartic unitary ensemble (QUE):
BE A is real-symmetric with iid Bernoulli ±1 entries on and below the diagonal. GOE A is real-symmetric with iid standard normal entries below the diagonal. The entries on the diagonal are iid normal with mean zero and variance two. GUE A is complex-Hermitian with iid standard complex normal entries below the diagonal. The
entries on the diagonal are iid complex normal mean zero and with variance two.
QUE A is complex-Hermitian with probability density ∝ e−trA4dA. See Deift (2000) for details on such an ensemble and Olver et al. (2015) for a method to sample such a matrix. Importantly, the entries of the matrix below the diagonal are correlated.
Here we have continuous and discrete, real and complex, and independent and dependent ensembles but nevertheless we see universality in Figure 2 where we take N = 150 and = 10−10.
Remark 1.1. The ensembles discussed above (GOE, GUE, BE and QUE) exhibit eigenvalue repulsion. That is, the probability that two eigenvalues are close1 is much smaller than if the locations of the eigenvalues were just given by iid points on the line. It turns out that choosing a random matrix with iid eigenvalues breaks the universality that is observed in Figure 2. See Pfrang et al. (2014) for a more in-depth discussion of this. Remark 1.2. To put the QR algorithm in the framework, let B = UAU∗ define F (A,U) by
min{j : √ N(N − k)‖B(k + 1 : N, 1 : k)‖∞ <
for some 1 ≤ k ≤ N − 1} We then use the QR algorithm to minimize F with respect to unitary matrices U using the initial condition U = I . If A is random then F (A,U) represents a random field on the unitary group.
1.3 CORE EXAMPLES: SPIN GLASS HAMILTONIANS AND DEEP LEARNING COST FUNCTIONS
A natural class of random fields is the class of Gaussian random functions on a high-dimensional sphere, known as p-spin spherical spin glass models in the physics literature (in the Gaussian process literature they are known as isotropic models). From the point of view of optimization, minimizing the spin glass model’s Hamiltonian is fruitful because a lot is known about its critical points. This allows us to experiment with questions regarding whether the local minima and saddle points, due to the non-convex nature of landscapes, present an obstacle in the training of a system. Such observations on the Hamiltonian doesn’t imply that it is a cost function or a simplified version of a cost function. Rather, the features that both systems have in common hint at a deeper underlying structure that needs to be discovered.
In recent years Dauphin et al. (2014) attacked the saddle point problem of non-convex optimization within deep learning. In contrast, Sagun et al. (2014) and the experimental second section of Choromanska et al. (2014) jointly argue that if the system is large enough, presence of saddle points is not an obstacle, and add that the local minimum practically gives a good enough solution within the limits of the model. However, Sagun et al. (2014) and Choromanska et al. (2014) hold different perspectives on what the qualitative similarities between optimization in spin glasses and deep learning might imply. The latter asserts a direct connection between the two systems based on these similarities. On the contrary, the former argues that these similarities hint at universal behaviors that are generically observed in vastly different systems rather than emphasizing a direct connection.
1By close, we mean that their distance is much less than O(1/N) where N is the size of the matrix.
In line with the asymptotic proof in Auffinger et al. (2013), the local minima are observed to lie roughly at the same energy level in spherical spin glasses. Auffinger et al. (2013) also gives asymptotic bounds on the value of the ground state and the exponential behavior of the average of the number of critical points below a given energy level. It turns out, when the dimension is large, the bulk of the local minima tend to have the same energy which is slightly above the global minimum. This level is called the floor level of the function. Simulations of the floor in spin glass can be found in Sagun et al. (2014). Sagun et al. (2014) also exhibits floor in a specially designed MNIST experiment: A student network is trained by the outputs of a pre-trained teacher network. Zero cost is achievable by the student, but the stochastic gradient descent cannot find zeros. It also does not have to because the floor level already gives a decent performance.
• Given data (i.e., from MNIST) and a measure L(x`, w) for determining the cost that is parametrized by w ∈ RN , the training procedure aims to find a point w∗ that minimizes the empirical training cost while keeping the test cost low. Here x` ∈ Z for ` ∈ {1, ..., S}, where Z is a random (ordered) sample of size S from the training examples. Total training cost is given by
F (Z,w) = LTrain(w) = 1
S S∑ `=1 L(x`, w). (2)
• Given couplings x(·) ∼ Gaussian(0, 1) that represent the strength of forces between triplets of spins. The state of the system is represented by w ∈ SN−1( √ N) ⊂ RN .
The Hamiltonian (or energy) of the simplest complex2 spherical spin glass model is given by:
F (x(·), w) = HN (w) = 1
N N∑ i,j,k xijkwiwjwk. (3)
The two functions are indeed different in two major ways. First, the domain of the Hamiltonian is a compact space and the couplings are independent Gaussian random variables whereas the inputs for (2) are not independent and the cost function has a non-compact domain. Second, at a fixed point w, variance of the function LTrain(w) is inversely proportional to the number of samples, but the variance of HN (w) is N . As a result a randomly initialized Hamiltonian can take vastly different values, but a randomly initialized cost tend to have very similar values. The Hamiltonian has macroscopic extensive quantities: its minimum scales with a negative constant multiple ofN . In contrast, the minimum of the cost function is bounded from below by zero. All of this indicates that landscapes with different geometries (glass-like, funnel-like, or another geometry) might still lead to similar phenomena such as existence of the floor level, and the universal behavior of the halting time.
1.4 SUMMARY OF RESULTS
We discuss the presence of universality in algorithms that are of a very different character. The conjugate gradient algorithm, discussed in Section 2.1, effectively solves a convex optimization problem. Gradient descent applied in the spin glass setting (discussed in Section 2.2) and stochastic gradient descent in the context of deep learning (MNIST, discussed in Section 2.3) are much more complicated non-convex optimization processes. Despite the fact that these algorithms share very little geometry in common, we demonstrate three things they share:
• A scaling region in which universality appears and performance is good. • Regions where the computation is either ineffective or inefficient. • A moment-based indicator for finding the universality class.
22-spin spherical spin glass, sum of xijwiwj terms, has exactly 2N critical points. When p ≥ 3, p−spin model has exponentially many critical points with respect to N . For the latter case, complexity is a measure on the number of critical points in an exponential scale. Deep learning problems are suspected to be complex in this sense.
2 EMPIRICAL OBSERVATION OF UNIVERSALITY
2.1 THE CONJUGATE GRADIENT ALGORITHM
The conjugate gradient algorithm (Hestenes & Stiefel, 1952) for solving the N × N linear system Ax = b, when A = A∗ is positive definite, is an iterative procedure to find the minimum of the convex quadratic form:
F (A, y) = 1
2 y∗Ay − y∗b,
where ∗ denotes the conjugate-transpose operation. Given an initial guess x0 (we use x0 = b), compute r0 = b−Ax0 and set p0 = r0. For k = 1, . . . , N ,
1. Compute rk = rk−1 − ak−1Apk−1 where3 ak−1 = 〈rk−1, rk−1〉/〈pk−1, Apk−1〉. 2. Compute pk = rk + bk−1pk−1 where bk−1 = 〈rk, rk〉/〈rk−1, rk−1〉. 3. Compute xk = xk−1 + ak−1pk−1.
If A is strictly positive definite xk → x = A−1b as k → ∞. Geometrically, the iterates xk are the best approximations of x over larger and larger affine Krylov subspaces Kk,
‖Axk − b‖A = minx∈Kk‖Ax− b‖A, Kk = x0 + span{r0, Ar0, . . . , Ak−1r0}, ‖x‖2A = 〈x,A−1x〉,
as k ↑ N . The quantity one monitors over the course of the conjugate gradient algorithm is the norm ‖rk‖:
T ,N,CG,E(A, b) := min{k : ‖rk‖ < }.
In exact arithmetic, the method takes at mostN steps: In calculations with finite-precision arithmetic the number of steps can be much larger than N and the behavior of the algorithm in finite-precision arithmetic has been the focus of much research (Greenbaum, 1989; Greenbaum & Strakos, 1992). What is important for us here is that it may happen that ‖rk‖ < but the true residual r̂k := b−Axk (which typically differs from rk in finite-precision computations) satisfies ‖r̂k‖ > .
Now, we discuss our choices for ensembles E of random data. In all computations, we take b = (bj)1≤j≤N where each bj is iid uniform on (−1, 1). We construct positive definite matrices A by A = XX∗ where X = (Xij)1≤i≤N, 1≤j≤M and each Xij ∼ D is iid for some distribution D. We make the following three choices for D, Positive definite Bernoulli ensemble (PDE), Laguerre orthogonal ensemble (LOE), Laguerre unitary ensemble (LUE):
3We use the notation ‖y‖2 = 〈y, y〉 = ∑
j |yj | 2 for y = (y1, y2, . . . , yN ) ∈ CN .
PBE D a Bernoulli ±1 random variable (equal probability). LOE D is a standard normal random variable. LUE D is a standard complex normal random variable.
The choice of the integer M , which is the inner dimension of the matrices in the product XX∗, is critical for the existence of universality. In Deift et al. (2014) and Deift et al. (2015) it is demonstrated that universality is present when M = N + bc √ Nc and the -accuracy is small, but fixed. Universality is not present when M = N and this can be explained by examining the distribution of the condition number of the matrix A in the LUE setting (Deift et al., 2015). We demonstrate this again in Figure 3(a). We also demonstrate that universality does indeed fail for M = N in Figure 3(b).
2.2 SPIN GLASSES AND GRADIENT DESCENT
The gradient descent algorithm for the Hamiltonian of the p-spin spherical glass will find a local minimum of the non-convex function (3). Since variance of HN (w) is typically of order N , a local minimum has size N . More precisely, by Auffinger et al. (2013), the energy of the floor level where most of local minima are located is asymptotically at −2 √ 2/3N ≈ −1.633N and the ground state is around−1.657N . The algorithm starts by picking a random element w of the sphere with radius √ N , SN−1( √ N), as a starting point for each trial. We vary the environment for each trial and introduce ensembles by setting x(·) ∼ D for a number of choices of distributions. For a fixed dimension N , accuracy that bounds the norm of the gradient, and an ensemble E: (1) Calculate the gradient steps: wt+1 = wt − ηt∇wH(wt), (2) Normalize the resulting vector to the sphere: √ N w t+1
||wt+1|| ← wt+1, and (3) Stop when the norm of the gradient size is below and record T ,N,GD,E . This procedure is repeated 10,000 times for different ensembles (i.e. different choices forD). Figure 4 exhibit the universal halting time presenting evidence that τ ,N,GD,E is independent of the ensemble.
2.3 DIGIT INPUTS VS. RANDOM INPUTS IN DEEP LEARNING
A deep learning cost function is trained on two drastically different ensembles. The first is the MNIST dataset, which consists of 60,000 samples of training examples and 10,000 samples of test examples. The model is a fully connected network with two hidden layers, that have 500 and 300 units respectively. Each hidden unit has rectified linear activation, and a cross entropy cost is attached at the end. To randomize the input data we sample 30K samples from the training set each time we set up the model and initialize the weights randomly. Then we train the model by the stochastic gradient descent method with a minibatch size of 100. This model gets us about 97%
accuracy without any further tuning. The second ensemble uses the same model and outputs, but the input data is changed from characters to independent Gaussian noise. This model, as expected, gets us only about 10% accuracy: it randomly picks a number! The stopping condition is reached when the average of successive differences in cost values goes below a prescribed value. As a comparison we have also added a deep convolutional network (convnet), and we used the fully connected model with a different stopping condition: one that is tied to the norm of the gradient. Figure 5 demonstrates universal fluctuations in the halting time in all of the four cases.
3 CONCLUSIONS
What are the conditions on the ensembles and the model that lead to such universality? What constitutes a good set of hyperparameters for a given algorithm? How can we go beyond inspection when tuning a system? How can we infer if an algorithm is a good match to the system at hand? What is the connection between the universal regime and the structure of the landscape? This research attempts to exhibit cases where one can extract answers to these questions in a robust and quantitative way. The examples we have presented clearly exhibit universality. The normalized moment analysis, presented in the Appendix, gives a quantitative way to test for universality. And we further believe that an algorithm that exhibits universality is running in a scaling region of “high performance”: universality is a measure of insensitivity to initial data which is a beneficial property of a numerical method. Establishing this claim is a difficult task, beyond the scope of this primarily empirical work.
More specifically, the current work gives empirical evidence that within an appropriate scaling region, the halting time can often be approximated as
T ,N,A,E ≈ µ+ στ∗A, where τ∗A is a mean-zero, variance one universal distribution. If this holds, a simple estimate of the mean µ = µ ,N,A,E and the standard deviation σ = σ ,N,A,E using a few samples will give a good a priori estimate of algorithm run time
P (|T ,N,A,E − µ| ≥ σ`) ≈ P (|τ∗A| ≥ `) . If τ∗A has (or is just conjectured to have) exponential tails, for example, this can be quite useful.
This work also validates the broad claims made in Deift et al. (2015) that universality is present in all or nearly all (sensible) computation. Future work will be along the lines of using these heuristics to identify when we have universality, to identify the different kinds of landscapes, and to guide both algorithm development and algorithm tuning. Furthermore, one would like theoretical estimates for the mean µ ,N,A,E and the standard deviation σ ,N,A,E .
ACKNOWLEDGMENTS
We thank Percy Deift for valuable discussions and Gérard Ben Arous for his mentorship throughout the process of this research. The first author thanks very much to Uğur Güney for his availability for support and valuable contributions in countless implementation issues. This work was partially supported by the National Science Foundation under grant number DMS-1303018 (TT).

Summary
For several algorithms, previous research has shown that the halting time follows a two-parameter distribution (the so-called universal property investigated by the authors). In this work, the authors extend the investigation to new algorithms (spin-glass, gradient descent in deep learning).
An algorithm is considered to satisfy the universality property when the centered/scaled halting time fluctuations (empirical distribution of halting times) depend on the algorithm but do not depend on the target accuracy epsilon, an intrinsic measure of dimension N, the probability distribution/random ensemble. (This is clear from Eq 1 where on the left the empirical halting time distribution depends on epsilon, N, A, E and on the right, the approximation only depends on the algorithm)
The authors argue that empirically, the universal property is observed when both algorithms (spin glass and deep learning) perform well and that it is not observed when they do not perform well.
A moment-based indicator is introduced to assess whether universality is observed.
Review
This paper presents several problems.
page 2: “[…] for sufficiently large N and eps = eps(N)”
The dependence of epsilon on N is troubling.
page 3: “Universality is a measure of stability in an algorithm […] For example […] halting time for the power method […] has infinite expectation and hence this type of universality is *not* present. One could use this to conclude that the power method is naive. Therefore the presence of universality is a desirable feature of a numerical method”
No. An algorithm is naive if there are better ways to answer the problem. One could not conclude from a halting time with infinite expectation (e.g. solving a problem extremely quickly 99% of the time, and looping forever in 1% of cases) or infinite variance, that the algorithm is naive.

Moreover the universal property is more restrictive than having a finite halting time expectation. Even if in many specific cases, having a finite halting time expectation is a desirable property, showing that the presence of universality is desirable would require a demonstration that the other more restrictive aspects are also desirable.

Also, the paragraph only concerns one algorithm. why would the conclusions generalise to all numerical methods ?

Even if the universality property is arguably desirable (i.e. event if the conclusion of this paragraph is assumed correct), the paragraph does not support the given conclusion.
Comparing Eq 1 and figures 2,3,4,5

From Eq 1, universality means that the centered/scaled halting time fluctuations (which depend on A, epsilon, N, E) can be approximated by a distribution that only depends on A (not on epsilon, N, E) but in the experiments only E varies (figures 2,3,4,5). The validity of the approximation with varying epsilon or N is never tested
The ensembles/distributions parameter E (on which halting fluctuations should not depend) and the algorithm A (on which halting fluctuations are allowed to depend) are not well defined, especially w.r.t. the common use of the words. In the optimisation setting we are told that the functional form of the landscape function is part of A (in answer to the question of a reviewer) but what is part of the functional form ? what about computations where the landscape has no known functional form (black box) ?
The conclusion claims that the paper “attempts to exhibit cases” where one can answer 5 questions in a robust and quantitative way.
Question 1: “What are the conditions on the ensembles and the model that lead to such universality ?”
The only quantitative way would be to use the moments based indicator however there is only one example of universality not being observed which concerns only one algorithm (conjugate gradient) and one type of failure (when M = N). This does not demonstrate robustness of the method.
Question 2: “What constitutes a good set of hyper parameters for a given algorithm ?”
The proposed way to choose would be to test whether universality is observed. If it is then the hyper parameters are good, if not the hyper parameters are bad. The correspondance between bad hyper-parameters and observing no universality concerns only one algorithm and one type of failure. Other algorithms may fail in the universal regime or perform well in the non universal regime. The paper does not show how to answer this question in a robust way.
Question 3: "How can we go beyond inspection when tuning a system ?
"
The question is too vague and general and there is probably no robust and quantitative way to answer it at all.
Question 4: "How can we infer if an algorithm is a good match to the system at hand ?
"
The paper fails to demonstrate convincingly that universality is either a good or robust way to approach the very few studied algorithms. The suggested generalisation to all systems and algorithms is extremely far fetched.
Question 5: "What is the connection between the universal regime and the structure of the landscape ?"

Same as before, the question is extremely vague and cannot be answered in a robust or quantitative way at all. The fact that what corresponds to A and what corresponds to E is not clear does not help.
In the conclusion it is written that the paper validates the claim that universality is present in all or nearly all sensible computation. It does not. The paper does not properly test whether universality is present (only 1 parameter in 3 that should not vary is tested). The paper does not properly test whether universality is lost when the computation is no longer sensible (only one failure case tested). Finally the experiments do not apply to all or nearly all computations but only to very few  specific algorithms.