LEARNING TO DISCOVER SPARSE GRAPHICAL MODELS
We consider structure discovery of undirected graphical models from observational data. Inferring likely structures from few examples is a complex task often requiring the formulation of priors and sophisticated inference procedures. In the setting of Gaussian Graphical Models (GGMs) a popular estimator is a maximum likelihood objective with a penalization on the precision matrix. Adapting this estimator to capture domain-specific knowledge as priors or a new data likelihood requires great effort. In addition, structure recovery is an indirect consequence of the data-fit term. By contrast, it may be easier to generate training samples of data that arise from graphs with the desired structure properties. We propose here to leverage this latter source of information as training data to learn a function mapping from empirical covariance matrices to estimated graph structures. Learning this function brings two benefits: it implicitly models the desired structure or sparsity properties to form suitable priors, and it can be tailored to the specific problem of edge structure discovery, rather than maximizing data likelihood. We apply this framework to several real-world problems in structure discovery and show that it can be competitive to standard approaches such as graphical lasso, at a fraction of the execution speed. We use convolutional neural networks to parametrize our estimators due to the compositional structure of the problem. Experimentally, our learnable graph-discovery method trained on synthetic data generalizes well: identifying relevant edges in real data, completely unknown at training time. We find that on genetics, brain imaging, and simulation data we obtain competitive (and generally superior) performance, compared with analytical methods.
1 INTRODUCTION
Probabilistic graphical models provide a powerful framework for describing the dependencies between a set of variables. Many applications infer the structure of a probabilistic graphical model from data to elucidate the relationships between variables. These relationships are often represented by an undirected graphical model also known as a Markov Random Field (MRF). We focus on a common MRF model, Gaussian graphical models (GGMs). GGMs are used in structure-discovery settings for rich data such as neuroimaging, genetics, or finance (Friedman et al., 2008; Ryali et al, 2012; Mohan et al., 2012; Belilovsky et al., 2016). Although multivariate Gaussian distributions are well-behaved, determining likely structures from few examples is a complex task when the data is high dimensional. It requires strong priors, typically a sparsity assumption, or other restrictions on the structure of the graph, which now make the distribution difficult to express analytically and use.
A standard approach to estimating structure with GGMs in high dimensions is based on the classic result that the zeros of a precision matrix correspond to zero partial correlation, a necessary and sufficient condition for conditional independence (Lauritzen, 1996). Assuming only a few conditional dependencies corresponds to a sparsity constraint on the entries of the precision matrix, leading to a combinatorial problem. Many popular approaches to learning GGMs can be seen as leveraging the
`1-norm to create convex surrogates to this problem. Meinshausen & Bühlmann (2006) use nodewise `1 penalized regressions. Other estimators penalize the precision matrix directly (Cai et al., 2011; Friedman et al., 2008; Ravikumar et al., 2011). The most popular being the graphical lasso
fglasso(Σ̂) = arg min Θ 0 − log |Θ|+ Tr (Σ̂Θ) + λ‖Θ‖1, (1)
which can be seen as a penalized maximum-likelihood estimator. Here Θ and Σ̂ are the precision and sample covariance matrices, respectively. A large variety of alternative regularization penalties extend the priors of the graphical lasso (Danaher et al., 2014; Ryali et al, 2012; Varoquaux et al., 2010). However, several problems arise in this approach. Constructing novel surrogates for structured-sparsity assumptions on MRF structures is challenging, as a prior needs to be formulated and incorporated into a penalized maximum likelihood objective which then needs an efficient optimization algorithm to be developed, often within a separate research effort. Furthermore, model selection in a penalized maximum likelihood setting is difficult as regularization parameters are often unintuitive.
We propose to learn the estimator. Rather than manually designing a specific graph-estimation procedure, we frame this estimator-engineering problem as a learning problem, selecting a function from a large flexible function class by risk minimization. This allows us to construct a loss function that explicitly aims to recover the edge structure. Indeed, sampling from a distribution of graphs and empirical covariances with desired properties is often possible, even when this distribution is not analytically tractable. As such we can perform empirical risk minimization to select an appropriate function for edge estimation. Such a framework gives more easy control on the assumed level of sparsity (as opposed to graph lasso) and can impose structure on the sampling to shape the expected distribution, while optimizing a desired performance metric.
For particular cases we show that the problem of interest can be solved with a polynomial function, which is learnable with a neural network (Andoni et al., 2014). Motivated by this fact, as well as theoretical and empricial results on learning smooth functions approximating solutions to combinatorial problems (Cohen et al., 2016; Vinyals et al., 2015), we propose to use a particular convolutional neural network as the function class. We train it by sampling small datasets, generated from graphs with the prescribed properties, with a primary focus on sparse graphical models. We estimate from this data small-sample covariance matrices (n < p), where n is the number of samples and p is the dimensionality of the data. Then we use them as training data for the neural network (Figure 2) where target labels are indicators of present and absent edges in the underlying GGM. The learned network can then be employed in various real-world structure discovery problems.
In Section 1.1 we review the related work. In Section 2 we formulate the risk minimization view of graph-structure inference and describe how it applies to sparse GGMs. Section 2.3 describes and motivates the deep-learning architecture we chose to use for the sparse GGM problem in this work. In Section 3 we describe the details of how we train an edge estimator for sparse GGMs. We then evaluate its properties extensively on simulation data. Finally, we show that this edge estimator trained only on synthetic data can obtain state of the art performance at inference time on real neuroimaging and genetics problems, while being much faster to execute than other methods.
1.1 RELATED WORK
Lopez-Paz et al. (2015) analyze learning functions to identify the structure of directed graphical models in causal inference using estimates of kernel-mean embeddings. As in our work, they demonstrate the use of simulations for training while testing on real data. Unlike our work, they primarily focus on finding the causal direction in two node graphs with many observations.
Our learning architecture is motivated by the recent literature on deep networks. Vinyals et al. (2015) have shown that neural networks can learn approximate solutions to NP-hard combinatorial problems, and the problem of optimal edge recovery in MRFs can be seen as a combinatorial optimization problem. Several recent works have been proposed which show neural architectures for graph input data (Henaff et al., 2015; Duvenaud et al, 2015; Li et al., 2016). These are based on multi layer convolutional networks, as in our work, or multi-step recurrent neural networks. The input in our approach can be viewed as a complete graph, while the ouput a sparse graph, thus none of these are directly applicable. A related use of deep networks to approximate a posterior distribution can be found in Balan et al. (2015). Finally, Gregor & LeCun (2010); Xin et al. (2016) use deep networks to approximate steps of a known sparse recovery algorithm.
Bayesian approaches to structure learning rely on priors on the graph combined with sampling techniques to estimate the posterior of the graph structure. Some approaches make assumptions on the decomposability of the graph (Moghaddam et al., 2009). The G-Wishart distribution is a popular distribution which forms part of a framework for structure inference, and advances have been recently made in efficient sampling (Mohammadi & Wit, 2015). These methods can still be rather slow compared to competing methods, and in the setting of p > n we find they are less powerful.
2 METHODS
2.1 LEARNING AN APPROXIMATE EDGE ESTIMATION PROCEDURE
We consider MRF edge estimation as a learnable function. Let X ∈ Rn×p be a matrix whose n rows are i.i.d. samples x ∼ P (x) of dimension p. Let G = (V,E) be an undirected and unweighted graph associated with the set of variables in x. Let L = {0, 1} and Ne = p(p−1)2 the maximum possible edges in E. Let Y ∈ LNe indicate the presence or absence of edges in the edge set E of G, namely
Y ij = { 0 xi ⊥ xj |xV \i,j 1 xi 6⊥ xj |xV \i,j
(2)
We define an approximate structure discovery method gw(X), which produces a prediction of the edge structure, Ŷ = gw(X), given a set of data X . We focus on X drawn from a Gaussian distribution. In this case, the empirical covariance matrix, Σ̂, is a sufficient statistic of the population covariance and therefore of the conditional dependency structure. We thus express our structure-recovery problem as a function of Σ̂: gw(X) := fw(Σ̂). fw is parametrized by w and belongs to the function class F . We note that the graphical lasso in Equation (1) is an fw for an appropriate choice of F . This view on the edge estimator now allows us to bring the selection of fw from the domain of human design to the domain of empirical risk minimization over F . Defining a distribution P on Rp×p × LNe such that (Σ̂, Y ) ∼ P, we would like our estimator, fw, to minimize the expected risk R(f) = E(Σ̂,Y )∼P[l(f(Σ̂), Y )] (3) Here l : LNe×LNe → R+ is the loss function. For graphical model selection the 0/1 loss function is the natural error metric to consider (Wang et al., 2010). The estimator with minimum risk is generally not possible to compute as a closed form expression for most interesting choices of P, such as those arising from sparse graphs. In this setting, Eq. (1) achieves the information theoretic optimal recovery rate up to a constant for certain P corresponding to uniformly sparse graphs with a maximum degree, but only when the optimal λ is used and the non-zero precision matrix values are bounded away from zero (Wang et al., 2010; Ravikumar et al., 2011).
The design of the estimator in Equation (1) is not explicitly minimizing this risk functional. Thus modifying the estimator to fit a different class of graphs (e.g. small-world networks) while minimizing R(f) is not obvious. Furthermore, in practical settings the optimal λ is unknown and precision matrix entries can be very small. We would prefer to directly minimize the risk functional. Desired structural assumptions on samples from P on the underlying graph, such as sparsity, may imply that the distribution is not tractable for analytic solutions. Meanwhile, we can often devise a sampling procedure for P allowing us to select an appropriate function via empirical risk minimization. Thus it is sufficient to define a rich enough F over which we can minimize the empirical risk over the samples generated, giving us a learning objective over N samples {Yk,Σk}Nk=1 drawn from P: min w 1 N ∑N k=1 l(fw(Σ̂k), Yk). To maintain tractability, we use the standard cross-entropy loss as a convex surrogate, l̂ : RNe × LNe , given by: l̂(fw(Σ̂), Y ) =
∑ i 6=j ( Y ij log(f ijw (Σ̂)) + (1− Y ij) log(1− f ijw (Σ̂)) ) . (4)
We now need to select a sufficiently rich function class for fw and a method to produce appropriate (Y, Σ̂) which model our desired data priors. This will allow us to learn a fw that explicitly attempts to minimize errors in edge discovery.
2.2 DISCOVERING SPARSE GAUSSIAN GRAPHICAL MODELS AND BEYOND
We discuss how the described approach can be applied to recover sparse Gaussian graphical models. A typical assumption in many modalities is that the number of edges is sparse. A convenient property of these GGMs is that the precision matrix has a zero value in the (i, j)th entry precisely when variables i and j are independent conditioned on all others. Additionally, the precision matrix and partial correlation matrix have the same sparsity pattern, while the partial correlation matrix has normalized entries.
Algorithm 1 Training a GGM edge estimator for i ∈ {1, .., N} do
Sample Gi ∼ P(G) Sample Σi ∼ P(Σ|G = Gi) Xi ← {xj ∼ N(0,Σi)}nj=1 Construct (Yi, Σ̂i) pair from (Gi,Xi)
end for Select Function Class F (e.g. CNN) Optimize: min
f∈F 1 N
∑N k=1 l̂(f(Σ̂k), Yk)) We propose to simulate our a priori assumptions of sparsity and Gaussianity to learn fw(Σ̂), which can then produce predictions of edges from the input data. We model P (x|G) as arising from a sparse prior on the graph G and correspondingly the entries of the precision matrix Θ. To obtain a single sample of X corresponds to n i.i.d. samples from N (0,Θ−1). We can now train fw(Σ̂) by generating sample pairs (Σ̂, Y ). At execution time we standardize the input data and compute the covariance matrix before evaluating fw(Σ̂). The process of learning fw for the sparse GGM is given in Algorithm 1. A weakly-informative sparsity prior is one where each edge is equally likely with small probability, versus structured sparsity where edges have specific configurations. For obtaining the training samples (Σ̂, Y ) in this case we would like to create a sparse precision matrix, Θ, with the desired number of zero entries distributed uniformly. One strategy to do this and assure the precision matrices lie in the positive definite cone is to first construct an upper triangular sparse matrix and then multiply it by its transpose. This process is described in detail in the experimental section. Alternatively, an MCMC based G-Wishart distribution sampler can be employed if specific structures of the graph are desired (Lenkoski, 2013).
The sparsity patterns in real data are often not uniformly distributed. Many real world networks have a small-world structure: graphs that are sparse and yet have a comparatively short average distance between nodes. These transport properties often hinge on a small number of high-degree nodes called hubs. Normally, such structural patterns require sophisticated adaptation when applying estimators like Eq. (1). Indeed, high-degree nodes break the small-sample, sparse-recovery properties of `1-penalized estimators (Ravikumar et al., 2011). In our framework such structural assumptions appear as a prior that can be learned offline during training of the prediction function. Similarly priors on other distributions such as general exponential families can be more easily integrated. As the structure discovery model can be trained offline, even a slow sampling procedure may suffice.
2.3 NEURAL NETWORK GRAPH ESTIMATOR
In this work we propose to use a neural network as our function fw. To motivate this let us consider the extreme case when n p. In this case Σ̂ ≈ Σ and thus entries of Σ̂−1 or the partial correlation that are almost equal to zero can give the edge structure. Definition 1 (P-consistency). A function class F is P-consistent if ∃f ∈ F such that
E(Σ̂,Y )∼P[l(f(Σ̂), Y )]→ 0 as n→∞ with high probability.
Proposition 1 (Existence of P-consistent neural network graph estimator). There exists a feed forward neural network function class F that is P-consistent. Proof. If the data is standardized, each entry of Σ corresponds to the correlation ρi,j . The partial correlation of edge (i, j) conditioned on nodes Z, is given recursively as
ρi,j|Z = (ρi,j|Z\zo − ρi,zo|Z\zoρj,zo|Z\zo) 1
D . (5)
We may ignore the denominator, D, as we are interested in I(ρi,j|Z = 0). Thus we are left with a recursive formula that yields a high degree polynomial. From Andoni et al. (2014, Theorem 3.1) using gradient descent, a neural network with only two layers can learn a polynomial function of degree d to arbitrary precision given sufficient hidden units.
Remark 1. Naïvely the polynomial from the recursive definition of partial correlation is of degree bounded by 2p−2. In the worst case, this would seem to imply that we would need an exponentially
growing number of hidden nodes to approximate it. However, this problem has a great deal of structure that can allow efficient approximation. Firstly, higher order monomials will go to zero quickly with a uniform prior on ρi,j , which takes values between 0 and 1, suggesting that in many cases a concentration bound exists that guarantees non-exponential growth. Furthermore, the existence result is shown already for a shallow network, and we expect a logarithmic decrease in the number of parameters to peform function estimation with a deep network (Cohen et al., 2016).
Moreover, there are a great deal of redundant computations in Eq. (5) and an efficient dynamic programming implementation can yield polynomial computation time and require only low order polynomial computations with appropriate storage of previous computation. Similarly we would like to design a network that would have capacity to re-use computations across edges and approximate low order polynomials. We also observe that the conditional independence of nodes i, j given Z can be computed equivalently in many ways by considering many paths through the nodes Z. Thus we can choose any valid ordering for traversing the nodes starting from a given edge.
We propose a series of shared operations at each edge. We consider a feedforward network where each edge i, j is associated with a fixed sized vector, oki,j , of dimensionality d at each layer, k > 0. o0i,j is initialized to the covariance entries at k = 0. For each edge we start with a neighborhood of the 6 adjacent nodes, i, j, i-1, i+1, j-1, j+1 for which we take all corresponding edge values from the covariance matrix illustrated in Figure 1. We proceed at each layer to increase the nodes considered for each edge, the output at each layer progressively increasing the receptive field making sure all values associated with the considered nodes are present. The receptive field here refers to the original covariance entries which are accessible by a given, oki,j (Luo et al., 2010). The equations defining the process are shown in Figure 1. Here a neural network fwk is applied at each edge at each layer and a dilation sequence dk is used. We call a network of this topology a D-Net of depth l. We use dilation here to allow the receptive field to grow fast, so the network does not need a great deal of layers. We make the following observations: Proposition 2. For general P it is a necessary condition for P-consistency that the receptive field of D-Net covers all entries of the covariance, Σ̂, at any edge it is applied. Proof. Consider nodes i and j and a chain graph such that i and j are adjacent to each other in the matrix but are at the terminal nodes of the chain graph. One would need to consider all other variables to be able to explain away the correlation. Alternatively we can see this directly from expanding Eq. (5).
Proposition 3. A p× p matrix Σ̂ will be covered by the receptive field for a D-Net of depth log2(p) and dk = 2k−1 Proof. The receptive field of a D-Net with dilation sequence dk = 2k−1 of depth l is O(2l). We can see this as oki,j will receive input from o k−1 a,b at the edge of it’s receptive field, effectively doubling it. It now follows that we need at least log2(p) layers to cover the receptive field.
Intuitively adjacent edges have a high overlap in their receptive fields and can easily share information about the non-overlapping components. This is analogous to a parametrized message passing. For example if edge (i, j) is explained by node k, as k enters the receptive field of edge (i, j − 1),
the path through (i, j) can already be discounted. In terms of Eq. 5 this can correspond to storing computations that can be used by neighbor edges from lower levels in the recursion.
Here fwk is shared amongst all nodes and thus we can implement this as a special kind of convolutional network. We make sure that to have considered all edges relevant to the current set of nodes in the receptive field which requires us to add values from filters applied at the diagonal to all edges. In Figure 1 we illustrate the nodes and receptive field considered with respect to the covariance matrix. This also motivates a straightforward implementation using 2D convolutions (adding separate convolutions at i, i and j, j to each i, j at each layer to achieve the specific input pattern described) shown in (Figure 2).
Ultimately our choice of architecture that has shared computations and multiple layers is highly scalable as compared with a naive fully connected approach and allows leveraging existing optimized 2-D convolutions. In preliminary work we have also considered fully connected layers but this proved to be much less efficient in terms of storage and scalibility than using deep convolutional networks.
Considering the general n p case is illustrative. However, the main advantages of making the computations differentiable and learned from data is that we can take advantage of the sparsity and structure assumptions on the target function to obtain more efficient results than naive computation of partial correlation or matrix inversion. As n decreases our estimate of ρ̂i,j becomes inexact and here a data driven model which can take advantage of the assumptions on the underlying distribution can more accurately recover the graph structure.
The convolution structure is dependent on the order of the variables used to build the covariance matrix, which is arbitrary. Permuting the input data we can obtain another estimate of the output. In the experiments, we leverage these various estimate in an ensembling approach, averaging the results of several permutations of input. We observe that this generally yields a modest increase in accuracy, but that even a single node ordering can show substantially improved performance over competing methods in the literature.
3 EXPERIMENTS
Our experimental evaluations focus on the challenging high dimensional settings in which p > n and consider both synthetic data and real data from genetics and neuroimaging. In our experiments we explore how well networks trained on parametric samples generalize, both to unseen synthetic data and to several real world problems. In order to highlight the generality of the learned networks, we apply the same network to multiple domains. We train networks taking in 39, 50, and 500 node graphs. The former sizes are chosen based on the real data we consider in subsequent sections. We refer to these networks as DeepGraph-39, 50, and 500. In all cases we have 50 feature maps of 3× 3 kernels. The 39 and 50 node network with 6 convolutional layers and dk = k + 1. For the 500 node network with 8 convolutional layers and dk = 2k+1. We use ReLU activations. The last layer has 1× 1 convolution and a sigmoid outputing a value of 0 to 1 for each edge. We sample P (X|G) with a sparse prior on P (G) as follows. We first construct a lower diagonal matrix, L, where each entry has α probability of being zero. Non-zero entries are set uniformly between −c and c. Multiplying LLT gives a sparse positive definite precision matrix, Θ. This gives us our P (Θ|G) with a sparse prior on P (G). We sample from the Gaussian N (0,Θ−1) to obtain
samples of X . Here α corresponds approximately to a specific sparsity level in the final precision matrix, which we set to produce matrices 92− 96% sparse and c chosen so that partial correlations range 0 to 1.
Each network is trained continously with new samples generated until the validation error saturates. For a given precision matrix we generate 5 possible X samples to be used as training data, with a total of approximately 100K training samples used for each network. The networks are optimized using ADAM (Kingma & Ba, 2015) coupled with cross-entropy loss as the objective function (cf. Sec. 2.1). We use batch normalization at each layer. Additionally, we found that using the absolute value of the true partial correlations as labels, instead of hard binary labels, improves results.
Synthetic Data Evaluation To understand the properties of our learned networks, we evaluated them on different synthetic data than the ones they were trained on. More specifically, we used a completely different third party sampler so as to avoid any contamination. We use DeepGraph-39 on a variety of settings. The same trained network is utilized in the subsequent neuroimaging evaluations as well. DeepGraph-500 is also used to evaluate larger graphs.
We used the BDGraph R-package to produce sparse precision matrices based on the G-Wishart distribution (Mohammadi & Wit, 2015) as well as the R-package rags2ridges (Peeters et al., 2015) to generate data from small-world networks corresponding to the Watts–Strogatz model (Watts & Strogatz, 1998). We compared our learned estimator against the scikit-learn (Pedregosa et al, 2011) implementation of Graphical Lasso with regularizer chosen by cross-validation as well as the Birth-Death Rate MCMC (BDMCMC) method from Mohammadi & Wit (2015).
For each scenario we repeat the experiment for 100 different graphs and small sample observations showing the average area under the ROC curve (AUC), precision@k corresponding to 5% of possible edges, and calibration error (CE) (Mohammadi & Wit, 2015).
For graphical lasso we use the partial correlations to indicate confidence in edges; BDGraph automatically returns posterior probabilities as does our method. Finally to understand the effect of the regularization parameter we additionally report the result of graphical lasso under optimal regularizer setting on the testing data.
Our method dominates all other approaches in all cases with p > n (which also corresponds to the training regime). For the case of random Gaussian graphs with n=35 (as in our training data), and graph sparsity of 95%, we have superior performance and can further improve on this by averaging permutations. Next we apply the method to a less straightforward synthetic data, with distributions typical of many applications. We found that, compared to baseline methods, our network performs particularly well with high-degree nodes and when the distribution becomes non-normal. In particular our method performs well on the relevant metrics with small-world networks, a very common family of graphs in real-world data, obtaining superior precision at the primary levels of interest. Figure 3 shows examples of random and Watts-Strogatz small-world graphs used in these experiments.
Training a new network for each number of samples can pose difficulties with our proposed method. Thus we evaluted how robust the network DeepGraph-39 is to input covariances obtained from fewer or more samples. We find that overall the performance is quite good even when lowering the number of samples to n = 15, we obtain superior performance to the other approaches (Table 1). We also applied DeepGraph-39 on data from a multivariate generalization of the Laplace distribution (Gómez et al., 1998). As in other experiments precision matrices were sampled from the G-Wishart at a sparsity of 95%. Gómez et al. (1998, Proposition 3.1) was applied to produce samples. We find that DeepGraph-39 performs competitively, despite the discrepancy between train and test distributions. Experiments with variable sparsity are considered in the supplementary material, which find that for very sparse graphs, the networks remain robust in performance, while for increased density performance degrades but remains competitive.
Using the small-world network data generator (Peeters et al., 2015), we demonstrate that we can update the generic sparse prior to a structured one. We re-train DeepGraph-39 using only 1000 examples of small-world graphs mixed with 1000 examples from the original uniform sparsity model. We perform just one epoch of training and observe markedly improved performance on this test case as seen in the last row of Table 1.
For our final scenario we consider the very challenging setting with 500 nodes and only n = 50 samples. We note that the MCMC based method fails to converge at this scale, while graphical lasso is very slow as seen in the timing performance and barely performs better than chance. Our method convincingly outperforms graphical lasso in this scenario. Here we additionally report precision at just the first 0.05% of edges since competitors perform nearly at chance at the 5% level.
Cancer Genome Data We perform experiments on a gene expression dataset described in Honorio et al. (2012). The data come from a cancer genome atlas from 2360 subjects for various types of cancer. We used the first 50 genes from Honorio et al. (2012, Appendix C.2) of commonly regulated genes in cancer. We evaluated on two groups of subjects, one with breast invasive carcinoma (BRCA) consisting of 590 subjects and the other colon adenocarcinoma (CODA) consisting of 174 subjects.
Evaluating edge selection in real-world data is challenging. We use the following methodology: for each method we select the top-k ranked edges, recomputing the maximum likelihood precision matrix with support given by the corresponding edge selection method. We then evaluate the likelihood on a held-out set of data. We repeat this procedure for a range of k. We rely on Algorithm 0 in Hara & Takemura (2010) to compute the maximum likelihood precision given a support. The experiment is repeated for each of CODA and BRCA subject groups 150 times. Results are shown in Figure 4. In all cases we use 40 samples for edge selection and precision estimation. We compare with graphical lasso as well as the Ledoit-Wolf shrinkage estimator (Ledoit & Wolf, 2004). We additionally consider the MCMC based approach described in previous section. For graphical lasso and Ledoit-Wolf, edge selection is based on thresholding partial correlation (Balmand & Dalalyan, 2016).
Additionally, we evaluate the stability of the solutions provided by the various methods. In several applications a low variance on the estimate of the edge set is important. On Table 3, we report
Spearman correlations between pairs of solutions, as it is a measure of a monotone link between two variables. DeepGraph has far better stability in the genome experiments and is competitive in the fMRI data.
Resting State Functional Connectivity We evaluate our graph discovery method to study brain functional connectivity in resting-state fMRI data. Correlations in brain activity measured via fMRI reveal functional interactions between remote brain regions. These are an important measure to study psychiatric diseases that have no known anatomical support. Typical connectome analysis describes each subject or group by a GGM measuring functional connectivity between a set of regions (Varoquaux & Craddock, 2013). We use the ABIDE dataset (Di Martino et al, 2014), a large scale resting state fMRI dataset. It gathers brain scans from 539 individuals suffering from autism spectrum disorder and 573 controls over 16 sites.1 For our experiments we use an atlas with 39 regions of interest derived in Varoquaux et al. (2011).
We use the network DeepGraph-39, the same network and parameters from synthetic experiments, using the same evaluation protocol as used in the genomic data. For both control and autism patients we use time series from 35 random subjects to estimate edges and corresponding precision matrices. We find that for both the Autism and Control group we can obtain edge selection comparable to graph lasso for very few selected edges. When the number of selected edges is in the range above 25 we begin to perform significantly better in edge selection as seen in Fig. 4. We evaluated stability of the results as shown in Tab. 3. DeepGraph outperformed the other methods across the board.
ABIDE has high variability across sites and subjects. As a result, to resolve differences between approaches, we needed to perform 1000 folds to obtain well-separated error bars. We found that the birth-death MCMC method took very long to converge on this data, moreover the need for many folds to obtain significant results amongst the methods made this approach prohibitively slow to evaluate.
1http://preprocessed-connectomes-project.github.io/abide/
We show the edges returned by Graph Lasso and DeepGraph for a sample from 35 subjects (Fig. 5) in the control group. We also show the result of a large-sample result based on 368 subjects from graphical lasso. In visual evaluation of the edges returned by DeepGraph we find that they closely align with results from a large-sample estimation procedure. Furthermore we can see several edges in the subsample which were particularly strongly activated in both methods.
4 DISCUSSION AND CONCLUSIONS
Our method was competitive with strong baselines. Even in cases that deviate from standard GGM sparsity assumptions (e.g. Laplacians, small-world) it performed substantially better. When finetuning on the target distribution performance further improves. Most importantly the learned estimator generalizes well to real data finding relevant stable edges. We also observed that the learned estimators generalize to variations not seen at training time (e.g. different n or sparsity), which points to this potentialy learning generic computations. This also shows potential to more easily scale the method to different graph sizes. One could consider transfer learning, where a network for one size of data is used as a starting point to learn a network working on larger dimension data.
Penalized maximum likelihood can provide performance guarantees under restrictive assumptions on the form of the distribution and not considering the regularization path. In the proposed method one could obtain empirical bounds under the prescribed data distribution. Additionally, at execution time the speed of the approach can allow for re-sampling based uncertainty estimates and efficient model selection (e.g. cross-validation) amongst several trained estimators.
We have introduced the concept of learning an estimator for determining the structure of an undirected graphical model. A network architecture and sampling procedure for learning such an estimator for the case of sparse GGMs was proposed. We obtained competitive results on synthetic data with various underlying distributions, as well as on challenging real-world data. Empirical results show that our method works particularly well compared to other approaches for small-world networks, an important class of graphs common in real-world domains. We have shown that neural networks can obtain improved results over various statistical methods on real datasets, despite being trained with samples from parametric distributions. Our approach enables straightforward specifications of new priors and opens new directions in efficient graphical structure discovery from few examples.
ACKNOWLEDGEMENTS
This work is partially funded by Internal Funds KU Leuven, FP7-MC-CIG 334380, DIGITEO 2013- 0788D - SOPRANO, and ANR-11-BINF-0004 NiConnect. We thank Jean Honorio for providing pre-processed Cancer Genome Data.
A SUPPLEMENTARY EXPERIMENTS
A.1 PREDICTING COVARIANCE MATRICES
Using our framework it is possible to attempt to directly predict an accurate covariance matrix given a noisy one constructed from few observations. This is a more challenging task than predicting the edges. In this section we show preliminay experiments which given an empirical covariance matrix from few observations attempts to predict a more accurate covariance matrix that takes into account underlying sparse data dependency structure.
One challenge is that outputs of our covariance predictor must be on the positive semidefinite cone, thus we choose to instead predict on the cholesky decompositions, which allows us to always produce positive definite covariances. We train a similar structure to DeepGraph-39 structure modifying the last layer to be fully connected linear layer that predicts on the cholesky decomposition of the true covariance matrices generated by our model with a squared loss.
We evaluate this network using the ABIDE dataset described in Section 3. The ABIDE data has a large number of samples allowing us to obtain a large sample estimate of the covariance and compare it to our estimator as well as graphical lasso and empirical covariance estimators. Using the large sample ABIDE empirical covariance matrix. We find that we can obtain competitive `2 and `∞ norm using few samples. We use 403 subjects from the ABIDE Control group each with a recording of 150 − 200 samples to construct covariance matrix, totaling 77 330 samples (some correlated). This acts as our very approximate estimate of the population Σ. We then evaluate covariance estimation on 35 samples using the empirical covariance estimator, graphical lasso, and DeepGraph trained to output covariance matrices. We repeat the experiment for 50 different subsamples of the data. We see in 5 that the prediction approach can obtain competitive results. In terms of `2 graphical lasso performs better, however our estimate is better than empirical covariance estimation and much faster then graphical lasso. In some applications such as robust estimation a fast estimate of the covariance matrix (automatically embedding sparsity assumptions) can be of great use. For `∞ error we see the empirical covariance estimation outperforms graphical lasso and DeepGraph for this dataset, while DeepGraph performs better in terms of this metric.
We note these results are preliminary, as the covariance predicting networks were not heavily optimized, moreover the ABIDE dataset is very noisy even when pre-processed and thus even the large sample covariance estimate may not be accurate. We believe this is an interesting alternate application of our paper.
A.2 ADDITIONAL SYNTHETIC RESULTS ON SPARSITY
We investigate the affect of sparsity on DeepGraph-39 which has been trained with input that has sparsity 96% − 92% sparse. We find that DeepGraph performs well at the 2% sparsity level despite not seeing this at training time. At the same time performance begins to degrade for 15% but is still competitive in several categories. The results are shown in Table 6. Future investigation can consider how alternate variation of sparsity at training time will affect these results.
A.3 APPLICATION OF LARGER NETWORK ON SMALLER INPUT
We perform preliminary investigation of application of a network trained for a larger number of nodes to a smaller set of nodes. Specifically, we consider the breast invasive carcinoma groups gene data. We now take all 175 valid genes from Appendix C.2 of Honorio et al. (2012). We take the network trained on 500 nodes in the synthetic experiments section. We use the same experimental setup as in the gene experiments. The 175× 175
covariance matrix from 40 samples and padded to the appropriate size. We observe that DeepGraph has similar performance to graph lasso while permuting the input and ensembling the result gives substantial improvement.
A.4 PERMUTATION AS ENSEMBLE METHOD
As discussed in Section 2.3, permuting the input and averaging several permutations can produce an improved result empirically. We interpret this as a typical ensembling method. This can be an advantage of the proposed architecture as we are able to easily use standard ensemble techniques. We perform an experiment to further verify that indeed the permutation of the input (and subsequent inverse permutation) allows us to produce separate classifiers that have uncorrelated errors.
We use the setup from the synthetic experiments with DeepGraph-39 in Section 3 with n = 35 and p = 39. We construct 20 permutation matrices as in the experimental section. Treating each as a separate classifier we compute the correlation coefficient of the errors on 50 synthetic input examples. We find that the average correlation coefficient of the errors of two classifiers is 0.028± 0.002, suggesting they are uncorrelated. Finally we note the individual errors are relatively small, as can already be inferred from our extensive experimental results in Section 3. We however compute the average absolute error of all the outputs across each permutation for this set of inputs as 0.03, notably the range of outputs is 0 to 1. Thus since prediction error differ at each permutation but are accurate we can average and yield a lower total prediction error.
Finally we note that our method is extremely efficient computationally thus averaging the results of several permutations is practical even as the graph becomes large.

The paper proposes a novel algorithm to estimate graph structures by using a convolutional neural network to approximate the function that maps from empirical covariance matrix to the sparsity pattern of the graph. Compared with existing approaches, the new algorithm can adapt to different network structures, e.g. small-world networks, better under the same empirical risk minimization framework. Experiments on synthetic and real-world datasets show promising results compared with baselines.
In general, I think it is an interesting and novel paper. The idea of framing structure estimation as a learning problem is especially interesting and may inspire further research on related topics. The advantage of such an approach is that it allows easier adaptation to different network structure properties without designing specific regularization terms as in graph lasso.
The experiment results are also promising. In both synthetic and real-world datasets, the proposed algorithm outperforms other baselines in the small sample region.
However, the paper can be made clearer in describing the network architectures. For example, in page 5, each o^k_{i,j} is said be a d-dimensional vector. But from the context, it seems o^k_{i,j} is a scalar (from o^0_{i,j} = p_{i,j}). It is not clear what o^k_{i,j} is exactly and what d is. Is it the number of channels for the convolutional filters?
Figure 1 is also quite confusing. Why in (b) the table is 16 x 16 whereas in (a) there are only six nodes? And from the figure, it seems there is only one channel in each layer? What do the black squares represent and why are there three blocks of them. There are some descriptions in the text, but it is still not clear what they mean exactly.
For real-world data, how are the training data (Y, Sigma) generated? Are they generated in the same way as in the synthetic experiments where the entries are uniformly sparse? This is also related to the more general question of how to sample from the distribution P, in the case of real-world data.


TATIONS WITH A LEXICON
A synonym of a polysemous word is usually only the paraphrase of one sense among many. When lexicons are used to improve vector-space word representations, such paraphrases are unreliable and bring noise to the vector-space. The prior works use a coefficient to adjust the overall learning of the lexicons. They regard the paraphrases equally. In this paper, we propose a novel approach that regards the paraphrases diversely to alleviate the adverse effects of polysemy. We annotate each paraphrase with a degree of reliability. The paraphrases are randomly eliminated according to the degrees when our model learns word representations. In this way, our approach drops the unreliable paraphrases, keeping more reliable paraphrases at the same time. The experimental results show that the proposed method improves the word vectors. Our approach is an attempt to address the polysemy problem keeping one vector per word. It makes the approach easier to use than the conventional methods that estimate multiple vectors for a word. Our approach also outperforms the prior works in the experiments.
1 INTRODUCTION
Vector-space representations of words are reported useful and improve the performance of the machine learning algorithms for many natural language processing tasks such as name entity recognition and chunking (Turian et al., 2010), text classification (Socher et al., 2012; Le & Mikolov, 2014; Kim, 2014; Joulin et al., 2016), topic extraction (Das et al., 2015; Li et al., 2016), and machine translation (Zaremba et al., 2014; Sutskever et al., 2014).
People are still trying to improve the vector-space representations for words. Bojanowski et al. (2016) attempt to improve word vectors by involving character level information. Other works (Yu & Dredze, 2014; Xu et al., 2014; Faruqui et al., 2015; Bollegala et al., 2016) try to estimate better word vectors by using a lexicon or ontology. The idea is simple: because a lexicon or ontology contains well-defined relations about words, we can use them to improve word vectors.
However, for a polysemous word, one of its synonym does not always mean the same thing with the original one under different contexts. For example, the word ”point” equals ”score” in ”Team A got 3 points”, but does not in ”my point of view.” A method to address this issue is to estimate a vector for each word sense (Huang et al., 2012; Chen et al., 2014) or per word type (Neelakantan et al., 2014). However, it requires additional word sense disambiguation or part-of-speech tagging to use such word vectors.
In this paper, we propose a method to improve the vector-space representations using a lexicon and alleviate the adverse effect of polysemy, keeping one vector per word. We estimate the degree of reliability for each paraphrase in the lexicon and eliminate the ones with lower degrees in learning. The experimental results show that the proposed method is effective and outperforms the prior works. The major contributions of our work include:
• We propose a novel approach involving fuzzy sets to reduce the noise brought by polysemous words in the word vector space when a lexicon is used for learning, and a model to use the fuzzy paraphrase sets to learn the word vector space.
• Although some prior works propose to solve the polysemy problem by estimating one vector per word sense or type, using such word vectors requires additional pre-process. Our proposed method keeps one vector per word. It makes the word vectors easier to use in practical terms: it is neither necessary to disambiguate the word senses nor to tag the part-of-speeches before we use the word vectors.
We give an introduction of our proposed method in section 2. We show the effects of different paraphrase sets, parameters, corpus size, and evaluate the effectiveness of our approach by comparing to simpler algorithms in section 3. We compare our approach with the prior works via an evaluation experiment in section 4. We give the findings, conclusions and outlook in section 5.
2 THE PROPOSED METHOD
2.1 FUZZY PARAPHRASES
As described in section 1, whether a polysemous word’s paraphrase is the same as the original depends on the context.
Henceforth, if we simply use all the paraphrases of a word in the lexicon to improve the word vector without discrimination, they may sometimes bring noise to the vector-space.
A conventional method for them is to give each word sense a vector. However, such vector-spaces require additional word sense disambiguation in practical use.
Here, we propose a method to alleviate the adverse effects of polysemous words’ paraphrases without word sense disambiguation. Our idea is to annotate each paraphrase with a degree about its reliability, like a member of a fuzzy set. We call such paraphrases as “fuzzy paraphrases”, and their degrees as the “memberships.”
2.2 LEARNING WITH FUZZY PARAPHRASES
We also propose a novel method to jointly learn corpus with a lexicon, in order to use fuzzy paraphrases to improve the word vectors.
If the meanings of two words are totally the same, they can replace each other in a text without changing the semantic features. Henceforth, we can learn the lexicon by replacing the words in the corpus with its lexical paraphrases.
We learn the word vectors by maximizing the probability of a word for a given context, and also for a generated context where words are replaced by their paraphrases randomly. The memberships of the fuzzy paraphrases are used here to control the probability that the replacements occur by a control function as shown in Figure 1.
For a text corpus T , denote wi the ith word in T , c the context window, wj a word in the context window, Lwj the paraphrase set of wj in the lexicon L, wk the kth fuzzy paraphrase in Lwj , and xjk the membership of wk for wj , the objective is
T∑ wi∈T ∑ (i−c)≤j≤(i+c) log p(wi|wj) + Lwj∑ wk∈Lwj f(xjk) log p(wi|wk)  . (1) The function f(xjk) of the membership xjk is a specified drop-out function. It returns 0 more for the paraphrases that have lower memberships, and 1 more for the others.
2.3 MEMBERSHIP ESTIMATION & CONTROL FUNCTION f(x)
Looking for a control function that is easy to train, we notice that if two words are more often to be translated to the same word in another language, the replacement of them are less likely to change the meaning of the original sentence. Thus, we use a function of the bilingual similarity (denoted as Sjk) as the membership function:
xjk = g(Sjk). (2)
There have been works about calculating the similarity of words using such bilingual information. A lexicon called the paraphrase database (PPDB) provides scores of the similarity of paraphrases on the basis of bilingual features (Ganitkevitch et al., 2013; Pavlick et al., 2015b;a).
We scale the similarity score of the paraphrase wk to [0, 1] in PPDB2.0 as the memberships, and draw the values of f(xjk) from a Bernoulli distribution subjected to them. Denote Sjk the similarity score of word wj and wk in PPDB2.0, the value of f(xjk) is drawn from the Bernoulli distribution:
f(xjk) ∼ Bernoulli(xjk), (3)
xjk = Sjk
max j∈T,k∈L
Sjk . (4)
2.4 TRAINING
We do not need to train f(xjk) using the method described above. The model can be trained by negative sampling (Mikolov et al., 2013b): For word wO and a word wI in its context, denote AI as the set of the paraphrases for wI accepted by f(xjk), we maximize log p(wO|wI) by distinguishing the noise words from a noise distribution Pn(w) from wO and its accepted paraphrases in AI by logistic regression:
log p(wO|wI) = log σ(vwOTvwI ) + n∑
i=1
Ewi ∼ Pn(w)[log σ(−vwiTvwI )], wi 6= wO, wi /∈ AI
(5)
Here, vwO T and vwi T stand for the transposed matrices of vwO and vwi , respectively. n is the number of negative samples used. σ(x) is a sigmoid function, σ(x) = 1/(1 + e−x).
3 MODEL EXPLORATION
3.1 CORPUS FOR EXPERIMENTS
We use enwiki91 mainly for tuning and model exploration. It has a balanced size(1 GB), containing 123,353,508 tokens. It provides enough data to alleviate randomness while it does not take too much time for our model to learn.
1http://mattmahoney.net/dc/enwiki9.zip
We use ukWaC (Baroni et al., 2009) to compare with the prior works in section 4. But we do not use it for model exploration, because it takes more than 20 hours to learn it, as an enormous corpus containing 12 GB text.
3.2 BENCHMARKS
We used several benchmarks. They are Wordsim-353 (WS353) (Finkelstein et al., 2001) (353 word pairs), SimLex-999 (SimLex) (Hill et al., 2016) (999 word pairs), the Stanford Rare Word Similarity Dataset (RW) (Luong et al., 2013) (2034 word pairs), the MEN Dataset (MEN) (Bruni et al., 2014) (3000 word pairs), and the Mikolov’s (Google’s) word analogical reasoning task (Mikolov et al., 2013a).
WS353, SimLex, and RW are gold standards. They provide the similarity of words labeled by humans. We report the Spearman’s rank correlation (ρ) for them.
Mikolov’s word analogical reasoning task is another widely used benchmark for word vectors. It contains a semantic part (SEM), and a syntactic part (SYN). We use the basic way suggested in their paper to find the answer for it: to guess word b′ related to b in the way how a′ is related to a, the word closest in cosine similarity to a′ − a+ b is returned as b′. We find that the benchmark scores change every time we learn the corpus, even under the same settings. It is because that the models involve random numbers. Therefore we should consider the margin of error of the changes when we use the benchmarks.
To test the margin of error, we firstly used our proposed method to repeat learning enwiki9 for 10 times under the same parameters. Then we tested the vectors under each benchmark, to find the margin of error. In each test, we used the same parameters: the vector dimension was set to 100 for speed, the window size was set to 8, and 25 negative samples were used. The results are shown in Table 1. We use them to analyze the other experimental results later.
3.3 DIFFERENT TYPES OF PARAPHRASES
In PPDB2.0, there are six relationships for paraphrases. For word X and Y , the different relationships between them defined in PPDB2.0 are shown in Table 2. We do not consider the exclusion and independent relations because they are not semantic paraphrases. Those of equivalence are the most reliable because they are the closest ones. But we still want to know whether it is better to take
the entailment and the other related paraphrases into consideration. We learn enwiki9 with different paraphrase sets and use SimLex to evaluate the trained vectors.
Figure 2 compares the performance using different paraphrase sets, tested by SimLex. We can see that it is best to use the equivalence and entailment (forward + reverse) paraphrases together or use only the equivalence paraphrases. Only using the entailment paraphrases is weak. Involving the other related paraphrases deteriorates the performance. We use the Equivalence and Entailment paraphrases in the experiments according to these results.
3.4 EFFECTS OF PARAMETERS
We use our proposed method to learn enwiki9 under different parameter settings to evaluate the effects of parameters. We firstly learn enwiki9 under different parameter settings and then test the vectors using SimLex, WS353, RW, MEN, SEM and SYN. We report Spearman’s rank correlation ρ for SimLex, WS353, RW and MEN, the percentage of correct answers for SEM and SYN.
3.4.1 EFFECTS OF VECTOR SPACE DIMENSION
We compare the benchmarks using different vector-space dimensions. Figure 3 shows the change of each benchmark’s scores under different dimensions.
We find that:
• The larger vectors do not bring the better performance for most of the benchmarks (except SimLex), although some previous works suggest that the higher dimensions brings better performance for their methods (Pennington et al., 2014; Levy & Goldberg, 2014b).
• The curves of SimLex and SYN are gradual. However, there are several abrupt changes in the others. And those of WS353 and RW do not change gradually.
• The best dimension for different benchmarks is not consistent.
The differences in the content of the benchmarks may cause the inconsistence. For example, SimLex rates related but dissimilar words lower than the other word similarity benchmarks (Hill et al., 2016; Chiu et al., 2016). The results suggest that the best dimensions for our method depends on the task.
3.4.2 EFFECTS OF CONTEXT WINDOW SIZE
We compared the benchmarks using different context window sizes. They are shown in Figure 4. Previous works argue that larger window sizes introduce more topic words, and smaller ones emphasize word functions (Turney, 2012; Levy & Goldberg, 2014a; Levy et al., 2015; Hill et al., 2016; Chiu et al., 2016). Different context window sizes provide different balances between relatedness and similarity. The best window size depends on what we want the vectors to be. We also see that in our results.
The relationship between the window size and performance depends on how they rate the pairs. For example, WS353 rates word pairs according to association rather than similarity (Finkelstein et al., 2001; Hill et al., 2016). As larger window capture relatedness rather than similarity, the results show
that the larger the window is, the better for WS353. The MEN dataset also prefer relatedness than similarity (Bruni et al., 2014), but they gave annotators examples involving similarity2. It may be the reason that the windows larger than 8 deteriorate the benchmarks based on MEN (Figure 4d). The standards of WS353 and MEN to rate the words are similar (Bruni et al., 2014). It leads to their similar curves (Figure 4b and 4d). The worst window sizes of them are also close. When the window size is set to about 2 or 3, respectively, the balance of similarity and relatedness is the worst for them.
Unlike the other word similarity dataset, SimLex rates synonyms high and related dissimilar word pairs low. Therefore, the smallest window is the most suitable for SimLex because it is best for capturing the functional similarity.
The results of RW differs from the others (Figure 4c). There are many abrupt changes. The best window size is 10, but 1 is better than 2-9. The dataset contains rare words. Because of their low frequencies, usage of broad context window may be better to draw features for them. However, additional words introduced by larger windows may also deteriorate the vectors of unusual words. For such tasks requiring rare word vectors of high quality, we should be careful in tuning the context window size.
For Google’s word analogical tasks (SEM and SYN), the questions are quite related to the topic or domain. For examples, there are questions about the capitals of the countries. They are associated but not synonymous. Therefore a larger window is usually better. However for SYN, using window size 9 is a little better than 10 in Figure 4d and for MEN 8 is best in Figure 4f. It may be because that if the window is too large, it introduces too many words and reduces the sparsity (Chiu et al., 2016).
We can consider that the best context window size depends on the task, but we should avoid using too large window.
3.4.3 EFFECTS OF NEGATIVE SAMPLES
We also explored the effects of the number of negative samples. The results are shown in Figure 5. 2According to their homepage: http://clic.cimec.unitn.it/ elia.bruni/MEN.html.
In Figures 5a, 5c and 5f, we see that overfitting occurs when we use more than 15 negative samples. In Figure 5b and Figure 5e, it occurs from 25 and 20, respectively. In Figure 5d, the performance does not change very much when we use more than 30 negative samples.
The results indicate that too many negative samples may cause overfitting. For 3 of the 6 benchmarks, it is best to use 15 negative samples. But we should be careful in practice use because the other different results suggest that the best number depends on the task.
The abrupt change at around 15 in Figure 5b is interesting. WS353 is the smallest dataset among those we used. Because of the small size, the effects of randomness may cause such singularities when the vector-space is not well trained.
3.5 EFFECTS OF THE CONTROL FUNCTION & THE CORPUS SIZE
In this section, we evaluate the effectiveness of our fuzzy approach, by comparing to the situations that set f(x) in Equation (1) as:
• f(x) = 1: It makes the model regard all paraphrases equally. They are all used without drop-out.
• f(x) = 0: It makes the model use no paraphrases, equivalent to CBOW.
It is also a good way to show the effects of corpus size by comparing the proposed method to the situations above using corpora in varying size. Therefore we discuss them together in this section.
We use text83 together with eEnwiki9 and ukWaC described in section 3.1. It is a small corpus containing 100 MB text. To show the difference, we report the benchmarks scores including not only SimLex, but also MEN, and the word analogical task (SEM and SYN). They are the other benchmarks that are shown relatively solid in section 3.2. The vector-space dimension is set to 300. The context window size is set to 8. 25 negative samples are used in learning. The results are shown in Figure 6.
3http://mattmahoney.net/dc/text8.zip
We can see that:
• The proposed function outperforms the others for SimLex and MEN under text8, for all the benchmarks under enwiki9, for SimLex, SEM and SYN under ukWaC.
• The proposed function is always better than f(x) = 1 in the experiments, no matter what the benchmark is or how big the corpus is.
• For SEM, the proposed function is weaker than f(x) = 0 under text8, slightly better under enwiki9, and obviously outperforms f(x) = 0 under ukWaC. As the proposed function outperforms under larger corpora, the relatively low scores under text8 may be caused by the effects of randomness: the proposed function involves random numbers; they bring huge instability under such tiny corpora. Another possible reason is that the control function is less useful for text8 because there are few polysemous words in the tiny corpus.
• There is no advantages to use f(x) = 1 instead of f(x) = 0 for both text8 and enwiki9. It shows that learning the context words replaced by paraphrases may be not a good idea without fuzzy approaches. However, if we use the proposed control function, the results are better and go beyond those of f(x) = 0 in most tests. It shows that the control function utilizing fuzzy paraphrases improves the performance.
Therefore, we can see that the proposed control function using the fuzzy paraphrases annotated with the degrees of reliability improves the quality of the learned word vector-space.
4 COMPARISON WITH THE PRIOR WORKS
We compared our work to the prior works using a lexicon to improve word vectors. However, we failed to use the public code to reproduce the works of Yu & Dredze (2014) and Bollegala et al. (2016). We also failed to find an available implementation of Xu et al. (2014). Hence, we use the
same corpus and benchmarks with Bollegala et al. (2016) and compare our results with the reported scores of the prior works in their paper. The benchmarks are:
• The MEN Dataset (MEN); • Word Analogical Reasoning Task (SEM and SYN).
Rubenstein-Goodenough dataset (RG) (Rubenstein & Goodenough, 1965) is also used in their works. However, we do not use it, because it fails the sanity check in Batchkarov et al. (2016): ρ may increase when noise is added.
We use ukWaC to learn the word vectors, the same with Bollegala et al. (2016). We also use the same parameters with the prior works: The vector-space dimension is set to 300; the context window size is set to 8; the number of negative samples is set to 25. Then we calculate the cosine similarity of the words and report 100 ∗ ρ for Men. We use the add method described in section 3.2 and report the percentage of correct answers, for the word analogical reasoning task.
Table 3 shows the results of the experiments. The of MEN and SEM is 0.86 and 0.44 as shown in Table 1. Therefore we see that our proposed method outperforms the prior works under these benchmarks. We consider our score for SYN is as good as Bollegala et al. (2016) achieved, and better than the others, because its margin of error is 1.79 as shown in Table 1.
5 CONCLUSION & THE FUTURE WORKS
We proposed a fuzzy approach to control the contamination caused by the polysemous words when a lexicon is used to improve the vector-space word representations. We annotate each paraphrase of a word with a degree of reliability, like the members of a fuzzy set with their memberships, on the basis of their multilingual similarities to the original ones. We use the fuzzy paraphrases to learn a corpus by jointly learning a generated text, in which the original words are randomly replaced by their paraphrases. A paraphrase is less likely to be put into the generated text if it has lower reliability than the others, and vice versa.
We tested the performance using different types of paraphrases in the lexicon PPDB2.0 and find that it is best to use the equivalence type and the entailment type. Using other related paraphrases deteriorates the performance.
We explored the effects of parameters. We find that the best parameter setting depends on the task. We should tune the model carefully in practical use.
We evaluated the effectiveness of our approach by comparing it to the situations that simpler functions are used to control replacements: f(x) = 1 which accepts all, and f(x) = 0 which rejects
all. We also repeated the experiments under a tiny, a medium sized, and a large corpus, to see the effects of the corpus size on the effectiveness. Our approach achieves the best in 3 of 4 benchmarks under the tiny corpus, and in all benchmarks under the medium sized and the large one. The results indicate that our approach is effective to improve the word vectors.
Our proposed method also achieved the top scores, compared with the prior works.
Unlike the previous works that solve the problems about polysemy by estimating a vector for each word sense or word type, our approach keeps one vector per word. It makes the word vectors easier to use in practical terms: it is neither necessary to disambiguate the word senses nor to tag the part-of-speeches before we use the word vectors.
The fuzzy paraphrases can also be employed for the other models with some changes. We are going to show it in the future. The proposed idea for the polysemy problem without word sense disambiguation is meaningful especially for practical use because it saves the effort of part-of-speech tagging and word sense disambiguation.
Besides, the control function may be more accurate if it considers all the context. We are also going to work on it in the future.
We have opened the source of a demo of the proposed method online4.

This paper introduces the concept of fuzzy paraphrases to aid in the learning of distributed word representations from a corpus augmented by a lexicon or ontology. Sometimes polysemy is context-dependent, but prior approaches have neglected this fact when incorporating external paraphrase information during learning. The main idea is to introduce a function that essentially judges the context-sensitivity of paraphrase candidates, down-weighting those candidates that depend strongly on context. This function is inferred from bilingual translation agreement.
The main argumentation leading to the model selection is intuitive, and I believe that the inclusion of good paraphrases and the elimination of bad paraphrases during training should in principle improve word representation quality. However, the main questions are how well the proposed method achieves this goal, and, even if it achieves it well, whether it makes much difference in practical terms.
Regarding the first question, I am not entirely convinced that the parameterization of the control function f(x_ij) is optimal. It would have been nice to see some experiments investigating different choices, in particular some baselines where the effect of f is diminished (so that it reduces to f=1 in the limit) would have been interesting. I also feel like there would be a lot to gain from having f be a function of the nearby word embeddings, though this would obvious incur a significant slowdown. (See for example 'Efficient Non-parametric Estimation of Multiple Embeddings per Word in Vector Space' by Neelakantan et al, which should probably be cited.) As it stands, the experimental results do not clearly distinguish the fuzzy paraphrase approach from prior work, i.e. tables 3 and 4 do not show major trends one way or the other.
Regarding the second question, it is hard to draw many conclusions from analogy tasks alone, especially when effects unrelated to good/bad paraphrasing such as corpus size/content, window size, vocabulary size, etc., can have an outsize effect on performance.
Overall, I think this is a good paper presenting a sensible idea, but I am not convinced by the experiments that the specific approach is achieving its goal. With some improved experiments and analysis, I would wholeheartedly recommend this paper for acceptance; as it stands, I am on the fence.