VISUALIZING DEEP NEURAL NETWORK DECISIONS: PREDICTION DIFFERENCE ANALYSIS
This article presents the prediction difference analysis method for visualizing the response of a deep neural network to a specific input. When classifying images, the method highlights areas in a given input image that provide evidence for or against a certain class. It overcomes several shortcoming of previous methods and provides great additional insight into the decision making process of classifiers. Making neural network decisions interpretable through visualization is important both to improve models and to accelerate the adoption of black-box classifiers in application areas such as medicine. We illustrate the method in experiments on natural images (ImageNet data), as well as medical images (MRI brain scans).
1 INTRODUCTION
Over the last few years, deep neural networks (DNNs) have emerged as the method of choice for perceptual tasks such as speech recognition and image classification. In essence, a DNN is a highly complex non-linear function, which makes it hard to understand how a particular classification comes about. This lack of transparency is a significant impediment to the adoption of deep learning in areas of industry, government and healthcare where the cost of errors is high.
In order to realize the societal promise of deep learning - e.g., through self-driving cars or personalized medicine - it is imperative that classifiers learn to explain their decisions, whether it is in the lab, the clinic, or the courtroom. In scientific applications, a better understanding of the complex dependencies learned by deep networks could lead to new insights and theories in poorly understood domains.
In this paper, we present a new, probabilistically sound methodology for explaining classification decisions made by deep neural networks. The method can be used to produce a saliency map for each (instance, node) pair that highlights the parts (features) of the input that constitute most evidence for or against the activation of the given (internal or output) node. See figure 1 for an example.
In the following two sections, we review related work and then present our approach. In section 4 we provide several demonstrations of our technique for deep convolutional neural networks (DCNNs) trained on ImageNet data, and further how the method can be applied when classifying MRI brain scans of HIV patients with neurodegenerative disease.
2 RELATED WORK
Broadly speaking, there are two approaches for understanding DCNNs through visualization investigated in the literature: find an input image that maximally activates a given unit or class score to visualize what the network is looking for (Erhan et al., 2009; Simonyan et al., 2013; Yosinski et al., 2015), or visualize how the network responds to a specific input image in order to explain a particular classification made by the network. The latter will be the subject of this paper.
One such instance-specific method is class saliency visualization proposed by Simonyan et al. (2013) who measure how sensitive the classification score is to small changes in pixel values, by computing the partial derivative of the class score with respect to the input features using standard backpropagation. They also show that there is a close connection to using deconvolutional networks for visualization, proposed by Zeiler & Fergus (2014). Other methods include Shrikumar et al. (2016), who compare the activation of a unit when a specific input is fed forward through the net to a reference activation for that unit. Zhou et al. (2016) and Bach et al. (2015) also generate interesting visualization results for individual inputs, but are both not as closely related to our method as the two papers mentioned above. The idea of our method is similar to another analysis Zeiler & Fergus (2014) make: they estimate the importance of input pixels by visualizing the probability of the (correct) class as a function of a gray patch occluding parts of the image. In this paper, we take a more rigorous approach at both removing information from the image and evaluating the effect of this.
In the field of medical image classification specifically, a widely used method for visualizing feature importances is to simply plot the weights of a linear classifier (Klöppel et al., 2008; Ecker et al., 2010), or the p-values of these weights (determined by permutation testing) (Mourao-Miranda et al., 2005; Wang et al., 2007). These are independent of the input image, and, as argued by Gaonkar & Davatzikos (2013) and Haufe et al. (2014), interpreting these weights can be misleading in general.
The work presented in this paper is based on an instance-specific method by Robnik-Šikonja & Kononenko (2008), the prediction difference analysis, which is reviewed in the next section. Our main contributions are three substantial improvements of this method: conditional sampling (section 3.1), multivariate analysis (section 3.2), and deep visualization (section 3.3).
3 APPROACH
Our method is based on the technique presented by Robnik-Šikonja & Kononenko (2008), which we will now review. For a given prediction, the method assigns a relevance value to each input feature with respect to a class c. The basic idea is that the relevance of a feature xi can be estimated by measuring how the prediction changes if the feature is unknown, i.e., the difference between p(c|x) and p(c|x\i), where x\i denotes the set of all input features except xi.
To find p(c|x\i), i.e., evaluate the prediction when a feature is unknown, the authors propose three strategies. The first is to label the feature as unknown (which only few classifiers allow). The second is to re-train the classifier with the feature left out (which is clearly infeasible for DNNs and high-dimensional data like images). The third approach is to simulate the absence of a feature by marginalizing the feature:
p(c|x\i) = ∑ xi p(xi|x\i)p(c|x\i, xi) (1)
(with the sum running over all possible values for xi). However, modeling p(xi|x\i) can easily become infeasible with a large number of features. Therefore, the authors approximate equation (1) by assuming that feature xi is independent of the other features, x\i:
p(c|x\i) ≈ ∑ xi p(xi)p(c|x\i, xi) . (2)
The prior probability p(xi) is usually approximated by the empirical distribution for that feature.
Once the class probability p(c|x\i) is estimated, it can be compared to p(c|x). We stick to an evaluation proposed by the authors referred to as weight of evidence, given by
WEi(c|x) = log2 (odds(c|x))− log2 ( odds(c|x\i) ) , (3)
Algorithm 1 Evaluating the prediction difference using conditional and multivariate sampling Input: classifier with outputs p(c|x), input image x of size n× n, inner patch size k, outer patch size l > k, class of interest c, probabilistic model over patches of size l × l, number of samples S Initialization: WE = zeros(n*n), counts = zeros(n*n) for every patch xw of size k × k in x do
x′ = copy(x) sumw = 0 define patch x̂w of size l × l that contains xw for s = 1 to S do
x′w ← xw sampled from p(xw|x̂w\xw) sumw += p(c|x′) . evaluate classifier
end for p(c|x\xw) := sumw/S WE[coordinates of xw] += log2(odds(c|x))− log2(odds(c|x\xw)) counts[coordinates of xw] += 1
end for Output: WE / counts . point-wise division
where odds(c|x) = p(c|x)/(1 − p(c|x)). To avoid problems with zero probabilities, Laplace correction p← (pN + 1)/(N +K) is used, where N is the number of training instances and K the number of classes.
The method produces a relevance vector (WEi)i=1...m (m being the number of features) of the same size as the input, which reflects the relative importance of all features. A large prediction difference means that the feature contributed substantially to the classification, whereas a small difference indicates that the feature was not important for the decision. A positive value WEi means that the feature has contributed evidence for the class of interest: removing it would decrease the confidence of the classifier in the given class. A negative value on the other hand indicates that the feature displays evidence against the class: removing it also removes potentially conflicting or irritating information and the classifier becomes more certain in the investigated class.
3.1 CONDITIONAL SAMPLING
In equation (3), the conditional probability p(xi|x\i) of a feature xi is approximated using the marginal distribution p(xi). This is a very crude approximation. In images for example, a pixel’s value is highly dependent on other pixels. We propose a much more accurate approximation, based on the following two observations: a pixel depends most strongly on a small neighborhood around it, and the conditional of a pixel given its neighborhood does not depend on the position of the pixel in the image. For a pixel xi, we can therefore find a patch x̂i of size l× l that contains xi, and condition on the remaining pixels in that patch:
p(xi|x\i) ≈ p(xi|x̂\i) . (4)
This greatly improves the approximation while remaining completely tractable.
For a feature to become relevant when using conditional sampling, it now has to satisfy two conditions: being relevant to predict the class of interest, and be hard to predict from the neighboring pixels. Relative to the marginal method, we therefore downweight the pixels that can easily be predicted and are thus redundant in this sense.
3.2 MULTIVARIATE ANALYSIS
Robnik-Šikonja & Kononenko (2008) take a univariate approach: only one feature at a time is removed. However, we would expect that a neural network is relatively robust to just one feature of a high-dimensional input being unknown, like a pixel in an image. Therefore, we will remove several features at once by again making use of our knowledge about images by strategically choosing these feature sets: patches of connected pixels. Instead of going through all individual pixels, we go through all patches of size k × k in the image (k × k × 3 for RGB images and k × k × k for 3D images like MRI scans), implemented in a sliding window fashion. The patches are overlapping, so that ultimately an individual pixel’s relevance is obtained by taking the average relevance obtained from the different patches it was in.
Algorithm 1 and figure 2 illustrate how the method can be implemented, incorporating the proposed improvements.
3.3 DEEP VISUALIZATION OF HIDDEN LAYERS
When trying to understand neural networks and how they make decisions, it is not only interesting to analyze the input-output relation of the classifier, but also to look at what is going on inside the hidden layers of the network. We can adapt the method to see how the units of any layer of the network influence a node from a deeper layer. Mathematically, we can formulate this as follows. Let h be the vector representation of the values in a layer H in the network (after forward-propagating the input up to this layer). Further, let z = z(h) be the value of a node that depends on h, i.e., a node in a subsequent layer. Then the analog of equation (2) is given by the expectation:
g(z|h\i) ≡ Ep(hi|h\i) [z(h)] = ∑ hi p(hi|h\i)z(h\i, hi) , (5)
which expresses the distribution of z when unit hi in layer H is unobserved. The equation now works for arbitrary layer/unit combinations, and evaluates to the same as equation (1) when the input-output relation is analyzed. To evaluate the difference between g(z|h) and g(z|h\i), we will in general use the activation difference, ADi(z|h) = g(z|h)− g(z|h\i) , for the case when we are not dealing with probabilities (and equation (3) is not applicable).
4 EXPERIMENTS
In this section, we illustrate how the proposed visualization method can be applied, on the ImageNet dataset of natural images when using DCNNs (section 4.1), and on a medical imaging dataset of MRI scans when using a logistic regression classifier (section 4.2). For marginal sampling we always use the empirical distribution, i.e., we replace a feature (patch) with samples taken directly from other images, at the same location. For conditional sampling we use a multivariate normal distribution. For both sampling methods we use 10 samples to estimate p(c|x\i) (since no significant difference was observed with more samples). Note that all images are best viewed digital and in color.
Our implementation is available at github.com/lmzintgraf/DeepVis-PredDiff.
4.1 IMAGENET: UNDERSTANDING HOW A DCNN MAKES DECISIONS
We use images from the ILSVRC challenge (Russakovsky et al., 2015) (a large dataset of natural images from 1000 categories) and three DCNNs: the AlexNet (Krizhevsky et al., 2012), the GoogLeNet (Szegedy et al., 2015) and the (16-layer) VGG network (Simonyan & Zisserman, 2014). We used the publicly available pre-trained models that were implemented using the deep learning framework caffe (Jia et al., 2014). Analyzing one image took us on average 20, 30 and 70 minutes for the respective classifiers AlexNet, GoogLeNet and VGG (using the GPU implementation of caffe and mini-batches with the standard settings of 10 samples and a window size of k = 10).
The results shown here are chosen from among a small set of images in order to show a range of behavior of the algorithm. The shown images are quite representative of the performance of the method in general. Examples on randomly selected images, including a comparison to the sensitivity analysis of Simonyan et al. (2013), can be seen in appendix A.
We start this section by demonstrating our proposed improvements (sections 3.1 - 3.3).
Marginal vs Conditional Sampling
Figure 3 shows visualizations of the spatial support for the highest scoring class, using marginal and conditional sampling (with k = 10 and l = 14). We can see that conditional sampling leads to results that are more refined in the sense that they concentrate more around the object. We can also see that marginal sampling leads to pixels being declared as important that are very easily predictable conditioned on their neighboring pixels (like in the saxophone example). Throughout our experiments, we have found that conditional sampling tends to give more specific and fine-grained results than marginal sampling. For the rest of our experiments, we therefore show results using conditional sampling only.
Multivariate Analysis
For ImageNet data, we have observed that setting k = 10 gives a good trade-off between sharp results and a smooth appearance. Figure 4 shows how different window sizes influence the resolution of the visualization. Surprisingly, removing only one pixel does have a measurable effect on the prediction, and the largest effect comes from sensitive pixels. We expected that removing only one pixel does not have any effect on the classification outcome, but apparently the classifier is sensitive even to these small changes. However when using such a small window size, it is difficult to make sense of the sign information in the visualization. If we want to get a good impression of which parts in the image are evidence for/against a class, it is therefore better to use larger windows. If k is chosen too large however, the results tend to get blurry. Note that these results are not just simple averages of one another, but a multivariate approach is indeed necessary to observe the presented results.
Deep Visualization of Hidden Network Layers
Our third main contribution is the extension of the method to neural networks; to understand the role of hidden layers in a DNN. Figure 5 shows how different feature maps in three different layers of the GoogLeNet react to the input of a tabby cat (see figure 6, middle image). For each feature map in a convolutional layer, we first compute the relevance of the input image for each hidden unit in that map. To estimate what the feature map as a whole is doing, we show the average of the relevance vectors over all units in that feature map. The first convolutional layer works with different types of simple image filters (e.g., edge detectors), and what we see is which parts of the input image respond
positively or negatively to these filters. The layer we picked from somewhere in the middle of the network is specialized to higher level features (like facial features of the cat). The activations of the last convolutional layer are very sparse across feature channels, indicating that these units are highly specialized.
To get a sense of what single feature maps in convolutional layers are doing, we can look at their visualization for different input images and look for patterns in their behavior. Figure 6 shows this for four different feature maps from a layer from the middle of the GoogLeNet network. We can directly see which kind of features the model has learned at this stage in the network. For example, one feature map is mostly activated by the eyes of animals (third row), and another is looking mostly at the background (last row).
Penultimate vs Output Layer
If we visualize the influence of the input features on the penultimate (pre-softmax) layer, we show only the evidence for/against this particular class, without taking other classes into consideration. After the softmax operation however, the values of the nodes are all interdependent: a drop in the probability for one class could be due to less evidence for it, or because a different class becomes more likely. Figure 7 compares visualizations for the last two layers. By looking at the top three scoring classes, we can see that the visualizations in the penultimate layer look very similar if the classes are similar (like different dog breeds). When looking at the output layer however, they look rather different. Consider the case of the elephants: the top three classes are different elephant subspecies, and the visualizations of the penultimate layer look similar since every subspecies can be identified by similar characteristics. But in the output layer, we can see how the classifier decides for one of the three types of elephants and against the others: the ears in this case are the crucial difference.
Network Comparison
When analyzing how neural networks make decisions, we can also compare how different network architectures influence the visualization. Here, we tested our method on the AlexNet, the GoogLeNet and the VGG network. Figure 8 shows the results for the three different networks, on two input images. The AlexNet seems to more on contextual information (the sky in the balloon image), which could be attributed to it having the least complex architecture compared to the other two networks. It is also interesting to see that the VGG network deems the basket of the balloon as very important compared to all other pixels. The second highest scoring class in this case was a parachute - presumably, the network learned to not confuse a balloon with a parachute by detecting a square basket (and not a human).
4.2 MRI DATA: EXPLAINING CLASSIFIER DECISIONS IN MEDICAL IMAGING
To illustrate how our visualization method can also be useful in a medical domain, we show some experimental results on an MRI dataset of HIV and healthy patients. In such settings, it is crucial that the practitioner has some insight into the algorithm’s decision when classifying a patient, to weigh this information and incorporate it in the overall diagnosis process.
The dataset used here is referred to as the COBRA dataset. It contains 3D MRIs from 100 HIV patients and 70 healthy individuals, included in the Academic Medical Center (AMC) in Amsterdam, The Netherlands. Of these subjects, diffusion weighted MRI data were acquired. Preprocessing of the data was performed with software developed in-house, using the HPCN-UvA Neuroscience Gateway and using resources of the Dutch e-Science Grid Shahand et al. (2015). As a result, Fractional Anisotropy (FA) maps were computed. FA is sensitive to microstructural damage and therefore expected to be, on average, decreased in patients. Subjects were scanned on two 3.0 Tesla scanner systems, 121 subjects on a Philips Intera system and 39 on a Philips Ingenia system. Patients and controls were evenly distributed. FA images were spatially normalized to standard space Andersson et al. (2007), resulting in volumes with 91× 109× 91 = 902, 629 voxels.
We trained an L2-regularized Logistic Regression classifier on a subset of the MRI slices (slices 29-40 along the first axis) and on a balanced version of the dataset (by taking the first 70 samples of the HIV class) to achieve an accuracy of 69.3% in a 10-fold cross-validation test. Analyzing one image took around half an hour (on a CPU, with k = 3 and l = 7, see algorithm 1). For conditional sampling, we also tried adding location information in equation (2), i.e., we split up the 3D image into a 20 × 20 × 20 grid and also condition on the index in that grid. We found that this slightly improved the interpretability of the results, since the pixel values in the special case of MRI scans does depend on spacial location as well.
Figure 9 (first row) shows one way via which the prediction difference results could be presented to a physician, for an HIV sample. By overlapping the prediction difference and the MRI image, the exact regions can be pointed out that are evidence for (red parts) or against (blue parts) the classifier’s decision. The second row shows the results using the weights of the logistic regression classifier, which is a commonly used method in neuroscientific literature. We can see that they are considerably noisier (in the sense that, compared to our method, the voxels relevant for the classification decisions are more scattered), and also, they are not specific to the given image. Figure 10 shows the visualization results for four healthy, and four HIV samples. We can clearly see that the patterns for the two classes are distinct, and there is some pattern to the decision of the classifier, but which is still specific to the input image. Figure 11 shows the same (HIV) sample as in figure 9 along different axes, and figure 12 shows how the visualization changes with different patch sizes. We believe that both varying the slice and patch size can give different insights to a clinician, and in clinical practice, a 3D animation where these parameters can be adjusted would be very useful for analyzing the visualization result.
In general we can assume that the better the classifier, the closer the explanations for its decisions are to the true class difference. For clinical practice it is therefore crucial to have very good classifiers. This will increase computation time, but in many medical settings, longer waiting times for test results are common and worth the wait if the patient is not in an acute life threatening condition (e.g., when predicting HIV or Alzheimer from MRI scans, or the field of cancer diagnosis and detection). The presented results here are for demonstration purposes of the visualization method, and we claim no medical validity. A thorough qualitative analysis incorporating expert knowledge was outside the scope of this paper.
5 FUTURE WORK
In our experiments, we used a simple multivariate normal distribution for conditional sampling. We can imagine that using more sophisticated generative models will lead to better results: pixels that are easily predictable by their surrounding are downweighted even more. However this will also significantly increase the computational resources needed to produce the explanations. Similarly, we could try to modify equation (4) to get an even better approximation by using a conditional distribution that takes more information about the whole image into account (like adding spatial information for the MRI scans).
To make the method applicable for clinical analysis and practice, a better classification algorithm is required. Also, software that visualizes the results as an interactive 3D model will improve the usability of the system.
6 CONCLUSION
We presented a new method for visualizing deep neural networks that improves on previous methods by using a more powerful conditional, multivariate model. The visualization method shows which pixels of a specific input image are evidence for or against a node in the network. The signed information offers new insights - for research on the networks, as well as the acceptance and usability in domains like healthcare. While our method requires significant computational resources, real-time 3D visualization is possible when visualizations are pre-computed. With further optimization and powerful GPUs, pre-computation time can be reduced a lot further. In our experiments, we have presented several ways in which the visualization method can be put into use for analyzing how DCNNs make decisions.
ACKNOWLEDGMENTS
This work was supported by AWS in Education Grant award. We thank Facebook and Google for financial support, and our reviewers for their time and valuable, constructive feedback.
This work was also in part supported by: Innoviris, the Brussels Institute for Research and Innovation, Brussels, Belgium; the Nuts-OHRA Foundation (grant no. 1003-026), Amsterdam, The Netherlands; The Netherlands Organization for Health Research and Development (ZonMW) together with AIDS Fonds (grant no 300020007 and 2009063). Additional unrestricted scientific grants were received from Gilead Sciences, ViiV Healthcare, Janssen Pharmaceutica N.V., Bristol-Myers Squibb, Boehringer Ingelheim, and Merck&Co.
We thank Barbara Elsenga, Jane Berkel, Sandra Moll, Maja Totté, and Marjolein Martens for running the AGEhIV study program and capturing our data with such care and passion. We thank Yolanda Ruijs-Tiggelman, Lia Veenenberg-Benschop, Sima Zaheri, and Mariska Hillebregt at the HIV Monitoring Foundation for their contributions to data management. We thank Aafien Henderiks and Hans-Erik Nobel for their advice on logistics and organization at the Academic Medical Center. We thank all HIV-physicians and HIV-nurses at the Academic Medical Center for their efforts to include the HIV-infected participants into the AGEhIV Cohort Study, and the Municipal Health Service Amsterdam personnel for their efforts to include the HIV-uninfected participants into the AGEhIV Cohort Study. We thank all study participants without whom this research would not be possible.
AGEhIV Cohort Study Group. Scientific oversight and coordination: P. Reiss (principal investigator), F.W.N.M. Wit, M. van der Valk, J. Schouten, K.W. Kooij, R.A. van Zoest, E. Verheij, B.C. Elsenga (Academic Medical Center (AMC), Department of Global Health and Amsterdam Institute for Global Health and Development (AIGHD)). M. Prins (co-principal investigator), M.F. Schim van der Loeff, M. Martens, S. Moll, J. Berkel, M. Totté, G.R. Visser, L. May, S. Kovalev, A. Newsum, M. Dijkstra (Public Health Service of Amsterdam, Department of Infectious Diseases). Datamanagement: S. Zaheri, M.M.J. Hillebregt, Y.M.C. Ruijs, D.P. Benschop, A. el Berkaoui (HIV Monitoring Foundation). Central laboratory support: N.A. Kootstra, A.M. Harskamp-Holwerda, I. Maurer, T. Booiman, M.M. Mangas Ruiz, A.F. Girigorie, B. Boeser-Nunnink (AMC, Laboratory for Viral Immune Pathogenesis and Department of Experimental Immunology). Project management and administrative support: W. Zikkenheiner, F.R. Janssen (AIGHD). Participating HIV physicians and nurses: S.E. Geerlings, M.H. Godfried, A. Goorhuis, J.W.R. Hovius, J.T.M. van der Meer, F.J.B. Nellen, T. van der Poll, J.M. Prins, P. Reiss, M. van der Valk, W.J. Wiersinga, M. van Vugt, G. de Bree, F.W.N.M. Wit; J. van Eden, A.M.H. van Hes, M. Mutschelknauss , H.E. Nobel, F.J.J. Pijnappel, M. Bijsterveld, A. Weijsenfeld, S. Smalhout (AMC, Division of Infectious Diseases). Other collaborators: J. de Jong, P.G. Postema (AMC, Department of Cardiology); P.H.L.T. Bisschop, M.J.M. Serlie (AMC, Division of Endocrinology and Metabolism); P. Lips (Free University Medical Center Amsterdam); E. Dekker (AMC, Department of Gastroenterology); N. van der Velde (AMC, Division of Geriatric Medicine); J.M.R. Willemsen, L. Vogt (AMC, Division of Nephrology); J. Schouten, P. Portegies, B.A. Schmand, G.J. Geurtsen (AMC, Department of Neurology); F.D. Verbraak, N. Demirkaya (AMC, Department of Ophthalmology); I. Visser (AMC, Department of Psychiatry); A. Schadé (Free University Medical Center Amsterdam, Department of Psychiatry); P.T. Nieuwkerk, N. Langebeek (AMC, Department of Medical Psychology); R.P. van Steenwijk, E. Dijkers (AMC, Department of Pulmonary medicine); C.B.L.M. Majoie, M.W.A. Caan, T. Su (AMC, Department of Radiology); H.W. van Lunsen, M.A.F. Nievaard (AMC, Department of Gynaecology); B.J.H. van den Born, E.S.G. Stroes, (AMC, Division of Vascular Medicine); W.M.C. Mulder (HIV Vereniging Nederland).
A RANDOM RESULTS

The paper presents a theoretically well motivated for visualizing what parts of the input feature map are responsible for the output decision. The key insight is that features that maximally change the output and are simultaneously more unpredictable from other features are the most important ones. Most previous work has focused on finding features that maximally change the output without accounting for their predictability from other features. Authors build upon ideas presented in the work of Robnik-Šikonja & Kononenko (2008).
The results indicate that the proposed visualization mechanism based on modeling conditional distribution identifies more salient regions as compared to a mechanism based on modeling marginal distribution. I like that authors have presented visualization results for a single image across multiple networks and multiple classes. There results show that the proposed method indeed picks up on class-discriminative features. Authors have provided a link to visualizations for a random sample of images in a comment – I encourage the authors to include this in the appendix of the paper.
My one concern with the paper is – Zeiler et al., proposed a visualization method by greying small square regions in the image. This is similar to computing the visualization using the marginal distribution. Authors compute the marginal visualization using 10 samples, however in the limit of infinite samples the image region would be gray. The conditional distribution is computed using a normal distribution that provides some regularization and therefore estimating the conditional and marginal distributions using 10 samples each is not justified.
I would like to see the comparison when grey image patches (akin to Zeiler et al.) are used for visualization against the approach based on the conditional distribution.


TWO METHODS FOR WILD VARIATIONAL INFERENCE
Variational inference provides a powerful tool for approximate probabilistic inference on complex, structured models. Typical variational inference methods, however, require to use inference networks with computationally tractable probability density functions. This largely limits the design and implementation of variational inference methods. We consider wild variational inference methods that do not require tractable density functions on the inference networks, and hence can be applied in more challenging cases. As an example of application, we treat stochastic gradient Langevin dynamics (SGLD) as an inference network, and use our methods to automatically adjust the step sizes of SGLD, yielding significant improvement over the hand-designed step size schemes.
1 INTRODUCTION
Probabilistic modeling provides a principled approach for reasoning under uncertainty, and has been increasingly dominant in modern machine learning where highly complex, structured probabilistic models are often the essential components for solving complex problems with increasingly larger datasets. A key challenge, however, is to develop computationally efficient Bayesian inference methods to approximate, or draw samples from the posterior distributions. Variational inference (VI) provides a powerful tool for scaling Bayesian inference to complex models and big data. The basic idea of VI is to approximate the true distribution with a simpler distribution by minimizing the KL divergence, transforming the inference problem into an optimization problem, which is often then solved efficiently using stochastic optimization techniques (e.g., Hoffman et al., 2013; Kingma & Welling, 2013). However, the practical design and application of VI are still largely restricted by the requirement of using simple approximation families, as we explain in the sequel.
Let p(z) be a distribution of interest, such as the posterior distribution in Bayesian inference. VI approximates p(z) with a simpler distribution q∗(z) found in a set Q = {qη(z)} of distributions indexed by parameter η by minimizing the KL divergence objective:
min η
{ KL(qη || p) ≡ Ez∼qη [log(qη(z)/p(z))] } , (1)
where we can get exact result p = q∗ if Q is chosen to be broad enough to actually include p. In practice, however, Q should be chosen carefully to make the optimization in (1) computationally tractable; this casts two constraints on Q: 1. A minimum requirement is that we should be able to sample from qη efficiently, which allows us to make estimates and predictions based on qη in placement of the more intractable p. The samples from qη can also be used to approximate the expectation Eq[·] in (1) during optimization. This means that there should exist some computable function f(η; ξ), called the inference network, which takes a random seed ξ, whose distribution is denoted by q0, and outputs a random variable z = f(η; ξ) whose distribution is qη .
2. We should also be able to calculate the density qη(z) or it is derivative in order to optimize the KL divergence in (1). This, however, casts a much more restrictive condition, since it requires us to use only simple inference network f(η; ξ) and input distributions q0 to ensure a tractable form for the density qη of the output z = f(η; ξ).
In fact, it is this requirement of calculating qη(z) that has been the major constraint for the design of state-of-the-art variational inference methods. The traditional VI methods are often limited to
using simple mean field, or Gaussian-based distributions as qη and do not perform well for approximating complex target distributions. There is a line of recent work on variational inference with rich approximation families (e.g., Rezende & Mohamed, 2015b; Tran et al., 2015; Ranganath et al., 2015, to name only a few), all based on handcrafting special inference networks to ensure the computational tractability of qη(z) while simultaneously obtaining high approximation accuracy. These approaches require substantial mathematical insights and research effects, and can be difficult to understand or use for practitioners without a strong research background in VI. Methods that allow us to use arbitrary inference networks without substantial constraints can significantly simplify the design and applications of VI methods, allowing practical users to focus more on choosing proposals that work best with their specific tasks.
We use the term wild variational inference to refer to variants of variational methods working with general inference networks f(η, ξ) without tractability constraints on its output density qη(z); this should be distinguished with the black-box variational inference (Ranganath et al., 2014) which refers to methods that work for generic target distributions p(z) without significant model-by-model consideration (but still require to calculate the proposal density qη(z)). Essentially, wild variational inference makes it possible to “learn to draw samples”, constructing black-box neural samplers for given distributions. This enables more adaptive and automatic design of efficient Bayesian inference procedures, replacing the hand-designed inference algorithms with more efficient ones that can improve their efficiency adaptively over time based on past tasks they performed.
In this work, we discuss two methods for wild variational inference, both based on recent works that combine kernel techniques with Stein’s method (e.g., Liu & Wang, 2016; Liu et al., 2016). The first method, also discussed in Wang & Liu (2016), is based on iteratively adjusting parameter η to make the random output z = f(η; ξ) mimic a Stein variational gradient direction (SVGD) (Liu & Wang, 2016) that optimally decreases its KL divergence with the target distribution. The second method is based on minimizing a kernelized Stein discrepancy, which, unlike KL divergence, does not require to calculate density qη(z) for the optimization thanks to its special form.
Another critical problem is to design good network architectures well suited for Bayesian inference. Ideally, the network design should leverage the information of the target distribution p(z) in a convenient way. One useful perspective is that we can view the existing MC/MCMC methods as (hand-designed) stochastic neural networks which can be used to construct native inference networks for given target distributions. On the other hand, using existing MC/MCMC methods as inference networks also allow us to adaptively adjust the hyper-parameters of these algorithms; this enables amortized inference which leverages the experience on past tasks to accelerate the Bayesian computation, providing a powerful approach for designing efficient algorithms in settings when a large number of similar tasks are needed.
As an example, we leverage stochastic gradient Langevin dynamics (SGLD) (Welling & Teh, 2011) as the inference network, which can be treated as a special deep residential network (He et al., 2016), in which important gradient information ∇z log p(z) is fed into each layer to allow efficient approximation for the target distribution p(z). In our case, the network parameter η are the step sizes of SGLD, and our method provides a way to adaptively improve the step sizes, providing speed-up on future tasks with similar structures. We show that the adaptively estimated step sizes significantly outperform the hand-designed schemes such as Adagrad.
Related Works The idea of amortized inference (Gershman & Goodman, 2014) has been recently applied in various domains of probabilistic reasoning, including both amortized variational inference
(e.g., Kingma & Welling, 2013; Rezende & Mohamed, 2015a) and date-driven designs of Monte Carlo based methods (e.g., Paige & Wood, 2016), to name only a few. Most of these methods, however, require to explicitly calculate qη(z) (or its gradient).
One well exception is a very recent work (Ranganath et al., 2016) that also avoids calculating qη(z) and hence works for general inference networks; their method is based on a similar idea related to Stein discrepancy (Liu et al., 2016; Oates et al., 2017; Chwialkowski et al., 2016; Gorham & Mackey, 2015), for which we provide a more detailed discussion in Section 3.2.
The auxiliary variational inference methods (e.g., Agakov & Barber, 2004) provide an alternative way when the variational distribution qη(z) can be represented as a hidden variable model. In particular, Salimans et al. (2015) used the auxiliary variational approach to leverage MCMC as a variational approximation. These approaches, however, still require to write down the likelihood function on the augmented spaces, and need to introduce an additional inference network related to the auxiliary variables.
There is a large literature on traditional adaptive MCMC methods (e.g., Andrieu & Thoms, 2008; Roberts & Rosenthal, 2009) which can be used to adaptively adjust the proposal distribution of MCMC by exploiting the special theoretical properties of MCMC (e.g., by minimizing the autocorrelation). Our method is simpler, more generic, and works efficiently in practice thanks to the use of gradient-based back-propagation. Finally, connections between stochastic gradient descent and variational inference have been discussed and exploited in Mandt et al. (2016); Maclaurin et al. (2015).
Outline Section 2 introduces background on Stein discrepancy and Stein variational gradient descent. Section 3 discusses two methods for wild variational inference. Section 4 discuss using stochastic gradient Langevin dynamics (SGLD) as the inference network. Empirical results are shown in Section 5.
2 STEIN’S IDENTITY, STEIN DISCREPANCY, STEIN VARIATIONAL GRADIENT
Stein’s identity Stein’s identity plays a fundamental role in our framework. Let p(z) be a positive differentiable density on Rd, and φ(z) = [φ1(z), · · · , φd(z)]> is a differentiable vector-valued function. Define∇z · φ = ∑ i ∂ziφ. Stein’s identity is
Ez∼p[〈∇z log p(z), φ(z)〉+∇z · φ(z)] = ∫ X ∇z · (p(z)φ(z))dx = 0, (2)
which holds once p(z)φ(z) vanishes on the boundary of X by integration by parts or Stokes’ theorem; It is useful to rewrite Stein’s identity in a more compact way:
Ez∼p[Tpφ(z)] = 0, with Tpφ def = 〈∇z log p, φ〉+∇z · φ, (3)
where Tp is called a Stein operator, which acts on function φ and returns a zero-mean function Tpφ(z) under z ∼ p. A key computational advantage of Stein’s identity and Stein operator is that they depend on p only through the derivative of the log-density ∇z log p(z), which does not depend on the cumbersome normalization constant of p, that is, when p(z) = p̄(z)/Z, we have ∇z log p(z) = ∇z log p̄(z), independent of the normalization constant Z. This property makes Stein’s identity a powerful practical tool for handling unnormalized distributions widely appeared in machine learning and statistics.
Stein Discrepancy Although Stein’s identity ensures that Tpφ has zero expectation under p, its expectation is generally non-zero under a different distribution q. Instead, for p 6= q, there must exist a φ which distinguishes p and q in the sense that Ez∼q[Tpφ(z)] 6= 0. Stein discrepancy leverages this fact to measure the difference between p and q by considering the “maximum violation of Stein’s identity” for φ in certain function set F :
D(q || p) = max φ∈F
{ Ez∼q[Tpφ(z)] } , (4)
where F is the set of functions φ that we optimize over, and decides both the discriminative power and computational tractability of Stein discrepancy. Kernelized Stein discrepancy (KSD) is a special
Stein discrepancy that takes F to be the unit ball of vector-valued reproducing kernel Hilbert spaces (RKHS), that is,
F = {φ ∈ Hd : ||φ||Hd ≤ 1}, (5)
where H is a real-valued RKHS with kernel k(z, z′). This choice of F makes it possible to get a closed form solution for the optimization in (4) (Liu et al., 2016; Chwialkowski et al., 2016; Oates et al., 2017):
D(q || p) = max φ∈Hd
{ Ez∼q[Tpφ(z)], s.t. ||φ||Hd ≤ 1 } , (6)
= √ Ez,z′∼q[κp(z, z′)], (7)
where κp(z, z′) is a positive definite kernel obtained by applying Stein operator on k(z, z′) twice:
κp(z, z ′) = T z
′
p (T zp ⊗ k(z, z′)), = sp(z)sp(z
′)k(z, z′) + sp(z)∇z′k(z, z′) + sp(z′)∇zk(z, z′) +∇z · (∇z′k(z, z′)), (8)
where sp(z) = ∇z log p(z) and T zp and T zp denote the Stein operator when treating k(z, z′) as a function of z and z′, respectively; here we defined T zp ⊗k(z, z′) = ∇x log p(x)k(z, z′)+∇xk(z, z′) which returns a d× 1 vector-valued function. It can be shown that D(q || p) = 0 if and only if q = p when k(z, z′) is strictly positive definite in a proper sense (Liu et al., 2016; Chwialkowski et al., 2016). D(q || p) can treated as a variant of maximum mean discrepancy equipped with kernel κp(z, z
′) which depends on p (which makes D(q || p) asymmetric on q and p). The form of KSD in (6) allows us to estimate the discrepancy between a set of sample {zi} (e.g., drawn from q) and a distribution p specified by∇z log p(z),
D̂2u({zi} || p) = 1 n(n− 1) ∑ i6=j [κp(zi, zj)], D̂2v({zi} || p) = 1 n2 ∑ i,j [κp(zi, zj)], (9)
where D̂2u(q || p) provides an unbiased estimator (hence called a U -statistic) for D2(q || p), and D̂2v(q || p), called V -statistic, provides a biased estimator but is guaranteed to be always nonnegative: D̂2v({zi} || p) ≥ 0.
Stein Variational Gradient Descent (SVGD) Stein operator and Stein discrepancy have a close connection with KL divergence, which is exploited in Liu & Wang (2016) to provide a general purpose deterministic approximate sampling method. Assume that {zi}ni=1 is a sample (or a set of particles) drawn from q, and we want to update {zi}ni=1 to make it “move closer” to the target distribution p to improve the approximation quality. We consider updates of form
zi ← zi + φ∗(zi), ∀i = 1, . . . , n, (10)
where φ∗ is a perturbation direction, or velocity field, chosen to maximumly decrease the KL divergence between the distribution of updated particles and the target distribution, in the sense that
φ∗ = arg max φ∈F { − d d KL(q[ φ] || p) ∣∣ =0 } , (11)
where q[ φ] denotes the density of the updated particle z′ = z + φ(z) when the density of the original particle z is q, and F is the set of perturbation directions that we optimize over. A key observation (Liu & Wang, 2016) is that the optimization in (11) is in fact equivalent to the optimization for KSD in (4); we have
− d d KL(q[ φ] || p) ∣∣ =0 = Ez∼q[Tpφ(z)], (12)
that is, the Stein operator transforms the perturbation φ on the random variable (the particles) to the change of the KL divergence. Taking F to be unit ball of Hd as in (5), the optimal solution φ∗ of (11) equals that of (6), which is shown to be (e.g., Liu et al., 2016)
φ∗(z′) ∝ Ez∼q[T zp k(z, z′)] = Ez∼q[∇z log p(z)k(z, z′) +∇zk(z, z′)].
Algorithm 1 Amortized SVGD and KSD Minimization for Wild Variational Inference for iteration t do
1. Draw random {ξi}ni=1, calculate zi = f(η; ξi), and the Stein variational gradient ∆zi in (13). 2. Update parameter η using (14) or (15) for amortized SVGD, or (17) for KSD minimization.
end for
By approximating the expectation under q with the empirical mean of the current particles {zi}ni=1, SVGD admits a simple form of update that iteratively moves the particles towards the target distribution,
zi ← zi + ∆zi, ∀i = 1, . . . , n, ∆zi = Êz∈{zi}ni=1 [∇z log p(z)k(z, zi) +∇zk(z, zi)], (13)
where Êz∼{zi}ni=1 [f(z)] = ∑ i f(zi)/n. The two terms in ∆zi play two different roles: the term with the gradient ∇z log p(z) drives the particles towards the high probability regions of p(z), while the term with ∇zk(z, zi) serves as a repulsive force to encourage diversity; to see this, consider a stationary kernel k(z, z′) = k(z − z′), then the second term reduces to Êz∇zk(z, zi) = −Êz∇zik(z, zi), which can be treated as the negative gradient for minimizing the average similarity Êzk(z, zi) in terms of zi.
It is easy to see from (13) that ∆zi reduces to the typical gradient ∇z log p(zi) when there is only a single particle (n = 1) and ∇zk(z, zi) when z = zi, in which case SVGD reduces to the standard gradient ascent for maximizing log p(z) (i.e., maximum a posteriori (MAP)).
3 TWO METHODS FOR WILD VARIATIONAL INFERENCE
Since the direct parametric optimization of the KL divergence (1) requires calculating qη(z), there are two essential ways to avoid calculating qη(z): either using alternative (approximate) optimization approaches, or using different divergence objective functions. We discuss two possible approaches in this work: one based on “amortizing SVGD” (Wang & Liu, 2016) which trains the inference network f(η, ξ) so that its output mimic the SVGD dynamics in order to decrease the KL divergence; another based on minimizing the KSD objective (9) which does not require to evaluate q(z) thanks to its special form.
3.1 AMORTIZED SVGD
SVGD provides an optimal updating direction to iteratively move a set of particles {zi} towards the target distribution p(z). We can leverage it to train an inference network f(η; ξ) by iteratively adjusting η so that the output of f(η; ξ) changes along the Stein variational gradient direction in order to maximumly decrease its KL divergence with the target distribution. By doing this, we “amortize” SVGD into a neural network, which allows us to leverage the past experience to adaptively improve the computational efficiency and generalize to new tasks with similar structures. Amortized SVGD is also presented in Wang & Liu (2016); here we present some additional discussion.
To be specific, assume {ξi} are drawn from q0 and zi = f(η; ξi) the corresponding random output based on the current estimation of η. We want to adjust η so that zi changes along the Stein variational gradient direction ∆zi in (13) so as to maximumly decrease the KL divergence with target distribution. This can be done by updating η via
η ← arg min η n∑ i=1 ||f(η; ξi)− zi − ∆zi||22. (14)
Essentially, this projects the non-parametric perturbation direction ∆zi to the change of the finite dimensional network parameter η. If we take the step size to be small, then the updated η by (14) should be very close to the old value, and a single step of gradient descent of (14) can provide a
good approximation for (14). This gives a simpler update rule: η ← η + ∑ i ∂ηf(η; ξi)∆zi, (15)
which can be intuitively interpreted as a form of chain rule that back-propagates the SVGD gradient to the network parameter η. In fact, when we have only one particle, (15) reduces to the standard gradient ascent for maxη log p(f(η; ξ)), in which fη is trained to “learn to optimize” (e.g., Andrychowicz et al., 2016), instead of “learn to sample” p(z). Importantly, as we have more than one particles, the repulsive term ∇zk(z, zi) in ∆zi becomes active, and enforces an amount of diversity on the network output that is consistent with the variation in p(z). The full algorithm is summarized in Algorithm 1.
Amortized SVGD can be treated as minimizing the KL divergence using a rather special algorithm: it leverages the non-parametric SVGD which can be treated as approximately solving the infinite dimensional optimization minq KL(q || p) without explicitly assuming a parametric form on q, and iteratively projecting the non-parametric update back to the finite dimensional parameter space of η. It is an interesting direction to extend this idea to “amortize” other MC/MCMC-based inference algorithms. For example, given a MCMC with transition probability T (z′|z) whose stationary distribution is p(z), we may adjust η to make the network output move towards the updated values z′ drawn from the transition probability T (z′|z). The advantage of using SVGD is that it provides a deterministic gradient direction which we can back-propagate conveniently and is particle efficient in that it reduces to “learning to optimize” with a single particle. We have been using the simple L2 loss in (14) mainly for convenience; it is possible to use other two-sample discrepancy measures such as maximum mean discrepancy.
3.2 KSD VARIATIONAL INFERENCE
Amortized SVGD attends to minimize the KL divergence objective, but can not be interpreted as a typical finite dimensional optimization on parameter η. Here we provide an alternative method based on directly minimizing the kernelized Stein discrepancy (KSD) objective, for which, thanks to its special form, the typical gradient-based optimization can be performed without needing to estimate q(z) explicitly.
To be specific, take qη to be the density of the random output z = f(η; ξ) when ξ ∼ q0, and we want to find η to minimize D(qη || p). Assuming {ξi} is i.i.d. drawn from q0, we can approximate D2(qη || p) unbiasedly with a U-statistics:
D2(qη || p) ≈ 1 n(n− 1) ∑ i 6=j κp(f(η; ξi), f(η; ξj)), (16)
for which a standard gradient descent can be derived for optimizing η:
η ← η − 2 n(n− 1) ∑ i 6=j ∂ηf(η; ξi)∇ziκp(zi, zj), where zi = f(η; ξi). (17)
This enables a wild variational inference method based on directly minimizing η with standard (stochastic) gradient descent. See Algorithm 1. Note that (17) is similar to (15) in form, but replaces ∆zi with a ∆̃zi ∝ − ∑ j : i 6=j ∇ziκp(zi, zj). It is also possible to use the V -statistic in (9), but we find that the U -statistic performs much better in practice, possibly because of its unbiasedness property.
Minimizing KSD can be viewed as minimizing a constrastive divergence objective function. To see this, recall that q[ φ] denotes the density of z′ = z + φ(z) when z ∼ q. Combining (11) and (6), we can show that
D2(q || p) ≈ 1
(KL(q || p)−KL(q[ φ] || p)).
That is, KSD measures the amount of decrease of KL divergence when we update the particles along the optimal SVGD perturbation direction φ given by (11). If q = p, then the decrease of KL
divergence equals zero and D2(q || p) equals zero. In fact, as shown in Liu & Wang (2016) KSD can be explicitly represented as the magnitude of a functional gradient of KL divergence:
D(q || p) = ∣∣∣∣∣∣ d dφ KL(q[φ] || p) ∣∣ φ=0 ∣∣∣∣∣∣ Hd ,
where q[φ] is the density of z = z +φ(z) when z ∼ q, and ddφF (φ) denotes the functional gradient of functional F (φ) w.r.t. φ defined in RKHSHd, and ddφF (φ) is also an element inH
d. Therefore, KSD variational inference can be treated as explicitly minimizing the magnitude of the gradient of KL divergence, in contract with amortized SVGD which attends to minimize the KL divergence objective itself.
This idea is also similar to the contrastive divergence used for learning restricted Boltzmann machine (RBM) (Hinton, 2002) (which, however, optimizes p with fixed q). It is possible to extend this approach by replacing z′ = z + φ(z) with other transforms, such as these given by a transition probability of a Markov chain whose stationary distribution is p. In fact, according the so called generator method for constructing Stein operator (Barbour, 1988), any generator of a Markov process defines a Stein operator that can be used to define a corresponding Stein discrepancy.
This idea is related to a very recent work by Ranganath et al. (2016), which is based on directly minimizing the variational form of Stein discrepancy in (4); Ranganath et al. (2016) assumes F consists of a neural network φτ (z) parametrized by τ , and find η by solving the following min-max problem:
min η max τ Ez∼q[Tpφτ (z)].
In contrast, our method leverages the closed form solution by taking F to be an RKHS and hence obtains an explicit optimization problem, instead of a min-max problem that can be computationally more expensive, or have difficulty in achieving convergence.
Because κp(x, x′) (defined in (8)) depends on the derivative ∇x log p(x) of the target distribution, the gradient in (17) depends on the Hessian matrix ∇2x log p(x) and is hence less convenient to implement compared with amortized SVGD (the method by Ranganath et al. (2016) also has the same problem). However, this problem can be alleviated using automatic differentiation tools, which be used to directly take the derivative of the objective in (16) without manually deriving its derivatives.
4 LANGEVIN INFERENCE NETWORK
With wild variational inference, we can choose more complex inference network structures to obtain better approximation accuracy. Ideally, the best network structure should leverage the special properties of the target distribution p(z) in a convenient way. One way to achieve this by viewing existing MC/MCMC methods as inference networks with hand-designed (and hence potentially suboptimal) parameters, but good architectures that take the information of the target distribution p(z) into account. By applying wild variational inference on networks constructed based on existing MCMC methods, we effectively provide an hyper-parameter optimization for these existing methods. This allows us to fully optimize the potential of existing Bayesian inference methods, significantly improving the result with less computation cost, and decreasing the need for hyper-parameter tuning by human experts. This is particularly useful when we need to solve a large number of similar tasks, where the computation cost spent on optimizing the hyper-parameters can significantly improve the performance on the future tasks.
Stochastic Gradient Langevin Dynamics We first take the original stochastic gradient Langevin dynamics (SGLD) algorithm (Welling & Teh, 2011) as an example. SGLD starts with a random initialization z0, and perform iterative update of form
zt+1 ← zt + ηt ∇z log p̂(zt; Mt) + √ 2ηt ξt, ∀t = 1, · · ·T, (18)
where log p̂(zt; Mt) denotes an approximation of log p(zt) based on, e.g., a random mini-batch Mt of observed data at t-th iteration, and ξt is a standard Gaussian random vector of the same size as z, and ηt denotes a (vector) step-size at t-th iteration; here “ ” denotes element-wise product. When running SGLD for T iterations, we can treat zT as the output of a T -layer neural network
parametrized by the collection of step sizes η = {ηt}Tt=1, whose random inputs include the random initialization z0, the mini-batch Mt and Gaussian noise ξt at each iteration t. We can see that this defines a rather complex network structure with several different types of random inputs (z0, Mt and ξt). This makes it intractable to explicitly calculate the density of zT and traditional variational inference methods can not be applied directly. But wild variational inference can still allow us to adaptively improve the optimal step-size η in this case.
General Langevin Networks Based on the original formula of SGLD, we proposed a more general langevin network structure, and each layer of the network has a form
zt+1 ← Atzt + h(BtBt>∇z log p̂(zt; Mt) +Btξt +Dt), ∀t = 1, · · ·T, (19)
where At, Bt and Dt are network parameters at t-th iteration(whose size is d× d, and d is the size of zt), and h(·) denotes a smooth element-wise non-linearity function; here ξt is still a standard gaussian random vector with the same size as z. With this more complex network, we can use fewer layers to construct more powerful back-box samplers.
5 EMPIRICAL RESULTS
5.1 SGLD INFERENCE NETWORK
We first test our algorithm with SGLD inference network with (18) formula on both a toy Gaussian mixture model and a Bayesian logistic regression example. We find that we can adaptively learn step sizes that significantly outperform the existing hand-designed step size schemes, and hence save computational cost in the testing phase. In particular, we compare with the following step size schemes, for all of which we report the best results (testing accuracy in Figure 3(a); testing likelihood in Figure 3(b)) among a range of hyper-parameters:
1. Constant Step Size. We select a best constant step size in {1, 2, 23, . . . , 229} × 10−6. 2. Power Decay Step Size. We consider t = 10a × (b + t)−γ where γ = 0.55, a ∈ {−6,−5, . . . , 1, 2}, b ∈ {0, 1, . . . , 9}. 3. Adagrad, Rmsprop, Adadelta, all with the master step size selected in {1, 2, 23, . . . , 229}× 10−6, with the other parameters chosen by default values.
Gaussian Mixture We start with a simple 1D Gaussian mixture example shown in Figure 2 where the target distribution p(z) is shown by the red dashed curve. We use amortized SVGD and KSD to optimize the step size parameter of the Langevin inference network in (18) with T = 20 layers (i.e., SGLD with T = 20 iterations), with an initial z0 drawn from a q0 far away from the target distribution (see the green curve in Figure 2(a)); this makes it critical to choose a proper step size to achieve close approximation within T = 20 iterations. We find that amortized SVGD and KSD allow us to achieve good performance with 20 steps of SGLD updates (Figure 2(b)-(c)), while the result of the best constant step size and power decay step-size are much worse (Figure 2(d)-(e)).
Bayesian Logistic Regression We consider Bayesian logistic regression for binary classification using the same setting as Gershman et al. (2012), which assigns the regression weights w with a Gaussian prior p0(w|α) = N (w,α−1) and p0(α) = Gamma(α, 1, 0.01). The inference is applied on the posterior of z = [w, logα]. We test this model on the binary Covertype dataset1 with 581,012 data points and 54 features.
To demonstrate that our estimated learning rate can work well on new datasets never seen by the algorithm. We partition the dataset into mini-datasets of size 50, 000, and use 80% of them for training and 20% for testing. We adapt our amortized SVGD/KSD to train on the whole population of the training mini-datasets by randomly selecting a mini-dataset at each iteration of Algorithm 1, and evaluate the performance of the estimated step sizes on the remaining 20% testing mini-datasets.
Figure 3 reports the testing accuracy and likelihood on the 20% testing mini-datasets when we train the Langevin network with T = 10, 50, 100 layers, respectively. We find that our methods outperform all the hand-designed learning rates, and allow us to get performance closer to the fully converged SGLD and SVGD with a small number T of iterations.
Figure 4 shows the testing accuracy and testing likelihood of all the intermediate results when training Langevin network with T = 100 layers. It is interesting to observe that amortized SVGD and KSD learn rather different behavior: KSD tends to increase the performance quickly at the first few iterations but saturate quickly, while amortized SVGD tends to increase slowly in the beginning and boost the performance quickly in the last few iterations. Note that both algorithms are set up to optimize the performance of the last layers, while need to decide how to make progress on the intermediate layers to achieve the best final performance.
5.2 GENERAL LANGEVIN INFERENCE NETWORK
We further test our algorithm with general Langevin inference network. We firstly construct one single layer general Langevin network to approach the posterior of Bayesian logistic regression parameters and we can achieve 74.58% average accuracy and −0.5216 average testing log-likelihood in 100 repeat experiments. This result proves the proposed general Langevin Inference Network is quite competitive and worth to explore. Moreover, we use it as a black-box sampler to approach more complicate Gaussian Mixture distributions.
Gaussian Mixture We consider 10 components Gaussian Mixture Models with mean and covariance matrix of each component randomly drawed from a uniform distribution, and we test our methods on different dimensions models.
We construct 6 layers of general Langevin networks as a black-box sampler, and our proposed two methods to train the black-box sampler to approximate the target distribution. Figure 5 shows our
1https://www.csie.ntu.edu.tw/˜cjlin/libsvmtools/datasets/binary.html
results on 50 dimension Gaussian Mixture case and figure 6 shows results of different dimensions of Gaussian Mixture. From the figures we can know that our proposed sampling structure is quite competive comparing with NUT sampler(Hoffman & Gelman, 2014), and these two variational inference methods can both train a good black-box sampler.
6 CONCLUSION
We consider two methods for wild variational inference that allows us to train general inference networks with intractable density functions, and apply it to adaptively estimate step sizes of stochastic gradient Langevin dynamics. More studies are needed to develop better methods, more applications and theoretical understandings for wild variational inference, and we hope that the two methods we discussed in the paper can motivate more ideas and studies in the field.

The authors propose two variational methods based on the theme of posterior approximations which may not have a tractable density. The first is from another ICLR submission on "amortized SVGD" (Wang and Liu, 2016), where here the innovation is in using SGLD as the inference network. The second is from a NIPS paper (Ranganath et al., 2016) on minimizing the Stein divergence with a parametric approximating family, where here the innovation is in defining their test functions to be an RKHS, obtaining an analytic solution to the inner optimization problem.
The methodology is incremental. Everything up to Section 3.2 is essentially motivation, background, or related work. The notion of a "wild variational approximation" was already defined in Ranganath et al. (2016), termed a "variational program". It would be useful for the authors to comment on the difference, if any.
Section 3.2 is at first interesting because it analytically solves the maximum problem that is faced in Ranganath et al. (2016). However, this requires use of a kernel which will certainly not scale in high dimensions, so it is then equivalent in practice to having chosen a very simple test function family. To properly scale to high dimensions would require a deeper kernel and also learning its parameters; this is not any easier than parameterizing the test function family as a neural network to begin with, which Ranganath et al. (2016) do.
Section 4 introduces a Langevin inference network, which essentially chooses the variational approximation as an evolving sequence of Markov transition operators as in Salimans et al. (2015). I had trouble understanding this for a while because I could not understand what they mean by inference network. None of it is amortized in the usual inference network sense, which is that the parameters are given by the output of a neural network. Here, the authors simple define global parameters of the SGLD chain which are used across all the latent variables (which is strictly worse?). (What then makes it an "inference network"?) Is this not the variational approximation used in Salimans et al. (2015), but using a different objective to train it?
The experiments are limited, on a toy mixture of Gaussians posterior and Bayesian logistic regression. None of this addresses the problems one might suspect on high-dimensional and real data, such as the lack of scalability for the kernel, the comparison to Salimans et al. (2015) for the Langevin variational approximation, and any note of runtime or difficulty of training.
Minor comments
+ It's not clear if the authors understood previous work on expressive variational families or inference networks. For example, they argue Rezende & Mohamed, 2015b; Tran et al., 2015; Ranganath et al., 2015 require handcrafted inference networks. However, all of them assume use of any neural network for amortized inference. None of them even require an inference network. Perhaps the authors mean handcrafted posterior approximations, which to some extent is true; however, the three mentioned are all algorithmic in nature: in Rezende & Mohamed (2015), the main decision choice is the flow length; Tran et al. (2015), the size of the variational data; Ranganath et al. (2015), the flow length on the auxiliary variable space. Each works well on different problems, but this is also true of variational objectives which admit intractable q (as the latter two consider, as does Salimans et al. (2015)). The paper's motivation could be better explained, and perhaps the authors could be clearer on what they mean by inference network.
+ I also recommend the authors not term a variational inference method based on the class of approximating family. While black box variational inference in Ranganath et al. (2014) assumes a mean-field family, the term itself has been used in the literature to mean any variational method that imposes few constraints on the model class.


BEYOND BILINGUAL: MULTI-SENSE WORD EMBED-
Word embeddings, which represent a word as a point in a vector space, have become ubiquitous to several NLP tasks. A recent line of work uses bilingual (two languages) corpora to learn a different vector for each sense of a word, by exploiting crosslingual signals to aid sense identification. We present a multiview Bayesian non-parametric algorithm which improves multi-sense word embeddings by (a) using multilingual (i.e., more than two languages) corpora to significantly improve sense embeddings beyond what one achieves with bilingual information, and (b) uses a principled approach to learn a variable number of senses per word, in a data-driven manner. Ours is the first approach with the ability to leverage multilingual corpora efficiently for multi-sense representation learning. Experiments show that multilingual training significantly improves performance over monolingual and bilingual training, by allowing us to combine different parallel corpora to leverage multilingual context. Multilingual training yields comparable performance to a state of the art monolingual model trained on five times more training data.
1 INTRODUCTION
Word embeddings (Turian, Ratinov, and Bengio, 2010; Mikolov, Yih, and Zweig, 2013, inter alia) represent a word as a point in a vector space. This space is able to capture semantic relationships: vectors of words with similar meanings have high cosine similarity. Use of embeddings as features has been shown to benefit several NLP tasks and serve as good initializations for deep architectures ranging from dependency parsing (Bansal, Gimpel, and Livescu, 2014) to named entity recognition (Guo et al., 2014b).
Although these representations are now ubiquitous in NLP, most algorithms for learning wordembeddings do not allow a word to have different meanings in different contexts, a phenomenon known as polysemy. For example, the word bank assumes different meanings in financial (eg. “bank pays interest”) and geographical contexts (eg. “river bank”) and which cannot be represented adequately with a single embedding vector. Unfortunately, there are no large sense-tagged corpora available and such polysemy must be inferred from the data during the embedding process.
Several attempts (Reisinger and Mooney, 2010; Neelakantan et al., 2014; Li and Jurafsky, 2015) have been made to infer multi-sense word representations by modeling the sense as a latent variable in a Bayesian non-parametric framework. These approaches rely on the ”one-sense per collocation” heuristic (Yarowsky, 1995), which assumes that presence of nearby words correlate with the sense of the word of interest. This heuristic provides only a weak signal for sense identification, and such algorithms require large amount of training data to achieve competitive performance.
Recently, several approaches (Guo et al., 2014a; Šuster, Titov, and van Noord, 2016) propose to learn multi-sense embeddings by exploiting the fact that different senses of the same word may be translated into different words in a foreign language (Dagan and Itai, 1994; Resnik and Yarowsky, 1999; Diab and Resnik, 2002; Ng, Wang, and Chan, 2003). For example, bank in English may be translated to banc or banque in French, depending on whether the sense is financial or geographical. Such bilingual distributional information allows the model to identify which sense of a word is being used during training.
However, bilingual distributional signals often do not suffice. It is common that polysemy for a word survives translation. Fig. 1 shows an illustrative example – both senses of interest get translated to intérêt in French. However, this becomes much less likely as the number of languages under consideration grows. By looking at Chinese translation in Fig. 1, we can observe that the senses translate to different surface forms. Note that the opposite can also happen (i.e. same surface forms in Chinese, but different in French). Existing crosslingual approaches are inherently bilingual and cannot naturally extend to include additional languages due to several limitations (details in Section4). Furthermore, works like (Šuster, Titov, and van Noord, 2016) sets a fixed number of senses for each word, leading to inefficient use of parameters, and unnecessary model complexity.1
This paper addresses these limitations by proposing a multi-view Bayesian non-parametric word representation learning algorithm which leverages multilingual distributional information. Our representation learning framework is the first multilingual (not bilingual) approach, allowing us to utilize arbitrarily many languages to disambiguate words in English. To move to multilingual system, it is necessary to ensure that the embeddings of each foreign language are relatable to each other (i.e., they live in the same space). We solve this by proposing an algorithm in which word representations are learned jointly across languages, using English as a bridge. While large parallel corpora between two languages are scarce, using our approach we can concatenate multiple parallel corpora to obtain a large multilingual corpus. The parameters are estimated in a Bayesian nonparametric framework that allows our algorithm to only associate a word with a new sense vector when evidence (from either same or foreign language context) requires it. As a result, the model infers different number of senses for each word in a data-driven manner, avoiding wasting parameters.
Together, these two ideas – multilingual distributional information and nonparametric sense modeling – allow us to disambiguate multiple senses using far less data than is necessary for previous methods. We experimentally demonstrate that our algorithm can achieve competitive performance after training on a small multilingual corpus, comparable to a model trained monolingually on a much larger corpus. We present an analysis discussing the effect of various parameters – choice of language family for deriving the multilingual signal, crosslingual window size etc. and also show qualitative improvement in the embedding space.
2 RELATED WORK
Work on inducing multi-sense embeddings can be divided in two broad categories – two-staged approaches and joint learning approaches. Two-staged approaches (Reisinger and Mooney, 2010; Huang et al., 2012) induce multi-sense embeddings by first clustering the contexts and then using the clustering to obtain the sense vectors. The contexts can be topics induced using latent topic models(Liu, Qiu, and Huang, 2015; Liu et al., 2015), or Wikipedia (Wu and Giles, 2015) or coarse part-of-speech tags (Qiu et al., 2014). A more recent line of work in the two-staged category is that of retrofitting (Faruqui et al., 2015; Jauhar, Dyer, and Hovy, 2015), which aims to infuse semantic ontologies from resources like WordNet (Miller, 1995) and Framenet (Baker, Fillmore, and Lowe, 1998) into embeddings during a post-processing step. Such resources list (albeit not exhaustively) the senses of a word, and by retro-fitting it is possible to tease apart the different senses of a word. While some resources like WordNet (Miller, 1995) are available for many languages, they are not exhaustive in listing all possible senses. Indeed, the number senses of a word is highly dependent on the task and cannot be pre-determined using a lexicon (Kilgarriff, 1997). Ideally, the senses should be inferred in a data-driven manner, so that new senses not listed in such lexicons can be discovered. While recent work has attempted to remedy this by using parallel text for retrofitting sense-specific embeddings (Ettinger, Resnik, and Carpuat, 2016), their procedure requires creation of sense graphs, which introduces additional tuning parameters. On the other hand, our approach only requires two tuning parameters (prior α and maximum number of senses T ).
1Most words in conventional English are monosemous, i.e. single sense (eg. the word monosemous)
In contrast, joint learning approaches (Neelakantan et al., 2014; Li and Jurafsky, 2015) jointly learn the sense clusters and embeddings by using non-parametrics. Our approach belongs to this category. The closest non-parametric approach to ours is that of (Bartunov et al., 2016), who proposed a multisense variant of the skip-gram model which learns the different number of sense vectors for all words from a large monolingual corpus (eg. English Wikipedia). Our work can be viewed as the multi-view extension of their model which leverages both monolingual and crosslingual distributional signals for learning the embeddings. In our experiments, we compare our model to monolingually trained version of their model.
Incorporating crosslingual distributional information is a popular technique for learning word embeddings, and improves performance on several downstream tasks (Faruqui and Dyer, 2014; Guo et al., 2016; Upadhyay et al., 2016). However, there has been little work on learning multi-sense embeddings using crosslingual signals (Bansal, DeNero, and Lin, 2012; Guo et al., 2014a; Šuster, Titov, and van Noord, 2016) with only (Šuster, Titov, and van Noord, 2016) being a joint approach. (Kawakami and Dyer, 2015) also used bilingual distributional signals in a deep neural architecture to learn context dependent representations for words, though they do not learn separate sense vectors.
3 MODEL DESCRIPTION
Let E = {xe1, .., xei , .., xeNe} denote the words of the English side and F = {x f 1 , .., x f i , .., x f Nf } denote the words of the foreign side of the parallel corpus. We assume that we have access to word alignments Ae→f and Af→e mapping words in English sentence to their translation in foreign sentence (and vice-versa), so that xe and xf are aligned if Ae→f (xe) = xf . We define Nbr(x, L, d) as the neighborhood in language L of size d (on either side) around word x in its sentence. The English and foreign neighboring words are denoted by ye and yf , respectively. Note that ye and yf need not be translations of each other. Each word xf in the foreign vocabulary is associated with a dense vector xf in Rm, and each word xe in English vocabulary admits at most T sense vectors, with the kth sense vector denoted as xek.
2 As our main goal is to model multiple senses for words in English, we do not model polysemy in the foreign language and use a single vector to represent each word in the foreign vocabulary.
We model the joint conditional distribution of the context words ye, yf given an English word xe and its corresponding translation xf on the parallel corpus:
P (ye, yf | xe, xf ;α, θ), (1) where θ are model parameters (i.e. all embeddings) and α governs the hyper-prior on latent senses.
Assume xe has multiple senses, which are indexed by the random variable z, Eq. (1) can be rewritten, ∫
β
∑ z P (ye, yfz, β | xe, xf , α; θ)dβ
where β are the parameters determining the model probability on each sense for xe (i.e., the weight on each possible value for z). We place a Dirichlet process (Ferguson, 1973) prior on sense assignment for each word. Thus, adding the word-x subscript to emphasize that these are word-specific senses,
P (zx = k | βx) = βxk ∏k−1
r=1 (1− βxr), βxk | α
ind∼ Beta(βxk | 1, α), k = 1, . . . . (2)
That is, the potentially infinite number of senses for each word x have probability determined by the sequence of independent stick-breaking weights, βxk, in the constructive definition of the DP (Sethuraman, 1994). The hyper-prior concentration α provides information on the number of senses we expect to observe in our corpus.
After conditioning upon word sense, we decompose the context probability P (ye, yf | z, xe, xf ; θ) into two terms, P (ye | xe, xf , z; θ)P (yf | xe, xf , z; θ). Both the first and the second terms are sense-dependent, and each factors as,
P (y |xe, xf , z=k; θ)∝Ψ(xe, z=k, y)Ψ(xf , y) = exp(yTxek) exp(yTxf ) = exp(yT (xek+xf )), 2We also maintain a context vector for each word in the English and Foreign vocabularies. The context
vector is used as the representation of the word when it appears as the context for another word.
The bank paid me [interest] on my savings.
Ψ(interest,2,savings)
where xek is the embedding corresponding to the k th sense of the word xe, and y is either ye or yf . The factor Ψ(xe, z = k, y) use the corresponding sense vector in a skip-gram-like formulation. This results in total of 4 factors,
P (ye, yf | z, xe, xf ; θ) ∝ Ψ(xe, z, ye)Ψ(xf , yf )Ψ(xe, z, yf )Ψ(xf , ye) (3) See Figure 2 for illustration of each factor. This modeling approach is reminiscent of (Luong, Pham, and Manning, 2015), who jointly learned embeddings for two languages l1 and l2 by optimizing a joint objective containing 4 skip-gram terms using the aligned pair (xe,xf )– two predicting monolingual contexts l1 → l1, l2 → l2 , and two predicting crosslingual contexts l1 → l2, l2 → l1.
Learning. Learning involves maximizing the log-likelihood, P (ye, yf | xe, xf ;α, θ) = ∫ β ∑ z P (ye, yf , z, β | xe, xf , α; θ)dβ
Let q(z, β) = q(z)q(β) where q(z) = ∏ i q(zi) and q(β) = ∏V w=1 ∏T k=1 βwk be the fully factorized variational approximation of the true posterior P (z, β | ye, yf , xe, xf , α), where V is the size of english vocabulary, and T is the maximum number of senses for any word. The optimization problem solves for θ,q(z) and q(β) using the stochastic variational inference technique (Hoffman et al., 2013) similar to (Bartunov et al., 2016) (refer for details).
The resulting learning algorithm is shown as Algorithm 1. The first for-loop (line 1) updates the English sense vectors using the crosslingual and monolingual contexts. First, the expected sense distribution for the current English word w is computed using the current estimate of q(β) (line 4). The sense distribution is updated (line 7) using the combined monolingual and crosslingual contexts (line 5) and re-normalized (line 8). Using the updated sense distribution q(β)’s sufficient statistics is re-computed (line 9) and the global parameter θ is updated (line 10) as follows,
θ ← θ + ρt∇θ ∑
k|zik> ∑ y∈yc zik log p(y|xi, k, θ) (4)
Note that in the above sum, a sense participates in a update only if its probability exceeds a threshold (=0.001). The final model retains sense vectors whose sense probability exceeds the same threshold. The last for-loop (line 11) jointly optimizes the foreign embeddings using English context with the standard skip-gram updates.
Disambiguation. Similar to (Bartunov et al., 2016), we can disambiguate the sense for the word xe given a monolingual context ye as follows,
P (z | xe, ye) ∝ P (ye | xe, z; θ) ∑
β P (z | xe, β)q(β) (5)
Although the model trains embeddings using both monolingual and crosslingual context, we only use monolingual context at test time. We found that so long as the model has been trained with multilingual context, it performs well in sense disambiguation on new data even if it contains only monolingual context. A similar observation was made by (Šuster, Titov, and van Noord, 2016).
4 MULTILINGUAL EXTENSION
Bilingual distributional signal alone may not be sufficient as polysemy may survive translation in the second language. Unlike existing approaches, we can easily incorporate multilingual distributional
Algorithm 1 Psuedocode of Learning Algorithm
Input: parallel corpus E = {xe1, .., xei , .., xeNe} and F = {x f 1 , .., x f i , .., x f Nf } and alignments Ae→f and
Af→e, Hyper-parameters α and T , window sizes d, d′ . Output: θ, q(β), q(z)
1: for i = 1 to Ne do . update english vectors 2: w← xei 3: for k = 1 to T do 4: zik ← Eq(βw)[log p(zi = k|, x e i )] 5: yc← Nbr(xei ,E,d) ∪ Nbr(xfi ,F ,d ′) ∪ {xfi } where x f i = Ae→f (x e i ) 6: for y in yc do 7: SENSE-UPDATE(xei , y, zi) 8: Renormalize zi using softmax 9: Update suff. stats. for q(β) like (Bartunov et al., 2016) 10: Update θ using eq. (4) 11: for i = 1 to Nf do . jointly update foreign vectors 12: yc← Nbr(xfi ,F ,d) ∪ Nbr(x e i ,E,d
′) ∪ {xei} where xei = Af→e(xfi ) 13: for y in yc do 14: SKIP-GRAM-UPDATE(xfi , y) 15: procedure SENSE-UPDATE(xi, y, zi) 16: zik ← zik + log p(y|xi, k, θ)
signals in our model. For using languages l1 and l2 to learn multi-sense embeddings for English, we train on a concatenation of En-l1 parallel corpus with an En-l2 parallel corpus. This technique can easily be generalized to more than two foreign languages to obtain a large multilingual corpus.
Value of Ψ(ye, xf ). The factor modeling the dependence of the english context word ye on foreign word xf is crucial to performance when using multiple languages. Consider the case of using French and Spanish contexts to disambiguate the financial sense of the english word bank. In this case, the (financial) sense vector of bank will be used to predict vector of banco (Spanish context) and banque (French context). If vectors for banco and banque do not reside in the same space or are not close, the model will incorrectly assume they are different contexts to introduce a new sense for bank. This is precisely why the bilingual models, like that of (Šuster, Titov, and van Noord, 2016), cannot be extended to multilingual setting, as they pre-train the embeddings of second language before running the multi-sense embedding process. As a result of naive pre-training, the French and Spanish vectors of semantically similar pairs like (banco,banque) will lie in different spaces and need not be close. A similar reason holds for (Guo et al., 2014a), as they use a two step approach instead of joint learning.
To avoid this, the vector for pairs like banco and banque should lie in the same space and close to each other and the sense vector for bank. The Ψ(ye, xf ) term attempts to ensure this by using the vector for banco and banque to predict the vector of bank. This way, the model brings the embedding space for Spanish and French closer by using English as a bridge language during joint training. A similar idea of using English as a bridging language was used in the models proposed in (Hermann and Blunsom, 2014) and (Coulmance et al., 2015). Beside the benefit in the multilingual case, the Ψ(ye, xf ) term improves performance in the bilingual case as well, as it forces the English and second language embeddings to remain close in space.
To show the value of Ψ(ye, xf ) factor in our experiments, we ran a variant of Algorithm 1 without the Ψ(ye, xf ) factor, by only using monolingual neighborhood Nbr(xfi , F ) in line 12 of Algorithm 1. We call this variant ONE-SIDED model and the model in Algorithm 1 the FULL model.
5 EXPERIMENTAL SETUP
Parallel Corpora. We use parallel corpora in English (En), French (Fr), Spanish (Es), Russian (Ru) and Chinese (Zh) in our experiments. Corpus statistics for all datasets used in our experiments are shown in Table 1. For En-Zh, we use the FBIS parallel corpus (LDC2003E14). For En-Fr, we use the first 10M lines from the Giga-EnFr corpus released as part of the WMT shared task (CallisonBurch et al., 2011). Note that the domain from which parallel corpus has been derived can affect the final result. To understand what choice of languages provide suitable disambiguation signal,
it is necessary to control for domain in all parallel corpora. To this end, we also used the En-Fr, En-Es, En-Zh and En-Ru sections of the MultiUN parallel corpus (Eisele and Chen, 2010). Word alignments were generated using fast_align tool (Dyer, Chahuneau, and Smith, 2013) in the symmetric intersection mode. Tokenization and other preprocessing were performed using cdec 3 toolkit. Stanford Segmenter (Tseng et al., 2005) was used to preprocess the chinese corpora.
Word Sense Induction (WSI). We evaluate our approach on word sense induction task. In this task, we are given several sentences showing usages of the same word, and are required to cluster all sentences which use the same sense (Nasiruddin, 2013). The predicted clustering is then compared against a provided gold clustering. Note that WSI is a harder task than Word Sense Disambiguation (WSD)(Navigli, 2009), as unlike WSD, this task does not involve any supervision or explicit human knowledge about senses of words. We use the disambiguation approach in eq. (5) to predict the sense given the word and four context words.
To allow for fair comparison with earlier work, we use the same benchmark datasets as (Bartunov et al., 2016) – Semeval-2007, 2010 and Wikipedia Word Sense Induction (WWSI). We report Adjusted Rand Index (ARI) (Hubert and Arabie, 1985) in the experiments, as ARI is a more strict and precise metric than F-score and V-measure.
Parameter Tuning. For fairness, we used five context words on either side to update each English word-vectors in all the experiments. In the monolingual setting, all five words are English; in the multilingual settings, we used four neighboring English words plus the one foreign word aligned to the word being updated (d = 4, d′ = 0 in Algorithm 1). We also analyze effect of varying d′.
We tune the parameters α and T by maximizing the log-likelihood of a held out English text.4 The parameters were chosen from the following values α = {0.05, 0.1, .., 0.25}, T = {5, 10, .., 30}. All models were trained for 10 iteration with a decaying learning rate of 0.025, decayed to 0. Unless otherwise stated, all embeddings are 100 dimensional.
Under various choice of α and T , we identify only about 10-20% polysemous words in the vocabulary using monolingual training and 20-25% polysemous using multilingual training. It is evident using the non-parametric prior has led to substantially more efficient representation compared to previous methods with fixed number of senses per word.
6 EXPERIMENTAL RESULTS
We performed extensive experiments to evaluate the benefit of leveraging bilingual and multilingual information during training. We also analyze how the different choices of language family (i.e. using more distant vs more similar languages) affect performance of the embeddings.
Word Sense Induction Results. The results for WSI are shown in Table 2. MONO refers to the AdaGram model of (Bartunov et al., 2016) trained on the English side of the parallel corpus. In all cases, the MONO model is outperformed by ONE-SIDED and FULL models, showing the benefit of using crosslingual signal in training. Best performance is attained by the multilingual model (EnFrZh), showing value of multilingual signal. The value of Ψ(ye, xf ) term is also verified by the fact that the ONE-SIDED model performs worse than the FULL model.
We can also compare (unfairly to FULL model) to the best results described in (Bartunov et al., 2016), which achieved ARI scores of 0.069, 0.097 and 0.286 on the three datasets respectively after
3github.com/redpony/cdec 4first 100k lines from the En-Fr Europarl (Koehn, 2005)
training 300 dimensional embeddings on English Wikipedia (≈ 100M lines). Note that, as WWSI was derived from Wikipedia, training on Wikipedia gives AdaGram model an undue advantage, resulting in high ARI score on WWSI. Nevertheless, even in the unfair comparison, it noteworthy that on S-2007 and S-2010, we can achieve comparable performance (0.067 and 0.094) with multilingual training to a model trained on almost 5 times more data and higher (300) dimensional embeddings.
Contextual Word Similarity Results. For completeness, we report correlation scores on Stanford contextual word similarity dataset (SCWS) (Huang et al., 2012) in Table 2. The task requires computing similarity between two words given their contexts. While the bilingually trained model outperforms the monolingually trained model, surprisingly the multilingually trained model does not perform well on SCWS. We believe this may be due to our parameter tuning strategy.5
Effect of Language Family Distance. Intuitively, choice of language can affect the result from crosslingual training as some languages may provide better disambiguation signals than others. We performed a systematic set of experiment to evaluate whether we should choose languages from a closer family (Indo-European languages) or farther family (Non-Indo European Languages) as training data alongside English.6 To control for domain here we use the MultiUN corpus. We use En paired with Fr and Es as Indo-European languages, and English paired with Ru and Zh for representing Non-Indo-European languages.
From Table 3, we see that using Non-Indo European languages yield a slightly higher average improvement in WSI task than using Indo-European languages. This suggests that using languages from a distance family aids better disambiguation. Our findings echo those of (Resnik and Yarowsky, 1999), who found that the tendency to lexicalize different senses of an English word differently in a second language correlated with language distance.
Effect of Window Size. Figure 3d shows the effect of increasing the crosslingual window (d′) on the average ARI on the WSI task for the En-Fr and En-Zh models. While increasing the window size improves the average score for En-Zh model, the score for the En-Fr model goes down. This suggests that it might be beneficial to have a separate window parameter per language. This also
5Most works tune directly on the test dataset for Word Similarity tasks (Faruqui et al., 2016) 6 (Šuster, Titov, and van Noord, 2016) compared different languages but did not control for domain.
aligns with the observation earlier that different language families have different suitability (bigger crosslingual context from a distant family helped) and requirements for optimal performance.
Qualitative Illustration. As an illustration for the effects of multilingual training, Figure 3 shows PCA plots for 11 sense vectors for 9 words using monolingual, bilingual and multilingual models. From Fig 3a, we note that with monolingual training the senses are poorly separated. Although the model infers two senses for bank, the two senses of bank are close to financial terms, suggesting their distinction was not recognized. The same can be observed about apple. In Fig 3b, with bilingual training, the model infers two senses of bank correctly, and two sense of apple become more distant. The model can still improve eg. pulling interest towards the financial sense of bank, and pulling itunes towards apple 2. Finally, in Fig 3c, all senses of the words are more clearly clustered, improving over the clustering of Fig 3b. The senses of apple, interest, and bank are well separated, and are close to sense-specific words, showing the benefit of multilingual training.
7 CONCLUSION
We presented a multi-view, non-parametric word representation learning algorithm which can leverage multilingual distributional information. Our approach effectively combines the benefits of crosslingual training and Bayesian non-parametrics. Ours is the first multi-sense representation learning algorithm capable of using multilingual distributional information efficiently, by combining several parallel corpora to obtained a large multilingual corpus. Our experiments show how this multi-view approach learns high-quality embeddings using substantially less data and parameters than prior state-of-the-art. While we focused on improving the embedding of English words here, the same algorithm could learn better multi-sense embedding for Chinese, for instance. Exciting avenues for future research include extending our approach to model polysemy in foreign language. The sense vectors can then be aligned across languages (thanks to our joint training paradigm), to generate a multilingual Wordnet like resource, in a completely unsupervised manner.

In this paper, the authors propose a Bayesian variant of the skipgram model to learn word embeddings. There are two important variant compared to the original model. First, aligned sentences from multiple languages are used to train the model. Therefore, the context words of a given target word can be either from the same sentence, or from an aligned sentence in a different language. This allows to learn multilingual embedding. The second difference is that each word is represented by multiple vectors, one for each of its different senses. A latent variable z models which sense should be used, given the context.
Overall, I believe that the idea of using a probabilistic model to capture polysemy is an interesting idea. The model introduced in this paper is a nice generalization of the skipgram model in that direction. However, I found the paper a bit hard to follow. The formulation might probably be simplified (e.g. why not consider a target word w and a context c, where c is either in the source or target language. Since all factors are independent, this should not change the model much, and would make the presentation easier). The performance of all models reported in Table 2 & 3 seem pretty low.
Overall, I like the main idea of the paper, which is to represent word senses by latent variables in a probabilistic model. I feel that the method could be presented more clearly, which would make the paper much stronger. I also have some concerns regarding the experimental results.
Pros:
Interesting extension of skipgram to capture polysemy.
Cons:
The paper is not clearly written.
Results reported in the paper seems pretty low.