Posterior Concentration for Sparse Deep Learning
We introduce Spike-and-Slab Deep Learning (SS-DL), a fully Bayesian alternative to dropout for improving generalizability of deep ReLU networks. This new type of regularization enables provable recovery of smooth input-output maps with unknown levels of smoothness. Indeed, we show that the posterior distribution concentrates at the near minimax rate for α-Hölder smooth maps, performing as well as if we knew the smoothness level α ahead of time. Our result sheds light on architecture design for deep neural networks, namely the choice of depth, width and sparsity level. These network attributes typically depend on unknown smoothness in order to be optimal. We obviate this constraint with the fully Bayes construction. As an aside, we show that SS-DL does not overfit in the sense that the posterior concentrates on smaller networks with fewer (up to the optimal number of) nodes and links. Our results provide new theoretical justifications for deep ReLU networks from a Bayesian point of view.

This paper translates some theory of posterior concentration from Bayesian non-parametric regression to deep ReLU networks. This allows for the development of spike-and-slab priors on the weights of the hidden units that both provably concentrates around the true posterior distribution and ought to allow for automatic choice of hidden units. The main proof of posterior concentration follows related work in asymptotic statistics by considering the size of neighborhoods in \norm{.}_n, where n is the number of data points, and considering the limit of n-> \infty.  Originality: This seems to be the first paper to apply non-parametric regression techniques to prove posterior concentration properties for spike-and-slab priors on DNNs. This is relatively novel, even if the results are a natural extension of existing work on spike-and-slab variable selection.   Significance: The results seem to be a quite natural extension of existing non-parametric regression results; however, this could be a useful addition to the community as it informs potentially more interesting prior choices for researchers applying Bayesian methods to DNNs.   Clarity: Presentation wise, the paper seems relatively well structured; however, too much space is spent defining terms and introducing non-parametric regression concepts, resulting in too little space (1.5 pages) being devoted to the new results. Even less space is spent explaining the importance of these results. Additionally, discussion as to how these results can be applied for practitioners is quite lacking.   However, the exposition on posterior concentration and spike-and-slab priors is quite nice, well-written, and easy to follow. One suggestion is that the introduction section could be condensed quite a bit, with the relevant references being pushed into the next sections on deep learning and deep ReLU networks.   Quality: The proofs in the supplementary material do seem to follow logically, and the comments that I have are mostly minor. The work would be made much stronger by the inclusion of an experiments section showing how these DNNs could be trained.  Comments: - A tighter bound on function approximation for deep ReLU networks does exist than is given in Section 5.1, as [2] show that a network of depth k and width O(n/k) exists to approximate any function on n data points in p dimensions. Perhaps that argument could be adapted to include sparsity and thus show a tighter bound in Section 5.1, allowing fixed depth and layers, rather than the requirement that the number of layers grow as a function of data. - One primary weakness in this work is in the lack of experiments using the spike-and-slab theory. It would be very nice to see a) that the spike-and-slab priors discussed could be trained in practice, whether by some sort of Gibbs sampling, MCMC, or by a variant of standard SGD, and b) if an architecture could be adaptively trained using the prior on network width using Equation 18. - It is interesting that the composed deep network suggested by [1] (~180 parameters) performs so much better than the wide shallow network (~8000 parameters). Perhaps the authors can comment on if they would expect this result to hold in the limit of width as the single layer neural network converges to a Gaussian process? - The results seem to hinge on the true generative model, P_f0^n, being in the same class of functions as the model. Could the authors clarify if this is the case, and if not, what would happen to both sparsity and the posterior distribution? - Proof of Theorem 5.1: In Section 3, the authors point to the fact that they must verify all three conditions (Equations 11-13); however, in the proof, they only verify Equations 11 and 12. A quick sentence demonstrating why they do not need to verify Equation 13 would be quite helpful. - Corollary 5.1: In line 274, s_n = n \epsilon_n^2, which has been assumed to approach infinity in the limit (line 183). Thus, it seems like the corollary is just showing that the sparsity level is finite (and thus the number of non-zero parameters is finite), when it should be finite by construction of the neural network. Could the authors comment on this interpretation?  Updates after Author Responses: I thank the authors for addressing my mathematical comments, questions about implementations, and comments about paper structuring. These resolve many of my concerns with the paper.   Minor Comments/Typos: Line 69: Similarly as dropout, -> Similarly to dropout Line 79: Section 3 defined … -> Section 3 defines (rest of paragraph is in present tense) Equation 7: A verification that this holds for all L>1, p>1 and N>1 would be nice. It is possible to find nonsensical parameter settings that do not satisfy this inequality under those conditions, but I could not find a counter-example to the inequality. Line 168: Potentially (T \choose s)^{-1} instead of 1/ (T \choose s) for clarity. Also applies to other locations. Line 178: The posterior distribution should also include dependence on X (the observed features) and not just Y (the observed classes). Line 201: A citation for “it is commonly agreed …” would be quite helpful. Line 297: We proof… -> We prove… Line 297: There seems to be a missing \Epsilon. log N (…) -> log N \Epsilon (…) Line 307: Where does \epsilon_n/2 come from? It seems like the triangle inequality would give \epsilon_n instead.  References: [1] Mhaskar, H., Liao, Q., and Poggio, T. A. When and why are deep networks better than shallow ones? AAAI, 2017. [2] Zhang, C., et al. Understanding Deep Learning Requires Rethinking Generalization, ICLR, 2017. (see particularly Theorem 1 and Lemma 1 in Appendix C.)


Variational Memory Encoder-Decoder
Introducing variability while maintaining coherence is a core task in learning to generate utterances in conversation. Standard neural encoder-decoder models and their extensions using conditional variational autoencoder often result in either trivial or digressive responses. To overcome this, we explore a novel approach that injects variability into neural encoder-decoder via the use of external memory as a mixture model, namely Variational Memory Encoder-Decoder (VMED). By associating each memory read with a mode in the latent mixture distribution at each timestep, our model can capture the variability observed in sequential data such as natural conversations. We empirically compare the proposed model against other recent approaches on various conversational datasets. The results show that VMED consistently achieves significant improvement over others in both metricbased and qualitative evaluations.

Summary This paper introduces a new model for natural conversation modelling, the Variational Memory Encoder-Decoder (VMED). The purpose of the model is to learn a distribution over latent variables that captures time dependence and has different modes corresponding to for instance different moods and intentions during the conversation. In order to ensure coherence over time and variability of the generated conversations, an external memory is used as a mixture of Gaussians prior over the latent space, which is updated during the conversation. VMED outperforms all baselines in four conversational datasets.   Originality As far as I understand the building blocks for this model include: 1. A Gaussian mixture model for the prior in the VAE. 2. The use of an external memory  [5,35] to produce the parameters of the Gaussian mixture prior, which is updated at each time step, especially useful for coherence and multimodality in language tasks. 3. A probabilistic framework for sequence generation. [8], in the form of a conditional generative model [17, 38]. To my knowledge, all of these ideas are joined for the first time for sequence generation.  Quality Besides being well motivated, the paper appears technically sound. The evaluation of the model as a whole is strong. The baselines are outperformed by the proposed model.  However, it is hard to understand the influence of the different components on the increased performance over the baselines: i) the use for a multimodal prior distribution for a conditional VAE, ii) an external memory that adapts this distribution over time. It would provide useful insights to show experiments on this, for instance by comparing against a CVAE without an external memory but with a Gaussian mixture model prior.  The details of the architectures used are limited.  Clarity Overall this paper is written in an excellent manner. The formation of ideas is clear and the embedding in relevant literature is good. The figures are helpful in understanding the method, although I would encourage the authors to extend the captions to be more standalone.                          Significance The approach seems novel and relevant for language modelling to me. Since the main application in this paper is conversation modeling, I wonder what modeling choices need to be adapted in order to perform well on other sequence generation tasks, such as translation. Can the authors comment on this?  Detailed comments/questions  1. In Eq. 2, There is a p missing inside the logarithm corresponding to the reconstruction term of the lower bound.  2. Should there be a solid arrow from x to z in figure 1? 3. Can the authors motivate why you choose to use a Gaussian mixture model as a prior instead of using for instance flows to create a multimodal distribution? 4. In Eq. 5, the minus sign in front of the sum should only be in front of the KL term.  5. In Eq. 6, the sum over i=1 to K is outside the log, whereas in the definition above for Dvar it is inside the log. Furthermore, it seems that z^l_t should be z^l_{<=t} in the last term. It would be good to explicitly mention that the last term is an MC estimate with samples from the posterior. Does r_{t-1} depends on z_


Clustering Redemption-Beyond the Impossibility of Kleinberg's Axioms
Kleinberg [20] stated three axioms that any clustering procedure should satisfy and showed there is no clustering procedure that simultaneously satisfies all three. One of these, called the consistency axiom, requires that when the data is modified in a helpful way, i.e. if points in the same cluster are made more similar and those in different ones made less similar, the algorithm should output the same clustering. To circumvent this impossibility result, research has focused on considering clustering procedures that have a clustering quality measure (or a cost) and showing that a modification of Kleinberg's axioms that takes cost into account lead to feasible clustering procedures. In this work, we take a different approach, based on the observation that the consistency axiom fails to be satisfied when the "correct" number of clusters changes. We modify this axiom by making use of cost functions to determine the correct number of clusters, and require that consistency holds only if the number of clusters remains unchanged. We show that single linkage satisfies the modified axioms, and if the input is well-clusterable, some popular procedures such as k-means also satisfy the axioms, taking a step towards explaining the success of these objective functions for guiding the design of algorithms.

This paper extends Kleinberg’s work on the impossibility of clustering. That is, Kleinberg introduced 3 axioms that any clustering procedure should satisfy and then showed that it is impossible to satisfy all three simultaneously.  This work suggests a refinement of Kleinberg’s 3rd axiom having to do with consistency.  In the original axiom, if the clustering distance function is perturbed such that all within-cluster distances do not increase and all across cluster distances do not decrease, then the optimal clustering should be the same with respect to both the perturbed and unperturbed distance functions. The refined consistency axiom proposed in this work says that under the perturbed distance function, the optimal clustering may be different so long as the number of clusters in the optimal partition is different as well. The authors define the optimal number of clusters using the commonly used elbow heuristic. The remainder of the paper analyzes prevalent clustering algorithms and objectives like single linkage and k-center variants and shows under what conditions they do or do not satisfy Kleinberg’s first two axioms and refined consistency.  The work has several strengths. The work positively contributes to the line of work on the impossibility of clustering, which helps to explain why clustering algorithms are effective in practice even when they are shown to be difficult to optimize--or impossible to optimize--in practice. The main new axiomatic ideas are simple, intuitive and crisp.  For the most part, the writing makes the high-level concepts easy to understand. The theoretical results contribute to our understanding of common clustering objectives and are an important research thread to pursue. I think that researchers interested in a theoretical understanding of clustering will be interested in this work.  The main issues with this paper seem to be easily corrected. In the proof of theorem 4.1 for the case of k-center, the authors claim that the optimal cost for k=1 cluster is 2\gamma - 4\epsilon, which they claim is achieved by opening a center at u_3, but I think this is incorrect; to me it seems that opening a center at u_3 incurs a cost of 2\gamma - 3\epsilon + 1. A better cost is achieved by opening a center at u_2 which incurs a cost of of 2\gamma - 4\epsilon (like what is claimed).  Also given that in the example gamma == 1.5 and epsilon == 0.1, the final line in the proof, which claims that \gamma + epsilon <= 1.5 is incorrect.  Perhaps what is meant is \gamma - \epsilon <= 1.5? Or perhaps that inequality should be eliminated?  The writing of the proof of theorem 4.2 should be simplified.  One thing that might help is a table showing what the various clusterings are, i.e., C1 = …, C1’ = …, C1’’ = …., C2 = …, etc. and their corresponding costs under the original and perturbed instances.  Again, I think there is an issue in this proof: on line 341 there is a claim that OPT^o_2 < OPT^p(C2), which I think is incorrect based on the values of x and y given. However, this incorrect claim doesn’t invalidate the proof.  Like in section 4, I think that the work would be strengthened by visual aids for a handful of the proofs in the supplement.  There are a handful of grammatical errors and notational sloppiness in this paper which make understanding the technical arguments much more cumbersome.  For example: in the first paragraph of 3.1 the optimal k is defined differently than the optimal k defined before axiom 2.4; on line 237 the authors write k \in {1, …, n-1} but then say that k cannot be 1; I think that k* and k^\star refer to the same thing--be consistent; after Theorem 3.4, the link to Axiom 2.3 should actually link to Axiom 2.4 (line 259); on line 274, it seems like \Gamma refers to a partition of the instances rather than a cost function--as it did in the beginning of the work; the supplement refers to Theorem 3.5 but I think this should be Lemma 3.5. There are other examples as well.  There are some grammatical errors as well; for example, in the paragraph starting on line 138.  Be sure to correct all of these mistakes.   The introduction makes it seem like the theorems presented will be in terms of stable clusterings. Clearly connecting stability to \alpha-center proximity will improve the presentation. There is no conclusion. While I think it would improve the paper to include a conclusion--even if very short--I don’t find it absolutely necessary.  As mentioned above, all of these issues can be corrected via a more diligent revision.


Learning Concave Conditional Likelihood Models for Improved Analysis of Tandem Mass Spectra
The most widely used technology to identify the proteins present in a complex biological sample is tandem mass spectrometry, which quickly produces a large collection of spectra representative of the peptides (i.e., protein subsequences) present in the original sample. In this work, we greatly expand the parameter learning capabilities of a dynamic Bayesian network (DBN) peptide-scoring algorithm, Didea [25], by deriving emission distributions for which its conditional log-likelihood scoring function remains concave. We show that this class of emission distributions, called Convex Virtual Emissions (CVEs), naturally generalizes the log-sum-exp function while rendering both maximum likelihood estimation and conditional maximum likelihood estimation concave for a wide range of Bayesian networks. Utilizing CVEs in Didea allows efficient learning of a large number of parameters while ensuring global convergence, in stark contrast to Didea's previous parameter learning framework (which could only learn a single parameter using a costly grid search) and other trainable models [12,13,14] (which only ensure convergence to local optima). The newly trained scoring function substantially outperforms the state-of-the-art in both scoring function accuracy and downstream Fisher kernel analysis. Furthermore, we significantly improve Didea's runtime performance through successive optimizations to its message passing schedule and derive explicit connections between Didea's new concave score and related MS/MS scoring functions.

The manuscript “Learning Concave Conditional Likelihood Models for Improved Analysis of Tandem Mass Spectra” extends a dynamic Bayesian network approach called DIDEA by introducing a new class of emission distributions. The conditional log-likelihood of those functions remains concave leading to an efficient global optimization method for parameter estimation. This is in stark contrast to the previous variant, for which the best parameter had to be found by grid search. In comparison to other state-of-the-art methods, the new approach outperforms the other methods, while being faster at the same time.  Quality Overall the quality of the manuscript is good. There were some errors in the formulas, but the authors promised to fix them at the author response period.  Clarity Overall the clarity is good, there are just some parts, where the reader has to guess certain details (e.g., regarding b and y). It is a little bit confusing that in the introduction it is mentioned that DRIP outperformed DIDEA (line 39), but then in the comparison on the data sets where the authors show the performance of the old DIDEA approach (Figure 4, Didea-0), the Didea-0 performance seems to be superior to the performance of DRIP (shown in Figure 3). It would have been better to show Didea-0 also in Figure 3. Showing this separately does not make a good impression especially in light of the above mentioned observation. Figure 4 also seems to imply that the performance gain is not as substantial as claimed in line 281.  Author response: ...Please see the introductory DRIP paper [1] being referred to on line 39, where the authors show that DRIP outperforms Didea... We note, though, that the authors of [1] only test algorithms by assuming a fixed observed precursor charge of 2, rather than the harder charge-varying case considered in this paper.  answer: But then you are saying now that in the charge-varying case DRIP is worse than the old DIDEA. One should at least discuss this and not just copy an old statement from a paper with a non-extensive evaluation.   It is good to see a statement about the runtime in the manuscript (with actual values) and not just in the supplement. This seems to be the main selling point in case the performance of Didea and Didea-0 is really that close.  Significance The introduced distributions, called convex virtual emissions (CVEs), seem interesting and might have applicability beyond DBNs, but the focus of the manuscript is only on this one application of DBNs with CVEs for Tandem Mass Spectra analysis. If the authors could make plausible that CVEs could be useful for other models, then this would be of even higher interest to the general NIPS community.


Preference Based Adaptation for Learning Objectives
In many real-world learning tasks, it is hard to directly optimize the true performance measures, meanwhile choosing the right surrogate objectives is also difficult. Under this situation, it is desirable to incorporate an optimization of objective process into the learning loop based on weak modeling of the relationship between the true measure and the objective. In this work, we discuss the task of objective adaptation, in which the learner iteratively adapts the learning objective to the underlying true objective based on the preference feedback from an oracle. We show that when the objective can be linearly parameterized, this preference based learning problem can be solved by utilizing the dueling bandit model. A novel sampling based algorithm DL 2 M is proposed to learn the optimal parameter, which enjoys strong theoretical guarantees and efficient empirical performance. To avoid learning a hypothesis from scratch after each objective function update, a boosting based hypothesis adaptation approach is proposed to efficiently adapt any pre-learned element hypotheses to the current objective. We apply the overall approach to multi-label learning, and show that the proposed approach achieves significant performance under various multi-label performance measures.

The paper discusses a problem of adapting an optimization objective to a complex test objective. A complex test objective can either be hard to define (revenue, attrition, etc.) or hard to optimize directly (e.g. ranking losses), and it may not be clear which optimization objective (continuous, possibly convex) should be used. The paper proposes an approach that jointly optimizes the parameter according to an optimization objective and adapts the optimization objective to the complex test objective.  The main idea is to consider multiple optimization objectives and find the best linear combination of them. The solution has two main components. The first assumes black box access to a solver of the optimization task and finds the linear combination of the different objectives best suited to the test objective. The second solves each of the optimization objectives efficiently given the previous solutions. The first component assumes oracle access to the test loss, and build a solution via the multi-armed bandit technology. The second component adapts methods from multi-task learning to solve the particular problem at hand.  Novelty / Impact: The problem proposed is very common, and I am surprised it did not receive more attention. The authors bring an educated solution to this problem, and this type of approach is likely to be proven useful in many real-life problems.   The solution proposed seems somewhat raw. The main idea of having two components for (1) tuning the optimization objective (2) efficiently solving each optimization problem by utilizing previous solutions feels like the right thing to do. The details probably have room for improvement (see details below). That being said, the problem doesn’t have to be completely solved by a single paper, and I think the impact of the paper is still significant based on the importance of the problem and the clean high level solution  Writing / Clarity: There are some parts of the paper that could be improved. There are many definitions that are not here, but can only be obtained by reading the cited papers. The motivation, at least for algorithm 2 can be better explained, especially given that the algorithm does not come with any guarantees. Furthermore, if one would like to implement algorithm 2 based on the paper, they would have a very hard time doing so. The authors should make a considerable effort to explain exactly how the algorithm works.  Experiments: The experiments contain one experiment measuring the bandit algorithm (component 1) and another for the full system.   In the section discussing the full system, the authors use the test set for measuring the test objective in the bandit algorithm. Thought claiming it does not leak information, it does, exactly as in the case of hyper parameter optimization. A cleaner experiment could have been done with a validation set used to tune the optimization objective.   Another part that seems to be missing is an experiment measuring the effectiveness of the 2nd component without the first. For example, given the solutions to weights w_1,..,w_n, apply algorithm 2 to solve a new objective, and compare its performance to a solution obtained directly. Given that algorithm 2 doesn’t come with guarantees, that experiment would be more useful than experiments showing that algorithm 1 works.  Overall the experiments paint an OK picture but can be made much more convincing.   Summary: The paper tackles an important problem, its high level solution is clean and elegant, but the details leave room for improvement. The experiments are probably sufficient but not much more. Still, everything considered the fixes are doable within the time for the camera ready version and the advantages of the paper out-weigh its disadvantages.  Comments: * For algorithm 1, the guarantee is for the regret, but the objective is clearly simple regret, i.e. identifying the best linear function. The authors should consider either presenting a different proof, or even adapting the algorithm with that in mind. In particular, the discussion about the UCB approach not working is not surprising as UCB may not work well for best arm identification.   * In line 151 there is an explanation of how to adapt from the unit ball to positive numbers. A naive implementation of what is written may take time exponential in the dimension (drawing a random vector until all its components are positive). Please provide a more detailed explanation of how this can be done efficiently. Generally the explanation in the paragraph about the implementation is very vague and short. Perhaps add a detailed section in the appendix.  * line 163. The parameter l^i seems to come out of nowhere * line 214: the parameter \kappa in alg. 1 is a matrix, but in the experiments it’s fixed as a constant number. Please fix this issue. Also, compare the method to a parameter kappa that is set according to what the algorithm dictates, and not just a constant. The choice of setting it as a constant is not clear, and is even shown to be wrong by the experiments.  * Equation (13): Please define the the paper all the optimization objectives * Table 1: Please bold the best experiments in each setting


cpSGD: Communication-efficient and differentially-private distributed SGD
Distributed stochastic gradient descent is an important subroutine in distributed learning. A setting of particular interest is when the clients are mobile devices, where two important concerns are communication efficiency and the privacy of the clients. Several recent works have focused on reducing the communication cost or introducing privacy guarantees, but none of the proposed communication efficient methods are known to be privacy preserving and none of the known privacy mechanisms are known to be communication efficient. To this end, we study algorithms that achieve both communication efficiency and differential privacy. For d variables and n ⇡ d clients, the proposed method uses O(log log(nd)) bits of communication per client per coordinate and ensures constant privacy. We also improve previous analysis of the Binomial mechanism showing that it achieves nearly the same utility as the Gaussian mechanism, while requiring fewer representation bits, which can be of independent interest.

--Post rebuttal:  The authors' answered my questions satisfactorily, and I raised my score by 1 to 8. I think this is a strong paper. I look forward to reading an updated paper that improves some of the writing and presentation.  Summary: This paper seeks to combine two aspects of distributed optimization: (i) quantization to reduce the total communication cost and (ii) privacy through an application of differential privacy. The authors propose to have each of the worker nodes add noise to its gradients before sending the result to the central aggregator node.   The challenge in this case is that adding Gaussian noise, as is usual for differential privacy mechanisms, means that the resulting gradients have infinite resolution, so they cannot easily be compressed. Instead, the authors use adding a binomial RV. They analyze this particular differential privacy mechanism. Finally, since the aggregation step requires the central node to estimate the mean, the authors study the error of distributed mean estimation.  This is a heavy theory paper. As far as I know, this is the first result of this type, which could be very useful in the federated optimization setting. The results are strong and the theory is a welcome building block for future works. The paper isn't particularly written or easy to understand, with many un-interpreted expressions, but this doesn't take away from the overall contribution.   Major comments:  - This is a theory paper, so it's important to explain each of the theoretical results. The huge formula in (6) in Theorem 1 isn't very interpretable as it stands. Which one of those three terms is typically dominant? Which of the three sensitivity parameters are typically important for problems of interest? Under what conditions is the resulting epsilon not meaningful (i.e., super huge)?   - The role of the scaling term s is also a little bit unclear. It must be sufficiently small (or else the noise magnitudes could be huge, huge enough to be past the representation limit for floats), but if it's too small, the noise may not be able to be distinguishable by floats either. Is there a heuristic rule for getting the right s?  - The paper's writing is sometimes pretty rough. Please add some signposting so we know what to expect and in which section.   Minor comments:  - Line 20: it's not necessarily true that "without a central server, each client saves a global model, broadcasts the gradient to all other clients". It's fine to have each client send its gradient to a potentially small neighborhood of clients (and receive a gradient from each of them). This is the standard setting for decentralized GD and related algorithms, which don't require a complete graph for the topology. There is a whole literature that shows how the graph topology impacts convergence, etc. See Yuan, Lin, and Yin, "On the Convergence of Decentralized Gradient Descent" for a discussion.  - What's the reason why you want to use cases other than p=1/2 for the binomial mechanism?  - One of the listed improvements is that N >= 8 log (2/delta)/epsilon^2 samples is enough... where does this come out?


MixLasso: Generalized Mixed Regression via Convex Atomic-Norm Regularization
We consider a generalization of mixed regression where the response is an additive combination of several mixture components. Standard mixed regression is a special case where each response is generated from exactly one component. Typical approaches to the mixture regression problem employ local search methods such as Expectation Maximization (EM) that are prone to spurious local optima. On the other hand, a number of recent theoretically-motivated Tensor-based methods either have high sample complexity, or require the knowledge of the input distribution, which is not available in most of practical situations. In this work, we study a novel convex estimator MixLasso for the estimation of generalized mixed regression, based on an atomic norm specifically constructed to regularize the number of mixture components. Our algorithm gives a risk bound that trades off between prediction accuracy and model sparsity without imposing stringent assumptions on the input/output distribution, and can be easily adapted to the case of non-linear functions. In our numerical experiments on mixtures of linear as well as nonlinear regressions, the proposed method yields high-quality solutions in a wider range of settings than existing approaches.

This paper considers a generalized version of the mixed regression problem, where we observe a collection of input-output samples, with the output corresponding to an additive combination of several mixture components/functions, and the goal is to find a collection of K functions that minimise the risk.   The corresponding ERM problem is NP-Hard to solve due to combinatorial constraints. The authors propose to relax these constraints by replacing them with an atomic norm regularizer they introduce as an "approximation" of the number of components. They propose to solve the resulting convex problem using a greedy algorithm.  Their analysis show that the solutions obtained by their approach achieve epsilon-optimal risk using a linear number of samples (both in terms of K and the dimension D) and O(K/epsilon) number of components, thus improving over the state-of-the-art in terms of number of sample complexity. The numerical experiments confirm this improvement.   The paper is written in a fairly clear way, and the approach of solving a convex relaxation of the mixed regression problem and the obtained sample complexity are compelling contributions. However, the introduced greedy algorithm (Algo. 1) seems unnecessarily computationally expensive!  In particular,  step 3 of  Algorithm 1 requires minimizing the loss function (7) over the current active set. The authors propose to solve this using proximal gradient algorithm (eq. 9), which requires solving a least squares problem at each iteration. Moreover, the convergence rate provided in Theorem 1 assumes that step (3) is solved exactly, which is not possible. Note also that the obtained convergence rate does not guarantee eps-optimal solution: F(c^T) - F(c^*) <= max{ O( ||c^*||_1 ), O(||c^*||^2_1 / T)}.   Why not solve instead the constrained formulation min g(M) s.t. ||M||_S <= K? This can simply be solved by Frank-Wolfe algorithm which only requires step 1 of Algorithm 1, and has a known convergence rate of O(1/T).   I think the paper could be improved by replacing Algorithm 1 with Frank-Wolfe and modifying the analysis in Theorem 2 accordingly.  Update: I thank the authors for their responses to my questions. Please note that Frank-Wolfe variants for the regularized form were also considered in the literature. For example, the paper "Conditional Gradient Algorithms for Norm-Regularized Smooth Convex Optimization" by Z. Harchaoui, A. Juditsky, and A. Nemirovski, provides a variant of Frank-Wolfe, which precisely address problems regularized with a norm, and is more computationally efficient than the proposed Algorithm 1. It is worth mentioning such algorithms in the paper, with an explanation (if any) on why Algorithm 1 might still be preferable in this case (for ex because it leads to easier analysis).


Processing of missing data by neural networks MarekŚmieja
We propose a general, theoretically justified mechanism for processing missing data by neural networks. Our idea is to replace typical neuron's response in the first hidden layer by its expected value. This approach can be applied for various types of networks at minimal cost in their modification. Moreover, in contrast to recent approaches, it does not require complete data for training. Experimental results performed on different types of architectures show that our method gives better results than typical imputation strategies and other methods dedicated for incomplete data.

The paper provides a theoretical and practical justification on using a density function to represent missing data while training a neural networks. An obvious upside is that training can be done with incomplete data, unlike denoising autoencoder for example; this can be very helpful in many applications. My comments are: - It is stated that if all the attributes are complete then the density is not used; if we have access to a huge amount of complete training data and relatively small amount of training missing data, how trustworthy is our estimation of density function? Can’t we benefit from the complete data? Do we really have to remove attributes as is done in ESR task? - In the above case, would denoising autoencoder outperform? - How would the generalized activation impact the training time? Any observation or input in this regard can be helpful to better assess the usefulness of this method. - With regards to the experiments, It would have been much better if some experiments were carried out to compare the proposed method with one of [2,3,4] when complete data is available for training. Also, some results showing performance vs amount of missing data (e.g. 75%, 50% and  25%) could give a helpful insight. -       Please also look at the following paper and see if a fair comparison can be made. GAIN: Missing Data Imputation using Generative Adversarial Nets - The second paragraph in introduction : ‘be understand’ -> be understood - Section 4 first paragraph: ‘lose’ -> loss -  In section 3, please define N. -        There are some old and nice works on dealing with NN and missing data that could be cited.  ###  Thanks for the reply. I am happy with the responses. With regard to the GAIN paper, what I meant was to try to conceptually compare your method with that and discuss the differences or similarities.


Bayesian multi-domain learning for cancer subtype discovery from next-generation sequencing count data
Precision medicine aims for personalized prognosis and therapeutics by utilizing recent genome-scale high-throughput profiling techniques, including next-generation sequencing (NGS). However, translating NGS data faces several challenges. First, NGS count data are often overdispersed, requiring appropriate modeling. Second, compared to the number of involved molecules and system complexity, the number of available samples for studying complex disease, such as cancer, is often limited, especially considering disease heterogeneity. The key question is whether we may integrate available data from all different sources or domains to achieve reproducible disease prognosis based on NGS count data. In this paper, we develop a Bayesian Multi-Domain Learning (BMDL) model that derives domain-dependent latent representations of overdispersed count data based on hierarchical negative binomial factorization for accurate cancer subtyping even if the number of samples for a specific cancer type is small. Experimental results from both our simulated and NGS datasets from The Cancer Genome Atlas (TCGA) demonstrate the promising potential of BMDL for effective multi-domain learning without "negative transfer" effects often seen in existing multi-task learning and transfer learning methods.

In this manuscript the authors introduce Bayesian Multi-Domain Learning (BMDL), a Bayesian model for learning latent spaces across related domains with high predictive power. The model’s sparsity structure allows some of the latent dimensions to be domain specific and others to be shared across domains and the authors derive an efficient blocked-gibbs sampler that allows for a negative binomial likelihood for NGS count data by using a data augmentation strategy. The authors apply BMDL to both synthetic data and real-world data, where they demonstrate their method on related domains (different RNA seq versions as an example of closely related domains, and different cancer types as an example of unrelated domains).   Overall, I think the model the authors propose is elegant and the problem they address (integrating multiple ‘omics datasets to improve predictive power) is important in a field with traditionally low predictive accuracy. However, I think the authors need to further convince the reader that their method is necessary over basic supervised machine learning techniques. The baseline model of a linear SVM on raw counts is inadequate: an implementation that appropriately normalizes (e.g. TPM) and scales (N(0,1)) the counts across a variety of techniques such as regularized logistic regression or even neural nets (see e.g. http://beamandrew.github.io/deeplearning/2017/06/04/deep_learning_works.html) could be used to get a true baseline.   It was difficult to fully evaluate this paper as the supplementary material submitted was identical to the main paper (I assume by mistake) - I would ask the authors submit the actual supplement as part of the rebuttal.   Further comments below -  The authors state “Traditional machine learning methods do not directly apply here as their fundamental assumption is that samples (training and testing) are from an identical probability distribution” - this is not addressed in the paper  - the authors “calculate a latent factor score for every sample in both the training and testing sets” - these come from same distribution for both training and test set so I do not see how the assertion in the introduction is addressed? (nor indeed how you could do any inference without making similarity assumptions about the training and test set)  In paragraph starting line 46 it would help if in the context of the problem the authors give examples of the domains they are targeting (RNA-seq technology version and cancer types)  Line 73 the authors say they exploit a “novel” data augmentation strategy for negative binomial distributions - is this the referenced [Zhou and Carin] (in which case it is not novel) or referencing the approximation scheme on line 183 for drawing from CRTs for large N ?  On line 127 it would help if the authors clarified what they mean by “cancer subtyping” - ultimately they use it to predict different cancer types based on gene expression, but it can mean other things in this context (e.g. identifying clusters of patients with high/low survival, etc).   In equation 2 the Gamma distribution has as its shape the product of s with a binary latent variable z - however, this can obviously lead to a Gamma with shape = 0 for which the density is undefined. It seems like the correct form for \theta would be a mixture of a point mass at 0 and a gamma distribution Gamma(s, 1/c) where the mixture component is controlled by z?  Line 223 d_t should be defined as the target domain when it is introduced  Line 277 - “1000 out of the top 5000 genes with higher log2 fc between LUAD and LUSC have been selected for analysis.” How were the 1000 out of the top 5000 selected? Importantly, this invalidates the accuracy claims in table 1 - the entire dataset has been used for feature selection rather than appropriately splitting the data into training and test (& validation) sets. While this has been uniformly applied across all algorithms and it appears BMDL has higher accuracy, if the point is to predict cancer subtypes based on gene expression then this is not proven.   Line 280 - what are the random SVM runs?  Line 286 - “From the table, BMDL clearly outperforms all the methods...more critically, BMDL consistently achieves better cancer subtyping.” What is the difference between these two? Isn’t the performance the subtyping?


Constructing Fast Network through Deconstruction of Convolution
Convolutional neural networks have achieved great success in various vision tasks; however, they incur heavy resource costs. By using deeper and wider networks, network accuracy can be improved rapidly. However, in an environment with limited resources (e.g., mobile applications), heavy networks may not be usable. This study shows that naive convolution can be deconstructed into a shift operation and pointwise convolution. To cope with various convolutions, we propose a new shift operation called active shift layer (ASL) that formulates the amount of shift as a learnable function with shift parameters. This new layer can be optimized end-to-end through backpropagation and it can provide optimal shift values. Finally, we apply this layer to a light and fast network that surpasses existing state-of-the-art networks.

In this paper, the authors propose a network compression framework by decomposing conventional convolution operator into a depth-wise learnable shifting operation, followed by 1x1 convolution. Intuitively, this work can be approximately seen as an improvement over MobileNet by replacing the depth-wise convolution into shift operations from [23]/[11]. The idea of learning continuous displacement in shift kernels also shares certain similar spirit to deformable convolution.  Pros of the paper: The addressed problem is clearly of importance to the machine learning community and practical applications. The proposed work is technically correct and is clearly presented. The whole paper is easy to follow. The presented method achieved promising, state of the art results in the experiments.  Cons of the paper: The novelty of this paper is somewhat weakened given the existence of previous related-works (ShiftNet/Active Convolution/MobileNet). In particular, the proposed work can be regarded as a combination of 1x1 convolution (can be found in MobileNet) and shift operations (similar ideas can be found in ShiftNet/Active Convolution). However, the authors provided some insights on the deconstruction of convolution in Section 3.1, which makes sense and justified the intuition of this combination.  A concern is that, since FLOP may not be an accurate indicator of the inference speed, the final speed performance can depend largely on the practical network architecture/implementation which is case-by-case. The fact that the network architectures are not unified between different comparing methods leaves the space for one to empirically pick a configuration that favors the proposed compression method. An example is what the authors mentioned in lines 188-189: "We noted that increasing depth caused an increase in inference time; therefore, a shallower network is preferred." As a result, it is not easy to find whether the inference speed gain mostly comes from the selected implementation/architecture or from the proposed method itself. (Of course, the same problem applies to previous methods as well.)  ======================================================================================== Update: I have read both the rebuttal and the other reviews. I decide to increase the score given the authors' additional results in response to R1. However, although the authors responded to my questions in the rebuttal, the answer somewhat dodged my concern without directly addressing it. In particular, the additional results from DW-ResNet proves that the proposed ASL module is better than MobileNet's depth-wise conv + 1x1 conv under current architecture. Since the authors' have specifically chosen the network architecture to be shallow and wide, whether the inference speed gain from the proposed ASL module is largely conditioned on the choice network architectures remains a question. It will be better if the authors can additionally include results on the comparison with ShiftNet/MobileNet under their proposed architectures (Say ASNet-110 vs ShiftNet-110). This will help people better understand the limits or properties of the proposed ASL module.


Gradient Descent for Spiking Neural Networks
Most large-scale network models use neurons with static nonlinearities that produce analog output, despite the fact that information processing in the brain is predominantly carried out by dynamic neurons that produce discrete pulses called spikes. Research in spike-based computation has been impeded by the lack of efficient supervised learning algorithm for spiking neural networks. Here, we present a gradient descent method for optimizing spiking network models by introducing a differentiable formulation of spiking dynamics and deriving the exact gradient calculation. For demonstration, we trained recurrent spiking networks on two dynamic tasks: one that requires optimizing fast (≈ millisecond) spike-based interactions for efficient encoding of information, and a delayed-memory task over extended duration (≈ second). The results show that the gradient descent approach indeed optimizes networks dynamics on the time scale of individual spikes as well as on behavioral time scales. In conclusion, our method yields a general purpose supervised learning algorithm for spiking neural networks, which can facilitate further investigations on spike-based computations.

This paper introduces a smooth thresholding technique which enables practically standard gradient descent optimization to be applied to spiking neural networks. Since the spiking threshold is usually set at a certain membrane potential, the function "spike or no spike" is a function of voltage whose distributional derivative is a dirac Delta at the threshold. By replacing this Dirac delta by a finite positive function g(v) with tight support around the threshold, and which integrates to 1, the step function "spike or no spike" is replaced by a function that increases continuously from 0 to 1 across the support of g. In turn, this setup can be placed into standard differential equation models governing spikes, while retaining the possibility of having meaningful gradient signal for parameter optimization.  Two experiments are evaluated, an autoencoding task and a delayed-memory-XOR task, which are both shown to be trainable with the proposed setup.  This is a good contribution which removes some previously required restrictions for training spiking neuron models (which are enumerated in the introduction).  An analysis that is missing and which imposes itself quite naturally is the behavior of the training as a function of support size/smoothness of g. One could for example evaluate g_a(v) = a g(a(v - v0) + v0) with g starting as very smooth and becoming more pointy as a increases. At some point training should become impossible due to convergence of this method to standard thresholding (+ numerical issues). Next, one could also evaluate making this function continuously sharper during training.


A Simple Proximal Stochastic Gradient Method for Nonsmooth Nonconvex Optimization
We analyze stochastic gradient algorithms for optimizing nonconvex, nonsmooth finite-sum problems. In particular, the objective function is given by the summation of a differentiable (possibly nonconvex) component, together with a possibly nondifferentiable but convex component. We propose a proximal stochastic gradient algorithm based on variance reduction, called ProxSVRG+. Our main contribution lies in the analysis of ProxSVRG+. It recovers several existing convergence results and improves/generalizes them (in terms of the number of stochastic gradient oracle calls and proximal oracle calls). In particular, ProxSVRG+ generalizes the best results given by the SCSG algorithm, recently proposed by [Lei et al., 2017] for the smooth nonconvex case. ProxSVRG+ is also more straightforward than SCSG and yields simpler analysis. Moreover, ProxSVRG+ outperforms the deterministic proximal gradient descent (ProxGD) for a wide range of minibatch sizes, which partially solves an open problem proposed in . Also, ProxSVRG+ uses much less proximal oracle calls than ProxSVRG . Moreover, for nonconvex functions satisfied Polyak-Łojasiewicz condition, we prove that ProxSVRG+ achieves a global linear convergence rate without restart unlike ProxSVRG. Thus, it can automatically switch to the faster linear convergence in some regions as long as the objective function satisfies the PL condition locally in these regions. Finally, we conduct several experiments and the experimental results are consistent with the theoretical results.

This paper focuses on the optimization problem min f(x) + h(x), where f is of a finite sum structure (with n functions in the sum), with nonconvex but smooth components, and h is a convex but possibly nonsmooth function. So, this is a nonconvex finite sum problem with a convex regularizer. Function h is treated using a prox step.  The authors propose a small modification to ProxSVRG (called ProxSVRG+), and prove that this small modification has surprisingly interesting consequences. The modification consists in replacing the full gradient computation in the outer loop of ProxSVRG by an approximation thereof through subsampling/minibatch (batch size B). In the inner loop, minibatch steps are run (with minibatch size b).   Main theoretical results:  • The SFO complexity (i.e., number of calls of stochastic first order oracle) of ProxSVRG+ is better than the complexity of ProxSVRG, ProxSGD and ProxGD in all minibatch regimes (any b). The results are nicely summarized in Tables 1, 2 and Figure 1 on page 3. The complexity is O( n/(eps*sqrt{b}) + b/eps ) under no additional assumptions.  • Further improvement is obtained when one assumes bounded variance of the stochastic gradients, with  sigma = O(1). The improvement consist in replacing the “n” in the complexity by the minimum of n and 1/eps. • The SFO complexity is better than that of SCSG in the regimes when b is not too small and not too large.  • ProxSVRG results hold for b <= n^{2/3}, and ProxSGD results hold for b >= 1/eps. However, ProxSVRG+ result holds for all values of b. • The PO complexity (number of proximal steps) is O(1/\eps), which matches the best PO complexity of ProxGD and ProxSGD, and is better than the PO complexity of Natasha1.5 and SCSG.  Additional results of similar nature (but with logarithmic dependence on epsilon, as expected) are obtained under the PL (Polyak-Lojasiewicz) condition. A new variant of PL condition is proposed and used in the analysis. Unlike ProxSVRG, no restart is needed for ProxSVRG+.  Computational results are provided to support the theory.   Strengths:  The paper is well-written (there is only a small number of grammatical issues) and well-motivated. The fact that only a minor modification of ProxSVRG is needed to obtain all the improvements should not be seen negatively. The new analysis with the modification is simpler, more elegant, and tighter. The computational experiments clearly show that ProSVRG+ can obtain similar performance as ProxSVRG with much smaller minibatch sizes b (order of magnitude smaller). This is both interesting and important. This is a very nice paper that should be accepted.   Weaknesses and Questions:  1) Can the results be improved so that ProxSVRG+ becomes better than SCSG for all minibatch sizes b? 2) How does the PL condition you use compare with the PL conditions proposed in “Global Convergence of Arbitrary-Block Gradient Methods for Generalized Polyak-Łojasiewicz Functions”, arXiv:1709.03014 ? 3) Line 114: Is the assumption really necessary? Why? Or just sufficient? 4) I think the paper would benefit if some more experiments were included in the supplementary, on some other problems and with other datasets. Otherwise the robustness/generalization of the observations drawn from the included experiments is unclear.   Small issues:  1) Lines 15-16: The sentence starting with “Besides” is not grammatically correct. 2) Line 68: What is a “super constant”? 3) Line 75: “matches” -> “match” 4) Page 3: “orcale” – “oracle” (twice) 5) Caption of Table 1: “are defined” -> “are given” 6) Eq (3): use full stop 7) Line 122: The sentence is not grammatically correct. 8) Line 200: “restated” -> “restarted” 9) Paper [21]: accents missing for one author’s name  *****  I read the rebuttal and the other reviews, and am keeping my score.


PAC-Bayes bounds for stable algorithms with instance-dependent priors
PAC-Bayes bounds have been proposed to get risk estimates based on a training sample. In this paper the PAC-Bayes approach is combined with stability of the hypothesis learned by a Hilbert space valued algorithm. The PAC-Bayes setting is used with a Gaussian prior centered at the expected output. Thus a novelty of our paper is using priors defined in terms of the data-generating distribution. Our main result estimates the risk of the randomized algorithm in terms of the hypothesis stability coefficients. We also provide a new bound for the SVM classifier, which is compared to other known bounds experimentally. Ours appears to be the first uniform hypothesis stability-based bound that evaluates to non-trivial values.

Summary: This paper proposes a theoretical analysis of stable algorithms by means of the PAC-Bayesian theory for randomized classifiers. The originality come from the fact that the authors use a stability with respect to a distance (norm) between hypothesis instead of the classic stability with respect to loss functions. The authors propose a new PAC-Bayes bound in the context of Gaussian randomization and exhibits a special case for SVM classifiers. The bound is discussed and compared with other stability based bounds.   Pros/Cons Pros: -New PAC-Bayes bound that appears to be informative and tight -In my point of view, the paper can offer a basis to other developments -As a side contribution, an extension of the McDiarmid inequality to vector-valued functions is proposed. Cons: -A more precise comparison/discussion to related notion of stability should be made -A detailed discussion with other PAC-Bayesian bounds is missing -Results specialized to Gaussian randomization, perspectives of development are not well developed  Overall:   Comments  +The notion of hypothesis sensitivity coefficients (Definition 1) seem very similar to the notion of uniform argument stability of [Liu et al., 2017] (Definition 2). Could the author discuss the difference between the two formulations.  There is also the notion of collective stability which is applied to vector-valued functions: Ben London et al.,PAC-Bayesian Collective Stability. AISTATS 2014 The authors should also discuss this notion and the point is also related to the comparison of PAC-Bayes results (see below)   +One appealing aspect of algorithmic stability is to provide some guarantees for the algorithm used to learn the hypothesis, and more technically with respect to the loss used. By using the hypothesis sensitivity coefficients, you are not directly dependent on the algorithm and the loss used but it seems to me that there is an indirect dependency. It could be interesting to discuss this dependency. It seems natural to think that if we have the hypothesis sensitivity then we have an hypothesis stability for the corresponding loss but what about the other side.  Considering randomized algorithms, this dependency (if confirmed) seems to influence the mean hypothesis of the distribution. Can we say that the learning algorithm can be interpreted as a way to bias the distribution.  +The authors have compared their work with previous bounds using a notion of stability which is good and interesting. But I am a bit disappointed that there is no real discussion with PAC-Bayesian bounds and in particular those using Gaussian distributions (mentioned for example in the paper of Germain, Lacasse, Laviolette, Marchand, Roy; JMLR 2015) It appears that the result seem to have a better convergence than existing bounds.  In general, for having a better rate than O(1/\sqrt(n)) one must rely on some (strong) assumption(s). Since the uniform stability framework and the classic PAC-Bayesian theorems (as used in the paper) with Gaussian distributions do not allow to obtain better rates easily taken separately, I wonder what is the assumption setting that makes this new result possible.  Is is the particular combination of  stability over coefficients with the assumption of Gaussian distributions over them or is there another underlying assumption?  +This is maybe related to the question above, but how the capacity to define good Gaussian  distributions over the Hilbert space of classifiers (as discussed in Appendix E) is key for the result and in particular for developing interesting perspectives?   +The result is based on a Seeger/Langford formulation, is it possible to obtain results based on other PAC-Bayesian theorems such as McAllester1999 or Catoni2007 for example? From new theorems, will it be possible to develop new learning algorithms specifically adapted to the PAC-Bayesian setting ?  +The bound found by the authors is penalized by small variances. This point is interesting because it illustrates that for a randomized algorithm (which can also be interpreted as a majority vote) the classifiers involved must have some diversity. If you have a large variance, the bound will become better and the average hypothesis will have less importance. Do the authors that the presence of the \sigma^2 parameter is reasonable and if there is room for improvement or a better understanding of randomized/majority vote algorithms?   The notion of diversity appears in the C-bound of (Lacasse et al.,NIPS'07), I do not know if some relationships can be made here.  +line 44 notation M_1(.) could be introduced at this line.  +Note that the reference [Liu et al.,2017] has been published at ICML 2017.    #### After Rebuttal ####  Thank you for your answers.  I think that the authors should precise the connections/differences between the works of Liu et al. and London et al. (at least in supplementary for example). The discussion with Germain et al,09 is also necessary.  For the sake of completeness, I would suggest to the authors to add the proof of the  expression of the Kullback-Leibler divergence for Gaussians over Hilbert spaces, line  174 (for example in supplementary).  The discussion about perspectives were rather short in the paper, the authors may try to improve the discussion about possible perspectives.


Causal Inference and Mechanism Clustering of A Mixture of Additive Noise Models
The inference of the causal relationship between a pair of observed variables is a fundamental problem in science, and most existing approaches are based on one single causal model. In practice, however, observations are often collected from multiple sources with heterogeneous causal models due to certain uncontrollable factors, which renders causal analysis results obtained by a single model skeptical. In this paper, we generalize the Additive Noise Model (ANM) to a mixture model, which consists of a finite number of ANMs, and provide the condition of its causal identifiability. To conduct model estimation, we propose Gaussian Process Partially Observable Model (GPPOM), and incorporate independence enforcement into it to learn latent parameter associated with each observation. Causal inference and clustering according to the underlying generating mechanisms of the mixture model are addressed in this work. Experiments on synthetic and real data demonstrate the effectiveness of our proposed approach.

This paper proposes an approach to estimate causal relationships between two variables X and Y when properties of the mechanism changes across the dataset. The authors propose and extension of the non-linear additive noise model [Hoyer et al. 2009] to the case of a mixture of a finite number of non-linear additive noise models, coined Additive Noise Model- Mixture Model (ANM-MM). The authors propose a theoretical identifiability result based on the proof of [Hoyer et al. 2009], then provide an estimation algorithm based on Gaussian Process Partially Observable Models (GPPOM), introduced as a generalization of Gaussian Process Latent Variable Models (GPLVM). Comparison of the approach to baseline for causal inference and clustering are provided on real and simulated data.  The problem addressed in this paper is definitively interesting. While some of the experimental results are promising, theoretical and empirical provide a limited understanding of the approach, which is rather complex, and in particular of its strength and limitations. Main concerns follow. Theoretical results:  It is unclear to me why the theoretical identifiability result of Theorem 1 is not a very simple corollary of [Hoyer et al., 2009]: assume both directions admit an ANM-MM, then there is a pair of function parameters values (\theta,\omega) corresponding to each direction, that happen simultaneously with non-zero probability. Then conditioning on this event, we get an ANM for both directions and can conclude that this cannot happen in a generic case, based on [Hoyer et al., 2009]. Am I missing something? At the same time, this theoretical result does not offer guaranties for finite samples. Given the complexity of the model, I can imagine several things that can go wrong when applying the model: - When the number of clusters gets too large, overfitting of the models (due to lack of samples) may reduce the information in additive noise that can be used for causal inference. How can we assess (theoretically or experimentally) and possibly handle this issue? This is partially shown in Fig. 3a), but the study restrict itself to the case of a number of clusters inferior or equal to ground truth. On the other hand, there are baseline methods that do not assume a mixture of mechanisms (such as LiNGAM or PNL) that seem quite robust to variations in the mechanism. - In the particular context of ANM, it would be good that the authors provide a toy example showing that mixing mechanisms can lead to misleading results. Intuitively, not capturing the variation in the mechanism with lead to an increase in additive noise. Is there any toy example where one can prove ANM would fail? Experimental results Regarding simulation, one simple baseline should use clustering combined with ANM inference (all “standard” causal inference approaches can handle a single mechanism). Moreover, clustering (such as K-means) in the PCA domain seems a fair and reasonable baseline that is likely to perform better than in the original space. Given the complexity of the algorithm, it seems difficult to choose the hyperparameters for a given dataset (this includes number of clusters and parameter of independence). How to we choose them to reproduce the results on the Tuebingen dataset? Also it is unclear whether clustering of mechanism is necessary in most of these datasets, so I wonder if it would not be worth focusing on more datasets where the mechanism variability is known. The FOEN data seems to be an interesting example, and I would suggest to put part of supplemental Fig.5 in main text.


Norm-Ranging LSH for Maximum Inner Product Search
Neyshabur and Srebro proposed SIMPLE-LSH [2015], which is the state-of-the-art hashing based algorithm for maximum inner product search (MIPS). We found that the performance of SIMPLE-LSH, in both theory and practice, suffers from long tails in the 2-norm distribution of real datasets. We propose NORM-RANGING LSH, which addresses the excessive normalization problem caused by long tails by partitioning a dataset into sub-datasets and building a hash index for each sub-dataset independently. We prove that NORM-RANGING LSH achieves lower query time complexity than SIMPLE-LSH under mild conditions. We also show that the idea of dataset partitioning can improve another hashing based MIPS algorithm. Experiments show that NORM-RANGING LSH probes much less items than SIMPLE-LSH at the same recall, thus significantly benefiting MIPS based applications.

Authors find that the inner-product-to-cosine-reduction technique by Neyshabur and Srebro suffers from a disbalance in the L2-norm distribution. They show that this can be improved by splitting the data set (plus some additional improvements). They present a proof that such a split has theoretical guarantees. In addition, they proposed an approach to improve probing order of the buckets in their norm-split index.   They demonstrate their methods is partly applicable to A2-LSH, but it will likely improve other SOTA methods.  This is very solid work: I would vote for acceptance despite it misses comparison to some state-of-the-art and uses mostly small data sets (even ImageNet is not really large-scale as authors claim).   PROS: This is a well-written paper: I enjoyed reading it.   Authors make an interesting finding that can potentially benefit several methods for k-NN search using the maximum-inner product as the similarity function. In particular, the method of Andoni et al [2015]. Possibly, data set splitting can be useful even for graph-based retrieval methods (despite some of your tricks are applicable to random-projection LSH only).  They present several other interesting findings, some of which I briefly mention in detailed comments.  CONS: Authors do not compare against state-of-the art graph-based algorithms (Malkov and Yashunin ). Evaluation leaves somewhat to be desired, because two out of three data sets are small (Netfix is in fact tiny). ImageNet is not very big either.  Additional baseline that would be nice to run:  1) Andoni et al method for the cosine similarity search combined with the inner-product-to-cosine-reduction by Neyshabur and Srebro. One should be quite curious if Andoni et al would benefit from your data set splitting approach. Again, I understand it wouldn't be exactly straightforward to apply: yet, even a simple data set splitting with parallel searching over split may be beneficial. I think this would be a strong baseline as Andoni et all improve over simple-LSH quite a bit.  2) My feeling is that your extension to A2-LSH (Section 5) should have been implemented and tested.  SOME FURTHER COMMENTS:   61-62: Wasn't the original definition specified in terms of the distance? Does it make a difference when you reformulate it using the similarity instead? I think it would beneficial to formally introduce similarity and explain how it relates to the distance (e.g., there's an obvious connection for the cosine similarity).  68-69: the probability GIVEN THAT A RANDOM FUNCTION is drawn randomly and independently FOR EACH PAIR of vectors. This nearly always silent assumption of the LSH community was deservedly criticized by Wang et al. I think there is nothing bad about using this simplified assumption, but it is quite bad NOT TO MENTION it. Nearly nobody gets this subtlety and I have seen a machine learning professor from a top university baffled by the question what is actually assumed to be random in the theory of LSH.  103: Complexity under which assumptions? Also, missing citation to back up this. 108: It would be nice to add: i.e., the higher is the similarity, the lower is the complexity. Otherwise: nice observation also backed up by theory! 117-118: Nice observation too!  226: 2 million is not large at all! And this is your only reasonably large data set.   REFERENCES  Andoni, Alexandr, et al. "Practical and optimal LSH for angular distance." Advances in Neural Information Processing Systems. 2015. Malkov, Yu A., and D. A. Yashunin. "Efficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphs." arXiv preprint arXiv:1603.09320 (2016).  Wang, Hongya, et al. "Locality sensitive hashing revisited: filling the gap between theory and algorithm analysis.


High Dimensional Linear Regression using Lattice Basis Reduction
We consider a high dimensional linear regression problem where the goal is to efficiently recover an unknown vector β * from n noisy linear observations Y = Xβ * + W ∈ R n , for known X ∈ R n×p and unknown W ∈ R n . Unlike most of the literature on this model we make no sparsity assumption on β * . Instead we adopt a regularization based on assuming that the underlying vectors β * have rational entries with the same denominator Q ∈ Z >0 . We call this Q-rationality assumption. We propose a new polynomial-time algorithm for this task which is based on the seminal Lenstra-Lenstra-Lovasz (LLL) lattice basis reduction algorithm. We establish that under the Q-rationality assumption, our algorithm recovers exactly the vector β * for a large class of distributions for the iid entries of X and non-zero noise W . We prove that it is successful under small noise, even when the learner has access to only one observation (n = 1). Furthermore, we prove that in the case of the Gaussian white noise for W , n = o (p/ log p) and Q sufficiently large, our algorithm tolerates a nearly optimal information-theoretic level of the noise.

The paper presents a novel method of exactly recovering a vector of coefficients in high-dimensional linear regression, with high probability as the dimension goes to infinity. The method assumes that the correct coefficients come from a finite discrete set of bounded rational values, but it does not - as is commonplace - assume that the coefficient vector is sparse. To achieve this, the authors extend a classical algorithm for lattice basis reduction. Crucially, this approach does not require the sample size to grow with the dimension, thus in certain cases the algorithm is able to recover the exact coefficient vector from just a single sample (with the dimension sufficiently large).  A novel connection between high-dimensional linear regression and lattice basis reduction is the main strength of the paper. The ideas and algorithms in the paper are presented clearly. The main limitation of the algorithm is the fact that all of the guarantees hold „with high probability as dimension p goes to infinity”. No finite dimension guarantees are offered, which raises the question of whether the proposed algorithm only achieves the desired behavior for combinatorially large dimensions. That would make the „polynomial runtime” guarantee somewhat misleading (since the runtime depends on p). It is worth noting that despite the authors’ claim that taking dimension p to infinity is „very popular in the literature”, after looking at some of the cited works I noticed that typically at least some finite dimension guarantees are offered in these works. Thus, to address this, in addtion to „w.h.p. when p tends to infinity”, the authors could - if at all possible - state the guarantees as „w.p. at least 1-delta for p larger than…”, for example.   Minor comments: - line 76: „so that to” -> „so as to” - line 212: „can be applied,.” -> „can be applied.”


Contour location via entropy reduction leveraging multiple information sources
We introduce an algorithm to locate contours of functions that are expensive to evaluate. The problem of locating contours arises in many applications, including classification, constrained optimization, and performance analysis of mechanical and dynamical systems (reliability, probability of failure, stability, etc.). Our algorithm locates contours using information from multiple sources, which are available in the form of relatively inexpensive, biased, and possibly noisy approximations to the original function. Considering multiple information sources can lead to significant cost savings. We also introduce the concept of contour entropy, a formal measure of uncertainty about the location of the zero contour of a function approximated by a statistical surrogate model. Our algorithm locates contours efficiently by maximizing the reduction of contour entropy per unit cost.

This paper introduces an active learning strategy for estimating some contour line of an expensive to evaluate function in the case where lower fidelity  evaluations of the objective function can be performed at lower cost. The proposed worklow relies on a multi-information source Gaussian Process model  following notably the approach of Poloczek et al. (on "Multi-information source optimization") presented at NIPS 2017. Here however the acquisition  function substantially differs from the one used in the former reference, and focuses specifically on contour line estimation rather than optimization.  The proposed acquisition function builds upon the entropy of discrete distributions indicating at each point whether the response is in a prescribed  interval around the contour level, below the interval lower bound or above the interval upper bound. At any given point, the entropy of this distribution can be written as a sum of three terms written in closed form (up to the Gaussian CDF) as a function of the predictive GP mean and standard deviation.  Then this entropy is integrated over the input space (up to a normalization by the domain volume) and the considered sampling criterion is obtained at any  given candidate evaluation point and fidelity level as the ratio between the expected decrease in this integrated entropy if a corresponding evaluation is  performed and the (known) cost of performing it. While the integrand is not calculated in closed form, a tight approximation is provided for which a closed  form expression is presened in the paper. An algorithm is then devised that consists, from a starting design, in performing at every iteration a new  evaluation at selected point and fidelity level so as to achieve optimal resource allocation in the sense of the proposed criterion, until budget exhaustion.  The contour line of the GP predictive mean at the final state is finally returned. The algorithm is tested and illustrated on two examples, one 2-dimensional  analytical one and one 1-dimensional test case stemming from a chemical engineering problem.   As motivated at the beginning of the paper, contour line estimation is an important topic arising in a number of applications and methodological contexts.  Also, integrating several information sources withing a GP model in order to study expensive to evaluate functions under a smart resource allocation scheme  is a timely and useful approach with promising benefits in a variety of settings. Adressing contour line estimation using a multi-information source GP and  a dedicated acquisition function thus appears as a legitimate and worthwhile development. From that perspective, I very much welcome the present contribution. This being said, the paper also has some weaknesses, and while I recommend acceptance, I think that it deserves to be substantially improved before publication.  One of the main concerns is that part of the relevant literature has been ignored, and also importantly that the proposed approach has not really been  extensively compared to potential competitors (that might need to be adapted to the multi-source framework; not e also that single-fidelity experiments could be  run in order to better understand how the proposed acquisition function compares to others from the literature). Another main concern in connection with the  previous one is that the presented examples remain relatively simple, one testcase being an analytical function and the other one a one-dimensional mototonic  function. While I am not necessarily requesting a gigantic benchmark or a list of complicated high-dimensional real-world test cases, the paper would significantly  benefit from a more informative application section. Ideally, the two aspects of improving the representativity of numerical test cases and of better benchmarking  against competitor strategies could be combined. As of missing approaches from the literature, some entry points follow:   * Adaptive Designs of Experiments for Accurate Approximation of a Target Region (Picheny et al. 2010) http://mechanicaldesign.asmedigitalcollection.asme.org/article.aspx?articleid=1450081  *Fast kriging-based stepwise uncertainty reduction with application to the identification of an excursion set (Chevalier et al. 2014a) http://amstat.tandfonline.com/doi/full/10.1080/00401706.2013.860918   * NB: approaches from the two articles above and more (some cited in the submitted paper but not benchmarked against) are coded for  instance in the R package "KrigInv". The following article gives an overview of some of the package's functionalities:  KrigInv: An efficient and user-friendly implementation of batch-sequential inversion strategies based on kriging (Chevalier et al. 2014b)  * In the following paper, an entropy-based approach (but not in the same fashion as the one proposed in the submitted paper) is used, in  a closely related reliability framework:  Gaussian process surrogates for failure detection: A Bayesian experimental design approach (Wang et al. 2016)  https://www.sciencedirect.com/science/article/pii/S002199911600125X  * For an overall discussion on estimating and quantifying uncertainty on sets under GP priors (with an example in contour line estimation),  see Quantifying Uncertainties on Excursion Sets Under a Gaussian Random Field Prior (Azzimonti et al. 2016) https://epubs.siam.org/doi/abs/10.1137/141000749 NB: the presented approaches to quantify uncertainties on sets under GP priors could also be useful here to return a more complete output  (than just the contour line of the predictive GP mean) in the CLoVER algorithm.   * Coming to the multi-fidelity framework and sequential design for learning quantities (e.g. probabilities of threshold exceedance) of  interest, see notably Assessing Fire Safety using Complex Numerical Models with a Bayesian Multi-fidelity Approach (Stroh et al. 2017) https://www.sciencedirect.com/science/article/pii/S0379711217301297?via%3Dihub    Some further points  * Somehow confusing to read "relatively inexpensive" in the abstract and then "expensive to evaluate" in the first line of the introduction! * L 55 "A contour" should be "A contour line"?  * L88: what does "f(l,x) being the normal distribution..." mean?    * It would be nice to have a point-by-point derivation of equation (10) in the supplementary material (that would among others help readers  including referees proofchecking the calculation). * About the integral appearing in the criterion, some more detail on how its computation is dealt with could be worth.  ##### added after the rebuttal  I updated my overall grade from 7 to 8 as I found the response to the point and it made me confident that the final paper would be improved (by suitably accounting for my remarks and those of the other referees) upon acceptance.  Let me add a comment about related work by Rémi Stroh. The authors are right, the Fire Safety paper is of relevance but does actually not address the design of GP-based multi-fidelity acquisition functions (in the BO fashion). However, this point has been further developed by Stroh et al; see "Sequential design of experiments to estimate a probability of exceeding a threshold in a multi-fidelity stochastic simulator" in conference contributions listed in  http://www.l2s.centralesupelec.fr/en/perso/remi.stroh/publications


Modelling and unsupervised learning of symmetric deformable object categories
We propose a new approach to model and learn, without manual supervision, the symmetries of natural objects, such as faces or flowers, given only images as input. It is well known that objects that have a symmetric structure do not usually result in symmetric images due to articulation and perspective effects. This is often tackled by seeking the intrinsic symmetries of the underlying 3D shape, which is very difficult to do when the latter cannot be recovered reliably from data. We show that, if only raw images are given, it is possible to look instead for symmetries in the space of object deformations. We can then learn symmetries from an unstructured collection of images of the object as an extension of the recently-introduced object frame representation, modified so that object symmetries reduce to the obvious symmetry groups in the normalized space. We also show that our formulation provides an explanation of the ambiguities that arise in recovering the pose of symmetric objects from their shape or images and we provide a way of discounting such ambiguities in learning.

Summary: This work propose an approach to model symmetries in deformable object categories in an unsupervised manner. This approach has been demonstrated to work for objects with bilateral symmetry (identifying symmetries in human faces using CelebA dataset, cats using cats head dataset, on cars with a synthetic car dataset), and finally for rotational symmetry on a protein structure.   Pros:  + Overview of the problem and associated challenges.  + The proposed approach seems a natural way to establish dense correspondences for non-rigid objects given two views of same object category (say example in Figure-3). In my opinion, correspondences for non-rigid/deformable objects is far more important problem than symmetry (with a potential impact on numerous problems including non rigid 3D reconstruction, wide baseline disparity estimation, human analysis etc). Reading the approach section made me actually think more in terms of correspondences than symmetry.  Cons:   - While the theory is interesting, I think the current submission is lacking on experiments. Here are my reasons for this statement:  (i)  The evaluation using CelebA face dataset cannot be termed completely "unsupervised". The face images are aligned using the 6 keypoints (left eye, right eye, nose, left side of mouth, right side of mouth, and chin) and everything is centered around nose. Most of the images have an "inherent" symmetry because of this pre-processing. Therefore, it is hard to tell the role of proposed approach in getting symmetries. I would suggest the authors to use the original CelebA dataset where faces are not aligned and show comparison.  (ii) The same argument goes for using Cats-head dataset. I would suggest to instead use Cats and Dogs dataset from Parkhi et al to demonstrate learning of symmetry. In my opinion that would be a sound approach to demonstrate the "unsupervised" part of the approach. I believe that current experiments have a hidden manual supervision and is not completely unsupervised.  (iii) This point is a suggestion to make the submission strong -- If the authors agree on (i) and (ii), than I think the most impactful experiment is demonstration of learning symmetries on ImageNet-1k dataset. Use the images from each category, and demonstrate how correspondences and symmetries naturally emerge using this approach without any manual supervision.  - An ideal comparison of this approach would be the evaluation on synthetic humans dataset (as done by Raviv et al [27]). The authors have provided a qualitative analysis in the supplementary material for synthetic humans but an extensive analysis with human study is the one that I would have loved to see. Even more interesting would have been the use of data from L. Wei et al (CVPR 2016) < http://www.hao-li.com/publications/papers/cvpr2016DHBCUCN.pdf > for demonstrating symmetries, and to verify the hypothesis of lemma-3, lemma-4, and bilateral symmetry.  Suggestion:  1. The proposed approach has a flavor of estimating symmetry or similarity via composition, so it would be good to include those work in Section-2 specifically the ones by Boiman and Irani (NIPS 2006, ICCV 2005).


Orthogonally Decoupled Variational Gaussian Processes
Gaussian processes (GPs) provide a powerful non-parametric framework for reasoning over functions. Despite appealing theory, its superlinear computational and memory complexities have presented a long-standing challenge. State-of-the-art sparse variational inference methods trade modeling accuracy against complexity. However, the complexities of these methods still scale superlinearly in the number of basis functions, implying that that sparse GP methods are able to learn from large datasets only when a small model is used. Recently, a decoupled approach was proposed that removes the unnecessary coupling between the complexities of modeling the mean and the covariance functions of a GP. It achieves a linear complexity in the number of mean parameters, so an expressive posterior mean function can be modeled. While promising, this approach suffers from optimization difficulties due to ill-conditioning and non-convexity. In this work, we propose an alternative decoupled parametrization. It adopts an orthogonal basis in the mean function to model the residues that cannot be learned by the standard coupled approach. Therefore, our method extends, rather than replaces, the coupled approach to achieve strictly better performance. This construction admits a straightforward natural gradient update rule, so the structure of the information manifold that is lost during decoupling can be leveraged to speed up learning. Empirically, our algorithm demonstrates significantly faster convergence in multiple experiments.

Edit 2: While reviewing the paper and discussing with other reviewers, I reframed the approximation of Cheng and Boots as a structured variational approximation in the Titsias's framework like this: https://imgur.com/a/E6uUKXh, and similarly for the method proposed in this paper: https://imgur.com/a/tbx1fMs. While these might seem trivial in hindsight and that they didn't bring much intuition, I think these could be useful in making connections between methods and previously published results. I hope the authors find these useful.  Edit after the rebuttal period 1: The author response has sufficiently addressed my concerns, though I would still want to understand how the approximation was discovered/derived. I also think that it could be good, as a future research direction, to relate the approximation of Cheng and Boots and the approximation that this paper is using to the Titsias' framework without the need of RKHS, and the natural gradient version of Hensman et al.   Summary: This paper introduces an orthogonally decoupled sparse variational approximation for Gaussian process regression and classification models, extending the decoupled framework of Cheng and Boots (2017). The idea is that a combination of a particular partitioning of the inducing points and a specific parameterisation of the mean and covariance functions lead to an efficient sparse posterior such that natural gradients can be obtained cheaply, enabling natural gradient methods to be used to optimise the variational lower bound. Experiments on multiple regression and classification datasets demonstrate the performance of the new decoupled approach and its version using natural gradients.  Comments:  1. the parameterisation of the mean and covariance in equations 9 is particularly interesting, as it allows useful cancellations to happen when moving to equations 10. 2. I'm not sure how this particular parameterisation pops up in the first place, could the thought process be elaborated -- I think this could be useful for deriving further approximations based on this decoupled framework? for example, the natural gradient version of Cheng and Boots as in appendix C2 was considered first, and then somehow you realised that inserting (I - \phi_\beta K_\beta^{-1} \phi_\beta) in front of \phi_gamma gives decoupled natural gradients. This does not look trivial. 3. just an observation: the hybrid approach seems to perform poorly compared to the natural gradient version of Orth and Coupled, but seems to be better than Orth. 4. I am particularly intrigued that the natural version of the coupled approach -- which I think is the natural gradient approach of Hensman et al -- performs *much better* than the decoupled approach of Cheng and Boots (2017). However, Cheng and Boots compared their decoupled approach to SVI [I thought this is the approach of Hensman et al], and the decoupled approach outperforms SVI in all experiments they considered. Have you looked at this contradictory results in detail? Or I'm missing something here? 5. While I think the experimental results, especially on the regression dataset, are pretty solid, I'm wondering what is the fairest comparison protocol here? Previous sparse GP works basically assumed the same number of pseudo points for all methods and looked at the predictive performance after a number of iterations [provided that all methods have the same computational complexity]. The decoupled approaches have two set of pseudo points, and the complexity is a little bit different compared to previous approaches. The deltas between methods in the experimental tables are also very small to concretely say one method is better than the other, given they all have slightly different complexities and hence running times.  6. Making some of the ranking results in table 2 bold is, in my opinion, a bit misleading, as the methods perform very similarly and the std error for the mean rank seems to be very large here. 7. The paper is generally well-written, but I found it's hard to understand what the approximation is intuitively doing. Perhaps some visualisation of the methods [coupled vs decoupled vs orthogonally decoupled] on some 1D, 2D tasks could be useful, i.e. what the pseudo-points are doing, what happens when you optimise them, will the methods get to exact GP when the number of pseudo points = number of training points, does the FITC pathology described in Bauer et al (2016) exist here, ...


Implicit Reparameterization Gradients
By providing a simple and efficient way of computing low-variance gradients of continuous random variables, the reparameterization trick has become the technique of choice for training a variety of latent variable models. However, it is not applicable to a number of important continuous distributions. We introduce an alternative approach to computing reparameterization gradients based on implicit differentiation and demonstrate its broader applicability by applying it to Gamma, Beta, Dirichlet, and von Mises distributions, which cannot be used with the classic reparameterization trick. Our experiments show that the proposed approach is faster and more accurate than the existing gradient estimators for these distributions.
• We develop implicit reparameterization gradients that provide unbiased estimators for continuous distributions with numerically tractable CDFs. This allows many other important distributions to be used as easily as the Normal distribution in stochastic computation graphs.
• We show that the proposed gradients are both faster and more accurate than alternative approaches.
• We demonstrate that our method can outperform existing stochastic variational methods at training the Latent Dirichlet Allocation topic model in a black-box fashion using amortized inference.
• We use implicit reparameterization gradients to train VAEs with Gamma, Beta, and von Mises latent variables instead of the usual Normal variables, leading to latent spaces with interesting alternative topologies.

EDITED AFTER REBUTTAL:  I appreciate the authors' effort to make the notation consistent and improve the experiments by taking into account the computational complexity.  There are a couple of comments that were not addressed in the rebuttal, but I still recommend acceptance. For completeness, I am listing these comments below: + Experiments without optimizing the hyperparameters in the LDA experiment. + Experiments to showcase truncated distributions. + Comparison / Discussion about the approach of Roeder et al. for mixtures (they also integrate out the discrete component).  ------------------  This paper proposes a method to reparameterize distribution that are considered "non-reparameterizable", such as the Gamma or vonMises, as well as truncated distributions and mixtures. The proposed approach is compared to other baselines such as RSVI and is shown to outperform them.  The paper addresses a relevant problem and the proposed solution is clear and elegant.  My first comment is that this method has already been proposed in the paper "Pathwise Derivatives Beyond the Reparameterization Trick" [Jankowiak & Obermeyer, ICML 2018]. However, I still think that there is value in the Implicit Reparameterization Gradients, for three reasons. First, this paper approximates the CDF of the gamma without resorting to Taylor approximations. Second, this paper has more experiments, including experiments with the vonMises distribution. Third, accepted ICML papers where not available at the time of the NIPS deadline, so [Jankowiak & Obermeyer 2018] should not be taken into account. That said, the citation should be included.  My two main criticisms about this paper are: 1. The notation is not consistent. 2. The experiments can be improved; e.g., by showing the evolution of the marginal lower bound.  1. The notation in the paper is not consistent, please revise and improve it. For example, the parameterization of a distribution (say, q) in terms of some parameters (say, phi) is sometimes written as q(z|phi) and sometimes as q_phi(z). Please use one. More comments on the notation: + Eq. 9: Replace u with epsilon. Variable u has not been defined anywhere else. + In the mixture example (line 116), it should be q_phi(z) instead of p(z|phi). + In the LDA experiment, alpha is both the hyperparameter of the Dirichlet (line 214) and the topics (Fig 2a). Please remove the "alpha" in Fig 2a.  2. Regarding experiments: + In both the LDA and VAE experiments, can you show the evolution of the variational lower bound w.r.t. wall-clock time? The paper states that each step RSVI step is 15% slower, but the information about convergence of each method is missing.  + In section 6.1, the paper states that "The results presented on Fig. 1 show that the implicit gradient is faster and has lower variance than RSVI". In Fig. 1, however, only the computational time for the vonMises distribution is reported, but not for the Dirichlet. Is it also faster in the Dirichlet case? Does that hold for any value of the shape augmentation? I think this should be added to the paper. + In the LDA experiment, I think it would be better not to optimize the hyperparameters, because the goal is to assess how good the different variational approximations are for a fixed model. If the model is not fixed (by fitting alpha), then the comparisons get more blurry. That said, I wouldn't expect any significant difference with the results reported in the paper, but it would be nice to check that.  Finally, I also have two more comments. The paper explains how to reparameterize mixtures and truncated distributions, but it does not include experiments with those. It would also be nice to see an experiment with truncated distributions. In addition, mixtures can be reparameterized by following the approach of "sticking the landing" [Roeder et al., 2017]; is there any reason why either method should be preferred?