Learning with Relaxed Supervision
For weakly-supervised problems with deterministic constraints between the latent variables and observed output, learning necessitates performing inference over latent variables conditioned on the output, which can be intractable no matter how simple the model family is. Even finding a single latent variable setting that satisfies the constraints could be difficult; for instance, the observed output may be the result of a latent database query or graphics program which must be inferred. Here, the difficulty lies in not the model but the supervision, and poor approximations at this stage could lead to following the wrong learning signal entirely. In this paper, we develop a rigorous approach to relaxing the supervision, which yields asymptotically consistent parameter estimates despite altering the supervision. Our approach parameterizes a family of increasingly accurate relaxations, and jointly optimizes both the model and relaxation parameters, while formulating constraints between these parameters to ensure efficient inference. These efficiency constraints allow us to learn in otherwise intractable settings, while asymptotic consistency ensures that we always follow a valid learning signal.

# Summary of paperThe paper proposes a method for handling models with latent variables where the latent variables have a complex, deterministic relationship with the output variables. Computing marginal probability in such models is intractable because it requires summing over the latent variables. The proposed method forms an alternate probability distribution that decomposes a functional representation of the determinism, which essentially penalizes disagreement with the constraints. The relaxation is theoretically analyzed, showing that the effect of the relaxation on loss is bounded by the inverse of the minimum value in the relaxation vector beta (which ranges from 0 to infinity), that the amount of data needed to learn increases by at most a similar ratio. The authors then provide analysis for the tradeoff between efficiency and these negative effects in setting the relaxation vector, which uses sampling to learn and adjust the relaxation parameter in a manner that guarantees tractable inference.The method is tested on synthetic data for translation and parsing. The evaluation essentially only compares against different variants of the proposed approach, so it doesn't compare against any existing methods for similar problems. I don't see any discussion of this either. It seems the approaches described in the 2nd paragraph of the paper would be useful comparisons. # QualityThis is a very good paper, with an innovative technical contribution, well evaluated and analyzed results, and good presentation. I'm actually left with a feeling the virtues of the proposed approach are too good. What are the weaknesses of this approach? What is the catch? This problem is related to the lack of comparison, in discussion or experiments, to other perhaps-less-principled approaches to problems with intractable hard constraints.# ClarityIt is obvious to me that the authors took a lot of care in presenting what could otherwise be difficult, dense material. The writing is some of the best I have seen, with useful examples, readable, but technically precise exposition, and well thought out structure. # OriginalityThe idea of relaxing hard constraints is not particularly new, but I think using the idea in this way is. # SignificanceThe approach should apply to a variety of applications, and the technical ideas may extend to other problems.


Recognizing retinal ganglion cells in the dark
Many neural circuits are composed of numerous distinct cell types that perform different operations on their inputs, and send their outputs to distinct targets. Therefore, a key step in understanding neural systems is to reliably distinguish cell types. An important example is the retina, for which present-day techniques for identifying cell types are accurate, but very labor-intensive. Here, we develop automated classifiers for functional identification of retinal ganglion cells, the output neurons of the retina, based solely on recorded voltage patterns on a large scale array. We use per-cell classifiers based on features extracted from electrophysiological images (spatiotemporal voltage waveforms) and interspike intervals (autocorrelations). These classifiers achieve high performance in distinguishing between the major ganglion cell classes of the primate retina, but fail in achieving the same accuracy in predicting cell polarities (ON vs. OFF). We then show how to use indicators of functional coupling within populations of ganglion cells (cross-correlation) to infer cell polarities with a matrix completion algorithm. This can result in accurate, fully automated methods for cell type classification.

The authors propose to identify different types of retinal ganglion cells based on their electrical image in high-density MEA recordings, their inter-spike interval distribution and their couplings to other cells. To this end, the authors focus on the problem of identifying suitable features from these measurements to then train logistic regression or other classifiers for inferring the cell type. In addition, they use matrix completion techniques for inferring the cell polarity (ON vs OFF). Identifying retinal ganglion cell types is clearly a timely topic (reviewed recently by Sanes and Masland, Annual Reviews of Neuroscience, 2015) and the approach chosen by the authors is creative and interesting: typical techniques either focus on anatomical reconstructions or functional responses. However, I found the the paper was not well structured, so that it was difficult to assess precisely what was done and what are the novel contributions. The first contribution was to use certain features and apply a logistic regression (or other) classifier. The features used for the classification algorithm in lines 79-101 are not well described. The authors should describe the manually chosen filter banks in the main paper, not only define them in the appendix. They should also define what slices are and how they look, how the filter banks were chosen and what the clusters were. It seems to me that much of the information later presented in section 4 should have already been worked into section 2. The second contribution is to estimate a "polarity" vector by exploiting the cross-correlation matrix and the fact that types of same polarity have positive correlations. It is a bit unclear how this task relates to the previous one. They describe their approach to this "matrix completion" problem in some detail on page 3, which again is very dense and full of jargon (e.g. what is a lifted matrix?). Theorem 1 seems oddly out of place - while this may be an interesting theoretical result, a data analysis paper seems to me not the place to advertise it. Section 5.2 seems of place after the results and I am not sure I understand its relevance (partially because Fig. 4 is hard to read and lacks meaningful labels). The evaluation of the methods seems mostly performed withcare and using interesting setups, e.g. the transfer between datasets. Unfortunately, the hand labeled data only includes the standard set of ON/OFF midget and parasol RGCs, so the evaluation is restricted to those types. In some sense, those are the easy ones, as they are so abundant in the data and can be clearly identified in primate recordings. Of course, for testing a new method that may be fine. The data augementation for building in invariances is a nice trick, albeit not new. The results seem are ok, but it is hard to tell what a good baseline would be to compare against - on the task of polarity identification, the method devised by the authors only performs a little better than just exploiting ISI/CCs. Indeed, it is unclear what the values in () in Table 1 are, so it may be that the author's method does not significantly outperforms the simpler methods. In line 351ff, the authors write that performance was >90% on 3 datasets for polarity inference, and ~66% for the rest. Are those included in the numbers reported in the table and the figures? What is the difference between the datasets? Then they say they only show the results for 4/10 datasets? Which ones? Regarding type identification, features from the electrical image clearly improve it, although random features seem to be as good as handcrafted ones (~93% correct). What about PCA features of the electrical image (or other standard feature sets one might think of)? In Fig 2 right, I am somewhat puzzled by the red curve. Wouldn't you expect that this should be high where green and blue curves are high? Can the authors provide any insight into why the red curve is only high for smaller distances, although the task is to infer polarity as for the green curve? In Fig. 3, error bars are again unclear. An analysis of the false positive/false negative rates would have been nice to see.


A Normative Theory of Adaptive Dimensionality Reduction in Neural Networks
To make sense of the world our brains must analyze high-dimensional datasets streamed by our sensory organs. Because such analysis begins with dimensionality reduction, modeling early sensory processing requires biologically plausible online dimensionality reduction algorithms. Recently, we derived such an algorithm, termed similarity matching, from a Multidimensional Scaling (MDS) objective function. However, in the existing algorithm, the number of output dimensions is set a priori by the number of output neurons and cannot be changed. Because the number of informative dimensions in sensory inputs is variable there is a need for adaptive dimensionality reduction. Here, we derive biologically plausible dimensionality reduction algorithms which adapt the number of output dimensions to the eigenspectrum of the input covariance matrix. We formulate three objective functions which, in the offline setting, are optimized by the projections of the input dataset onto its principal subspace scaled by the eigenvalues of the output covariance matrix. In turn, the output eigenvalues are computed as i) soft-thresholded, ii) hard-thresholded, iii) equalized thresholded eigenvalues of the input covariance matrix. In the online setting, we derive the three corresponding adaptive algorithms and map them onto the dynamics of neuronal activity in networks with biologically plausible local learning rules. Remarkably, in the last two networks, neurons are divided into two classes which we identify with principal neurons and interneurons in biological circuits.

### Summary A normative approach is presented to derive local learning rules for a rate network that optimise a classic multidimensional scaling objective (MDS). The work extends a recent publication on the same topic in three ways: (1) it incorporates a low-rank regularisation on the network output, (2) adds another set of latent variables to match the distribution of Hebbian/Anti-Hebbian plasticity rules in cortical networks and (3) adds a penalty term to whiten the outputs.### Advances The manuscript represents a welcome addition to the (recently) growing set of publications that strive to derive local learning rules in a normative way. The ansatz presented here is interesting, though much of the novelty is already presented in the earlier publication [22]. The additions are nonetheless big enough to justify another publication. For one, it had so far been unclear whether low-rank regularisations can actually be incorporated into local learning rules, and the work here sets a nice precedence. Second, anti-hebbian rules are usually observed on inhibitory synapses, and so the extension by another set of latent variables with similar characteristics is welcome. ### Major First and foremost, the paper is not the first one to derive biologically plausible learning rule from a normative ansatz (as suggested in line 065). Besides the earlier publication on which this manuscript is build upon, there is at least the publication by Vertechi et al. (2014) [1] which derive neural networks and local learning rules for a rate-based neural networks that converges to an efficient population code (for both the over- as well as under complete case, the latter of which performs dimensionality reduction).The objectives (3) and (5) are difficult to understand in the way written here, and should be rather formulated in terms of the standard MDS with an additional nuclear norm regularisation on Y^TY. E. g. (3) is equivalent tomin_Y ||X^TX - Y^TY||_2^2 + alpha*T*||Y^TY||_*where ||.||_* is the nuclear norm (i.e. the sum of singular values). The authors note this relation in passing, but it would really help the reader if the intuition behind (3) and (5) is explained and formulated more extensively.More severely, I could not develop any intuition behind objective (7). This equation is essentially (5) without the quadratic terms, but there are no explanations whatsoever as to where this objective is coming from, what it is supposed to capture, and what it adds to the objectives (3) and (5).### Minor 1) Line 039 "each individual neuron still has to compress hundreds or thousands of its inputs into a single output". I do not doubt the importance of compression for the brain, but that is a very odd argument given that it is true for any linear transformation (being it expansive or contractive). 2) Line 044 "Because the signal-to-noise ratio of natural stimuli varies significantly [...]". I have no idea what that should mean - are you referring to measurement noise like in the retina? 3) Practical problems: how do you initialise the network?### Verdict The manuscript provides an interesting extension of a recent work on multidimensional scaling in plausible neural networks that can incorporate some rough statistics of synaptic plasticity as well as low-rank approximations using local learning rules alone. The manuscript could gain from more polishing, especially by providing intuition on the different objectives (3), (5) and (7).[1] Vertechi, Brendel & Machines, Unsupervised learning of an efficient short-term memory network, NIPS (2014)Update: I have seen the (helpful) response of the authors and I am confident that the points raised above will be sufficiently addressed during the revision.


Optimal Rates for Random Fourier Features
Kernel methods represent one of the most powerful tools in machine learning to tackle problems expressed in terms of function values and derivatives due to their capability to represent and model complex relations. While these methods show good versatility, they are computationally intensive and have poor scalability to large data as they require operations on Gram matrices. In order to mitigate this serious computational limitation, recently randomized constructions have been proposed in the literature, which allow the application of fast linear algorithms. Random Fourier features (RFF) are among the most popular and widely applied constructions: they provide an easily computable, low-dimensional feature representation for shift-invariant kernels. Despite the popularity of RFFs, very little is understood theoretically about their approximation quality. In this paper, we provide a detailed finite-sample theoretical analysis about the approximation quality of RFFs by (i) establishing optimal (in terms of the RFF dimension, and growing set size) performance guarantees in uniform norm, and (ii) presenting guarantees in L r (1 ≤ r < ∞) norms. We also propose an RFF approximation to derivatives of a kernel with a theoretical study on its approximation quality.

Given the attention that has been drawn to the random Fourier features, it is glad to see a paper trying to given a detailed analysis on how much features it needs to have descent performance. It is definitely an important problem as well to obtain better understanding of the kernel methods and the efforts to make it work for very large scale data set.The paper is to provide optimal rates for the convergence of the kernel function in terms of the uniform and L^r norm. In the main results, the authors gave an improved a rate over the previous work: reducing the linear dependence of the diameter |S| to a logarithmic dependence.This is definitely an important result for certain settings. However, I am more curious about the consistency of random Fourier features in the probabilistic setting given a probability measure, because in many real world problems, data are usually not sampled from a uniform distribution.Moreover, I would also like to see some discussions about how the convergence of the kernel function will influence the performance of classification or other applications that use kernel method.To sum up, the paper does deliver what the authors promise, even though I didn't carefully go through the proof of the main results. The results are interesting in its own right, but I feel its contribution to the literature is relatively incremental.


Discrete Rényi Classifiers
Consider the binary classification problem of predicting a target variable Y from a discrete feature vector X = (X 1 , . . . , X d ). When the probability distribution P(X, Y ) is known, the optimal classifier, leading to the minimum misclassification rate, is given by the Maximum A-posteriori Probability (MAP) decision rule. However, in practice, estimating the complete joint distribution P(X, Y ) is computationally and statistically impossible for large values of d. Therefore, an alternative approach is to first estimate some low order marginals of the joint probability distribution P(X, Y ) and then design the classifier based on the estimated low order marginals. This approach is also helpful when the complete training data instances are not available due to privacy concerns. In this work, we consider the problem of finding the optimum classifier based on some estimated low order marginals of (X, Y ). We prove that for a given set of marginals, the minimum Hirschfeld-Gebelein-Rényi (HGR) correlation principle introduced in [1] leads to a randomized classification rule which is shown to have a misclassification rate no larger than twice the misclassification rate of the optimal classifier. Then, under a separability condition, it is shown that the proposed algorithm is equivalent to a randomized linear regression approach. In addition, this method naturally results in a robust feature selection method selecting a subset of features having the maximum worst case HGR correlation with the target variable. Our theoretical upper-bound is similar to the recent Discrete Chebyshev Classifier (DCC) approach [2], while the proposed algorithm has significant computational advantages since it only requires solving a least square optimization problem. Finally, we numerically compare our proposed algorithm with the DCC classifier and show that the proposed algorithm results in better misclassification rate over various UCI data repository datasets.

The problem of fitting a classifier to some data is investigated in this paper. Since full join distributions are infeasible when the number of features is large, the authors follow a standard approach of using marginal distributions to build a classifier. The extreme example of this approach is a Naive Bayes classifier which assumes that all features are independent given the class, and marginal for individual features only are considered. In this work, having pairwise marginal distributions the authors search for stochastic decision rules which minimise error for the worst case joint probability consistent with marginal distributions; those marginal distributions are computed from the data.The authors could also mention that even if the full joint could be modelled, it is not always wise to do that because it usually leads to over-fitting as, e.g., in unbiased Bayesian learning. But, that's a rather minor thing.The paper is very well written, and it contains several, important, technical contributions. The discussion of related work seems to be sufficient. Especially the connection with existing work is very well explained which adds to the significance of this paper.It would be useful if the authors could add a citation showing another example where harmonic mean was used to obtain a smooth objective function. It is a nice transformation.I am not sure if section 4 should not be restructured. Perhaps it would be better to emphasize the connection with Eq. 5. This could be done if the observation in section 4 was replaced by a theorem. This is an important part, and a clearly explained connection would be nice here.An important question, how does your a 2-factor approximation gap relate to the problem of over-fitting. It looks that over-fitting is somehow ignored in this work until feature selection is considered? Could you please explain?The connection with the group lasso regulariser applied to the standard linear regression problem is very interesting.Practical implications of this work are not clear at this point, but the amount of analytical contribution in this paper makes me believe that this could be a great paper if accepted.


Decomposition Bounds for Marginal MAP
Marginal MAP inference involves making MAP predictions in systems defined with latent variables or missing information. It is significantly more difficult than pure marginalization and MAP tasks, for which a large class of efficient and convergent variational algorithms, such as dual decomposition, exist. In this work, we generalize dual decomposition to a generic power sum inference task, which includes marginal MAP, along with pure marginalization and MAP, as special cases. Our method is based on a block coordinate descent algorithm on a new convex decomposition bound, that is guaranteed to converge monotonically, and can be parallelized efficiently. We demonstrate our approach on marginal MAP queries defined on real-world problems from the UAI approximate inference challenge, showing that our framework is faster and more reliable than previous methods.

The authors present a dual decomposition method for marginal MAP problems that makes use of Holder's inequality.This derivation stands apart from previous work in that it requires a fixed elimination ordering for the bounds to hold - the order of operations is not commutative.Still, the authors show that the resulting optimization problem has a similar form to known results in the literature on approximate variational inference.The paper is well-written and easy to read.General comments:-pg. 2, "the resulting optimization is only guaranteed to be a bound at convergence..." I'm not sure that this is true.I know for a fact it isn't true for TRW max-product despite some published statements to the contrary (though it does require a slightly different reparameterization vantage point).You should take another look at "Convergent message passing algorithms - a unifying view" by Meltzer et al.I think that the argument for the max-product case can be applied to the sum-product case as well.The authors made several comments on this point (which I admittedly didn't fully understand).To clarify, I was thinking of techniques based on reparameterization and pushing max through a product to get an inequality.In these situations (much like yours), you have a bound for any choice of messages even when the consistency conditions are not satisfied.-The reparameterization approach described here is similar to "Message-passing algorithms: reparameterizations and splittings" by Ruozzi et al.You should probably cite it. Typos:-pg. 2, extra space in "Section2" -throughout:"message passing algorithm" -> "message-passing algorithm" -throughout:I don't think, grammatically, that citations should be used as nouns."[8] showed" -> "Hazan et al. [8] showed"


Hidden Technical Debt in Machine Learning Systems
Machine learning offers a fantastically powerful toolkit for building useful complex prediction systems quickly. This paper argues it is dangerous to think of these quick wins as coming for free. Using the software engineering framework of technical debt, we find it is common to incur massive ongoing maintenance costs in real-world ML systems. We explore several ML-specific risk factors to account for in system design. These include boundary erosion, entanglement, hidden feedback loops, undeclared consumers, data dependencies, configuration issues, changes in the external world, and a variety of system-level anti-patterns.

The paper shines the light on the systems aspect of the design and maintenance of the ML systems.This is not a typical NIPS paper as it does not propose new methodology.However, there are barely any papers of its ilk despite the fact that most major corporations which make decisions based on data are employing some version of a ML system.Technical debt leads to high overhead in maintenance which carries significant monetary costs to the companies.By summarizing major issues associated with the maintenance of such systems (and thus potentially helping avoid some of the pointed out issues), this paper will have a *very* significant impact on the development of new ML systems.I very much enjoyed reading the paper.The paper resonates very strongly with me as after many years as a university ML researcher, I recently left to join industry, and over the last two years, I have observed first-hand the systems issues described in the paper.The authors did a great job summarizing the main pitfalls.I could add a few points to the taxonomy even though it is already quite comprehensive: - (Intro) Deploying ML systems carries its own overhead as well manifesting in running of A/B experiments (and then cleaning up after them) and maintaining an archive of artefacts for deployed models. - (ML-System Anti-Patterns) When reading about the Glue Code, I cannot help but think of incorporating open source packages into the system, especially if additional custom modifications need to be made.Using these packages leads to several potential (unmentioned) headaches: (1) someone needs to maintain this package internally, (2) whenever there is an updated version of the package, it needs to be re-integrated into the internal code-base, and (3) if internal modifications we made, they often need to be committed back to the open source (to maintain future compatibility); this may require jumping through administrative hoops. - (Configuration Debt) It is worth mentioning that interruptions in data logging may lead not only to stale features but also to (potentially cascading) failures in feature builds. - (Monitoring and Testing) Additional aspect could be changes in available bandwidth, both to extract/build the features and to train the model.Typos: - Page 2 (Correction Cascades) $a$ -> $m_a$ (two places) - Page 3, line 131, the -> them


Spherical Random Features for Polynomial Kernels
Compact explicit feature maps provide a practical framework to scale kernel methods to large-scale learning, but deriving such maps for many types of kernels remains a challenging open problem. Among the commonly used kernels for nonlinear classification are polynomial kernels, for which low approximation error has thus far necessitated explicit feature maps of large dimensionality, especially for higher-order polynomials. Meanwhile, because polynomial kernels are unbounded, they are frequently applied to data that has been normalized to unit 2 norm. The question we address in this work is: if we know a priori that data is normalized, can we devise a more compact map? We show that a putative affirmative answer to this question based on Random Fourier Features is impossible in this setting, and introduce a new approximation paradigm, Spherical Random Fourier (SRF) features, which circumvents these issues and delivers a compact approximation to polynomial kernels for data on the unit sphere. Compared to prior work, SRF features are less rank-deficient, more compact, and achieve better kernel approximation, especially for higher-order polynomials. The resulting predictions have lower variance and typically yield better classification accuracy.

== Summary ==This paper proposes a Spherical Random Fourier (SRF) features for polynomial kernels for data on the unit sphere. Although the polynomial kernels on the unit sphere is known to be shift-invariant, the paper shows that it is not always a positive definite function which prevents one from applying Bochner's theorem and obtain random Fourier features. Based on Schoenberg's characterization of positive definite function on an interval [0,2], they propose to approximate the kernel as an indefinite sum of Gaussians. The optimal probability p(w) from which the random features are sampled is obtained by minimizing the MSE between the true kernel K(z) and its approximation. Compared to Random Maclaurin (RM) and Tensor Sketch (TS), the proposed technique seems to enjoy better approximation as well as predictive accuracy, especially for polynomial of large degree p. In many datasets, however, it seems to perform as good as TS in term of accuracy, e.g., USPS. It's very nice to also see the extensive experimental results in the supplementary.In summary, this is a nice paper, although I see some disconnection between motivation in using Schoenberg's result (Theorem 4). However, I believe Schoenberg's characterization of kernel can be applied more generally. The empirical results are encouraging.== Quality ==The paper is technically sound. The major criticism I have is on Section 3.2 and Section 4. Regarding Section 3.2, it is not clear why Schoenberg's characterization is a suitable way of approximating the K(z) on [0,2]. It looks like one can construct an approximation of K(z) directly on [0,2] using arbitrary functions. Better justification should be provided.In Section 4, the approximation error decays at a rate of O(p^{-2.5}) which is a bit obscure as the analysis is done with fixed N=1. How does the number of Gaussians, N, affect the approximation? In experiment, it is mentioned that the specific number of N had negligible effect on the result. Why is this the case? Can you incorporate N into the analysis in Section 4 so it becomes clearer?Some comments/questions: [Line 086-089] To interpret the Fourier transform as a probability distribution, should one also assume that K(0) = 1? == Clarity ==The paper is clearly written. I only have minor comments:The notation d^d in Eq. (1) is confusing. There seems to be a disconnection between the first part and the approximation of the kernel. Comments/questions: [Line 298] what does (2k - 1)!! mean mathematically?== Originality ==Previous works in large-scale kernel learning oftens employed Bochner's characterization of positive definite kernel. On the other hand, this paper took another path and used the characterization of Schoenberg which is less known in the literature. One can argue, however, that using Schoenberg's characterization may not be the best option.== Significance ==It looks like the Schoenberg's characterization (Theorem 4) can be applied more generally, not just to polynomial kernels.One drawback of this paper is the restriction to polynomial kernel although the Schoenberg's characterization can already be applied to more general class of shift-invariant kernels. It is not clear why this characterization is particularly useful for polynomial kernels. What is the main difficulty?Minor commentsIn many datasets, the performance of SRF is comparable to that of TS. What is the intuition?


Inferring Algorithmic Patterns with Stack-Augmented Recurrent Nets
Despite the recent achievements in machine learning, we are still very far from achieving real artificial intelligence. In this paper, we discuss the limitations of standard deep learning approaches and show that some of these limitations can be overcome by learning how to grow the complexity of a model in a structured way. Specifically, we study the simplest sequence prediction problems that are beyond the scope of what is learnable with standard recurrent networks, algorithmically generated sequences which can only be learned by models which have the capacity to count and to memorize sequences. We show that some basic algorithms can be learned from sequential data using a recurrent network associated with a trainable memory.

The authors motivate the study of problems that are generated from algorithms, as the next step in learning algorithms that goes beyond just learning of statistical regularities. The authors propose a recurrent model that uses augmented memory that is independent from the computational model itself. Specifically, the authors use RNNs (and LSTM-RNNs) to control a memory store through soft PUSH/POP actions (for stacks) (or soft INSERT/LEFT/RIGHT for doubly linked lists). The current contents of memory stores themselves are used to compute the hidden states of the RNNs, along with data and past hidden states. At each step the RNN first computes the hidden state using the memory stores and the inputs and the past hidden state. From this hidden state the next step action is generated, which is then used, along with the hidden state to update the memory store at the next time step.Learning is done with SGD and a strategy that amounts to beam searching on actions. Results presented on toy problems show that indeed the method can perform better than RNNs and LSTMs. Results on the Penn Tree bank task on the other hand are worse than LSTMs. My comments on the paper follow in somewhat of a random order: Some of the results are curious and unintuitive. For example the LIST RNN gets 100% on the anbncndn task but worse on the easier task of anbncn. Also anbmc(n+m) does worse then anbncndn task on Stack RNN 40+10, even though the latter are generally longer and possibly more complicated. Is this probably just a relic of the difficulty of training this models and the stochasticity in the optimization procedure described in the supplementary materials ? Thepaper does try to control overfitting on Penn Tree bank task carefully, as can be seen from the results, which are much worse than the best results on this task by Vinyals et al. As such it does disservice to the exploration of the method itself, since its not clear if its actually performing worse than LSTMs or is just overfitting more. I wish that the optimization procedure was described in the paper itself, not shuffled off to the appendix since it is a crucial part of the paper. It is not clear to me if the extra memory that is provided actually serves as anything other than extra computation, since memory is read /write and can be used only for the current problem. When a new instance of a problem is encountered, the memory is reset. This would be different from a model that could separate algorithms from data needed to solve a problem. I was looking to see if the stacks had a size to them.. I would expect in practical implementation where the authors blend the stack contents from one time to the next, there would a certain size of the stack that's stored ? Is it perhaps tied to the number of time steps ? Small note: Is the equation for linked lists updates correct ? I would have expected that insert would add an element at i, and for i < HEAD all the elements would be unchanged and copied from the last time step, whereas for i > HEAD things would be copied from i-1 elements of the last time step.. Perhaps insert is defined slightly different here, or I'm mistaken..


Subspace Clustering with Irrelevant Features via Robust Dantzig Selector
This paper considers the subspace clustering problem where the data contains irrelevant or corrupted features. We propose a method termed "robust Dantzig selector" which can successfully identify the clustering structure even with the presence of irrelevant features. The idea is simple yet powerful: we replace the inner product by its robust counterpart, which is insensitive to the irrelevant features given an upper bound of the number of irrelevant features. We establish theoretical guarantees for the algorithm to identify the correct subspace, and demonstrate the effectiveness of the algorithm via numerical simulations. To the best of our knowledge, this is the first method developed to tackle subspace clustering with irrelevant features.

The authors define a convex optimization problem for each data point and aim to recover the support of a data point in terms of other data points. The main contribution is to replace the standard inner product between two vectors with its robust counterpart, by selecting the smallest (hopefully) non-irrelevant features.Comments - Could the authors please define 'robustness' in the abstract or briefly describe it.- line 37 'motivates'- line 77, lin 82, can club WLOG together, or show a picture.- line 149, where does the robustness come from ? What is intuitively the benefit of neglecting the larger dot-products ?- what happens when all irrelevant features are really small ? they would satisfy line 207 as well enter the dot-products, what would fail then ?- The current set of numerical simulations although limited, show some insight. Could the authors also please the add the result of choosing the top 'D - k' for various values of 'k', rather than just D1 ? This would show the benefit of robustness in the first place.


Learning to Linearize Under Uncertainty
Training deep feature hierarchies to solve supervised learning tasks has achieved state of the art performance on many problems in computer vision. However, a principled way in which to train such hierarchies in the unsupervised setting has remained elusive. In this work we suggest a new architecture and loss for training deep feature hierarchies that linearize the transformations observed in unlabeled natural video sequences. This is done by training a generative model to predict video frames. We also address the problem of inherent uncertainty in prediction by introducing latent variables that are non-deterministic functions of the input into the network architecture.

Summary:The paper addresses two main issues with the current autoencoder representations for the video frame prediction task:-- Using L2 loss to minimize the difference between the predicted and reference frame leads to blurry representations, as the network averages under uncertainty. A latent variable formulation is introduced to address this, where separate variables that are not coupled deterministically with the frame inputs are allowed to affect the output.-- It is generally difficult to learn expressive representations, as the identify function from the most recent frame is a strong predictor of the result. The paper proposes decoupling the representation into a locally stable "what" component (the weighted average of the final feature map) and a locally linear "where" component (the phase, or soft argmax of the final feature map activations). A loss function enforcing that the phase of the predicted frame can be linearly extrapolated from the phases of the previous two frames.The experimental results demonstrate mostly anecdotally that representations with the desired properties (what are where components, and latent variables) are indeed learned and positively impact video frame reconstruction quality. Quality:I did not see notable issues in the intuitions or technical implementation. The arguments are substantiated with intuitive results on a toy experimental dataset (NORB). Some questions:-- Is the beta parameter from Eqns 2 and 3 learned. If not (as it is not present in Algorithm 1 pseudocode), then how was it set and what was its effect?-- The paper mentions that "at test time  can be selected via sampling, assuming its distribution on the training set has been previously estimated". It would be helpful to visualize how sampling delta affects the results. It would not only help us visualize the uncertainty that is being learned, but also verify the hypothesis above whether sampling is a viable options of getting believable-looking frames.-- Deep architecture 3 has a bit of an unusual design. What is the intuition behind computing a fully-connected layer only to reshape it back into a convolutional layer? Does a fully-convolutional design work and if not, why not?-- The exposition mentions the L2 reconstruction errors of different deep architectures, but I did not see a table with the values anywhere (or how they compare to the other frame-reconstruction methods). It would be helpful to have numeric estimates, not just visual-anecdotal ones. Clarity:The paper is clearly written, except a couple of spots. I had trouble parsing Table1 -- while succinct, without knowing the filter sizes and phase pooling sizes it is hard to intuit what happens in the architectures. I also had trouble following some of the exact details of Sec 4.1 for the same reason -- but the deep network experiment explanations were fine. Originality:The paper has several original ideas: 1) phase pooling, including its soft-argmax computation and the associated phase linear extrapolation loss 2) the introduction of the latent variables delta. To the best of my knowledge, these are original ideas and a creative way of using autoencoder models for video frame prediction. Significance:The paper expands the kinds of auto-encoder representations that can be learned from video and provide intriguing enough results that I believe may spur interesting follow-up work.


Beyond Convexity: Stochastic Quasi-Convex Optimization
Stochastic convex optimization is a basic and well studied primitive in machine learning. It is well known that convex and Lipschitz functions can be minimized efficiently using Stochastic Gradient Descent (SGD). The Normalized Gradient Descent (NGD) algorithm, is an adaptation of Gradient Descent, which updates according to the direction of the gradients, rather than the gradients themselves. In this paper we analyze a stochastic version of NGD and prove its convergence to a global minimum for a wider class of functions: we require the functions to be quasi-convex and locally-Lipschitz. Quasi-convexity broadens the concept of unimodality to multidimensions and allows for certain types of saddle points, which are a known hurdle for first-order optimization methods such as gradient descent. Locally-Lipschitz functions are only required to be Lipschitz in a small region around the optimum. This assumption circumvents gradient explosion, which is another known hurdle for gradient descent variants. Interestingly, unlike the vanilla SGD algorithm, the stochastic normalized gradient descent algorithm provably requires a minimal minibatch size.

Summary: This paper analyzes the stochastic version of normalized version of normalized gradient descent (NGD), which is the first effort to explore the efficacy and property of stochastic normalized gradient descent (SNGD). In order to verify the benefits of NGD in training non-convex optimization problems, this paper introduces a new property, local-quasi-convexity, to prove its convergence to a global minimum. Particularly, they prove that NGD finds an \epsilon-optimal minimum for local quasi convex functions within O(1/ \epsilon^2) iterations. In addition, this paper introduces a new setup: stochastic optimization of locally-quasi convex functions, in which the gradient is estimated using a minibatch of examples. Empirically, this paper reports experimental results by training deep neural networks by comparing with the-state-of-the-arts methods, minibatch SGD and Nesterov's accelerated gradient method.Quality: This paper is technically sound without obvious mistakes. The theoretical analyses are easy to follow and achieve the expected outcome. The empirical study shows the improvement of the proposed algorithm than the existing methods. This is the first effort for the provable gradient-based algorithm for stochastic quasi-convex optimization. Nevertheless, its training model is relatively small network, that is, a single hidden layer of 100 units. This setup is very likely to be too easy to show the advantage of the algorithm, not to mention the real application of training deep neural network on large datasets. Clarity:This paper is easy to follow and well structured. Several typos do exist, however. For example, line 37: 'SGD is guaranteed find' should be 'SGD is guaranteed to find'; line 134: 'Inforamlly' should be 'Informally'; line 185: 'its is' should be 'it is'; line 250: 'and and' should be 'and', etc. The draft should be better checked if accepted.  Originality: In keeping with NIPS objective of rewarding new unexplored areas and research directions (more than incremental advancement of the state of the art), this paper studies an important problem for non-convex optimization and tries the first effort on provable results, which should be scored a high score for originality. It is of significance to move the focus from stochastic convex optimization to stochastic non-convex optimization. Significance: This work studies an important research problem and is in a good research direction since stochastic non-convex optimization has shown its importance for several years and needs to be solved with solid theoretical guarantee. They define the property, local-quasi-convexity, to extend quasi-convexity and capture the unimodal functions which are not quasi-convex, and then leads to the proof the convergence to the global optimal. However, the empirical studies shown in the paper are not comprehensive or convincing to show that the proposed algorithm could work for training deep neural network since only a single layer model is presented.Other comments:The key strength of this paper is to introduce the property, local-quasi-convexity, to extend quasi-convexity and capture the unimodal functions that are not quasi-convex. They propose solutions for locally-quasi-convex optimization with NGD and strictly-locally-quasi-convex optimization with stochastic NGD with both theoretical proofs. The main concern about this paper lies in the empirical studies. I believe that the proposed algorithm aims to solve the problem of training deep neural network finally. However, in this paper, the model for training is too simple, which is just a single layer with 100 hidden units. Therefore, whether the proposed algorithm could really work for training deep neural network in practice beating SGD is yet to be verified.


Enforcing balance allows local supervised learning in spiking recurrent networks
To predict sensory inputs or control motor trajectories, the brain must constantly learn temporal dynamics based on error feedback. However, it remains unclear how such supervised learning is implemented in biological neural networks. Learning in recurrent spiking networks is notoriously difficult because local changes in connectivity may have an unpredictable effect on the global dynamics. The most commonly used learning rules, such as temporal back-propagation, are not local and thus not biologically plausible. Furthermore, reproducing the Poisson-like statistics of neural responses requires the use of networks with balanced excitation and inhibition. Such balance is easily destroyed during learning. Using a top-down approach, we show how networks of integrate-and-fire neurons can learn arbitrary linear dynamical systems by feeding back their error as a feed-forward input. The network uses two types of recurrent connections: fast and slow. The fast connections learn to balance excitation and inhibition using a voltage-based plasticity rule. The slow connections are trained to minimize the error feedback using a current-based Hebbian learning rule. Importantly, the balance maintained by fast connections is crucial to ensure that global error signals are available locally in each neuron, in turn resulting in a local learning rule for the slow connections. This demonstrates that spiking networks can learn complex dynamics using purely local learning rules, using E/I balance as the key rather than an additional constraint. The resulting network implements a given function within the predictive coding scheme, with minimal dimensions and activity. The brain constantly predicts relevant sensory inputs or motor trajectories. For example, there is evidence that neural circuits mimic the dynamics of motor effectors using internal models [1]. If the dynamics of the predicted sensory and motor variables change in time, these models may become false [2] and therefore need to be readjusted through learning based on error feedback.
From a modeling perspective, supervised learning in recurrent networks faces many challenges. Earlier models have succeeded in learning useful functions at the cost of non local learning rules that are biologically implausible [3,4]. More recent models based on reservoir computing [5][6][7] transfer the learning from the recurrent network (with now "random", fixed weights) to the readout weights. Using this simple scheme, the network can learn to generate complex patterns. However, the majority of these models use abstract rate units and are yet to be translated into more realistic spiking networks. Moreover, to provide a sufficiently large reservoir, the recurrent network needs to be large, balanced and have a rich and high dimensional dynamics. This typically generates far more activity than strictly required, a redundancy that can be seen as inefficient.
On the other hand, supervised learning models involving spiking neurons have essentially concentrated on the learning of precise spike sequences [8][9][10]. With some exceptions [10,11] these models use feed-forward architectures [12]. In a balanced recurrent network with asynchronous, irregular and highly variable spike trains, such as those found in cortex, the activity has been shown to be 1
chaotic [13,14]. This leads to spike timing being intrinsically unreliable, rendering a representation of the trajectory by precise spike sequences problematic. Moreover, many configurations of spike times may achieve the same goal [15].
Here we derive two local learning rules that drive a network of leaky integrate-and-fire (LIF) neurons into implementing a desired linear dynamical system. The network is trained to minimize the objective x(t) −x(t)
2 + H(r), Wherex(t) is the output of the network decoded from the spikes, x(t) is the desired output, and H(r) is a cost associated with firing (penalizing unnecessary activity, and thus enforcing efficiency). The dynamical system is linear,ẋ = Ax + c, with A being a constant matrix and c a time varying command signal. We first study the learning of an autoencoder, i.e., a network where the desired output is fed to the network as a feedforward input. The autoencoder learns to represent its inputs as precisely as possible in an unsupervised fashion. After learning, each unit represents the encoding error made by the entire network. We then show that the network can learn more complex computations if slower recurrent connections are added to the autoencoder. Thus, it receives the command c along with an error signal and learns to generate the outputx with the desired temporal dynamics. Despite the spike-based nature of the representation and of the plasticity rules, the learning does not enforce precise spike timing trajectories but, on the contrary, enforces irregular and highly variable spike trains.

I greatly appreciate the problem the authors are trying to solve: supervised learning is difficult to implement in a biologically plausible manner, indeed known algorithms (such as backprop through time) are typically non-local. The authors propose a neat solution: unsupervised learning of fast recurrent connections establishes a detailed balance of E and I currents, which is then shown to make relevant aspects of the global output of the network available to each neuron locally. These global-made-local signals can then be used to learn slower connections in order to implement a linear dynamical system.I find the idea really appealing, but I think the results are rather preliminary and the whole study could have been more thorough.- while the spike-based rule used to learn a detailed balance can certainly be shown to achieve its goal in an isolated postsynaptic neuron, it is by no means guaranteed to work in a recurrent network (for just about the same reasons that make local approximations to error backpropagation close to useless in a recurrent net). Why is it that it works here?- learning the slow connections requires very strong feedback (K large). The authors claim that after learning, since the error has vanished anyways, K could be set to zero without affecting performance. Is that true?Is there a figure where we can see that? If the error was exactly zero, then I would buy that, but since it's never perfect, one is allowed to wonder about the stability of the K=0 solution to small errors. Indeed the error can be reduced arbitrarily by making K very large, without even touching the slow weights, so clearly in that case a very small error isn't at all predictive of successful outputs with K=0; for a stable solution to emerge with K small, the error with K large should be further reduced (through learning) by an amount that ought to be made more precise mathematically. I guess what I'm uncomfortable with is that if the authors' claim was true, we could imagine decreasing K progressively as W_slow is being learned, but then at some point we would hit the limit of validity of the first-order expansion of (I-M/K)^{-1} used to derive Eq7 and run into trouble. Am I missing something here?- somehow, the authors postpone the difficult problem of local supervised learning to the (in my opinion, not much easier) learning of identical input, output, and error feedback weights that must all be equal to the same matrix F. Such learning is not addressed here. Could the authors discuss/comment on that? - in their derivations, the authors assume that the inputs to the dynamical system they are trying to learn are constant (or very slow changing). What happens if inputs are faster? (btw, what was the author's choice of input c in the learning setup of Fig3?)  Some minor comments/typos: - line 94: what does it mean for the feedforward weights to "evenly span the input space"?- isn't there a sign issue in Eq2? If W_ij > 0 and V_i > 0, the learning rule is going to strengthen W_ij which makes the imbalance even worse.- l175: adjust to bold, "vector font"- trying to derive Eq.7, I instead got \delta M_{ij} \approx (c_j + Kx_j) e_i. Did the authors drop the constant term on the basis of K being large?- the whole paragraph l.278-l.285 was rather obscure to me; in the end, don't the authors want the error to be fed back through F because of the Fe term in Eq. 9? Couldn't that be stated more simply/clearly?- l221: different from - ref list Gutig - Guetig of Guetig - ref list Deneve - Deneve - caption of fig3B - replace top/bottom by left/right - l387 the the- l403 different fromUPDATE following author's responses: I'm updating my evaluation to 8 due to a satisfying response to my second comment in particular.


Taming the Wild: A Unified Analysis of HOGWILD!-Style Algorithms
Stochastic gradient descent (SGD) is a ubiquitous algorithm for a variety of machine learning problems. Researchers and industry have developed several techniques to optimize SGD's runtime performance, including asynchronous execution and reduced precision. Our main result is a martingale-based analysis that enables us to capture the rich noise models that may arise from such techniques. Specifically, we use our new analysis in three ways: (1) we derive convergence rates for the convex case (HOGWILD!) with relaxed assumptions on the sparsity of the problem; (2) we analyze asynchronous SGD algorithms for non-convex matrix problems including matrix completion; and (3) we design and analyze an asynchronous SGD algorithm, called BUCKWILD!, that uses lower-precision arithmetic. We show experimentally that our algorithms run efficiently for a variety of problems on modern hardware.

This paper is an essentially theoretical contribution regarding convergence rates for the so-called "Hogwild"-style algorithms for stochastic gradient descent.In these algorithms, the gradient step is produces asynchronously over different chunks of the dataset in parallel, with results updating current weights as they are completed, independent of other parallel updates.Previously, demonstrating theoretical convergence has been difficult and somewhat brittle.In this paper, the authors present an analysis framework that is able to extend convergence results to a variety of specializations/generalizations of the algorithm.They show that one of their proven variants, "Buckwild" provides significant real-world speedups by using lower precision arithmetic to compute the gradient steps.As far as the paper goes, it is generally good.I had little trouble reading and understanding the paper (I think), and they make a point to explain the maths in an intuitive fashion, insofar as it is possible.The authors do a good job placing their work in the context of other research.I admit that I'm not an expert in convergence proofs of this type, so I'll have to leave it to others to do a robust check of the correctness thereof.If the proof is correct, it seems to satisfy the usual criteria for a reasonable proof of this type (part of which is that it is by itself not sufficient to show the practical usefulness of the underlying algorithm).One concern I have is that it's not obvious where the work of others ends and the work done by the authors begins.All of the citations are there, I'm just not sure how closely related previous results are to what's being show here.For example, on p1, 51 the authors mention at least one other version of SGD that uses reduced precision arithmetic, but no details from the paper are referenced in section 3.2.Is Buckwild closely related?Are the results similar?Moreover, I could swear I remember a paper in which supermartingale methods are use to prove convergence results in some Hogwild-related paper, but neither Google nor memory is serving me well, so I'll take the authors word that they're first to this idea as they seem to claim (in any case, they should please give the literature one last look to make sure that this is so).Another concern is the broadness of the result:I'm not sure how popular fixed point arithmetic is in this area, especially considering the dominance of GPU calculations in this area.The result for the non-convex optimizations is nice, but we decide the fixed point result isn't that important, it starts to seem a bit thin to me.Of course, you often have to do limited precision calculations to use the GPU (e.g., 64 -> 32 bit floats).Do these results apply in that case?Might be a nice addition.  The empirical results are somewhat lightweight, but it's to be expected in a mostly theoretical paper.I'd like to see confidence bars on the speed comparison, and I'd like to see every thread size from 1 to 24 if they can manage it.Why does the 32-bit version slow down at high numbers of threads?Figure 1b. seems unnecessary to me:I'm willing to take the authors word for it that the two algorithms converge to roughly the same answers in roughly the same fashion.Table 1 is kind of a fun result by itself which I'd love to see explored more, maybe in a different paper.It seems crazy to me that we can throw out a whole bunch of information and have virtually no impact on performance.Honestly, let's just use a single bit and see what happens.Could you reference other papers that show something similar here?Incidentally, Googling related terms brings up this paper both at arvix and at the author's homepage.Is this what we're doing now?At a minimum, it makes blind review kind of meaningless.


Measuring Sample Quality with Stein's Method
To improve the efficiency of Monte Carlo estimation, practitioners are turning to biased Markov chain Monte Carlo procedures that trade off asymptotic exactness for computational speed. The reasoning is sound: a reduction in variance due to more rapid sampling can outweigh the bias introduced. However, the inexactness creates new challenges for sampler and parameter selection, since standard measures of sample quality like effective sample size do not account for asymptotic bias. To address these challenges, we introduce a new computable quality measure based on Stein's method that bounds the discrepancy between sample and target expectations over a large class of test functions. We use our tool to compare exact, biased, and deterministic sample sequences and illustrate applications to hyperparameter selection, convergence rate assessment, and quantifying bias-variance tradeoffs in posterior inference.

The main contribution of the paper is a proposal for IPM with certain properties and geared towards a specific application. The IPM proposed is based on Stein operator, which is one of the standard methods, by now, for studying convergence in distribution. The motivating application is measuring the quality of biased MCMC procedures. The assumption are: (i) the expectation under the target distribution is zero (which is made because of the Stein's method) (ii) the operator used is the infinitesimal generator of Markov process and (iii) the set of functions in defining the IPM is what is called as 'classic Stein set' in page 3 of the paper. The main contributions of the paper are 1) Theorem 7 (and theorem 2) relating Stein and (Wasser)stein IPMs for log-concave densities under certain assumptions in the multivariate setting. 2) Providing computable optimization programs for Stein discrepancy - I especially liked the use ofWhitney's extension theorem for computing a graph based discrepancy. It would be interesting to explore rates of convergence results for Stein discrepancy rigorously. Furthermore, it would be interesting to explore relationship to MMD (as one of the main focus in on computability). Using MMD in this context would correspond to goodness-of-fit style testing for target distribution (say P) using MMD. The authors mention MMD would work when 'expected kernel evaluations are easily computed under the target'. But since the space of rkhs function is dense in L_2(P) essentially it could approximate a large class of functions (not in the sample sense as mentioned in the paper but in a population setting). I went over the proofs and they seem to be correct at least at a high level. The contributions of this paper are interesting and I would recommend accepting the paper. I think extending and exploring ideas along the lines of this paper should lead to interesting results.


Learning to Transduce with Unbounded Memory
Recently, strong results have been demonstrated by Deep Recurrent Neural Networks on natural language transduction problems. In this paper we explore the representational power of these models using synthetic grammars designed to exhibit phenomena similar to those found in real transduction problems such as machine translation. These experiments lead us to propose new memory-based recurrent networks that implement continuously differentiable analogues of traditional data structures such as Stacks, Queues, and DeQues. We show that these architectures exhibit superior generalisation performance to Deep RNNs and are often able to learn the underlying generating algorithms in our transduction experiments.

This paper presents neural stack, queue, and neural deque data structures with the aim of drawing the 'attention' of the network to the relevant memory/information in the past. More specifically, the state of the data structure (its memory content and the strength signal) is updated by control signals from a 'controller'. The controller is the core part of the model which 'reads' the relevant information from the past from the neural data structure, and combines it with its previous state and the input to generate the output and the control signal to change the neural data structure content for the next step. The paper presents experimental results on synthetic data showing that the neural data structures provide suitable structured memory to draw the 'attention' of the model to the relevant part of the history. The experiments are performed across a range of transduction tasks, and compared with LSTM as the baseline.I like the paper and I believe it present interesting neural data structures to implement tractable structured memory networks. However, there are some issues there: - The presentation of the model is not good, i.e. it is very difficult to understand. Authors need to improve the presentation to make it understandable for more people.- Authors claim the neural Turing machines are intractable, hence justifying the need for more restricted but tractable models of computation. They need to show the intractability of neural Turing machines and the tractability of neural data structures on some simple experiments to back up their claim.- Authors need to compare in more details with the Facebook work, both in the 'Related Work' section AND in the experiments section. I would very much like to see how these works are compared empirically. - Why the number of baseline LSTM layers is different across tasks and how it has been specified for each task? - Finally, authors need to show the usefulness of the neural data structures on real life datasets, even small. For example, they could have used BTEC for machine translation, if they cannot run the models on large bilingual corpora.I would very much appreciate authors' comments on the above issues.


Smooth and Strong: MAP Inference with Linear Convergence
Maximum a-posteriori (MAP) inference is an important task for many applications. Although the standard formulation gives rise to a hard combinatorial optimization problem, several effective approximations have been proposed and studied in recent years. We focus on linear programming (LP) relaxations, which have achieved state-of-the-art performance in many applications. However, optimization of the resulting program is in general challenging due to non-smoothness and complex non-separable constraints. Therefore, in this work we study the benefits of augmenting the objective function of the relaxation with strong convexity. Specifically, we introduce strong convexity by adding a quadratic term to the LP relaxation objective. We provide theoretical guarantees for the resulting programs, bounding the difference between their optimal value and the original optimum. Further, we propose suitable optimization algorithms and analyze their convergence.

Overview: This paper studies the benefits of augmenting the linear programming relaxation of the maximum a-posteriori (MAP) inference problem in graphical models with a quadratic term, thereby achieving strong convexity.Such augmented formulations are obtained both from the original primal and dual formulations, and in each case the resulting primal-dual relationship is studied.Prior work has mostly focused on smoothing the LP formulation using a softmax/entropy term, with a few notable exceptions, such as [5], [17] and [18].Rather than those previous approaches, which employ a quadratic term in the sub-problems of either a *proximal* or a *alternating direction* scheme, in the present manuscript, the quadratic smoothing term is added directly.This can in some way be seen as a naive approach: In comparison to proximal or alternating direction schemes, convergence to the global optimum of the original problem is no longer guaranteed, and the approximation quality directly depends on the strength of the augmentation term.The aforementioned approaches adjust the strength in an adaptive manner and thus find the global optimum, at least in the limit.However, the formulations derived from this direct augmentation are very insightful and reveal several interesting connections to other formulations.Moreover, the authors demonstrate in several experiments (both synthetic and real-world), that a particular optimization scheme for the primal form of the augmented dual (based on block-coordinate Frank-Wolfe) compares very favorably to other practical schemes if a commensurate strength of the quadratic penalty is chosen.The Frank-Wolfe method has recently seen numerous useful applications in machine learning, and this is yet another interesting example.Positive points: + The paper is exceptionally well-written and organized. Related work is cited and the authors do an excellent job at connecting the various existing approaches and putting them into perspective. + The suggested optimization scheme (FW) seems practical.It is reasonably efficient and has the advantage that only maximization (as opposed to max-marginalization, or even softmax-marginalization) is needed, which allows for the use of certain structured potentials. + The experimental evaluation is unusually thorough for a paper of this type, involving even inference experiments based on real-world learned structured prediction models. + The paper seems to be technically correct as far as I can tell. + Some of the derivations are novel to my knowledge, and the established optimization scheme employes the recently suggested block-coordinate Frank-Wolfe scheme for simplex-constrained quadratic optimization in a novel context. + Table 1 is extremely helpful in connecting the dots.Negative points: - From a convex optimization point of view, the proposed scheme (FW) is less advanced than either of [5], [17], and [18]. - In fact, as the authors point out, their augmented formulation is also not entirely novel, but very closely related to the sub-problem in [17] (or the one in [5], if the primal is directly augmented).As such, some of the derivations and bounds in the manuscript also build on material developed in previous work.A comment: The quadratic programming relaxation suggested by Kumar [4] is also a simplex-constrained convex quadratic program.It might be interesting to extend the discussion in the paper to highlight which, if any, similarities exist between the soft-constrained primal formulation suggested in this manuscript and Kumar's convex QP relaxation - in particular from an approximation quality point of view. (Note that by adding a quadratic augmentation term, as suggested in the present manuscript, the approximation also becomes one that differs from the original LP relaxation.)


Probabilistic Variational Bounds for Graphical Models
Variational algorithms such as tree-reweighted belief propagation can provide deterministic bounds on the partition function, but are often loose and difficult to use in an "any-time" fashion, expending more computation for tighter bounds. On the other hand, Monte Carlo estimators such as importance sampling have excellent any-time behavior, but depend critically on the proposal distribution. We propose a simple Monte Carlo based inference method that augments convex variational bounds by adding importance sampling (IS). We argue that convex variational methods naturally provide good IS proposals that "cover" the target probability, and reinterpret the variational optimization as designing a proposal to minimize an upper bound on the variance of our IS estimator. This both provides an accurate estimator and enables construction of any-time probabilistic bounds that improve quickly and directly on state-of-the-art variational bounds, and provide certificates of accuracy given enough samples relative to the error in the initial bound.

The authors propose using an importance sampling procedure based on proposal distributions obtained by running TRW/BP to estimate the partition function of graphical models.They demonstrate experimentally that such methods are effective at constructing good estimates of the partition function for easy instances but are also useful for more challenging instances (though at the price of higher sample complexity).The paper is well-written and more or less self-contained.General thoughts:-There isn't much deep in the way of theoretical content here, so it would have been nice to see how IS with a TRW/BP proposal performs versus other common proposals that are used to estimate partition functions (there is quite a bit of work in this area).The authors make a good point in their rebuttal that this approach does provide bounds which is arguably more useful than simple convergence rates.-Having to actually run TRW/BP to convergence to get the proposal seems unfortunate, and in practice, these methods aren't necessarily particularly fast (especially for large models).Once you do this, you need a large number of additional samples (if the model is hard) to get tight bounds.So, I'm not sure how practical this approach really is.-Z_TRW can be an exponentially bad bound (especially for very frustrated models).This means that \Delta could be quite large.Do you have some sense as to how the number of samples scales with the number of nodes and interaction strength on a single highly frustrated cycle?-On page 7, you mention that LBP is a lower bound on these models and cite [26].Sudderth et al. [26] showed only part of the claimed result under certain conditions on LBP fixed points.The result you are claiming was actually proved later on in:  N. Ruozzi.The Bethe partition function of log-supermodular graphical models.Advances in Neural Information Processing Systems (NIPS), December 2012.Typos:-pg. 1, "optimization and Monte Carlo" -> "optimization and Monte Carlo". -pg. 1, "mean field type" -> "mean-field type". -pg. 2, "important advantages:" -> "important advantages". -pg. 3,It is helpful for readers when expectations use a subscript to denote the distribution with which the expectation is being taken. -pg. 8, "state-of-the-art variational bounds," -> "state-of-the-art variational bounds"


Optimization Monte Carlo: Efficient and Embarrassingly Parallel Likelihood-Free Inference
We describe an embarrassingly parallel, anytime Monte Carlo method for likelihood-free models. The algorithm starts with the view that the stochasticity of the pseudo-samples generated by the simulator can be controlled externally by a vector of random numbers u, in such a way that the outcome, knowing u, is deterministic. For each instantiation of u we run an optimization procedure to minimize the distance between summary statistics of the simulator and the data. After reweighing these samples using the prior and the Jacobian (accounting for the change of volume in transforming from the space of summary statistics to the space of parameters) we show that this weighted ensemble represents a Monte Carlo estimate of the posterior distribution. The procedure can be run embarrassingly parallel (each node handling one sample) and anytime (by allocating resources to the worst performing sample). The procedure is validated on six experiments.

Likelihood-free inference refers to methods used when it is expensive or impossible to evaluate the likelihood function and which instead perform inference by simulating data from the model. Existing methods, such as ABC, simulate data from many parameter values and accept only close matches to the observed data. This is wasteful as most simulations are typically poor matches. This paper proposes Optimisation Monte Carlo (OMC), a more efficient approach which views the simulator as a function f(,u) where  is the unknown parameters and u is a vector of U(0,1) draws. The idea is perform many iterations of the following: draw a u vector, minimise ||y-f(,u)|| (distance between observations and simulator output) to find the best , and weight it according to the the volume of nearby  values providing reasonably good fits. This weight is shown to depend on the Jacobian of f.OMC is an appealing novel and simple strategy. The paper is clearly written, original and seems technically correct. There are also several examples in which OMC convincingly beats state-of-the-art alternatives. However I have some doubts about significance: the examples are for fairly simple models and I am not convinced the approach will succeed for more complicated scenarios. In particular, I think in all the examples f(,u) is a smooth function of . If the simulator involves branching decisions affected by  this will not necessarily be the case and optimisation could become a lot more difficult, as it may be necessary to search every branch.A few other points are as follows:* I'd be interested to know how expensive a typical optimisation step is and whether it affects the runtime per sample of OMC compared to ABC methods. * For some models I imagine only a subset of u space is interesting and it would be nice to have a method which learns and concentrates on this. This is not really a criticism of OMC, as ABC methods don't do this either. * It might be interesting to comment on links with "Efficient likelihood-free Bayesian Computation for household epidemics" by Peter Neal, which seems to share some of the ideas used in OMC.Finally some possible typos:* Pg 3, "volume proportional" -> "volume is proportional" * Pg 4 equation (8), is hat needed on first f? * Pg 5, "this problem is" -> "this problem has" * Pg 6, "is simply the" -> "is simply"


Bounding errors of Expectation-Propagation
Expectation Propagation is a very popular algorithm for variational inference, but comes with few theoretical guarantees. In this article, we prove that the approximation errors made by EP can be bounded. Our bounds have an asymptotic interpretation in the number n of datapoints, which allows us to study EP's convergence with respect to the true posterior. In particular, we show that EP converges at a rate of O(n −2 ) for the mean, up to an order of magnitude faster than the traditional Gaussian approximation at the mode. We also give similar asymptotic expansions for moments of order 2 to 4, as well as excess Kullback-Leibler cost (defined as the additional KL cost incurred by using EP rather than the ideal Gaussian approximation). All these expansions highlight the superior convergence properties of EP. Our approach for deriving those results is likely applicable to many similar approximate inference methods. In addition, we introduce bounds on the moments of log-concave distributions that may be of independent interest.

The paper derives inference bounds for the Expectation-Propagation (EP) algorithm in variational inference. The results are developed for log-concave distributions with slowly-changing logs, relating the finite-sample bounds to these obtained via Bernstein-von Mises limits for the Canonical Gaussian Approximation (CGA). Kullback-Leibler divergence costs of EP versus CGA are also discussed. EP is a popular method that works well in practice but comes with few theoretical guarantees, so these developments are of interest.There are of course several directions in which these results could be improved, which the authors already discuss. In practice, the current results are still restrictive, but they are certainly a step in the right direction. My main discussion point is that EP is often more powerful for prediction than inference, but the paper only discusses inference approximation bounds. I expect that, even in cases when the sites are not log-concave, EP could perform badly in inference but reasonably in prediction. In that sense, the current bounds are not as interesting as they could be. Some minor points:* the notation $\phi_p$ to denote $\sum \phi_i$ is a bit confusing. * many of the formulae don't have punctuation marks at the end.* The authors could cite "Expectation propagation as a way of life" by Andrew Gelman, Aki Vehtari, Pasi Jylaenki, Christian Robert, Nicolas Chopin, John P. Cunningham. * The references should be cleaned up.


Bayesian Manifold Learning: The Locally Linear Latent Variable Model
We introduce the Locally Linear Latent Variable Model (LL-LVM), a probabilistic model for non-linear manifold discovery that describes a joint distribution over observations, their manifold coordinates and locally linear maps conditioned on a set of neighbourhood relationships. The model allows straightforward variational optimisation of the posterior distribution on coordinates and locally linear maps from the latent space to the observation space given the data. Thus, the LL-LVM encapsulates the local-geometry preserving intuitions that underlie non-probabilistic methods such as locally linear embedding (LLE). Its probabilistic semantics make it easy to evaluate the quality of hypothesised neighbourhood relationships, select the intrinsic dimensionality of the manifold, construct out-of-sample extensions and to combine the manifold model with additional probabilistic models that capture the structure of coordinates within the manifold.

The paper introduces a model which is probabilistic for non linear manifold discovery. It is based on a generative model with missing variables and required a variational EM implementation which is standard but nevertheless technical to derive in this specific context.As regards the bibliography, I can see at least two relevant references missing: [1] A. Deleforge, F. Forbes and R. Horaud, High-Dimensional Regression with Gaussian Mixtures and Partially-Latent Response Variables. Statistics and Computing.[2] A. Deleforge, F. Forbes, S. Ba and R. Horaud, Hyper-spectral image analysis with Partially-Latent Regression and spatial Markov dependencies. IEEE journal of selected topics in signal processing 2015.Note that [2] is an extension of [1] with a Markov neighborhood and a VEM approach to high-to-low dimensional mapping. In terms of results, the authors seem to stop their EM algorithm after a rather small (30-40) number of iterations without any mention of any more formal convergence or stoppping criterion. Why is it so? This looks a little suspicious. Do the results degrade when more iterations are done? In figure 3 B, I cannot see the mentioned graph structure? Comparison with GP-LVM: this paragraph is missing a clear conclusion at the end. I'd suggest the authors explain better what they conclude from this comparison. Fig 5 F: is the difference between the first 3 methods actually significant? Fig 5-A: The authors mention multiple initializations of EM but what is their conclusion wrt the sensitivity of their algorithm to initialization? Fig 6: I would need more information as why the authors think that the result of their method, although better than the others, is satisfying. It seems to me, at least visually, that the recovered mapping is quite far from the ground truth one.In their conclusion, the authors mention that their model permits evaluation of hypothetical neighborhood but I do not see where in the paper this is illustrated.


Expectation Particle Belief Propagation
We propose an original particle-based implementation of the Loopy Belief Propagation (LPB) algorithm for pairwise Markov Random Fields (MRF) on a continuous state space. The algorithm constructs adaptively efficient proposal distributions approximating the local beliefs at each note of the MRF. This is achieved by considering proposal distributions in the exponential family whose parameters are updated iterately in an Expectation Propagation (EP) framework. The proposed particle scheme provides consistent estimation of the LBP marginals as the number of particles increases. We demonstrate that it provides more accurate results than the Particle Belief Propagation (PBP) algorithm of [1] at a fraction of the computational cost and is additionally more robust empirically. The computational complexity of our algorithm at each iteration is quadratic in the number of particles. We also propose an accelerated implementation with sub-quadratic computational complexity which still provides consistent estimates of the loopy BP marginal distributions and performs almost as well as the original procedure.

The authors propose a variation on particle belief propagation in which the proposal distribution is adaptively selected at each step of the algorithm by using an expectation-propagation like calculation: the particle messages are mapped to an exponential family distribution (Gaussian for the experiments).The idea is simple but appealing.The authors show that it is more efficient than a previously proposed MCMC procedure for resampling the particles (whose inner loop is comparatively slow and requires user selection or tuning), and a "fixed" proposal based on EP which is not updated during the particle message passing.The experiments are sufficient to support the claims but could have been stronger.The authors assess the convergence behavior of the algorithm in very simple models (few variables; mildly multimodal or skewed potentials; amenable to simple discretization for comparison), with a simple synthetic denoising example as well.While sufficient to demonstrate the point of the paper, that the EP-like proposals improve convergence in the number of samples, a practical application or example would have strengthened the submission.The authors also mention a practical speed-up based on Briers et al. (2010), which uses subsampling to reduce the computational overhead.While potentially useful, I did not see a significant contribution here.Also, its usefulness and appropriateness will likely depend on the model and number of samples being used.Given that fact, the simple empirical assessment was not strongly compelling. Overall, the paper provides a simple but appealing idea for designing proposals during particle BP.However, it could also benefit from a stronger and more detailed experimental evaluation, in more compelling or practical domains.


Statistical Model Criticism using Kernel Two Sample Tests
We propose an exploratory approach to statistical model criticism using maximum mean discrepancy (MMD) two sample tests. Typical approaches to model criticism require a practitioner to select a statistic by which to measure discrepancies between data and a statistical model. MMD two sample tests are instead constructed as an analytic maximisation over a large space of possible statistics and therefore automatically select the statistic which most shows any discrepancy. We demonstrate on synthetic data that the selected statistic, called the witness function, can be used to identify where a statistical model most misrepresents the data it was trained on. We then apply the procedure to real data where the models being assessed are restricted Boltzmann machines, deep belief networks and Gaussian process regression and demonstrate the ways in which these models fail to capture the properties of the data they are trained on.

The function that attains the supremum of the statistic used in MMD two-sample tests can be used to identify regions in attribute space in which a statistical model most misrepresents the data used for automatic induction. This "witness function" can be interpreted as the difference of two kernel density estimates one corresponding to the actual data and one to the data generated with the fitted model. The authors then apply the procedure to real and synthetic learning problems and illustrate (rather than demonste, as the authors claim) the ways in which automatically induced models, such as restricted Boltzmann machines, deep belief networks or Gaussian processes regressors fail to capture the properties of the data on which they are trained.The paper is very well written and is a pleasure to read. However the originality and significance of the contribution is rather limited. Part of the problem stems from the lack of a clear focus: it is presented as a "an exploratory approach" that touches on many aspects of Statistical Model Criticism but fails to provide sufficient background or depth in the discussion of the issues addressed. This would be resolved in a longer version that provides a more detailed discussion.  According to the authors, the main contribution of this work is highlighting the importance for model criticism of the (known) property that the witness function in MMD two-sample tests is the difference of two kernel density estimates. This releveance apparently stems from the fact that the MMD statistic is the supremum among a large space o possible statistics. Leaving aside the connection with MMD, the witness function is simply the difference of kernel density estimates. If one were asked to identify regions in which a statistical model most misrepresents the data it was trained on, this method is probably the most obvious choice. In criticising a model one should have the use of the model in mind. Therefore, model evaluation ought to be guided by a utility function. I agree with the authors that focusing on the differences of kernel density estimates for the actual data and for data generated by the fitted model is probably an excellent general purpose model criticism tool, especially if I am interested in local differences. I would argue (rather trivially) that the important feature is the local character of RBF kernels. The maximization property is reassuring, albeit not of much practical use:The crucial issue seems to be the choice of kernel and kernel parameters. Unfortunately, in MMD two-sample tests, these need to be specified by the user.In summary, the paper is based on a fairly straightforward result: that differences of kernel densitie estimates help us understand how and where a model fails to fit the data. The experimental investigation is promising but seems to be rather preliminary and unsystematic. On the question posed by the authors: would it be worth this paper being published now as a conference paper to bring model criticism to the attention of the machine learning community quickly or waiting a year or two for a review article?Yes, it would be worth waiting.   A minor comment:"While it is clear that simple methods will fail to capture all of the structure in this data, it is not clear a priori how much better the more advanced methods will fair."In line 297, the authors probably meant "will fare".


Practical and Optimal LSH for Angular Distance
We show the existence of a Locality-Sensitive Hashing (LSH) family for the angular distance that yields an approximate Near Neighbor Search algorithm with the asymptotically optimal running time exponent. Unlike earlier algorithms with this property (e.g., Spherical LSH [1, 2]), our algorithm is also practical, improving upon the well-studied hyperplane LSH [3] in practice. We also introduce a multiprobe version of this algorithm and conduct an experimental evaluation on real and synthetic data sets. We complement the above positive results with a fine-grained lower bound for the quality of any LSH family for angular distance. Our lower bound implies that the above LSH family exhibits a trade-off between evaluation time and quality that is close to optimal for a natural class of LSH functions.

The authors propose a novel LSH family for angular distance which (a) matches the theoretical guarantees of Spherical LSH (i.e., an asymptotically optimal runtime exponent) while at the same time (unlike Spherical LSH) being practical in that they outperform Hyperplane LSH for the same task by up to an order of magnitude.This result is achieved by the combination of techniques where (a) a data point is first hashed using a random rotation and then the after-image matched to the closest point y from {+/-e_i}, where e_i is the ith standard basis vector of \R^d, (b) the random rotation is performed as a sequence of 3 pseudo-random rotations (reducingthe asymototic overhead from O(d^2) to O(d log d), (c) feature hashing to reduce the effective dimensionality d, and (d) using a variant of multi-probe LSH for retrieval which generates additional probing locations.The proposed work is novel, original and has the rare property that it combines theoretical optimality with a practical, realistic (C++ instead of MATLAB) implementation.