Dual Swap Disentangling
Learning interpretable disentangled representations is a crucial yet challenging task. In this paper, we propose a weakly semi-supervised method, termed as Dual Swap Disentangling (DSD), for disentangling using both labeled and unlabeled data. Unlike conventional weakly supervised methods that rely on full annotations on the group of samples, we require only limited annotations on paired samples that indicate their shared attribute like the color. Our model takes the form of a dual autoencoder structure. To achieve disentangling using the labeled pairs, we follow a "encoding-swap-decoding" process, where we first swap the parts of their encodings corresponding to the shared attribute, and then decode the obtained hybrid codes to reconstruct the original input pairs. For unlabeled pairs, we follow the "encoding-swap-decoding" process twice on designated encoding parts and enforce the final outputs to approximate the input pairs. By isolating parts of the encoding and swapping them back and forth, we impose the dimension-wise modularity and portability of the encodings of the unlabeled samples, which implicitly encourages disentangling under the guidance of labeled pairs. This dual swap mechanism, tailored for semi-supervised setting, turns out to be very effective. Experiments on image datasets from a wide domain show that our model yields state-of-the-art disentangling performances.

This papers presents a model that elicits interpretable disentangled representations from weakly supervised data. It does so through a dual-autoencoder model fit for both unlabelled and labelled data, where labelled data assumes knowledge of which  part of the representation should be the shared one. The core of the model is the operation of swapping the shared part of the representation, when data is labelled, and dual-swapping of randomly selected parts of the representation in the case of unlabelled data.   Strengths - the method makes sense, and is a well-thought combination of existing approaches (swapping, dual mechanism and parts of representations as labels) - it presents a nice way to add possibly softer labels, e.g. which part of the representation should be shared, as opposed to stronger labels (without forcing semantics)  Weaknesses - the presented results do not give me the confidence to say that this approach is better than any of the other due to a lot of ad-hoc decisions in the paper (e.g. digital identity part of the code vs full code evaluation, the evaluation itself, and the choice of the knn classifier) - the results in table 1 are quite unusual - there is a big gap between the standard autoencoders and the variational methods which makes me ask whether there’s something particular about the classifier used (knn) that is a better fit for autoencoders. the particularities of the loss or the distribution used when training. why was k-nn used? a logical choice would be a more powerful method like svm or a multilayer perceptron. there is no explanation for this big gap - there is no visual comparison of what some of the baseline methods produce as disentangled representations so it’s impossibly to compare the quality of (dis) entanglement and the semantics behind each factor of variation - the degrees of freedom among features of the code seem binary in this case, therefore it is important which version of vae and beta-vae, as well as infogan are used, but the paper does not provide those details - the method presented can easily be applied on unlabeled data only, and that should have been one point of comparison to the other methods dealing with unlabeled data only - showing whether it works on par with baselines when no labels are used, but that wasn’t done. the only trace of that is in the figure 3, but that compares the dual and primary accuracy curves (for supervision of 0.0), but does not compare it with other methods - though parts of the paper are easy to understand, in whole it is difficult to get the necessary details of the model and the training procedure (without looking into the appendix which i admittedly did not do, but then, i want to understand the paper fully (without particular details like hyperparameters) from the main body of the paper). i think the paper would benefit from another writing iteration   Questions: - are the features of the code binary? because i didn’t find it clear from the paper. if so, then the effects of varying the single code feature is essentially a binary choice, right? did the baseline (beta-)vae and infogan methods use the appropriate distribution in that case? i cannot find this in the paper - is there a concrete reason why you decided to apply the model only in a single pass for labelled data, because you could have applied the dual-swap on labelled data too - how is this method applied at test time - one needs to supply two inputs? which ones? does it work when one supplies the same input for both? - 57 - multi dimension attribute encoding, does this essentially mean that the projection code is a matrix, instead of a vector, and that’s it? - 60-62 - if the dimensions are not independent, then the disentanglement is not perfect - meaning there might be correlations between specific parts of the representation. did you measure/check for that? - can you explicitly say what labels for each dataset in 4.1 are - where are they coming from? from the dataset itself? from what i understand, that’s easy for the generated datasets (and generated parts of the dataset), but what about cas-peal-r1 and mugshot? - i find the explanation in 243-245 very unclear. could you please elaborate what this exactly means  - why is it 5*3 (and not 5*2, e.g. in the case of beta-vae where there’s a mean and stdev in the code) - 247 - why was k-nn used, and not some other more elaborate classifier? what is the k, what is the distance metric used? - algorithm 1, ascending the gradient estimate? what is the training algorithm used? isn’t this employing a version of gradient descent (minimising loss)? - what is the effect of the balance parameter? it is a seemingly important parameter, but there are no results showing a sweep of that parameter, just a choice between 0.5 and 1 (and why does 0.5 work better)? - did you try beta-vae with significantly higher values of beta (a sweep between 10 and 150 would do)?  Other: - the notation (dash, double dash, dot, double dot over inputs is a bit unfortunate because it’s difficult to follow) - algorithm 1 is a bit too big to follow clearly, consider revising, and this is one of the points where it’s difficult to follow the notation clearly - figure 1, primary-stage I assume that f_\phi is as a matter of a fact two f_\phis with shared parameters. please split it otherwise the reader can think that f_\phi accepts 4 inputs (and in the dual-stage it accepts only 2) - figure 3 a and b are very messy / poorly designed - it is impossible to discern different lines in a because there’s too many of them, plus it’s difficult to compare values among the ones which are visible (both in a and b). log scale might be a better choice. as for the overfitting in a, from the figure, printed version, i just cannot see that overfitting - 66 - is shared - 82-83 Also, require limited weaker…sentence not clear/gramatically correct - 113 - one domain entity - 127 - is shared - 190 - conduct disentangled encodings? strange word choice - why do all citations in parentheses have a blank space between the opening parentheses and the name of the author? - 226 - despite the qualities of hybrid images are not exceptional - sentence not clear/correct - 268 - is that fig 2b or 2a? - please provide an informative caption in figure 2 (what each letter stands for)  UPDATE: I’ve read the author rebuttal, as well as all the reviews again, and in light of a good reply, I’m increasing my score.  In short, I think the results look promising on dSprites and that my questions were well answered. I still feel i) the lack of clarity is apparent in the variable length issue as all reviewers pointed to that, and that the experiments cannot give a fair comparison to other methods, given that the said vector is not disentangled itself and could account for a higher accuracy, and iii) that the paper doesn’t compare all the algorithms in an unsupervised setting (SR=0.0) where I wouldn’t necessarily expect better performance than the other models.


Beauty-in-averageness and its contextual modulations: A Bayesian statistical account
Understanding how humans perceive the likability of high-dimensional "objects" such as faces is an important problem in both cognitive science and AI/ML. Existing models generally assume these preferences to be fixed. However, psychologists have found human assessment of facial attractiveness to be context-dependent. Specifically, the classical Beauty-in-Averageness (BiA) effect, whereby a blended face is judged to be more attractive than the originals, is significantly diminished or reversed when the original faces are recognizable, or when the blend is mixedrace/mixed-gender and the attractiveness judgment is preceded by a race/gender categorization, respectively. This "Ugliness-in-Averageness" (UiA) effect has previously been explained via a qualitative disfluency account, which posits that the negative affect associated with the difficult race or gender categorization is inadvertently interpreted by the brain as a dislike for the face itself. In contrast, we hypothesize that human preference for an object is increased when it incurs lower encoding cost, in particular when its perceived statistical typicality is high, in consonance with Barlow's seminal "efficient coding hypothesis." This statistical coding cost account explains both BiA, where facial blends generally have higher likelihood than "parent faces", and UiA, when the preceding context or task restricts face representation to a task-relevant subset of features, thus redefining statistical typicality and encoding cost within that subspace. We use simulations to show that our model provides a parsimonious, statistically grounded, and quantitative account of both BiA and UiA. We validate our model using experimental data from a gender categorization task. We also propose a novel experiment, based on model predictions, that will be able to arbitrate between the disfluency account and our statistical coding cost account of attractiveness.

Beauty-in-averageness and its contextual modulations: A Bayesian statistical account  --Summary--  The paper introduces a model to explain the beauty-in-averageness and ugliness-in-averageness phenomena: The more "typical" a face, the more attractive it is. Typicality of an object is in some sense its information complexity with respect to an underlying model, perhaps conditioned on a specific context.  The paper proposes a specific model that validates the proposed explanation for the phenomena, as shown by an empirical study.  A face is modeled as a vector of features, which is in turn a mixture of Gaussians: Some Gaussians correspond to features for prototypical faces, whereas other Gaussians correspond to typical features for different categories. This model makes per-category average faces easy to represent, and also faces closer to prototypes. On the other hand, a face that deviates from either prototypes or categories is harder to encode---say an average of two largely distinct category-average faces.  The paper proposes an explanation for previously unexplained phenomena that have been subject interest in previous work, and in this sense it seems interesting to me.  The claims of the paper are somewhat imprecise and should be clarified, especially so that it is clear what is novel in the paper (see the first comment in the comments section). The main contribution of the paper is a model for faces that explains data & phenomena observed in other studies. The claims that this model is a plausible _predictive_ model for existing data is satisfactorily backed up by the empirical study.  The paper is otherwise clear about what it is doing, the results are clear and I am convinced that they can be reproduced. The paper seems a bit vague about potential impact of the result (especially at the discussion in the end), and that should be clarified.  --Comments--  Reference 43 seems to quite clearly apply to BiA, and seems more related to the paper than suggested in the literature review. In particular, the idea that compressibility and attractiveness are related is not new and is also present in 43. From the purely abstract view of the model proposed in the submitted paper, there is very little difference between category prototypes and individual prototypes (familiar faces), which somewhat suggests the hypothesis that closeness to prototypes dictates attractiveness (and the beauty-vs-ugliness in average is an observation of presence or absence of average prototype).  I found myself confused by the paper's assumptions on the data model---not what they are, but the fact that they are assumptions. It would be more clear to make no assumptions about what is really happening, but rather say "we chose to model the random variable X as ...". In this way, the reader can understand that the conclusions are not vacuous when reality does not conform to the model used. Indeed, the data sort of conforms to the model used and that is enough for the model to be deemed explanatory of different phenomena. In contrast, if the language "we assume that X ~ ..." is used, the reader will either read this as "we model X as" or be confused, asking theirself "what if the assumption is violated?"  The paper's contributions to explaining BiA and UiA are interesting, but the paper is a bit vague about other ways the work can have impact (lines 233-246). In particular, the proposed model is effectively a mixture of Gaussians, so it should be clarified how the paper's contributions can be informative to modeling human preferences in other domains, before what mixtures of Gaussians already give us.  If space can be spared, I suggest that the paper have some comments about what the current model may fail to capture. For example, we see asymmetry in Fig. 4-e, and we see that uncertainties do not match in Fig. 2-b&c. This will naturally outline directions for future work and will be constructive to the paper's empirical study.  The recognition model in Sec. 4.1.1 may be further simplified, by restricting the h_r from the outset to what is described in the "Approximation" paragraph. The point is that if the model used for the empirical study collapses to that, then there is no need to describe "intermediate" model ideas. (One exception is if the intermediate model is more common in the literature, but in this case it would be informative to make a remark about this commonality and about the choice of not using this common model in favor or a simpler one.)  For the response, please clarify how your predictive model can have impact in future work. Relevance and potential impact are in this reviewer's opinion an important factor for the 8-10 score band.  --Minor comments--  [57] induce a larger [137] |G| = 2 [179] There is a dot below the legend of Figure 1. [193] contains a basis that describes [225] ``liking'' [Fig 3-d] the distribution is similar to a mixture of normals - Please fix your citation formatting. There are some titles written in all capital letters, and different formats for the same proceedings (including past NIPS proceedings). - The paper seems to make no distinction between non-beauty and ugliness, but that seems to bear a difference in related work, so perhaps it would be useful to remark that Ugliness in Average may be conflated with "Non-Beauty in Average".   -- Post Feedback Response --  Thank you for the response. I am happy with the submission and the motivation in the response, and I think they enrich the paper. After the discussion, I am maintaining my original score.


Accelerated Stochastic Matrix Inversion: General Theory and Speeding up BFGS Rules for Faster Second-Order Optimization
We present the first accelerated randomized algorithm for solving linear systems in Euclidean spaces. One essential problem of this type is the matrix inversion problem. In particular, our algorithm can be specialized to invert positive definite matrices in such a way that all iterates (approximate solutions) generated by the algorithm are positive definite matrices themselves. This opens the way for many applications in the field of optimization and machine learning. As an application of our general theory, we develop the first accelerated (deterministic and stochastic) quasi-Newton updates. Our updates lead to provably more aggressive approximations of the inverse Hessian, and lead to speed-ups over classical non-accelerated rules in numerical experiments. Experiments with empirical risk minimization show that our rules can accelerate training of machine learning models.

This paper presents an accelerated version of the sketch-and-project algorithm, an accelerated algorithm for matrix inversion and accelerated variants of deterministic and stochastic quasi-Newton updates. I strongly believe that this line of research is of interest for the ML and optimization communities, and that the algorithms and theoretical results presented in this paper are significant and novel. Moreover, the numerical results presented in the paper clearly illustrate the effectiveness of the approaches presented in the paper. For this reason, I strongly recommend this paper for publication. Below I list my minor concerns with the paper.  - The paper is clearly written and easy to read, however, I believe that in several places there are grammatical errors or minor typos. These are easy to fix. A few examples are: — Line 25: liner system in each iteration -> linear system at each iteration — Line 27: in particular the BFGS -> in particular BFGS — Line 116: which in term -> which in turn — Line 122-123: Sentence starting with “This is a very general problem…” has a grammatical issue. — Line 142: in the special case -> for the special case — Line 247: inverting a matrix symmetric positive matrix A -> inverting a symmetric positive definite matrix A — Line 263-264: Sentence starting with “We also observe from…” is not clear. — Line 266: as fast as nonaccelerated algorithm -> as fast as the nonaccelerated algorithm  - I strongly believe that a Final Remarks and Future Work (Extensions) section is missing from the paper. This section would help the reader summarize the contributions of the paper and understand the significance and possible extensions of this work.  - Other Minor Issues: — The authors should explain how alpha, beta and gamma are chosen in the Algorithms, and why these choices were made.  — Line 207: Issue with reference to (4.2)? — Line 230: Issue with equation reference? — Reference [32] and [33] is the same — Reference section lacks consistency (names of authors)   - I believe that there are some missing references in the paper.  — stochastic quasi-Newton methods: 1) A self-correcting variable-metric algorithm for stochastic optimization (http://proceedings.mlr.press/v48/curtis16.pdf), 2) A multi-batch l-bfgs method for machine learning (http://papers.nips.cc/paper/6145-a-multi-batch-l-bfgs-method-for-machine-learning.pdf) — subsampled Newton methods: 1) Sub-sampled Newton methods I: globally convergent algorithms (https://arxiv.org/pdf/1601.04737.pdf), 2) Newton-type methods for non-convex optimization under inexact hessian information (https://arxiv.org/pdf/1708.07164.pdf), 3) On the use of stochastic hessian information in optimization methods for machine learning (https://pdfs.semanticscholar.org/ba21/5739785c21704dc2a6c8fa235c52cab0f45d.pdf), (4) Convergence rates of sub-sampled Newton methods (https://arxiv.org/abs/1508.02810)


Constrained Generation of Semantically Valid Graphs via Regularizing Variational Autoencoders
Deep generative models have achieved remarkable success in various data domains, including images, time series, and natural languages. There remain, however, substantial challenges for combinatorial structures, including graphs. One of the key challenges lies in the difficulty of ensuring semantic validity in context. For example, in molecular graphs, the number of bonding-electron pairs must not exceed the valence of an atom; whereas in protein interaction networks, two proteins may be connected only when they belong to the same or correlated gene ontology terms. These constraints are not easy to be incorporated into a generative model. In this work, we propose a regularization framework for variational autoencoders as a step toward semantic validity. We focus on the matrix representation of graphs and formulate penalty terms that regularize the output distribution of the decoder to encourage the satisfaction of validity constraints. Experimental results confirm a much higher likelihood of sampling valid graphs in our approach, compared with others reported in the literature.

This paper investigates a method for learning deep generative models of graphs, subject to constraints of the types that arise in many practical situations of interest. Rather than specify the model in such a way that it explicitly encodes valid graphs, instead a Lagrangian is formulated which penalizes generation of graphs which violate the constraints.   In a VAE setting, the constraints themselves only apply to the generated graphs (x) not the latent variables (z). To ensure the constraints hold over the entire domain, the constraints are evaluated via Monte Carlo integration with values separately simulated from the prior p(z), rather than the values drawn from the approximating posterior q(z|x).  The overall idea is fairly straightforward — Lagrange multipliers are the standard approach for converting constrained to unconstrained optimization problems. The most interesting aspect of this paper is in Theorem 1, which makes explicit that regularization of the VAE objective in this manner is appropriate. However, the exact form used is not justified particularly well in the main text. ** In particular: why in eq (6) would one consider the marginalization over the squared constraint, rather than the marginalization itself? Is this necessary for Theorem 1 to hold? The use of a ramp function in equation (7) seems a bit of a hack. ** If this were addressed more clearly, or better motivated, I would adjust my score upwards.  There is also an overloading of notation which becomes a bit confusing, for the functions g_i(…, …). In equations (4) and (6), these are functions of x and z, while in e.g. equation (7) g_i is a function of theta and z. It would be good to separate the process which transforms a sampled z into a graph x (a process parameterized by theta) from the definitions of the constraints themselves, which typically seem to be only a function of the simulated x, not directly of z (the constraints are on graph properties, not on z).  The experimental validation is fairly minimal, and does not compare against other graph generation methods, but rather only against VAEs which produce SMILES strings. I would think there should be some baselines for graph-based methods available for some of the proposed tasks. (For example, would it be possible to define exponential random graph models (ERGMs) which would include the constraints explicitly as likelihood terms, at the expense of easily simulating graphs?) As an example, the arXiv paper https://arxiv.org/abs/1805.09076 considers similar molecule generation tasks, but includes a couple explicitly graph-based baselines.


Learning safe policies with expert guidance
We propose a framework for ensuring safe behavior of a reinforcement learning agent when the reward function may be difficult to specify. In order to do this, we rely on the existence of demonstrations from expert policies, and we provide a theoretical framework for the agent to optimize in the space of rewards consistent with its existing knowledge. We propose two methods to solve the resulting optimization: an exact ellipsoid-based method and a method in the spirit of the "follow-the-perturbed-leader" algorithm. Our experiments demonstrate the behavior of our algorithm in both discrete and continuous problems. The trained agent safely avoids states with potential negative effects while imitating the behavior of the expert in the other states.

Learning from demonstrations usually faces an ill-posed problem of inferring the expert reward functions. To facilitate safe learning from demonstrations, the paper formulates a maximin learning problem over a convex reward polytope, in order to guarantee that the worst possible consistent reward would yield a policy that is not much worse than optimal. The assumption is that the reward is linear in known features. The authors proposed two method: (i) ellipsoid method and (ii) follow-the-perturbed leader using separation oracles and a given MDP solver. The experiment is done in a grid world setting, and a modified version of the cart-pole problem.   One piece that I failed to understand is from line 61-63: why apprenticeship learning won't return the same policy as the proposed solution? This seems like a major point for me, as the formulation looks very similar to Syed & Schapire 2008 (although they used Multiplicative Weights Update as the online learning procedure. There they also used linear reward assumption, and reduction of policy learning to feature expectation matching, similar to Abbeel & Ng 2004). Since the contrast to previous method was not discussed beyond this paragraph, I'm curious what the comparison would look like in the experiment? It seems that this discussion is missing from the experiment section. I think the paper will benefit from at least some additional intuition in the experiment / discussion sections into what's wrong with Syed & Schapire method (or also Apprenticeship Learning via Linear Programming, where iteratively invoking the oracles is not needed).   The other concern, as perhaps is also clear to the authors and other readers, is the scalability of the algorithms. Repeatedly invoking separation oracle and MDP solver may mean that this method will have limited applicability to higher dimensionalities.   Gridworld experiment: How many environments did you use for training? How many demonstrations? Cartpole experiment: how well would the method work without the addition of the two question blocks?  This is a relatively minor point, but I think the focus of the paper has a lot more to do with robust policy learning (due to worst case guarantee) than safe learning, which to me implies safe exploration / satisfying safety criteria during training in addition to high reward learning.   The paper is nicely written and well-organized.


Training deep learning based denoisers without ground truth data
Recently developed deep-learning-based denoisers often outperform state-of-the-art conventional denoisers, such as the BM3D. They are typically trained to minimize the mean squared error (MSE) between the output image of a deep neural network and a ground truth image. In deep learning based denoisers, it is important to use high quality noiseless ground truth data for high performance, but it is often challenging or even infeasible to obtain noiseless images in application areas such as hyperspectral remote sensing and medical imaging. In this article, we propose a method based on Stein's unbiased risk estimator (SURE) for training deep neural network denoisers only based on the use of noisy images. We demonstrate that our SURE-based method, without the use of ground truth data, is able to train deep neural network denoisers to yield performances close to those networks trained with ground truth, and to outperform the state-of-the-art denoiser BM3D. Further improvements were achieved when noisy test images were used for training of denoiser networks using our proposed SURE-based method. Code is available at https://github.com/Shakarim94/Net-SURE.
in medical imaging, ultra high resolution 3D MRI (sub-millimeter resolution) often requires several hours of acquisition time for a single, high quality volume, but reducing acquisition time leads to increased noise. In X-ray CT, image noise can be substantially reduced by increasing the radiation dose. Recent studies on deep learning based image denoisers [19,20] used CT images generated with normal doses as the ground truth so that denoising networks would be able to be trained to yield excellent performance. However, increased radiation dose leads to harmful effects in scanned subjects, while excessively high doses may saturate the CT detectors (e.g., in a similar manner to the acquisition of a photo of the sun without any filter). Thus, acquiring ground truth data with newly developed CT scanners seems challenging without compromising the subjects' safety.
Conventional denoising methods do not usually require noiseless ground truth images to perform denoising, but often require them for tuning parameters of image filters to elicit the best possible results (minimum MSE). In order to identify the optimal parameters of conventional denoisers without ground truth data, several works have been conducted [21,22] using Stein's unbiased risk estimator (SURE) [23], which is an unbiased MSE estimator. For the popular non-local means (NLM) filter [24], the analytical form of SURE was used to optimize the denoiser performance [25,26,27]. For denoisers whose analytical forms of SURE are not available, Ramani et al. [28] proposed a MonteCarlo-based SURE (MC-SURE) method to determine near-optimal denoising parameters based on the brute-force search of the parameter space. Deledalle et al. [29] investigated the approximation of a weak gradient of SURE to optimize parameters using the quasi-Newton algorithm. However, since this method requires the computation of the weak Jacobian, it is not applicable to high dimensional parameter spaces, such as deep neural networks.
We propose a SURE-based training method for deep neural network denoisers without ground truth data. In Section 2, we review key results elicited from SURE and MC-SURE. Subsequently, in Section 3, we describe our proposed method using MC-SURE and a stochastic gradient for training deep learning based image denoisers. In Section 4, simulation results are presented for (a) conventional state-of-the-art denoiser (BM3D), (b) deep learning based denoiser trained with BM3D as the ground truth, (c) the same deep neural network denoiser with the proposed SURE training without the ground truth, and (d) the same denoiser network with ground truth data as a reference. Section 5 concludes this article by discussing several potential issues for further studies.

The submission "Training deep learning denoisers without ground truth data" transfers methods from risk estimation to the setting of convolutional neural networks for denoising. The usual minimization of the l2-loss between ground truth and training data is replaced by the minimization over an unbiased estimator over training data, sampled in a Monte-Carlo fashion. The submission highlights how previous techniques for unbiased parameter estimation can be translated into the CNNs and shows very intriguing results, training without ground truth data.  A missing aspect that has to be addressed is the existence of minimizer of the SURE estimator (equation (13)) - it is easy to contruct simple (e.g. linear) networks and corresponding training data such that no minimizer \theta of (13) exists. The function value is not necessarily bounded from below, and the infimum over (13) becomes minus infinity. How can such cases be excluded, either by assumptions on the data / the number of free parameters, or by additional regularization on theta?  As minor comments  - I think it would be fair to state that the interesting idea of including test data during training also increases the inference speed significantly (e.g. to the 15 hours mentioned). - Along the lines of including the test data during training, the authors could train on as few as a single test image, yielding an interesting connection to the recent work on "Deep Image Priors" by Ulyanov et al. - Possibly related prior work on using the SURE estimator for denoising can be found, e.g., in Donoho and Johnstone, "Adapting to unknown smoothness via Wavelet Shrinkage" or Zhang and Desai, "Adaptive Denoising based on SURE risk".   - It should be noted, that although  equation (13) is an unbiased risk estimator for the MSE loss for fixed parameters theta, the minimimum over theta is not necessarily an unbiased estimation of the MSE loss of the estimator parametrized by this minimizer, see, e.g., section 1.3 in Tibsharani and Rosset's "Excess Optimism: How Biased is the Apparent Error of an Estimator Tuned by SURE?". - Please have your submission proof-read for English style and grammar issues with a particular focus on missing articles.  - I am wondering if completing the square in equation (14), ariving at a term \|y^{(j)} + \frac{\sigma^2}{\epsilon} \tilde{n}^{(j)} - h(y^{(j)}; \theta)\|^2 and one remaining linear term provides an intering point of view. What happens if one just trains with the above quadratic term? This is just a question out of curiosity and does not have to be addressed.      In summary, the work is interesting and the ideas are clearly explained. The experiments are detailed and well-formulated and show promising results. The paper is, in general, well structured and easy to read. I recommend the acceptance of this manuscript.     I appreciate the clarifications of the authors in their rebuttal. Although it is a minor technical issue (typically avoided e.g. by weight decay), I'd like to point out that the existence of minimizers is not only a question of the network architecture, but also of the training data. I agree with the other reviewers that the limitation to (fixed) Gaussian noise is severe in practice, but I'm willing to accept such a limitation as this is one of the very first papers to train denoising networks without any ground truth. Because the general idea of the paper is very interesting and the first results are promising, I keep my score and recommend the acceptance of this manuscript.


Provably Correct Automatic Subdifferentiation for Qualified Programs
The Cheap Gradient Principle [Griewank and Walther, 2008] -the computational cost of computing the gradient of a scalar-valued function is nearly the same (often within a factor of 5) as that of simply computing the function itself -is of central importance in optimization; it allows us to quickly obtain (high dimensional) gradients of scalar loss functions which are subsequently used in black box gradient-based optimization procedures. The current state of affairs is markedly different with regards to computing subderivatives: widely used ML libraries, including TensorFlow and PyTorch, do not correctly compute (generalized) subderivatives even on simple examples. This work considers the question: is there a Cheap Subgradient Principle? Our main result shows that, under certain restrictions on our library of nonsmooth functions (standard in nonlinear programming), provably correct generalized subderivatives can be computed at a computational cost that is within a (dimension-free) factor of 6 of the cost of computing the scalar function itself.

In this submission, the authors consider the problem of computing sub-differentiation for a class of non-smooth functions automatically and correctly. They give a very nice example that illustrates problems with current automated differentiation frameworks, such as tensorflow and pytorch. Then, the authors prove a chain rule for the one-sided directional derivative of a composite non-smooth function satisfying certain assumptions. Based on this rule, the authors derive a (randomized) algorithm for computing such a derivative for a particular kind of programs only with constant overhead. The algorithm is very similar to the one for back-ward automatic differentiation except that its forward computation is based on the newly-proved chain rule in the submission, rather than the standard chain rule for differentiation.  I vote for the acceptance of the submission. Although the message in the submission is unlikely to change the practice of machine-learning engineers, it will at least make them be aware of potential pitfalls arising from differentiating ReLU or other non-smooth functions using tensorflow and pytorch. In the past, I annoyed my colleagues multiple times with the question: why is it ok to differentiate a non-smooth function like ReLU as in tensorflow or pytorch? I got a wide range of answers, such as justification based on sub-differentiation and argument based on measure-zero non-differentiability, but all of them were slightly ad-hoc. This submission seems to give a principled answer and to expose a line of past research and new results on this question. I learned a lot by reading the paper. I feel that other participants in NIPS would experience the same.  * I think that it is better to give the definition of Clark sub-differential of f in the paper or in the appendix.  * Assumption 3.1: How flexible is this requirement?   * Don't Theorem 3.2 and Corollary 3.1 need Assumption 3.1?  * Why do you have to sample v1,...,vd in Algorithm 6 uniformly in the sphere? If not, what would go wrong?  * line 60: software is has ===> software has  * line 101: it computes ===> computes  * line 120: L is used here without definition. Say that it is a set of library functions.  * line 146: a function a function ===> a function  * line 181: allowed use ===> allowed to use  * formula right after line 187: -g(v) ===> -g(x)  * formula right after line 214: partial[f_2;v] ===> partial[f;v]  * line 218: algorithm, that ===> algorithm that  * I found it difficult to follow Example 3.3.


Quantifying Learning Guarantees for Convex but Inconsistent Surrogates
We study consistency properties of machine learning methods based on minimizing convex surrogates. We extend the recent framework of Osokin et al. [14] for the quantitative analysis of consistency properties to the case of inconsistent surrogates. Our key technical contribution consists in a new lower bound on the calibration function for the quadratic surrogate, which is non-trivial (not always zero) for inconsistent cases. The new bound allows to quantify the level of inconsistency of the setting and shows how learning with inconsistent surrogates can have guarantees on sample complexity and optimization difficulty. We apply our theory to two concrete cases: multi-class classification with the tree-structured loss and ranking with the mean average precision loss. The results show the approximation-computation trade-offs caused by inconsistent surrogates and their potential benefits.

Summary  The paper provides a new lower bound in consistency analysis of convex surrogate loss functions. Section 1 provides an introduction which discusses previous work and contributions. The main previous work is Osokin et al, which is frequently referenced in the text. The main contribution is a generalization of the results of Osokin et al to inconsistent surrogates, and a new tighter lower bound. An additional contribution is a study the behavior of the proposed bound for two prediction problems.  Section 2 provides definitions of the task loss and surrogate loss, while section 3 defines the calibration function which links them. The concept of level-eta consistency is defined. Level-0 consistency is the same as Fisher consistency, whereas with eta>0 "the surrogate is not consistent meaning that the actual risk cannot be minimized globally." Section 4 provides a new lower bound, with discussion that clarifies how the paper is novel wrt previous work. Section 5 analyzes the bounds in two problem settings: multi-class classification with tree-structured loss and ranking with mean average precision loss. Section 6 provides a discussion and conclusion.  Pros: - frequent references to previous work which help clarify the novelty. - new proof technique and lower bound. - results include a comparison with previous work (Figure 1b) in order to show a visual example of how the previous bound is trivial and the proposed bound is tighter.  Cons: - no clear discussion of how the analysis may lead to new loss functions or efficient learning algorithms.


Adversarial Examples that Fool both Computer Vision and Time-Limited Humans
Machine learning models are vulnerable to adversarial examples: small changes to images can cause computer vision models to make mistakes such as identifying a school bus as an ostrich. However, it is still an open question whether humans are prone to similar mistakes. Here, we address this question by leveraging recent techniques that transfer adversarial examples from computer vision models with known parameters and architecture to other models with unknown parameters and architecture, and by matching the initial processing of the human visual system. We find that adversarial examples that strongly transfer across computer vision models influence the classifications made by time-limited human observers.

This paper introduces an interesting experiment that tries to show that adversarial examples that transfer across recently proposed deep learning models can influence the visual classification produced by time-limited humans (at most 2.2 to 2.5 seconds for looking at the image and making the classification).  The idea is to take a set of 38 observers to classify images into 2 classes from within each of three separate groups (pets - cat or dog, hazard - snake or spider, vegetables - broccoli or cabbage).  The images are presented in different ways: 1) original image, 2) adversarial image manipulated to be classified by an ensemble classification method (of 10 recently proposed deep learning models), 3) flip, which is the original image manipulated by a vertically flipped adversarial perturbation (this is a control case to show that low signal-to-noise ratio alone does not explain poor human classification), and 4) false, which is an image outside the three groups above, manipulated by an adversarial perturbation to be classified as one of the classes within one of the three groups.  In general, results show that 1) adversarial perturbations in the false class successfully biased human classification towards the target class; and 2) adversarial perturbations cause observers to select the incorrect class even when the correct class is available (this is the 2nd manipulation described above).  I found this paper quite interesting to read and I support its publication.  I have a couple of comments that may influence the final writing of the paper.  Can there be an error due to mistakes in pressing the buttons given that the assignment of classes to buttons was randomized?  Was there some calibration phase before each experiment to rule that issue out?  Another experiment would be to measure how humans perform as time limit increases or decreases?


Learning to Solve SMT Formulas
We present a new approach for learning to solve SMT formulas. We phrase the challenge of solving SMT formulas as a tree search problem where at each step a transformation is applied to the input formula until the formula is solved. Our approach works in two phases: first, given a dataset of unsolved formulas we learn a policy that for each formula selects a suitable transformation to apply at each step in order to solve the formula, and second, we synthesize a strategy in the form of a loop-free program with branches. This strategy is an interpretable representation of the policy decisions and is used to guide the SMT solver to decide formulas more efficiently, without requiring any modification to the solver itself and without needing to evaluate the learned policy at inference time. We show that our approach is effective in practice -it solves 17% more formulas over a range of benchmarks and achieves up to 100× runtime improvement over a state-of-the-art SMT solver.

Thank you for the detailed rebuttal! I have accordingly updated my score from 6 to 7. ---------------------------- This paper proposes a framework for learning solving strategies for Satisfiability Modulo Theories (SMTs). SMT generalizes SAT, and as such can be used to model a variety of verification applications, among others. Given an instance, an SMT solver such as Z3 will select a strategy, i.e. a sequence of transformations, that will be applied step by step to the input formula, culminating in solving it and returning the answer (true or false). The choice of strategies in SMT solvers is currently heuristic or manually optimized for a set of instances. Alternatively, this paper proposes to learn strategies based on training instances and label-strategies resulting from exploration. A set of learned strategies are then synthetized into a decision tree structure than can be easily fed into the Z3 solver. Experimentally, it is shown that the proposed approach can boost the performance of Z3 significantly.  I am rather positive about this paper, hence my score of 6/10. The paper is relatively well-written, although some points could be clarified here and there. The problem this paper addresses is definitely interesting, and part of a recent effort that aims at improving constraint solvers for MIP/SAT/CSP/SMT using machine learning. I really appreciate the 2-step methodology proposed by the authors, as they do not try to ignore the advances in SMT solvers, but rather hybridize Z3 with their learned strategies. I think that the main weakness of the paper, in its current version, is the experimental results. This is not to say that the proposed method is not promising - it definitely is. However, I have some questions that I hope the authors can address.  - Time limit of 10 seconds: I am quite intrigued as to the particular choice of time limit, which seems really small. In comparison, when I look at the SMT Competition of 2017, specifically the QF_NIA division (http://smtcomp.sourceforge.net/2017/results-QF_NIA.shtml?v=1500632282), I find that all 5 solvers listed require 300-700 seconds. The same can be said about QF_BF and QF_NRA (links to results here http://smtcomp.sourceforge.net/2017/results-toc.shtml). While the learned model definitely improves over Z3 under the time limit of 10 seconds, the discrepancy with the competition results on similar formula types is intriguing. Can you please clarify? I should note that while researching this point, I found that the SMT Competition of 2018 will have a "10 Second wonder" category (http://smtcomp.sourceforge.net/2018/rules18.pdf).  - Pruning via equivalence classes: I could not understand what is the partial "current cost" you mention here. Thanks for clarifying.  - Figure 3: please annotate the axes!!  - Bilinear model: is the label y_i in {-1,+1}?  - Dataset statistics: please provide statistics for each of the datasets: number of formulas, sizes of the formulas, etc.  - Search models comparison 5.1: what does 100 steps here mean? Is it 100 sampled strategies?  - Missing references: the references below are relevant to your topic, especially [a]. Please discuss connections with [a], which uses supervised learning in QBF solving, where QBF generalizes SMT, in my understanding. [a] Samulowitz, Horst, and Roland Memisevic. "Learning to solve QBF." AAAI. Vol. 7. 2007. [b] Khalil, Elias Boutros, et al. "Learning to Branch in Mixed Integer Programming." AAAI. 2016.  Minor typos: - Line 283: looses -> loses


Learning Loop Invariants for Program Verification
A fundamental problem in program verification concerns inferring loop invariants. The problem is undecidable and even practical instances are challenging. Inspired by how human experts construct loop invariants, we propose a reasoning framework CODE2INV that constructs the solution by multi-step decision making and querying an external program graph memory block. By training with reinforcement learning, CODE2INV captures rich program features and avoids the need for ground truth solutions as supervision. Compared to previous learning tasks in domains with graph-structured data, it addresses unique challenges, such as a binary objective function and an extremely sparse reward that is given by an automated theorem prover only after the complete loop invariant is proposed. We evaluate CODE2INV on a suite of 133 benchmark problems and compare it to three state-of-the-art systems. It solves 106 problems compared to 73 by a stochastic search-based system, 77 by a heuristic search-based system, and 100 by a decision tree learning-based system. Moreover, the strategy learned can be generalized to new programs: compared to solving new instances from scratch, the pre-trained agent is more sample efficient in finding solutions.

= Summary  The paper presents a new method for reinforcement learning-based counterexample-guided loop invariant generation. First the program's source code is encoded into a vector representation by transforming it into a graph and applying a graph neural network. Then, a tree decoder is used to generate an arithmetic formula in conjunctive normal form, using attention over the encoded program.  Overall, the approach seems promising, and the shown experimental results indicate that it is an interesting alternative to standard deductive and more recently introduced data-driven CEGAR approaches to program verification. However, the paper does not actually show the core of its contribution, the smoothed reward function used to train the system, and so I believe that the submission in its current form can not be accepted. I am inviting the authors to clarify this point (cf. below for details) in the rebuttal and update the paper subsequently, and would be happy to increase my score then. [UPDATE: This happened.]  = Quality  The paper follows recent trends in the literature (programs as graphs, syntax-directed decoding of formulas, ...), but combines it with a novel reinforcement learning approach, sidestepping the issue of a lack of labeled examples and making it easier to adapt the method to new domains. However, the core of the main contribution, namely the reward function that smoothes over the very sparse, binary valid/invalid signal, is not defined in the main paper. Instead, it refers to the supplement, where the reward is again not defined. Instead, some handwavy text about the use of counterexamples is provided (for example, do their values / traces matter, or just their number? Are UNSAT cores used to identify which parts of the candidate invariant are contradictory?). Without this, the contribution is not remotely reproducible and the paper cannot be accepted.  The main text of the paper has to contain at least an approximate definition of the reward function, and the supplement should provide a full definition. Documenting the use of counterexamples is especially important, as Sect. 5 even provides an ablation study underlining their importance.  [UPDATE: The authors have provided details on the reward function in their response, and promised to include these details in a revision of their paper. Based on this, I think an interesting question for a future revision would be to experimentally valid the design of the reward function. For example, inverting the order of reward components from pre->inv->post to post->inv->pre would be interesting (corresponding to a backwards-looking style, as in Hoare logic proofs); similarly not ordering the components at all could be an interesting ablation.]  Similarly, the paper should be more honest about the runtime performance of the proposed approach. Lines 260-265 introduce the notion of calls to the backing SMT solver as a metric; which is fine to show the data efficiency of the approach, but wallclock time is interesting for actual use of an invariant generation tool in a verifier. The timeout of 12 hours used in the paper contrasts with 1 hour allowed in the SyGuS competition from which many benchmarks are taken, and the fact that the ICE-DT solver requires less than 5 _seconds_ for most benchmarks (cf. https://dl.acm.org/citation.cfm?doid=2837614.2837664). Even if the method is currently not competitive in terms of runtime, this should be stated explicitly in the paper.  [UPDATE: The authors have promised to include runtime in the next revision.]  = Clarity  The paper succeeds reasonably well in introducing basic verification concepts in the little space it has, and apart from the lack of description of the reward function is reasonably clear. I found Sect. 3.1 not particularly enlightening (especially because I don't see that it supports conclusions (1) and (2) in lines 128-131) and would rather use some of the space used for the example to go into more detail on the reward function, which is the actual contribution of the paper.  = Originality  Finding a smoothing of the reward function that actually works is a novel and interesting contribution that hasn't been provided before. The authors review most of the related literature and discuss how they are clearly differing from it, though they seem to be unaware of one related work that infers loop invariants for heap programs using a tree decoder and a graph encoding of its inputs (but uses fully supervised training on synthetic data):    Marc Brockschmidt, Yuxin Chen, Pushmeet Kohli, Siddharth Krishna, and    Daniel Tarlow. Learning Shape Analysis. In SAS'17.  = Significance  Pushing beyond existing heuristics in deductive invariant generation tools is of great importance to making program verification more generally useful; several large-scale efforts (e.g., the verified TLS stack, the seL4 kernel, ...) towards formally verified software could profit from an evolution of the presented approach to more interesting domains. [This is minor pet peeve that I have with the paper: The fact that integer invariants are interesting, but not all that useful without heap invariants is never stated]  = Minor notes - lines 31/32: Calling Octagons "hand-crafted features" seems    unnecessarily harsh. It would be more correct to call them a    restricted hypothesis space. - line 74: This statement is just wrong, for example for arbitrary    polynomials or for the very relevant domain of separation logic.    It's true for the domain of linear integer arithmetic that is    investigated in the paper, but that distinction isn't stated. - line 141: "... module summarizes what have been" - s/have/has/ - line 159: "To convert the graph into vector representation" - missing   article.


Differentially Private Robust Low-Rank Approximation
In this paper, we study the following robust low-rank matrix approximation problem: given a matrix A 2 R n⇥d , find a rank-k matrix M , while satisfying differential privacy, such that kA M k p  ↵ · OPT k (A) + ⌧, where kBk p is the entry-wise`p-norm of B and OPT k (A) := min rank(X)k kA Xk p . It is well known that low-rank approximation w.r.t. entrywise`p-norm, for p 2 [1, 2), yields robustness to gross outliers in the data. We propose an algorithm that guarantees
We study extensions to the streaming setting where entries of the matrix arrive in an arbitrary order and output is produced at the very end or continually. We also study the related problem of differentially private robust principal component analysis (PCA), wherein we return a rank-k projection matrix ⇧ such that kA A⇧k p  ↵ · OPT k (A) + ⌧.

SUMMARY: This is a theory paper proposing a new algorithm for low-rank matrix approximation (LRA) that satisfies feature-level, approximate (delta > 0) differential privacy (DP). (The 'robust' in the title simply refers to using the pointwise l_p norm with 1<=p<2 as the LRA loss function, which is a setting that does not appear to have been studied under the DP constraint before.) The paper analyses the multiplicative and additive error of the proposed algorithm.  Content-wise the paper attempts to provide a good description of (1) how the proposed algorithm fits into the landscape of existing (DP) LRA algorithms, (2) why existing algorithms fail in the 'robust' and DP setting considered here, and (3) what are the intuitions and key ideas behind the proposed algorithm. Unfortunately, the clarity of presentation is at times hindered by poor writing.  (As I'm quite new to this field (especially matrix factorisation), please excuse any erroneous comments and please feel free to directly point them out.)   MAIN COMMENTS: 1/ [significance] This could appear as another paper that takes an existing task X and simply looks at what happens when one wants to solve it under DP, proposing a new DP-X algorithm. However, I think X=LRA can be a particularly interesting problem to look at under the DP constraint, for the following reason. In DP we are constantly looking for ways to reduce the amount of noise required in order to reach a given privacy guarantee, and one folk wisdom general advice seems to be to avoid privatising high-dimensional objects (especially when some dimensions can carry little useful statistical information, but still need to be privacy-protected as we don't know a priori which dimensions these are). In studying DP-LRA one is forced to look at this question whether interest in a lower-dimensional description of the data allows one to get away with less noise being added, and this paper answers this question in the positive for the setting studied. I wonder if there might be some substance in this view and if it could help differentiate this paper from other DP-X papers.  2/ [significance] Is the proposed algorithm practical, or is it theoretical at this stage? Specifically, could you compute exact values of epsilon and delta when the algorithm is run? I can see there is an empirical evaluation in Appendix C, but I cannot find the corresponding values of epsilon and delta.  3/ [quality] The 'robustness' requirement appears to be only used as motivation for using the pointwise l_p norm with 1<=p<2 as the LRA loss function (which is known to impart some notion of robustness). However, the paper could be more explicit about the exact ranges of p in different statements. - Are all statements supposed to hold for 1<=p<2 only? (Some sentences, including one in the Abstract, specifically refer to the p=2 case as well.) - Which statements would fail for other values of p?  4/ [quality] Section 3.2 describing the extension to the streaming setting and in particular the section on local DP appears a bit too informal to me. For example, it would be helpful to state explicitly the privacy setup in the streaming setting (when are two streams of updates considered neighbouring?) and how the privacy loss would accumulate under the continuous release model. Would one simply use an (advanced) composition theorem from the DP literature, or can one do better here?  5/ [originality] To my knowledge the paper is novel, and it is well discussed against existing work, describing the differences in assumptions, theoretical results and algorithmic ideas used.  6/ [clarity] The clarity of writing could be substantially improved at places. The submission would also really benefit from just careful proof-reading for grammar (although I'm not penalising in my review for bad grammar, fixing even singular vs plural in many places would have made reading the submission easier). Please see also minor comments below for some concrete suggestions.  7/ [quality] Re line 174: Why would any privacy preserving mechanism need to necessarily privatise the matrix A? Perhaps it could instead select x using something like the Exponential mechanism, or privacy-protect C_i instead of A, or use a different approach entirely.   STRENGTHS: [+] LRA seems to be an interesting problem to study under DP, and it appears the case considered here (pointwise l_p norm loss function with 1<=p<2) had not been studied yet. [+] Content-wise the paper is on the right track in providing the reader with necessary background, explaining differences to prior work, and the key ideas in constructing the new algorithm.  WEAKNESSES: [-] Unclear whether the paper is theoretical, or if the proposed algorithm gives a meaningful privacy guarantee with reasonable accuracy. [-] The quality of the writing (and grammar) should be improved. [-] Being a theory paper first and foremost, a more suitable venue than NIPS could be a journal where the reviewing process scope would be such that the proofs (currenty in the Appendix) are also carefully peer-reviewed.   MINOR COMMENTS: (12, 76) In robust subspace learning, wouldn't it make sense to restrict the X in the definition to projection matrices as well, so that an algorithm would have a chance of having no approximation error at least when k=d? (15) Is "measure" meant in a technical sense here? Also, the problem could potentially be generalised by considering general distances d(A, B) rather than just those stemming from norms via d(A, B) = ||A-B||. (21) What is meant by a "robust" privacy guarantee? Confusing given there is another notion of robustness at play here. (23) Missing object after "de facto". (24) If the output of M is not discrete, a rigorous definition would only consider measurable sets S, rather than all subsets of the range. (41) May want to define "efficient". (46) Based on line 224, Big-O with tilde also hides poly log factors in other variables (e.g. poly log k), not just in n. (47) Definition 2 needs to specify the range of p, otherwise there is no connection to robustness. (Equation 1) Tau in this equation was called gamma in the Abstract. (52) The value of alpha here doesn't depend on the privacy parameters epsilon, delta. Is this the best alpha one can achieve even in the non-private setting, or is there a hidden privacy cost here? (59) "l_p-norm" -> "l_1-norm" (60-62) Holder's inequality only seems to provide an *upper bound* on the error when measured in terms of the l_p norm for p<2 (using a different approach, one might obtain a tighter bound). So strictly speaking the "exponential improvement" would only apply to the p=2 case, which is however outside of the robustness scope considered? (66) I can see how feature selection relates to projection matrices, but I'm not sure how projection matrices are related to representation learning, where the learned representations need not correspond to projections. Can you please clarify? (70) "Problem 1" -> "Definition 2" (91) It is said that the result of Song et al. [37] holds even when an additive error o(k(n+d)) is allowed. However, this does not seem to subsume the case considered in this submission, where an O(k(n+d)) additive error is allowed. (106) "Table 1.2" -> "Table 1" (115) The sentence is false as written. (126) Missing brackets around v_1, ..., v_n. (133, 135) Just checking, intuitively I would have swapped the definitions of Dilation and Contraction. (Lemma 7) Variable naming conflict for "p", which is used to denote two different quantities. (150) I think it's a bit unfair to say the two main approaches to DP are input and output perturbation, as many DP algorithms operate by privacy-protecting an intermediate object in the computation pipeline (e.g. DP-SGD protects gradients). (184) Do you mean "c" instead of "s"? (246-255) Could also refer to Table 1 again, to compare the result obtained in the paper being described. (261) Should say A_{i_t, j_t} instead of A_{i_{t-1}, j_{t-1}}.  ===== Update after author response and reviewer discussion =====  Thank you for the responses to our reviews. I'm happy with the answers to our technical comments, and I would like to suggest incorporating some of these clarifications into the next revision. Adding a comment on motivating the neighbouring relation (as requested by fellow Reviewer #1) would be particularly helpful. I don't think the quality of writing was good enough for NIPS in the submitted version of the paper, but I believe this can be fixed in the next revision. Please do have your paper proof-read for style and grammar issues. Assuming these changes are made, I've increased my score from 5 to 6.


L4: Practical loss-based stepsize adaptation for deep learning
We propose a stepsize adaptation scheme for stochastic gradient descent. It operates directly with the loss function and rescales the gradient in order to make fixed predicted progress on the loss. We demonstrate its capabilities by conclusively improving the performance of Adam and Momentum optimizers. The enhanced optimizers with default hyperparameters consistently outperform their constant stepsize counterparts, even the best ones, without a measurable increase in computational cost. The performance is validated on multiple architectures including dense nets, CNNs, ResNets, and the recurrent Differential Neural Computer on classical datasets MNIST, fashion MNIST, CIFAR10 and others.

The paper proposes a scheme for adaptive choice of learning rate for stochastic gradients descent and its variants. The key idea is very simple and easy to implement: given the loss value L at the global minimum, L_min, the idea is to choose learning rate eta, such that the update along the gradient reaches L_min from the current point i.e. solving L(theta-eta*v)=L_min in eta, where v is for example dL/dtheta in gradient descent. Due to nonlinearity of L, the authors suggest linearizing L around theta: L(theta)+(-eta*v)^T dL/dtheta=L_min which yields eta=(L(theta)-L_min)/(v^T dL/dtheta). Finally, to make the adaptive learning rate pessimistic to the possible linearization error, the authors introduce a coefficient alpha, so the effective learning rate used by the optimizer is eta*alpha.  The authors empirically show (on badly conditioned regression, MNIST, CIFAR-10, and neural computer) that using such adaptive scheme helps in two ways: 1. the optimization performance is less sensitive to the choice of the coefficient alpha vs the learning rate (in non-adaptive setting), and 2. the optimizer can reduce the loss faster or at worst in equal speed with commonly used optimizers.  At the same time, the paper has some shortcomings as admitted by the authors: 1. There is no theoretical justification for the proposed scheme. 2. Despite training faster, the resulted model may generalize worse than models trained with non-adaptive step size.  In spite of the above shortcomings, I think this is a nice paper which may motivate further research around the proposed idea and eventually create an impact.  Some comments that may improve the manuscript: 1. I think providing a graphical illustration on the trajectories taken by this scheme vs plain gradient descent can be very insightful and maybe provide some intuition why the idea works. Perhaps this can be shown on a simple task: a poorly conditioned quadratic function. We know that in this case the gradient may not necessarily point toward the minimum. So it is interesting to visually inspect how by merely modifying the step size, and yet moving along the gradient directions over this poorly condition surface, the adaptive step size method can reach the minimum faster. 2. For CIFAR-10 graphs (Figure 5), what happens after 250 epochs? Is it that all methods eventually converge to the same loss value or some methods get stuck at larger values?  One major concern I have about this paper is the way L_min is used. Is L_min defined locally (i.e. the minimal point along the gradient ray only) or globally (over the entire space)? I suppose it is global, because the notation does not show any dependency of L_min on theta. If it is defined globally, then how would you justify solving L(theta-eta*v)=L_min, because this equation may not even have a solution( and I think often it does not!). For example, for a poorly conditioned quadratic, the direction of gradient may not necessarily point toward the minimum. Hence, no matter how much you move along the gradient direction, you never reach the minimum value L_min. Therefore, It seems the idea of solving  L(theta-eta*v)=L_min has some fundamental problem.  POST REBUTTAL COMMENTS: I read author's response. I was not convinced about L_min justification. Specially, the fact that setting L_min to "0" did not work shows the crticality of L_min for the algorithm to work. However, I am also not aware of a cheap workaround, so I believe the setting is okay although not idea. I keep my rating as it was.


Optimal Algorithms for Continuous Non-monotone Submodular and DR-Submodular Maximization
In this paper we study the fundamental problems of maximizing abcontinuous non-monotone submodular function over a hypercube, with and without coordinatewise concavity. This family of optimization problems has several applications in machine learning, economics, and communication systems. Our main result is the first 1 2 -approximation algorithm for continuous submodular function maximization; the approximation factor of 1 2 is the best possible for algorithms that use only polynomially many queries. For the special case of DR-submodular maximization, i.e., when the submodular functions is also coordinate-wise concave along all coordinates, we provide a faster 1 2 -approximation algorithm that runs in almost linear time. Both of these results improve upon prior work [Bian et al., 2017a,b,  Soma and Yoshida, 2017, Buchbinder et al., 2012. Our first algorithm is a single-pass algorithm that uses novel ideas such as reducing the guaranteed approximation problem to analyzing a zero-sum game for each coordinate, and incorporates the geometry of this zero-sum game to fix the value at this coordinate. Our second algorithm is a faster single-pass algorithm that exploits coordinate-wise concavity to identify a monotone equilibrium condition sufficient for getting the required approximation guarantee, and hunts for the equilibrium point using binary search. We further run experiments to verify the performance of our proposed algorithms in related machine learning applications.
This paper concerns a different well-studied setting, where L is a hypercube (i.e., [0,1] n ), with componentwise maximum and minimum serving as the join and meet, respectively. 3 We consider the fundamental problem of (approximately) maximizing a continuous and nonnegative submodular function over the hypercube. 4 The function F is given as a "black box" and can only be accessed by querying its value at a point. We are interested in algorithms that use at most a polynomial (in n) number of queries. We do not assume that F is monotone (otherwise the problem is trivial).
We next briefly mention four applications of maximizing a non-monotone submodular function over a hypercube that are germane to machine learning and other related application domains.

The paper is about maximizing non-monotone continuous submodular functions on the hypercube [0,1]^n. The main result of the paper is an algorithm that provides a tight (1/2-eps)-optimal solution using O(n^2/eps) complexity. The algorithm is built on the double-greedy algorithm proposed by Buchbinder et al for maximizing non-monotone submodular set functions, however, it turns out that a number of novel ideas have to be added to extend the algorithm to continuous submodular functions (e.g. the coordinate-wise zero-sum game strategy to decide the value of the final solution at each coordinate). A 1/2-optimal algorithm was already known for DR-submodular functions and a 1/3-opt efficient algorithm was known for continuous submodular functions, both based on the double-greedy method, however, I find the case of general continuous submodular functions much more difficult than maximizing non-monotone DR-submodular functions. All in all, after a careful read, I find the results of the paper quite novel and I recommend acceptance.  I don't have any major comments and only have a few suggestions. First of all, the dependency to the dimension n, i.e.  n^2, can perhaps be improved by a more intelligent (and adaptive over rounds using submodularity and lazy updates) discretization method. Second, it's worth to mention how many function computations your algorithm needs (i.e. is it O(n/eps) or O(n^2/eps)--I suspect that the latter is the case ).


New Insight into Hybrid Stochastic Gradient Descent: Beyond With-Replacement Sampling and Convexity
As an incremental-gradient algorithm, the hybrid stochastic gradient descent (HS-GD) enjoys merits of both stochastic and full gradient methods for finite-sum problem optimization. However, the existing rate-of-convergence analysis for HSGD is made under with-replacement sampling (WRS) and is restricted to convex problems. It is not clear whether HSGD still carries these advantages under the common practice of without-replacement sampling (WoRS) for non-convex problems. In this paper, we affirmatively answer this open question by showing that under WoRS and for both convex and non-convex problems, it is still possible for HSGD (with constant step-size) to match full gradient descent in rate of convergence, while maintaining comparable sample-size-independent incremental first-order oracle complexity to stochastic gradient descent. For a special class of finite-sum problems with linear prediction models, our convergence results can be further improved in some cases. Extensive numerical results confirm our theoretical affirmation and demonstrate the favorable efficiency of WoRS-based HSGD.

This paper lies in the well-developed field of variance-reduced stochastic gradient algorithms. It proposes a theoretical study of the well-known HSGD for sampling without replacement scheme, which is known to perform better than sampling with replacement in practice.  The considered setting makes the analysis of the algorithm more difficult, since in this case the mini-batch gradients are not unbiased anymore. Rates for strongly-convex, non-strongly convex and non-convex objectives are provided, with a special care for linearly structured problems (such as generalized linear models). Numerical experiments are convincing enough, although the datasets used in experiments are not very large (largest one is covtype) (while authors argue that this methods is interesting in the large n medium precision setting). I think that this is an nice piece of work, which provides an incremental contribution to the already widely developed theory of stochastic optimisation algorithms for ML. The paper can certainly be accepted for publication, although I think that the contribution is mostly incremental, since (to the best of my knowledge) the proofs do not seem to introduce a major novelty.


Deep Neural Nets with Interpolating Function as Output Activation
We replace the output layer of deep neural nets, typically the softmax function, by a novel interpolating function. And we propose end-to-end training and testing algorithms for this new architecture. Compared to classical neural nets with softmax function as output activation, the surrogate with interpolating function as output activation combines advantages of both deep and manifold learning. The new framework demonstrates the following major advantages: First, it is better applicable to the case with insufficient training data. Second, it significantly improves the generalization accuracy on a wide variety of networks. The algorithm is implemented in PyTorch, and the code is available at https://github.com/ BaoWangMath/DNN-DataDependentActivation.

This paper develops a new data-dependent output activation function base on interpolation function. It is a nonparametric model based on a subset of training data. The activation function  is defined in an implicit manner by solving a set of linear equations. Therefore, it cannot be solved directly by backpropagation. Instead it proposes an auxiliary network with linear output to approximate the gradient. Experiment results show that it can improve the generalization ability on a variety of networks. And it also improves the performance and robustness when the data size is small. The strength of the paper is that the proposed algorithm consistently improves performance by combining nonparametric method with DNN. More importantly, it improves the robustness of DNN for small data case. On the other hand, the weakness of the paper is that the complexity of using nonparametric approach is usually high and it requires storage of the template samples. It seems that the algorithm need to store a lot of template samples in experiments (half of the training set)? It is not clear how the number of required template samples scale with problem size. Furthermore, the argument in lines 96-102 for developing the approximate approach for training the model is a bit heuristic.   Comments: *If we are using so many template data samples, why can’t we learn a parametric interpolation function by optimizing the objective function (1)?  *Figure 1(b) is not correct. The groundtruth label Y should not be used as input to the model during testing time. *In line 75, “is need to be” should be “need to be”. *In the equation above line 93, it is not explained how \hat{X}^{te} is obtained. From Algorithm 1, it can be conjectured that it is obtained by passing X^{te} through DNN and buffer layer. It should be clarified after this equation in the main text. *The discussion in Section 3 is not clear. And the discussion about (5) is unclear and cannot be understood. *In the caption of Figure 3, “plot the generation” should be “plot the generalization”. *In line 179, “(1-400 and 406-805 epochs corresponds to linear activation)”. I think 406-805 should be the part after switching to WNLL? *In lin 199, why mean is used? Why not use mean and standard deviation? *In line 232, “tTis” should be “This”.


Generalized Cross Entropy Loss for Training Deep Neural Networks with Noisy Labels
Deep neural networks (DNNs) have achieved tremendous success in a variety of applications across many disciplines. Yet, their superior performance comes with the expensive cost of requiring correctly annotated large-scale datasets. Moreover, due to DNNs' rich capacity, errors in training labels can hamper performance. To combat this problem, mean absolute error (MAE) has recently been proposed as a noise-robust alternative to the commonly-used categorical cross entropy (CCE) loss. However, as we show in this paper, MAE can perform poorly with DNNs and challenging datasets. Here, we present a theoretically grounded set of noise-robust loss functions that can be seen as a generalization of MAE and CCE. Proposed loss functions can be readily applied with any existing DNN architecture and algorithm, while yielding good performance in a wide range of noisy label scenarios. We report results from experiments conducted with CIFAR-10, CIFAR-100 and FASHION-MNIST datasets and synthetically generated noisy labels.
log p(e y = e y n |x n , ✓)
p(e y = e y n |y = i)p(y = i|x n , ✓)), (1) where c represents number of classes, e y represents noisy labels, y represents the latent true labels and p(e y = e y n |y = i) is the (e y n , i)'th component of the confusion matrix. Usually, the real confusion matrix is unknown. Several methods have been proposed to estimate it [11,14,32,17,12]. Yet, accurate estimations can be hard to obtain. Even with the real confusion matrix, training with the above loss function might be suboptimal for DNNs. Assuming (1) a DNN with enough capacity to memorize the training set, and (2) a confusion matrix that is diagonally dominant, minimizing the cross entropy with confusion matrix is equivalent to minimizing the original CCE loss. This is because the right hand side of Eq. 1 is minimized when p(y = i|x n , ✓) = 1 for i = e y n and 0 otherwise, 8 n.
In the context of support vector machines, several theoretically motivated noise-robust loss functions like the ramp loss, the unhinged loss and the savage loss have been introduced [5,38,27]. More generally, Natarajan et al. [29] presented a way to modify any given surrogate loss function for binary classification to achieve noise-robustness. However, little attention is given to alternative noise robust loss functions for DNNs. Ghosh et al. [10,9] proved and empirically demonstrated that MAE is robust against noisy labels. This paper can be seen as an extension and generalization of their work.
Another popular approach attempts at cleaning up noisy labels. Veit et al. [39] suggested using a label cleaning network in parallel with a classification network to achieve more noise-robust prediction. However, their method requires a small set of clean labels. Alternatively, one could gradually replace noisy labels by neural network predictions [33,36]. Rather than using predictions for training, Northcutt et al. [31] offered to prune the correct samples based on softmax outputs. As we demonstrate below, this is similar to one of our approaches. Instead of pruning the dataset once, our algorithm iteratively prunes the dataset while training until convergence.
Other approaches include treating the true labels as a latent variable and the noisy labels as an observed variable so that EM-like algorithms can be used to learn true label distribution of the dataset [41,18,37]. Techniques to re-weight confident samples have also been proposed. Jiang et al. [16] used a LSTM network on top of a classification model to learn the optimal weights on each sample, while Ren, et al. [34] used a small clean dataset and put more weights on noisy samples which have gradients closer to that of the clean dataset. In the context of binary classification, Liu et al. [24] derived an optimal importance weighting scheme for noise-robust classification. Our method can also be viewed as re-weighting individual samples; instead of explicitly obtaining weights, we use the softmax outputs at each iteration as the weightings. Lastly, Azadi et al. [2] proposed a regularizer that encourages the model to select reliable samples for noise-robustness. Another method that uses knowledge distillation for noisy labels has also been proposed [23]. Both of these methods also require a smaller clean dataset to work.
3 Generalized Cross Entropy Loss for Noise-Robust Classifications

I acknowledge that I read the author's response, and I feel that the revised version of the manuscript will be even stronger. As a result, I am raising my score to an 8. This paper is definitely in the Top 50%.  -----  Summary:  This paper shows that using the negative Box-Cox transformation as a loss function ($L_q$ or Lq loss) generalizes both categorical cross-entropy (CCE) and mean absolute error (MAE) losses and offers an elegant way to trade off their respective virtues when training on noisy labels. The key insight comes from analyzing the loss function gradients: they are equivalent, except that CCE includes a term that implicitly assigns higher weights to incorrect predictions. This makes training with CCE faster than with MAE but also makes it more susceptible to overfitting label noise. Like CCE, the gradient of Lq loss yields a weighting term but with an exponent parameter that we can choose. When q=0, we get CCE, and when q=1, the weighting term disappears, which is equivalent to MAE. The paper shows that a variant of a known risk bound for MAE under uniform label noise applies to Lq loss as q approaches 1. Experimental results are noticeably strong: Lq consistently outperforms CCE and MAE both and is competitive with several alternative strong baselines.  My current feeling is that this paper is a pretty clear accept, but I look forward to reading the other reviewers' comments and authors' responses.  Quality:  This submission appears sound to me -- the bounds quoted from the Ghosh work are correct and the gradients for the CCE, MAE, and Lq losses are correct. The analyses of Lq loss (lines 147-153) and truncated Lq loss (lines 165-178) appear correct to me, but I can't guarantee their correctness. For what it's worth, I'd recommend including a more detailed proof in either an appendix or an extended version of the paper on arXiv.  The experiments provide convincing evidence that Lq loss is superior to CCE and MAE in the presence of uniform and class conditional label noise. Lq loss is competitive with Sukhbaatar and Fergus algorithm, even when it has access to the noise rate.  I do have two questions for the authors:  (1) How do these results apply to binary classification (presumably they do since binary classification is a special case)? However, Ghosh showed in an earlier paper [4] that there for binary classification, there are other noise robust losses, e.g., ramp loss, that work quite well and might provide a stronger baseline than MAE.  (2) Ghosh's results apply to more general label noise scenarios, including non-uniform noise. Does the analysis here also apply to this more challenging (and realistic) scenario?  The experiments do have one weakness: there has been a lot of research on noisy labels in the last few years, so I think that a comparison with additional baselines is warranted. Examples include [1][2][3]. Frankly, I'm not sure ANY baseline comparisions is strictly necessary since the paper's central claim is that Lq loss is superior to CCE and MAE -- not that it's superior to all noisy label learning techniques. However, the inclusion of the "Forward" method opens the door to the "why didn't you include this other baselinee" critique.  Clarity:  This paper is clearly written for the most part, though as noted, the theoretical analysis of Lq loss (lines 147-153, 165-178) would benefit from greater detail, even in a separate document.  The analysis of the gradients could use further clarification -- this sentence on lines 122-23 is particularly challenging to understand: "...less congruent with provided labels are implicitly weighed more in the gradient update." Staring at Equation 5, it's not immediately clear to me what this means.  The discussion of truncated Lq loss could benefit from further discourse. For example, what is the motivation behind the introduction of the threshold k in Equation 9? How can we intuitively understand what it's doing, and how does it connect to Equation 11?  Also, the details of the pruning step aren't entirely clear: since neural net training typically involves unconstrained SGD, how do we optimize v_i bounded between 0 and 1?  Originality:  This paper has a marginal amount of originality -- it leans heavily on the Ghosh and Ferrari/Yang papers, synthesizing their respective ideas (risk bounds for noise robust symmetric loss functions and Lq loss), rather than proposing entirely new ones. Nonetheless, I think it demonstrates innovtative thinking, especially when looking at the gradients of the loss functions and considering how they will impact the dynamics of neural net learning (wereas Ghosh focuses primarily on the statistical properties).  Significance:  I'm really uncertain about the significance of the paper. But my takeaway from the paper is "use Lq loss instead of CCE and MAE, in most cases." Given that Lq loss appears pretty easy to implement and differentiate, there seems to be no reason to replace CCE with Lq in any experiment where there may be nontrivial amounts of label noise.   References  [1] Reed, et al.  Training Deep Neural Networks on Noisy Labels with Bootstrapping. ICLR 2015 Workshop Track. [2] Natarajan, et al. Learning with Noisy Labels Nagarajan. ICML 2013. [3] Ma, et al. Dimensionality-Driven Learning with Noisy Labels. ICML 2018. [4] Ghosh, et al. Making Risk Minimization Tolerant to Label Noise. Neurocomputing 2015.


Robustness of conditional GANs to noisy labels
We study the problem of learning conditional generators from noisy labeled samples, where the labels are corrupted by random noise. A standard training of conditional GANs will not only produce samples with wrong labels, but also generate poor quality samples. We consider two scenarios, depending on whether the noise model is known or not. When the distribution of the noise is known, we introduce a novel architecture which we call Robust Conditional GAN (RCGAN). The main idea is to corrupt the label of the generated sample before feeding to the adversarial discriminator, forcing the generator to produce samples with clean labels. This approach of passing through a matching noisy channel is justified by accompanying multiplicative approximation bounds between the loss of the RCGAN and the distance between the clean real distribution and the generator distribution. This shows that the proposed approach is robust, when used with a carefully chosen discriminator architecture, known as projection discriminator. When the distribution of the noise is not known, we provide an extension of our architecture, which we call RCGAN-U, that learns the noise model simultaneously while training the generator. We show experimentally on MNIST and CIFAR-10 datasets that both the approaches consistently improve upon baseline approaches, and RCGAN-U closely matches the performance of RCGAN.

Conditional generative adversarial network (GAN) learn conditional distribution from a joint probability distribution but corrupted labels hinder this learning process. The paper highlights a unique algorithm to learn this conditional distribution with corrupted labels. The paper introduces two architectures i) RCGAN which relies on availability of matrix C which contains information regarding the errors, and ii) RCGAN-U which does not contain any information about the matrix C.   The paper claims that there is no significant loss in performance with regards to knowledge about the matrix C. Even though the problem is unique in nature the paper contains details of some of the related work and references to techniques utilized in the paper such as projection discriminator. I believe the in-depth analysis of the assumptions with theorems and proofs solidify the claims made in the paper although the math requires a more careful check. The paper provides experimentation results on two separate datasets    As far as I understand the permutation regularizer penalizes the model if generator produces samples from a class different from true one and mixing regularizer penalizes if it produces a mix of classes. The paper states that in practice we require a very weak classifier, an explanation of the intuition behind this or theoretical analysis of this statement should help concrete this claim. The paper utilizes recently proposed neural network distance which only guarantees some weak notions of generalization. Ambient GAN as stated in the related work section most closely related to the paper but the paper lacks a comparison with it under the experiments section.  To summarize the paper highlights a unique problem. It would be interesting to see its application to fix corrupted labels in the training data.


Relating Leverage Scores and Density using Regularized Christoffel Functions
Statistical leverage scores emerged as a fundamental tool for matrix sketching and column sampling with applications to low rank approximation, regression, random feature learning and quadrature. Yet, the very nature of this quantity is barely understood. Borrowing ideas from the orthogonal polynomial literature, we introduce the regularized Christoffel function associated to a positive definite kernel. This uncovers a variational formulation for leverage scores for kernel methods and allows to elucidate their relationships with the chosen kernel as well as population density. Our main result quantitatively describes a decreasing relation between leverage score and population density for a broad class of kernels on Euclidean spaces. Numerical simulations support our findings.

*** UPDATE *** Thank you for your author feedback. I feel that the paper is strong, but discussion with fellow reviewers persuaded me to reduce my score from 9 to 8 due to their feedback that aspects of the presentation are unclear. ***  This paper makes a strong mathematical contribution to the concept of leverage, linking this to notions from the theory of reproducing kernels and, ultimately, proposing and studying (mathematically) a "regularised" Christoffel function. Such a contribution is, to my mind, timely and its theoretical emphasis is welcome at NIPS.   The paper reads well and the only comments that I have are extremely minor:  - for transpose in latex, ^T ought to be ^\top  - on p2 "which degree is" -> "whose degree is"  - on p4 "approximate integration" -> "approximates integration" and "we first precise" -> "we first make precise"  - on p4, technically L^\infty hasn't been defined.  - the overview of numerical cubature in methods in sec 2.4 was rather limited, and the authors could emphasise that in general a cubature rule should be selected based on properties of the function being integrated. For example, arbitrarily fast convergence for smooth integrands can sometimes be achieved with quasi-Monte Carlo methods.  - on p7 "numerical evidences" -> "numerical evidence"