Practical exact algorithm for trembling-hand equilibrium refinements in games
Nash equilibrium strategies have the known weakness that they do not prescribe rational play in situations that are reached with zero probability according to the strategies themselves, for example, if players have made mistakes. Tremblinghand refinements-such as extensive-form perfect equilibria and quasi-perfect equilibria-remedy this problem in sound ways. Despite their appeal, they have not received attention in practice since no known algorithm for computing them scales beyond toy instances. In this paper, we design an exact polynomial-time algorithm for finding trembling-hand equilibria in zero-sum extensive-form games. It is several orders of magnitude faster than the best prior ones, numerically stable, and quickly solves game instances with tens of thousands of nodes in the game tree. It enables, for the first time, the use of trembling-hand refinements in practice.

*** POST-REBUTTAL *** Despite the missing appendices, I had tried to work out the authors' proofs based on the sketches they provided in the main paper: whatever questions I had, they addressed in the rebuttal (I upgraded my assessment from 6 to 7 to indicate just this). As far as I can tell, the authors' analysis is sound and I believe the paper clears the bar for acceptance.   *** PRE-REBUTTAL  *** In this paper, the authors propose a novel algorithm for calculating trembling-hand equilibria in two-player zero-sum games in extensive form. In a nutshell, the authors reformulate the zero-sum game at hand as a linear program (a standard trick), and they then describe an oracle mechanism for deciding whether a basis of the resulting linear program is stable or not. When such a basis is computed, the algorithm lets the magnitude of the trembles go to zero and ultimately outputs a trembling hand equilibrium.  I found the paper to be overall well-written, clearly structured, and the analysis morally sound (at least, to the best of my understanding). At the same time however, there are some important points where I found the authors' presentation unsatisfactory. In particular, I have the following concerns:  - A good amount of the authors' analysis relies on the optimality certificate  t_B of a given basis being an analytic function of epsilon (the perturbation magnitude). From what I could gather, the polynomial (in epsilon) nature of the LP has to do with the fact that the underlying game is finite (though the specifics of how epsilon enters the various expressions are not particularly well-explained either). However, given that the expression for t_B involves the inverse of B (which is a rational function of its components), it is not clear why the resulting function does not have a pole at 0 (or below the perturbation threshold predicted by Theorem 7 for that matter). This is further compounded by the last sentence of Section 5 which states that "this task is easy after having computed the Laurent [as opposed to Taylor] series of x(epsilon)", which also seems to suggest that t_B is meromorphic - not holomorphic. This is a central point in the authors' reasoning so I would expect a more rigorous and more detailed treatment.  - Regarding the algorithm's runtime: the authors claim that it is polynomial, but there's a couple of points that are not clear to me:     1. The lowest non-zero term in the Taylor expansion of t_B might have an exponentially high degree: in that case, the algorithm would need to go through exponentially many derivatives before getting to the order of the expansion. [To be clear: I am not claiming that this is the case, I am just saying that this cannot be a priori excluded.] The authors need to show that this cannot happen, otherwise I fail to see why the algorithm would terminate in polynomial time.     2. The number of bases that need to be checked might also be super-polynomial: why is this not the case?  All in all, statements like the one in L244 ("the optimality certificate is extremely easy to deal with") seem too cavalier to me, especially when considering subtle complexity issues as the authors are doing.  - The argument in p.2 that "no NE refinement was used [in Libratus] to capitalize on the opponent's mistakes" is vague at best, misleading at worst. Even though I appreciate the algorithmic importance of calculating NE refinements (trembling hand/sub-game perfect, QRE or otherwise), this statement is neither correct nor sufficient to single out the relevance of trembling hand equilibria. The cited book of van Damme (1984) contains many other refinements that coud "capitalize on mistakes" (basically, any Nash equilibrium that is not supported on weakly dominated strategies would do a fine job at exploiting mistakes).  All in all, I'm on the fence about this submission: I did like it, but some points could be better explained and I also feel a better venue for it would be a conference like EC, SODA or FOCS.


GILBO: One Metric to Measure Them All
We propose a simple, tractable lower bound on the mutual information contained in the joint generative density of any latent variable generative model: the GILBO (Generative Information Lower BOund). It offers a data-independent measure of the complexity of the learned latent variable description, giving the log of the effective description length. It is well-defined for both VAEs and GANs. We compute the GILBO for 800 GANs and VAEs each trained on four datasets (MNIST, FashionMNIST, CIFAR-10 and CelebA) and discuss the results.

[Response to rebuttal: I am happy and thoroughly satisfied with the author's response to the reviews, which is why I have increased my rating. I only want to touch upon one aspect once more: Reviewer 2 has raised an interesting issue (and in my view the authors have responded well and convincingly), namely the problem that the continuous MI might diverge for a differentiable deterministic model. The paper might benefit from explicitly including and addressing this issue, at least in the appendix. The issue is trivially resolved by acknowledging the fact that we perform computation with limited bit-precision, so the authors might even choose to make this explicit by replacing integrals with sums in their equations. For the continuous case, the problem remains when using a deterministic, reversible generator and I think it is interesting to comment on this in the paper and discuss how (and potentially under what circumstances) typical Relu networks tend to produce non-reversible functions ("many-to-one" mapping) while still being (mostly) differentiable. A full investigation of this is beyond the scope of the paper, but it might spark interest to look into this in some future work and thoroughly investigate what kind of compression GAN decoders perform (in the continuous limit).]  The paper introduces a new metric to quantitatively characterize the complexity of a generative latent-variable model with the ultimate goal of providing a performance indicator to quantitatively characterize generative models obtained from both GANs and VAEs. The aim is to quantify the mutual information between the latent variable z and the generated data x. This quantity is interesting since it provides a training-data independent measure, is invariant under re-parametrization and has interesting and well-known information-theoretic properties and interpretations. Since the computation of the mutual information involves the intractable posterior p(x|z), the paper proposes to instead consider a lower bound (GILBO) based on a variational approximation of the posterior instead. The measure is demonstrated in an exhaustive experimental study, where 800 models (VAEs and GANs) are compared on four different datasets. The authors find that the proposed measure provides important insight into the complexity of the generative model which is not captured by previously proposed data-dependent and sample-quality focused measures such as the FID score. The GILBO allows to qualitatively divide trained models into different regimes such as “data memorization” or “failure to capture all classes in latent space”. Additionally the GILBO highlights some qualitative differences between GANs and VAEs. The paper concludes by addressing some potential shortcomings of the method: reproducibility of quantitative values, tightness of lower bound, quality of variational approximation and consistency with qualitative “behavior” of the generative model.  The recent literature reports several attempts at objectively comparing the quality of complex generative models obtained from GANs or VAEs. While many methods so far focused on capturing the match between the true and the generated data-distribution or quantitative measures that correlate well with visual quality of generated samples, these measures seem to be insufficient to fully capture all important aspects of a generative model. Reporting the GILBO as an additional quantity that characterizes complexity of the generative model is a novel and original proposal. The paper is generally well written (except for some hiccups at the beginning of section 2) and the experimental section is impressive (certainly in terms of invested computation). Main points of criticism regarding the method are already addressed in the paper. For these reasons, I argue for accepting the paper. My comments below are meant as suggestions to authors for further increasing the quality of the paper.  (1 Intro of the GILBO in Sec. 2) Section 2 (lines 23 to 48) could use some polishing and additional details, particularly for readers unfamiliar with standard notation in generative latent-variable models. In particular: -) please explain what the variables X and Z correspond to in the context of generative models, ideally even mention that such a generative model consists of a prior over the latent variable and a (deterministic) generator p(x|z) which is a deep neural network in GAN and VAE. -) line(27): I guess to be precise, it is the reduction in the number of questions to guess X if you observe Z and know p(X). -) line (32): I think it would not hurt if you (in-line) add the equation for marginalization to make it obvious why p(x) is intractable. Also consider giving the equation for the posterior p(z|x) as you have it in line 34 before mentioning p(x), since p(x) does not appear in Eq. (1) but p(z|x) does. This might help readers follow the reasoning more easily. -) Eq (2): perhaps consider being explicit about the fact that GILBO is a function which depends on the parameters of e(z|x), for instance by writing GILBO(\theta)=… and then using e_\theta(z|x) or similar. -) line (42 – 44): “maximizing the likelihood…” – it could be helpful to write down the same as an equation, just to make it precise for readers.  (2 Proofs) There are two arguments that you might want to back up with a proof (e.g. in the supplementary material). Though the proofs should not be too difficult, some readers might appreciate them and they are actually central statements for the appealing properties of the GILBO. Additionally it might be beneficial because it would require being more specific about the technical conditions that need to be fulfilled to get the appealing properties of GILBO.  -) line 46: show that the GILBO is actually a lower bound to the mutual information. -)line 48: show that failure to converge, etc. simply makes the bound looser, i.e. show that Eq. (2) cannot diverge due to some ill-parameterized e(z|x).  (3 Discussion) Three more issues that you might want to include in your discussion. -) Do you think it would be possible/computationally feasible to replace the variational approximation of the posterior with some sampling-approximation, which would in principle allow for arbitrary precision and thus guarantee convergence to the actual mutual information (while computationally expensive, this could be used to compute a gold-standard compared to the variational approximation)? -) How sensitive is the GILBO to different settings of the optimization process (optimizer, hyper-parameters)? Ideally, this answer is backed up by some simulations. This would be of practical importance, since comparing GILBOs of different models could in principle require following very closely the same procedure (and of course encoder architecture) across papers. -) Besides qualitative “sanity-checks” (as in Fig. 2: which regime does my generative model fall into), how do you propose to use the GILBO to quantitatively compare models. Say I have two models with different GILBO (but in the same qualitative regime) and different FID score – which model is favorable, higher GILBO, higher FID, …?  Misc: -) Footnote 1, last sentence: … it could also achieve a looser bound -) Figure 1a: Any specific reason for a different color-map here, otherwise please use the same color-map as in the other plots. -) Fig. 7 caption vs. line 133: did you use 128 or 127 samples? -) Fig. 8 caption vs. line 165/166: did you use 16 or 15 samples? -) citations: I quite like the author, year citations (even though they “cost” extra space in a conference paper) – please check that all your citations have the year properly set (see line 194, where it is missing) -) line 207/208: Reporting the GILBO as an additional measure could be a good idea, please be a bit more specific that this would require fixing the encoder architecture and optimization procedure as well and comment on practical aspects such as whether you would use a different encoder architecture per data-set or the same across many data-sets, etc.


Supervised autoencoders: Improving generalization performance with unsupervised regularizers
Generalization performance is a central goal in machine learning, with explicit generalization strategies needed when training over-parametrized models, like large neural networks. There is growing interest in using multiple, potentially auxiliary tasks, as one strategy towards this goal. In this work, we theoretically and empirically analyze one such model, called a supervised auto-encoder: a neural network that jointly predicts targets and inputs (reconstruction). We provide a novel generalization result for linear auto-encoders, proving uniform stability based on the inclusion of the reconstruction error-particularly as an improvement on simplistic regularization such as norms. We then demonstrate empirically that, across an array of architectures with a different number of hidden units and activation functions, the supervised auto-encoder compared to the corresponding standard neural network never harms performance and can improve generalization.

This paper performs an analysis of linear supervised autoencoders, showing that they have nice O(1/t) stability properties that in turn transfers to a nice generalization error bound. They also show that in practice, regularizing a deep model with an autoencoder penalty outperforms L2 and dropout regularizations in some scenarios. This is most notable if the model is allowed more capacity: the capacity is used to fit the reconstruction (with apparent benefits) rather than overfit on the predictive task.  This result seems to be an interesting improvement over Liu et al. if one cares about/believes in reconstruction. While I think most in our community would agree that capturing factors of variation through reconstruction is very appealing (and previous literature favorably points that way), there are instances (unfortunately rarely documented) of auxiliary reconstruction losses harming performance or requiring extensive tuning, and the bound that you compute does not necessarily suggest a better performance either. I intuitively agree that reconstruction seems preferable to L2, and as you say, “potentially without strongly biasing the solution”, but I would not be surprised if under certain mild conditions the bias that a reconstruction loss adds was demonstrably larger than that of L2.  The empirical part nicely demonstrates the point of the authors (and has a good methodology). Something feels like it is missing though. The main result of this paper is a O(1/t) bound over t the number of dataset samples, yet, there is not a single plot that reflects varying the size of a dataset.  The paper itself was well written, and although I had to read a few of the referenced papers to get familiar with stability bounds, the paper felt clear. I think overall this is a neat contribution that could maybe be stronger but is nonetheless relevant to deep learning training understanding.  I gave myself a confidence score of 3: while I think I understand the paper and its results, I do not have sufficient understanding of the theory to say with certainty that the math is 100% correct.  Minor comments: The equation following (2) has a typo: the first L_p should be a function of x’_m and y’_p,m (rather than x’_i and y’_p,i). wrt Theorem 1. In practice, people rarely bound their parameters (except maybe for RNNs, and even then people usually only clip gradients). Without L2, what stops say B_F or rather ||F|| from growing in O(t)? (Maybe this is wrong since this is for the linear case, but since more data points can require a more complex hypothesis they might require larger weights) CIFAR-10 is a specific subset of 60k examples of a larger dataset (the tiny images dataset), which itself has 80 million examples. I’d either correct the number or explicit what CIFAR-10 is, because I assume that when you “used a random subset of 50,000 images for training and 10,000 images for testing” you only shuffled the actual 60k image dataset, but referencing the 80M images suggests otherwise. It’s great that you do cross-validation. I think mentioning meta-parameter ranges in the appendix would be useful for reproducibility (or the optimal ones you’ve found). I think you could easily do a plot that’s exactly like Figure 3, but with the x axis being the number of samples used, from say 1 to 50k   Rebuttal update: I thank the authors for their response. Again I think this could be a great paper and encourage the authors to add varieties of experiments that would make this work even more convincing.


Adversarial Regularizers in Inverse Problems
Inverse Problems in medical imaging and computer vision are traditionally solved using purely model-based methods. Among those variational regularization models are one of the most popular approaches. We propose a new framework for applying data-driven approaches to inverse problems, using a neural network as a regularization functional. The network learns to discriminate between the distribution of ground truth images and the distribution of unregularized reconstructions. Once trained, the network is applied to the inverse problem by solving the corresponding variational problem. Unlike other data-based approaches for inverse problems, the algorithm can be applied even if only unsupervised training data is available. Experiments demonstrate the potential of the framework for denoising on the BSDS dataset and for computed tomography reconstruction on the LIDC dataset.

Summary:  The paper proposes to learn the regularization function in inverse problem. This idea is that the regularization should take large values for non-plausible images and low values (not necessaritly bounded below) for true images. While many variational formulations use hard coded functionals such as TV for images, here it is learnt using an approach inspired by the Wasserstein GAN method.  To learn the regularization the GAN has to discrimitate between measurements backprojected to image space using a simple pseudo-inverse strongly affected by noise, and some ground truth images.  By promoting a regularization function which is "close" to have 1-Lipschitz gradient, results on stability can be obtained and hyperparameter \lambda can be calibrated using the same network that does not depend on the noise level.  The paper is well written.  Qualitative evaluation:  - While the GAN promotes a regularizer with a 1-Lipschitz gradient it is not a hard constraint of the learning problem. However theoretical results make the assumption that the function has a 1-Lipschitz gradient everywhere. This weakens the theoretical claims of the paper, if not makes the theorems innapropriate. This should be discussed.  - One strong claim of the paper is that the method does not need lots of ground truth data.  "Our approach is particularly well-suited for applications in medical imaging, where usually very few training samples are available and ground truth images to a particular measurement are hard to obtain, making supervised algorithms impossible train.""  However the procedure described in Algorithm 1 requires to simulate data using ground truth images. This seems to be a paradox and there is no experimental evidence that shows that the method needs very little ground truth data to perform well.  Misc / Typos:  - Careful with capital letters in the titles of the papers in the bibliography.  - Kontorovich -> Kantorovich


On preserving non-discrimination when combining expert advice
We study the interplay between sequential decision making and avoiding discrimination against protected groups, when examples arrive online and do not follow distributional assumptions. We consider the most basic extension of classical online learning: Given a class of predictors that are individually non-discriminatory with respect to a particular metric, how can we combine them to perform as well as the best predictor, while preserving non-discrimination? Surprisingly we show that this task is unachievable for the prevalent notion of equalized odds that requires equal false negative rates and equal false positive rates across groups. On the positive side, for another notion of non-discrimination, equalized error rates, we show that running separate instances of the classical multiplicative weights algorithm for each group achieves this guarantee. Interestingly, even for this notion, we show that algorithms with stronger performance guarantees than multiplicative weights cannot preserve non-discrimination.

The paper theoretically studies the suitability of achieving a particular definition of fairness, equalized odds (which relates to the false positive rate), in the context of online learning with experts advise (Cesa-Bianchi et al. 2006). In particular, the authors show that achieving an online algorithm that jointly satisfies zero-regret and equalized odds is not possible. Afterward, they show that this is not the case when considering fairness in terms of the total number of errors per group. They also discuss that unfortunately this definition of fairness (also previously discussed in  Zafar et al., 2017) is not realistic (or even fair) in many real-world scenarios.  In the positive side, I believe that (im)possibility theoretical studies on when a fairness definition can be accomplished is definitely a major contribution to the field.  However, I also believe that the paper has important gaps to be filled:  1) Their definition of online learning comes from the game theory literature and does not corresponds to the standard ML view on online learning. However, the authors do not clarify this particular setting in the abstract (neither the title of the paper) and provide any reference to the considered  "benign setting"--where at time t there is a set of expert providing their advise about the decision to be made, and the learner select one expert advise (decision) with a fixed probability i~p^t, getting a loss l(t,i) that depends on the selected expert. Is this the actual setting? Please clarify this point in the paper, and add the necessary references.  2) Although the paper is presented in the context of fairness, which is definitely a real problem, the authors do not provide a single real-world example where their setting (based on the game theoretical game benign) would fit.  As a consequence, it is hard to evaluate the potential impact of the presented theoretical results in the field.  In summary, I believe although the paper presents good ideas and results, it does not provide the necessary context and details to judge the contribution of the paper.   Cesa-Bianchi, Nicolo, and Gábor Lugosi. Prediction, learning, and games. Cambridge university press, 2006.  B. Zafar, I. Valera, M. Gomez-Rodriguez and K. Gummadi, "Fairness Beyond Disparate Treatment & Disparate Impact: Learning Classification without Disparate Mistreatment", (Full paper) 26th International World Wide Web Conference (WWW), Perth (Australia), April 2017.


Connecting Optimization and Regularization Paths
We study the implicit regularization properties of optimization techniques by explicitly connecting their optimization paths to the regularization paths of "corresponding" regularized problems. This surprising connection shows that iterates of optimization techniques such as gradient descent and mirror descent are pointwise close to solutions of appropriately regularized objectives. While such a tight connection between optimization and regularization is of independent intellectual interest, it also has important implications for machine learning: we can port results from regularized estimators to optimization, and vice versa. We investigate one key consequence, that borrows from the well-studied analysis of regularized estimators, to then obtain tight excess risk bounds of the iterates generated by optimization techniques.

The authors explore the relation between the trajectory of Gradient Descent (GD) initiated in the origin and the regularization path for l2-regularized minimization of the same objective.  They first study the continuous-time setting where GD is replaced Gradient Flow, assuming that the objective is smooth and strongly convex. The main result (Theorem 1 whose proof I have verified) is as follows: under the appropriate scaling between the time $t$ in GD and the inverse regularization parameter $\eta$, the two trajectories do not diverge much. This result is obtained by quantifying the shrinkage of the gradients as $t$ and $eta$ tend to infinity. In the continuous-time setting, the authors manage to reduce this task to formulating and solving certain ODEs. Then, a discrete-time counterpart of Theorem 1 is established (Theorem 2); its proof is more technical but mostly goes along the same lines (the main technical difference is that instead of solving ODEs one deals with linear recursions). Honestly, I did not check the proof of this later result, but overall it seems credible to me taking into account my verification of Theorem 1 and related technical lemmas.  The authors then briefly state the extensions of these results for non-Euclidean proximal setups (i.e. Mirror Descent), assuming that the potential is smooth and strongly convex.  Finally, the obtained results are applied to establish statistical bounds for the excess risk when Gradient Flow is used to minimize the empirical risk objective. The main result here is Corollary 5, which provides a recipe for early stopping. These results are then tightened for random-design linear regression and classification under separability assumptions. These results are close in spirit to (1) the results of (Hardt et al, 2015), where implicit regularization properties of SGD with early stopping are established in the general setting of empirical risk minimization (ERM), and (2) a more recent line of work (Soudry et al. 2017), (Ji and Telgarsky 2018), (Gunasekar et a;. 2018), where self-regularization properties of SGD are established in the case of separable data.   First let me discuss the main highlight of the paper, Theorems 1-2. The technique that relates the two trajectories through a differential equation is easy-to-use and powerful, and I consider it the main highlight of the paper. I also like the idea of using the known results for l2-regularized ERM to analyze GD with early stopping. That said, it has long been known that the analysis of Gradient Flow is reduced to solving ODEs, see e.g. (D. Scieur, Integration Methods and Accelerated Optimization Algorithms 2017). I would like to see a review of the related literature since I suspect that some of the results in Section 1 can be already known; as far as I can see, the real novelties here are (1) using the ODE-analysis to relate Gradient Flow/Descent to the l2-regularization path, and (2) providing a result for the case of discrete step.   I should also mention that I find the results stated in Sections 3 and 4 particularly interesting and useful for the computational learning theory community. That said, a more thorough discussion of these results and comparison with the existing literature on optimal stopping / margin-type results would be appreciated.  I have decided to evaluate the paper only as weak accept, for a number of reasons listed below:  1. On several occasions, mathematical results are claimed without a proof or reference (lines 69-70, line 80, lines 134-137).  2. The last (non-trivial) step in the proof of Theorem 1 is missing — the authors do not detail how they solve the ODE in the display between l.337 and l.338. I did manage to arrive at l.339 after some algebraic manipulations, making the estimate m\nu + 1 \ge m\nu, and solving an inhomogeneous ODE obtained in the end; I think that all these steps should have been at least hinted on!  3. Theorem 2 is not properly commented upon. 4. The results in Section 2.1 seem rather conceptual: the only known to me non-trivial case of a strongly convex *and* smooth potential is that of Mahalanobis distance which can be reduced to the case of Euclidean norm and |.|^2/2-potential by a linear change of variables. As noted by the authors themselves, the ``truly’’ non-Euclidean setups, such as minimization on the simplex or l1-ball, or on the nuclear-norm ball, do not admit smooth potentials, see e.g.  (Nesterov and Nemirovski, On first-order algorithms for l 1/nuclear norm minimization).  Instead, one imposes a weaker property of the slow growth of the potential on the feasible set. Most importantly, as noted in (Nesterov 2005) and also by the authors themselves, non-Euclidean setups are never better than the Euclidean one when the objective is both smooth and strongly convex. 5. The quality of writing in general could be improved. For example, on several occasions I spotted contractions in the text, such as “till” or “don’t”; I personally consider their use inappropriate in a scientific paper. 6. The plots are hard to read because of very small axis tick labels and non-uniform grid. If the paper gets accepted, I would appreciate better-looking plots in the camera-ready version, as well as more ample explanations for them.


Fast Approximate Natural Gradient Descent in a Kronecker-factored Eigenbasis

This paper proposes a new Kronecker approximation of the empirical Fisher information, EKFAC, that matches the eigenvalues of the Fisher in the approximation; this method is then used for natural gradient descent like the original Kronecker factored approximate curvature method (KFAC).   The authors claim that the proposed method has both enhanced computational benefits over the original KFAC scheme and increased accuracy than the previous KFAC variants. However, it does not seem to outperform well-tuned SGD with momentum and batch norm and has approximately double the computational costs according to the figures in the paper. The tradeoff with the proposed method seems to be a marginal increase in test accuracy at the expense of a significant increase in computational effort. Overall, the experiments section is quite confusing to me and the figures do not seem to align well with the descriptions in the text of the paper.  Originality: The proposed method seems to be quite novel. Instead of directly computing an approximation to the empirical Fisher information, EKFAC computes an approximation to the eigen-decomposition. This has the interpretation of projecting the gradients into an eigen-basis, performing natural gradient descent in the eigen-space, before projecting back into the standard weight space.   Significance: The task of developing new second-order stochastic optimization methods that scale to deep neural networks is quite significant. The proposed method seems to be one of the first methods to use an eigen-decomposition to perform natural gradient descent. However, the improvement in practice seems to be quite minimal when compared to SGD and SGD + momentum.  Clarity: The paper is relatively well-written, with the exception of the experiments section (see below). However, I would suggest more paragraph breaks, particularly in Section 1 and Section 3.1, to further encourage readability.  Quality: The method developed seems to be sound. However, I have some issues with the experiment section (see below), particularly with respect to the presentation. Much emphasis is placed on the wall-clock time, but no description of the system architectures or units on the time (seconds? minutes?) is mentioned.  Comments on Experiments: - The color scheme in the plots with respect to the SGD/Momentum/Adam variants is quite unclear. For instance, in Figure 4c, it is obvious that one of these baseline optimizers gives quite similar validation loss as KFAC and EKFAC but noticing this is hindered by both the color scheme and the KFAC loss curve lines being on top of that curve. This is true for Figure 6 as well, particularly Figure 6c, which has the additional complexity of having dashed lines for the test set in the same plot as the train set accuracy. Perhaps tables could be used to report accuracy and wall-clock time instead. - Figure 4b: Could the authors comment on why the two EKFAC_ra curves (yellow and orange) are by far the slowest methods on the deep auto-encoding task? By contrast, in the text, EKFAC_ra is described as a quicker than the EKFAC and KFAC methods.  - Figure 5a and 5b: From these figures, it is relatively clear that KFAC performs better with respect to training loss when updated more frequently; the trade-off is of course computation time. However, there is no line for EKFAC updated every 1 iteration (like for KFAC). This makes it hard to substantiate the claim that “EKFAC preserves better its optimization performances as we decrease the frequency of eigendecomposition,” as there are only two data points for EKFAC (50 and 100). Additionally, could the authors comment on if the running average version of EKFAC should also maintain this performance as it is not in this plot? - Figure 6b and 6d: Could the authors please comment on the vastly improved speed of KFAC (yellow line) in terms of wall-clock time on the CIFAR 10 experiment? From these figures, it seems like KFAC requires a similar amount of run time to SGD + momentum, and considerably less wall-clock time than EKFAC. - A suggestion for another experiment: what is the accuracy on a problem when comparing SGD (and friends) to EKFAC for the same amount of wall-clock time?  - Finally, could the authors test the accuracy of the KFAC/EKFAC approximations on a small MLP (say one or two hidden layers and 20 hidden units) where the empirical Fisher information and its eigenvalues can be computed exactly? This might be more effective in showing that EKFAC is more accurate than KFAC. The experiment with the fourth layer of the deep autoencoder is quite nice; however, it would be interesting to see how EKFAC and KFAC model the cross-correlations between the two sets of weight parameters.  Updates after Author Response: I thank the authors for their clarifications, particularly with respect my comments on the figures in the main paper. I do hope that all of their updated figures look as interpretable as the ones that they showed in the response.  As a small comment, I would also like to see the error figure that they included in the response in the main paper (and with more y-axis labels), as it shows the error associated with using KFAC and EKFAC very nicely. However, I'm still unconvinced by the author's claims that EKFAC_ra is slower in convnets due to issues with computing a running average of the second moment of gradients, as this seems like it would just require a square operation.  Typos: Line 25: 1st -> 1st order  Line 28: mention that \eta is the learning rate. Line 34: Diagonal approximate -> Diagonal approximations Line 35: are proven to be efficient -> have been proven to be efficient Line 38: view) -> view  Line 54: while computing Kronecker-factored eigenbasis is an expensive operation that need to be -> while computing [a] Kronecker-factored eigenbasis is an expensive operation that [needs] to be Line 75: Whereas … uses is not a sentence and should probably be joined with the previous sentence. Alternatively, change Whereas to (By contrast,). Lines 61-80: This paragraph seems to be quite run-on and should be separated into two separate paragraphs.  Line 230: missing . after (Laurent et al, 2018).


Clebsch-Gordan Nets: a Fully Fourier Space Spherical Convolutional Neural Network
Recent work by Cohen et al. [1] has achieved state-of-the-art results for learning spherical images in a rotation invariant way by using ideas from group representation theory and noncommutative harmonic analysis. In this paper we propose a generalization of this work that generally exhibits improved performace, but from an implementation point of view is actually simpler. An unusual feature of the proposed architecture is that it uses the Clebsch-Gordan transform as its only source of nonlinearity, thus avoiding repeated forward and backward Fourier transforms. The underlying ideas of the paper generalize to constructing neural networks that are invariant to the action of other compact groups.

This paper proposes a generalized version of SO(3)-equivariant architectures including Spherical CNN. By utilizing the algebraic properties of Fourier transform and the tools in non-commutative harmonic analysis, the authors are able to construct (and prove) a most generalized version of SO(3)-equivariant architecture. Specifically, it only requires that, when an input image is rotated, each fragment (i.e., the output, Fourier coefficient vectors) of each layer will be multiplied by a Wigner-D matrix. To include non-linearities without performing inverse Fourier transform, the authors propose to use Clebsch-Gordon transformation. The experiments show that the proposed CG-Net can outperform Spherical CNN in several tasks.  Strength: - The paper proposes a generalized version of Spherical CNN. Plus, the paper proves that this architecture is the most generalized version of SO(3)-equivariant architecture. - A principled treatment on SO(3)-equivarient operations by looking into the algebraic properties of them. As a result, almost all operations in CG-Net are performed in Fourier space. - CG-Net works well even without traditional non-linearity activation functions like ReLU.  Weakness: - The improvement compared to Spherical CNN seems incremental. - The paper claims that it is costly to compute forward and backward Fourier transform in Spherical CNN. However, it does not compare the runtime of Spherical CNN and CG-Net.  - The experiment seems unfair as the number of parameters in CG-Net is almost 5 times the number in Spherical CNN. - The paper does not include an ablation study on the hyperparameter tau, which is the maximal number of fragments for each spherical harmonic function index l in each layer (to avoid exponential growth in number of fragments) - It would be better if the authors include a visualization to provide a graphical explanation of the non-linearity in CG-transform.  ---------------------- Authors' rebuttal only paritally addresses my concerns. I will keep the original rating


SLANG: Fast Structured Covariance Approximations for Bayesian Deep Learning with Natural Gradient
Uncertainty estimation in large deep-learning models is a computationally challenging task, where it is difficult to form even a Gaussian approximation to the posterior distribution. In such situations, existing methods usually resort to a diagonal approximation of the covariance matrix despite the fact that these matrices are known to result in poor uncertainty estimates. To address this issue, we propose a new stochastic, low-rank, approximate natural-gradient (SLANG) method for variational inference in large, deep models. Our method estimates a "diagonal plus low-rank" structure based solely on back-propagated gradients of the network log-likelihood. This requires strictly less gradient computations than methods that compute the gradient of the whole variational objective. Empirical evaluations on standard benchmarks confirm that SLANG enables faster and more accurate estimation of uncertainty than mean-field methods, and performs comparably to state-of-the-art methods.

Update: I have considered the author response. I maintain my assessment.  ==========================================================  This paper proposed an interesting idea of learning efficient structured multivariate Gaussian variational distributions. Since directly making a mean-field approximation may make the variational distribution too far from the model posterior distribution while making a full Gaussian approximation may be intractable, the authors proposed that, we could approximate the inverse of covariance matrix of the variational distribution as the sum of a diagonal matrix and a low-rank matrix. To make the inference faster, the authors also built a natural gradient update algorithm based on the method proposed in [1].  Strengths:   In my opinion, this paper addresses an interesting question and many people in the machine learning community may be interested in it. Variational inference is useful since it is fast but the most common variational distribution family, the mean-field family, could not yield satisfactory approximations in some cases. Hence, the problem that this paper focuses on is interesting. The design of their structured variational approximation is guaranteed to work at least as good as the regular mean-field approximation, and could outperform it in many cases, even if the added low-rank approximation only have rank L=1. Moreover, the paper is well-written and it is easy to follow the ideas.   Weakness:  It might show the success of the proposed method better if some more experiment results are presented.   First, for both the Bayesian Logistic regression experiment and the Bayesian neural network experiment, the authors mentioned that a bigger value of L improved the performances, but they only show the results for L=1 (on the metrics in Table 1 and 2). It will be more interesting if the experiment results for larger L could be shown and be compared to the baselines since for both experiments, there are baseline methods work better than the proposed methods. Though we understand that, for the Bayesian Logistic regression experiment, it is not possible for the SLANG method to outperform the full Gaussian method, but the difference between them might be smaller if L is larger.   Second, as shown in the right subfigure in Figure 3, the authors tried to compare the efficiency of SLANG with some baseline methods. It will be great if the authors could also compare the efficiency of their methods with some other second-order variational inference algorithms, such as the Hessian-Free Stochastic Gaussian Variational Inference (HFSGVI) algorithm [2].  In addition to that, it will be good if the authors could provide some theoretical analysis on their method. It will be interesting to see the performance improvements of this method compared to the mean-field approximation.  Minor comment:   Some sentences are a little bit confusing. For example, the sentence "Our goal is to approximate it to bring the cost down to O(D)" on line 110. We know that the time complexity is O(DL^2+DLS), not O(D). Though we understand L and S are small, and the precise time complexity is also mentioned later. It is still a little bit confusing to have this sentence. Probably it is better to say "bring the cost down to be linear in D".  References  [1] Mohammad Emtiyaz Khan and Wu Lin. Conjugate-computation variational inference: Converting variational inference in non-conjugate models to inferences in conjugate models. arXiv preprint arXiv:1703.04265, 2017.  [2] Fan, K., Wang, Z., Beck, J., Kwok, J., & Heller, K. A. (2015). Fast second order stochastic backpropagation for variational inference. In Advances in Neural Information Processing Systems (pp. 1387-1395).


Maximum Entropy Fine-Grained Classification
Fine-Grained Visual Classification (FGVC) is an important computer vision problem that involves small diversity within the different classes, and often requires expert annotators to collect data. Utilizing this notion of small visual diversity, we revisit Maximum-Entropy learning in the context of fine-grained classification, and provide a training routine that maximizes the entropy of the output probability distribution for training convolutional neural networks on FGVC tasks. We provide a theoretical as well as empirical justification of our approach, and achieve stateof-the-art performance across a variety of classification tasks in FGVC, that can potentially be extended to any fine-tuning task. Our method is robust to different hyperparameter values, amount of training data and amount of training label noise and can hence be a valuable tool in many similar problems.

This paper presents a simple and effective approach for fine-grained image recognition. The core idea is to introduce max-entropy into loss function, because regular image classification networks often fail to distinguish semantically close visual classes in the feature space. The formulation is clear and the performance is very good in fine-grained tasks. I like the ablation study on CIFAR10/100 and different subsets of ImageNet, showing that this idea really works in classifying fine-grained concepts.  The major drawback of this paper lies in its weak technical contribution. Max-entropy is a well-known theory, and this paper seemed not to provide any improvement on it, but just applied it to fine-grained recognition (I searched on Web, but I did not see any work with the same solution, so I vote for moderate novelty). It would be a good contribution to the computer vision community, but not a significant one to NIPS which leans more to theory.  In addition, another concern is that the formulation is closely similar to that in [30] (see the first equation in Sec 3 (arXiv: 1701.06548), KL divergence and cross-entropy differ from each other by a constant). Please clarify the difference, otherwise this work would be considered much less novel.  ===== POST-REBUTTAL COMMENTS =====  After reading the rebuttal and other reviewers' comments, I can accept that this paper is an extension to [30] with a proof why a max-entropy-based approach works well on fine-grained recognition. Again, this is a nice work, but the technical contribution is not significant, which limits its quality especially for NIPS. Considering other reviewers' comments, I would change my rating to weak accept.


Learning and Testing Causal Models with Interventions
We consider testing and learning problems on causal Bayesian networks as defined by Pearl [Pea09]. Given a causal Bayesian network M on a graph with n discrete variables and bounded in-degree and bounded "confounded components", we show that O(log n) interventions on an unknown causal Bayesian network X on the same graph, and O(n/ 2 ) samples per intervention, suffice to efficiently distinguish whether X = M or whether there exists some intervention under which X and M are farther than in total variation distance. We also obtain sample/time/intervention efficient algorithms for: (i) testing the identity of two unknown causal Bayesian networks on the same graph; and (ii) learning a causal Bayesian network on a given graph. Although our algorithms are non-adaptive, we show that adaptivity does not help in general: Ω(log n) interventions are necessary for testing the identity of two unknown causal Bayesian networks on the same graph, even adaptively. Our algorithms are enabled by a new subadditivity inequality for the squared Hellinger distance between two causal Bayesian networks.

This paper answers a series of important questions about hypothesis testing and learning of causal models that can contain both measured and unmeasured variables, and contain only discrete variables. (The causal models are represented by semi-Markov causal models that contain a directed edge from A to B representing that A is a direct cause of B, and a bidirected edge between A and B when there is an unmeasured common cause of A and B). It provides bounds on the number of interventions, samples and time steps required by the algorithms. The distance between two models is measured by the maximum difference between interventions on the models. For an unknown model it is assumed that an experiment can perform experiments on any subset of the variables, and then measure the results of the intervention on a sample. The problems include determining when a hypothesized causal model is within epsilon of the unknown true model, when two unknown models are within epsilon of each other, and how to output a model that is within epsilon of the true model (given the correct graph). The paper also shows that the number of interventions required by their algorithms are almost optimal for the latter two problems. The fundamental strategy is to find a small "covering set" of interventions that supports a locality result that shows that when two causal models are far apart, there exists a marginal distribution of some intervention in the covering set that is far apart.   I am familiar with the causal part of the arguments, but less familiar with the complexity part of the arguments. I don't understand the cases where d represents not only the maximum indegree of the model, but also the maximum degree of the model Lemma 2 states that outputting a covering set of interventions is O(K^(ld)*((3d)^l)(ld^2 log(K), where K is the number of different values the variables can take, l is the maximum size of a c-component, and d is the maximum degree of a vertex. However, there is nothing in this formula which mentions n, the number of vertices in the graph (unlike the case where d represents only the maximum indegree of the model). Similarly, Theorem 4 says that same formula is the complexity of determining when two unknown causal models are the same. But given a DAG that consists of a number of subgraphs that are disconnected from each other, and each subgraph consists of a variable and a fixed number of different parents, the number of disconnected subgraphs does not change K, l or d. Nevertheless all of the individual subgraphs would have to be experimented on, and the number of interventions needed would have to increase with n in the worst case. Also, the number of subtests performed by the algorithm is said to depend on n.   There is a subtlety about the models that they are testing and finding that is not mentioned in the article. Once there are latent variables that can't be intervened on, there are multiple causal models that can generate the results of all of the possible interventions. For example, if a latent variable L causes measured X1 and X2, there are parameter values which make both X1 and X2 dependent on L, but independent of each other. In addition, intervening on X1 or X2 will produce no change in the other measured variable. So the intervention distributions can be represented by a semi-Markovian causal model in which X1 and X2 have no edge between them in the causal graph (M1) , as well as a particular parameterization of a semi-Markovian causal model in which X1 and X2 are connected by a bi-connected edge (M2). No experiment on just X1 and X2 can distinguish M1 from M2. So while the "true" causal model (i.e. the one that generated the data) might be M2, the algorithm will state that a hypothesized properly parameterized M1 is correct. For many purposes that is correct (if understood as being about correct predictions about interventions) but it is not the model that generated the data, and while M2 might suggest looking for an unmeasured common cause to manipulate, M1 does not.   There were a couple of minor typos. Theorem 2 uses Sigma before it is defined. Lemma 2 refers to delta - is that the same delta as defined in Lemma 1?  This paper has theoretical significance, particularly with respect to lower bounds on causal inference by any algorithm, as in Theorem 2. It is not clear what the practical significance of the algorithms or the theoretical results are for several reasons. First, most of the algorithms assume that the causal graph is completely known. Second, because the number of interventions (and sample sizes and time) grows so quickly with the size of the number of values of variables, the maximum indegree, and the size of the largest c-component. Even with l = 3, d =  4, K =4, K^(ld) which appears in the algorithm for choosing the covering set of interventions, is more than a million.


Poison Frogs! Targeted Clean-Label Poisoning Attacks on Neural Networks
Data poisoning is an attack on machine learning models wherein the attacker adds examples to the training set to manipulate the behavior of the model at test time. This paper explores poisoning attacks on neural nets. The proposed attacks use "clean-labels"; they don't require the attacker to have any control over the labeling of training data. They are also targeted; they control the behavior of the classifier on a specific test instance without degrading overall classifier performance. For example, an attacker could add a seemingly innocuous image (that is properly labeled) to a training set for a face recognition engine, and control the identity of a chosen person at test time. Because the attacker does not need to control the labeling function, poisons could be entered into the training set simply by leaving them on the web and waiting for them to be scraped by a data collection bot. We present an optimization-based method for crafting poisons, and show that just one single poison image can control classifier behavior when transfer learning is used. For full end-to-end training, we present a "watermarking" strategy that makes poisoning reliable using multiple (≈ 50) poisoned training instances. We demonstrate our method by generating poisoned frog images from the CIFAR dataset and using them to manipulate image classifiers.

== Update after author response == The authors' points about the comparison to Suciu et al. and Koh and Liang are well taken, thanks. I still think that the evidence for the conclusions about end-to-end training vs. transfer learning is weak. While training the Inception network from scratch might be infeasible, the authors should still have done a more controlled experiment, e.g., transfer learning vs. end-to-end training on CIFAR. The claims in the paper about that comparison are currently too speculative.  == Original review == The authors propose a method for creating data poisoning attacks that crafts attack points so that they resemble the target point (that the attacker wants to get misclassified) in feature space, while resembling an innocuous base point in input space. The authors run a carefully-designed series of experiments to show that their targeted attack succeeds in both a transfer learning setting as well as a more challenging end-to-end training setting.  The paper is well-written and easy to follow, and the authors take pains to do an ablative analysis of their attacks, as well as try to analyze them beyond just noting their effectiveness (e.g., by looking at the rotation of the decision boundary). The attack idea is simple (a good thing), though somewhat specific to neural networks, and seems quite workable in practice (which is bad news for most of us, but a good contribution).  My main concern is about novelty, especially compared to the Suciu et al. (2018), which the authors acknowledge. Conceptually, the Stingray attack introduced in Suciu et al. is very similar to the Poison Frogs attack that the present paper introduces: the former also tries to craft poisoned points that are similar in feature space to the attack point but similar in input space to a benign base point. The attacks differ in that Suciu et al. measure similarity in input space in the L_\inf norm (i.e., max deviation of each pixel), which is I think more justified than the L2 norm that the present authors use. Given the comparisons to adversarial training throughout the paper -- where the L_\inf norm is much more commonly used -- the choice of the L2 norm is unorthodox, and I think the onus is on the present authors to defend it.  The drawback of Suciu et al. is in their evaluation, because they weave in a constant fraction of the poisoned points into each training minibatch: but this seems to be a flaw with the evaluation and not a fundamental issue with the attack.   In addition, the present authors state that the “clean-label” case that they consider is a “new type of attack” (line 72); but Suciu et al. also consider the clean-label case, though they do not explicitly use that phrase. The present authors also say that this is the first time that “clean-label poisoning in the end-to-end training scenario” has been studied, but doesn’t Suciu et al. do that too (with the caveat about mini-batching)?  Suciu et al. also performs a more comprehensive evaluation of data poisoning attacks and defenses, which the current paper does not consider. The main benefit of the current work is in the additional analysis of the image/NN poisoning setting, but the change from L_\inf to L2 norm makes it hard to compare (for example, Suciu et al. report successful poisoning attempts against CNNs without using watermarking; I’m not sure if that’s because of the change in norm, or something else).   Overall, the paper is well-written and informative but seems to have a high degree of overlap with Suciu et al.; it could be strengthened by a better comparison with that work.  Other comments: 1) I’m a bit skeptical of the experiments showing that end-to-end training is significantly harder than the transfer learning setting, because the mode architecture and dataset are different between those experiments. Is there a more comparable setting?  2) In Section 3.1 (one-shot kill attack), the empirical comparison to Koh and Liang (2017) is somewhat misleading. That paper uses the L_\inf norm to measure similarity in input space, whereas here the authors use the L2 norm. I’m not sure how to think about this: are the attack images used here more or less similar than those used in Koh and Liang? At any rate, it is not an apples-to-apples comparison, as the current writing suggests.  Minor comments: 3) Lines 125-126: Isn’t the L2 norm the same as the Frobenius norm? I’m confused as to why you’re using two different terms for the same quantity. 4) Algorithm 1: Please specify lambda 5) There’s a missing word in the first sentence of the abstract: “Data poisoning an attack”... 6) Line 59 of the supplement has a missing reference


Dropping Symmetry for Fast Symmetric Nonnegative Matrix Factorization
Symmetric nonnegative matrix factorization (NMF)-a special but important class of the general NMF-is demonstrated to be useful for data analysis and in particular for various clustering tasks. Unfortunately, designing fast algorithms for Symmetric NMF is not as easy as for the nonsymmetric counterpart, the later admitting the splitting property that allows efficient alternating-type algorithms. To overcome this issue, we transfer the symmetric NMF to a nonsymmetric one, then we can adopt the idea from the state-of-the-art algorithms for nonsymmetric NMF to design fast algorithms solving symmetric NMF. We rigorously establish that solving nonsymmetric reformulation returns a solution for symmetric NMF and then apply fast alternating based algorithms for the corresponding reformulated problem. Furthermore, we show these fast algorithms admit strong convergence guarantee in the sense that the generated sequence is convergent at least at a sublinear rate and it converges globally to a critical point of the symmetric NMF. We conduct experiments on both synthetic data and image clustering to support our result.

[Summary] In this paper, an algorithm for symmetric NMF is proposed.  The key idea of the proposed method is to optimize a matrix by optimizing two different matrices with a penalty term for the equivalence of those two matrices.  The main contribution is to prove the relationship between the original and the modified optimization problems and an equivalent condition about the penalty weight.  Computational experiments show some advantages.  [Related work] Actually, the idea of variable splitting for symmetric NMF is not original, and the same algorithm has been proposed in [10].  The proposed formulation Eq.(3) is completely equivalent to Eq.(15) in [10].  Furthermore, the proposed alternating nonnegative least squares (ANLS) algorithms is the same as in [10].  [Strengths] Several theorems were proven to clarify the relationship between the original and the modified optimization problems.  An equivalent condition about the penalty weight is provided.  Sublinear convergence properties of ANLS and HALS algorithms were proven.  [Weakness]   1) Originality is limited because the main idea of variable splitting is not new and the algorithm is also not new.     2) Theoretical proofs of existing algorithm might be regarded as some incremental contributions.   3) Experiments are somewhat weak:    3-1) I was wondering why Authors conducted experiments with lambda=1.  According to Corollary 1 and 2 lambda should be sufficiently large, however it is completely ignored for experimental setting.  Otherwise the proposed algorithm has no difference from [10].    3-2) In Figure 3, different evaluations are shown in different dataset.  It might be regarded as subjectively selected demonstrations.    3-3) I think clustering accuracy is not very significant because there are many other sophisticated algorithms, and initializations are still very important for nice performance.  It show just the proposed algorithm is OK for some applications.  [Minor points]   -- comma should be deleted at line num. 112: "... + \lambda(u-v), = 0 ...".   -- "close-form" --> "closed-form" at line num.195-196.  --- after feedback --- I understand that the contribution of this paper is a theoretical justification of the existing algorithms proposed in [10].  In that case, the experimental validation with respect to the sensitivity of "lambda" is more important rather than the clustering accuracy.  So Fig. 1 in feedback file is nice to add paper if possible.  I think dropping symmetry is helpful, however it is not new idea that is already being used. So, it will not change anything in practice to use it. Furthermore, in recent years, almost all application researchers are using some application specific extensions of NMF such as sparse NMF, deep NMF, semi-NMF, and graph-regularized NMF, rather than the standard NMF.  Thus, this paper is theoretically interesting as some basic research, but weak from an application perspective.  Finally, I changed my evaluation upper one as:  "Marginally below the acceptance threshold. I tend to vote for rejecting this submission, but accepting it would not be that bad."


Adaptive Skip Intervals: Temporal Abstraction for Recurrent Dynamical Models
We introduce a method which enables a recurrent dynamics model to be temporally abstract. Our approach, which we call Adaptive Skip Intervals (ASI), is based on the observation that in many sequential prediction tasks, the exact time at which events occur is irrelevant to the underlying objective. Moreover, in many situations, there exist prediction intervals which result in particularly easy-to-predict transitions. We show that there are prediction tasks for which we gain both computational efficiency and prediction accuracy by allowing the model to make predictions at a sampling rate which it can choose itself.

The authors propose a method to learn temporally abstract transition models for time series prediction/classification. Starting with the insight that for many tasks not all time steps need to be accurately predicted, they propose to predict the "best" inputs within a fixed horizon H instead. This is done by evaluating the reconstruction loss for all H future inputs, but back-propagate only on the smallest one, akin to pooling over losses. As a result, the method can learn a sequence of future inputs, with varying delays, that are easy to predict. The target label of a time series can be predicted based either on the last or the average predicted input. The authors evaluate their method on two complex tasks based on input-images to show it outperforms time series prediction with fixed step-sizes.  The idea to pool over the loss of future prediction is, to the best knowledge of the reviewer, quite original and appears highly relevant. It is important to distinguish the presented method from model learning for control, though, which have to be conditioned on control signals or actions. The paper is well written and the approach is clearly explained. Experiments are sufficiently complex and show the method's advantage clearly, although the reviewer was missing the comparison with another state-of-the-art algorithm as baseline. Note that the classification problem is somewhat artificially combined with the model-learning, which makes the presented method an unsupervised pre-processing step, rather than a combined classification method. It would be interesting to see if one could backprop the gradient of the classification problem somehow in the model-learning procedure. To this end, the authors are encouraged to try a RNN for classification. RNN yield gradients at each abstract time step, which could be propagated into the model-prediction.  The reviewer recommends to accept this paper. It is a well written and presents a simple, but interesting twist on time series prediction. The technique should only work in tasks where the labels depend almost exclusively on the final predicted states, but the basic idea may be applicable beyond that.   COMMENTS: l.144: this paragraph was hard to follow l.172+: "separately" twice in one sentence l.221: use the term "NN with seven convolutional layers" instead of "seven-layer fully-convolutional NN" to avoid confusion with seven fully-connected layers l.269: "proposes proposes" fig.7: plotting the best 3 out of 6 runs is a metric that can be easily manipulated and should be avoided fig.9+10: as you mention yourself, these plots are not informative fig.12: Figure 12 refers to itself and the end of the sentence is missing fig.1-12: showing the interquantile range of 3 samples is misleading. Use STD, SEM or minimum/maximum instead


Learning from discriminative feature feedback
We consider the problem of learning a multi-class classifier from labels as well as simple explanations that we call discriminative features. We show that such explanations can be provided whenever the target concept is a decision tree, or can be expressed as a particular type of multi-class DNF formula. We present an efficient online algorithm for learning from such feedback and we give tight bounds on the number of mistakes made during the learning process. These bounds depend only on the representation size of the target concept and not on the overall number of available features, which could be infinite. We also demonstrate the learning procedure experimentally.

Main Idea This paper analyzes a novel setting of interactive learning where the user gives a discriminative feature as feedback in addition to the labels. The setting is quite simple: we suppose data points form m clusters, where each cluster is pure in its label. The discriminative feature feedback is defined as the feature that differentiate two clusters. The paper shows that under this setting all possible classifiers can be represented as a certain kind of DNF, and gives an algorithm with regret at most m^2. Comments Overall, the paper is written nicely and easy to follow. However I am a bit concerned about its contribution in terms of theoretical results: The setting here seems quite simplified and resulting algorithm is not practical for most interested applications. E.g., in the Siri example in section 1, the feedback of human for why he likes the movie might be of many reasons, and the resulting feature space will be very large and sparse (and therefore there will be a large number of clusters). In this case it would be prohibitive to make up to m^2 mistakes. It is therefore important to analyze cases where (1) clusters are not pure, (2) human are not perfect, and (3) features are continuous. Minor comments  1. It will be interesting to know some lower bounds of a general algorithm in this setting.  2. Line 179: It should be "total number of conjunction terms [in] F_{y,i} is m"  Quality  The paper is well written and the analysis seems complete. Clarity The paper is mostly clear.  Originality The paper is original. Significance The paper analyzes the important problem of learning with discriminative features. However, the setting seems too restricted and simplified. --Update-- Although the authors argue that their algorithm can work under even an infinite number of features, it is not shown that how the number of groups behaves in such settings. The paper can be more influential if, e.g., the authors can related the number of groups to sparsity conditions and/or logarithm of # features. I'm therefore not changing my review score. --update 2-- After seeing discussions I would raise my score by 1 in this situation.


Adding One Neuron Can Eliminate All Bad Local Minima
One of the main difficulties in analyzing neural networks is the non-convexity of the loss function which may have many bad local minima. In this paper, we study the landscape of neural networks for binary classification tasks. Under mild assumptions, we prove that after adding one special neuron with a skip connection to the output, or one special neuron per layer, every local minimum is a global minimum.

[Updates]  After reading the author's response, I think my concerns about the existence of minima has been partly, but not completely addressed. In the begging of the response a proof for the existence of a global minima is provided. However, an important distinction has been ignored. The author wrote that "By assumption 2, there exists θ* such that f(·; θ*) achieves zero training error". However, the existence a parameter for which all the data can be correctly classified (Assumption 2) is not the same as having a parameter for which the loss function is zero. That is precisely how the counterexample I provided works. Of course, the author could avoid this problem by modifying Assumption 2 to "there exists a parameter for which the loss function is zero", or by adding one assumption stating that $f$ can be rescaled as they did in the response, which I believe they should.   Another thing I'm very interested in is how difficult is to find a local minima of the modified network. If I understand correctly, after adding the neuron, each stationary point in the previous network becomes a corresponding saddle point (if one just keep the added neuron inactive) in the modified network (except for the global minima). How does such a loss landscape affect the optimization process? Is it computationally efficient to actually find the minima? How well does the minima generalize? It would be more convincing if the authors can provide some numerical experiments.  Overall I believe this is a very good paper, and should be accepted. I've changed my overall score to 7.   [Summary of the paper]  This paper presents a rather surprising theoretical result for the loss surface of binary classification models. It is shown that under mild assumptions, if one adds a specific type of neuron with skip connection to a binary classification model, as well as a quadratic regularization term to the loss function, then every local minima on the loss surface is also a global minima. The result is surprising because virtually no assumptions have been made about the classification model itself, other than that the dataset is realizable by the model (namely, there exists a parameter under which the model can classify all samples in the dataset correctly), hence the result is applicable to many models. The paper also provides some extensions to their main result.  [Quality]  I have concerns about the main result of the paper:  -- In section 3.2, the authors add the output of an exponential neuron to the output of the network, namely, the new architecture $\tilde{f}$ is defined by  $\tilde{f}(x, \theta) = f(x, \theta) + a \exp (w^T x + b)$  Note that the added term has an invariance in its parameters, namely, if one perform the following transformation:  $a^\prime = a / C, b^\prime = b + \log C$  then the model will stay exactly the same, i.e., $a^\prime \exp (w^T x + b^\prime) = a \exp (w^T x + b)$ holds for any input $x$.   Now, consider the loss function $\tilde{L}_n(\theta, a/C, w, b + \log C)$. Because there is also a regularization term for $a$, by the invariance argument above we can see that $\tilde{L}_n(\theta, a/C, w, b + \log C)$ decreases monotonically as $C$ increases (assuming $a \neq 0$). But as $C$ increases, $b$ is pushed to infinitely faraway. This argument naturally leads to the following concern: Theorem 1 is stated as "if there is a local minima for $\tilde{L}_n, then it is a global minima", but how does one ensure that $\tilde{L}_n$ actually has a local minima at all?   To further illustrate my point, consider the following very simple example I constructed. Suppose $f(x, \theta) = 1$, i.e., the model always output 1. Suppose we only have one sample in the dataset, $(x, y) = (0, 1)$. Note that the realizability assumption (Assumption 2) is satisfied. Let the loss function be $l(z) = \max(0, 2 + z)^3$, so that Assumption 1 is satisfied. Finally let $\lambda = 2$. Now, we have   $\tilde{L}_n = \max(0, 1 - a \exp(b))^3 + a^2$  One can immediately see that this function has no local minima. To see this, note that when $a = 0$, we have $\tilde{L}_n = 1$; on the other hand, let $a = t$ for some $t > 0$, and $b = - \log t$, and we have $L_n -> 0$ as $t -> 0$, but this would also make $b -> +\infty$. Hence the function has no global minima, and by Theorem 1 it cannot have any local minima.   While this observation does not mean Theorem 1 is wrong (because theorem 1 assumes the existence of a local minima), it does limit the scope of Theorem 1 in the case where local minimas do not exist.   [Clarity]  The paper is well written and well organized. The proof sketch is also easy to follow.   [Originality]  To the best of my knowledge, the results presented in the paper are original.   [Significance]  I really like the results presented in this paper. It is quite surprising that by making very simple modifications of the model, one can eliminate bad local minimas, especially given the fact that little assumptions on the model itself are needed. Despite so, I feel that the significance of the results might be slightly less than it appears:  -- As mentioned in the [Quality] part, there are cases where the loss function of the modified model has no local minima at all. In such cases, the theorems in the paper do not apply. It is not clear to me what conditions are needed to guarantee the existence of local minimas. It would be nice if the authors can address this issue.   -- The theorems in the paper do not actually make any assumptions on the model $f$ except that there exist parameters with which $f$ can correctly classify all samples in the dataset. While this makes the results very general, this unfortunately also implies that the paper is not really about loss surface of neural networks, but rather a general way to modify the loss surface that can be applied to any model so long as the realizability assumption is satisfied. The results seem to have nothing to do with neural networks, and hence it does not really add anything to our understanding of the loss surface of neural networks.  The assumptions made in the paper seem reasonable enough to be satisfied in realistic settings. It would be nice if the authors can present some numerical experiments.


Generative Modeling for Protein Structures
Analyzing the structure and function of proteins is a key part of understanding biology at the molecular and cellular level. In addition, a major engineering challenge is to design new proteins in a principled and methodical way. Current computational modeling methods for protein design are slow and often require human oversight and intervention. Here, we apply Generative Adversarial Networks (GANs) to the task of generating protein structures, toward application in fast de novo protein design. We encode protein structures in terms of pairwise distances between ↵-carbons on the protein backbone, which eliminates the need for the generative model to learn translational and rotational symmetries. We then introduce a convex formulation of corruption-robust 3D structure recovery to fold the protein structures from generated pairwise distance maps, and solve these problems using the Alternating Direction Method of Multipliers. We test the effectiveness of our models by predicting completions of corrupted protein structures and show that the method is capable of quickly producing structurally plausible solutions.

Overview In silico protein folding is an essential part of many bioengineering pipelines; however, all existing methods are highly computationally expensive, which makes the generation of large numbers of candidate protein structures infeasible. The authors propose an alternative to the incumbent biophysical models (the most common of which is Rosetta) which uses instead a generative adversarial network (GAN) to generate a pairwise distance matrix for alpha-carbons within a variable-length amino acid chain. The authors then use an existing algorithm called Alternating Direction Method of Multipliers (ADMM) to reconstruct the 3D position of the alpha-carbons, and finally use a partial solution from Rosetta to obtain the full amino acid backbone structure.  Technical quality  The authors’ method is clearly and adequately described. The experiments are well thought out and results are presented fairly. In the de novo protein folding experiment, the authors compare their method to five other protein folding methods. Aside from 3DGAN (Wu et al, NIPS 2016), it is not clear which (or any) of these are published methods, or whether they were partial methods invented by the authors. Regardless of the origin of these methods, the authors show that potential alternative formulations of their method using torsion angles, full-atom GANs, or Hidden Markov Models are insufficient for de novo protein generation.  However, between the GAN folded by fragment sampling and the GAN folded by ADMM, it is worth noting that none of the proteins generated by the authors’ chosen method (ADMM) contain any beta sheets. This pattern is clearly repeated in the inpainting experiment, where not one of the proteins in which the original inpainted section contained a beta sheet (1mva, 3h3i, 4ang, 1ahk, 3ame) has a beta sheet when inpainted by ADMM GAN. As one of the most common tertiary amino acid structures, the failure to recapitulate a beta sheet is a severe technical flaw in the proposed method.  Additionally, as noted by the authors, the metrics used for numerical comparison of the tested methods are significantly flawed. The authors note that “in the case where the inpainted solution is plausible but deviates from the native structure, the rigid-body alignment error will be high”. However, the authors could at the very least verify that the GAN is capable of producing *a* plausible solution by examining the distribution of rigid-body alignment errors for all possible GAN inpainting solutions. If the minimum (or e.g. 10th percentile) rigid-body alignment error across hundreds of inpainting solutions approaches the scores achieved by Rosetta, this would be a convincing argument that the GAN is capable of generating biologically plausible solutions, and that other solutions with higher rigid-body alignment error are simply alternative solutions less similar to the original protein that may still be valid folded proteins. Novelty While the authors’ proposed algorithm is indeed a novel pipeline, and to date the first attempt to apply GANs to protein modeling, it should be noted that none of the component parts of the algorithm are novel. ADMM was invented in 2011 and has been previously applied to protein folding, and the semantic inpainting method used in section 4 was previously described in 2016. Thus the authors present a novel application of existing technologies.  Potential impact  The authors’ work has the potential to be highly impactful, contingent on the resolution of the technical failure to produce beta sheet structures. De novo protein design is a highly computationally expensive task, and additionally requires human intervention. A fast, automated and performant method to design proteins in a high-throughput manner could revolutionize drug design; however, the authors have not yet demonstrated that their algorithm is genuinely competitive with slower alternatives.  Clarity The paper is well written and easy to read.   Minor comments: - “Our model for the scope of this paper is not necessarily learning protein structures which will fold, but rather is learning building block features that define secondary and tertiary structural elements.” – this sentence is confusing.  - The authors claim that the semidefinite program to infer 3D coordinates from pairwise distances is quadratic, and that ADMM by contrast converges “quickly”. What is quickly? Could this be shown, perhaps in a supplementary figure? - It would also be good to see the difference in performance of the ADMM folding method compared to the GAN + fragment sampling shown in Figure 3. - Equation 3 fails to define \lambda and Z. - Equation 4: should G_{k+1} be dependent on G_k, rather than G? - If is not clear how the alpha-carbon trace algorithm works, and whether it is a custom script written by the authors, or included in the Rosetta package.  The authors present a novel application of GANs in a field where the impact of such a technology could be very high.  (post response) The authors have addressed my major concern with the manuscript, the inability to produce beta sheets, as this inability is caused not by the ADMM GAN but by the protein rendering software which creates the cartoons. Discussion of how to overcome this issue, in addition to added reporting on runtimes and the distribution of rigid-body alignment error will make more convincing the capability of GANs to revolutionize the field of in silico protein modeling. Recommendation: weak accept.


Multi-value Rule Sets for Interpretable Classification with Feature-Efficient Representations
We present the Multi-value Rule Set (MRS) for interpretable classification with feature efficient presentations. Compared to rule sets built from single-value rules, MRS adopts a more generalized form of association rules that allows multiple values in a condition. Rules of this form are more concise than classical singlevalue rules in capturing and describing patterns in data. Our formulation also pursues a higher efficiency of feature utilization, which reduces possible cost in data collection and storage. We propose a Bayesian framework for formulating an MRS model and develop an efficient inference method for learning a maximum a posteriori, incorporating theoretically grounded bounds to iteratively reduce the search space and improve the search efficiency. Experiments on synthetic and realworld data demonstrate that MRS models have significantly smaller complexity and fewer features than baseline models while being competitive in predictive accuracy. Human evaluations show that MRS is easier to understand and use compared to other rule-based models.

Review:  The paper proposes a bayesian rule based model that allows for the inclusion of multiple values of the same feature inside a condition. The model is composed of a set of rules, and an example is classified as positive if it is covered by at least one rule. The generative model has knobs to encourage three interpretability-related properties :  - Small number of rules  - Small number of conditions within each rule  - For each rule, small number of features used (i.e. conditions are clustered into few features)  The improvements of MRS over other rule-based frameworks are compelling. Rule sets that use less features are easier to grasp, and rules with multiple values for the same feature are more intuitive and easier to grasp than the same condition fragmented over multiple conditions.  The paper is mostly clear, but there is a lot of notation. Some suggestions to improve clarity:  - Maybe draw the generative model  - Change the notation of multiple values to [state \in {California, Texas, Arizona, Oregon}] rather than [state =California or Texas or Arizona or Oregon], or in the very least change the spacing around the =  - Drop the second _m in {L_m}_m, {z_m}_m, etc.  - The subscript plus and minus in 4.1 makes the equations very hard to read  Comments / questions:   - Is there a reason for not constraining alpha_+ > beta_+ and alpha_- > beta_- in the formulation? This would avoid the repetition of this condition around Lemma 1.  - It seems that removing a condition reduces the coverage, while adding a condition increases the coverage. These are reversed in 4.2.  - The comparison of number of conditions in Table 1 does not seem entirely fair, as the conditions for MRS are much longer, unless the authors are considering each value in a condition as a separate condition for this count.  - The discussion in lines 326 - 330 does not seem to make sense given that "All of them" is a value included in the condition. This value breaks the intuition presented.  - It's not clear how the authors selected the best model in cross validation after doing grid search. Was the selection done in a systematic or ad hoc way?  Overall, I think this is an interesting interpretable model, with the caveats that it only works for binary classification (and the extensions to more labels is not obvious) and that there are many hyperparameters that need tuning. The results dot not present significant improvements over prior art in terms of accuracy, but there seem to be some improvements in terms of interpretability (although the comparison may not be fair, see comment above).  Typos: 26: primitive concept level -> primitive concepts 51: "an examples" -> examples, "fo example" -> for example 55: for its appealing property (weird phrasing) 68: remove 'the' 73: remove stop before (Millions) 75: 'and it was done…'  weird phrasing 78: various work  79: conjunction connector -> the conjunction connector 81: conjunction/literal -> conjunction / literal (add space) 94: 'simply adapt' -> be adapted 306: uniqeu -> unique 310: 'accuracy 67 conditions' - something is missing here 316: two stops after 'meaningful' 342: MARS -> MRS  Supplementary line 6: MARS -> MRS  # Comments on feedback: - Thanks for addressing the concern about fair comparison - I think that the user study makes the submission stronger, and so do the added results in the new Table 1. I've changed my score to reflect that. - I had understood that 'all of them' is a value like the others. I am question the intuition presented, which states that people prefer to say they are not sure or refuse to answer. This intuition does not fit well with the inclusion of 'all of them' on the right hand side of the rule.


Evidential Deep Learning to Quantify Classification Uncertainty
Deterministic neural nets have been shown to learn effective predictors on a wide range of machine learning problems. However, as the standard approach is to train the network to minimize a prediction loss, the resultant model remains ignorant to its prediction confidence. Orthogonally to Bayesian neural nets that indirectly infer prediction uncertainty through weight uncertainties, we propose explicit modeling of the same using the theory of subjective logic. By placing a Dirichlet distribution on the class probabilities, we treat predictions of a neural net as subjective opinions and learn the function that collects the evidence leading to these opinions by a deterministic neural net from data. The resultant predictor for a multi-class classification problem is another Dirichlet distribution whose parameters are set by the continuous output of a neural net. We provide a preliminary analysis on how the peculiarities of our new loss function drive improved uncertainty estimation. We observe that our method achieves unprecedented success on detection of outof-distribution queries and endurance against adversarial perturbations.

- Novelty is below the NIPS level. The claim of using the DST theory was stated in the beginning (L36) as: achieving a "richer" representation. I did not see through the rest of the paper why this is a "richer" representation.   - The experiments are rigorously performed and analysed. They well serve the cause of the paper.   - The text in the introduction is trying to be too verbose for a scientific paper at this level. There is no need for that in my opinion. An example is: "All these (no longer) futuristic ... Are they futuristic or are they not? I'd go in favour of a more rigorous and unequivocal (and still eye-catching) presentation style.   - L85: 13]. "Hence, it allows one to use the principles of evidential theory to quantify belief masses and uncertainty through a well-defined theoretical framework.": Ok, but why is using the principles of evidential theory preferred? A Dirichlet can be used without resorting to this theory, i.e. in a Bayesian setting. The motivation can be a bit clearer here too (in the beginning of Section 3, I mean).  - There are some grammar mistakes and typos. Examples include:   -- L64: "The consequence is then unreliability of ..."  -- L75: "our approach proposed in this paper ..."  - L22: Who said that the "saturation" is "fast approaching"? I mean are there citations demonstrating that? Or is that just a speculation?


Predictive Approximate Bayesian Computation via Saddle Points
Approximate Bayesian computation (ABC) is an important methodology for Bayesian inference when the likelihood function is intractable. Sampling-based ABC algorithms such as rejection-and K2-ABC are inefficient when the parameters have high dimensions, while the regression-based algorithms such as K-and DR-ABC are hard to scale. In this paper, we introduce an optimization-based ABC framework that addresses these deficiencies. Leveraging a generative model for posterior and joint distribution matching, we show that ABC can be framed as saddle point problems, whose objectives can be accessed directly with samples. We present the predictive ABC algorithm (P-ABC), and provide a probabilistically approximately correct (PAC) bound for its learning consistency. Numerical experiment shows that P-ABC outperforms both K2-and DR-ABC significantly.

Thank you for your response that answers my main concerns. I am still voting for accepting the paper. --------------------------------------------------------------------------------------------  This paper considers the problem of performing Bayesian inference when dealing with intractable likelihood. Approximate Bayesian Computation (ABC) is now a well established method to handle complex likelihood which does not admit closed form or is computationally too expensive to be computed. Nevertheless, such methods present limitations like : - they are delicate to tune as they require a non one-to-one function of the data (summary statistics) and the definition of a neighbourhood of the data with respect to some metric, - they may be difficult to scale up. To overcome such issues, the authors propose to move from the initial ABC paradigm towards a saddle point formulation of the problem. They suggest to use the ability to sample from complex model (like in ABC) in order to solve an optimisation problem which provides an approximation of the posterior of interest. The authors compare the performance of their approach with some particular examples of ABC algorithms on synthetic and real datasets.  The motivation of the paper and the advocated solution are well explained. The existing literature on ABC is rather dense but the authors managed to give a broad overview of the different solutions. I have only few comments about the existing literature that is not mentioned in the paper: - to be completely fair, the authors could have mentioned ABC random forests for Bayesian parameter inference (Raynal et al. 2016) as it is a solution which avoids tuning the neighbourhood and reduces the need for summary statistic selection, - regarding the formulation in terms of optimisation problem, it would have been interesting to provide a discussion, a comparison or a mention of the work of Moreno, et al. (2016, Automatic Variational ABC) and Fasiolo, et al. (2017, An Extended Empirical Saddlepoint Approximation for Intractable Likelihoods). Do you have any input or comment on this?  The paper is overall properly written and easily to follow. I just have few minor comments about the readability: - the definition of the representation of $\theta$ for the posterior distribution in section section 3.1 (l.199 - l.204) is redundant as it is already defined on line l.97 - l.104. I would suggest to do the mention from l.199 - l.204 straight before section 2.1 (sticking to the word representation rather than reparametrization) as it is used in Equations (4), (5) and so on... and remove it from section 3.1, - Equation (7) and equation (10) are the same. In section 3.1, the authors refer alternatively to (10) or (7). I would suggest to remove equation (10) and introduce the definition of $\Phi$ in equation (7) and then specifying $f(Y,\i)$ and $u(\theta,Y)$ when needed.  The idea is original and very interesting as it brings some efficient machine learning techniques where existing ABC methods may fail or struggle. The results seem promising though it would have been interesting to have a more comprehensive comparison with existing methods.   The main limitation I see to this approach is the inability to reach the optimal solution of the saddle point formulation (7) as in some variational approaches. This would end up to rough or wrong approximations of $f*$ and $u*$ and hence to a misspecified posterior approximations. Do you have any comments on this?  I spotted few typos that I listed below :  l.101 modelstive? l.102 mdoels -> models l.108 p(Y|\theta)p(\theta) -> p(Y|\theta)\pi(\theta) l.112 Shouldn't be the expectation of equation (4) with respect to q(\theta|Y)p(Y) instead of with respect to p(\theta|Y)p(Y)? l.134 experessive -> expressive