Contrastive Learning from Pairwise Measurements
Learning from pairwise measurements naturally arises from many applications, such as rank aggregation, ordinal embedding, and crowdsourcing. However, most existing models and algorithms are susceptible to potential model misspecification. In this paper, we study a semiparametric model where the pairwise measurements follow a natural exponential family distribution with an unknown base measure. Such a semiparametric model includes various popular parametric models, such as the Bradley-Terry-Luce model and the paired cardinal model, as special cases. To estimate this semiparametric model without specifying the base measure, we propose a data augmentation technique to create virtual examples, which enables us to define a contrastive estimator. In particular, we prove that such a contrastive estimator is invariant to model misspecification within the natural exponential family, and moreover, attains the optimal statistical rate of convergence up to a logarithmic factor. We provide numerical experiments to corroborate our theory.

The paper considers a family of statistical models for pairwise measurements, in which we observe scores that come from triplets of the form (user, item1, item2). The problem is to estimate the underlying score function of user-item pair, which is assumed to be low-rank, and the observation to be generated by a natural exponential family.   The authors address the problem from a novel angle as a semiparametric estimation problem, which avoids the need to specify the log-partition function in the natural exponential family.  A penalized contrastive estimator is derived and its consistency and error bound are analyzed.  Numerical simulation supports the rate established in the bound.  I find this a nice addition to the learning from pairwise comparison literature. I recommend accept.    Detailed Comments:   1. Equation (3.2) (3.4)  effectively reduces the problem to a logistic loss matrix sensing problem with a specific randomly generated sensing matrix.  Can a similar robustness to model misspecification be stated for other sensing matrices and other link function? 2. The exponential family distribution of p(y|Theta) really needs to be a natural exp. family.   In general, one still needs to specify the model class in order to reparameterize a given exp family to a natural one. In that sense, the semiparametric view is not as strong as it seems.  3. Assumption 4.3 is somewhat strong as it rules out Gaussian, Poisson and many others. Perhaps it suffices to use sub-gaussian or sub-exponential?


SING: Symbol-to-Instrument Neural Generator
Recent progress in deep learning for audio synthesis opens the way to models that directly produce the waveform, shifting away from the traditional paradigm of relying on vocoders or MIDI synthesizers for speech or music generation. Despite their successes, current state-of-the-art neural audio synthesizers such as WaveNet and SampleRNN [24,17] suffer from prohibitive training and inference times because they are based on autoregressive models that generate audio samples one at a time at a rate of 16kHz. In this work, we study the more computationally efficient alternative of generating the waveform frame-by-frame with large strides. We present SING, a lightweight neural audio synthesizer for the original task of generating musical notes given desired instrument, pitch and velocity. Our model is trained end-to-end to generate notes from nearly 1000 instruments with a single decoder, thanks to a new loss function that minimizes the distances between the log spectrograms of the generated and target waveforms. On the generalization task of synthesizing notes for pairs of pitch and instrument not seen during training, SING produces audio with significantly improved perceptual quality compared to a state-of-the-art autoencoder based on WaveNet [4] as measured by a Mean Opinion Score (MOS), and is about 32 times faster for training and 2, 500 times faster for inference.

Neural audio synthesis is a important and rapidly evolving field of study. Recent breakthroughs rely on expensive audio-sample level autoregressive models. The authors demonstrate a simpler frame-wise regression approach can do better than expected by generating raw audio, but then performing an STFT and regressing to targets in the log-spectral domain. This results in considerable speedups in inference time, as the sound can be generated in parallel. The authors wisely focus on NSynth as a controlled dataset to test conditional generation in a limited domain, and demonstrate generation that generalizes to unseen conditioning, generates audio that is low in distortion, and scores well on MOS tests.  Strengths: + STFTs as differentiable operations are a very promising direction. Despite being an obvious choice, sucessful examples of their use are relatively rare, and is the core of the novelty of this paper.   + First demonstration of low-distorion audio generated framewise with learned convolutional filters. Framewise filters are usually somewhat noisy when learned to generate raw audio. The combination of the spectral loss and the 'overlap-add' windowing are likely responsible, and demonstrating this is of broad value to those working with machine learning and audio.  + Conditional generation of pairs unseen pairs of conditioning demonstrates generalization of the model and the strong use of the conditioning variables. Forcing autoregressive models to utilize conditioning information is sometimes still a challenge as the model can sometimes learn to ignore them and rely on the autoregression for modeling dependencies.  Room for improvement: - The paper is a bit confused about its treatement of phase. The model uses a log power spectra as the regression loss function, which is invariant to phase values of local bins. The authors seem to indicate in L45-46 that this is good because audio perception is also invariant to phase, but this is the not the case. Audio perception is very sensitive to phase alignment from frame-to-frame and between harmonics in a single frame. Humans are relatively unaware of *constant phase offsets* applied to the whole spectrogram, which equate to a translation of the audio forward or backwards by a few milliseconds. The log-magnitude loss is also insensitive to constant phase offsets, but gives no information to the model as to how to realistically align the generated phases between frames and frequencies. This is very apparent in the generated audio, as indicated by a percieved lower frequency "warble" or "ringing". Autoregressive models handle the phase problem by working at the lowest possible stride, and it is clear that this model does not comperably handle phase. Section 3.2.1 also misrepresents the situation, by considering an example signal that is a single harmonic and a single frame. Even for a single harmonic, the value of the phase will precess at a constant rate in a stft porportional to the difference of frequency between the fft bin and the harmonic frequency. This is why the derivative of the unwrapped phase for a constant tone is a constant, giving the solid colors in the "Rainbowgrams" of the original NSynth paper. That said, the generated audio sounds much better than magnitudes with random phases. It would improve the paper greatly to more accurately describe the relationships of phase alignment and directly address phase as a limitation of the current technique. For example, presenting rainbowgrams of real, wavenet, and sing generated audio could highlight what aspects sing does well at, despite not receiving any phase error info during training, and what needs improvement. It would be good to include those graphs in the supplemental for all audio in the supplemental. Similarly, a baseline model that just estimates magnitudes for each frame and then uses Griffin-Lim to estimate phase would be very important for arguing that generating directly in the audio domain is an improvement for frame-wise generation. An extension to the current model, using log-magnitude and derivative of unwrapped phase as regression targets would perhaps generate more phase coherent audio.  - Reliance on temporal encodings reduces the generality of the proposed architecture. This requires the dataset to be preprocessed and well-aligned, which by design is the case for for the NSynth dataset.  - There are a few simple simple metrics that could be added to evaluations to make them more thorough. Since there are labels for the dataset, training a separate classifier could give classification accuracy (pitch, velocity, instrument). The authors could also use the network to compute Frechet Inception Distance (FID), as a proxy for "quality". It would allow for investigation of which pitches/velocities/instruments for which SING does better.  Small other comments: * The authors of the NSynth paper actually note that it takes 10 days on 32 K40 GPUs, which are significantly slower than P100 GPUs. (https://github.com/tensorflow/magenta/tree/master/magenta/models/nsynth). While the SING model still is much faster train it's probably more like a factor of 30-40 in reality. Either demphasizing the specific numbers of the cliam (not mentioning in the abstract), or coming up with a more quantitative measure (ex. FLOPS) would be more transparent.  * The text of the paragraph on L61 could be clearer. It could just be said that prior work focuses on speech and compositions of symbolic notes, while the current goal is learning timbre of individual notes.   * L94, the NSynth model can generate conditional waveforms (pitch, instrument, velocity) since it is an autoencoder (this is how the authors make a playable instrument with it). Also the baseline model generalized to generate pitches that were not seen at training. I think the distinction the authors are trying to make, that should be emphasized is that SING is a generative model, not an autoencoder, so requires no initial audio to condition on.  * L137, "spectal loss instead of a regression loss". The spectral loss _is_ a regression loss. I think the authors are trying to distinguish between regression to spectral coefficients and regression to wavefunction values.   * L193, "fasten", I think the authors mean to say "speed up" or "hasten"  * MOS of random mechanical turkers is perhaps less meaningful than in the case of speech, because while average people are experts at producing and listening to speech, the same cannot be said for music. For example, the phase artifacts present in SING might stand out quite strongly as unrealistic to a music producer or researcher, but less so to a casual listener who might associate "distortion" more with white noise or quantization errors. There's not a lot that the authors can do about that in the context of their experiments, but I just wanted to note, because it was strange to me that the MOS was similar for the SING and wavenet models, while to my ears the wavenet was far preferable most of the time.


Non-Local Recurrent Network for Image Restoration
Many classic methods have shown non-local self-similarity in natural images to be an effective prior for image restoration. However, it remains unclear and challenging to make use of this intrinsic property via deep networks. In this paper, we propose a non-local recurrent network (NLRN) as the first attempt to incorporate non-local operations into a recurrent neural network (RNN) for image restoration. The main contributions of this work are: (1) Unlike existing methods that measure self-similarity in an isolated manner, the proposed non-local module can be flexibly integrated into existing deep networks for end-to-end training to capture deep feature correlation between each location and its neighborhood. (2) We fully employ the RNN structure for its parameter efficiency and allow deep feature correlation to be propagated along adjacent recurrent states. This new design boosts robustness against inaccurate correlation estimation due to severely degraded images. (3) We show that it is essential to maintain a confined neighborhood for computing deep feature correlation given degraded images. This is in contrast to existing practice [41] that deploys the whole image. Extensive experiments on both image denoising and super-resolution tasks are conducted. Thanks to the recurrent non-local operations and correlation propagation, the proposed NLRN achieves superior results to state-of-the-art methods with many fewer parameters. The code is available at https://github.com/Ding-Liu/NLRN.

This paper introduced a mathematical framework to describe nonlocal operations in image restoration, such as nonlocal means, low-rank models, etc. Then the authors use the framework as a module of an RNN and validated the network on a few datasets for image denoising and super-resolution. Overall, I see merits of this paper, while I have a few questions/concerns. (See below.)  Pros:  1) The proposed framework is a nice summary of exsiting nonlocal operations commonly used in image restoration.  2) The proposed deep model works epseically well on texture images, and has advantage over several exsiting models.   Cons:  1) It has already known to the literature that some nonlocal operations can be put in matrix/operator forms. The authors claims they were the first to propose such framework. However, I think they should address the relation and differences between their framework and existing ones, such as the ones listed below: -- Yin, Rujie, et al. "A tale of two bases: Local-nonlocal regularization on image patches with convolution framelets." SIAM Journal on Imaging Sciences 10.2 (2017): 711-750. -- Danielyan, Aram, Vladimir Katkovnik, and Karen Egiazarian. "BM3D frames and variational image deblurring." IEEE Transactions on Image Processing 21.4 (2012): 1715-1728.  2) The authors emphasized a multiple places as one of their contributions that local search regions for similarity should be used. This is one of the main differences between their model with [37]. However, such treatment is a common practice for nonlocal means. The other two differences from [37] seems minor. From the title of the paper, the non-localness is the main selling point of the paper, which has already appeared in [37].  3) In the mathematical framework, the use of correlation features is not included. However, it appears in RNN model. From Table 2, with or without correlation propergation does not seem to make much difference. Then why bother introducing the additional path of s_corr in the network? I might have misinterpreted the paper, but in any case, I think this should be further clarified.   4) How does the proposed network compare with some exsiting nonlocal+dl denoising models such as  Lefkimmiatis, Stamatios. "Non-local color image denoising with convolutional neural networks." In Proc. IEEE Int. Conf. Computer Vision and Pattern Recognition, pp. 3587-3596. 2017.


Improving Explorability in Variational Inference with Annealed Variational Objectives
Despite the advances in the representational capacity of approximate distributions for variational inference, the optimization process can still limit the density that is ultimately learned. We demonstrate the drawbacks of biasing the true posterior to be unimodal, and introduce Annealed Variational Objectives (AVO) into the training of hierarchical variational methods. Inspired by Annealed Importance Sampling, the proposed method facilitates learning by incorporating energy tempering into the optimization objective. In our experiments, we demonstrate our method's robustness to deterministic warm up, and the benefits of encouraging exploration in the latent space.

POST-REBUTTAL  Having read the rebuttal, I am inclined to keep my score as is. I like the paper and the main reason why I am not giving a higher score is because I think the proposed solution AVO (an ad hoc collection of objectives) seems rather crude. I look forward to future work that improves upon the currently proposed AVO.  I still think that one needs to be careful with the "optimization process" claim. In the rebuttal, the authors stated that "both HVI and NF are highly expressive families of distribution that can provide a good approximate of a potentially highly multimodal target."  There are two problems with this line of argument:  1. The correctness of the aforementioned quote depends on the expressivity of amortization family (aka encoder size). In the limit of universal function approximation, I agree with the statement. For finite-depth networks used in practice, I do not think we have the necessarily empirical evidence to back up the claim.  2. Let's suppose the encoder has "infinite capacity." Any argument that there is an issue with the optimization process is *very difficult to support*. If one readily admits that the optimization process is suboptimal, It is difficult to say for sure that the suboptimal optimization process will necessarily result in mode-seeking behavior---even if the objective itself is known to be mode-seeking. This is because the mode-seeking characterization of the objective is only guaranteed for the optimal solution.   Ultimately, this issued is resolved via their empirical experiments, which showed consistent mode-collapsing behavior. I therefore think the authors could be more careful about the way they structured their claim.  --------------------------------------------------------------------------------  SUMMARY OF PAPER  The paper makes the interesting observation that ELBO objective encourages the variational posterior to adopt a mode-seeking behavior, and that this bias will persist even when a more expressive variational family is used due to optimization issues. To remedy this, they introduce a novel regularizer inspired by Annealed Importance Sampling that supposedly biases the variational posterior to be less mode-seeking. They demonstrate the value of the proposed regularizer on several illuminating toy examples as well as MNIST/OMNIGLOT density estimation.  --------------------------------------------------------------------------------  SUMMARY OF EVALUATION  This is quite an interesting paper that highlights the mode-seeking behavior of variational inference. While the mode-seeking behavior of variational inference is well-known, little has thus far been written in the VAE community about how this biases the generative model to adopt unimodal posteriors. I think this paper will help draw attention to this important issue.  Pros:  + Generally well-written + Addresses an important problem + Proposes an interesting solution (AVO) + Insightful experiments  Cons:  - AVO is kind of hacky - Not completely clear why AVO encourages exploration  --------------------------------------------------------------------------------  NARRATIVE  The narrative is sound; there are only a few things I want to remark:  1. Is the problem the optimization *process* or the optimization objective?  To me, optimization process suggests that the problem has to do with the choice of optimizer, and that access to an oracle optimizer will solve the issue. The issue as presented in this paper, however, is regarding the inherent mode-seeking behavior of the variational inference objective which trains the variational posterior via reverse-mode KL.  2. More discussion about the AVO   There are a couple of things that make AVO peculiar and worthy of more exposition.  First, it is interesting that AVO is not a single optimization problem, but a heuristic collection of T separate optimization problems. My guess is that this design is forced by the fact that optimization of the normalized f-tilde (w.r.t. theta) and the marginal q(z) are intractable. The fact that the AVO is an ad-hoc collection of T objective problems makes it harder to truly characterize how AVO will behave. It also raises the question how the behavior of AVO might have differed had it been a single, fully-differentiable objective trained on both inference and generative model parameters. I think these issues should be discussed in the paper so that future readers are inspired to come up with potentially better solutions than AVO.  Second, it is not obvious why AVO necessarily encourages exploration. I would appreciate a more thorough discussion on why the authors believe exploration is implied by AVO. At a very high level, OVA still more or less minimizes the KL(q || f*r), which still (arguably) biases q toward mode-seeking (w.r.t. f*r). Why then is this not a source of concern? On a related note, I think the sentence "in the case of amortized VI, we set f0 to be the prior distribution of the VAE, so each intermediate target has a more widely distributed density function, which facilitates exploration of the transition operators" should be expanded upon to emphasize the potential importance of designing f0 to be a broad distribution. In fact, I would appreciate additional experiments for Table 1 showing what would happen if f0 (and q0) is instead trainable.  3. Checking the unimodality claim  The unimodality-bias of ELBO popped up a number of times. While I agree that this is probably true, it would be nice to empirically check whether the ELBO-trained VAEs tend to end up with unimodal posteriors, and whether AVO-trained VAEs end up with non-unimodal posteriors (or something else?). If there is anyway of checking the number of modes in either the true posterior or variational posterior, this would be an interesting experiment.  --------------------------------------------------------------------------------  EXPERIMENTS  1. Show ELBO values for Figure 3  I would like to see the ELBO values for the q learned by HVI-AVO v. HVI-ELBO. Ideally, we should see that HVI-AVO has a worse ELBO.   2. Where is appendix D?  I only see appendix A and B in the supplementary file. I would like to check how the entropy of q_T(z_T) is computed to get a sense of how tightly the lower bound approximates the the log marginal.   3. Beta-annealing schedule used in 5.3  Can you describe more precisely what is the experimental set-up for 5.3? I'm afraid I don't understand the sentence "x-axis: the percentage of total training time it takes to anneal β back to 1." Is it stated anywhere that the beta-annealing schedule is also linear?  4. AVO applied to amortized VI  There are a number of ways one could potentially construct the amortized forward and backward transition operators depending on how you choose to condition on x. It would be nice if graphical model figure is provided somewhere in the paper, showing the exact choices of generative/inference model structure used in the experiments (e.g. see Auxiliary Deep Generative Model Fig. 1 for example).  5. Log-likelihood estimation for Table 1  Please state in the paper how the log-likelihood is estimated.  6. Not setting f0 to be the prior  I'm quite interested in what would happen if f0 is not set to be the prior distribution of the VAE.


Optimization for Approximate Submodularity
We consider the problem of maximizing a submodular function when given access to its approximate version. Submodular functions are heavily studied in a wide variety of disciplines since they are used to model many real world phenomena and are amenable to optimization. There are many cases however in which the phenomena we observe is only approximately submodular and the optimization guarantees cease to hold. In this paper we describe a technique that yields strong guarantees for maximization of monotone submodular functions from approximate surrogates under cardinality and intersection of matroid constraints. In particular, we show tight guarantees for maximization under a cardinality constraint and 1/(1 + P ) approximation under intersection of P matroids.

The authors consider the problem of approximate submodular maximization, i.e. the maximization of functions that can are defined via a submodular function whose set evaluatons are scaled by iid (static) noise. The goal is to maximize the underlying submodular function under cardinality and matroid constraints. The authors develop algorithms for that purpose and proof certain approximation guarantees for them.  While the problem itself in its basic version has been studied already, this paper presents improved algorithms and analysis as well as extensions of the considered constraints involving substantial different analysis. I didn't check proofs in details, but on a high-level the author's approach appears to be correct. So novelty and technical contribution certainly make the paper qualify for NIPS.  In terms of presentation of the paper, improvements could be made. For instance, I believe that putting parts of appendix E (additive approximation guarantees, correlated approximations, and more connections to related work) would make the paper more compelling than showing partial proofs (I see little value added compared to the high level steps as large parts of the technical contributions are in the appendix anyway).  Furthermore, an empirical evaluation would help the reader get an intuition of the importance of parts of the results and where to focus for future algorithmic developments. In particular, the authors argue that they overcome fundamental limitations of previous algorithms, which I understand from their theory, but it would be interesting to see whether this makes a difference in practice. It should be relatively straightforward to implement such an experiment.  * f_S(z) seems to be undefined.  --  I have read the author response and thank the authors for their comments. I think this is an interesting paper with significant relevance to a group of NIPS attendees.


Power-law efficient neural codes provide general link between perceptual bias and discriminability
Recent work in theoretical neuroscience has shown that efficient neural codes, which allocate neural resources to maximize the mutual information between stimuli and neural responses, give rise to a lawful relationship between perceptual bias and discriminability in psychophysical measurements (Wei & Stocker 2017, [1]). Here we generalize these results to show that the same law arises under a much larger family of optimal neural codes, which we call power-law efficient codes. These codes provide a unifying framework for understanding the relationship between perceptual bias and discriminability, and how it depends on the allocation of neural resources. Specifically, we show that the same lawful relationship between bias and discriminability arises whenever Fisher information is allocated proportional to any power of the prior distribution. This family includes neural codes that are optimal for minimizing L p error for any p, indicating that the lawful relationship observed in human psychophysical data does not require information-theoretically optimal neural codes. Furthermore, we derive the exact constant of proportionality governing the relationship between bias and discriminability for different choices of power law exponent q, which includes information-theoretic (q = 2) as well as "discrimax" (q = 1/2) neural codes, and different choices of decoder. As a bonus, our framework provides new insights into "anti-Bayesian" perceptual biases, in which percepts are biased away from the center of mass of the prior. We derive an explicit formula that clarifies precisely which combinations of neural encoder and decoder can give rise to such biases.

This paper generalizes previous work by Wei & Stocker (2017), by showing that a general class of variational objectives (“power-law efficient codes”) can produce the same psychophysical law that relates perceptual bias and discriminability. The work derives a strong form of the law, that holds in the asymptotically high SNR range (in which the prior is dominated by the likelihood), and a “weak” form which is nonlinear but works in a regime of non-negligible noise. In both cases, the paper explicitly derives the proportionality constants for the psychophysical law. The framework of this manuscript is slightly different than that used originally by Wei & Stocker (2017), but the authors show (mostly in the Supplementary Material) that the results hold also for the original framework (in particular, the encoding differs between the two, even though the two have the same Fisher information).  Comments:  This paper is technically sound, very well written and clear despite the complexity of the subject. The paper takes off from previous work by Wei and Stocker (2017) which proposed a novel psychophysical law (a lawful relation between perceptual bias and discriminability), but provides substantial novel content in a derivation of an entire class of “power-law” codes that would induce the same relation. This is important as it implies that the proposed psychophysical law could emerge for many different kinds of codes, and not only for an information maximizing code as it was assumed in previous work.  At the moment, it’s not clear where the ansatz for Eq. 21 comes from (I can guess it emerged from visual inspection of Figure 5C, and trying out a few functional forms). The authors might want to refer to Fig 5 here. (In fact, there is no reference to Fig 5 in the main text.)  This is a strong submission with novel theoretical results that lead to experimental predictions for psychophysical experiments (e.g., in terms of the coefficient of proportionality between bias and discriminability), and novel ideas that elaborate on the concept of efficient coding.  Typos: line 39: disriminability → discriminability line 138: differ → differs line 151.5 (unnumbered equation): remove either log on the left or exp on the right Eq 19: missing a minus sign in the argument of the exp line 315.5 (SI): in the first line of the equation, right-hand side, p(x) is missing line 430: minimzation → minimization line 432: wherein → therein line 444: supercede → supersede  After authors' feedback: Nothing to add - well done.


Dirichlet belief networks for topic structure learning
Recently, considerable research effort has been devoted to developing deep architectures for topic models to learn topic structures. Although several deep models have been proposed to learn better topic proportions of documents, how to leverage the benefits of deep structures for learning word distributions of topics has not yet been rigorously studied. Here we propose a new multi-layer generative process on word distributions of topics, where each layer consists of a set of topics and each topic is drawn from a mixture of the topics of the layer above. As the topics in all layers can be directly interpreted by words, the proposed model is able to discover interpretable topic hierarchies. As a self-contained module, our model can be flexibly adapted to different kinds of topic models to improve their modelling accuracy and interpretability. Extensive experiments on text corpora demonstrate the advantages of the proposed model.

The paper introduces the Dirichlet Belief Net as a general and deeper drop-in replacement for Dirichlet prior distributions for multinomial data. The author(s) derive an elegant data augmented Gibbs sampler to sample the parameters in what they call the Dirichlet Belief Net. They also give three application examples where this prior can replace standard Dirichlet distribution models, such as in Poisson Factor Analysis, MetaLDA, and the Gamma Belief Networks. The work is motivated by the fact that in the three mentioned models, previous work in deep topic modeling has not been focused on deep structures in the word-topic distribution. I’m no expert in the field of Deep Belief Net, so I’m not sure that similar approaches have not been proposed previously. I’m also not sure exactly how big a contribution this is compared to other similar approaches. But I think the idea is both novel, general and the proposed data augmented Gibbs sampling algorithm make the paper an attractive contribution that could be studied and developed further in other model settings. The proposed approach also seems to produce a better perplexity as well as better topic coherence.  Quality:  The technical content of the paper appears to be correct and the experimental result is convincing that this is a promising approach. There may be a couple of minor typos in the Inference section in the paper:: i) At line 115, I suspect that $q^\beta_{\cdot k}$ should be $q^\psi_{\cdot k_1}}$ ii) At line 116, can it be so that $z_{vk_{2}k_{1}}^{(1)}$ should be $\sum_{k_{2}}^{K_{2}}z_{vk_{2}k_{1}}^{(1)}=y_{vk_{1}}^{(1)}$? That would make more sense for me, but I may be wrong.  One issue is the question of using documentation classification as a method for evaluating topic models. The documentation classification does not show any significant results with the new DirBN. I wonder about the argument of including this as an evaluation metric in the paper. Why would we expect a better classification accuracy if we have a better WP part of the model? I think this should be explained and/or argued for in the paper.  Clarity: The paper is, in general, is well written, I think there are some parts that can be improved to make it easier and more clear for the reader. As an example, I find the Related Work being partly overlapping with the introduction. As an example, see paragraph 2 in the introduction and the last paragraph of the related work section.  Figure 1 was initially a little difficult to follow and understand, but after a while, I think it was quite informative. But maybe the author(s) could add an explanation in the caption, helping the reader to understand the Figure easier, maybe by explaining what the three slides are, since they are different things, even though they look similar.  There are parts in Table 1 that are not explained anywhere in the text, such as the different percentage of the different corpora (I guess this is the percentage of the corpus used as training set?). This should be explained in the paper or in the title caption. Also, I guess that the +- indicate the standard deviation of the different estimates. This should also be stated together with how this was computed in the experiment section. This is not possible to follow or reproduce in the current version.  Minor things: i) At line 80, define what k_{t+1} is in a similar way as k is defined previously. ii) There are also errors in the boldfacing of the results in Table 1 (a) I think MetaLDA DirBN3 has better performance than the boldfaced results for WS20% and TM20%.  Originality: Since I’m not an expert in the field of deep probabilistic models, I am not sure if this work has been done before. But the idea of Dirichlet distributed hidden layers seem to be a good idea and the results are promising.  Significance: The experimental results using the DirBN is promising. Including the DirBN in three different models all improve the model perplexity and topic coherence. The only question is how well the approach would work compared with other similar approaches that has been used to do deep modeling of TP.   The straight-forward MCMC sampling scheme using local conjugacy makes the approach promising and straightforward to use in other models, this in turn make the approach an attractive contribution. The approach can easily be incorporated in other models, studied and developed further.  UPDATE AFTER REBUTTAL:  I think most of my issues were explained. But it did not clarify \emph{why} DPN for WP should increase classification accuracy. Why would we expect this to happen? But this is a minor issue.


On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transport
Many tasks in machine learning and signal processing can be solved by minimizing a convex function of a measure. This includes sparse spikes deconvolution or training a neural network with a single hidden layer. For these problems, we study a simple minimization method: the unknown measure is discretized into a mixture of particles and a continuous-time gradient descent is performed on their weights and positions. This is an idealization of the usual way to train neural networks with a large hidden layer. We show that, when initialized correctly and in the many-particle limit, this gradient flow, although non-convex, converges to global minimizers. The proof involves Wasserstein gradient flows, a by-product of optimal transport theory. Numerical experiments show that this asymptotic behavior is already at play for a reasonable number of particles, even in high dimension.

This paper considers the problem of optimizing over measures instead of parameters directly ( as is standard in ML), for differentiable predictors with convex loss. This is an infinite dimensional convex optimization problem. The paper considers instead optimizing with m particles (dirac deltas). As m tends to infinity this corresponds to optimizing over the measure space. Proposition 2.3 shows existence and uniqueness of the particle gradient flow for a given initialization. The main results show convergence of the particle gradient flow to the minimizer under infinite particle limit and under the assumptions of  separability of the support of initialization.   The paper has experiments on toy synthetic problems for sparse deconvolution and training one hidden layer neural nets, evaluating the performance of the finite particle method. The experiments show that for these toy problems, gradient flow with smaller number of particles (~100) converges to a global minimizer. Even though the theoretical results do not apply for ReLU networks, experiments show favorable performance for this class as well.     Comments:   The paper is a theoretical paper studying the optimization over measures using particle gradient flow,  and provides conditions for convergence to global optima. It also has  encouraging experiments on toy examples. The paper currently lacks discussion of limitations of this approach and what is required to scale it to solving reasonable ML tasks.   The results also hold under some fairly strong assumptions on initialization and technical conditions on the functions. It will also be useful to discuss these assumptions a bit more on when do we expect them to hold in practice.   In the introduction, it might be useful to introduce the optimization problem slowly with more details, as it is optimization on \mu and not \theta (as normally done). May be even highlight this difference to make it more clear.  In line 25, the rectified linear unit doesn't fit the framework that requires \phi be differentiable.   Subdifferential as defined in line 135 need not exist for a non-convex function right?. What happens to the subgradient flow then?  Why is the condition on initialization required for proposition 2.5 but not for proposition 2.3?   The description about gap between stationarity and optimality in lines 189-190 is not very clear.    Other comments:    R() is a poor choice of notation for the loss function, as it is confusing with the Real line and is also commonly used for the regularizer part.  There is div() notation used in line 157 and  referred to in appendix without defining it in the main paper, which reduces the readability of the paper.   There has been some empirical work in neural nets that separately optimizes magnitude and direction of units of neural nets (https://arxiv.org/abs/1602.07868). It seems there is some high level similarity between these methods that might be interesting to explore more.   Typos in lines 181, 220


Precision and Recall for Time Series
Classical anomaly detection is principally concerned with point-based anomalies, those anomalies that occur at a single point in time. Yet, many real-world anomalies are range-based, meaning they occur over a period of time. Motivated by this observation, we present a new mathematical model to evaluate the accuracy of time series classification algorithms. Our model expands the well-known Precision and Recall metrics to measure ranges, while simultaneously enabling customization support for domain-specific preferences.

Summary: The authors present a parameterized model that generalizes precision and recall to "range-based" anomaly detection. A range-based anomaly is defined as "an anomaly that occurs over a consecutive sequence of time points, where no non-anomalous data points exist between the beginning and the end of the anomaly". The model is carefully and precisely defined. Then it is evaluated by comparing it against the metric proposed in the Numenta Anomaly Benchmark. The authors conclude that their metric is superior to the NAB metric.   Assessment: I like this paper, because it reflects clear thinking about problem formulation that will be valuable to people working in this area.   Clarity: The paper is well-written. I had a few questions. At line 269, what does "recessing" mean? In general, the paper would be improved if the authors provided better justification for biases beyond the front-end bias. I didn't see how their examples related to mid- and back-end bias.  I can't imagine a case in which one would want a back-end bias. You need to persuade the reader that a problem exists before providing a solution.  I think you have an error at line 22. In the context of anomaly detection, recall is the fraction of true anomalies that were detected and precision is the fraction of detections that were true anomalies. This is how you define things later, but the definitions here seem focused on detecting non-anomalies.   Technical Soundness:  The main claim of the paper is that their model can be used to "evaluate, rank, and compare results of AD algorithms". However, their model does not give a single measure but instead amounts to a domain-specific language for expressing various notions of precision and recall. While any specific instantiation of the model can be applied to evaluate, rank, and compare AD algorithms, it isn't clear how to do this in general. Indeed, is that a realistic goal?  Every anomaly detection application has its unique costs and interaction protocol.  For example, if we consider fraud detection, we are typically interested in the precision @ K: how many of the K top-ranked alarms are true alarms?  This is because our scarce resource is the fraud analyst, and he/she can only examine K candidates per day.  The measure described in this paper could give us a good definition of precision, but I would want to then rank by precision @ K.  I might also want to rank by the financial cost or other features of the anomaly. These are application-specific measures that I don't think can be expressed by the current model (except for the simple measure of the size (duration) of the anomaly).   If we consider equipment diagnosis, the natural measure is precision @ 100% recall (or some high percentile such as 95%). Again, the definitions in this paper would correctly define precision and recall, but I would then compare algorithms based on a measure computed from those.   It is hard to assess the comparison with the NAB scoring model. Are F1, F2, and F0.5 the right measures? They make sense for information retrieval, but in my experience, they rarely make sense for anomaly detection applications.   A stronger paper would have performed a series of case studies where, in each case, the application-specific criterion would have been expressed in the model. I would suggest replacing section 6.4 with this kind of analysis. To me, it is obvious that the metrics could be computed efficiently, but I understand that other readers might want some reassurance on this point. (Minor note: You could express the asymptotic complexity as O(max{N_r,N_p}).)  I thought the experiment reported in Table 2 was very nice.   Originality: I believe the work is original.  Significance: The paper reflects clear thinking about the problem which will be valuable to anyone who works in this area.


Boolean Decision Rules via Column Generation
This paper considers the learning of Boolean rules in either disjunctive normal form (DNF, OR-of-ANDs, equivalent to decision rule sets) or conjunctive normal form (CNF, AND-of-ORs) as an interpretable model for classification. An integer program is formulated to optimally trade classification accuracy for rule simplicity. Column generation (CG) is used to efficiently search over an exponential number of candidate clauses (conjunctions or disjunctions) without the need for heuristic rule mining. This approach also bounds the gap between the selected rule set and the best possible rule set on the training data. To handle large datasets, we propose an approximate CG algorithm using randomization. Compared to three recently proposed alternatives, the CG algorithm dominates the accuracy-simplicity tradeoff in 8 out of 16 datasets. When maximized for accuracy, CG is competitive with rule learners designed for this purpose, sometimes finding significantly simpler solutions that are no less accurate.
see also the survey [30]) of learning rules one by one and removing "covered" examples, a bottomup strategy of combining more specific rules into more general ones [45,22,41], and associative classification in which association rule mining is followed by rule selection using various criteria [38,36,54,50,12,13]. Broadly speaking, these approaches employ heuristics and/or multiple criteria not directly related to classification accuracy. Moreover, they do not explicitly consider model complexity, a problem that has been noted especially with associative classification. Rule set models have been generalized to rule ensembles [17,29,20], using boosting and linear combination rather than logical disjunction; the interpretability of such models is again not comparable to rule sets. Models produced by logical analysis of data [9,31] from the operations research community are similarly weighted linear combinations.
In recent years, spurred by the demand for interpretable models, several researchers have revisited Boolean and rule set models and proposed methods that jointly optimize accuracy and simplicity within a single objective function. These works however have both restricted the problem and approximated its solution. In [33,52,51], frequent rule miners are first used to produce a set of candidate rules. A greedy forward-backward algorithm [33], simulated annealing [52], or integer programming (IP) (in an unpublished manuscript [51]) are then used to select rules from the candidates. The drawback of rule mining is that it limits the search space while often still producing a large number of rules, which then have to be filtered using criteria such as information gain. [51] also presented an IP formulation (but no computational results) that jointly constructs and selects rules without pre-mining. [46] developed an IP formulation for DNF and CNF learning in which the number of clauses (conjunctions or disjunctions) is fixed. The problem is then solved approximately by decomposing into subproblems and applying a linear programming (LP) method [39], which requires rounding of fractional solutions.
In this paper, we also propose an IP formulation for Boolean rule (DNF or CNF) learning but one that avoids the above limitations. Rather than mining rules, we use the large-scale optimization technique of column generation (CG) to intelligently search over the exponential number of all possible clauses, without enumerating even a pre-mined subset (which can be large). Instead, only those clauses that can improve the current solution are generated on the fly. In practice, our approach solves the IP formulation to provable optimality for smaller datasets. For large datasets we employ an approximate version of CG by randomly selecting samples and candidate features that can be used in a clause. To speed up computation, we also generate additional clauses using a greedy algorithm that still optimizes the correct objective.
A numerical evaluation is presented using 16 datasets, including one from the ongoing FICO Explainable Machine Learning Challenge [1]. In terms of the trade-off achieved between accuracy and rule simplicity, our CG algorithm dominates three other recent proposals on 8 datasets, whereas each of the others dominates on at most two. When optimized for accuracy using cross-validation, CG remains competitive with rule learners such as RIPPER [16] that are designed for maximum accuracy. In some instances it provides significantly less complex models with no sacrifice in accuracy.
We note that CG has been proposed for other machine learning tasks such as boosting [21,7] and hash learning [37]. In [21] however, the pricing problem (see Section 2.2) is solved approximately by a weak learning algorithm ("weak" in the boosting sense), not IP, whereas in [7], pricing can be done tractably through enumeration.

The authors propose a mathematical programming approach to build interpretable machine learning models. In this case, the interpretable model is a system of Boolean rules in disjunctive (or conjunctive) normal form which is constructed using column generation for the linear relaxation of a mixed integer program (MIP) designed to minimize the number of positive samples classified incorrectly and the complexity of the learned system subject to complexity constraints. To remedy the fact that there are exponentially many potential clauses to optimize over, the authors propose a standard column generation approach that prices potential columns to add and solves a secondary integer program to find such potential columns. The authors also note that the column generation can also be done via heuristics or a greedy algorithm. Once the linear programming program is solved or reaches its time limit, the approach then solves the global mixed integer formulation to get a final set of rules. The authors perform many experiments comparing their model to a wide range of benchmark models testing improvements in model complexity as well as accuracy. Several benchmark models are built with only one of these two objectives in mind and the proposed model seems to give good tradeoffs in both objectives.  The goal of having an interpretable decision making model is very well motivated. Additionally, the proposed methodology is very clearly explained and the experimental design is nicely laid out. Overall, the column generation mechanism seems to scale better than previous work and the mathematical programming formulation allows easy access to bound the optimal accuracy and complexity. Furthermore, the mathematical programming formulation is very flexible and allows for various loss formulations as well as differing tradeoffs between accuracy and complexity. This methodology is seemingly extensible enough to account for other desired traits of a machine learning model such as fairness constraints. The approach can also be extended to explicitly trade off complexity and accuracy. Finally, the approach seems to do very well against several benchmark in terms of test accuracy and model complexity.   For these reasons, I recommend that this paper be accepted, and hope the authors can improve it by addressing the following questions/suggestions:  - The MIP is missing bounds on the 0 <= \xi_i <= 1 (same for \delta_i in the pricing problem). While you are right that you don’t need \xi_i \in \{0,1\}, if you don’t include lower/upper bounds, then if for a given i in constraint (2), the sum of w_k variables is large, then the optimal solution would make the corresponding \xi_i negative (to make the objective smaller) while maintaining feasibility of constraint (2). I think you probably have the bounds right in the implementation, but not in the text. - Derivation of the pricing problem: since this IP approach may be new to many NIPS readers, it may be better to give a more gentle derivation of the pricing problem, defining reduced costs; just a suggestion.  - Tables 1 and 2 are somewhat difficult to read and would benefit from either bolding the best value in the row or writing the value of the best entry in a final column. It would also be helpful to explain some of the more interesting rows such as the tic-tac-toe dataset having 100 test accuracy with 0.0 standard error for CG while the random forest has less accuracy. This could be resolved by reporting training accuracy or demonstrating that the CG approach enables better generalization due to its lack of complexity.  - It would be helpful to demonstrate the ability for the system to give lower bounds on complexity by reporting these metrics for the different experiments. - Can you elaborate on the FICO example end of page 7? What are these features, and why is this an intuitive rule? Without a few sentences explaining it, the example is kind of useless. - The related work section could include brief references to work that uses column generation for machine learning tasks, examples below.  [1] Demiriz, Ayhan, Kristin P. Bennett, and John Shawe-Taylor. "Linear programming boosting via column generation." Machine Learning 46.1-3 (2002): 225-254. [2] Bi, Jinbo, Tong Zhang, and Kristin P. Bennett. "Column-generation boosting methods for mixture of kernels." Proceedings of the tenth ACM SIGKDD international conference on Knowledge discovery and data mining. ACM, 2004. [3] Li, Xi, et al. "Learning hash functions using column generation." arXiv preprint arXiv:1303.0339 (2013).


Adversarial Text Generation via Feature-Mover's Distance
Generative adversarial networks (GANs) have achieved significant success in generating real-valued data. However, the discrete nature of text hinders the application of GAN to text-generation tasks. Instead of using the standard GAN objective, we propose to improve text-generation GAN via a novel approach inspired by optimal transport. Specifically, we consider matching the latent feature distributions of real and synthetic sentences using a novel metric, termed the featuremover's distance (FMD). This formulation leads to a highly discriminative critic and easy-to-optimize objective, overcoming the mode-collapsing and brittle-training problems in existing methods. Extensive experiments are conducted on a variety of tasks to evaluate the proposed model empirically, including unconditional text generation, style transfer from non-parallel text, and unsupervised cipher cracking. The proposed model yields superior performance, demonstrating wide applicability and effectiveness.

The paper proposes a "feature-mover's distance" as the optimization objective of GAN applied to text. Experiments are conducted for text generation, text style transfer and text decipher tasks.  The proposed idea is a variation on the choice of loss functions compared to previous GAN models for text. However, both the motivation and the experimental results are not significant enough to show an advantage of the proposed loss form.  First of all, the paper mentions that "mode collapsing" and "brittle training" are 2 main problems facing GAN for text. These problems are true, but the paper shows no justification for why the proposed loss form could be helpful for mitigating these 2 problems, neither theoretically nor empirically.  Secondly, since the main contribution is the loss form, the paper should have included fair comparisons with some other major loss forms using the same generator and discriminator network design. This is missing in the paper, and it is very hard to grasp why the particular choice of loss form is made. Besides, a recent paper [1] shows that loss choices of GAN do not matter too much for the generative results (albeit for image generation), which makes is even more necessary to include such comparisons in this paper.  Thirdly, the evaluation results are not statistically significant. The first problem is the choice of evaluation metric (BLEU and self-BLEU), which are not well established scores for text generative models. It is questionable whether any form of BLEU is useful in this case, since it measures whether a phrase exists in a target sentence. Text generative models could simply generate phrases that could make sense, but beyond the measurement of such statistical matching scores because they do not exist as a target sample. One would argue that a large enough dataset would have solved this, but the datasets used in the paper are quite small compared to other available text corpora. Also, the self-BLEU score requires multiple generated texts, which is a hyper-parameter not clearly detailed in the paper.  Furthermore, it is unclear whether the improvement obtained in the paper under such metric is significant enough to demonstrate an advantage of the proposed loss form.  Finally, it is hard for me to perceive a difference from the generated samples in the supplemental material. The authors should perhaps consider including an explanation on why the proposed form generate better results from these samples.  In a summary, due to the lack of proper motivation, comparison and significance in results, I recommend rejection for the paper in its current form. That said, the loss form is an interesting idea. If the authors can develop the idea further and show a clear advantage, it could perhaps be submitted again.  [1] Mario Lucic, Karol Kurach, Marcin Michalski, Sylvain Gelly, Olivier Bousquet. Are GANs Created Equal? A Large-Scale Study. arXiv 1711.10337  ------  After reading the author rebuttal, I realized I missed figure 2 in the first round of reviewing. As a result, the main reason for recommending rejection -- lack of comparison with other GAN losses -- has been significantly weakened. I changed the score to recommend for acceptance now.


DeepProbLog: Neural Probabilistic Logic Programming
We introduce DeepProbLog, a probabilistic logic programming language that incorporates deep learning by means of neural predicates. We show how existing inference and learning techniques can be adapted for the new language. Our experiments demonstrate that DeepProbLog supports (i) both symbolic and subsymbolic representations and inference, (ii) program induction, (iii) probabilistic (logic) programming, and (iv) (deep) learning from examples. To the best of our knowledge, this work is the first to propose a framework where general-purpose neural networks and expressive probabilistic-logical modeling and reasoning are integrated in a way that exploits the full expressiveness and strengths of both worlds and can be trained end-to-end based on examples.

The paper proposes DeepProbLog, a combination of probabilistic logic programming (PLP) and neural networks. In PLP, each atom is a Boolean random variable. The integration is achieved by considering NN that output a probability distribution, such as those that have a final softmax layer. In that case, the output of a network can be associated to a PLP construct such as a probabilistic fact or annotated disjunction and, in turn, to ground facts for so called neural predicates. Inference can then be performed by computing the probability of the atoms for neural predicates by forward evaluation of the network plus knowledge compilation to sentential decision diagrams. Learning is performed by gradient descent using algebraic ProbLog and the gradient semiring. The computation of the gradient is performed by a dynamic programming algorithm that also computes the gradient of the output wrt to the neural network output, so that the gradient can be propagated to the network for its training. DeepProbLog is applied to problems combining logical and perceptual reasoning such as computing the addition of two numbers given as images, inducing programs and probabilistic programming problems. The results of the comparison with CNN for addition and differentiable Forth for program induction shows that it converges faster to a better solution.  The integration of logic and NN is a hot topic nowadays. Among the many proposal, I find this one particularly interesting for its simplicity that makes it a good candidate to become a reference for future systems. The idea of associating the output of NNs to the probability of being true of ground atoms makes the integration direct and clean. Then inference and learning are obtained by propagating probabilities and gradients through the circuit and network. The integration is made possible by the use of probabilistic logic programming that attaches real values to ground atoms.  DeepProbLog is correctly placed in the context of related work. In particular, Rockta¨schel and Riedel (2017) integrates logic and NN by embedding logical symbols and modifying theorem proving. Learning then consists of tuning the embeddings. DeepProbLog differs because logical reasoning is retained with its soundness guarantees. Cohen et al. (2018) is more similar to DeepProbLog but uses a language that does not have a sound probabilistic semantics.  So the paper falls in a very active area but presents an approach that is nevertheless novel and promising. The experiments are particularly interesting because they show the effectiveness of DeepProbLog and because they can form benchmarks for systems aiming to integrate logical and perceptual reasoning.  The paper is overall clear and well written, with a balanced presentation that is easy to follow. I have only a remark on the example in section 4, which contains some imprecisions: 1) in the text you say the program has 6 parameters, yet nodes in Figure 2c have vectors with 5 or 6 elements 2) given that 3 of the 6 parameters are fixed given the other 3, why didn't you use just 3 parameters? 3) what is the meaning of p(0.5) in Figure 2b? 4) You say: "According to the neural network, the first coin is most likely heads (p = 0.9), but is entirely unsure about the  second (p = 0.5)." However, Fig 2c shows p=0.2 for side(coin2,heads)  After reading the rebuttal, I think the authors answered sufficiently well the reviewers' concerns.


Training Deep Neural Networks with 8-bit Floating Point Numbers
The state-of-the-art hardware platforms for training Deep Neural Networks (DNNs) are moving from traditional single precision (32-bit) computations towards 16 bits of precision -in large part due to the high energy efficiency and smaller bit storage associated with using reduced-precision representations. However, unlike inference, training with numbers represented with less than 16 bits has been challenging due to the need to maintain fidelity of the gradient computations during back-propagation. Here we demonstrate, for the first time, the successful training of DNNs using 8-bit floating point numbers while fully maintaining the accuracy on a spectrum of Deep Learning models and datasets. In addition to reducing the data and computation precision to 8 bits, we also successfully reduce the arithmetic precision for additions (used in partial product accumulation and weight updates) from 32 bits to 16 bits through the introduction of a number of key ideas including chunk-based accumulation and floating point stochastic rounding. The use of these novel techniques lays the foundation for a new generation of hardware training platforms with the potential for 2 − 4× improved throughput over today's systems.

This manuscript provides a clear step forward in low precision training, through a mixture of novel ideas and solid experimental results. To summarize the paper: 1. Propose a new data format, or rather, recipe for combining data formats, that allows for training with a majority of the operations performed in an 8-bit floating point format. This includes some novel tricks like chunk-based accumulation 2. Show comprehensive experiments on Cifar10-ResNet and ImageNet1k-AlexNet to validate the method.  3. Demonstrate in silicon that that these methods are useful for optimized deep learning hardware.  Overall it's a solid paper, and there are lots of things I like about it. The authors clearly understand the challenges of low precision training, and are familiar with tricks of the trade such as stochastic rounding and loss scaling. Figure 1 provides a great summary of the 3 major issues that practitioners encounter in low precision training, disentangling accumulation, weight update and storage of weights, activations and gradients. It's commendable that the authors take the space to explain the basics like this even in the 8 page NIPS format. The same can be said about Figure 2, it's a great way to succinctly capture all of the operations and what precision they are in.   There are no major flaws in the manuscript, but a few minor points that could use clarification:   Why was the (1,5,2) format chosen for fp8 and (1,6,9) for fp16? The 5 exponent bits for fp8 might seem natural, since it's the same as IEEE fp16, but did the authors explore e.g. (1,4,3) or (1,6,1) and find them to work less well? If IEEE (1,5,10) does not have enough dynamic range, wouldn't (1,7,8) do even better?   Figure text is too small and unreadable in print or of PDF viewers that don't easily allow zooming in. I realize space is at a premium, but I would consider removing Figure 7 and section 4.4, as it's independent of the rest of the paper, and I am sure there is a separate publication in the works that will going into a great deal more detail on the hardware. Otherwise consider shortening the introduction, or removing some of the detail on Stochastic Rounding. Line 140 onwards provides a useful refresher, but the results in Table 4 and section 4.3 are not novel and could be removed / shortened.   Minor comments - I was not familiar with the term "Swamping", is there a reference for where this expression comes from? - 161 seems to have a stray link - 192, does ResNet18, 50 refer to the i1k sized network, or CIFAR? Please clarify. - Table 2, what does the column "FP32" mean? Is this the accuracy reported by various authors for training the same fp32 model? Should they not all be the same? - Figure 4, ResNet50 test error is around 29%, but should be closer to 24% based on He 2016. Benchmarks such as DawnBench and MLPerf generally require 25% or better on ResNet50. Can you describe differences between this ResNet and the original one and explain the discrepancy? - 253, typo, dominated -> dominant    ——-  Update after rebuttal: I don’t have much to add in response to the rebuttal, but would like to urge the authors to incorporate the suggestions from Reviewer 1, which I found very valuable.


Generalizing graph matching beyond quadratic assignment model
Graph matching has received persistent attention over several decades, which can be formulated as a quadratic assignment problem (QAP). We show that a large family of functions, which we define as Separable Functions, can approximate discrete graph matching in the continuous domain asymptotically by varying the approximation controlling parameters. We also study the properties of global optimality and devise convex/concave-preserving extensions to the widely used Lawler's QAP form. Our theoretical findings show the potential for deriving new algorithms and techniques for graph matching. We deliver solvers based on two specific instances of Separable Functions, and the state-of-the-art performance of our method is verified on popular benchmarks.

Summary:  This paper shows that a large family of functions, defined as Separable Functions, can asymptotically approximate the discrete matching problem by varying the approximation controlling parameter.  They present Theorem 1 that The optimal objective p^* to problem (2) is equal to the optimal objective p_delta^* to problem (4), in which problem (2) is original graph matching problem in integer domain and problem (4) is the problem relaxed to continuous domain. They define Separable Function (SF), f_theta(x, y), that satisfies there properties mentioned in Definition 1 in Section 4.1, and theta in univariate SF h_theta is defined as controlling parameter. The SF has three fine properties for computation: 1) it shows similar behavior as a probabilistic distribution on two independent variables; 2) it eases gradient computing; 3) we can construct new approximation function via reweighted summation and multiplication of existing Separable Functions. They show that original problem (4) can be asymptotically approximated using SFs as theta -> 0. They analyze the properties of convexity/concavity under such approximations, as there is technique to convert non-convex/non-concave problems into convex/concave ones.  They describe two optimization strategies for generalized graph matching (GGM): Path Following Strategy and Multiplication Strategy. The first one starts by obtaining a local optimum from a relatively tractable problem , then we shrink the value of theta by a factor of two. For the second one, each step amounts to a multiplication x(t+1) = Ax(t) and the objective score over the solution path is non-decreasing, under the assumption that A is positive semi-definite. It is found the multiplicative method converges much faster and hence the overall run time is less compared with the path following method.  They perform experiments on two popular random graph matching datasets, CMU house sequence and Caltech-101/MSRC object matching, with comparison to Spectral Matching (SM), Integer Projected Fixed Point (IPFP), Graduated Assignment (GAGM), Reweighted Random Walk (RRWM), Soft-restricted Graduated Assignment (SRGA), Factorized Graph Matching (FGM), and Branching Path Following Matching (BPM). The experiment shows their GMM methods reach similar performance or outperform other state-of-the-art methods.   Strengths:  - This paper is technically sound with experiment results to support the effectiveness of their approach. - This paper provides novelty algorithms for GMM problems with strong theoretical analysis. - The theoretical analysis and experiments in this  paper is written well and clear.  Weaknesses:  - It may be better to provide asymptotic analysis of running time for both proposed GMM optimization algorithms.  Rebuttal Response:  After reading the rebuttal, the author addresses some comments on computation complexity, convergence criterion, parameter sensitivity, and others. I support accepting the paper, but I will not change my score.


e-SNLI: Natural Language Inference with Natural Language Explanations
In order for machine learning to garner widespread public adoption, models must be able to provide interpretable and robust explanations for their decisions, as well as learn from human-provided explanations at train time. In this work, we extend the Stanford Natural Language Inference dataset with an additional layer of human-annotated natural language explanations of the entailment relations. We further implement models that incorporate these explanations into their training process and output them at test time. We show how our corpus of explanations, which we call e-SNLI, can be used for various goals, such as obtaining full sentence justifications of a model's decisions, improving universal sentence representations and transferring to out-of-domain NLI datasets. Our dataset 1 thus opens up a range of research directions for using natural language explanations, both for improving models and for asserting their trust.

Update after Author Feedback:  Thanks for all of the clarification, especially with regard to the BLEU scores. I think the idea of explicable models is worth pursuing, and this is a decent contribution to showing how one might do that. It is unfortunate that this work shows a huge tradeoff between models that perform at high levels and those that explain well (from 4.1 it seems like we can get good performance, but then can't generate correct explanations very often and from 4.2 we can generate correct explanations more often at the expense of good performance). It also seems disappointing that the BLEU scores in the PREDICT setting are already so close to the inter-annotator agreement even though they are not correct explanations very often; this seems to suggest that we really do need to rely on the percent correct given by human evaluation and that the BLEU scores are not very meaningful. This seems like a bottleneck for this resource being widely adopted. Nonetheless, these findings are a solid contribution and so is the data if others are willing to do human evaluation or work on a new automatic metric for a task like this.  Original Review:  This work augments the SNLI corpus with explanations for each label. They then run several different kinds of experiments to show what can be done with such an augmented datasets. First, they show that it is possible to train a model to predict the label and then generate an explanation based on that label. Second, they show that given a ground truth explanation, the labels can be easily predicted. Third, they show that the model that both predicts and explains (e-SNLI) can provide useful sentence representations. Fourth, they try to transfer e-SNLI to MultiNLI. These seem like the perfect experiments for a very interesting attempt to make deep learning models more interpretable. As the authors mentioned, this line of work has some history with Park et al, Ling et al, and Jansen et al, but I have not seen anything quite like this, especially for a task like SNLI, which does hold potential for transfer learning. It is an especially important task to learn how to interpret if we want to have a better sense of how our models are thinking of logical inference.  Subsection 4.1  I’m confused/concerned by the evaluation of the generated explanations and the resulting low BLEU scores. First of all, why use only BLEU-4? Why not use the same BLUE that is typically used for tasks like neural machine translation? This would make the numbers more interpretable to other NLP researchers. Second, is it really .2 in the same scale that NMT researchers would report 30+ BLEU scores for NMT. The obvious answer is no because it is just BLEU-4, but how do I relate those? .2 BLEU on translation would give complete nonsense, but the generated explanations in Table 2 look quite good. How do I make sense of this as a reader? Does this mean the explanations are coherent but mostly ungrounded in the prediction? If the inter-annotator agreement is truly so low, then how can we ever really evaluate these systems? Will we always have to rely on human judgment in comparing future systems to the one you’ve proposed?  I’m also worried by these numbers: if 99% of labels are consistent with their explanations, why is there such a wide gap between the 84% correct labels and the 35.7% correct explanations? Does this mean that the explanations are usually not explaining at all? They just happen to give the same label if you were to infer from them alone? Based on the discussion in the next section, I’m assuming the ‘them’ in ’35.7% of them’ refer to the 84 out of 100 examples correctly predicted rather than out of all 100 qualitatively examined examples, but that was ambiguous until I read 4.2.  How exactly were partial points awarded to get the 35.7% correct explanations?  You mention it is done in the footnote, but I’d like more detail there if there is going to be work that builds off this in the future. Maybe it would just be less ambiguous to not award these partial points.  Subsection 4.2  The 67.75% is quite low in this section. Using the hypothesis alone can give you a score right around there (Gururangan et al. 2018); that should probably be mentioned.  I like this conclusion here. The GENERATE model can just generate explanations after the fact that don’t have any real incentive to be correct. Perhaps with a higher alpha value (Eq. 1), this would change, but we don’t have those experiments to verify.  4.3 and 4.4 appear to show some empirical validation of the method for transfer learning and universal embeddings.   I think there might be a typo in Table 5’s std deviation for InferSent on MultiNLI.   line 203: “a model on model” seems like a typo  I really like this idea, the experiments, and the example generated explanations, but I have major concerns about evaluation for this kind of task. Even though the language for the generated explanations is surprisingly coherent, it seems like in most cases, the explanations are not actually aligning with the predictions in a meaningful way. What’s worse is that the only way to find out is to manually inspect. This doesn’t seem like a sustainable way to compare which models in the future are better at the explanation part of e-SNLI, which is really the crux of the paper. In short, I’m left feeling like the evaluation setup just isn’t quite there yet for the community to be able to use these ideas. This is a hard problem. I don’t have great ideas for how to fix the evaluation, and I’m definitely looking forward to hearing what the authors have to say about this, especially when it comes to the BLEU evaluation and how I might better interpret those results. For now, the overall is a 5 with a confidence of 4.


Query K-means Clustering and the Double Dixie Cup Problem
We consider the problem of approximate K-means clustering with outliers and side information provided by same-cluster queries and possibly noisy answers. Our solution shows that, under some mild assumptions on the smallest cluster size, one can obtain an (1 + )-approximation for the optimal potential with probability at least 1 − δ, where > 0 and δ ∈ (0, 1), using an expected number of O(
noiseless same-cluster queries and comparison-based clustering of complexity O(ndK + K 3 δ ); here, n denotes the number of points and d the dimension of space. Compared to a handful of other known approaches that perform importance sampling to account for small cluster sizes, the proposed query technique reduces the number of queries by a factor of roughly O(
3 ), at the cost of possibly missing very small clusters. We extend this settings to the case where some queries to the oracle produce erroneous information, and where certain points, termed outliers, do not belong to any clusters. Our proof techniques differ from previous methods used for K-means clustering analysis, as they rely on estimating the sizes of the clusters and the number of points needed for accurate centroid estimation and subsequent nontrivial generalizations of the double Dixie cup problem. We illustrate the performance of the proposed algorithm both on synthetic and real datasets, including MNIST and CIFAR 10.

This paper investigates the problem of active-semi-supervised clustering, by considering both noiseless (perfect oracle) and noisy (imperfect oracle) query responses. The authors provide probabilistic guarantees for low approximation errors to the true optimal k-means objective. The corresponding query complexities are substantially lower than in the existing literature. Importantly, as noted by the authors, their query complexity is independent of the size of the dataset.  The main strength of the paper lies in the considerable technical rigour with which the subject has been handled. Numerous results are presented which are either very novel, or, where similar to existing results, provide substantial improvements in terms of the resulting complexities. The experiments while not extensive, are adequate to show considerable improvement over the apparently only available relevant pre-existing method. The paper discusses very technical material, but the authors have done well to elucidate the important points within the text.  My major concern lies in the practical relevance of the theory. If I understand correctly the assumption is that the oracle has perfect knowledge of the optimal k-means solution, and not of the true solution. However, this seems quite implausible in most situations. Indeed, as far as I can tell in the experiments the queries are answered based on whether or not the points are in the same true cluster, and not the optimal k-means assignment. Certainly this seems to be a much more realistic implementation, especially in the crowdsourcing scenario mentioned by the authors. That said, the performance on the real datasets is clearly good. It still does not take much of a stretch to imagine very simple examples where the true clusters' means are markedly different from the k-means solutions'.  Given the relative infancy of this area, it would be interesting to compare with existing (non active) semi-supervised clustering or semi-supervised classification algorithms (the latter both active and non active). For example, how many true labels are needed for semi-supervised classification to do as well as your approach? This comparison seems relevant if weighing up the relative cost of label queries against pairwise assignment queries.  I recorded only one significant typo: in the description of Algorithm 2 there is repetition of "without replacement".  ########################################  In the authors' response the justification of the assumptions underlying their theoretical contributions were revisited. Of course I completely recognise that it is necessary to have a well defined mathematical or probabilistic description of the target solution. I also recognise that in many real clustering problems the true cluster labels may not be consistent with any intuitive mathematical formulation of a clustering objective. I would still argue that the assumption of the availability of the optimal k-means solution to an oracle (or crowd) is very unrealistic. I would find it far more satisfying if (even relatively strong) assumptions on the true clusters were made, e.g., that the data arise from a GMM with perhaps an upper bound on the overlap/misclassification rate w.r.t. the true model. Even if this is equally unrealistic in practice, it at least does not create a disconnect between the assumption (that optimal k-means solution is known) and implementation (queries are answered w.r.t. the true clusters and not w.r.t the clusters in the optimal kmeans solution). That said, I am satisfied that this issue is more relevant to the entire literature on this problem and not the paper specifically. Also that there is more than sufficient precedent for these assumptions in high quality publications.  I also accept that given the page limit it is not possible to include results comparing active-semi-supervised clustering with semi-supervised classification. The authors' applications in bioinformatics sound extremely interesting and relevant given the importance of same cluster queries being more realistic than label queries. I would find it interesting as a reader to have an example described in the paper where label queries are not relevant. This does not necessitate inclusion of experiments on such a problem, just a description perhaps in the introduction, motivating the relevance of the problem the authors are studying.  Given the above, and reading the other reviewers' comments, I more strongly recommend acceptance.


Discovery of Latent 3D Keypoints via End-to-end Geometric Reasoning
This paper presents KeypointNet, an end-to-end geometric reasoning framework to learn an optimal set of category-specific 3D keypoints, along with their detectors. Given a single image, KeypointNet extracts 3D keypoints that are optimized for a downstream task. We demonstrate this framework on 3D pose estimation by proposing a differentiable objective that seeks the optimal set of keypoints for recovering the relative pose between two views of an object. Our model discovers geometrically and semantically consistent keypoints across viewing angles and instances of an object category. Importantly, we find that our end-to-end framework using no ground-truth keypoint annotations outperforms a fully supervised baseline using the same neural network architecture on the task of pose estimation. The discovered 3D keypoints on the car, chair, and plane categories of ShapeNet [6]  are visualized at keypointnet.github.io.

The paper presents a self-supervised approach for detecting 3D key-points using end-to-end geometric reasoning.  Given two views of the same object with a known rigid transformation, the network is indirectly trained for keypoint localization by seeking key-points that best relate both views. An objective function is proposed that incorporates multiple domain-specific geometric cues to train the network. Multi-view consistency between the two sets of points under ground-truth transformation is used to enforce that the estimated key-points from both views can be transformed to each other via the known rigid transformation. To avoid degenerated solutions where all key-points collapse to the same location, two additional loss functions are introduced. The first loss measures the misfit between the ground-truth relative rotation and the estimated relative rotation using estimated key-points, while the second loss discourages proximity of the key-points in 3D. Finally, a silhouette consistency loss is introduced to ensure that the 2D projections of 3D key-joints lie inside the object silhouettes. Experiments are performed on the ShapeNet dataset, where the authors have obtained the annotations for three object categories via Mechanical Turk. Experimental evaluation demonstrates that the proposed self-supervised approach outperform supervised counterpart.   Pros: + The paper is well-written and easy to read. + The idea of discovering 3D key-point is very nice, though not completely novel (see below) + I like the idea of relative pose estimation as a supervisory signal.  + The qualitative results look quite good.  + The proposed approach can be extended to real applications such human pose estimation and hand pose estimation. The used supervisory signals for human pose estimation are also available, for example, in HUMAN3.6M dataset or for hand pose estimation in CMU's Panoptic Studio.  Cons: - The idea of self-supervised 3D keypoint localization is not entirely novel and have been addressed in the literature. For example in [a].  - I would have liked to see quantitative ablative studies in the paper, that how much each loss contributes to the final performance. At the moment this is only provided in the supp. material without any quantification.  - I would have liked to see some results on realistic data such as for HUMAN3.6M or some other application where same supervisory signals can be used.    Summary: I think it is a good paper with sufficient novel contributions to be accepted to NIPS. The proposed method can be extended to many other applications.   Some references that should be added:  [a] Tung et al., Self-supervised Learning of Motion Capture, NIPS'17 [b] Rohdin et al., Unsupervised Geometry-Aware Representation for 3D Human Pose Estimation, CVPR'18 [c] Molchanov et al., Improving Landmark Localization with Semi-Supervised Learning, CVPR'18 [d] Zhang et al., Unsupervised Discovery of Object Landmarks as Structural Representations, CVPR'18  or other recent CVPR'18 papers that I might have missed.    The rebuttal addressed most of my concerns. It still misses experiments on a real application, but I agree with the authors that it is an interesting future direction.


Dirichlet-based Gaussian Processes for Large-scale Calibrated Classification
This paper studies the problem of deriving fast and accurate classification algorithms with uncertainty quantification. Gaussian process classification provides a principled approach, but the corresponding computational burden is hardly sustainable in large-scale problems and devising efficient alternatives is a challenge. In this work, we investigate if and how Gaussian process regression directly applied to classification labels can be used to tackle this question. While in this case training is remarkably faster, predictions need to be calibrated for classification and uncertainty estimation. To this aim, we propose a novel regression approach where the labels are obtained through the interpretation of classification labels as the coefficients of a degenerate Dirichlet distribution. Extensive experimental results show that the proposed approach provides essentially the same accuracy and uncertainty quantification as Gaussian process classification while requiring only a fraction of computational resources.

The authors of this paper introduce a novel approach to GP classification, called GPD. The authors use a GP to produce the parameters of a Dirichlet distribution, and use a categorical likelihood for multi-class classification problems. After applying a log-normal approximation to the Dirichlet distribution, inference for GPD is the same as exact-GP inference (i.e. does not require EP, Laplace approximation, etc.) The authors show that GPD has competitive accuracy, is well calibrated, and offers a speedup over existing GP-classificaiton methods.  Quality:  The method introduced by this paper is a clever probabilistic formulation of Bayesian classification. The method is well-grounded in Bayesian statistics and makes good comparisons to similar existing approaches. The one issue I have with this paper are the claims about the ``speedup'' afforded by this method (see detailed comments). The authors are not specific about the speedup (is it training time or inference time?) and I'm concerned they are missing an important baseline.  Clarity:  The paper is clearly written.  Originality:  To the best of my knowledge this is a novel approach to Bayesian classification.  Significance:  The idea is an interesting new approach to approximate inference for classification, though it does not perform better than other classification methods (in terms of accuracy). Nevertheless, it will be of interest to the Bayesian ML community.  Overall:  The proposed method is a novel approach to classification - and based on these merits I would be willing to accept the paper. However, the authors need to substantiate the speedup results with more detail and analysis if they wish to report this as a primary advantage.  Detailed comments:  My main concern with this paper is with regards to the ``speedup'' that the authors claim. There are many missing details.  The authors claim that their GPD method will be faster than GP classification because "GPC requires carrying out several matrix factorizations." (What matrix factorizations are the authors referring to?) The inducing-point GP-classification method in the results (SVGP) is asymptotically the same as the inducing-point method used for GPD (SGPR). The NMLL computations (used for training) and inference computations for SVGP and SGPR both require the Cholesky decomposition of an m x m matrix (where m is the number of inducing points). This is the primary computational bottleneck of both SVGP (for GP classification) and SGPR (for GPD). I'm assuming the authors claim that GPD is faster because GPD does not require learning variational parameters. The ``training'' of GPD - as far as I understand - only requires optimizing the hyperparameters, which presumably requires fewer optimization iterations. However, one key advantage of SVGP (for GP classification) is that it can be optimized with SGD. So even though SVGP requires more optimization iterations, each iteration will be fast since it only requires a minibatch of data. On the other hand, every optimization iteration of GPD will require the entire dataset. Therefore, there is not an obvious reason why GPD (and GP regression) should be that much faster than GP classification. The authors should elaborate with more detail in Sections 2 and 3 as to why GP-classification is slow.  While the authors do report some empirical speedups, the experiments are missing several important details. Firstly, what is the ``speedup'' measuring - training or inference? (I'm assuming that it is measuring training, but this must be clarified.) Secondly, was the author's GP-classification baseline trained with or without SGD? If it was trained without SGD, I would argue that the speed comparisons are unfair since stochastic optimization is key to making GP-classification with SVGP a fast method.  Small comments:  - 149: "For a classifier to be well-calibrated, it should accurately approximate fp." - This is not necessarily true. Assuming a balanced dataset without loss of generality, a latent function that outputs zero (in the case of CE) or 0.5 (in the case of least-squares classification) for all samples is well calibrated. - Figure 1: it is difficult to differentiate between the lines in the n=60 and n=80 plots - 241: do you mean the initial inducing points were chosen by k-means? SVGP and SGPR should optimize the inducing point locations.   ----------------------- Post author response:  While I would still vote to accept the paper, I will note that the author's response did not clarify my questions about the speedup. (Was it training or inference?) Additionally, the author's claim about speed still does not acknowledge many recent advances in scalable GP classification:  "Our claim in the paper about multiple matrix factorizations pertains to training times and it is specific to the case of standard implementations of EP and the Laplace approximation."  Though the authors make a statement about GPC speed based on the EP/Laplace approximations, yet they do not compare to these methods in their experiments! Regarding SGD-based GPC methods: there have been a number of recent works in scalable GPC methods that rely on SGD [1-3]. These papers report large empirical speedups when using SGD-based optimization over batch GD optimization. By not comparing against SGD-based methods, the authors are disregarding GPC advances from the past 4 years.  That being said, I still do believe that this method is novel, and I would vote for acceptance. I would encourage the authors to be much more rigorous about their speedup claims. There are some scenarios where a non-GPC-based approach would be clearly advantageous (e.g. in an active learning setting, when the kernel hyperparameters can be reused between data acquisition), and I would suggest the authors to focus on these specific scenarios rather than making broad claims about speed.  [1] Hensman, James, Alexander Matthews, and Zoubin Ghahramani. "Scalable Variational Gaussian Process Classification." Artificial Intelligence and Statistics. 2015. [2] Hernández-Lobato, Daniel, and José Miguel Hernández-Lobato. "Scalable Gaussian process classification via expectation propagation." Artificial Intelligence and Statistics. 2016. [3] Wilson, Andrew G., et al. "Stochastic variational deep kernel learning." Advances in Neural Information Processing Systems. 2016.