Generative models such as Variational Auto Encoders (VAEs) and Generative Adversarial Networks (GANs) are typically trained for a fixed prior distribution in the latent space, such as uniform or Gaussian. After a trained model is obtained, one can sample the Generator in various forms for exploration and understanding, such as interpolating between two samples, sampling in the vicinity of a sample or exploring differences between a pair of samples applied to a third sample. In this paper, we show that the latent space operations used in the literature so far induce a distribution mismatch between the resulting outputs and the prior distribution the model was trained on. To address this, we propose to use distribution matching transport maps to ensure that such latent space operations preserve the prior distribution, while minimally modifying the original operation. Our experimental results validate that the proposed operations give higher quality samples compared to the original operations.

The authors demonstrate experimentally a problem with the way common latent space operations such as linear interpolation are performed for GANs and VAEs. They propose a solution based on matching distributions using optimal transport. Quite heavy machinery to solve a fairly simple problem, but their approach is practical and effective experimentally (though the gain over the simple SLERP heuristic is often marginal). The problem they describe (and so the solution) deserves to be more widely known.
Major comments:
The paper is quite verbose, probably unnecessarily so. Firstly, the authors devote over 2 pages to examples that distribution mismatches can arise in synthetic cases (section 2). This point is well made by a single example (e.g. section 2.2) and the interesting part is that this is also an issue in practice (experimental section). Secondly, the authors spend a lot of space on the precise derivation of the optimal transport map for the uniform distribution. The fact that the optimal transport computation decomposes across dimensions for pointwise operations is very relevant, and the matching of CDFs, but I think a lot of the mathematical detail could be relegated to an appendix, especially the detailed derivation of the particular CDFs.
Minor comments:
It seems worth highlighting that in practice, for the common case of a Gaussian, the proposed method for linear interpolation is just a very simple procedure that might be called "projected linear interpolation", where the generated vector is multiplied by a constant. All the optimal transport theory is nice, but it's helpful to know that this is simple to apply in practice.
Might I suggest a very simple approach to fixing the distribution mismatch issue? Train with a spherical uniform prior. When interpolating, project the linear interpolation back to the sphere. This matches distribution, and has the attractive property that the entire geodesic between two points lies in a region with typical probability density. This would also work for vicinity sampling.
In section 1, overfitting concerns seem like a strange way to motivate the desire for smoothness. Overfitting is relatively easy to compensate for, and investigating the latent space is interesting regardless.
When discussing sampling from VAEs as opposed to GANs, it would be good to mention that one has to sample from p(x | z) not just p(z).
Lots of math typos such as t - 1 should be 1 - t in (2), "V times a times r" instead of "Var" in (3) and "s times i times n" instead of "sin", etc, sqrt(1) * 2 instead of sqrt(12), inconsistent bolding of vectors. Also strange use of blackboard bold Z to mean a vector of random variables instead of the integers.
Could cite an existing source for the fact that most mass for a Gaussian is concentrated on a thin shell (section 2.2), e.g. David MacKay Information Theory, Inference and Learning Algorithms.
At the end of section 2.4, a plot of the final 1D-to-1D optimal transport function (for a few different values of t) for the uniform case would be incredibly helpful.
Section 3 should be a subsection of section 2.
For both SLERP and the proposed method, there's quite a sudden change around the midpoint of the interpolation in Figure 2. It would be interesting to plot more points around the midpoint to see the transition in more detail. (A small inkling that samples from the proposed approach might change fastest qualitatively near the midpoint of the interpolation perhaps maybe be seen in Figure 1, since the angle is changing fastest there??)


Published as a conference paper at ICLR 2018 STABILIZING ADVERSARIAL NETS WITH PREDICTION METHODS
Adversarial neural networks solve many important problems in data science, but are notoriously difficult to train. These difficulties come from the fact that optimal weights for adversarial nets correspond to saddle points, and not minimizers, of the loss function. The alternating stochastic gradient methods typically used for such problems do not reliably converge to saddle points, and when convergence does happen it is often highly sensitive to learning rates. We propose a simple modification of stochastic gradient descent that stabilizes adversarial networks. We show, both in theory and practice, that the proposed method reliably converges to saddle points, and is stable with a wider range of training parameters than a non-prediction method. This makes adversarial networks less likely to "collapse," and enables faster training with larger learning rates.

NOTE:
I'm very willing to change my recommendation if I turn out to be wrong
about the issues I'm addressing and if certain parts of the experiments are fixed.
Having said that, I do (think I) have some serious issues:
both with the experimental evaluation and with the theoretical results.
I'm pretty sure about the experimental evaluation and less sure about the theoretical results.
THEORETICAL CLAIMS:
These are the complaints I'm not as sure about:
Theorem 1 assumes that L is convex/concave.
This is not generally true for GANs.
That's fine and it doesn't necessarily make the statement useless, but:
If we are willing to assume that L is convex/concave,
then there already exist other algorithms that will provably converge
to a saddle point (I think). [1] contains an explanation of this.
Given that there are other algorithms with the same theoretical guarantees,
and that those algorithms don't magically make GANs work better,
I am much less convinced about the value of your theorem.
In [0] they show that GANs trained with simultaneous gradient descent are locally asymptotically stable,
even when L is not convex/concave.
This seems like it makes your result a lot less interesting, though perhaps I'm wrong to think this?
Finally, I'm not totally sure you can show that simultaneous gradient descent won't converge
as well under the assumptions you made.
If you actually can't show that, then the therom *is* useless,
but it's also the thing I've said that I'm the least sure about.
EXPERIMENTAL EVALUATION:
Regarding the claims of being able to train with a higher learning rate:
I would consider this a useful contribution if it were shown that (by some measure of GAN 'goodness')
a high goodness was achieved faster because a higher learning rate was used.
Your experiments don't support this claim presently, because you evaluate all the models at the same step.
In fact, it seems like both evaluated Stacked GAN models get worse performance with the higher learning rate.
This calls into question the usefulness of training with a higher learning rate.
The performance is not a huge amount worse though (based on my understanding of Inception Scores),
so if it turns out that you could get that performance
in 1/10th the time then that wouldn't be so bad.
Regarding the experiment with Stacked GANs, the scores you report are lower than what they report [2].
Their reported mean score for joint training is 8.59.
Are the baseline scores you report from an independent reproduction?
Also, the model they have trained uses label information.
Does your model use label information?
Given that your reported improvements are small, it would be nice to know what the proposed mechanism is by
which the score is improved.
With a score of 7.9 and a standard deviation of 0.08, presumably none of the baseline model runs
had 'stability issues', so it doesn't seem like 'more stable training' can be the answer.
Finally, papers making claims about fixing GAN stability should support those claims by solving problems
with GANs that people previously had a hard time solving (due to instability).
I don't believe this is true of CIFAR10 (especially if you're using the class information).
See [3] for an example of a paper that does this by generating 128x128 Imagenet samples with a single generator.
I didn't pay as much attention to the non-GAN experiments because
a) I don't have as much context for evaluating them, because they are a bit non-standard.
b) I had a lot of issues with the GAN experiments already and I don't think the paper should be accepted unless those are addressed.
[0] https://arxiv.org/abs/1706.04156 (Gradient Descent GAN Optimization is Locally Stable)
[1] https://arxiv.org/pdf/1705.07215.pdf (On Convergence and Stability of GANs)
[2] https://arxiv.org/abs/1612.04357 (Stacked GAN)
[3] https://openreview.net/forum?id=B1QRgziT (Spectral Regularization for GANs)
EDIT:
As discussed below, I have slightly raised my score.
I would raise it more if more of my suggestions were implemented (although I'm aware that the authors don't have much (any?) time for this - and that I am partially to blame for that, since I didn't respond that quickly).
I have also slightly raised my confidence.
This is because now I've had more time to think about the paper, and because the authors didn't really address a lot of my criticisms (which to me seems like evidence that some of my criticisms were correct).


Regularization is a big issue for training deep neural networks. In this paper, we propose a new information-theory-based regularization scheme named SHADE for SHAnnon DEcay. The originality of the approach is to define a prior based on conditional entropy, which explicitly decouples the learning of invariant representations in the regularizer and the learning of correlations between inputs and labels in the data fitting term. We explain why this quantity makes our model able to achieve invariance with respect to input variations. We empirically validate the efficiency of our approach to improve classification performances compared to standard regularization schemes on several standard architectures.

Summary:
The paper presents an information theoretic regularizer for deep learning
algorithms. The regularizer aims to enforce compression of the learned
representation while conditioning upon the class label so preventing the
learned code from being constant across classes. The presentation of the Z
latent variable used to simplify the calculation of the entropy H(Y|C) is
confusing and needs revision, but otherwise the paper is interesting.
Major Comments:
- The statement that I(X;Y) = I(C;Y) + H(Y|C) relies upon several properties
  of Y which are not apparent in the text (namely that Y is a function of X,
so I(X;Y) should be maximal, and Y is a smaller code space than X so it should
be H(Y)). If Y is a larger code space than X then it should still be true, but
the logic is more complicated.
- The latent code for Z is unclear. Given the use of ReLUs it seems like Y
  will be O or +ve, and Z will be 0 when Y is 0 and 1 otherwise, so I'm
unclear as to when the value H(Y|Z) will be non-zero. The data is then
partitioned within a batch based on this Z value, and monte carlo sampling is
used to estimate the variance of Y conditioned on Z, but it's really unclear
as to how this behaves as a regularizer, how the z is sampled for each monte
carlo run, and how this influences the gradient. The discussion in Appendix C
doesn't mention how the Z values are generated.
- The discussion on how this method differs from the information bottleneck is
  odd, as the bottleneck is usually minimising the encoding mutual information
I(X;Y) minus the decoding mutual information I(Y;C). So directly minimising
H(Y|C) is similar to the IB, and also minimising H(Y|C) will affect I(C;Y) as
I(C;Y) = H(Y) - H(Y|C).
- The fine tuning experiments (Section 4.2) contain no details on the
  parameters of that tuning (e.g. gradient optimiser, number of epochs,
batch size, learning rates etc).
- Section 4.4 is obvious, and I'd consider it a bug if regularising with label
  information performed worse than regularising without label information.
Essentially it's still adding supervision after you've removed the
classification loss, so it's natural that it would perform better. This
experiment could be moved to the appendix without hurting the paper.
- In appendix A an upper bound is given for the reconstruction error in terms
  of the conditional entropy. This bound should be related to one of the many
upper bounds (e.g. Hellman & Raviv) for the Bayes rate of a predictor, as
there is a fairly wide literature in this area.
Minor Comments:
- The authors do not state what kind of input variations they are trying to
  make the model invariant to, and as it applies to CNNs there are multiple
different kinds, many of which are not amenable to a regularization based
system for inducing invariance.
- The authors should remind the reader once that I(X;Y) = H(Y) - H(Y|X) = H(X) -
  H(X|Y), as this fact is used multiple times throughout the paper, and it may
not necessarily be known by readers in the deep learning community.
- Computing H(Y|C) does not necessarily require computing c separate
  entropies, there are multiple different approaches for computing this
entropy.
- The exposition in section 3 could be improved by saying that H(X|Y) measures
  how much the representation compresses the input, with high values meaning
large amounts of compression, as much of X is thrown away when generating Y.
- The figures are difficult to read when printed in grayscale, the graphs
  should be made more readable when printed this way (e.g. different symbols,
dashed lines etc).
- There are several typos (e.g. pg 5 "staking" -> "stacking").


MULTI-VIEW DATA GENERATION WITHOUT VIEW SUPERVISION
The development of high-dimensional generative models has recently gained a great surge of interest with the introduction of variational auto-encoders and generative adversarial neural networks. Different variants have been proposed where the underlying latent space is structured, for example, based on attributes describing the data to generate. We focus on a particular problem where one aims at generating samples corresponding to a number of objects under various views. We assume that the distribution of the data is driven by two independent latent factors: the content, which represents the intrinsic features of an object, and the view, which stands for the settings of a particular observation of that object. Therefore, we propose a generative model and a conditional variant built on such a disentangled latent space. This approach allows us to generate realistic samples corresponding to various objects in a high variety of views. Unlike many multiview approaches, our model doesn't need any supervision on the views but only on the content. Compared to other conditional generation approaches that are mostly based on binary or categorical attributes, we make no such assumption about the factors of variations. Our model can be used on problems with a huge, potentially infinite, number of categories. We experiment it on four image datasets on which we demonstrate the effectiveness of the model and its ability to generalize.

The paper proposes a GAN-based method for image generation that attempts to separate latent variables describing fixed "content" of objects from latent variables describing properties of "view" (all dynamic properties such as lighting, viewpoint, accessories, etc). The model is further extended for conditional generation and demonstrated on a range of image benchmark data sets.
The core idea is to train the model on pairs of images corresponding to the same content but varying in views, using adversarial training to discriminate such examples from generated pairs. This is a reasonable procedure and it seems to work well, but also conceptually quite straightforward -- this is quite likely how most people working in the field would solve this problem, standard GAN techniques are used for training the generator and discriminator, and the network architecture is directly borrowed from Radford et al. (2015) and not even explained at all in the paper. The conditional variant is less obvious, requiring two kinds of negative images, and again the proposed approach seems technically sound.
Given the simplicity of the algorithmic choices, the potential novelty of the paper lies more in the problem formulation itself, which considers the question of separating two sets of latent variables from each other in setups where one of them (the "view") can vary from pair to pair in arbitrary manner and no attributes characterising the view are provided. This is an interesting problem setup, but not novel as such and unfortunately the paper does not do a very good job in putting it into the right context. The work is contrasted only against recent GAN-based image generation literature (where covariates for the views are often included) and the aspects related to multi-view learning are described only at the level of general intuition, instead of relating to the existing literature on the topic. The only relevant work cited from this angle is Mathieu et al. (2016), but even that is dismissed lightly by saying it is worse in generative tasks. How about the differences (theoretical and empirical) between the proposed approach and theirs in disentangling the latent variables? One would expect to see more discussion on this, given the importance of this property as motivation for the method.
The generative story using three sets of latent variables, one shared, to describe a pair of objects corresponds to inter-battery factor analysis (IBFA) and is hence very closely related to canonical correlation analysis as well (Tucker "An inter-battery method of factor analysis", Psychometrika, 1958; Klami et al. "Bayesian canonical correlation analysis", JMLR, 2013). Linear CCA naturally would not be sufficient for generative modeling and its non-linear variants (e.g. Wang et al. "Deep variational canonical correlation analysis", arXiv:1610.03454, 2016; Damianou et al. "Manifold relevance determination", ICML, 2012) would not produce visually pleasing generative samples either, but the relationship is so close that these models have even been used for analysing setups identical to yours (e.g. Li et al. "Cross-pose face recognition by canonical correlation analysis", arXiv:1507.08076, 2015) but with goals other than generation. Consequently, the reader would expect to learn something about the relationship between the proposed method and the earlier literature building on the same latent variable formulation. A particularly interesting question would be whether the proposed model actually is a direct GAN-based extension of IBFA, and if not then how does it differ. Use of adversarial training to encourage separation of latent variables is clearly a reasonable idea and quite likely does better job than the earlier solutions (typically based on some sort of group-sparsity assumption in shared-private factorisation) with the possible or even likely exception of Mathieu at al. (2016), and aspects like this should be explicitly discussed to extend the contribution from pure image generation to multi-view literature in general.
The empirical experiments are somewhat non-informative, relying heavily on visual comparisons and only satisfying the minimum requirement of demonstrating that the method does its job. The results look aesthetically more pleasing than the baselines, but the reader does not learn much about how the method actually behaves in practice; when does it break down, how sensitive it is to various choices (network structure, learning algorithm, amount of data,  how well the content and view can be disentangled from each other, etc.). In other words, the evaluation is a bit lazy somewhat in the same sense as the writing and treatment of related work; the authors implemented the model and ran it on a collection of public data sets, but did not venture further into scientific reporting of the merits and limitations of the approach.
Finally, Table 1 seems to have some min/max values the wrong way around.
Revision of the review in light of the author response:
The authors have adequately addressed my main remarks, and while doing so have improved both the positioning of the paper amongst relevant literature and the somewhat limited empirical comparisons. In particular, the authors now discuss alternative multi-view generative models not based on GANs and the revised paper includes considerably extended set of numerical comparisons that better illustrate the advantage over earlier techniques. I have increased my preliminary rating to account for these improvements.


Many types of relations in physical, biological, social and information systems can be modeled as homogeneous or heterogeneous concept graphs. Hence, learning from and with graph embeddings has drawn a great deal of research interest recently, but only ad hoc solutions have been obtained this far. In this paper, we conjecture that the one-shot supervised learning mechanism is a bottleneck in improving the performance of the graph embedding learning algorithms, and propose to extend this by introducing a multi-shot unsupervised learning framework. Empirical results on several real-world data set show that the proposed model consistently and significantly outperforms existing state-of-the-art approaches on knowledge base completion and graph based multi-label classification tasks.

The paper proposes a new method to compute embeddings of multirelational graphs. In particular, the paper proposes so-called E-Cells and R-Cells to answer queries of the form (h,r,?), (?,r,t), and(h,?,t). The proposed method (GEN), is evaluated on standard datasets for link prediction as well as datasets for node classification.
The paper tackles an interesting problem, as learning from graphs via embedding methods has become increasingly important. The experimental results of the proposed model, especially for the node classification tasks, look promising. Unfortunately, the paper makes a number of claims which are not justified or seem to result from misconceptions about related methods. For instance, the abstract labels prior work as "ad hoc solutions" and claims to propose a principled approach. However, I do not see how the proposed method is a more principled than previously proposed methods. For instance, methods such as RESCAL, TransE, HolE or ComplEx can be motivated as compositional models that reflect the compositional structure of relational data. Furthermore, RESCAL-like models can be linked to prior research in cognitive science on relational memory [3]. HolE explicitly motivates its modeling through its relation to models for associative memory.
Furthermore, due to their compositional nature, these model are all able to answer the queries considered in the paper (i.e, (h,r,?), (h,?,t), (?,r,t)) and are implicitly trained to do so. The HolE paper discusses this for instance when relating the model to associative memory. For RESCAL, [4] shows how even more complicated queries involving logical connectives and quantification can be answered. It is therefore not clear how to proposed method improves over these models.
With regard to the evaluation: It is nice that the authors provided an evaluation which compares to several SOTA methods. However, it is unclear under which setting these results where obtained. In particular, how were the hyperparameter for each model chosen and which parameters ranges were considered in the grid search. Appendix B.2 in the supplementary seems to specify the parameter setting for GEN, but it is unclear whether the same parameters where chosen for the competing models and whether they were trained with similar methods (e.g., dropout, learning rate decay etc.). The big difference in performance of HolE and ComplEx is also surprising, as they are essentially the same model (e.g. see [1,2]). It is therefore not clear to me which conclusions we can draw from the reported numbers.
Further comments:
- p.3: The statement "This is the actual way we humans learn the meaning of concepts expressed by a statement" requires justification
- p.4: The authors state that the model is trained unsupervised, but eq. 10 clearly uses supervised information in form of labels.
- p.4: In 3.1, E-cells are responsible to answer queries of the form (h,r,?) and (?, r, t), while Section 3.2 says E-Cells are used to answer (h, ?, t). I assume in the later case, the task is actually to answer (h,r,?)?
- p.2: Making a closed-world assumption is quite problematic in this context, especially when taking a principled approach. Many graphs such as Freebase are very incomplete and make an explicit open-world assumption.
- The paper uses a unusual definition of one-shot/multi-shot learning, which makes it confusing to read at first. The authors might consider using different terms to improve readability.
- Paper would benefit if the model is presented earlier. GEN Cells are defined only in Section 3.2, but the model is discussed earlier. Reversing the order might improve presentation.
[1] K. Hayashi et al: "On the Equivalence of Holographic and Complex Embeddings for Link Prediction", 2017
[2] T.Trouillon et al: "Complex and holographic embeddings of knowledge graphs: a comparison", 2017
[3] G. Halford et al: "Processing capacity defined by relational complexity: Implications for comparative, developmental, and cognitive psychology", 1998.
[4] D. Krompaß et al: "Querying factorized probabilistic triple databases", 2014


We study the properties of common loss surfaces through their Hessian matrix. In particular, in the context of deep learning, we empirically show that the spectrum of the Hessian is composed of two parts: (1) the bulk centered near zero, (2) and outliers away from the bulk. We present numerical evidence and mathematical justifications to the following conjectures laid out by : Fixing data, increasing the number of parameters merely scales the bulk of the spectrum; fixing the dimension and changing the data (for instance adding more clusters or making the data less separable) only affects the outliers. We believe that our observations have striking implications for non-convex optimization in high dimensions. First, the flatness of such landscapes (which can be measured by the singularity of the Hessian) implies that classical notions of basins of attraction may be quite misleading. And that the discussion of wide/narrow basins may be in need of a new perspective around over-parametrization and redundancy that are able to create large connected components at the bottom of the landscape. Second, the dependence of a small number of large eigenvalues to the data distribution can be linked to the spectrum of the covariance matrix of gradients of model outputs. With this in mind, we may reevaluate the connections within the data-architecturealgorithm framework of a model, hoping that it would shed light on the geometry of high-dimensional and non-convex spaces in modern applications. In particular, we present a case that links the two observations: small and large batch gradient descent appear to converge to different basins of attraction but we show that they are in fact connected through their flat region and so belong to the same basin.

This paper has at its core an interesting, novel, tentative claim, backed up by simple experiments, that small batch gradient descent and large batch gradient descent may converge to points in the same basin of attraction, contrary to the discussion (but not the actual experimental results) of Keskar et al. (2016). In general, there is a pressing need for insight into the qualitative behavior of gradient-based optimization and this area is of immense interest to many machine learning practitioners. Unfortunately the interesting tentative insights are surrounded by many unsubstantiated and only tangentially related theoretical discussions. Overall the paper has the appearance of lacking a sharp focus. This is a shame since I found the core of the paper very interesting and thought provoking.
Major comments:
While the paper has some interesting tentative experimental insights, the relationship between theory and experiment is complicated. The theoretical claims are vague and wide ranging, and are not all individually well supported or even tested by the experiments. Rather than including lots of small potential insights which the authors have had about what may be going on during gradient-based optimization, I'd prefer to see a paper with much tighter focus with a small number of theoretical claims well supported by experiments (it's fine if the experiments are simplistic as here; that's still interesting).
A large amount of the paper hinges on being able to ignore the second term in (6), and this fact is referred to many times, but the theoretical and experimental justification for this claim is very thin.
The authors mention overparameterization repeatedly, and it's in the title, but they never define it. It also doesn't appear to take center stage in their experimental investigations (if it is in fact critical to the experiments then it should be made clearer how).
Throughout this paper there is not a clear distinction between eigenvalues being zero and eigenvalues being close to zero, or similarly between the Hessian being singular and ill-conditioned. This distinction is particularly important in the theoretical discussion.
It would be helpful to be clearer about the differences between this work and that presented in Sagun et al. (2016).
Minor comments:
The assumption that the target y is real is at odds with many regression problems and practically all classification. It might be worth generalizing the discussion to multidimensional targets.
It would be good to have some citations to support the claim that often "the number of parameters M is comparable to the number of examples N (if not much larger)". With 1-dimensional targets as considered here, that sounds like a recipe for extreme overfitting and poor generalization. Generically based on counting constraints and free parameters one would expect to be able to fit exactly any dataset of N output values using a model with M free parameters. (With P-dimensional targets the relevant comparison would be M vs N P rather than M vs N).
At the end of intro to section 1, "loss is non-degenerate" should be "Hessian of the loss is non-degenerate"? Also, didn't the paper cited assume at least one negative eigenvalue at any saddle point, rather than non-degeneracy?
In section 1.1, it would be helpful to explain the precise sense in which "overparameterized" is being used. Hopefully it is in the sense that there are more parameters than needed for good performance at the true global minimum (the additional parameters helping with the process of *finding* a good minimum rather than its existence) or in the sense that M -> infinity for N "equal to" infinity. If it is in the sense that M >> N then I'm not sure of the relevance to practical machine learning.
It would be helpful to use a log scale for the plot in Figure 1. The claim that the Hessian is ill-conditioned depends on the condition number, which is impossible to estimate from the plot.
The fact that "wide basins, as opposed to narrow ones, generalize better" is not a new claim of the Keskar et al. paper. I'd argue it's well-known and part of the classical explanation of why maximum likelihood methods overfit and Bayesian ones don't. See for example MacKay, Information Theory Inference and Learning Algorithms.
"It turns out that the Hessian is degenerate at any given point" makes it sound like the result is a theoretical one. As I understand it, the experimental investigation in Sagun et al. (2016) just shows that the Hessian may often be ill-conditioned. As above, more clarity is also needed about whether it is literally degenerate or just approximately so, in which case ill-conditioned is probably a more appropriate word. Ill-conditioned is also more appropriate than singular in "slightly singular but extremely so".
How much data was used for the simple experiments in Figure 1? Infinite data? What data was used?
It would be helpful to spell out the intuition in "Intuitively, this kind of singularity...".
I don't think the decomposition (5) is required to "explain why having more parameters than samples results in degenerate Hessian matrices". Generically one would expect that with 1-dimensional targets, N datapoints and N + Q parameters, there would be a Q-dimensional submanifold of parameter space on which the loss would be zero. Of course there would be a few conditions needed to make this into a precise statement, but no need for assuming the second term is negligible.
Is the conventional decomposition of the loss into l o f used for the generalized Gauss Newton that f is a function only of the input to the neural net and the model parameters, but not the target? I could be wrong, but that was always my interpretation.
It's not clear whether the phrase "bottom of the landscape" used several times in the paper refers to the neighborhood of local minima or of global minima.
What is the justification for assuming l'(f(w)) and grad f(w) are not correlated? That seems unlikely to be true in general! Also spell out why this implies the second term can be ignored. I'm a bit skeptical of the claim in general. It's easy to come up with counterexamples. For example take l to be the identity (say f has a relu applied to it to ensure everything is well formed).
"Immediately, this implies that there are at least M - N trivial eigenvalues of the Hessian". Make it clear that trivial here means approximately not exactly zero (in which case a good word would be "small"); this follows since the second term in (5) is only approximately zero. In fact it should be possible to prove there are M - N values which are exactly zero, but that doesn't follow from the argument presented. As above I'd argue this analysis is somewhat beside the point since N should be greater than M in practice to prevent severe overfitting.
In section 3.1, "trivial eigenvalues" should be "non-trivial eigenvalues".
What's the relevance of using PCA on the data in Figure 2 when it comes to analyzing training neural nets? Also, is there any reason 2 classes breaks the trend?
What size of data was used for the experiments to plot figure 2 and figure 3? Infinite?
It's not completely clear what the takeaway is from Figure 3. I presume this is supporting the point that the eigenvalues of the Hessian at convergence consist of a bulk and outliers. The could be stated explicitly. Is there any significance to the fact that the number of clusters is equal to the number of outliers? Is this supporting some broader claim of the paper?
Figure 4, 5, 6 would benefit from being log plots, and make the claim that the bulk has the same shape independent of data much stronger.
The x-axis in Figure 5 is not "ordered counts of eigenvalues" but "index of eigenvalues", and in Figure 6 is not "ratios of eigenvalues" but ratio of the index. In the caption for Figure 6, "scaled by their ratio" is not clear.
I don't follow why Figure 6 confirms that "the effects of the ignored term in the decomposition is small" for negative eigenvalues.
In section 3.3, when saying the variances of the steps are different but the means are similar, it may interesting to note that the variance is often the dominant term and much greater in magnitude than the mean when doing SGD (at least that's what I've experienced).
What's the meaning of "elbow at similar levels"? What's the significance?
In section 4 it is claimed that overparameterization is what "leads to flatness at the bottom of the landscape which is easy to optimize". The bulk-outlier view suggests that adding extra parameters may just add extra dimensions to the flat region, but why is optimizing 100 values in a flat 100-dimensional space easier than optimizing 10 values in a flat 10-dimensional space?
In section 4.1, "fair comparison" is misleading since it depends on perspective. If one cares about compute time then certainly measuring steps rather than epochs would not be a fair comparison!
What's the relevance of the fact that random initial points in high-dimensional spaces are almost always nearly orthogonal (N.B. the "nearly" should be added)? This seems to be assuming something about the mapping from initial point to basin of attraction.
What's the meaning of "extending away from either end points appear to be confirming the sharpness of [the] LB solution"? Is this shown somewhere?
It would be helpful to highlight the key difference to Keskar et al. (which I believe is initializing SB training from LB point rather than from scratch). I presume the claim is that Keskar et al. only found their "inverted camel hump" linear interpolation results due to the random initialization, and that this would also often be observed for, say, two random LB-from-scratch trainings (which may randomly fall into different basins of attraction). If this is the intended point then it would be good to make this explicit.
In "the first terms starts to dominate", to dominate what? The gradient, or the second term in (5)? If the latter, what is the relevance of this?
Why "even" in "Even when the weight space has large flat regions"?
In the last paragraph of section 4.1, it might be worth spelling out that (as I understand it) the idea is that the small batch method finds itself in a poor region to begin with, since the average loss over an SB-noise-sized neighborhood of the LB point is actually not very good, and so there is a non-zero gradient through flat space to a place where the average loss over an SB-noise-sized neighborhood is good.
In section 5, "we see that even large batch methods are able to get to the level where small batch methods go" seems strange. Isn't this of training set loss? Isn't the "level" people care about the test set loss?
In appendix A, the meaning of consecutive in "largest consecutive gap" and "largest consecutive ratio" was not clear to me.
Appendix B is only referred to in a footnote. What is its significance for the main theme of the paper? I'd suggest either making it more prominent or putting it in a separate paper.


Generative Adversarial Networks (GANs) have become a popular method to learn a probability model from data. Many GAN architectures with different optimization metrics have been introduced recently. Instead of proposing yet another architecture, this paper aims to provide an understanding of some of the basic issues surrounding GANs. First, we propose a natural way of specifying the loss function for GANs by drawing a connection with supervised learning. Second, we shed light on the generalization peformance of GANs through the analysis of a simple LQG setting: the generator is linear, the loss function is quadratic and the data is drawn from a Gaussian distribution. We show that in this setting: 1) the optimal GAN solution converges to population Principal Component Analysis (PCA) as the number of training samples increases; 2) the number of samples required scales exponentially with the dimension of the data; 3) the number of samples scales almost linearly if the discriminator is constrained to be quadratic. Thus, linear generators and quadratic discriminators provide a good balance for fast learning.

*Paper summary*
The paper considers GANs from a theoretical point of view. The authors approach GANs from the 2-Wasserstein point of view and provide several insights for a very specific setting. In my point of view, the main novel contribution of the paper is to notice the following fact:
(*) It is well known that the 2-Wasserstein distance W2(PY,QY) between multivariate Gaussian PY and its empirical version QY scales as $n^{-2/d}$, i.e. converges very slow as the dimensionality of the space $d$ increases. In other words, QY is not such a good way to estimate PY in this setting. A somewhat better way is use a Gaussian distribution PZ with covariance matrix S computed as a sample covariance of QY. In this case W2(PY, PZ) scales as $\sqrt{d/n}$.
The paper introduces this observation in a very strange way within the context of GANs. Moreover, I think the final conclusion of the paper (Eq. 19) has a mistake, which makes it hard to see why (*) has any relation to GANs at all.
There are several other results presented in the paper regarding relation between PCA and the 2-Wasserstein minimization for Gaussian distributions (Lemma 1 & Theorem 1). This is indeed an interesting point, however the proof is almost trivial and I am not sure if this provides any significant contribution for the future research.
Overall, I think the paper contains several novel ideas, but its structure requires a *significant* rework and in the current form it is not ready for being published.
*Detailed comments*
In the first part of the paper (Section 2) the authors propose to use the optimal transport distance Wc(PY, g(PX)) between the data distribution PY (or its empirical version QY) and the model as the objective for GAN optimization. This idea is not novel: WGAN [1] proposed (and successfully implemented) to minimize the particular case of W1 distance by going through the dual form, [2] proposed to approach any Wc using auto-encoder reformulation of the primal (and also shoed that [5] is doing exactly W2 minimization), and [3] proposed the same using Sinkhorn algorithm. So this point does not seem to be novel.
The rest of the paper only considers 2-Wasserstein distance with Gaussian PY and Gaussian g(PX) (which I will abbreviate with R), which looks like an extremely limited scenario (and certainly has almost no connection to the applications of GANs).
Section 3 first establishes a relation between PCA and minimizing 2-Wasserstein distance for Gaussian distributions (Lemma 1, Theorem 1). Then the authors show that if R minimizes W2(PY, R) and QR minimizes W2(QY, QR) then the excess loss W2(PY, QR) - W2(PY, R) approaches zero at the rate $n^{-2/d}$ (both for linear and unconstrained generators). This result basically provides an upper bound showing that GANs need exponentially many samples to minimize W2 distance. I don't find these results novel, as they already appeared in [4] with a matching lower bound for the case of Gaussians (Theorem B.1 in Appendix can be modified easily to show this). As the authors note in the conclusion of Section 3, these results have little to do with GANs, as GANs are known to learn quite quickly (which contradicts the theory of Section 3).
Finally, in Section 4 the authors approach the same W2 problem from its dual form and notice that for the LQG model the optimal discriminator is quadratic. Based on this they reformulate the W2 minimization for LQG as the constrained optimization with respect to p.d. matrix A (Eq 16). The same conclusion does not work unfortunately for W2(QY, R), which is the real training objective of GANs. Theorem 3 shows that nevertheless, if we still constrain discriminator in the dual form of W2(QY, R) to be quadratic, the resulting soliton QR* performs the empirical PCA of Pn.
This leads to the final conclusion of the paper, which I think contains a mistake. In Eq 19 the first equation, according to the definitions of the authors, reads
\[
W2(PY, QR) = W2(PY, PZ),   (**)
\]
where QR is trained to minimize min_R W2(QY, R) and PZ is as defined in (*) in the beginning of these notes.
However, PZ is not the solution of min_R W2(QY, R) as the authors notice in the 2nd paragraph of page 8. Thus (**) is not true (at least, it is not proved in the current version of the text). PZ is a solution of min_R W2(QY, R) *where the discriminator is constrained to be quadratic*. This mismatch is especially strange, given the authors emphasize in the introduction that they provide bounds on divergences which are the same as used during the training (see 2nd paragraph on page 2) --- here the bound is on W2, but the empirical GAN actually does a regularized training (with constrained discriminator).
Finally, I don't think the experiments provide any convincing insights, because the authors use W1-minimization to illustrate properties of the W2. Essentially the authors say "we don't have a way to perform W2 minimization, so we rather do the W1 minimization and assume that these two are kind of similar".
* Other comments *
(1) Discussion in Section 2.1 seems to never play a role in the paper.
(2) Page 4: in p-Wasserstein distance, ||.|| does not need to be a Euclidean metric. It can be any metric.
(3) Lemma 2 seems to repeat the result from (Canas and Rosasco, 2012) as later cited by authors on page 7?
(4) It is not obvious how does Theorem 2 translate to the excess loss?
(5) Section 4. I am wondering how exactly the authors are going to compute the conjugate of the discriminator, given the discriminator most likely is a deep neural network?
[1] Arjovsky et al., Wasserstein GAN, 2017
[2] Bousquet et al, From optimal transport to generative modeling: the VEGAN cookbook, 2017
[3] Genevay et al., Learning Generative Models with Sinkhorn Divergences, 2017
[4] Arora et al, Generalization and equilibrium in GANs, 2017
[5] Makhazani et al., Adversarial Autoencoders, 2015


Published as a conference paper at ICLR 2018 OVERCOMING CATASTROPHIC INTERFERENCE USING CONCEPTOR-AIDED BACKPROPAGATION
Catastrophic interference has been a major roadblock in the research of continual learning. Here we propose a variant of the back-propagation algorithm, "conceptor-aided backprop" (CAB), in which gradients are shielded by conceptors against degradation of previously learned tasks. Conceptors have their origin in reservoir computing, where they have been previously shown to overcome catastrophic forgetting. CAB extends these results to deep feedforward networks. On the disjoint and permuted MNIST tasks, CAB outperforms two other methods for coping with catastrophic interference that have recently been proposed.

[Reviewed on January 12th]
This article applies the notion of “conceptors” -- a form of regulariser introduced by the same author a few years ago, exhibiting appealing boolean logic pseudo-operations -- to prevent forgetting in continual learning,more precisely in the training of neural networks on sequential tasks. It proposes itself as an improvement over the main recent development of the field, namely Elastic Weight Consolidation.  After a brief and clear introduction to conceptors and their application to ridge regression, the authors explain how to inject conceptors into Stochastic Gradient Descent and finally, the real innovation of the paper, into Backpropagation. Follows a section of experiments on variants of MNIST commonly used for continual learning.
Continual learning in neural networks is a hot topic, and this article contributes a very interesting idea. The notion of conceptors is appealing in this particular use for its interpretation in terms of regularizer and in terms of Boolean logic.  The numeric examples, although quite toy, provide a clear illustration.
A few things are still missing to back the strong claims of this paper:
* Some considerations of the computational costs: the reliance on the full NxN correlation matrix R makes me fear it might be costly, as it is applied to every layer of the neural networks and hence is the largest number of units in a layer.  This is of course much lighter than if it were the covariance matrix of all the weights, which would be daunting, but still deserves to be addressed, if only with wall time measures.
* It could also be welcome to use a more grounded vocabulary, e.g. on p.2 “Figure 1 shows examples of conceptors computer from three clouds of sample state points coming from a hypothetical 3-neuron recurrent network that was drive with input signals from three difference sources” could be much more simply said as “Figure 1 shows the ellipses corresponding to three sets of R^3 points”. Being less grandiose would make the value of this article nicely on its own.
* Some examples beyond the contrived MNIST toy examples would be welcome. For example, the main method this article is compared to (EWC) had a very strong section on Reinforcement learning examples in the Atari framework, not only as an illustration but also as a motivation. I realise not everyone has the computational or engineering resources to try extensively on multiple benchmarks from classification to reinforcement learning. Nevertheless, without going to that extreme, it might be worth adding an extra demo on something bigger than MNIST. The authors transparently explain in their answer that they do not (yet!) belong to the deep learning community and hope finding some collaborations to pursue this further. If I may make a suggestion, I think their work would get much stronger impact by  doing it the reverse way: first finding the collaboration, then adding this extra empirical results, which then leads to a bigger impact publication.
The later point would normally make me attribute a score of "6: Marginally above acceptance threshold" by current DL community standards, but because there is such a pressing need for methods to tackle this problem, and because this article can generate thinking along new lines about this, I give it a 7 : Good paper, accept.


Optimization algorithms for training deep models not only affects the convergence rate and stability of the training process, but are also highly related to the generalization performance of trained models. While adaptive algorithms, such as Adam and RMSprop, have shown better optimization performance than stochastic gradient descent (SGD) in many scenarios, they often lead to worse generalization performance than SGD, when used for training deep neural networks (DNNs). In this work, we identify two problems regarding the direction and step size for updating the weight vectors of hidden units, which may degrade the generalization performance of Adam. As a solution, we propose the normalized direction-preserving Adam (ND-Adam) algorithm, which controls the update direction and step size more precisely, and thus bridges the generalization gap between Adam and SGD. Following a similar rationale, we further improve the generalization performance in classification tasks by regularizing the softmax logits. By bridging the gap between SGD and Adam, we also shed some light on why certain optimization algorithms generalize better than others.

Method:
The paper is missing analysis of some important related works such as
"Beyond convexity: Stochastic quasi-convex optimization" by E. Hazan et al. (2015)
where Stochastic Normalized Gradient Descent (SNGD) was proposed.
Then, normalized gradient versions of AdaGrad and Adam were proposed in
"Normalized Gradient with Adaptive Stepsize Method for Deep
Neural Network Training" by A. W. Yu et al. (2017).
Another work which I find to be relevant is
"Follow the Signs for Robust Stochastic Optimization" by L. Balles and P. Hennig (2017).
From my personal experiments, restricting w_i to have L2 norm of 1, i.e., to be +-1
leads to worse generalization. One reason for this is that weight decay is not
really functioning since it cannot move w_i to 0 or make its amplitude any smaller.
Please correct me if I misunderstand something here.
The presence of +-1 weights moves us to the area of low-precision NNs,
or more specifically, NNs with binary / binarized weights as in
"BinaryConnect: Training Deep Neural Networks with
binary weights during propagations" by M. Courbariaux et al. (2015)
and
"Binarized neural networks: Training deep neural networks with weights and activations constrained to+ 1 or-1" by M. Courbariaux et al. (2016).
Regarding
"Moreover, the magnitude of each update does not depend on themagnitude of the gradient. Thus, ND-Adam is more robust to improper initialization, and vanishing or exploding gradients."
If the magnitude of each update does not depend on the magnitude of the gradient, then the algorithm heavily depends on the learning rate. Otherwise, it does not have any means to approach the optimum in a reasonable number of steps *when* it is initialized very / unreasonably far from it. The claim of your second sentence is not supported by the paper.
Evaluation:
I am not confident that the presented experimental validation is fair. First, the original WRN paper and many other papers with ResNets used weight decay of 0.0005 and not 0.001 or 0.002 as used for SGD in this paper. It is unclear why this setting was changed. One could just use \alpha_0 = 0.05 and \lambda = 0.0005.
Then, I don't see why the authors use WRN-22-7.5 which is different from WRN-28-10 which was suggested in the original study and used in several follow-up works. The difference between WRN-22-7.5 and WRN-28-10 is unlikely to be significant,
the former might have about only 2 times less parameters which should barely change the final validation errors. However, the use of WRN-22-7.5 makes it impossible to easily compare the presented results to the results of Zagoruyko who had 3.8\% with WRN-28-10. I believe that the use of the setup of Zagoruyko for WRN-22-7.5 would allow to get much better results than 4.5\% and 4.49\% shown for SGD and likely better 4.14\% shown for ND-Adam. I note that the use of WRN-22-7.5 is unlikely to be due to the used hardware because later in paper the authors refer to WRN-34-7.5.
My intuition is that the proposed ND-Adam moves the algorithm back to SGD but with potentially harmful constraints of w_i=+-1. Even the values of \alpha^v_0 found for ND-Adam (e.g., \alpha^v_0=0.05 in Figure 1B) are in line of what would be optimal values of \alpha_0 for SGD.
I find it uncomfortable that BN-Softmax is introduced here to support the use of an optimization algorithm, moreover, that the values of \gamma_c are different for CIFAR-10 and CIFAR-100. I wonder if the proposed values are optimal (and therefore selected) for all three tested algorithms  or only for Adam-ND. I expect that hyperparameters of SGD and Adam would also need to be revised to account for BN-Softmax.


Conventional methods model open domain dialogue generation as a black box through end-to-end learning from large scale conversation data. In this work, we make the first step to open the black box by introducing dialogue acts into open domain dialogue generation. The dialogue acts are generally designed and reveal how people engage in social chat. Inspired by analysis on real data, we propose jointly modeling dialogue act selection and response generation, and perform learning with human-human conversations tagged with a dialogue act classifier and a reinforcement approach to further optimizing the model for long-term conversation. With the dialogue acts, we not only achieve significant improvement over state-of-the-art methods on response quality for given contexts and long-term conversation in both machine-machine simulation and human-machine conversation, but also are capable of explaining why such achievements can be made.

The authors use a distant supervision technique to add dialogue act tags as a conditioning factor for generating responses in open-domain dialogues. In their evaluations, this approach, and one that additionally uses policy gradient RL with discourse-level objectives to fine-tune the dialogue act predictions, outperform past models for human-scored response quality and conversation engagement.
While this is a fairly straightforward idea with a long history, the authors claim to be the first to use dialogue act prediction for open-domain (rather than task-driven) dialogue. If that claim to originality is not contested, and the authors provide additional assurances to confirm the correctness of the implementations used for baseline models, this article fills an important gap in open-domain dialogue research and suggests a fruitful future for structured prediction in deep learning-based dialogue systems.
Some points:
1. The introduction uses "scalability" throughout to mean something closer to "ability to generalize." Consider revising the wording here.
2. The dialogue act tag set used in the paper is not original to Ivanovic (2005) but derives, with modifications, from the tag set constructed for the DAMSL project (Jurafsky et al., 1997; Stolcke et al., 2000). It's probably worth citing some of this early work that pioneered the use of dialogue acts in NLP, since they discuss motivations for building DA corpora.
3. In Section 2.1, the authors don't explicitly mention existing DA-annotated corpora or discuss specifically why they are not sufficient (is there e.g. a dataset that would be ideal for the purposes of this paper except that it isn't large enough?)
3. The authors appear to consider only one option (selecting the top predicted dialogue act, then conditioning the response generator on this DA) among many for inference-time search over the joint DA-response space. A more comprehensive search strategy (e.g. selecting the top K dialogue acts, then evaluating several responses for each DA) might lead to higher response diversity.
4. The description of the RL approach in Section 3.2 was fairly terse and included a number of ad-hoc choices. If these choices (like the dialogue termination conditions) are motivated by previous work, they should be cited. Examples (perhaps in the appendix) might also be helpful for the reader to understand that the chosen termination conditions or relevance metrics are reasonable.
5. The comparison against previous work is missing some assurances I'd like to see. While directly citing the codebases you used or built off of is fantastic, it's also important to give the reader confidence that the implementations you're comparing to are the same as those used in the original papers, such as by mentioning that you can replicate or confirm quantitative results from the papers you're comparing to. Without that there could always be the chance that something is missing from the implementation of e.g. RL-S2S that you're using for comparison.
6. Table 5 is not described in the main text, so it isn't clear what the different potential outputs of e.g. the RL-DAGM system result from (my guess: conditioning the response generation on the top 3 predicted dialogue acts?)
7. A simple way to improve the paper's clarity for readers would be to break up some of the very long paragraphs, especially in later sections. It's fine if that pushes the paper somewhat over the 8th page.
8. A consistent focus on human evaluation, as found in this paper, is probably the right approach for contemporary dialogue research.
9. The examples provided in the appendix are great. It would be helpful to have confirmation that they were selected randomly (rather than cherry-picked).


Published as a conference paper at ICLR 2018 SELF-ENSEMBLING FOR VISUAL DOMAIN ADAPTATION
This paper explores the use of self-ensembling for visual domain adaptation problems. Our technique is derived from the mean teacher variant (Tarvainen & Valpola (2017)) of temporal ensembling (Laine & Aila (2017)), a technique that achieved state of the art results in the area of semi-supervised learning. We introduce a number of modifications to their approach for challenging domain adaptation scenarios and evaluate its effectiveness. Our approach achieves state of the art results in a variety of benchmarks, including our winning entry in the VISDA-2017 visual domain adaptation challenge. In small image benchmarks, our algorithm not only outperforms prior art, but can also achieve accuracy that is close to that of a classifier trained in a supervised fashion.

This paper presents a domain adaptation algorithm based on the self-ensembling method proposed by [Tarvainen & Valpola, 2017]. The main idea is to enforce the agreement between the predictions of the teacher and the student classifiers on the target domain samples while training the student to perform well on the source domain. The teacher network is simply an exponential moving average of different versions of the student network over time.
Pros:
+ The paper is well-written and easy to read
+ The proposed method is a natural extension of the mean teacher semi-supervised learning model by [Tarvainen & Valpola, 2017]
+ The model achieves state-of-the-art results on a range of visual domain adaptation benchmarks (including top performance in the VisDA17 challenge)
Cons:
- The model is tailored to the image domain as it makes heavy use of the data augmentation. That restricts its applicability quite significantly. I’m also very interested to know how the proposed method works when no augmentation is employed (for fair comparison with some of the entries in Table 1).
- I’m not particularly fond of the engineering tricks like confidence thresholding and the class balance loss. They seem to be essential for good performance and thus, in my opinion, reduce the value of the main idea.
- Related to the previous point, the final VisDA17 model seems to be engineered too heavily to work well on a particular dataset. I’m not sure if it provides many interesting insights for the scientific community at large.
In my opinion, it’s a borderline paper. While the best reported quantitative results are quite good, it seems that achieving those requires a significant engineering effort beyond just applying the self-ensembling idea.
Notes:
* The paper somewhat breaks the anonymity of the authors by mentioning the “winning entry in the VISDA-2017”. Maybe it’s not a big issue but in my opinion it’s better to remove references to the competition entry.
* Page 2, 2.1, line 2, typo: “stanrdard” -> “standard”
Post-rebuttal revision:
After reading the authors' response to my review, I decided to increase the score by 2 points. I appreciate the improvements that were made to the paper but still feel that this work a bit too engineering-heavy, and the title does not fully reflect what's going on in the full pipeline.


Published as a conference paper at ICLR 2018 LEARNING TO TEACH
Teaching plays a very important role in our society, by spreading human knowledge and educating our next generations. A good teacher will select appropriate teaching materials, impact suitable methodologies, and set up targeted examinations, according to the learning behaviors of the students. In the field of artificial intelligence, however, one has not fully explored the role of teaching, and pays most attention to machine learning. In this paper, we argue that equal attention, if not more, should be paid to teaching, and furthermore, an optimization framework (instead of heuristics) should be used to obtain good teaching strategies. We call this approach "learning to teach". In the approach, two intelligent agents interact with each other: a student model (which corresponds to the learner in traditional machine learning algorithms), and a teacher model (which determines the appropriate data, loss function, and hypothesis space to facilitate the training of the student model). The teacher model leverages the feedback from the student model to optimize its own teaching strategies by means of reinforcement learning, so as to achieve teacher-student co-evolution. To demonstrate the practical value of our proposed approach, we take the training of deep neural networks (DNN) as an example, and show that by using the learning to teach techniques, we are able to use much less training data and fewer iterations to achieve almost the same accuracy for different kinds of DNN models (e.g., multi-layer perceptron, convolutional neural networks and recurrent neural networks) under various machine learning tasks (e.g., image classification and text understanding).

This paper focuses on the problem of "machine teaching", i.e., how to select a good strategy to select training data points to pass to a machine learning algorithm, for faster learning. The proposed approach leverages reinforcement learning by defining the reward as how fast the learner learns, and use policy gradient to update the teacher parameters. I find the definition of the "state" in this case very interesting. The experimental results seem to show that such a learned teacher strategy makes machine learning algorithms learn faster.
Overall I think that this paper is decent. The angle the authors took is interesting (essentially replacing one level of the bi-level optimization problem in machine teaching works with a reinforcement learning setup). The problem formulation is mostly reasonable, and the evaluation seems quite convincing. The paper is well-written: I enjoyed the mathematical formulation (Section 3). The authors did a good job of using different experiments (filtration number analysis, and teaching both the same architecture and a different architecture) to intuitively explain what their method actually does.
At the same time, though, I see several important issues that need to be addressed if this paper is to be accepted. Details below.
1. As much as I enjoyed reading Section 3, it is very redundant. In some cases it is good to outline a powerful and generic framework (like the authors did here with defining "teaching" in a very broad sense, including selecting good loss functions and hypothesis spaces) and then explain that the current work focuses on one aspect (selecting training data points). However, I do not see it being the case here. In my opinion, selecting good loss functions and hypothesis spaces are much harder problems than data teaching - except maybe when one use a pre-defined set of possible loss functions and select from it. But that is not very interesting (if you can propose new loss functions, that would be way cooler). I also do not see how to define an intuitive set of "states" in that case. Therefore, I think this section should be shortened. I also think that the authors should not discuss the general framework and rather focus on "data teaching", which is the only focus of the current paper. The abstract and introduction should also be modified accordingly to more honestly reflect the current contributions.
2. The authors should do a better job at explaining the details of the state definition, especially the student model features and the combination of data and current learner model.
3. There is only one definition of the reward - related to batch number when the accuracy first exceeds a threshold. Is accuracy stable, can it drop back down below the threshold in the next epoch? The accuracy on a held-out test set is not guaranteed to be monotonically increasing, right? Is this a problem in practice (it seems to happen on your curves)? What about other potential reward definitions? And what would they potentially lead to?
4. Experimental results are averaged over 5 repeated runs - a bit too small in my opinion.
5. Can the authors show convergence of the teacher parameter \theta? I think it is important to see how fast the teacher model converges, too.
6. In some of your experiments, every training method converges to the same accuracy after enough training (Fig.2b), while in others, not quite (Fig. 2a and 2c). Why is this the case? Does it mean that you have not run enough iterations for the baseline methods? My intuition is that if the learner algorithm is convex, then ultimately they will all get to the same accuracy level, so the task is just to get there quicker. I understand that since the learner algorithm is an NN, this is not the case - but more explanation is necessary here - does your method also reduces the empirical possibility to get stuck in local minima?
7. More explanation is needed towards Fig.4c. In this case, using a teacher model trained on a harder task (CIFAR10) leads to much improved student training on a simpler task (MNIST). Why?
8. Although in terms of "effective training data points" the proposed method outperforms the other methods, in terms of time (Fig.5) the difference between it and say, NoTeach, is not that significant (especially at very high desired accuracy). More explanation needed here.
Read the rebuttal and revision and slightly increased my rating.


Published as a conference paper at ICLR 2018 IDENTIFYING ANALOGIES ACROSS DOMAINS
Identifying analogies across domains without supervision is an important task for artificial intelligence. Recent advances in cross domain image mapping have concentrated on translating images across domains. Although the progress made is impressive, the visual fidelity many times does not suffice for identifying the matching sample from the other domain. In this paper, we tackle this very task of finding exact analogies between datasets i.e. for every image from domain A find an analogous image in domain B. We present a matching-by-synthesis approach: AN-GAN, and show that it outperforms current techniques. We further show that the cross-domain mapping task can be broken into two parts: domain alignment and learning the mapping function. The tasks can be iteratively solved, and as the alignment is improved, the unsupervised translation function reaches quality comparable to full supervision.

This paper adds an interesting twist on top of recent unpaired image translation work. A domain-level translation function is jointly optimized with an instance-level matching objective. This yields the ability to extract corresponding image pairs out of two unpaired datasets, and also to potentially refine unpaired translation by subsequently training a paired translation function on the discovered matches. I think this is a promising direction, but the current paper has unconvincing results, and it’s not clear if the method is really solving an important problem yet.
My main criticism is with the experiments and results. The experiments focus almost entirely on the setting where there actually exist exact matches between the two image sets. Even the partial matching experiments in Section 4.1.2 only quantify performance on the images that have exact matches. This is a major limitation since the compelling use cases of the method are in scenarios where we do not have exact matches. It feels rather contrived to focus so much on the datasets with exact matches since, 1) these datasets actually come as paired data and, in actual practice, supervised translation can be run directly, 2) it’s hard to imagine datasets that have exact but unknown matches (I welcome the authors to put forward some such scenarios), 3) when exact matches exist, simpler methods may be sufficient, such as matching edges. There is no comparison to any such simple baselines.
I think finding analogies that are not exact matches is much more compelling. Quantifying performance in this case may be hard, and the current paper only offers a few qualitative results. I’d like to see far more results, and some attempt at a metric. One option would be to run user studies where humans judge the quality of the matches. The results shown in Figure 2 don’t convince me, not just because they are qualitative and few, but also because I’m not sure I even agree that the proposed method is producing better results: for example, the DiscoGAN results have some artifacts but capture the texture better in row 3.
I was also not convinced by the supervised second step in Section 4.3. Given that the first step achieves 97% alignment accuracy, it’s no surprised that running an off-the-shelf supervised method on top of this will match the performance of running on 100% correct data. In other words, this section does not really add much new information beyond what we could already infer given that the first stage alignment was so successful.
What I think would be really interesting is if the method can improve performance on datasets that actually do not have ground truth exact matches. For example, the shoes and handbags dataset or even better, domain adaptation datasets like sim to real.
I’d like to see more discussion of why the second stage supervised problem is beneficial. Would it not be sufficient to iterate alpha and T iterations enough times until alpha is one-hot and T is simply training against a supervised objective (Equation 7)?
Minor comments:
1. In the intro, it would be useful to have a clear definition of “analogy” for the present context.
2. Page 2: a link should be provided for the Putin example, as it is not actually in Zhu et al. 2017.
3. Page 3: “Weakly Supervised Mapping” — I wouldn’t call this weakly supervised. Rather, I’d say it’s just another constraint / prior, similar to cycle-consistency, which was referred to under the “Unsupervised” section.
4. Page 4 and throughout: It’s hard to follow which variables are being optimized over when. For example, in Eqn. 7, it would be clearer to write out the min over optimization variables.
5. Page 6: The Maps dataset was introduced in Isola et al. 2017, not Zhu et al. 2017.
6. Page 7: The following sentence is confusing and should be clarified: “This shows that the distribution matching is able to map source images that are semantically similar in the target domain.”
7. Page 7: “This shows that a good initialization is important for this task.” — Isn’t this more than initialization? Rather, removing the distributional and cycle constraints changes the overall objective being optimized.
8. In Figure 2, are the outputs the matched training images, or are they outputs of the translation function?
9. Throughout the paper, some citations are missing enclosing parentheses.


Topic modeling of text documents is one of the most important tasks in representation learning. In this work, we propose iTM-VAE, which is a Bayesian nonparametric (BNP) topic model with variational auto-encoders. On one hand, as a BNP topic model, iTM-VAE potentially has infinite topics and can adapt the topic number to data automatically. On the other hand, different with the other BNP topic models, the inference of iTM-VAE is modeled by neural networks, which has rich representation capacity and can be computed in a simple feed-forward manner. Two variants of iTM-VAE are also proposed in this paper, where iTM-VAE-Prod models the generative process in products-of-experts fashion for better performance and iTM-VAE-G places a prior over the concentration parameter such that the model can adapt a suitable concentration parameter to data automatically. Experimental results on 20News and Reuters RCV1-V2 datasets show that the proposed models outperform the state-of-the-arts in terms of perplexity, topic coherence and document retrieval tasks. Moreover, the ability of adjusting the concentration parameter to data is also confirmed by experiments.

"topic modeling of text documents one of most important tasks"
Does this claim have any backing?
"inference of HDP is more complicated and not easy to be applied to new models"  Really an artifact of the misguided nature of earlier work. The posterior for the $\vec\pi$ of a elements of DP or HDP can be made a Dirichlet, made finite by keeping a "remainder" term and appropriate augmentation.  Hughes, Kim and Sudderth (2015) have avoided stick-breaking and CRPs altogether, as have others in earlier work. Extensive models building on simple HDP doing all sorts of things have been developed.
Variational stick-breaking methods never seemed to have worked well.  I suspect you could achieve better results by replacing them as well, but you would have to replace the tree of betas and extend your Kumaraswamy distribution, so it may not work.  Anyway, perhaps an avenue for future work.
"infinite topic models" I've always taken the view that the use of the word "infinite" in machine learning is a kind of NIPSian machismo. In HDP-LDA at least, the major benefit in model performance comes from fitting what you call $\vec\pi$, which is uniform in vanilla LDA, and note that the number of topics "found" by a HDP-LDA sampler can be made to vary quite widely by varying what you call $\alpha$, so any statement about the "right" number of topics is questionable.  So the claim in 3rd paragraph of Section 2, "superior" and "self-determined topic number" I'd say are misguided.  Plenty of experimental work to support this.
In Related Work, you seem to only mention HDP for non-parametric topic models.  More work exists, for instance using Pitman-Yor distributions for modelling words and using Gibbs samplers that are efficient and don't rely on the memory hungry HCRP.
Good to see a prior is placed on the concentration parameter.  Very important and not well done in the community, usually.
ADDED:  Originally done by Teh et al for HDP-LDA, and subsequently done
by several, including Kim et al 2016.   Others stress the importance of this.  You need to
cite at least Teh et al. in 5.4 to show this isn't new and the importance is well known.
The Prod version is a very nice idea.  Great results.  This looks original, but I'm not expert enough in the huge masses of new deep neural network research popping up.
You've upped the standard a bit by doing good experimental work.  Oftentimes this is done poorly and one is left wondering.  A lot of effort went into this.
ADDED:   usually like to see more data sets experimented with
What code is used for HDP-LDA?  Teh's original Matlab HCRP sampler does pretty well because at least he samples hyperparameters and can scale to 100k documents (yes, I tried). The comparison with LDA makes me suspicious. For instance, on 20News, a good non-parametric LDA will find well over 400 topics and roundly beat LDA on just 50 or 200.  If reporting LDA, or HDP-LDA, it should be standard to do hyperparameter fitting and you need to mention what you did as this makes a big difference.
ADDED:   20News results still poor for HPD, but its probably the implementation used ... their
        online variational algorithm only has advantages for large data sets
Pros:
* interesting new prod model with good results
* alternative "deep" approach to a HDL-LDA model
* good(-ish) experimental work
Cons:
* could do with a competitive non-parametric LDA implementation
ADDED:   good review responses generally


Existing black-box attacks on deep neural networks (DNNs) so far have largely focused on transferability, where an adversarial instance generated for a locally trained model can "transfer" to attack other learning models. In this paper, we propose novel Gradient Estimation black-box attacks for adversaries with query access to the target model's class probabilities, which do not rely on transferability. We also propose strategies to decouple the number of queries required to generate each adversarial sample from the dimensionality of the input. An iterative variant of our attack achieves close to 100% adversarial success rates for both targeted and untargeted attacks on DNNs. We carry out extensive experiments for a thorough comparative evaluation of black-box attacks and show that the proposed Gradient Estimation attacks outperform all transferability based black-box attacks we tested on both MNIST and CIFAR-10 datasets, achieving adversarial success rates similar to well known, state-of-the-art white-box attacks. We also apply the Gradient Estimation attacks successfully against a real-world content moderation classifier hosted by Clarifai. Furthermore, we evaluate black-box attacks against state-of-theart defenses. We show that the Gradient Estimation attacks are very effective even against these defenses.

The authors consider new attacks for generating adversarial samples against neural networks. In particular, they are interested in approximating gradient-based white-box attacks such as FGSM in a black-box setting by estimating gradients from queries to the classifier. They assume that the attacker is able to query, for any example x, the vector of probabilities p(x) corresponding to each class.
Given such query access it’s trivial to estimate the gradients of p using finite differences. As a consequence one can implement FGSM using these estimates assuming cross-entropy loss, as well as a logit-based loss. They consider both iterative and single-step FGSM attacks in the targeted (i.e. the adversary’s goal is to switch the example’s label to a specific alternative label) and un-targeted settings (any mislabelling is a success). They compare themselves to transfer black-box attacks, where the adversary trains a proxy model and generates the adversarial sample by running a white-box attack on that model.  For a number of target classifiers on both MNIST and CIFAR-10, they show that these attacks outperform the transfer-based attacks, and are comparable to white-box attacks, while maintaining low distortion on the attack samples.
One drawback of estimating gradients using finite differences is that the number of queries required scales with the dimensionality of the examples, which can be prohibitive in the case of images. They therefore describe two practical approaches for query reduction — one based on random feature grouping, and the other on PCA (which requires access to training data). They once again demonstrate the effectiveness of these methods across a number of models and datasets, including models deploying adversarially trained defenses.
Finally, they demonstrate compelling real-world deployment against Clarifai classification models designed to flag “Not Safe for Work” content.
Overall, the paper provides a very thorough experimental examination of a practical black-box attack that can be deployed against real-world systems. While there are some similarities with Chen et al. with respect to utilizing finite-differences to estimate gradients, I believe the work is still valuable for its very thorough experimental verification, as well as the practicality of their methods. The authors may want to be more explicit about their claim in the Related Work section that the running time of their attack is “40x” less than that of Chen et al. While this is believable, there is no running time comparison in the body of the paper.


Modern neural network architectures take advantage of increasingly deeper layers, and various advances in their structure to achieve better performance. While traditional explicit regularization techniques like dropout, weight decay, and data augmentation are still being used in these new models, little about the regularization and generalization effects of these new structures have been studied. Besides being deeper than their predecessors, could newer architectures like ResNet and DenseNet also benefit from their structures' implicit regularization properties? In this work, we investigate the skip connection's effect on network's generalization features. Through experiments, we show that certain neural network architectures contribute to their generalization abilities. Specifically, we study the effect that low-level features have on generalization performance when they are introduced to deeper layers in DenseNet, ResNet as well as networks with 'skip connections'. We show that these low-level representations do help with generalization in multiple settings when both the quality and quantity of training data is decreased.

The paper studies the effect of different network structures (plain CNN, ResNet and DenseNet). This is an interesting line of research to pursue, however, it gives an impression that a large amount of recent work in this direction has not been considered by the authors. The paper contains ONLY 4 references.
Some references that might be useful to consider in the paper:
- K. Greff et. al. Highway and Residual Networks learn Unrolled Iterative Estimation.
- C. Zang et. al. UNDERSTANDING DEEP LEARNING REQUIRES RETHINKING GENERALIZATION
- Q. Liao el. al. Bridging the Gaps Between Residual Learning, Recurrent Neural Networks and Visual Cortex
- A. Veit et. al. Residual Networks Behave Like Ensembles of Relatively Shallow Networks
- K. He at. Al Identity Mappings in Deep Residual Networks
The writing and the structure of the paper could be significantly improved. From the paper, it is difficult to understand the contributions. From the ones listed in Section 1, it seems that most of the contributions were shown in the original ResNet and DenseNet papers. Given, questionable contribution and a lack of relevant citations, it is difficult to recommend for acceptance of the paper.
Other issues:
Section 2: “Skip connection …. overcome the overfitting”, could the authors comment on this a bit more or point to relevant citation?
Section 2: “We increase the number of skip connections from 0 to 28”, it is not clear to me how this is done.
Section 3.1.1 “deep Linear model”, what the authors mean with this? Multiple layers without a nonlinearity? Is it the same as Cascade Net?
Section 3.2 From the data description, it is not clear how the training data was obtained. Could the authors provide more details on this?
Section 3.2 “…, only 3 of them are chosen to be displayed…”, how the selection process was done?
Section 3.2 “Instead of showing every layer’s output we exhibit the 3th, 5th, 7th, 9th, 11th, 13th and the final layer’s output”, according to the description in Fig. 7 we should be able to see 7 columns, this description does not correspond to Fig. 7.
Section 4 “This paper investigates how skip connections works in vision tasks…” I do not find experiments with vision datasets in the paper. In order to claim this, I would encourage the authors to run tests on a CV benchmark dataset (e. g. ImageNet)


Model-free deep reinforcement learning (RL) algorithms have been demonstrated on a range of challenging decision making and control tasks. However, these methods typically suffer from two major challenges: very high sample complexity and brittle convergence properties, which necessitate meticulous hyperparameter tuning. Both of these challenges severely limit the applicability of such methods to complex, real-world domains. In this paper, we propose soft actor-critic, an offpolicy actor-critic deep RL algorithm based on the maximum entropy reinforcement learning framework. In this framework, the actor aims to maximize expected reward while also maximizing entropy-that is, succeed at the task while acting as randomly as possible. Prior deep RL methods based on this framework have been formulated as either off-policy Q-learning, or on-policy policy gradient methods. By combining off-policy updates with a stable stochastic actor-critic formulation, our method achieves state-of-the-art performance on a range of continuous control benchmark tasks, outperforming prior on-policy and off-policy methods. Furthermore, we demonstrate that, in contrast to other off-policy algorithms, our approach is very stable, achieving very similar performance across different random seeds.

This paper proposes a soft actor-critic method aiming at lowering sample complexity and achieving a new convergence guarantee. However, the current paper has some correctness issues, is missing some related work and lacks a clear statement of innovation.
The first issue is that augmenting reward by adding an entropy term to the original RL objective is not clearly innovative. The connections, and improvements upon, other approaches need to be made more clear. In particular, the connection to the work by Haarnoja is unclear. There is this statement: “Although the soft Q-learning algorithm proposed by Haarnoja et al. (2017) has a value function and actor network, it is not a true actor-critic algorithm: the Q- function is estimating the optimal Q-function, and the actor does not directly affect the Q-function except through the data distribution. Hence, Haarnoja et al. (2017) motivates the actor network as an approximate sampler, rather than the actor in an actor-critic algorithm. Crucially, the convergence of this method hinges on how well this sampler approximates the true posterior. In contrast, we prove that our method converges to the optimal policy from a given policy class, regardless of the policy parameterization.” The last sentence suggests that the key difference is that any policy parameterization can be used, making the previous sentences less clear. Is the key extension on the proof, and so on the use of the projection with the KL-divergence?
Further, there is a missing connection to the paper “Guided policy search”, Levine and Koltun. Though it is a different framework, it clearly mentioned it uses the augmented reward to learn the sub-optimal policies (for differential dynamic program). The DDPG paper mentioned that DDPG can be also used within the GPS framework. That work is different, but a discussion should nonetheless be included about connections.

If the key novelty in this work is an extension on the theory, to allow any policy parameterization, and empirical results demonstrating improved performance over Haarnoja et al., there appear to be correctness issues in both, as laid out below.
The key novelty in the theory seems to be to use a projection onto the space of policies, using a KL divergence. There are, however, currently too many unclear or misspecified steps to verify correctness.
1. The definition of pinew in Equation (6) is for one specific state s_t; shouldn’t this be across all states? If it is for one state, then E_{pinew} makes sense, since pi is only specified as a conditional distribution (not a joint distribution); if it is supposed to be expected value across all states, then what is E_{pinew}? Is it with the stationary distribution of pinew?
2. The proof for Lemma 1 is hard to follow, because Z is not defined. I mostly was able to guess, based on Haarnoja et al., but the step between (18) and (19) where E_{pinew} [Zpiold] -  E_{piold} [Zpiold] is dropped is unclear to me. Zpiold does not depend on actions, so if the expectation is only w.r.t. to the action, then it cancels. This goes back to point 1, where it wouldn’t make much sense for the KL to only depend on actions. In fact, if pinew has to be computed separately for each state, then we are really back to tabular policies.
3. “There is no need in principle to include a separate function approximator for the state value, since it is related to the Q-function and policy according to Qθ (st , at ) − log πφ (at |st )” This is not true, since you rely on the fact that you have separate network parameters to get an unbiased gradient estimate in (10).
The experimental results also appear to have some correctness issues.
1. For the claim that the algorithm does better, this is also difficult to gauge because the graphs are unclear. In particular, it is not explained why the lines end early. How were multiple gradients incorporated into the update? Did you wait 4 (or 16) steps until computing a gradient update? This might explain why the lines end early, but then invalidates how the lines are drawn. Rather, the lines should be extended, where each point is plotted each 4 (or 16) steps. Doing this would remove the effect that the lines with 4 or 16 seem to learn faster (but really are just plotted on a different x-axis).
2. There is a lack of experimental details. This includes missing details about neural network architectures used by each algorithm, parameter tuning details, how multiple gradients are used, etc. This omission makes the experiments not reproducible.
3. Although DDPG is claimed to be very sensitive to parameter changes, and the proposed algorithm is more stable, there is no parameter sensitivity results showed.
Minor comments:
1. Graph font is much too small.
2. Typo in (10), should be V(s_{t+1})
3. Because the proof of Lemma 1 is so straightforward (just redefining reward), it would be better to actually spell it out, give a definition of entropy, etc.


Lifelong learning is the problem of learning multiple consecutive tasks in a sequential manner where knowledge gained from previous tasks is retained and used for future learning. It is essential towards the development of intelligent machines that can adapt to their surroundings. In this work we focus on a lifelong learning approach to generative modeling where we continuously incorporate newly observed streaming distributions into our learnt model. We do so through a student-teacher architecture which allows us to learn and preserve all the distributions seen so far without the need to retain the past data nor the past models. Through the introduction of a novel cross-model regularizer, the student model leverages the information learnt by the teacher, which acts as a summary of everything seen till now. The regularizer has the additional benefit of reducing the effect of catastrophic interference that appears when we learn over streaming data. We demonstrate its efficacy on streaming distributions as well as its ability to learn a common latent representation across a complex transfer learning scenario.

The paper proposed a teacher-student framework and a modified objective function to adapt VAE training to streaming data setting. The qualitative experimental result shows that the learned model can generate reasonable-looking samples. I'm not sure about what conclusion to make from the numerical result, as the test negative ELBO actually increased after decreasing initially. Why did it increase?
The modified objective function is a little ad-hoc, and it's unclear how to relate the overall objective function to Bayesian posterior inference (what exactly is the posterior that the encoder tries to approximate?). There is a term in the objective function that is synthetic data specific. Does that imply that the objective function is different depending on if the data is synthetic or real? What is the motivation/justification of choosing KL(Q_student||Q_teacher) as regularisation instead of the other way around? Would that make a difference in the goodness of the learned model? If not, wouldn't KL(Q_teacher||Q_student) result reduction in the variance of gradients and therefore a better choice?
Details on the minimum number of real samples per interval for the model to be able to learn is also missing. Also, how many synthetic samples per real samples are needed? How is the update with respect to synthetic sample scheduled? Given infinite amount of streaming data with a fixed number of classes/underlying distributions and interval length, and sample the class of each interval (uniformly) randomly, will the model/algorithm converge? Is there a minimum number of real examples that the student learner needs to see before it can be turned into a teacher?
Other question: How is the number of latent category J of the latent discrete distribution chosen?
Quality: The numerical experiment doesn't really compare to any other streaming benchmark and is a little unsatisfying. Without a streaming benchmark or a realistic motivating example in which the proposed scheme makes a significant difference, it's difficult to judge the contribution of this work.
Clarity: The manuscript is reasonably well-written. (minor: Paragraph 2, section 5, 'in principle' instead of 'in principal')
Originality: Average. The student-teacher framework by itself isn't novel. The modifications to the objective function appears to be novel as far as I am aware, but it doesn't require much special insights.
Significance: Below average. I think it will be very helpful if the authors can include a realistic motivating example where lifelong unsupervised learning is critical, and demonstrate that the proposed scheme makes a difference in the example.