We introduce an exploration bonus for deep reinforcement learning methods that is easy to implement and adds minimal overhead to the computation performed. The bonus is the error of a neural network predicting features of the observations given by a fixed randomly initialized neural network. We also introduce a method to flexibly combine intrinsic and extrinsic rewards. We find that the random network distillation (RND) bonus combined with this increased flexibility enables significant progress on several hard exploration Atari games. In particular we establish state of the art performance on Montezuma's Revenge, a game famously difficult for deep reinforcement learning methods. To the best of our knowledge, this is the first method that achieves better than average human performance on this game without using demonstrations or having access to the underlying state of the game, and occasionally completes the first level. This suggests that relatively simple methods that scale well can be sufficient to tackle challenging exploration problems.

The algorithm proposed in this paper consists in driving exploration in RL through an intrinsic reward, computed as the prediction error of a neural network whose target is the output of a randomly initialized network (with the state reached by the agent as input). The intuition is that rarely seen states will have a large prediction error, thus encouraging the agent to visit them (until they have been seen often enough that the error goes down). Among potential benefits of this method, compared to previously proposed intrinsic curiosity techniques for RL, are its simplicity and its robustness to environment stochasticity. Extensive experiments on the Atari game Montezuma’s Revenge investigate several variants of this idea (combined with PPO), with the best results significantly outperforming the current state-of-the-art. Other results on five other hard exploration Atari games show competitive performance as well.
The proposed technique definitely exhibits impressive performance on some tasks, in spite of its simplicity. Despite lacking theoretical grounding, I believe such results should be quite interesting to the RL research & applied community, as a novel and easy way to encourage exploration in sparse rewards tasks. I also really appreciate that the authors have included “negative” results contradicting their expectations, and are sharing their code: this is the kind of openness that in my opinion should be highly encouraged.
The paper is overall well written and easy to follow, except (from my point of view) section 2.2.2, which I found rather confusing and not very convincing. First, eq. 1 is a bit surprising since one expects the posterior to be in the same family of functions, i.e. of the form f_theta rather than f_theta + f_theta*. After a (very superficial) look at Osband et al (2018) I see that this particular lemma holds for linear functions, and the extension to nonlinear function approximation seems to be essentially based on intuition. Then the sentence “the optimization problem (...) is equivalent to distilling a randomly drawn function from the prior” ignores the sign mismatch (we are actually distilling the opposite of f_theta*, though I agree it can still make sense with a symmetric prior around 0, which is not mentioned). Finally, the reasoning to reach the conclusion “the distillation error could be seen as a quantification of uncertainty in predicting the constant zero function” seems somewhat unconvincing to me, considering the significant differences compared to Osband et al (2018), in particular: sharing weights among models in the ensemble, ignoring the specific regularization term R(theta), and not adding noise to the training data. As a result I find this link rather weak and I would appreciate if this section could be improved (at the very least with a better explanation of its limitations)
Among the various findings from experiments, one puzzled me in particular: the striking difference between episodic and non episodic intrinsic rewards in Fig. 3. I think this would have deserved a more thorough empirical investigation than the intuitive explanation from 2.3 (e.g. by checking whether the agent trained with non episodic rewards was indeed taking more risks and thus dying more often). What I find particularly surprising is that the beginning of the game should not yield much intrinsic reward relatively fast, since it should be the part the agent sees most often initially. As a result, I would expect that getting zero reward when dying (episodic rewards) should not be much different from getting future (small and discounted) intrinsic rewards, unless maybe early in training. What am I missing here?
I also have some comments regarding a couple of other findings and associated hypotheses:
- Section 3.3 shows some surprising results when varying discount factors (“This is at odds with the results in Figure 3 where increasing gamma_I did not significantly impact performance”). I wonder however to which extent these may be caused by the difference in the scale of discounted returns: for instance increasing gamma_I from 0.99 to 0.999 will (roughly) multiply V_I by 10, giving it more weight in the sum V = V_E + V_I. A fair comparison would either rescale V_I accordingly , or use a weighted sum and optimize the weights (the hyper-parameters table in the Appendix suggests that weights were actually used, but they are not mentioned in the main text and it is not clear how they were chosen).
- 3.7 shows an interesting behavior (“dancing with skulls”). The authors hypothetize it may be due to the inherent danger of such behavior. But could it be also (and possibly more) related to the fact the skulls are moving? (which leads to many varied different states, that the predictor network will take time to learn perfectly).
Here are a few more questions for the authors regarding specific details:
1. In 3.1, “The best return achieved by 4 out 5 runs of this setting was 6,700.” What does this mean?
2. In 3.5 a downsampling scheme is used to keep the training speed of the predictor network constant when increasing the number of actors. This raises the question of the impact of this training speed on the results, which is not investigated in the current experiments: do hyper-parameters influencing the predictor’s training speed (e.g. downsampling ratio, learning rate) need to be very carefully tuned, or are results robust across a wide range of speeds?
3. In A.5 there is mention of “a CNN policy with access to only the last 16 most recent frames”: does that mean the number of “frames stacked” (Table 2) was increased from 4 to 16? If so, why? (it is not clear to me what we learn compared to Fig. 4)
4. Your technique implicitly relies on the assumption that the predictor network’s weights will never be exactly the same as the target network’s (as otherwise nothing will be novel anymore, regardless of the states being visited). Do you foresee potential issues with this, and if yes do you have any idea to solve them? (a short discussion in the paper on this topic would be good as well)
And finally some suggestions for small improvements:
- Please try to find another name than “target” network since it is already widely used in the deep RL literature for something completely different (suggestions: “random”, “distillation”, “feature”, “reference”)
- In 2.1 (last paragraph) there are various papers cited regarding forward or inverse dynamics, but several of them contain both, while the way they are cited suggests they deal only with one. Just moving “and inverse dynamics” before the full list of citations would fix it.
- In first paragraph of section 3 please mention that the algorithm is based on PPO
- In Fig. 3 the x axis seems to be missing a multiplication by 1K (?)
- At end of 3.2, “having two value heads is necessary for combining reward streams with different characteristics”, please specify what are these characteristics.
- On p. 7, last paragraph: please (briefly) explain how the “random features” are computed
- The reference Ostrovski et al appears twice
- In Alg. 1, “Update reward normalization parameters using it”: the “s” in parameters can be misleading, suggesting that both mean and standard deviation are used for normalization => explicitly saying “Update running standard deviation” would avoid such confusion (or say it on the “Normalize” step below)
- Alg. 1 is not very clear on how returns and advantages are computed (and the corresponding code is not super easy to read). It also seems to be missing the update of the critic V.
- Alg. 1 mentions “number of optimization steps” while Table 4 says “Number of optimization epochs”: I guess they are the same, so they should probably have the same name
- After reading the paper, I felt like one learning was that CNN models worked better than RNN ones. However Table 5 shows that this can vary between games (ex: RND RNN outperforms RND CNN on Gravitar and Solaris) and/or algorithms (ex: PPO RNN outperforms PPO CNN on 3 games). I think the main text should at least point to this table when mentioning the superiority of the CNN.
- In the “Related work” section there is a very short paragraph about “vectorized value functions”. It seems to be overlooking the whole field of multi-objective reinforcement learning. Maybe you could cite a related survey paper like “A Survey of Multi-Objective Sequential Decision-Making”.
- The paper’s title and the OpenReview submission name should probably match
Update following author and reviewer discussion: I agree with others regarding the weakness of the empirical comparison to pseudo-counts in particular, but still believe that the paper deserves to be accepted due to the fact that (1) some of the results are really good, and (2) this is a simple original idea that has the potential to drive further advances (hopefully addressing the empirical and theoretical limitations of the current work)


We present an artificial intelligence research platform inspired by the human game genre of MMORPGs (Massively Multiplayer Online Role-Playing Games, a.k.a. MMOs). We demonstrate how this platform can be used to study behavior and learning in large populations of neural agents. Unlike currently popular game environments, our platform supports persistent environments, with variable number of agents, and open-ended task descriptions. The emergence of complex life on Earth is often attributed to the arms race that ensued from a huge number of organisms all competing for finite resources. Our platform aims to simulate this setting in microcosm: we conduct a series of experiments to test how large-scale multiagent competition can incentivize the development of skillful behavior. We find that population size magnifies the complexity of the behaviors that emerge and results in agents that out-compete agents trained in smaller populations.

Revised Review:
The authors of this work has taken my concerns, and concerns of other reviewers, and revised their paper during the rebuttal period. They have increased the quality of the writing / clarity, restructured the presentation (i.e. put many details in the Appendix section), and committed to open-sourcing the platform post publication. For these reasons I believe this work is now at a state that should be published at ICLR, and I revised my score from 5 to 7. I hope other reviewers can reread the work and post their updated comments.
I'm excited about the work, because it incorporates good ideas from A-Life / evolution / open-endedness communities, to introduce new paradigms and new ways of thinking to the RL community. I look forward to using this environment in my own research going forward, regardless of whether this work gets accepted or not. Good luck!
Minor comment: On page 4, the section 5 Experiments, I think "Technical details" should be in bold font before the sentence "We run each experiment using 100 worlds." so it is distinguished from being part of that sentence.
Original Review:
The authors present a new game environment inspired by MMORPGs. The environment supports a "massive" number of agents, each have different neural net brains (*) and have some foraging and combat skills. They use distributed RL to train the policies (using REINFORCE) and over time can observe the dynamics of the population of these artificial life agents interact with each other, where the only reward is survival. There are many interesting insights, such as looking at how multi-agent cooperative (and deceptive) strategies emerge, and how some agents with different niche skills co-evolve with agents with other niche skills. They also plan to open source the platform and I have high hopes that this will be a fantastic research environment. While I'm very optimistic about this work and direction, there are issues with this particular paper, and I feel it is not ready for publication in its current form. While I have no doubt that the software project will be great, as a reviewer I'm evaluating this particular paper, and I want to highlight flaws about the paper and what can be done to fix it during the rebuttal/review period.
My recommendations to improve the article:
(1) Writing - I really enjoyed this work, but frankly, the writing is horrible. It took me days of effort to decipher every paragraph and understand all the terms and what is going on. The article reads like it is written by the person who programmed the game, and played MMORPGs almost every day of his childhood and adult life, so someone who is not reading the article thru the lens of the author might have an incredibly tough time digesting the content. For instance, there are sentences like "It adds melee, ranged, and magic-based combat"... "Melee, ranged, and magic combat have maximum Manhattan distance of effect of 1, 2, and 3 cells respectively. They do 10, 2, and 1 damage respectively"... "This prevents uninteresting 'spawn killing' and is a common technique in human games". These are only a small selection of samples. There are also terminology like "#ent and #pop" which I feel should be replaced by $N_{ent}$ and $N_{pop}$ for a paper. In contrast, older works related to population-based RL training like [2], or RL in games like [3] are examples of clear and understandable writing. I highly recommend you give the draft to someone outside of your team, who is sufficiently isolated from this project (or perhaps to a professional writer if your lab has one), to go over each paragraph, and make the writing more clear. This would benefit the work in the long term as people refer back to the paper when they run your code.
(2) Diagrams - While the diagrams look interesting, IMO they are poorly made. When I look at Figure 1, 4, and 9, it is really difficult to understand what is going on. I recommend redoing the diagrams, perhaps get some inspiration from distill.pub or OpenAI blog posts. There are things that are not clear, like what the inputs are into each agent, and how the training works. I recommend having some pseudocode snippets (like the Gym framework) to explain parts of the overall picture in more detail as figures.
Given a work of this magnitude, I'm personally okay that they went over 8 pages, as long as it is properly used for clarity.
Discussion:
Concepts from Artificial Life and Evolution has been introduced in this work. There is some confusion between what is "learning" and what has been "evolved" in your setup. Some readers coming from the evolution, or biology fields (who I bet will find your paper interesting to read and experiment with) might interpret "learning" to be weight changes during a life time, while "evolution" would be changes to the weight parameters from one generation to the next, but I think in policy-gradient RL, "learning" means weight changes after an agent dies and is reborn. Should consider clarifying in the introduction, the definition of learning, and whether it is inter-life or intra-life.
You cited some of Stanley's talks on open-endedness, but I wonder if you considered their work [1] where they proposed that having a minimum criteria condition might encourage diversity of solutions. For instance, perhaps in your environment, an agent doesn't have to be the very best, but only manage to survive, to move on to the next generation, which might cause very interesting multi-agent population dynamics. A parallel to modern life is that people (at least those in wealthy nations) live with such a good social safety net that people don't really have to be the best "agent" to reproduce and survive, and this might explain the large diverse cultures and ideas we end up with as human species, compared to other animals (where the current game is probably a suitable model of). An experiment to explore an experiment where only the very weakest agents die, but leaving agents with mediocre foraging and combat skills still live on (and pursue their own interests, whatever they may be) will be super interesting, and I encourage you to explore these ideas of open-endedness.
Bugs: In the appendix, the citation for OpenAI Five needs fixing.
Currently it pains me that I can only assign a score of 5 of this work (NOTE: this has been since revised upwards to 7 upon reading revision after rebuttal period), since I don't think the current writing is up to standards. In my opinion, it deserves a score of 7-8. If you work on points (1) and (2) and submit a revised draft with much better writing, visualization, figures to explain the work, I'll happily revise my score and improve it by 1-3 points depending on how much improvement is made.
[1] Brand and Stanley. "Minimal Criterion Coevolution: A New Approach to Open-Ended Search" (GECCO 2017) http://eplex.cs.ucf.edu/papers/brant_gecco17.pdf
[2] https://arxiv.org/abs/1703.03864
[3] https://arxiv.org/abs/1804.03720
(*) well, sort of, due to compute limits they are clustered to some extent to their species, so agents within a species have identical brains, unlike the real world.


Recent work in network quantization has substantially reduced the time and space complexity of neural network inference, enabling their deployment on embedded and mobile devices with limited computational and memory resources. However, existing quantization methods often represent all weights and activations with the same precision (bit-width). In this paper, we explore a new dimension of the design space: quantizing different layers with different bit-widths. We formulate this problem as a neural architecture search problem and propose a novel differentiable neural architecture search (DNAS) framework to efficiently explore its exponential search space with gradient-based optimization. Experiments show we surpass the state-of-the-art compression of ResNet on CIFAR-10 and ImageNet. Our quantized models with 21.1x smaller model size or 103.9x lower computational cost can still outperform baseline quantized or even full precision models.

In this work the authors introduce a new method for neural architecture search (NAS) and use it in the context of network compression. Specifically, the NAS method is used to select the precision quantization of the weights at each layer of the neural network. Briefly, this is done by first defining a super network, which is a DAG where for each pair of nodes, the output node is the linear combination of the outputs of all possible operations (i.e., layers with different precision quantizations). Following [1], the weights of the linear combination are regarded as the probabilities of having certain operations (i.e., precision quantization), which allows for learning a probability distribution over the considered operations. Differently from [1], however, the authors bridge the soft sampling in [1] (where all operations are considered together but weighted accordingly to the corresponding probabilities) to a hard sampling (where a single operation is considered with the corresponding probability) through an annealing procedure based on the Gumbel Softmax technique. Through the proposed NAS algorithm, one can learn a probability distribution on the operations by minimizing a loss that accounts for both accuracy and model size. The final output of this search phase is a set of sampled architectures (containing a single operation at each connection between nodes), which are then retrained from scratch. In applications to CIFAR-10 and ImageNet, the authors achieve (and sometime surpass) state-of-the-art performance in model compression.
The two contributions of this work are
1)	A new approach to weight quantization using principles of NAS that is novel and promising;
2)	New insights/technical improvements in the broader field of NAS. While the utility of the method in the more general context of NAS has not been shown, this work will likely be of interest to the NAS community.
I only have one major concern. The architectures are sampled from the learnt probability distribution every certain number of epochs while training the supernet. Why? If we are learning the distribution, would not it make sense to sample all architectures only after training the supernet at our best?
This reasoning leads me to a second question. In the CIFAR-10 experiments, the authors sample 5 architecture every 10 epochs, which means 45 architectures (90 epochs were considered). This is a lot of architectures, which makes me wonder: how would a “cost-aware” random sampling perform with the same number of sampled architectures?
Also, I have some more questions/minor concerns:
1)	The authors say that the expectation of the loss function is not directly differentiable with respect to the architecture parameters because of the discrete random variable. For this reason, they introduce a Gumbel Softmax technique, which makes the mask soft, and thus the loss becomes differentiable with respect to the architecture parameters. However, subsequently in the manuscript, they write that Eq 6 provides an unbiased estimate for the gradients. Do they here refer to the gradients with respect to the weights ONLY? Could we say that the advantage of the Gumbel Softmax technique is two-fold? i) make the loss differentiable with respect to the arch parameters; ii) reduce the variance of the estimate of the loss gradients with respect to the network weights.
2)	Can the author discuss why the soft sampling procedure in [1] is not enough? I have an intuitive understanding of this, but I think this should be clearly discussed in the manuscript as this is a central aspect of the paper.
3)	The authors use a certain number of warmup steps to train the network weights without updating the architecture parameters to ensure that “the weights are sufficiently trained”. Can the authors discuss the choice on the number of warmup epochs?
I gave this paper a 5, but I am overall supportive. Happy to change my score if the authors can address my major concern.
[1] Liu H, Simonyan K, Yang Y. Darts: Differentiable architecture search. arXiv preprint arXiv:1806.09055. 2018 Jun 24.
-----------------------------------------------------------
Post-Rebuttal
---------------------------------------------------------
The authors have fully addressed my concerns. I changed the rating to a 7.


Generative models are important for several practical applications, from low level image processing tasks, to model-based planning in robotics. More generally, the study of generative models is motivated by the long-standing endeavor to model uncertainty and to discover structure by leveraging unlabeled data. Unfortunately, the lack of an ultimate task of interest has hindered progress in the field, as there is no established way to compare models and, often times, evaluation is based on mere visual inspection of samples drawn from such models. In this work, we aim at addressing this problem by introducing a new benchmark evaluation suite, dubbed GenEval. GenEval hosts a large array of distributions capturing many important properties of real datasets, yet in a controlled setting, such as lower intrinsic dimensionality, multi-modality, compositionality, independence and causal structure. Any model can be easily plugged for evaluation, provided it can generate samples. Our extensive evaluation suggests that different models have different strenghts, and that GenEval is a great tool to gain insights about how models and metrics work. We offer GenEval to the community 1 and believe that this benchmark will facilitate comparison and development of new generative models.

Overall, this is a thorough attempt at a system for evaluating various generative models on synthetic problems vaguely representative of the kinds of problems claimed to be covered by GANs. I think the approach and the conclusions drawn are mostly reasonable, with one major caveat discussed shortly.
I also think it would help in a revision to add evaluations of more recent successors to RealNVP, such as MAF (NIPS 2017, https://arxiv.org/abs/1705.07057 ), Glow ( https://arxiv.org/abs/1807.03039 ), and (although of course this paper came out concurrently with your submission) the promising FFJORD ( https://openreview.net/forum?id=rJxgknCcK7 ). The scale of comparison of GAN variants is also much smaller than that of Lucic et al. or their followup, Kurach et al. ( https://arxiv.org/abs/1807.04720 ), which is not cited here (and should be).
But primarily, I think there are some serious concerns with your choice of metrics that make the results as they are difficult to interpret.
"Note that OT is not a distance in the standard mathematical sense, as for instance the 'distance' between two sets of points sampled from the same distribution is not zero." -- You've confused some notions here. The Wasserstein-1 distance, which is a scalar times the variant of OT you use here, absolutely is a proper distance metric between distributions: W(P, Q) is a metric. But when you compute the OT distance between *samples*, OT(S, T) with S ~ P and T ~ Q, you're equivalently computing the distance W(\hat{P}, \hat{Q}) between the empirical distributions of the samples, \hat{P} = 1/N \sum_i \delta_{S_i} and the similar \hat{Q}, which of course are not the same thing as the source distributions themselves. You can, though, view OT(S, T) as an *estimator* of W(P, Q); the distance between *distributions* is what we actually care about.
It is well-known that these empirical distributions of samples \hat{P} converge to the true distribution P (in the Wasserstein sense, W(P, \hat{P})) exponentially slowly in the dimension, which is what your example about high-dimensional distributions demonstrates. Incidentally, this is exactly the example used in Arora et al. (ICML 2017, https://arxiv.org/abs/1703.00573 ). This means that, viewed as an estimator of the true distance between distributions, the empirical-distribution OT estimator is strongly biased. Thus it becomes very difficult to tell what the true OT value is at any sample size, and moreover this amount of bias might differ for different distribution pairs even at the same sample size, so *comparing* OT estimates at a fixed sample size is a tricky business. For example, in your Figure 2, when the "oracle" score is significantly more than zero, you know that all of your estimates are very strongly biased. There is not, as far as I know, any strong reason to suspect that this amount of bias should be comparable for different distribution pairs, making any conclusions drawn from these numbers suspect.
Your scheme you call "Two-Sample Test," first, should have a more specific name. Two-sample testing is an extremely broad field, with instances including the classical Kolmogorov-Smirnov test and t tests, the popular-in-ML kernel MMD-based tests, and even Wasserstein-based tests (e.g. https://arxiv.org/abs/1509.02237 ). Previous applications of these tests in GANs and generative models include Bounliphone et al. (ICLR 2016, https://arxiv.org/abs/1511.04581 ), Lopez-Paz and Oquab (2016 - which you cite without a venue but which was at ICLR 2017), Sutherland et al. (ICLR 2017, https://arxiv.org/abs/1611.04488 ), Huang et al. (2018), and more, using a variety of schemes. Your name for this should include "nearest neighbor" or something along those lines to avoid confusion.
Also, you call this an "extension of the original formulation," but in the common case where n(x) is more often right than wrong, your v is exactly \hat t - 1 of Lopez-Paz and Oquab; see their (2). If it's usually wrong, then v = 1 - \hat t; only when the signs differ per class does it significantly differ from theirs, and in any case I don't see a real motivation to put the absolute values for each class separately rather than just taking |\hat t - 1/2|.
Moreover, it's kind of crazy to term your v statistic a two-sample *test* -- you have nothing in there about its sampling distribution, which is key to hypothesis testing to obtain e.g. a p-value. (Maybe the variance of v is very different between different distributions; this is likely the case. In any case the variance will probably become extremely large as the dimension increases.) Comparing this score is thus difficult, but in any case calling it a "test" is potentially very misleading. You could, though, estimate the variance as described by Lopez-Paz and Oquab to construct a test.
Also: you can imagine the statistic v(S, T) as an estimator of the distance between distributions given as
  D(P, Q) = |1/2 - \int ( 1 if p(x) > q(x), 0 o.w.) p(x) dx|
          + |1/2 - \int (-1 if p(x) > q(x), 0 o.w.) q(x) dx|.
But v(S, T) is, like for the OT distance, a biased estimator of this distance, whose bias will get worse with the dimension. Thus, like with the OT, it's hard to meaningfully compare v(S, T) as an attempt to compare *distributions* based on D, which is what we actually care about. Here the oracle score does not show strong bias: assuming a reasonable number of samples, when P = Q the v estimator is always going be approximately 0. But this doesn't mean that other estimators aren't strongly biased, and indeed this is exactly what your Appendix C shows. The strong change in performance for KDE is somewhat hard to interpret, but maybe has something to do with the connection between KDE and NN-based methods?
Your log-likelihood score is an unbiased and asymptotically normal estimate of the true distribution score (the cross-entropy), so it's easy to compare. But it accounts only for a very small portion of comparing distributions.
There is at least one score in common use for this kind of evaluation with easy-to-compare estimators: the squared MMD. It has an easy-to-compute unbiased and asymptotically normal estimator, so it's easy to get confidence intervals for the true value between distributions at any sample size, making comparing the numbers based on a reasonable number of samples easy. There's also a well-devolped theory for how to construct p-values for a test if you want those; Bounliphone et al. above even developed a relative test to compare the MMDs of two models accounting for the correlations due to using the same "target" set, though if you use separate target sets (because you can easily sample more points from your synthetic distribution) then it's simpler. The choice of kernel does matter, but I think the median-heuristic Gaussian kernel would be a very reasonable score to add to your repertoire, and for particular distributions you also might be able to pick a better kernel (e.g. based on the causal factors when those exist). See also Binkowski et al. (ICLR 2018, https://arxiv.org/abs/1801.01401 ) for a detailed discussion of these issues in comparison to the FID score.
Using a metric whose estimation can be understood, and whose estimators can be reliably compared, is I think vital to any evaluation process. This also prevents issues like when RealNVP outperforms the oracle, which should be impossible with any proper evaluation metric.
Minor points:
- Why is Pedregosa et al. (2011) cited for fitting multivariate Gaussians by maximum likelihood? This is something that doesn't need a citation, especially not to scikit-learn, which doesn't even (I don't think) contain an implementation of fitting Gaussians beyond (np.mean(X, axis=0), np.cov(X, rowvar=False)).
- Mode coverage and related scores: this is based on assigning sample points to their single most likely clusters? I'd imagine that sometimes a model will output points far from any cluster, in which case the cluster that happens to be closest might happen to be the most likely, but it's strange to really count that point as part of that cluster for these scores. Or similarly, a point might be relatively evenly spaced between two clusters, in which case the assignment could be fairly arbitrary, again making these scores a little strange.


We introduce a bottleneck method for learning data representations based on channel deficiency, rather than the more traditional information sufficiency. A variational upper bound allows us to implement this method efficiently. The bound itself is bounded above by the variational information bottleneck objective, and the two methods coincide in the regime of single-shot Monte Carlo approximations. The notion of deficiency provides a principled way of approximating complicated channels by relatively simpler ones. The deficiency of one channel w.r.t. another has an operational interpretation in terms of the optimal risk gap of decision problems, capturing classification as a special case. Unsupervised generalizations are possible, such as the deficiency autoencoder, which can also be formulated in a variational form. Experiments demonstrate that the deficiency bottleneck can provide advantages in terms of minimal sufficiency as measured by information bottleneck curves, while retaining a good test performance in classification and reconstruction tasks.

This paper used the concept based on channel deficiency to derive a variational bound similar to variational information bottleneck. Theoretical analysis shows that this bound is an lower bound on the VIB objective. The empirical analysis shows it outperforms VIB in some sense.
I think this paper's contribution is rather theoretical than practical. The experiments section can be improved in the following aspect:
-  Figure 2 are hard to read for different M's. It would be better if the authors can show the exact accuracy numbers rather than the overlapped lines
- I(Z;Y) vs I(Z;X) graph is typically used in a VIB setting. In the paper's variational deficiency setting, although plotting I(Z;Y) vs I(Z;X) is necessary, it would be also helpful for the authors' to plot Deficiency vs I(Z;X), because this is what new objective is trading-off.
- Again, Figure 3, it is hard to see the benefits for increasing M from the visualizations for different clusterings.
- How do the paper estimate I(Z;Y) and I(Z;X) for plotting these figures? Does the paper use lower bound or some estimators? It should be made clear in the paper since these are non-trivial estimations.
Last comment is that, although the concept of `deficiency` in a bottleneck setting is novel, the similar idea for tighter bound of log likelihood has already been pursed in the following paper:
- Yuri Burda, Roger Grosse, Ruslan Salakhutdinov. Importance Weighted Autoencoders. ICLR 2016
It was kind of surprising that the authors did not cite this paper given the results are pretty much the same. It would also be helpful for the authors to do a comparison or connection section with this paper.
I like the paper in general, but given it still has some space for improvement, I would keep my decision as boarder line for now.


MULTIPLE-ATTRIBUTE TEXT REWRITING
The dominant approach to unsupervised "style transfer" in text is based on the idea of learning a latent representation, which is independent of the attributes specifying its "style". In this paper, we show that this condition is not necessary and is not always met in practice, even with domain adversarial training that explicitly aims at learning such disentangled representations. We thus propose a new model that controls several factors of variation in textual data where this condition on disentanglement is replaced with a simpler mechanism based on back-translation. Our method allows control over multiple attributes, like gender, sentiment, product type, etc., and a more fine-grained control on the trade-off between content preservation and change of style with a pooling operator in the latent space. Our experiments demonstrate that the fully entangled model produces better generations, even when tested on new and more challenging benchmarks comprising reviews with multiple sentences and multiple attributes.

This work proposes a new model that controls several factors of variation in textual data where the condition on disentanglement is replaced with a simpler mechanism based on back-translation. It allows control over multiple attributes, and a more fine-grained control on the trade-off between content preservation and change of style with a pooling operator in the latent space.
One of the major arguments is it is unnecessary to have attribute-disentangled latent representations in order to have good style-transferring rewriting. In Table 2, the authors showed that "a classifier that is separately trained on the resulting encoder representations has an easy time recovering the sentiment" when the discriminator during training has been fooled. Is there any difference between the two discriminators/classifiers? If the post-fit classifier on top of the encoder representation can easily predict the correct sentiment, there should be enough signal from the discriminator to adapt the encoder in order to learn a more disentangled representation. On the other hand, this does not answer the question if a "true" disentangled representation would give better performance. The inferior performance from the adversarially learned models could be because of the "entangled" representations.
As the author pointed out, the technical contributions are the pooling operator and the support for multiple attributes since the loss function is the same as that in (Lample et. al 2018). These deserve more elaborated explanation and quantitative comparisons. After all, the title of this work is "multiple-attribute text rewriting". For example, the performance comparison between the proposed how averaged attribute  embeddings and simple concatenation, and the effect of the introduced trade-off using temporal max-pooling.
How important is the denoising autoencoder loss in the loss function (1)? From the training details in the supplementary material, it seems like the autoencoder loss is used as "initialization" to some degree. As pointed out by the authors, the main task is to get fluent, attribute-targeted, and content-preserving rewriting. As long as the "back-translation" gives expected result, it seems not necessary to have "meaningful" or hard "content-preserving" latent representations when the generator is powerful enough.
I think the last and most critical question is what the expected style-transferred rewriting look like. What level or kind of "content-preserving" do we look for? In Table 4, it shows that the BLEU between the input and the referenced human rewriting is only 30.6 which suggest many contents have been modified besides the positive/negative attribute. This can also be seen from the transferred examples. In Table 8, one of the Male example: "good food. my wife and i always enjoy coming here for dinner. i recommend india garden." and the Female transferred rewriting goes as "good food. my husband and i always stop by here for lunch. i recommend the veggie burrito". It's understandable that men and women prefer different types of food even though it is imagination without providing context. But the transfer from "dinner" to "lunch" is kind of questionable. Is it necessary to change the content which is irrelevant to the attributes?
Other issues:
- Towards the end of Section 3, it says that "without back-propagating through the back-translation generation process". Can you elaborate on this and the reason behind this choice?
- What does it mean by "unknown words" in "... with 60k BPE codes, eliminating the presence of unknown words" from Section 4?
- There is no comparison with (Zhang et. al. 2018), which is the "most relevant work".
- In Table 4, what is the difference among the three "Ours" model?
- In Table 4, the perplexity of "Input Copy" is very high compared with generated sentences.
- In Table 7, what does the "attention" refer to?
- In the supplementary material, there are lambda_BT and lambda_AE. But there is only one lambda in the loss function (1).
- Please unify the citation style.


Embeddings are a fundamental component of many modern machine learning and natural language processing models. Understanding them and visualizing them is essential for gathering insights about the information they capture and the behavior of the models. State of the art in analyzing embeddings consists in projecting them in two-dimensional planes without any interpretable semantics associated to the axes of the projection, which makes detailed analyses and comparison among multiple sets of embeddings challenging. In this work, we propose to use explicit axes defined as algebraic formulae over embeddings to project them into a lower dimensional, but semantically meaningful subspace, as a simple yet effective analysis and visualization methodology. This methodology assigns an interpretable semantics to the measures of variability and the axes of visualizations, allowing for both comparisons among different sets of embeddings and fine-grained inspection of the embedding spaces. We demonstrate the power of the proposed methodology through a series of case studies that make use of visualizations constructed around the underlying methodology and through a user study. The results show how the methodology is effective at providing more profound insights than classical projection methods and how it is widely applicable to many other use cases.

The idea of analyzing embedding spaces in a non-parametric (example-based) way is well-motivated. However, the main technical contribution of this paper is otherwise not clear - the methodology section covers a very broad set of techniques but doesn't provide a clear picture of what is novel; furthermore, it makes a strong assumption about linear structure in the embedding space that may not hold. (It's worth noting that t-SNE does not make this assumption.)
The visualization strategies presented don't appear to be particularly novel. In particular, projection onto a linear subspace defined by particular attributes was done in the original word2vec and GloVe papers for the analogy task. There's also a lot of other literature on interpreting deeper models using locally-linear predictors, see for example LIME (Ribeiro et al. 2016) or TCAV (Kim at el. 2018).
Evaluations are exclusively qualitative, which is disappointing because there are quantitative ways of evaluating a projection - for example, how well do the reduced dimensions predict a particular attribute relative to the entire vector. Five-axis polar plots can pack in more information than a 2-dimensional plot in some ways, but quickly become cluttered. The authors might consider using heatmaps or bar plots, as are commonly used elsewhere in the literature (e.g. for visualizing activation maps or attention vectors).
User study is hard to evaluate. What were the specific formulae used in the comparison? Did subjects just see a list of nearest-neighbors, or did they see the 2D projection? If the latter, I'd imagine it would be easy to tell which was the t-SNE plot, since most researchers are familiar with how these look.


Recent work has introduced attacks that extract the architecture information of deep neural networks (DNN), as this knowledge enhances an adversary's capability to conduct attacks on black-box networks. This paper presents the first in-depth security analysis of DNN fingerprinting attacks that exploit cache side-channels. First, we define the threat model for these attacks: our adversary does not need the ability to query the victim model; instead, she runs a co-located process on the host machine where the victim's deep learning (DL) system is running and passively monitors the accesses of the target functions in the shared framework. Second, we introduce DeepRecon, an attack that reconstructs the architecture of the victim network using the internal information extracted via Flush+Reload, a cache side-channel technique. Once the attacker observes function invocations that map directly to architecture attributes of the victim network, the attacker can reconstruct the victim's entire network architecture. In our evaluation, we demonstrate that an attacker can accurately reconstruct two complex networks (VGG19 and ResNet50) having observed only one forward propagation. Based on the extracted architecture attributes, we also demonstrate that an attacker can build a meta-model that accurately fingerprints the architecture and family of the pretrained model in a transfer learning setting. From this meta-model, we evaluate the importance of the observed attributes in the fingerprinting process. Third, we propose and evaluate new framework-level defense techniques that obfuscate our attacker's observations. Our empirical security analysis represents a step toward understanding DNNs' vulnerability to cache side-channel attacks.

This paper considers the problem of fingerprinting neural network architectures using cache side channels. In the considered threat model, the attacker runs a process co-located with the victim's, and uses standard FLUSH+RELOAD attacks to infer high-level architectural information such as the number and types of layers of the victim's ML model. The paper concludes with the discussion of some "security-through-obscurity" defenses.
I don't quite understand the threat model considered in this paper. The main motivating factor given by the authors for uncovering model architecture details is for facilitating black-box attacks against ML models (e.g., for adversarial examples or membership inference).
Yet, in the case of adversarial examples for instance, knowledge of the architecture is often considered a given as keeping it secret has very little influence on attacks. There are black-box attacks that require no knowledge of the architecture and only a few queries (e.g., Black-box Adversarial Attacks with Limited Queries and Information, Ilyas et al., ICML'18).
So overall, learning such coarse-grained features about a model just doesn't seem particularly useful, especially since architecture-level details are often not considered private or secret to begin with.
After architectural details have been extracted, the end-goal attacks on ML models considered by the authors (e.g., model stealing, adversarial examples, etc.) require query access anyways. Thus, additionally assuming co-location between the adversary and the victim's model seems to unnecessarily strengthen the attacker model.
Maybe the most interesting scenario to consider for cache side-channels in ML is when ML models are run on trusted hardware (e.g., Oblivious Multi-Party Machine Learning on Trusted Processors, Ohrimenko et al.; or this work also submitted to ICLR: https://openreview.net/forum?id=rJVorjCcKQ).
Cache side channels are much more relevant to that threat model (i.e., ML code running in a trusted hardware enclave hosted by a malicious party). And indeed, there have been many cache side-channel attack papers against trusted hardware such as Intel's SGX (e.g., Software Grand Exposure: SGX Cache Attacks Are Practical, Brasser et al.)
But given what we know about the strength of these cache side channel attacks, one would expect to be able to extract much more interesting information about a target model, such as its weights, inputs or outputs. In the above trusted hardware scenario, solely extracting architecture-level information would also not be considered a very strong attack, especially since coarse-grained information (e.g., a rough bound on the number of layers), can be trivially obtained via timing side channels.
Minor comments:
- In the introduction, you say that white-box attacks for adversarial examples are rendered ineffective by gradient masking. This isn't true in general. Only "weak" white-box attacks can be rendered ineffective this way. So far, there are no examples of models that resist white-box attacks yet are vulnerable to black-box attacks.
- What exactly causes the cache-level differences you observe? Can you give some  code examples in the paper that showcase what happens? Are the TensorFlow code lines listed in Table 1 from a specific commit or release?
- The defenses discussed in Section 5 are all forms of "security through obscurity" that seem easily defeated by a determined attacker that adapts its attack (and maybe uses a few additional observations).
--REVISION--
I thank the authors for their rebuttal and clarifications on the threat model and end goals of their attacks. I remain somewhat unconvinced by the usefulness of extracting architectural information. For most of the listed attacks (e.g., building substitute models for adversarial examples, or simply for model extraction) it is not clear from prior work that knowledge of the architecture is really necessary, although it is of course always helpful to have this knowledge. As I mentioned in my review, with current (undefended) ML libraries, it should be possible to extract much more information (e.g., layer weights) using cache side channels.


DEEP GRAPH INFOMAX
We present Deep Graph Infomax (DGI), a general approach for learning node representations within graph-structured data in an unsupervised manner. DGI relies on maximizing mutual information between patch representations and corresponding high-level summaries of graphs-both derived using established graph convolutional network architectures. The learnt patch representations summarize subgraphs centered around nodes of interest, and can thus be reused for downstream node-wise learning tasks. In contrast to most prior approaches to unsupervised learning with GCNs, DGI does not rely on random walk objectives, and is readily applicable to both transductive and inductive learning setups. We demonstrate competitive performance on a variety of node classification benchmarks, which at times even exceeds the performance of supervised learning.
encoder models based on graph convolutions (Gilmer et al., 2017), it is unclear whether randomwalk objectives actually provide any useful signal, as these encoders already enforce an inductive bias that neighboring nodes have similar representations.
In this work, we propose an alternative objective for unsupervised graph learning that is based upon mutual information, rather than random walks. Recently, scalable estimation of mutual information was made both possible and practical through Mutual Information Neural Estimation (MINE, Belghazi et al., 2018), which relies on training a statistics network as a classifier of samples coming from the joint distribution of two random variables and their product of marginals. Following on MINE,  introduced Deep InfoMax (DIM) for learning representations of highdimensional data. DIM trains an encoder model to maximize the mutual information between a high-level "global" representation and "local" parts of the input (such as patches of an image). This encourages the encoder to carry the type of information that is present in all locations (and thus are globally relevant), such as would be the case of a class label. DIM relies heavily on convolutional neural network structure in the context of image data, and to our knowledge, no work has applied mutual information maximization to graph-structured inputs. Here, we adapt ideas from DIM to the graph domain, which can be thought of as having a more general type of structure than the ones captured by convolutional neural networks. In the following sections, we introduce our method called Deep Graph Infomax (DGI). We demonstrate that the representation learned by DGI is consistently competitive on both transductive and inductive classification tasks, often outperforming both supervised and unsupervised strong baselines in our experiments.

This paper adapts the Deep Informax (DIM; Hjelm et al. 2018) method, which was used on
image data, into the graph domain. The architecture of the neural network and
the learning cost function are given by figure 1 and eq.(1), respectively.
The idea is to maximize the mutual information between a local representation
(of a "patch" defined by graph adjacency) and a global representation (of the entire graph),
so those different local patches are encouraged to carry some shared
global information.
This is in contrast to most unsupervised graph encoders, where the objective is
to fit the random walk similarities (node adjacency on the graph).
In an unsupervised learning scenario, where the graph structure and node features
are given, the authors achieved state-of-the-art performance on transductive and
inductive node classification tasks, in some cases even better than supervised baselines.
The paper is well written. I recommend acceptance and have the following concerns.
Main comment 1
The title suggests that there are some information theory contents.
However, section 3 does not include much information theory.
Rather, the author(s) directly give eq.(1) with pointers to references and informal discussions.
This is not so helpful. It is not straightforward for
the reader to relate eq.(1) with the definition of mutual information.
Ideally, before eq.(1) there should be one or two equations (with text)
to introduce the Jesen-Shannon MI estimation and information theoretic bounds etc.
Overall, due to this, the contribution is mainly on adapting the DIM method info the graph domain. Although the experimental results are good, there is not much theoretical insight or "recreative" introduction of the DIM method from the authors' perspectives. This is the main reason for that it is not a strong accept.
Main comment 2
A motivation of the proposition is to "not rely on random walks", or graph node adjacency.
Notice that random walks can be intuitively regarded as higher order node adjacency.
However, the encoder, which is based on GCN, does rely on the adjacency matrix,
as the convolution is done in local neighborhoods (that can also be defined based on
random-walk similarities). The authors are therefore suggested to make it
clear in related places that, it is the cost function which is not based
on node adjacency, although the neural network structure does rely on it.
As a related question, in the inductive experiments, in the mini-batch of 256 nodes
randomly selected, or selected by a local patch of the graph which is connected or nearby?
If it is the latter case, the cost function does rely on random-walk similarities,
as the summary vector will be a local patch average.
Questions:
-The summary vector is the average of all node features. On large graphs, the
average may carry less information as compared to small graphs. It can be
observed that on Pubmed and Reddit, the performance improvement is not as
high as the other small graphs. Could you comment on this?
-In the baseline "DeepWalk+features", are the two different types of features directly concatenated?
-Is it straightforward to apply DGI to link prediction tasks?
-It that a concern that the random corruption function will cause a high variance of the gradient?


Machine translation is an important real-world application, and neural networkbased AutoRegressive Translation (ART) models have achieved very promising accuracy. Due to the unparallelizable nature of the autoregressive factorization, ART models have to generate tokens one by one during decoding and thus suffer from high inference latency. Recently, Non-AutoRegressive Translation (NART) models were proposed to reduce the inference time. However, they could only achieve inferior accuracy compared with ART models. To improve the accuracy of NART models, in this paper, we propose to leverage the hints from a welltrained ART model to train the NART model. We define two hints for the machine translation task: hints from hidden states and hints from word alignments, and use such hints to regularize the optimization of NART models. Experimental results show that the NART model trained with hints could achieve significantly better translation performance than previous NART models on several tasks. In particular, for the WMT14 En-De and De-En task, we obtain BLEU scores of 25.20 and 29.52 respectively, which largely outperforms the previous non-autoregressive baselines. It is even comparable to a strong LSTM-based ART model (24.60 on WMT14 En-De), but one order of magnitude faster in inference.

This work proposes a non-autoregressive Neural Machine Translation model which the authors call NART, as opposed to an autoregressive model which is referred to as an ART model. The main idea behind this work is to leverage a well trained ART model to inform the hidden states and the word alignment of NART models. The joint distribution of the targets y given the inputs x, is factorized into two components as in previous works on non-autoregressive MT: an intermediate z which is first predicted from x, which captures the autoregressive part, while the prediction of y given z is non-autoregressive. This is the approach taken e.g., in Gu et al, Kaiser et al, Roy et al., and this also seems to be the approach of this work.  The authors argue that improving the expressiveness of z (as was done in Kaiser et al, Roy et al), is expensive and so the authors propose a simple formulation for z. In particular, z is a sequence of the same length as the targets, where the j^{th} entry z_j is a weighted sum of the embedding of the inputs x (the weights depend in a deterministic fashion on j) . Given this z, the model predicts the targets completely non-autoregressively. However, this by itself is not entirely sufficient, and so the authors also utilize "hints": 1) If the pairwise cosine similarity between two successive hidden states in the student  NART model is above a certain threshold, while the similarity is lower than another threshold in the ART model, then the NART model incurs a cost proportional to this similarity 2) A KL term is used to encourage the distribution of attention weights of the student ART model to match that of the teacher NART model. These two loss terms are used in different proportions (using additional hyperparameters) together with maximizing the likelihood term.
Quality: The paper is not very well written and is often hard to follow in parts. Here are some examples of the writing that feel awkward:
--  Consequently, people start to develop Non-AutoRegressive neural machine
Translation (NART) models to speed up the inference process (Gu et al., 2017; Kaiser et al., 2018;
Lee et al., 2018).
-- In order to speed up to the inference process, a line of works begin to develop non-autoregressive
translation models.
Originality: The idea of using an autoregressive teacher model to improve a non-autoregressive translation model has been used in Gu et al., Roy et al., where knowledge distillation is used. So knowledge distillation paper from Hinton et al., should be cited. Moreover, the authors have missed comparing their work to that of Roy et al. (https://arxiv.org/abs/1805.11063), which greatly improves on the work of Kaiser et al., and almost closes the gap between a non-autoregressive model and an autoregressive model (26.7 BLEU vs 27 BLEU on En-De) while being orders of magnitude faster. So it is not true that:
-- "While the NART models achieve significant speedup during inference (Gu et al., 2017), their accuracy
is considerably lower than their ART counterpart."
-- "Non-autoregressive translation (NART) models have suffered from low-quality translation results"
Significance: The work introduces the idea of using hints for non-autoregressive machine translation. However, I have a technical concern: It seems that the authors complain that previous works like Kaiser et al, Roy et al, use sophisticated submodules to help the expressiveness of z and this was the cause for slowness. However, the way the authors define z seems to have some problems:
- z_j does not depend on z_1, ..., z_{j-1}, so where is the autoregressive dependencies being captured?
- z_1, z_2, ..., z_{T_y} depends only on the length of y, and does not depend on y in any other way. Given x, predicting z is trivial and I don't see why that should help the model f(y | z, x) help at all?
- Given such a trivial z, one can just assume that your model is completely factorial i.e. P(y|x) = \prod_{i} P(y_i|x) since the intermediate z has no information on the y's except it's length.
This is quite suspicious to me, and it seems that if this works, then a completely factorial model should work as well if we only use the "hints" from the ART teacher model. This is a red flag to me, and I am finding this hard to believe.


Shallow supervised 1-hidden layer neural networks have a number of favorable properties that make them easier to interpret, analyze, and optimize than their deep counterparts, but lack their representational power. Here we use 1-hidden layer learning problems to sequentially build deep networks layer by layer, which can inherit properties from shallow networks. Contrary to previous approaches using shallow networks, we focus on problems where deep learning is reported as critical for success. We thus study CNNs on image recognition tasks using the large-scale ImageNet dataset and the CIFAR-10 dataset. Using a simple set of ideas for architecture and training we find that solving sequential 1-hidden-layer auxiliary problems leads to a CNN that exceeds AlexNet performance on ImageNet. Extending our training methodology to construct individual layers by solving 2-and-3-hidden layer auxiliary problems, we obtain an 11-layer network that exceeds VGG-11 on ImageNet obtaining 89.8% top-5 single crop. To our knowledge, this is the first competitive alternative to end-to-end training of CNNs that can scale to ImageNet. We conduct a wide range of experiments to study the properties this induces on the intermediate layers.

Summary:
This paper proposes layer wise training of neural networks using classification auxiliary tasks for training each layer. Experiments are presented on CIFAR10 and Imagenet. Accuracies close to end to end training are obtained.
The layer wise training is repeated for J steps, the auxiliary tasks are trained on top of the shallow one layer (of width M ) with a network  of depth k and width tilde{M}. Layerwise training is done using sgd with momentum, and the learning rate is decayed through epochs.
Note that the layer wise training is done with large width M than typical end to end networks in use.
The authors argue and test the hypothesis that auxiliary tasks  encourage the linear separability of CNN features.
To reduce the size of the learned network the author propose a layer wise compression using the filter removal technique of Molchanov et al .
Reproducibility:
This empirical  work has been investigated for a while with mild success, the authors should make their code available to the community to confirm and reproduce  their findings.  I encourage the authors to make their code available during the review/discussion period.
Significance of the work:
From reading the paper it is not clear what is the main ingredient that makes this layer wise training  successful, negative results would help in understanding what is important for the accuracy.
Some more ablation studies and negative results will be insightful and here are few suggestions in that direction:
- Authors claim that they used invertible downsampling as max or average pooling  lead to information loss. Does the layer wise training give worst results with average or max pooling? If so please report those numbers to know what is the implications of this choice of pooling.
- On the width of the networks, seems it is key for the success of the approach.  What if  you train wider networks with J that is small? (  J=3 for instance but much  wider networks, instead of J=8 now for imagenet.)
- To answer the same question above one needs also to see what are the accuracies for J=8 with thiner networks (smaller M )?
- Would the accuracy  with the layer wise training reach a  plateau if one uses an architecture with J higher than 8?
- Transferability of the learned features: end to end features are know to be transferable. It would be good to see if this still holds using the network layer wise trained on imagenet for CIFAR10 or other datasets.
Other Questions:
- Section 3.2 is vague. In Proposition 3.1 and  Proposition 3.2 can you add some text to explain what are the implicitions of the claims? “Thus our training permits to extend some results on shallow CNN to deeper CNNs …” which shallow results ?
- “For k>1 batch normalization was useful “ is this only on the auxiliary problems networks  or you used also batch norm for the layer ?
- The ensemble used is Z=\sum_{j=1}^J 2^j z_j , this uses the network of J layers ,  also the O(J) auxiliary networks  of depth k. Please report the number of parameters for all models (single and ensembles) in Table 1 and Table 2.
- In the conclusion: “The framework can be extendable to the parallel training …” how would this possible since one needs the output of the first training to do the training of the next layer. can you elaborate on what is meant here?
Minor:
page 2 bottom have competitorsand -> have competitors and
the non linearity rho in equation 1 and throughout the paper put a bracket for its argument \rho(x) not \rho x
Page 6 , Imagenet paragraph : W —> We
section 4.2 we define linear separability etc… a space is missing before Further
section 4.3 we report our results .. (Imagenet) a space is missing after ImageNet)
Overall Assessment:
This is a good paper, making the code available and adding more ablation studies and explanations of width versus depth and the choice of pooling will make the contributions easier to understand.


New estimates for the generalization error are established for a nonlinear regression problem using a two-layer neural network model. These new estimates are a priori in nature in the sense that the bounds depend only on some norms of the underlying functions to be fitted, not the parameters in the model. In contrast, most existing results for neural networks are a posteriori in nature in the sense that the bounds depend on some norms of the model parameters. The error rates are comparable to that of the Monte Carlo method in terms of the size of the dataset. Moreover, these bounds are equally effective in the over-parametrized regime when the network size is much larger than the size of the dataset.

This paper studies the generalization properties of a two layer neural network for a nonlinear regression problem where the target function has a finite spectral norm. The generalization bound comprises of an approximation term (dependent on the width) and an estimation term (dependent on spectral norm of the target and scaling as 1/sqrt{n}).
The key contribution is the derivation of the generalization bound where the estimation term depends on the properties of the target function rather than properties of the class of two layer neural networks. These bounds are instantiated for a class of regularized estimators (with path norm regularization).
1. Theoretical Novelty: While both Theorem 3 (approximation) and Theorem 4 (a posterior generalization) were mostly following known results, the key development seems to be the bound on the path norm of the regularized solution in terms of the spectral norm of the target function. Given that the estimator is a path-norm regularized estimator, this seemed to be an incremental contribution. What would be more interesting is to obtain such a bound for an unregularized estimator: either saying something about the optimization procedure or relating this kind of regularization to properties of the dataset over which it is trained.
2. Regression vs Classification: While the focus of the paper is on a regression problem, the experiments and problem motivation seems to arise from a classification setting. This creates a mismatch between the what the paper is about and the problem that has been motivated. Would it be possible to extend these results to loss functions (other than squared loss ) like cross-entropy loss or hinge loss which indeed work in the classification setting?
3. Comparison with Klusowski & Barron (2016): In the comparison section, it is mentioned that  Klusowski & Barron (2016) analyze a "similar" problem and obtain worse generalization bounds. It would be important to know the exact setting in which they obtained their bounds and how do their assumptions compare with the ones made in this paper. The comparison seems incomplete without this.
4. The experiments showcase that the regularized estimator has a better path norm (and expectedly so) but almost similar (in case of MNIST actually better) test accuracy. This defeats the purpose of  showing that the regularized estimator has better generalization ability which is claimed in the introduction as well as the experiment section (calling it "well-posed"). What this indeed shows is that even though the path norm might be big, the generalization of the estimator is till very good contradicting the statements made.
5. The numbers shown in Figure 1 and the numbers reported in Table 2 do no match: while the plot shows that the scaled path norm is around 60 for both MNIST and CIFAR-10, the corresponding numbers in the table are 507 and 162. Can you please point out the reason for this discrepancy?
6. Theorem 5 seems to suggest that in the noiseless case, the estimation error would scale as the spectral norm of f^*. Rather, in the noiseless setting, it seems that the correct scaling of the generalization error should be with respect to properties of the regularized estimator and the function class. Even though the spectral norm can be arbitrarily high, the generalization bound should only be dependent on the complexity of functions which can fit the current data well. It would be good to have a comment in the draft on why the current dependence is a better thing and examples where such generalization bounds are indeed better.


A NEW DOG LEARNS OLD TRICKS: RL FINDS CLASSIC OPTIMIZATION ALGORITHMS
We ask whether reinforcement learning can find theoretically optimal algorithms for online optimization problems, and introduce a novel learning framework in this setting. To answer this question, we introduce a number of key ideas from traditional algorithms and complexity theory. Specifically, we introduce the concept of adversarial distributions (universal and high-entropy training sets), which are distributions that encourage the learner to find algorithms that work well in the worst case. We test our new ideas on the AdWords problem, the online knapsack problem, and the secretary problem. Our results indicate that the models have learned behaviours that are consistent with the optimal algorithms for these problems derived using the online primal-dual framework.

The overall goal of this paper is to solve online combinatorial optimization (OCO) problems using reinforcement learning. Importantly, the authors are not seeking to establish new results for unsolved problems, but instead they are motivated by analyzing and comparing the quality of solutions predicted by reinforcement learners with respect to the well-known near-optimal strategies for some OCO tasks. In doing so, the authors focused on an MDP framework using policy gradient and DQN methods. This framework was trained on three OCO tasks; online budget allocation, online knapsack, and the secretary problem. For each, the trained model is consistent with the near-optimal “handcrafted’’ algorithms.
The idea of checking whether a standard RL framework, without prior information about “how to” solve a given OCO task, is capable from experience to reach the performance of existing optimal strategies (especially primal-dual approaches), is clearly interesting. But I am not entirely convinced that the paper is making novel contributions in this direction. My comments are detailed below:
(1) OCO problems have been a subject of extensive research in online learning (see e.g. [1,2,3]). Notably, the main issues related to “input length independence” (Sec 1.1) and “adversarially chosen input distributions” (Sec 1.2) have already been addressed in online learning frameworks. Input length independence is related to “horizon-independence” in online learning (the number $T$ of trials is not known in advance), and well-known approaches have been developed for devising horizon-independent forecasters, or transforming a horizon-dependent forecaster into a horizon-independent one (see e.g. [4]). Also, the study of online learners with respect to different properties of input sequences (stochastic, adversarial, or hybrid), is a well-known topic of research which have been conceptualized and formalized with appropriate metrics (see e.g. [5,6]).
(2) Although the authors are interested in making connection between RL and “primal-dual approaches” in online learning, this connection was not made clear in the paper. Namely, the overall contribution was to show that deep RL architectures can compete with existing, handcrafted online strategies, on three specific tasks. But in order to make a concrete connection with primal-dual approaches, this study should be extended to more general primal (covering) and dual (packing) problems as described in [7], and the RL framework should be compared with “generic” online primal-dual algorithms also described in [7]. We may note in passing that the offline versions of the three tasks examined in the present paper belong to the approximation classes PTAS or APX. By contrast, the complexity class of general covering/packing problems is much higher (a constant approximation ratio is not achievable unless P=NP) and, to this point, it would be interesting to examine whether a standard deep RL framework can compete with existing online strategies (for example in [7,8]) on such hard problems.
(3) Even if we stick to the three problems examined in the paper, the neural nets vary between tasks, with different numbers of layers, different widths, different batch sizes, etc. On the one hand, it is legitimate to seek for an appropriate learning architecture for the input problem. On the other hand, such adjustments are conveying some prior knowledge about “how to” solve this problem using a deep RL model. Moreover, for knapsack and secretary tasks, additional knowledge about the history (i.e. state augmentation) is required for establishing convergence, but the resulting model is no longer a standard MDP. So, unless I missed the point about the overall goal of this paper, these different aspects are somewhat in contradiction with the idea of starting with a deep RL architecture with default settings, in which the varying components are essentially the states, transitions, and rewards of the MDP that encodes the problem description.
[1] Bubeck, S., Introduction to Online Optimization. Lecture Notes, Princeton University, 2011.
[2] Audibert, J-Y., Bubeck, S., and Lugosi, G. Minimax policies for combinatorial prediction games. In COLT, 2011.
[3] Rajkumar, A. and Agarwal, S. Online decision-making in general combinatorial spaces. In NIPS, 2014.
[4] N. Cesa-Bianchi and G. Lugosi. Prediction, Learning, and Games. Cambridge University Press, 2006.
[5] A. Rakhlin, K. Sridharan, and A. Tewari. Online learning: Stochastic, constrained, and smoothed adversaries. In NIPS, 2011.
[6] N. Buchbinder, S. Chen, J. Naor, O. Shamir: Unified Algorithms for Online Learning and Competitive Analysis. In COLT, 2012.
[7] N. Buchbinder and J. Naor. The Design of Competitive Online Algorithms via a Primal-Dual Approach. Foundations and Trends in Theoretical Computer Science, 2009.
[8] S. Arora, E. Hazan, and S. Kale. The Multiplicative Weights Update Method: a Meta-Algorithm and Applications, Theory of Computing, 2012.


Deep generative models seek to recover the process with which the observed data was generated. They may be used to synthesize new samples or to subsequently extract representations. Successful approaches in the domain of images are driven by several core inductive biases. However, a bias to account for the compositional way in which humans structure a visual scene in terms of objects has frequently been overlooked. In this work we propose to structure the generator of a GAN to consider objects and their relations explicitly, and generate images by means of composition. This provides a way to efficiently learn a more accurate generative model of real-world images, and serves as an initial step towards learning corresponding object representations. We evaluate our approach on several multi-object image datasets, and find that the generator learns to identify and disentangle information corresponding to different objects at a representational level. A human study reveals that the resulting generative model is better at generating images that are more faithful to the reference distribution.

[Overview]
In this paper, the authors proposed a compositional image generation methods that combines multiple objects and background into the final images. Unlike the counterpart which compose the images sequentially, the proposed method infer the relationships between multiple objects through a relational network before sending the hidden vectors to the generators. This way, the method can model the object-object interactions during the image generation. From the experimental results, the authors demonstrated that the proposed k-GAN can generate the images with comparable or slightly better FID compared with baseline GAN, and achieve plausible performance under the human study.
[Strengthes]
1. This paper proposed an interesting method for compositional image generation. Unlike the counterparts like LR-GAN, which generate foreground objects recurrently, this method proposed to derive the relationships between objects in parallel simultaneously. This kind of relational modeling has been seen in many other domains. It would be nice to see it can be applied to the compositional image generation domain.
2. The authors tried multiple synthesized datasets, including multi-MNIST and its variants, CLEVR. From the visualization, it is found that the proposed k-GAN can learn to disentangle different objects and the objects from the background. This indicates that the proposed model indeed capture the hidden structure of the images through relational modeling. The human study on these generated images further indicate that the generated images based on k-GAN is better than those generated by baseline GAN.
[Weaknesses]
1. The main contribution of this paper fall to the proposed method for modeling the relational structure for multiple objects in the images. In the appendix, the authors presented the results for the ablated version which does not consider the relationships. As the authors pointed out, these results are a bit counterintuitive and concluded that FID is not a good metrics for evaluating compositional generation. However, as far as I know, the compositional generation can achieve much better Inception scores on CIFAR-10 in LR-GAN paper (Yang et al). Combining the results on MNIST in LR-GAN paper, I would suspect that the datasets used in this paper are fairly simple and all methods can achieve good results without much efforts. It would be good to show some results on more complicated datasets, such as face images with background, or cifar-10. Also, the authors did not present the qualitative results for independent version of k-GAN. Meanwhile, they missed an ablated human study when the relational modeling is muted. I would like to see how the generated images without modeling relationships look like to humans.
2. Following the above comment, I think the datasets used in this paper is relatively simpler. In MM and CLEVR, the foreground objects are digits or simple cubes, spheres or cylinders. Also, the background is also simpler for these two datasets. Though CIFAR10+MM has a more complicated background, it is trivial for the model to distinguish the foregrounds from the backgrounds. Again, the authors should try more complicated datasets.
3. Though the proposed method can model the relationship between objects simultaneously, I doubt its ability to  really being able to disentangle the foregrounds from the backgrounds. Since the background and foregrounds are both whole images, which are then combined with an alpha blending, the model cannot discover the conceptually different properties for foreground and background that foregrounds are usually small than background and scattered at various locations. Actually, this has been partially demonstrated by Figure 4. In the last row, we can find one sphere is in the background image. I tend to think the proposed model performs similar to Kwak & Zhang's paper without a strong proof for the authors that the relational modeling plays an important role in the model.
4. It would be nice to perform more analysis on the trained k-GAN. Given the training set, like MM or CLEVR, I am wondering whether k-GAN can learn some reasonable relationship from the datasets. That is, whether it is smart enough to infer the right location for each objet by considering the others. This analysis can be performed, how much occlusions the generated images have compared with the real images. For example, on CLEVR, I noticed from the appendix that the generated CLEVR images base on k-GAN actually have some severe occlusions/overlaps.
[Summary]
In this paper, the authors proposed an interesting method for image generation compositionally. Instead of modeling the generation process recurrently, the authors proposed to model the relationships simultaneously in the hidden vector space. This way, the model can generate multiple foreground objects and backgrounds more flexibly. However, as pointed above, the paper missed some experiment, ablation study and analysis to demonstrate the relational modeling in the image generation. The author need to either try more complicated images or add deeper analysis on the recent experimental results.


We present a neural network architecture based upon the Autoencoder (AE) and Generative Adversarial Network (GAN) that promotes a convex latent distribution by training adversarially on latent space interpolations. By using an AE as both the generator and discriminator of a GAN, we pass a pixel-wise error function across the discriminator, yielding an AE which produces sharp samples that match both high-and low-level features of the original images.

Update:
I’d like to thank the authors for their thoroughness in responding to the issues I raised. I will echo my fellow reviewers in saying that I would encourage the authors to submit to another venue, given the substantial modifications made to the original submission.
The updated version provides a clearer context for the proposed approach (phychophysical experimentation) and avoids mischaracterizing GAIA as a generative model.
Despite more emphasis being put on mentioning the existence of bidirectional variants of GANs, I still feel that the paper does not adequately address the following question: “What does GAIA offer that is not already achievable by models such as ALI, BiGAN, ALICE, and IAE, which equip GANs with an inference mechanism and can be used to perform interpolations between data points and produce sharp interpolates?” To be clear, I do think that the above models are inadequate for the paper’s intended use (because their reconstructions tend to be semantically similar but noticeably different perceptually), but I believe this is a question that is likely to be raised by many readers.
To answer the authors’ questions:
- Flow-based generative models such as RealNVP relate to gaussian latent spaces in that they learn to map from the data distribution to a simple base distribution (usually a Gaussian distribution) in a way that is invertible (and which makes the computation of the Jacobian’s determinant tractable). The base distribution can be seen as a Gaussian latent space which has the same dimensionality as the data space.
- Papers on building more flexible approximate posteriors in VAEs: in addition to the inverse autoregressive flow paper already cited in the submission, I would point the authors to Rezende and Mohamed’s “Variational Inference with Normalizing Flows”, Huang et al.’s “Neural Autoregressive Flows”, and van den Berg et al.’s “Sylvester Normalizing Flows for Variational Inference”.
-----
The paper title summarizes the main claim of the paper: "adversarial training on latent space interpolations encourage[s] convex latent distributions". A convex latent space is defined as a space in which a linear interpolation between latent codes obtained by encoding a pair of points from some data distribution yields latent codes whose decoding also belongs to the same data distribution. The authors argue that current leading approaches fall short of producing convex latent spaces while preserving the "high-dimensional structure of the original distribution". They propose a GAN-AE hybrid, called GAIA, which they claim addresses this issue. The proposed approach turns the GAN generator and discriminator into autoencoders, and the adversarial game is framed in terms of minimizing/maximizing the discriminator’s reconstruction error. In addition to that, interpolations between pairs of data points are computed in the generator’s latent space, and the interpolations are decoded and treated as generator samples. A regularization term is introduced to encourage distances between pairs of data points to be mirrored by their representation in the generator’s latent space. The proposed approach is evaluated through qualitative inspection of latent space interpolations, attribute manipulations, attribute vectors, and generator reconstructions.
Overall I feel like the problem presented in the paper is well-justified, but the paper itself does not build a sufficiently strong argument in favor of the proposed approach for me to recommend its acceptance. I do think there is a case to be made for a model which exhibits sharp reconstructions and which allows realistic latent space manipulations -- and this is in some ways put forward in the introduction -- but I don’t feel that the way in which the paper is currently cast highlights this very well. Here is a detailed breakdown of why, and where I think it should be improved, roughly ordered by importance:
- The main reason for my reluctance to accept the paper is the fact that its main subject is convex latent spaces, yet I don’t see that reflected in the evaluation. The authors do not address how to evaluate (quantitatively or qualitatively) whether a certain model exhibits a convex latent space, and how to compare competing approaches with respect to latent space convexity. Figure 2 does present latent space interpolations which help get a sense of the extent to which interpolates also belong to the data distribution, however in the absence of a comparison to competing approaches it’s impossible for me to tell whether the proposed approach yields more convex latent spaces.
- I don’t agree with the premise that current approaches are insufficient. The authors claim that autoencoders produce blurry reconstructions; while this may be true for factorized decoders, autoregressive decoders should alleviate this issue. They also claim that GANs lack bidirectionality but fail to mention the existing line of work in that direction (ALI, BiGAN, ALICE, and more recently Implicit Autoencoders). Finally, although flow-based generative models are mentioned later in the paper, they are not discussed in Section 1.2 when potential approaches to building convex latent spaces are enumerated and declared insufficient. As a result, the paper feels a little disconnected from the current state of the generative modeling literature.
- The necessity for latent spaces to "respect the high-dimensional structure of the [data] distribution" is stated as a fact but not well-justified. How do we determine whether a marginal posterior is "a suboptimal representation of the high-dimensional dataset"? I think a more nuanced statement would be necessary. For instance, many recent approaches have been proposed to build more flexible approximate posteriors in VAEs; would that go some way towards embedding the data distribution in a more natural way?
- I also question whether latent space convexity is a property that should always hold. In the case of face images a reasonable argument can be made, but in a dataset such as CIFAR10 how should we linearly interpolate between a horse and a car?
- The proposed model is presented in the abstract as an "AE which produces non-blurry samples", but it’s not clear to me how one would sample from such a model. The generator is defined as a mapping from data points to their reconstruction; does this mean that the sampling procedure requires access to training examples? Alternatively one could fit a prior distribution on top of the latent codes and their interpolations, but as far as I can tell this is not discussed in the paper. I would like to see a more thorough discussion on the subject.
- When comparing reconstructions with competing approaches there are several confounding factors, like the resolution at which the models were trained and the fact that they all reconstruct different inputs. Removing those confounding factors by comparing models trained at the same resolution and reconstructing the same inputs would help a great deal in comparing each approach.
- The structure-preserving regularization term compares distances in X and Z space, but I doubt that pixelwise Euclidian distances are good at capturing an intuitive notion of distance: for example, if we translate an image by a few pixels the result is perceptually very similar but its Euclidian distance to the original image is likely to be high. As far as I can tell, the paper does not present evidence backing up the claim that the regularization term does indeed preserve local structure.
- Figures 2 and 3 are never referenced in the main text, and I am left to draw my own conclusions as to what claim they are supporting. As far as I can tell they showcase the general capabilities of the proposed approach, but I would have liked to see a discussion of whether and how they improve on results that can be achieved by competing approaches.
- The decision of making the discriminator an autoencoder is briefly justified when discussing related work; I would have liked to see a more upfront and explicit justification when first introducing the model architecture.
- When discussing feature vectors it would be appropriate to also mention Tom White’s paper on Sampling Generative Networks.


MODELING UNCERTAINTY WITH HEDGED INSTANCE EMBEDDING
Instance embeddings are an efficient and versatile image representation that facilitates applications like recognition, verification, retrieval, and clustering. Many metric learning methods represent the input as a single point in the embedding space. Often the distance between points is used as a proxy for match confidence. However, this can fail to represent uncertainty which can arise when the input is ambiguous, e.g., due to occlusion or blurriness. This work addresses this issue and explicitly models the uncertainty by "hedging" the location of each input in the embedding space. We introduce the hedged instance embedding (HIB) in which embeddings are modeled as random variables and the model is trained under the variational information bottleneck principle (Alemi et al., 2016;Achille & Soatto, 2018). Empirical results on our new N-digit MNIST dataset show that our method leads to the desired behavior of "hedging its bets" across the embedding space upon encountering ambiguous inputs. This results in improved performance for image matching and classification tasks, more structure in the learned embedding space, and an ability to compute a per-exemplar uncertainty measure which is correlated with downstream performance.

While most works consider embedding as the problem of mapping an input into a point in an embedding space, paper 1341 considers the problem of mapping an input into a distribution in an embedding space. Computing the matching score of two inputs (e.g. two images) involves the following steps: (i) assuming a Gaussian distribution in the embedding space, computing the mean and standard deviation for each input, (ii) drawing a set of samples from each distribution, (3) computing the normalized distances between the samples and (iv) averaging to obtain a global score.
The proposed approach is validated on a new benchmark built on MNIST.
On the positive side:
-	The topic of injecting uncertainty in neural networks should be of broad interest to the ICLR community.
-	The paper is generally clear.
-	The qualitative evaluation provides intuitive results.
On the negative side:
-	The whole idea of drawing samples to compute the distance between two Gaussian distributions seems unnecessarily complicated. Why not computing directly a distance between distributions? There exist kernels between distributions, such as the Probability Product Kernel (PPK). See Jebara, Kondor, Howard “Probability product kernels”, JMLR’04. The PPK between two distributions p(x) and q(x) writes as: \int_x p^a(x) q^a(x) dx, where a is a parameter. When a=1, it is known as the Expected Likelihood Kernel (ELK). When a=1/2, this is known as the Hellinger or Bhattacharyya kernel (BK). In p and q are Gaussian distributions, then the PPK can be computed in closed form. If p and q are mixtures of Gaussians, then the ELK can be computed in closed form.
-	The Mixture of Gaussians embedding extension is lacking in details. How does the network generate C Gaussian distributions? By having 2C output branches generating C means and C standard deviation vectors?
-	It might be useful to provide more details about why the self-similarity measure makes sense as an uncertainty measure. In its current state, the paper does not provide much intuition and it took me some time to understand (I actually understood when I made the connection with the ELK). Also, why not using a simpler measure of uncertainty such as the trace of the covariance matrix?
-	The experiments are lacking in some respects:
o	It would be useful to report results without the VIB regularization.
o	The focus on the cases D=2 and D=3 (embedding in a 2D or 3D space) shades some doubt on the practical usefulness of this framework in a higher-dimensional case.
Miscellaneous:
-	It seems there is a typo between equations (4) and (5). It should write z_1^{(k_1)} \sim p(z_1|x_1)
---
In their rebuttal, the authors satisfyingly addressed my concerns. Hence, I am upgrading my overall rating.