Imitation learning, followed by reinforcement learning algorithms, is a promising paradigm to solve complex control tasks sample-efficiently. However, learning from demonstrations often suffers from the covariate shift problem, which results in cascading errors of the learned policy. We introduce a notion of conservativelyextrapolated value functions, which provably lead to policies with self-correction. We design an algorithm Value Iteration with Negative Sampling (VINS) that practically learns such value functions with conservative extrapolation. We show that VINS can correct mistakes of the behavioral cloning policy on simulated robotics benchmark tasks. We also propose the algorithm of using VINS to initialize a reinforcement learning algorithm, which is shown to outperform prior works in sample efficiency.

This paper tackles an issue imitation learning approaches face. More specifically, policies learned in this manner can often fail when they encounter new states not seen in demonstrations. The paper proposes a method for learning value functions that are more conservative on unseen states, which encourages the learned policies to stay within the distribution of training states. Theoretical results are derived to provide some support for the approach. A practical algorithm is also presented and experiments on continuous control tasks display the effectiveness of the method, with particularly good results on imitation learning followed by reinforcement learning.
The proposed algorithm makes use of a natural intuition, that states visited by the expert probably have higher values, and the paper generally does a good job of supporting the approach through theory and experiments. Although the experiments seem sound, certain experimental details are not completely clear. The theory may also have some restrictive assumptions, limiting its significance.
Overall, I am divided about this paper. While this submission has the elements of a good paper and the presentation is great, certain concerns make me hesitant to recommend acceptance. I would be willing to increase my score if those points are addressed.
Theory:
1) My main concern is the applicability of the theorem, due to Assumption 4.2. While the intuition is that there is an action that corrects the next state towards the demonstration states, the theoretical condition is more restrictive. In particular, the following part (paraphrasing): "there exists an action a_cx that is close to a^bar and that makes a correction towards U". This condition implies that there are correcting actions near any action a^bar, which sems unrealistic in most cases. For example, in a driving task, say s^bar is a state such that moving back to U require the vehicle to move to the left. Then, consider the action a^bar of steering towards the right (with some angle). There could be no action near a^bar that makes the vehicle turn left towards U. Note that this is not necessarily a pathological situation as described in the text.
2) Also concerning assumption 4.2, I do not see why s^bar' is included in the quantifier of the statement since it is not used afterwards; after "there exists an action..." no mention of s^bar' is made.
3) The projection function may not be well-defined if there are multiple states that are closest to the one being projected.
4) It could be said explicitly that the expert policy is assumed to be deterministic. Currently, this is not said outright.
Experiments:
1) It seems like VINS relies heavily on the assumption that the environment is deterministic to learn an effective model. Was VINS tried in stochastic environments?
2) Data augmentation is used for VINS. This seems like an unfair advantage compared to the baseline competitors since sample efficiency is a key concern to reinforcement learning algorithms.  To make the comparisons fair, either it should be removed or the other algorithms should also receive additional data. How is the performance of VINS without this addition?
3) A description of how the hyperparameters were chosen and their final values would be needed for reproducibility. Also, a discussion of the importance of the hyperparameters and their sensitivity would be informative. For example, I was curious to know the value of \alpha in Algorithm 2 compared to the ranges the actions could take.
4) I am not convinced that using Q functions would necessarily fail. On p.6, the paragraph "can we learn conservatively-extrapolated Q-function" gives some reasoning why this could fail, that we may not want to penalize unseen actions. This is in opposition to the BCQ algorithm [1], which explicitly tries to avoid unseen actions and still has good performance. Trying a variant with Q(s,a) could be worthwhile.
I am not exactly sure if I understood Appendix A properly but, from my understanding, I do not think the argument made there necessarily invalidates using Q functions. It seems to apply mostly to deterministic expert policies and also Q(s,a) could still have reasonable values due to function approximation (even if the particular action 'a' is not seen in demonstrations).
5) Which RL algorithms were used for the imitation learning + RL set of experiments?
6) For table 1, are the results also averaged over different sets of demonstrations?
7) Are error bars one standard deviation or one standard error (divided by sqrt(n))?
8) For figure 3, using RL without imitation learning would serve as a good additional baseline
9) Ablation study: Trying no negative sampling with a perfect model could isolate the effect of negative sampling.
10) Ablation study: What is the no behavior cloning and perfect model experiment trying to show?
11) I think the name of the algorithm should be modified as "value iteration" refers to a specific dynamic programming algorithm for learning value functions, while the proposed algorithm does not resemble this at all.
Minor comments and typos (no impact on score):
- Using the cross-entropy method as in QTOpt [2] could be used to pick actions in a more refined manner.
- There is a large amount of blank space on p.8
- p.3 "At the test time" -> "At test time"
- p.4 "entire states space" -> "entire state space", "burned to warm up them" -> "burned to warm them up"
- p.9  "option 2 by search the action uniformly."  -> "option 2 to search the actions uniformly"
[1] "Off-Policy Deep Reinforcement Learning without Exploration" by Fujimoto et al.
[2] "QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation" by Kalashnikov et al.


This paper presents a generalized approach to provide an individual optimal temperature trajectory to each class. Temperature scaling is an effective technique to control the smoothness of the distribution. Current implementations of temperature scaling assume either a fixed or manually-crafted dynamically changing schedule. This schedule is used to identify the shared optimal trajectory, which is applied to every class. However, our studies indicate that under the change of context, we can identify the corresponding individual optimal trajectory for each class. To this end, we propose contextual temperature, a mechanism that allows the temperature to be learned during training along with the remaining model parameters, over the context for each vocabulary. Experimental results confirm that contextual temperature significantly improves state-of-the-art language models. The proposed method achieves a perplexity of 55.31 and 62.89 on the test set of Penn Treebank and WikiText-2, respectively. Additionally, in-depth analyses indicate that (a) every vocabulary possesses its individual schedule of temperature, and (b) due to the increased context from every time step, the optimal trajectory drops in order to suppress the uncertainties.

This work proposes a learned and context dependent way to calculate the temperatures for the softmaxes. More specifically, a low-rank affine-transformation, taking the hidden state at the current step as input, is used to calculate scalar weighting for every token in the vocabulary. The method is very general, and can be used in combination with other techniques in tasks such as language modeling and text generation. Experiments on language modeling with Penn TreeBank and WikiText-2 show that the proposed method yields strong performance.
Overall I found the paper well-motivated and easy to follow. The empirical results are solid and strong. The analysis is also interesting. I vote for an acceptance, if the authors can polish the writing.
Details:
- Eq. 5. The temperature scalar for each token competes with each other, since they are calculated with a softmax (and then rescaled). Another way is to use, e.g., a sigmoid function. Can the authors explain the motivation behind the use of softmax?
- Another view of the proposed method is that it learns a context-dependent weighting of the tokens in the vocabulary, such that "important" tokens (those with smaller \tau) receive more gradient updates. Can the authors comment on this? Also, I don't see the thermodynamics connection and find calling the proposed method `temperature` a bit misleading.
- Adding onto above. [1] discusses the low-rank bottleneck of using a single softmax. Since elementwise matrix product can blow up the rank, how do the authors think the proposed method can serve as a more efficient way to deal with the softmax bottleneck?
- Last but not least, the paper can be improved a lot if the authors can thoroughly polish the writing.
[1] Breaking the Softmax Bottleneck: A High-Rank RNN Language Model. https://arxiv.org/abs/1711.03953


Reinforcement learning encounters major challenges in multi-agent settings, such as scalability and non-stationarity. Recently, value function factorization learning emerges as a promising way to address these challenges in collaborative multiagent systems. However, existing methods have been focusing on learning fully decentralized value function, which are not efficient for tasks requiring communication. To address this limitation, this paper presents a novel framework for learning nearly decomposable Q-functions (NDQ) via communication minimization, with which agents act on their own most of the time but occasionally send messages to other agents in order for effective coordination. This framework hybridizes value function factorization learning and communication learning by introducing two information-theoretic regularizers. These regularizers are maximizing mutual information between agents' action selection and communication messages while minimizing the entropy of messages between agents. We show how to optimize these regularizers in a way that is easily integrated with existing value function factorization methods such as QMIX. Finally, we demonstrate that, on the StarCraft unit micromanagement benchmark, our framework significantly outperforms baseline methods and allows to cut off more than 80% communication without sacrificing the performance. The videos of our experiments are available at https://sites.google.com/view/ndvf.

The paper tackles the collaborative multi-agent RL problem as the problem of finding almost-decentralized value functions, where the loss tries to minimize communications between the agents. The core idea is around maximizing the mutual information between each massage and the existing knowledge of its receiver. This way, redundant messages are naturally removed. The authors then assert an entropy regularization to (almost) prevent the agents from *cheating*. The paper is in general well-written and motivated. There are however certain issues that should be addressed. [The second one is my main issue.]
1. The term *message* has been used repeatedly but not defined. You should define precisely what you mean by a message in the background section. In particular, it is not clear from the text what a message looks like mathematically. You may also consider giving intuitive examples of how different ways of designing a message alters the behaviour in a given context before start talking about how to optimise them.
2. Section 3.1 needs a revisit. I assume by "optimal action policy of agent j" you mean "optimal policy of agent j". Formally, an optimal policy is greedy to the optimal value function and is deterministic unless there exist multiple actions with same optimal value at a given state. Therefore, the mutual information is not mathematically well-defined since most of the time the optimal policy is deterministic and does not induce a probability distribution.
2.1 What is the optimal value function of an agent? The agents are not optimizing their own value function, so even if your complex model converges, it does not imply optimality of agent-level value functions (and they shouldn’t be locally optimal). If by that you mean the agent-level value functions *after* convergence of $Q_tot$, then you need to clarify it to avoid the confusion. Even so, it is still not clear how you conceive the agent-level policies from these value functions.
2.2. Minor: As for the notation, I found $A_j$ quite strange for a policy; you may want to consider something like $\pi_j$.
3. If $f_m$ is learnt, then message encoding is not stationary at least in the training time. It may make the training potentially become unstable. Specifically, there is no condition to assure stability of your model. Suggestion: your entropy regularization might be sufficient to induce stability. More discussion on this (or a simple experiment to show if an instability exists and will be alleviated with regularization) would be quite helpful.
4. If I understand it correctly [and it wasn't clear from the text], the encoder $f_m$ is conditioned on local history, which induces that all the agents must share same history shape (tensor-wise), which in turn means that all the local agents has to have same local state shape (computationally, they should for example have same output/internal-state shape in their neural networks). This sounds like a limitation, and it may only be OK in domains where agents are homogenous.


Through multi-agent competition, the simple objective of hide-and-seek, and standard reinforcement learning algorithms at scale, we find that agents create a selfsupervised autocurriculum inducing multiple distinct rounds of emergent strategy, many of which require sophisticated tool use and coordination. We find clear evidence of six emergent phases in agent strategy in our environment, each of which creates a new pressure for the opposing team to adapt; for instance, agents learn to build multi-object shelters using moveable boxes which in turn leads to agents discovering that they can overcome obstacles using ramps. We further provide evidence that multi-agent competition may scale better with increasing environment complexity and leads to behavior that centers around far more human-relevant skills than other self-supervised reinforcement learning methods such as intrinsic motivation. Finally, we propose transfer and fine-tuning as a way to quantitatively evaluate targeted capabilities, and we compare hide-and-seek agents to both intrinsic motivation and random initialization baselines in a suite of domain-specific intelligence tests.

# Review ICLR20, Emergent Tool Use...
This review is for the originally uploaded version of this article. Comments from other reviewers and revisions have deliberately not been taken into account. After publishing this review, this reviewer will participate in the forum discussion and help the authors improve the paper.
I apologize in advance for being reviewer 2.
## Overall
**Summary**
The article introduces a new multi-agent physics environment called "hide-and-seek". The authors trained agents in this environment and studied the emergence of and changes in strategies. The authors also study the performance of these same agents in new "targeted intelligence tests" compared to training from scratch and compared to agents trained with curiosity.
**Overall Opinion**
I think the environment is very appealing and the paper is overall well-structured and demonstrates novel work. Therefore I'd recommend this paper to be accepted. That being said, there are glaring issues with some of the writing that need to be addressed before I think this work conforms to the standards of ICLR. However, if these issues are addressed, I have no issue increasing my review score.
Main problems:
- The majority of the paper presents essentially a case study of what happened during a single seed of policy training. For RL literature that's very uncommon and I think it's consensus that DRL is very sensitive to random seeds. I know that you do have additional seeds in the appendix, but why didn't you mention those in the main body of the paper? You seem to have found some robustness against multiple seeds, so why not show it? And also the fact that Figure 1 & 3 only apply to 1 seed is not mentioned. I think this is easy enough to fix - I suggest since you're already at 10 pages, to just bring in the additional seeds from the appendix and average over their performance in Fig.1&3.
- The contributions section is overselling the work: (1) states that autocurricula lead to changes in agent strategy - Maybe I'm mistaken here but that sounds like a tautology. In other words, "a self-generated sequence of challenges" ("Autocurricula", according to [Leibo et al., 2019][1]) lead to changes in strategy. And (3) advertises "a proposed framework for evaluating agents in open-ended environments" and also "a suite of targeted intelligence tests for our domain". The former of those two is either not in the paper or you mean your section "6.2 Transfer and Fine-Tuning as Evaluation", which isn't novel (see e.g. [Alain & Bengio, 2016][2])
- Your acknowledgments should be anonymized until publication. Otherwise, reviewers might draw conclusions which group published this work, thus violating the double-blind review procedure.
[1]: https://arxiv.org/pdf/1903.00742.pdf
[2]: https://arxiv.org/pdf/1610.01644.pdf
Like I mentioned above, I think these are all easy to address, which should allow acceptance of this work. Here are some additional questions, comments, and nitpicks:
## Specific comments and questions
### Abstract
- "evidence that ... competition may scale better with increasing environment complexity" - that's only shown in the appendix
### Intro
- You mention TD-Gammon as a game, but I think it's an algorithm for the game Backgammon, similarly to how "Go" is the game and "AlphaGo" is an algorithm for playing.
### Rel. Work
- all good
### Hide And Seek
- Arena boundaries: What's the penalty and what's " too far outside the play area"? And in all depictions, it looks like the geometry of the arena is elevated around the edges and the agents don't have a jump action, so how would they ever go out of borders? After watching the videos: Apparently, the jagged-looking arena boundary in the videos is purely cosmetic and agents can still access that space. This is unclear from just the paper and the renderings in Figure 1.
### Policy Optimization
- Policy network and fusion are underspecified: How do you deal with the varying number of agents, boxes, obstacles? Do you just set the x/v of the missing pieces to zero or is the observation actually of a different shape in case there are more/fewer objects or agents? How's the embedding done that is depicted in Figure 2? Also, I didn't see the embedding being mentioned in the text - any reason for that?
- Figure 2 - This diagram is visually appealing but confusing and needs to be improved. Why does the agent's embedding say "1"? Why do the other agents' embeddings have a "-1" at the end of the orange box and the others don't? Is the agent's embedding concatenated with the other embeddings? If so, why and how (concat, sum, multiply, conditional batch norm, etc.)? In the center and on the right you use a blue block to indicate the agent's embedding and then at the bottom right you seem to use it as a network component or something (between the "LSTM")? If you're trying to signal that this is the agent's perception at different stages in the network, I'd use a different color to separate it from the agent's lidar and pos/vel. You don't mention that "x,v" stands for "position, velocity".

### Auto-curriculum and Emergent Behavior
- Figure 3: "environment-specific" (add dash). Draw skill development boundaries like in Fig.1.
- How exactly does the "surfing" work? The seekers step (not jump, right, since there is no jumping?) onto the boxes and then what? The momentum propels the box forward? Do other seekers push the box? Their movement on top of the box somehow moves the box (this seems to be the case judging by the videos but this is the least physically plausible)? This is a super interesting adaptation but I'd suspect the physics simulation to have a bug/glitch that's being exploited here.
- You mention in footnote 3 that the developmental stage and changes in reward aren't necessarily correlated. The same seems to be true for the metrics in Fig.3, which raises the question how did you come up with those boundaries for the different developmental stages in Fig.1? Did someone look at rollouts from the trained policy every couple of million steps? And do all agents learn new skills at the same time or is there a delay? From my understanding, they are all using the same policy and critic networks but maybe dependent on the proximity of an agent to an object/obstacle, it's easier or harder to execute.
### Evaluation
- clear and well-written, slightly too much content in the appendix and not enough in the main paper. Weird appendix numbering - A.6 appears in the main paper pages after A.7
### Discussion and Future Work
- all good
### Appendix
- I appreciate the TOC. I did not look into Appendix B-D because it's another 10 pages on top of the 10 pages of the article.
All in all an interesting work. Good luck with the rebuttal/discussion.


The learning rate warmup heuristic achieves remarkable success in stabilizing training, accelerating convergence and improving generalization for adaptive stochastic optimization algorithms like RMSprop and Adam. Pursuing the theory behind warmup, we identify a problem of the adaptive learning rate -its variance is problematically large in the early stage, and presume warmup works as a variance reduction technique. We provide both empirical and theoretical evidence to verify our hypothesis. We further propose Rectified Adam (RAdam), a novel variant of Adam, by introducing a term to rectify the variance of the adaptive learning rate. Experimental results on image classification, language modeling, and neural machine translation verify our intuition and demonstrate the efficacy and robustness of RAdam.

I haven't worked in this area before, and my knowledge to the topic of designing optimizers for improving stochastic gradient descent is limited to the level of advanced ML courses at graduate school. Nevertheless, below I try my best to evaluate the technicality of this paper
====================
In this work the authors studied the variance issue of the adaptive learning rate and aim to justify the warm-start heuristic. They also demonstrate that  the convergence issue of many of the stochastic gradient descent algorithms is due to large variance induced by the adaptive learning rate in the early stage of training. To tackle this issue, they proposed a variant of ADAM, which is known as rectified ADAM, whose learning rate not only takes the momentum into the account, but it also adapts to the variance of the previous gradient updates.
In the first part of the paper, the authors analyzed the variance issue exists in the existing ADAM algorithm, such that with limited samples in the early stage of training, the variance of the adaptive learning rate becomes rather large and it induces high variance to the gradient update to ADAM. In general I found this theoretical justification on the observation of variance issue in ADAM sound, and quite intuitive. In the second part, they proposed the algorithm, namely rectified ADAM, where the difference here to take the second moment  of the gradient into account when updating the adaptive learning rate. They showed that the variance of the adaptive learning rate with such rectification is more numerically stable (especially when variance is intractable in vanilla ADAM), and under some regularity assumption it decreases in the order or O(1/\rho_t).
In extensive numerical studies of supervised learning, the authors showed that RADAM achieves a better accuracy than ADAM (although in Table 1, I am a bit puzzled why the best accuracy is indeed from SGD, if so, what's the point of all adaptive learning rates, is that because SGD requires extensive lr tuning?) Because the superiority in accuracy they also showed that RADAM manages to have more stable training and achieves lower training loss, which is quite interesting.


We propose laconic classification as a novel way to understand and compare the performance of diverse image classifiers. The goal in this setting is to minimise the amount of information (aka. entropy) required in individual test images to maintain correct classification. Given a classifier and a test image, we compute an approximate minimal-entropy positive image for which the classifier provides a correct classification, becoming incorrect upon any further reduction. The notion of entropy offers a unifying metric that allows to combine and compare the effects of various types of reductions (e.g., crop, colour reduction, resolution reduction) on classification performance, in turn generalising similar methods explored in previous works. Proposing two complementary frameworks for computing the minimal-entropy positive images of both human and machine classifiers, in experiments over the ILSVRC test-set, we find that machine classifiers are more sensitive entropy-wise to reduced resolution (versus cropping or reduced colour for machines, as well as reduced resolution for humans), supporting recent results suggesting a texture bias in the ILSVRC-trained models used. We also find, in the evaluated setting, that humans classify the minimal-entropy positive images of machine models with higher precision than machines classify those of humans.

The paper proposes and studies a task where the goal is to classify an image that has been intentionally degraded to reduce information content - hence the name "laconic" image classification. The motivation for this task is to compare human and machine performance in a task that deviates from the standard ImageNet setup. To make different content reductions comparable, the authors measure the (approximate) entropy of an image via its PNG-compressed file size. As image transformations, the authors utilize quantization, downsampling, cropping, and a combination of the above. The authors find that convnets with higher accuracy are also more robust to these perturbations, and that humans perform well on the minimum-entropy examples of the networks (but not vice-versa).
Overall I find the comparison of human and machine performance interesting and hence recommend accepting the paper. However, there are multiple directions that could possibly strengthen the core experiment. Hence I only give a weak accept at this point. Concretely, these directions are:
- Do the results in the paper change if a different entropy measure is used (e.g., JPEG compression)?
- As suggested by the authors, training networks to be robust to the "laconic" image perturbations could be an interesting direction. For instance, standard data augmentation with the proposed perturbations would be a relevant baseline.
- To aid replicability and to compare the performance of different human test subjects, it would be interesting to conduct the experiment also on a crowdsourcing platform such as Mechanical Turk (as a complement to, not a replacement for, the university population in the paper).
- Also to add replicability and to make it easier to compare different human accuracy evaluations, it would be good to measure how well the annotators perform on the unperturbed images and on a simple noise transformation (e.g., Gaussian noise).
- It would be good to know how approximate the entropy measures in the paper are, e.g., to understand why humans perform worse in the "combined" perturbation setting.
- How did the results from the control group and the open / online evaluation differ?
In addition, I have the following suggestions for improving the paper:
- For the related work section, the authors may find the following papers on robustness of convnets to distortions interesting:
* Manitest: Are classifiers really invariant?
https://arxiv.org/abs/1507.06535
* Exploring the Landscape of Spatial Robustness
https://arxiv.org/abs/1712.02779
* Spatially Transformed Adversarial Examples
https://arxiv.org/abs/1801.02612
* Semantic Adversarial Examples
https://arxiv.org/abs/1804.00499
- It could be helpful for the reader to see some example images of the different transformations in the main text.
- Section 6: "[...] a bias in texture for images trained on the ILSVRC dataset [...]" - should this be "classifiers" instead of "images"?
- Section 6: "[...] would be interesting to explore in future" - insert "the" before "future"?


The ability to learn over changing task distributions without forgetting previous knowledge, also known as continual learning, is a key enabler for scalable and trustworthy deployments of adaptive solutions. While the importance of continual learning is largely acknowledged in machine vision and reinforcement learning problems, this is mostly under-documented for sequence processing tasks. This work focuses on characterizing and quantitatively assessing the impact of catastrophic forgetting and task interference when dealing with sequential data in recurrent neural networks. We also introduce a general architecture, named Gated Incremental Memory, for augmenting recurrent models with continual learning skills, whose effectiveness is demonstrated through the benchmarks introduced in this paper.

The goal of this work is to best understand the performance and benchmarking of continual learning algorithms when applied to sequential data processing problems like language or sequence data sets. The contributions of the paper are 3 fold - new benchmarks for CL with sequential data for RNN processing, new architecture introduced for more effective processing and a thorough empirical evaluation.
Introduction:
I think a little more insight into why the sequential data processing CL scenario is any different than the vision scenario would be quite helpful. Specifically, it would be quite impactful to tell us more about what the additional challenges with RNNs for CL vs feedforward for CL are in the intro.
The paper is written as if the benchmark is the main contribution and the architecture improvement is just a delta on top of this, but it gets confusing when the methods section starts off with just directly stating the new architecture.
The algorithm seems like a straightforward combination of recurrent progressive nets and gated autoencoders for CL. Can the authors provide more justification if that is the contribution or there is more to the insight than has been previously suggested in prior work?
Figure 1 has a very uninformative caption. It also doesn’t show how modules feed into one another properly.
The motivation for why one needs GIM after one already has A-LSTM or A-LMN is not very clear?
Overall the contribution does seem a bit incremental based on prior work and the description lacks enough detail to properly indicate why this is a very important contribution?
Experiments:
What does it mean to be application agnostic but restricted to particular datasets and losses? This doesn’t quite parse to me.
The description of the tasks is very informal and hard to follow. It’s not clear what exactly the tasks and datasets look like
“using morehidden units can bridge this gap” -> why not just do it? Its a benchmark after all.
Overall the task descriptions should be in a separate section where the setup is described in a lot of detail and motivated properly.
The results in the experiments section are very hard to parse. The captions need much more detail for eg Table 2.
Could we also possibly have more baselines from continual learning? For instance EWC (Kirkpatrick) or generative replay might be competitive baselines.
Overall I think that the GIM and A-LMN and A-LSTM methods are reasonable although somewhat incremental. But the proposed benchmarks are pretty unclear and the results are a bit hard to really interpret well. It would also be important to run comparisons with more baselines and to provide more ablation/analysis experiments to really see the benefit of GIM/A-LMN or A-LSTM. I also think that the task descriptions should be much earlier in the paper and desribed in much more rigorous detail.


Equivariant neural networks are special types of neural networks that preserve some symmetries on the data set. In this paper, we provide a method to modify a neural network to an equivariant one, which we call equivarification.

In this work, the authors employ concepts from group theory to turn an arbitrary feed forward neural network into an equivariant one, i.e. a network whose output transforms in a way that is consistent with the transformation of the input. To this end, the authors first introduce the basic concepts of group theory required to follow their work and provide a comprehensive definition of equivariance. They then explain how to equivarify (w.r.t. a finite group G) a given neural network, and present experimental results on rotated MNIST digits to support their approach.
I find that the proposed approach is very elegant and addresses a highly important question, namely how to devise or modify an architecture such that it preserves symmetries. In my opinion, the current paper makes an interesting theoretical contribution towards achieving this goal, but it also has several shortcomings that are detailed below. Based on these shortcomings, I recommend rejecting the paper but I would be willing to increase the score if these points were addressed in sufficient detail.
Major comments:
1) Scaling
The authors mention in the abstract that ‘although the network size scales up, the constructed equivariant neural network does not increase the complexity of the network compared with the original one in terms of the number of the parameters.’ Based on Eq. (3.0.2) and Fig. 3, it is my understanding that n evaluations of each data point (input to a layer) are required for a cyclic group of order n. If the outputs of a layer are then concatenated, the input dimension of the subsequent layer grows by a factor of n. I would therefore argue that out-the-box application of the proposed approach does increase the number of variables dramatically and that the abstract is misleading in this respect. The authors briefly comment on this point with one sentence in the third paragraph of the introduction. However, I would appreciate if this point was addressed in more detail, for example in a dedicated paragraph after the theory is introduced. Please also address the question of whether variable sharing is essential from an equivariance point of view, or whether it’s simply a necessity to prevent an explosion of the number of parameters. Furthermore, convolutions encode translational symmetry which may be beneficial for the current application but may not be desirable for other datasets. A few comments for clarification would be very helpful. Finally, the equivarified network seems to increase the required number of computations significantly compared to the original one, which I find worth a comment .
2) Experiment
The authors only consider a convolutional architecture and say that ‘in order to illustrate flexibility we choose not to simply equivarify the original network’. However, to me one of the main advantages of the paper seems to be that you can take this approach to equivarify any FFN. It would therefore be interesting to see this approach be applied to different networks starting with a simpler one, e.g. a 2-layer MLP. The authors could then compare the original network to the equivarified one with and without variable sharing. That would not only help the reader understand the approach better but also be much more in line with the main motivation of the paper. Then adding a second experiment, e.g. a convolutional architecture, to demonstrate flexibility would be very interesting. With regard to Fig. 4, I think there may be better ways of summarising the results than dumping 160 numbers of which only 4 seem to be of interest. The message seems to be that the network yields identical probabilities irrespective of the degree of rotation. What I find surprising is that all numbers are actually identical (shifted by 10). Is this by construction?
3) Limitations
As indicated in the second paragraph of Sec. 4, this approach is limited to finite groups and the authors only consider image rotations w.r.t. the cyclic group of degree 4. Although I appreciate that this is meant to serve as a toy problem to illustrate that the approach works, I do not think that rotations by a constant angle are very interesting. What would be really interesting is equivariance w.r.t continuous rotations (Lie Groups), e.g. the SO(2) in this particular case. I doubt that an extension to the SO(2) is straightforward within the current theoretical framework. However, even if that is the case, I would appreciate if the authors could comment on this in a paragraph.
Minor comments:
i) There are many typos and grammar mistakes in the paper:
‘any feedforward networks’ -> ‘any feedforward network’.
‘enables to design’ -> ‘enables us to design’
‘our proposed equivarification method use’ -> ‘our proposed equivarification method uses’
‘traditional sense multiplication’ -> ‘traditional sense of multiplication’
‘a group G acts’ -> ‘a group G that acts’
‘neural network that defined on the’ -> ‘neural network that is defined on the’
‘which achieves promising performance’ -> ‘which achieve promising performance’
‘supplymentary material’ -> ‘supplementary material’
Etc.
ii) I think there may be a mistake in the 3rd point of Definition 3.0.3: For consistency with the previous definitions and with Fig. 1, shouldn’t F map from X to Z and \hat F from X to \hat Z?
iii) Last paragraph of Sec 3: ‘then after the identification \hat F becomes a map from Z to...'.  Should it be ‘a map from X to ..’?
iv) In Definition 3.0.3 you define the tuple (\hat Z, T, p) to be a G-equivarification, but in the paragraph below you call the G-product itself a G-equivarification (without including T and p).
v) Footnote 2: You could correct for that and present the theory shifting by g instead of g^-1 to make it easier for the reader to follow. Or, at least, give a reference to the footnote earlier on in Example 4.0.1 to avoid confusion.
vi) Unless there is a special reason for this, I would suggest changing the enumeration of definitions, lemmas, examples and equations, i.e. (3.0.1) -> (3.1), etc...
*********************************************************
I increased my rating based on the authors addressing many of my comments.
*********************************************************


Text generation is ubiquitous in many NLP tasks, from summarization, to dialogue and machine translation. The dominant parametric approach is based on locally normalized models which predict one word at a time. While these work remarkably well, they are plagued by exposure bias due to the greedy nature of the generation process. In this work, we investigate un-normalized energy-based models (EBMs) which operate not at the token but at the sequence level. In order to make training tractable, we first work in the residual of a pretrained locally normalized language model and second we train using noise contrastive estimation. Furthermore, since the EBM works at the sequence level, we can leverage pretrained bi-directional contextual representations, such as BERT and RoBERTa. Our experiments on two large language modeling datasets show that residual EBMs yield lower perplexity compared to locally normalized baselines. Moreover, generation via importance sampling is very efficient and of higher quality than the baseline models according to human evaluation.

The authors make good points, starting from the exposure bias and label bias suffered by the mainstream neural auto-regressive models.
Residual EBMs are defined and trained using NCE. Experiments on two large language modeling datasets show that residual EBMs yield lower perplexity and generation via importance sampling is of higher quality, compared to locally normalized baselines.
In generally, the paper is well motivated and interesting. But I have some concerns.
1. Missing important relevant references.
EBMs (a.k.a. un-normalized models, random fields) have been successfully developed in language modeling in recent years. A large body of this paper has been studied in [5,6], including the model and the NCE estimation method. The model Eq.(2) is exactly the model in [5], defining the model in the form of exponential tilting of a reference distribution.
Connecting and comparing to these previous works are needed.
[1] R. Rosenfeld, S. F. Chen, and X. Zhu, “Whole-sentence exponential language models: a vehicle for linguistic-statistical integration,” Computer Speech & Language,  2001.
[2] B. Wang, Z. Ou, and Z. Tan, “Trans-dimensional random fields for language modeling,” ACL, 2015.
[3] B. Wang, Z. Ou, and Z. Tan, “Learning transdimensional random fields with applications to language modeling,” IEEE transactions on pattern analysis and machine intelligence, 2018.
[4] B. Wang and Z. Ou, “Language modeling with neural trans-dimensional random fields,” IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), 2017.
[5] B. Wang and Z. Ou, “Learning neural trans-dimensional random field language models with noise-contrastive estimation,” IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2018.
[6] B. Wang and Z. Ou, “Improved training of neural trans-dimensional random field language models with dynamic noise-contrastive estimation,” IEEE Spoken Language Technology Workshop (SLT), 2018.
2. I am a little bit concerned that the theoretical contribution seems weak.
Though Eq. (4) and (5) seem to be novel, I am not sure whether such a contribution is substantial enough to motivate acceptance.
I'm happy to adjust the score if the paper can be better placed in the literature and the authors take efforts to improve the paper.
--------update after reading the response-----------
Being well-placed in the literature and properly claiming contribution with respect to prior work is one of the key questions in reviewing a paper. The first version of the paper clearly lacks in this respect. That's the main concern when I gave a 1.
I appreciate the authors' response. The updated paper has been improved to address my main concern, although the added discussions presented in the updated paper is not as clear as the authors' clarifications in the response. I suggest to polish the main text incorporating these clarifications.
Generally, it is nice to see the successful application of energy-based/random-field-based models in text generation, besides in speech recognition. I update the score to 6 (Weak Accept).
It would have been better that the following can be further clarified.
"the partition function estimated via importance sampling would lead to bias favoring the random field language model" --- this comment is not clear to me.
Both Eq.4 and Eq.5 give estimates for perplexity. It would be better to clarify different uses of the two equations. If the perplexities are estimated using Eq.4 (as in Table 1), then what is the purpose of developing Eq.5?
How to calculate the lower and upper bounds of the step-wise perplexity gain at each position in Figure 1?
Under Figure 1, "At each position the lower and upper bounds (see Eq. 4) are estimated using 20,000 samples." But in the main text, it is said that "We therefore break down perplexity per position in the generated sequences as in Eq. 5" at page 8. It is confusing.


Positive-unlabeled (PU) learning learns a binary classifier using only positive and unlabeled examples without labeled negative examples. This paper shows that the GAN (Generative Adversarial Networks) style of adversarial training is quite suitable for PU learning. GAN learns a generator to generate data (e.g., images) to fool a discriminator which tries to determine whether the generated data belong to a (positive) training class. PU learning is similar and can be naturally casted as trying to identify (not generate) likely positive data from the unlabeled set also to fool a discriminator that determines whether the identified likely positive data from the unlabeled set (U) are indeed positive (P). A direct adaptation of GAN for PU learning does not produce a strong classifier. This paper proposes a more effective method called Predictive Adversarial Networks (PAN) using a new objective function based on KL-divergence, which performs much better. Empirical evaluation using both image and text data shows the effectiveness of PAN.

<Paper summary>
The authors proposed a novel method for positive-unlabeled learning. In the proposed method, adversarial training is adopted to extract positive samples from unlabeled data. In the experiments, the proposed method achieves better performance compared with state-of-the-art methods.
<Review summary>
Although the idea to utilize adversarial training for PU learning is interesting, the proposed method is not sufficiently validated in theory. In addition, the manuscript is hard to follow due to confusing notations and lack of figures. I vote for rejection.
<Details>
* Strength
 + The main idea is simple and interesting.
 + The proposed method performs well in the experiments.
* Weakness and concerns
 - Confusing notations and lack of figures.
  -- Lack of mathematical definition of C and D.
  -- The argument of P^p and that of P^u are different (x^p and x^u), which implies that those distributions are defined at different space (but actually same).
  -- Shared index ``i" for positive and unlabeled data in Eq. (3).
  -- The notation with ``hat" often imply the empirically estimated (or approximated) value in the field of ML.
  -- No figures about the proposed method. Specifically, it is hard to understand the relationship between C and D.
 - Since Eq. (3) looks totally different from Eq. (2), why Eq. (3) is reasonable remains unclear.
  -- About I: first, P^{pu} cannot be calculated, because it requires unavailable labels of x^u. If you treat unlabeled data as negative, it should not be called ``ground-truth," and the term I cannot help D correctly recognize positive samples. Second, the positive samples are almost ignored in this term, because the number of positive data should be substantially small in a common setting of PU learning.
  -- About II: the authors explain the role of this term by min-max game between C and D during optimization, but the most important point here is what will happen when we obtain the optimal C and D after the optimization. What property or behavior do the optimal C and D have?
 - What do the authors want to claim with Proposition 1? The right-hand side of Eq. (5) cannot be easily calculated due to the density ratio between P^p and P^u. There is no explanation about what f and eps mean. What ``optimal" means is also ambiguous.
* Minor concerns that do not have an impact on the score
 - Although the problem setting is quite different, the idea of this paper is partially similar to the importance weighting technique adopted in some recent domain adaptation methods [R1, R2]. Do you have any comment on that?
[R1] ``Reweighted adversarial adaptation network for unsupervised domain adaptation," CVPR2018
[R2] ``Importance weighted adversarial nets for partial domain adaptation," CVPR2018


Recent theoretical work has established connections between over-parametrized neural networks and linearized models governed by the Neural Tangent Kernels (NTKs). NTK theory leads to concrete convergence and generalization results, yet the empirical performance of neural networks are observed to exceed their linearized models, suggesting insufficiency of this theory. Towards closing this gap, we investigate the training of over-parametrized neural networks that are beyond the NTK regime yet still governed by the Taylor expansion of the network. We bring forward the idea of randomizing the neural networks, which allows them to escape their NTK and couple with quadratic models. We show that the optimization landscape of randomized two-layer networks are nice and amenable to escaping-saddle algorithms. We prove concrete generalization and expressivity results on these randomized networks, which leads to sample complexity bounds (of learning certain simple functions) that match the NTK and can in addition be better by a dimension factor when mild distributional assumptions are present. We demonstrate that our randomization technique can be generalized systematically beyond the quadratic case, by using it to find networks that are coupled with higher-order terms in their Taylor series.

This paper studies the training of over-parameterized neural networks. Specifically, the authors propose a novel method to study the training beyond the neural tangent kernel regime by randomizing the network and eliminating the effect of the first order term in the network’s Taylor expansion. Both optimization guarantee and generalization error bounds are established for the proposed method. It is also shown that when learning polynomials, the proposed randomized networks outperforms NTK by a factor of d, where d is the input dimension.
Overall, I enjoy reading this paper. The presentation is clear, the arguments are insightful, and the proofs seem to be solid. Moreover, this paper offers some interesting ideas to show that neural networks can outperform NTK, which might be impactful. However, this paper also has certain weak points, mainly due to the less common problem setting.
Although it is believed that NTK cannot fully explain the success of deep learning, results in the NTK regime have the advantage that the problem setting (initialization method, training algorithm) is very close to what people do in practice. Therefore, ideally, a result beyond NTK should demonstrate the advantage of NN over NTK under similar, or at least practical settings. If the problem setting is changed in some strange way, then it might not be that meaningful even if the training behavior is different from the NTK setting. In my opinion, the following four points about the problem setting in this paper are less desired:
(1) Assumption A is not satisfied by any commonly used activation functions. The authors only provided cubic ReLU as an example.
(2) The randomization technique in this paper is not standard, and is pretty much an artifact to make the first order term in the Taylor expansion of neural networks small.
(3) The $(\| \cdot \|_{2,4})^8$ regularization term introduced on page 6 is highly unusual. Due to reparameterization, this regularization is on the distance towards initialization, indead of the norms of the weight parameters.
(4) The training algorithm used in this paper is noisy SGD due to the need to escape from saddle points.
Despite the issues mentioned above, I still think this paper is of good quality, and these limitations are understandable considering the difficulty to escape from the NTK regime. It would be interesting if the authors could provide some direct comparison between the generalization bound in this submission and existing generalization bounds in the NTK regime, for example the results in
Yuan Cao, Quanquan Gu, Generalization Error Bounds of Gradient Descent for Learning Over-parameterized Deep ReLU Networks
Yuan Cao, Quanquan Gu, Generalization Bounds of Stochastic Gradient Descent for Wide and Deep Neural Networks
Moreover, since this paper studies optimization with regularization on the distance towards initialization, it would also be nice to compare with the following paper:
Wei Hu, Zhiyuan Li, Dingli Yu, Understanding Generalization of Deep Neural Networks Trained with Noisy Labels


Loss functions play a crucial role in deep metric learning thus a variety of them have been proposed. Some supervise the learning process by pairwise or tripletwise similarity constraints while others take advantage of structured similarity information among multiple data points. In this work, we approach deep metric learning from a novel perspective. We propose instance cross entropy (ICE) which measures the difference between an estimated instance-level matching distribution and its ground-truth one. ICE has three main appealing properties. Firstly, similar to categorical cross entropy (CCE), ICE has clear probabilistic interpretation and exploits structured semantic similarity information for learning supervision. Secondly, ICE is scalable to infinite training data as it learns on mini-batches iteratively and is independent of the training set size. Thirdly, motivated by our relative weight analysis, seamless sample reweighting is incorporated. It rescales samples' gradients to control the differentiation degree over training examples instead of truncating them by sample mining. In addition to its simplicity and intuitiveness, extensive experiments on three real-world benchmarks demonstrate the superiority of ICE.

Overview
This paper proposes a new objective function called Instance Cross Entropy (ICE) for metric learning. Compared to the triplet loss (or contrastive loss) and its many variants, the distance between points in feature space is defined to be the dot/inner product, rather than computing the euclidian distance between feature vectors. Since the L2 norm of the final features is constrained  to be 1, the dot product represents the cosine distance between 2 feature vectors. Compared to the softmax/categorical loss, this objective has the advantage that there is no need to learn a per-class weight vector in the output softmax layer and therefore this method can be used even when the number of classes during training is unbounded. This is a very useful design feature, since the output softmax layer grows linearly with the size of the classes and can quickly become prohibitively large. Furthermore, mini batches for training can be randomly sampled without requiring expensive negative data mining strategies which are necessary when using the triplet loss. This is another very useful property.
Overall, I think the ideas presented are interesting and the paper provides a useful summary of existing approaches for metric learning. However, I think the technical presentation needs to be improved significantly before  the paper can be accepted for publication. I found the paper quite difficult to follow. I had to re-read the motivation and the description of the loss function many times before being able to understand what is going on and how the proposed loss compares and contrasts with existing approaches. I also think some of the terminology and notation used in the paper is very confusing and should be updated to help the reader get to the main argument quickly.
Comments
1. Title: What exactly does instance cross entropy (ICE) mean? And how is it different from categorical cross entropy (CCE)? Isn’t categorical cross-entropy also calculating the entropy between the predicted distribution for *each instance* and the ground truth? I think the authors should reconsider the name for the proposed loss and choose a more descriptive name for the algorithm.
2.  I also found the use of the term “matching distribution” a bit confusing since it has a well-defined meaning in statistics. However ML is full of overloaded terms and I understand if the authors want to keep this description.
3. None of the references in the main text have brackets around them. I think this makes the paper appear very cluttered and makes parsing each paragraph quite difficult. I would highly recommend the paper be reformatted and the authors use brackets around references.
4. In the abstract, the authors mention that the proposed method has a “clear probabilistic interpretation”. From my understanding, I see ICE as a blend of the categorical cross-entropy loss and the triplet loss, retaining useful properties of each without increasing the complexity of the loss computation. However, I do not see a clear probabilistic interpretation of the loss function. The softmax computation yields a probability distribution like output given a query and an anchor point, which is hand-designed in the objective function. I  think the authors should more clearly motivate what the probabilistic angle for the loss function is.
5. I found Figure 1 to be very difficult to interpret. This is a missed opportunity, since this figure alone could communicate some of the key ideas in the paper to the reader. However, there are many details that have not been explicitly mentioned. What do the colours (white, blue, yellow) mean? The figure shapes probably mean classes. What do the terms p_* mean? What do i,j mean in Figure 1 e and how are they different from the previous figures. These details should all be present in the title for the figure. Even with these details, the figure could still be a bit more explicit. I found the yellow arrow with cross entropy and the notion of “ground truth” in the figure difficult to follow.
6. How was the constrained optimisation performed? In Equation 6, we see that there is a constraint associated with each example in the mini batch which says the norm of the feature vector should be equal to 1. However I could not find any implementation details of how this constraint was satisfied. Did the authors use a Lagrangian formulation? I think the paper is irreproducible without this detail.
Summary
I think this is an interesting proposal to combine the useful features of the softmax and triplet losses. However, I think the technical presentation needs to be improved significantly in order for the paper to be accepted for publication.


We present a neural architecture search algorithm to construct compact reinforcement learning (RL) policies, by combining ENAS (Vinyals et al., 2015;Pham et al., 2018;Zoph & Le, 2017) and ES (Salimans et al., 2017) in a highly scalable and intuitive way. By defining the combinatorial search space of NAS to be the set of different edge-partitionings (colorings) into same-weight classes, we represent compact architectures via efficient learned edge-partitionings. For several RL tasks, we manage to learn colorings translating to effective policies parameterized by as few as 17 weight parameters, providing > 90% compression over vanilla policies and 6x compression over state-of-the-art compact policies based on Toeplitz matrices (Choromanski et al., 2018), while still maintaining good reward. We believe that our work is one of the first attempts to propose a rigorous approach to training structured neural network architectures for RL problems that are of interest especially in mobile robotics (Gage, 2002) with limited storage and computational resources.

Summary: This paper focuses on neural architecture search for constructing compact RL policies. It combines ideas from the popular ENAS and ES methods for optimisation. Recent work defined a family of compact policies by imposing a fixed Toeplitz structure. This paper introduces the so-called “chromatic network” architecture, which partitions weights of the RL network into tied sub-groups. The partitioning is searched with ENAS and shared weights are updated via ES. Experiments on continuous control benchmarks show that good performance can be obtained using a very small number of parameters. Favourable reward-compression outcomes can be achieved compared to some baseline alternatives.
To my understanding the main contributions are (1) the Chromatic parameter sharing scheme and (2) the combined ENAS+ES learning procedure.
Pros:
+ Chromatic networks provide a neat idea for managing compact networks via parameter sharing. (Although not dramatically novel given cited work by Salimans, Gaier, etc)
+ Searching both the partitioning + weight values with ENAS and ES provides a nice way to do learning & compact architecture search simultaneously.
+ Results generally show favourable compression/reward performance vs Mask, Toeplitz  and Circulant baselines.
Cons & Questions:
0. Motivation. The paper claims that RL can be high-dimensional with millions of parameters, and so there is a need to apply NAS to RL. But experiments are conducted in low dimensional environments with only 100s of parameters. The empirical validation doesn’t match the motivating scenario. We would want to see results on successful compression of larger vision-based networks to be fully persuaded. Without this it undermines the significance of the paper. The paper makes a claim about embedded devices, but this is unconvincing, as for these proprioceptive control tasks, the uncompressed networks are already small enough to run on most embedded devices.

1. ENAS vs Chromatic. Both use RL controllers. The main difference to ENAS seems to be the Chromatic sharing scheme, and use of ES rather than Backprop to update the weights. Some points of motivation/justification are not very clear after reading: (1) Why vanilla ENAS can’t be used for RL? (with the modification of replacing standard backdrop with any standard continuous control RL algorithm for weight updates). It is sort of claimed that ENAS can’t be used, it’s not obvious to me that this is true. (2) How is the  Chromatic strategy specific for RL? In principle it should also be usable in SL. Assessing its performance in SL would be more broadly relevant, interesting and significant since there is more prior compression work in SL.
2. The paper is a bit vague about how number of partitions is set. Is it a user-specifiable parameter, if so how do we set it? It’s suggested that partition number is included in the reward function, but then it’s even more unclear how to calibrate this to hit a particular performance or size target (or optimize performance for fixed size, size for fixed performance constraint) that a user may require in practice.
3. The real (wall-clock) running time for different NAS methods should be reported in the results.
4. Baselines. The presented results are a reasonable start, but they are all very much variations within the same family. One would like to know how the current method compares to: (i) Direct application of ENAS (See also Q1), (ii) Training the full network and applying standard NN pruning techniques (such as magnitude based), (iii) Baseline of training a comparably small sized network directly. (ivv) Importantly, since the size of networks here are very small, classic neuro-evolution algorithms from evolutionary robotics such as NEAT (Stanley & Miikkulainen) seem to provide a reasonable alternative. It’s hard to know if to be impressed with the current result or not without seeing the results for a decent NEAT-like competitor. (See also Q0).
Other:
A. A presentation of the algorithm with pseudocode would help the reader follow an overview of the algorithm.
B. Is the RL network/method used for the tasks the same as the network used in paper “Structured Evolution with Compact Architectures for Scalable Policy Optimization”? If so what is the source of the discrepancy between Tab 2 here and Tab 1 in that paper?

C. The title doesn’t make the content obvious to a naive prospective reader. At least something like “RL with Chromatic Networks for Compact Architecture Search” would be more informative.
D. Eq (1) has a vspace error.


While generative adversarial networks (GAN) have been widely adopted in various topics, in this paper we generalize the standard GAN to a new perspective by treating realness as a random variable that can be estimated from multiple angles. In this generalized framework, referred to as RealnessGAN, the discriminator outputs a distribution as the measure of realness. While RealnessGAN shares similar theoretical guarantees with the standard GAN, it provides more insights on adversarial learning. More importantly, compared to multiple baselines, RealnessGAN provides stronger guidance for the generator, achieving improvements on both synthetic and real-world datasets. Moreover, it enables the basic DCGAN (Radford et al., 2015) architecture to generate realistic images at 1024*1024 resolution when trained from scratch.

Post rebuttal: The authors' responses have addressed most of my concerns, and I've raised my rating from 3 to 6.
----------------------------------------
Summary:
This paper extends the discriminator of GAN to use a distributional output (multiple scalars) instead of a single scalar. As a result, the trained GAN becomes robust to the mode collapse.
Pros:
- The proposed method is clearly written and well-justified (e.g., Theorem 2).
- Extension of the relativistic GAN [1] to the proposed setting is interesting.
- The authors demonstrate that vanilla DCGAN architecture can generate high-fidelity (1024x1024) images.
Cons:
1. An ensemble of discriminators?
The authors use multiple scalars to consider diverse factors of the realness. However, it is simply an ensemble of discriminators [2] in a spirit. As each discriminator focus on different factors, it is not surprising that the generator becomes robust to the mode collapse. Also, recent work on mode collapse (e.g., [3]) shows better results on the mixture of gaussian experiments even using a single discriminator. At least, the authors should compare their method with the ensemble methods and claim the advantage over them.
2. Choice of the anchor distributions.
The choice of anchor distributions A_0 and A_1 are not specified. While the authors provide some partial results in Table 2, it would be worthwhile to clarify the experimental details and justify them.
3. Role of each outcome u_i?
The authors claim that each outcome u_i corresponds to the different factors of realness. However, the role of learned u_i is not investigated. Also, one may enforce u_i to learn different factors by promoting diversity of them, e.g., decrease their cosine similarity [4].
Minor comments:
- The word "support" [5] is misused. The support itself means the set of non-zero elements, hence the authors should use the word "outcome" (or "sample") instead of "support".
- The notation is not consistent. For example, the authors may use "x \sim p_data(x)" (specify variable) or "z \sim p_z" (omit variable), but not both.
- Numbering is not consistent. For example, "Tab.4.2." should be changed to "Tab.2." for consistency.
[1] Jolicoeur-Martineau. The relativistic discriminator: a key element missing from standard GAN. ICLR 2019.
[2] Durugkar et al. Generative Multi-Adversarial Networks. ICLR 2017.
[3] Xiao et al. BourGAN: Generative Networks with Metric Embeddings. NeurIPS 2018.
[4] Elfeki et al. GDPP: Learning Diverse Generations Using Determinantal Point Process. ICML 2019.
[5] https://en.wikipedia.org/wiki/Support_(mathematics)


We present a reinforcement learning (RL) based guidance system for automated theorem proving geared towards Finding Longer Proofs (FLoP). FLoP focuses on generalizing from short proofs to longer ones of similar structure. To achieve that, FLoP uses state-of-the-art RL approaches that were previously not applied in theorem proving. In particular, we show that curriculum learning significantly outperforms previous learning-based proof guidance on a synthetic dataset of increasingly difficult arithmetic problems.

In the paper, the authors present a new algorithm for training neural networks used in an automated theorem prover using theorems with or without proofs as training data. The algorithm casts this training task as a reinforcement learning problem, and employs curriculum learning and the Proximal Policy Optimization algorithm to find appropriate neural network parameters, in particular, those that make the prover good at finding long proofs. The authors also propose a new dataset for theorems and proofs for a simple equational theory of arithmetic, which is again suitable for improving (via learning) and testing the ability of the prover for finding long proofs. The proposed prover is tested against existing theorem provers, and for the authors' dataset, it outperforms those provers.
I found it difficult to make up my mind on this paper. On the one hand, the paper tackles an interesting problem of improving an automated theorem prover via learning, in particular, its ability for finding long nontrivial proofs. Also, I liked a qualitative analysis of the failure of the curriculum learning for tackling hard tasks in the paper. On the other hand, I couldn't quite make me excited with the dataset used to test the prover in the paper. The dataset seems to consist of easy variable-free equational formulas about arithmetic that can be proved by evaluation. Of course, I may be completely wrong about the value of the dataset. Also, if the dataset includes variables and other propositional logic formulas, such as disjunction, negation and conjunction, so that the prover can be applied to any formulas from Peano arithmetic via Skolemization, I would be much more supportive for the paper. Another thing that demotivated me is that I couldn't find the discussion about the subtleties in using curriculum learning and PPO for the theorem-proving task in the paper. What are the possible design choices? Why does the authors' choice work better than others?
I added a few minor comments below.
* abstract, p1: "significantly outperforms previous learning-based". When I read the experimental result section, I couldn't quite get this sense of huge improvement of the proposed approach over the existing provers. Specifically, from Table 4, I can see FLoP performs better than rlCoP, but I wasn't sure that the improvement was that significant (especially because rlCoP might not have given a chance to be tuned to the type of questions used to train FLoP -- I may be wrong here). I suggest you to add some further explanation so that a reader can share your sentiment and excitement on the improvement brought by your technique.
* p2: The related work section is great. I learned a lot by reading it. Thanks.
* p4: I think that you used the latex citation command incorrectly in "learning Resnick ... Chen (2018)"
and "features Kaliszyk ... Kaliszyk et al. (2015a; 2018)".
* p6: discount factor) parameters related ===> discount factor), parameters related
* p8: a a well ==> a well


Characterizing the underlying mechanism of graph topological evolution from a source graph to a target graph has attracted fast increasing attention in the deep graph learning domain. However, there lacks expressive and efficient that can handle global and local evolution patterns between source and target graphs. On the other hand, graph topological evolution has been investigated in the graph signal processing domain historically, but it involves intensive labors to manually determine suitable prescribed spectral models and prohibitive difficulty to fit their potential combinations and compositions. To address these challenges, this paper proposes the deep Graph Spectral Evolution Network (GSEN) for modeling the graph topology evolution problem by the composition of newly-developed generalized graph kernels. GSEN can effectively fit a wide range of existing graph kernels and their combinations and compositions with the theoretical guarantee and experimental verification. GSEN has outstanding efficiency in terms of time complexity (O(n)) and parameter complexity (O(1)), where n is the number of nodes of the graph. Extensive experiments on multiple synthetic and real-world datasets have demonstrated outstanding performance.

The paper proposed a method to model the graph topological evolution from the spectral domain by developing a new generalized graph kernel. The new graph kernels cover many existing graph kernels as well as their combination and composition as special cases. The idea of spectral graph translation and its integration with deep learning is interesting, especially considering that most previous spectral graph neural networks only transform the graph signal instead of graph structures. However, I do have some concerns of papers.
1.	My major concern is the soundness of keeping eigenvectors unchanged in the evolution. Although the authors claim that in previous studies eigenvectors are found stable in evolution, it is very counter-intuitive, and I am not sure it is the case for all types of graphs. Let us look at the proof of Lemma 3.1, obviously $L^\prime$ does not necessarily have the same eigenvectors, and $U^TL^\prime U$ is not a diagonal matrix, so this loss is actually very large in many cases. That is to say, the evolution model does not have enough expressive power to recover the $L^\prime$.
2.	Spectral graph translation looks interesting, but the main idea comes from Kunegis et al. (2010). Despite of a new designed graph kernel and adding nonlinear activations, the contributions seem not so significant.
3.	In Kunegis et al. (2010), they consider the evolution of adjacency matrix $A$, but in this paper the authors use the Laplacian matrix $L$. If there any reason to make this choice? Also, I think some of the conclusions (e.g. stable eigenvectors) in Kunegis et al. (2010) may not work since $L$ is used instesd of $A$.
4.    The correlation metric is acceptable, but it will be much better if the authors can do more analysis. For example, why not add the link prediction task as in Kunegis et al. (2010)? BTW, is correlation analysis used before in previous graph evolution papers? (if so, please add a reference)


Given the variety of the visual world there is not one true scale for recognition: objects may appear at drastically different sizes across the visual field. Rather than enumerate variations across filter channels or pyramid levels, dynamic models locally predict scale and adapt receptive fields accordingly. The degree of variation and diversity of inputs makes this a difficult task. Existing methods either learn a feedforward predictor, which is not itself totally immune to the scale variation it is meant to counter, or select scales by a fixed algorithm, which cannot learn from the given task and data. We extend dynamic scale inference from feedforward prediction to iterative optimization for further adaptivity. We propose a novel entropy minimization objective for inference and optimize over task and structure parameters to tune the model to each input. Optimization during inference improves semantic segmentation accuracy and generalizes better to extreme scale variations that cause feedforward dynamic inference to falter.

Summary:
The following work proposes a test-time optimization over scales to improve semantic segmentation. Specifically, at test time, they iteratively optimize over the score and scale parameters of Shellhamer et al 2019, where a Gaussian receptive field is used to allow for dynamic scale adaptation of each convolutional layer. They optimize the parameters with respect to an entropy minimization objective. Experiments on PASCAL VOC, reported at multiple scales, demonstrate improvements in IOU over the vanilla architecture.
Strengths:
-The work was well-motivated
-Formulation is pretty elegant and outperforms the baseline
Weaknesses:
While I liked the somewhat elegant formulation of dynamic test-time scaling proposed by the following work, I don't think this work introduced many novel results nor insights
-Multiscale test-time inference is already standard in state-of-the-art architectures such as DeepLab. Specifically, DeepLab runs inference at multiple scales, then max pools the logit responses across all scales. (see Table 3 of https://arxiv.org/pdf/1511.03339.pdf)
-Using entropy as a measure of network uncertainty is a good idea, but also not a novel finding.
-COCO and Cityscapes would probably have been better choices for datasets with larger variations in scale
Improvements:
-Try comparing against the multiscale logit-max-pooling inference procedure as a baseline -- or demonstrate that the proposed technique can further improve upon the results of the logit-pooling technique.
-Some details weren't very well explained. Specifically, what is the $\theta_{score}$ task parameter for?
** Post Rebuttal Response
I'd like to thank the authors for clarifying some points in their response. Overall, I maintain that I think the optimization-based scale inference solution they present is interesting from an implementation standpoint, but the findings in this work did not yield sufficient new insight for the task. While I agree that their approach is fairly different from common approaches such as discrete image pyramids, a thorough quantitative comparison of these differences would make this work a lot stronger. As such, I maintain my original rating.


A key feature of intelligent behavior is the ability to learn abstract strategies that transfer to unfamiliar problems. Therefore, we present a novel architecture, based on memory-augmented networks, that is inspired by the von Neumann and Harvard architectures of modern computers. This architecture enables the learning of abstract algorithmic solutions via Evolution Strategies in a reinforcement learning setting. Applied to Sokoban, sliding block puzzle and robotic manipulation tasks, we show that the architecture can learn algorithmic solutions with strong generalization and abstraction: scaling to arbitrary task configurations and complexities, and being independent of both the data representation and the task domain.

This paper introduces a neural controller architecture for learning abstract algorithmic solutions to search and planning problems. By combining abstract and domain-specific components, the model is able to mimic two classical algorithms quite closely across several domains. The precision of the learning is very high; verified by generalization to substantially larger problem sizes and different domains. One notable conclusion is that Evolutionary Strategies is able to learn algorithmic solutions whose precision is on par with deterministic algorithms. The method of triggering learning based on curriculum level performance is a notable feature that nicely couples generalization progress with learning, and yields insightful learning curves.
Although the paper shows that learning abstract algorithmic solutions is possible, it is not clear that the framework is able to learn such solutions when the training signals have not already been broken down into their smallest constituent parts. In other words, it is not clear what this framework could be used for, since it appears the experimenter must already possess a complete specification of the target algorithm and breakdown of the domain. Is there a setting where this framework could be used to learn something the experimenter is not already aware of? Or is the main point that it is technically possible to get an NN to learn this behavior?
Although it is clear that models are achieved that satisfy R1-R3, it is not clear exactly what problem formulation is being considered. It would be very helpful if the paper included a formal problem definition so that the purpose of each framework component and differences w.r.t. prior work are clear.
Similarly, the motivation for each of the data dependent modules is not clear. Is there something fundamental about this particular decomposition into modules? Or are these just the modules that were necessary given the specifics of the algorithms that were learned in experiments? How does this framework generalize to other kinds of algorithms?
Are the comparison methods (DNC and stackRNN) unable to generalize to larger problem sizes? Including the full comparisons on generalization would give a more complete picture. Similarly, the figures are missing the comparisons for Learning to Plan for DNC and stackRNN.
Is the comparison w.r.t. training time in Figure 3c fair, since the proposed framework pretrains the submodules?
Is there a fundamental problem of DNC being addressed here? E.g., are there some critical types of submodules where making them differentiable is not an option?
Is the algorithm limited to cases where the number of actions at each state is equal? I.e., could it be applied to algorithms like shortest path in the DNC paper?
Finally, the last line talks about intriguing applications to the real world, but the running example in the paper is sorting. Is there some hypothetical but concrete example of how this framework could help in the real world, and do something better than a hard-coded classical algorithm? Or discover a new algorithm?
----------
After the rebuttal and discussion, I am convinced the paper is a useful contribution, and have increased my rating. However, I still think the presentation can be much improved to improve the clarity of the contribution. I would hope to see the following addressed in the final version:
1. More detailed and precise discussion of how the approach relates to prior work. As is, this discussion is informal and scattered throughout the paper. Enumerating the distinctions in one place would make the contribution much more clear.
2. The above would also help make the explicit contribution more clear. E.g., the "Contribution" paragraph currently does not contain the fact that the machine has only been applied to "planning" problems. This should be included to avoid making the contribution seem overly general.
3. More formal description of what each module does. Right now, they are described informally. Actually seeing the equations of what each computes would make it easier to understand how and why they all fit together.
I still see the main contribution as "It is technically possible to train the abstract controller of a neural computer for planning using NES, so that R1-R3 are satisfied." Ideally, the above clarifications would make it more clear that this is the main contribution, or that there are some other key contributions beyond this.


Interest has been rising lately towards methods representing data in non-Euclidean spaces, e.g. hyperbolic or spherical, that provide specific inductive biases useful for certain real-world data properties, e.g. scale-free, hierarchical or cyclical. However, the popular graph neural networks are currently limited in modeling data only via Euclidean geometry and associated vector space operations. Here, we bridge this gap by proposing mathematically grounded generalizations of graph convolutional networks (GCN) to (products of) constant curvature spaces. We do this by i) introducing a unified formalism that can interpolate smoothly between all geometries of constant curvature, ii) leveraging gyro-barycentric coordinates that generalize the classic Euclidean concept of the center of mass. Our class of models smoothly recover their Euclidean counterparts when the curvature goes to zero from either side. Empirically, we outperform Euclidean GCNs in the tasks of node classification and distortion minimization for symbolic data exhibiting non-Euclidean behavior, according to their discrete curvature.

Summary:
The authors propose using non-Euclidean spaces for GCNs. This is inspired by the recent work into non-Euclidean, and especially hyperbolic, embeddings. A few papers have recently tried to go past embeddings into building non-Euclidean models, requiring the lifting of standard operations in Euclidean space to non-Euclidean settings. This has been done in particular in hyperbolic space, but some datasets benefit from more complex spaces. The authors combine the mixed-curvature product formalism that uses products of Euclidean, hyperbolic, and spherical spaces for embeddings, but use these for GCN operations.
Doing this requires, in particular, developing a reasonable way to perform these operations in spherical space (since Euclidean is trivial and hyperbolic has been recently worked on). The authors do a nice lifting via complex operations, and both the hyperbolic and spherical spaces can devolve into the flat Euclidean space when their curvature goes to 0. The authors implement these GCNs, train the curvatures, and demonstrate performance improvements over Euclidean only versions on node classification on benchmark datasets. They also give a fairly nice introduction to all of these ideas in an extended appendix.
Strengths, Weaknesses, Recommendation:
This paper is reasonably interesting---it joins an effort to produce non-Euclidean models in a tractable way, which is fairly challenging, but could have a good impact. On the plus side, it's great that the authors added the nice development for the spherical operations, since that will come in handy for many models. The experiments are also good. On the downside, everything here is an extension of existing work, and the body of the paper is hard to read (though this may be inevitable, there's a lot of background to go over here). Overall I recommend accepting it; I think it's a solid contribution.
Comments:
- I don't understand why the authors say that their space "interpolates smoothly" just because the limit in the curvature is the same from the left and right side. For example, the absolute value function has the same limit from the left and the right at 0, but it's not differentiable there. Is it actually true that if we take the derivatives of the piecewise hyperbolic/spherical distance function that it's differentiable at c=0?
- There are a couple of recent papers that also consider hyperbolic GCNs, and in fact use  similar ideas for the aggregation and update steps (i.e., same lift to hyperbolic space). However, these were recently NeurIPS papers, and the text is not yet out, so I don't think this should affect the authors' independent work (and also the product part is new). I do recommend that the authors compare against those results in a future update of this work. The papers are "Hyperbolic Graph Convolutional Neural Networks" by Chami et al and Hyperbolic Graph Neural Networks by Liu et al.
- One thing that I didn't see discussed by the authors is that there are subtle difference between hyperbolic and spherical spaces. For example, the weighted midpoint of Def. 3.2 doesn't immediately extend to spherical space (or at least won't be unique). As an example, consider S^2 and the mean of two antipodal points on it---there's many choices for the midpoint. You probably have to limit the operation to a half-sphere (there's some ideas for this in Gu et al).
- For the synthetic tree, why is the number of edges 2(|V|-1) rather than |V|-1?
- Are the curvatures the same for each layer for the GCNs? This is an interesting point to discuss (some of the NeurIPS papers I mentioned train the curvature for each layer). Also, how do you select the number of factors of each type?
- Minor, but some of these citations can be updated. The "De Sa" et al 2018 arxiv citation is really Sala et al and is an ICML '18 paper. Similarly, Gulcehre et al is a 2019 ICLR paper, and so on. It's always good to get these right.
- Is there any actual empirical importance from recovering the Euclidean case exactly for 0 curvature? The reason I ask is that my experience is that the hyperboloid is typically easier to work with.
- One useful thing to point out in B.3.3 is that in general, it need not be a diffeomorphism for all of M for any manifold, which leads to non-uniqueness. In differential geometry, the "cut locus" is the region beyond which there is this non-uniqueness.
- In the appendix, the statement "Sarkar (2011) show that a similar statement as in Theorem 2 holds for a very general class of trees" is confusing to me. The "general class", as far as I know, is actually *all* trees, weighted or unweighted.


Neural networks have been shown vulnerable to adversarial samples. Slightly perturbed input images are able to change the classification of accurate models, showing that the representation learned is not as good as previously thought. To aid the development of better neural networks, it would be important to evaluate to what extent are current neural networks' representations capturing the existing features. Here we propose a way to evaluate the representation quality of neural networks using a novel type of zero-shot test, entitled Raw Zero-Shot. The main idea lies in the fact that some features are present on unknown classes and that unknown classes can be defined as a combination of previous learned features. To evaluate the soft-labels of unknown classes, two metrics are proposed. One is based on clustering validation techniques (Davies-Bouldin Index) and the other is based on soft-label distance of a given correct soft-label. Experiments show that such metrics are in accordance with the robustness to adversarial attacks and might serve as a guidance to build better models as well as be used in loss functions to create new types of neural networks. Interestingly, the results suggests that dynamic routing networks such as CapsNet have better representation while current deeper DNNs are trading off representation quality for accuracy.

=== A. Summary ===
This paper proposes to evaluate the representation (here, the output of 9 probabilities for 9 classes at the last softmax output layer) of an image taken from an unseen category by two metrics:
- The first metric (DBI) essentially measures how compact the learned manifold is (i.e. how nearby are the softmax probability vectors of images from an unseen category).
- The second metric basically measures how close a 9-D probability output vector ("soft labels" in the paper) of a classifier trained on the 9 classes is to a "ground-truth" which is the same 9-D vector but obtained when the classifier is trained on the full 10 CIFAR-10 classes.
The authors computed these "representation measures" for a set of 7 networks (6 convnets + CapsuleNet) trained on CIFAR-10.
Then, a set of white-box adversarial attacks were performed on these 7 networks and the results (both how accurate the classifier is, and the amount of L2 perturbations required to change a label) were used as a "robustness measures".
The authors then claim that the ranking of the 7 classifiers based on the representation measures match the rankings derived from the robustness measures.
The paper attempts to find the correlation between the representation quality of a classifier with its adversarial robustness. This general direction is important and worth pursuing!
However, the claimed correlation is very weakly supported by the evidence in the paper.
=== B. Decision ===
Reject.
The aim of connecting the representation quality with adversarial robustness is interesting!
However, this paper has several major issues:
0. The paper uses the wrong citation format of ICLR (changing from LastName et al. 2019 --> (12) ). The paper maybe 8.5 to 9 pages long if using the original format. If this format violation was accepted, it would be unfair to other submissions and be a bad example this citation violation would be OK.
1. The paper has many claims (e.g. "DBI metric matches extremely well") that were not supported by evidence or clarified.
2. The paper claims to study classifiers' "representation quality". However, all the experiments were conducted on only the 10-class CIFAR-10 dataset and the "representations" are taken at the softmax layer as opposed to some mid-CNN feature layer (e.g. the popular fc7 in AlexNet trained on the 1000-class ImageNet).
3. The key Definition in Sec. 2 that the paper hinges on has 0 references and is a debatable definition.
=== C. Suggestions for Improvement ===
- I'd advise the authors to perform careful literature review to identify where the gap to fill is before conducting the research. There is a ton of work that has been done in the intersection of zero-shot, adversarial examples, deep features, and classifier caliberation.
For example, [1] has looked at how out-of-distribution samples can be represented by the deep features of a classifier:
[1] Bendale, A., & Boult, T. E. (2016). Towards open set deep networks. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 1563-1572).
- This work [2] has shown adding a "center loss" to force the deep features to be closer to the feature centroid (i.e. related to your DBI measure) helps improve the adversarial robustness. Indeed, the visualization in Fig. 3 is very interesting and is consistent with the result in [2].
[2] Agarwal, C., Nguyen, A., & Schonfeld, D. (2019, September). Improving Robustness to Adversarial Examples by Encouraging Discriminative Features. In 2019 IEEE International Conference on Image Processing (ICIP) (pp. 3801-3505). IEEE.
- The word "representation quality" here is confusing when it refers to the soft labels, which is more relevant to *caliberation* than quality of deep features as the title implies. And the paper misses this body of work in the related work. See here: https://nicholas.carlini.com/writing/2019/all-adversarial-example-papers.html
- Try to avoid "extremely" or "clearly" in claims (and in general scientific writing) because they tend to be overclaims and it is hard for readers to interpret those intensifiers.
- I don't see how the rankings by DBI or AM are similar to the original rankings by adversarial measures at all (except for CapsNet). At least, reporting Pearson correlation (or any statistical similarity measure) would be more convincing.
- Section 2 is highly debatable and has no references for the "Definition".
- I'd perform this work on ImageNet, at least to follow the intuition in Sec. 2. With only 10 CIFAR-10 classes, it is unrealistic how to semantically represent a "frog" using the other 9 classes (truck, airplane, etc).