Strong improvements in neural network performance in vision tasks have resulted from the search of alternative network architectures. Prior work has shown that this search process can be automated and guided by evaluating candidate network performance following limited training (Performance Guided Architecture Search or PGAS). However, because of the large architecture search spaces and the high computational cost associated with evaluating each candidate network, further gains in computational efficiency are needed. Here we present a method termed Teacher Guided Search for Architectures by Generation and Evaluation (TG-SAGE) that produces up to an order of magnitude in search efficiency over PGAS methods. Specifically, TG-SAGE guides each step of the architecture search by evaluating the similarity of internal representations of the candidate networks with those of the (fixed) teacher network. We show that this procedure leads to significant reduction in required per-sample training and that, this advantage holds for two different search spaces of architectures, and two different search algorithms. We further show that in the space of convolutional cells for visual categorization, TG-SAGE finds a cell structure with similar performance as was previously found using other methods but at a total computational cost that is two orders of magnitude lower than Neural Architecture Search (NAS) and more than four times lower than progressive neural architecture search (PNAS). These results suggest that TG-SAGE can be used to accelerate network architecture search in cases where one has access to some or all of the internal representations of a teacher network of interest, such as the brain.

This work tries to accelerate neural architecture search by reducing the computational workload spend to evaluate a single architecture. Given a network with premature performance (i.e. one trained only for few epochs), it tries to compute the mature performance (accuracy achieved when training the network to completion). A score for each architecture is given by "P+TG" which considers the validation accuracy and the similarity of instance representations at different layers to a given teacher network. In an experiment against using the validation accuracy as a score only, they achieve higher validation accuracies on CIFAR-10/100. In a comparison against NASNet and PNASNet, their experiments indicate higher validation accuracy in orders of magnitudes faster run time.
This is a well-written paper with an innovative idea to forecast the mature performance. I am not aware of any other work using the internal representation in order to obtain that. However, there is plenty of other work aiming at accelerating the architecture search by early stopping based on premature performance. Hyperband [1] uses the premature performance (considered in this paper), many others try to forecast the learning curve based on the partial observed learning curve [2,3]. ENAS [4] learns shared parameters for different architectures. Predictions on batches are then used as a surrogate. SMASH [5] uses a hypernetwork to estimate the surrogate. While this work was partly mentioned, neither the close connection to this work is discussed nor serves any of these methods as a baseline.
Typically, the premise of automatic neural architecture search is that the user does not know deep learning well. However, the authors assume that a teacher exist which is an already high performing architecture. It is unclear whether this is a realistic scenario in real-life. How do you find the teacher's architecture in the first place? Does the method also work in cases where the teacher is performing poorly?
The P+TG score depends on a hyperparameter alpha. There is no empirical evidence supporting that alpha can be fixed once and applied for any arbitrary dataset. So I have concerns whether this approach is able to generalize over different datasets. The experiment in the appendix confirms this by showing that a badly chosen alpha will lead to worse performance.
Additionally, I think the paper would benefit from spending more space on explaining the proposed method. I would rather see more details about RDM and maybe a nice plot explaining the motivation rather than repeating NAS and TPE.
[1] Lisha Li, Kevin G. Jamieson, Giulia DeSalvo, Afshin Rostamizadeh, Ameet Talwalkar: Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization. Journal of Machine Learning Research 18: 185:1-185:52 (2017)
[2] Tobias Domhan, Jost Tobias Springenberg, Frank Hutter: Speeding Up Automatic Hyperparameter Optimization of Deep Neural Networks by Extrapolation of Learning Curves. IJCAI 2015: 3460-3468
[3] Bowen Baker, Otkrist Gupta, Ramesh Raskar, Nikhil Naik: Practical Neural Network Performance Prediction for Early Stopping. CoRR abs/1705.10823 (2017)
[4] Hieu Pham, Melody Y. Guan, Barret Zoph, Quoc V. Le, Jeff Dean: Efficient Neural Architecture Search via Parameter Sharing. ICML 2018: 4092-4101
[5] Andrew Brock, Theodore Lim, James M. Ritchie, Nick Weston: SMASH: One-Shot Model Architecture Search through HyperNetworks. CoRR abs/1708.05344 (2017)


Despite their enormous success, there is still no solid understanding of deep neural network's working mechanism. As such, researchers have demonstrated DNNs are vulnerable to small input perturbation, i.e., adversarial attacks. This work proposes the effective path as a new approach to exploring DNNs' internal organization. The effective path is an ensemble of synapses and neurons, which is reconstructed from a trained DNN using our activation-based backward algorithm. The per-image effective path can be aggregated to the class-level effective path, through which we observe that adversarial images activate effective path different from normal images. We propose an effective path similarity-based method to detect adversarial images and demonstrate its high accuracy and broad applicability.

This paper proposes a method for the detection of adversarial examples based on identification of critical paths (called "effective paths") in DNN classifiers. Borrowing from the analysis of execution paths of control-flow programs, the authors use back-propagation from the neuron associated from the final class decision to identify a minimal subset of input synapses accounting for more than a threshold proportion ("theta") of the total input weight. The identification process is then recursively applied at the preceding layer for those neurons associated with the selected minimal subset of synapses, forming a tree of synapses (the "effective path"). The authors then propose to compare the effective paths (actually, unions of paths) of different examples using simple structural dissimilarity measures, which they extend to allow comparison to a typical (aggregated) path for multiple examples drawn from a common class.
In their experimentation with their measure, they noted that examples generated by a number of adversarial attacks tend to be less similar to their first-ranked estimated class than normal examples are to their own first-ranked classes. Similarly, they note that these same adversarial attacks tend to be *more* similar to their second-ranked classes than normal examples are to their own second-ranked classes (as the authors point out, this is likely due to the increased likelihood of the second-ranked class of adversarial examples being the true class for the original example from which it was perturbed). The authors then propose the difference between these two similarities (that is, first-ranked dissimilarity minus second-ranked dissimilarity) as a characterization of adversarial examples.
The idea of using critical paths in the DNN to detect adversarial examples is interesting, and the authors deserve credit for showing that these critical paths (as defined in this paper) do show differences from those of normal examples. However, the originality of the approach is undercut by the recent work of Wang et al. (CVPR, 2018), which the authors acknowledge only in the discussion of experimental results. Although the details are different as to how critical paths are identified, and how adversarial examples can be detected using them, the strategies are definitely related - a more detailed explanation of this should have been given in the introduction of the paper. More troubling is the fact that a head-to-head experimental comparison is not provided, neither with Wang et al. nor with other state of the art detectors, other than a qualitative assessment of the capabilities of some detectors in Table 1. Note that even this qualitative discussion does not include some of the recent detection approaches, such as BPDA (Athalye et al., ICML 2018) or LID (Ma et al., ICLR 2018).
The question of how best to define critical paths and their similarities is still very much open - the authors' approach is rather simplistic and straightforward. For example, is their similarity measure biased towards the contributions from early layers? Can a layer-by-layer weighting of contributions improve the performance?
The authors do not always interpret their own experimental results correctly. For example, their results in Figures 7i and 7j don't really support their conclusion that performance "remains almost unchanged" when theta is in the range 0.5 - 1.0. Also, Figure 4 does not show that their effective path similarity is not *directly* "a great metric to distinguish between normal and adversarial" examples, because a large proportion of adversarial examples have scores that fall in the typical range for normal examples (however, there are differences in tendency which can be exploited, as the authors do show).
The organization of the paper is in some need of improvement. For example, the discussion of densities of "effective paths" (Section 2) comes well before the details of the choice of threshold value theta used to generate them (Section 4.1).
To summarize:
Pros:
* A good case is made for the use of critical paths as a way of differentiating adversarial examples from normal examples.
* The reported improvement in similarity of adversarial examples with respect to their second-ranked classes is particularly intriguing.
* The paper is generally well written and easy to follow.
Cons:
* The experimental treatment is insufficient; in particular, a more carefully considered experimental justification is needed with respect to other detection strategies.
* The question of how best to define critical paths and their similarities is still very much open.
* The authors do not always interpret their own experimental results correctly.
* The organization of the paper is in some need of improvement.


Efficient exploration plays a key role in reinforcement learning tasks. Commonly used dithering strategies, such as -greedy, try to explore the action-state space randomly; this can lead to large demand for samples. In this paper, We propose an exploration method based on the uncertainty in reward space. There are two policies in this approach, the exploration policy is used for exploratory sampling in the environment, then the benchmark policy try to update by the data proven by the exploration policy. Benchmark policy is used to provide the uncertainty in reward space, e.g. td-error, which guides the exploration policy updating. We apply our method on two grid-world environments and four Atari games. Experiment results show that our method improves learning speed and have a better performance than baseline policies.

This paper suggests an exploration driven by uncertainty in the reward space.
In this way, the agent receives a bonus based on its squared error in reward estimation.
The resultant algorithm is then employed with DQN, where it outperforms an e-greedy baseline.
There are several things to like about this paper:
- The idea of exploration wrt uncertainty in the reward is good (well... actually it feels like it's uncertainty in *value* that is important).
- The resultant algorithm, which gives bonus to poorly-estimated rewards is a good idea.
- The algorithm does appear to outperform the basic DQN baseline on their experiments.
Unfortunately, there are several places where the paper falls down:
- The authors wrongly present prior work on efficient exploration as "exploration in state space" ... rather than "reward space"... in fact prior work on provably-efficient exploration is dominated by "exploration in value space"... and actually I think this is the one that makes sense. When you look at the analysis for something like UCRL2 this is clear, the reason we give bonus on rewards/transitions is to provide optimistic bounds on the *value function*... now for tabular methods this often degenerates to "counts" but for analysis with generalization this is not the case: https://arxiv.org/abs/1403.3741, https://arxiv.org/abs/1406.1853
- So this algorithm falls into a pretty common trope of algorithms of "exploration bonus" / UCB, except this time it is on the squared error of rewards (why not take the square root of this, so at least the algorithm is scale-invariant??)
- Once you do start looking at taking a square root as suggested (and incorporating a notion of transition uncertainty) I think this algorithm starts to fall back on something a lot more like lin-UCB *or* the sort of bonus that is naturally introduced by Thompson (posterior) sampling... for an extension of this type of idea to value-based learning maybe look at the line of work around "randomized value functions"
- I don't think the experimental results are particularly illuminating when comparing this method to other alternatives for exploration. It might be helpful to distill the concept to simpler settings where the superiority of this method can be clearly demonstrated.
Overall, I do like the idea behind this paper... I just think that it's not fully thought through... and that actually there is better prior work in this area.
It could be that I am wrong, but in this case I think the authors need to include a comparison to existing work in the area that suggests "exploration by uncertainty in value space"... e.g. "deep exploration via randomized value functions"


INFORMATION-DIRECTED EXPLORATION FOR DEEP REINFORCEMENT LEARNING
Efficient exploration remains a major challenge for reinforcement learning. One reason is that the variability of the returns often depends on the current state and action, and is therefore heteroscedastic. Classical exploration strategies such as upper confidence bound algorithms and Thompson sampling fail to appropriately account for heteroscedasticity, even in the bandit setting. Motivated by recent findings that address this issue in bandits, we propose to use Information-Directed Sampling (IDS) for exploration in reinforcement learning. As our main contribution, we build on recent advances in distributional reinforcement learning and propose a novel, tractable approximation of IDS for deep Q-learning. The resulting exploration strategy explicitly accounts for both parametric uncertainty and heteroscedastic observation noise. We evaluate our method on Atari games and demonstrate a significant improvement over alternative approaches.

The authors propose a way of extending Information-Directed Sampling (IDS) to reinforcement learning. The proposed approach uses Bootstrapped DQN to estimate parametric uncertainty in Q-values, and distributional RL to estimate intrinsic uncertainty in the return. The two types of uncertainty are combined to obtain a simple exploration strategy based on IDS. The approach outperforms a number of strong baselines on a subset of 12 Atari 2600 games.
Clarity - I found the paper to be very well-written and easy to follow. Both the background material and the experimental setup were explained very clearly. The main ideas were also motivated quite well. It would have been nice to include a bit more discussion of why IDS is a good strategy, i.e. what are the theoretical guarantees in the bandit case? Section 3.2 could also provide a more intuitive argument.
Novelty - The paper essentially combines the IDS formulation of Kirschner & Krause, Bootstrapped DQN of Osband et al., and the C51 distributional RL method of Bellemare et al. Most of the novelty is in how to combine these ideas effectively in the deep RL setting, which I found sufficient.
Significance - Improving over existing exploration strategies for deep RL would be a significant achievement. While the results are impressive, I have a few concerns regarding some of the claims.
The subset of games used to evaluate the proposed approach seems to be biased towards games where there is either a dense reward or exploration is known to be easy. Almost every deep RL paper on exploration includes results for at least some of the hard exploration games (see “Unifying Count-Based Exploration and Intrinsic Motivation”). Why were these games excluded from the evaluation? The results would be much stronger if results on all 57 games were included.
The main difference between DQN-IDS and C51-IDS is that C51-IDS will tend to favor actions with lower return uncertainty. Doesn’t this mean that the improved performance of C51-IDS is due to an improved ability to exploit rather than explore? If this is indeed the case, then I would expect more evidence that this doesn't come at a cost of reduced performance on tasks where exploration is difficult.
Finally, the comparison between Bootstrapped DQN and DQN-IDS conflates the exploration strategies (IDS vs Thompson sampling) with the choice of optimizer (Adam vs RMSProp), so the claim that simply changing the exploration strategy to IDS leads to a major improvement is not valid. It would be interesting to see results for Bootstrapped DQN using the authors’ implementation and choice of optimizer to fully separate the effect of the exploration strategy.
Overall quality - This is an interesting paper with some promising results. I’m not convinced that the proposed method leads to better exploration, but I think it still makes a valuable contribution to the work on balancing exploration and exploitation in RL.
-------
The rebuttal and revisions addressed some of my concerns so I am increasing my score to 7


STABLE RECURRENT MODELS
Stability is a fundamental property of dynamical systems, yet to this date it has had little bearing on the practice of recurrent neural networks. In this work, we conduct a thorough investigation of stable recurrent models. Theoretically, we prove stable recurrent neural networks are well approximated by feed-forward networks for the purpose of both inference and training by gradient descent. Empirically, we demonstrate stable recurrent models often perform as well as their unstable counterparts on benchmark sequence tasks. Taken together, these findings shed light on the effective power of recurrent networks and suggest much of sequence learning happens, or can be made to happen, in the stable regime. Moreover, our results help to explain why in many cases practitioners succeed in replacing recurrent models by feed-forward models.

This is an interesting paper that I expect will generate some interest within the ICLR community and from deep learning researchers in general. The definition of stability is both intuitive and sound and the connection to exploding gradients is perhaps the most interesting and useful part of the paper. The sufficient conditions yield practical techniques for increasing the stability of, e.g., an LSTM, by constraining the weight matrices. They also show that stable recurrent models can be approximated by models with finite historical windows, e.g., truncated RNNs. Experiments in Sec 4 suggest that stable models produced by constraining standard RNN architectures can compete with their unconstrained unstable counterparts, and often without necessitating significant changes to architecture or hyperparameters. The perhaps most interesting observations are in Sec 4.3, in which the authors claim that even fundamentally unstable models, e.g., unconstrained RNNs, often operate in a stable regime, at least when being applied to in-sample data. I lean toward acceptance at the moment, but I am eager to discuss with the authors and other reviewers as I am not 100% confident that I fully understood the theory.
SUMMARY
This paper proposes a simple, generic definition of “stability” for recurrent, non-linear dynamical systems such as RNNs: that given two hidden states h, h’, the difference between their updated states given input x is bounded by the product between the difference between the states themselves and a small multiplier. The paper then immediately draws a connection between stability, asserting that unstable models are prone to gradient explosions during gradient descent-based training. In Sec 2.2, the paper presents sufficient conditions for basic RNNs and LSTMs to be stable. Secs 3.2 and 3.3 argue that stable recurrent models can be approximated by feedforward models during both inference and training with a finite history horizon, such as a RNN with a truncated history. Experiments in language and music modeling substantiate this claim: constrained, stable models are competitive with standard unconstrained models. Sec 4.3 sheds some light on this phenomenon, arguing that there is a weaker form of data-dependent stability and that even unstable models may operate in a stable regime for some problems, thus explaining the parity between stable and unstable models.
STRENGTHS
* This paper is surprisingly engaging and easy to read.
* The theorems are clearly stated and the proofs appear sound to me, though I will admit that I am not confident that I would catch a significant bug.
* This paper provides a new (to me, anyway) and thought-provoking analysis of RNNs. In particular, I was especially interested in the observation that stable models can be approximated by truncated models and that there is a connection between stability and long-term dependencies. This seems consistent with the fact that for many problems, non-recurrent models (ConvNets, Transformers, etc.) are often competitive with more complex architectures.
WEAKNESSES
* In practice it seems as though stability may depend on not only choice of  model architecture but also the data themselves. There is probably no good way to know a priori what the stability characteristics of a given data set are, making it tough to apply the ideas of this paper in practice
* The literature review seems a bit limited and appears to ignore the growing body of work on constraining RNN weight matrices to address both exploding and vanishing gradients. For example, I am pretty confident that the singular thresholding trick for renormalizing neural net weights has been  described in the literature previously.
* Although stable and unstable models appear to be competitive in experiments, the theoretical analysis provides no insights into stability and how it relates to accuracy.


We propose a framework for learning a Kolmogorov model, for a collection of binary random variables. More specifically, we derive conditions that link (in the sense of implications in mathematical logic) outcomes of specific random variables and extract valuable relations from the data. We also propose an efficient algorithm for computing the model and show its first-order optimality, despite the combinatorial nature of the learning problem. We exemplify our general framework to recommendation systems and gene expression data. We believe that the work is a significant step toward interpretable machine learning.

The paper considers a solution to a statistical association problem. The proposed solution involves a decomposition they call a kolmogorov model (what sort is not justified in any way and confused me a lot). The decomposition has two parts 1) a discrete basis function that needs to be discovered and 2) a discrete distribution over the basis elements. The define an optimization problem (2) which has a data term and some binary and simplex constraints and they propose a relaxation and decomposition of this optimization problem. They go on to claim that (mutual) causal relations can be then inferred by inspecting the representations they have learnt but they give little details on how and what impacts this distinction has in practice.  This may be obvious to a subfield expert but it is not clear to me at all. The paper is locally consistent but I have trouble understanding the contribution and placing in the broad machine learning field.
I am not an expert in causality so I cannot evaluate the contribution but I can say that what interests me are section 2.2 and sec. 4-5. And they both require a lot better writing. 2.2 made things much more intuitive but i fail to see how the indicator variable annotations (action, scifi, etc.) can possibly come out of the data. I think this is an important point to support the interpretability claim. As for 4 I think there is room for intuition building there as well as limitations (e.g. what sort of inferences can be made and not etc.) Finally for 5 i find that very interesting but i find it difficult to have the right intuition about what the support condition means and how that helps in a practical setting.
pros:
- causality and interpretability are major directions of research
- seems like a valid contribution on an interesting problem
cons:
- the highlevel picture is relatively clear but i find important things very difficult to grasp
- the kolmogorov model definition i find confusing but i am not an expert in causality (the introduction should give some intuition about what that is and why it is a good idea).
- find it very hard to have a coherent picture of the limitations and assess the contributions of the paper.


VERIFICATION OF NON-LINEAR SPECIFICATIONS FOR NEURAL NETWORKS
Prior work on neural network verification has focused on specifications that are linear functions of the output of the network, e.g., invariance of the classifier output under adversarial perturbations of the input. In this paper, we extend verification algorithms to be able to certify richer properties of neural networks. To do this we introduce the class of convex-relaxable specifications, which constitute nonlinear specifications that can be verified using a convex relaxation. We show that a number of important properties of interest can be modeled within this class, including conservation of energy in a learned dynamics model of a physical system; semantic consistency of a classifier's output labels under adversarial perturbations and bounding errors in a system that predicts the summation of handwritten digits. Our experimental evaluation shows that our method is able to effectively verify these specifications. Moreover, our evaluation exposes the failure modes in models which cannot be verified to satisfy these specifications. Thus, emphasizing the importance of training models not just to fit training data but also to be consistent with specifications.

This paper uses convex relaxation to verify a larger class of specifications
for neural network's properties. Many previous papers use convex relaxations on
the ReLU activation function and solve a relaxed convex problem to give
verification bounds.  However, most papers consider the verification
specification simply as an affine transformation of neural network's output.
This paper extends the verification specifications to a larger family of
functions that can be efficiently relaxed.
The author demonstrates three use cases for non-linear specifications,
including verifying specifications involving label semantics, physic laws and
down-stream tasks, and show some experiments that the proposed verification
method can find non-vacuous bound for these problems. Additionally, this paper
shows some interesting experiments on the value of verification - a more
verifiable model seems to provide more interpretable results.
Overall, the proposed method seems to be a straightforward extension to
existing works like [2]. However the demonstrated applications of non-linear
specifications are indeed interesting, and the proposed method works well on
these tasks.
I have some minor questions regarding this paper:
1) For some non-linear specifications, we can convert these non-linear elements
into activation functions, and build an equivalent network for verification
such that the final verification specification becomes linear. For example, for
verifying the quadratic specification in physics we can add a "quadratic
activation function" to the network and deal with it using techniques in [1] or
[2].  The authors should distinguish the proposed technique with these existing
techniques. My understanding is that the proposed method is more general, but
the authors should better discussing more on the differences in this paper.
2) The authors should report the details on how they solve the relaxed convex
problem, and report verification time. Are there any tricks used to improve
solving time? What is the largest scale of network that the algorithm can
handle within a reasonable time?
3) The detailed network architecture (Model A, Model B) is not shown. How many
layers and neurons are there in these networks? This is important to show the
scalability of the proposed method.
4) For the Mujoco experiment, I am not sure how to interpret the delta values
in Figure 1. For CIFAR I know it is the delta of pixel values but it is not
clear about the delta in Mujoco model. What is the normal range of predicted
numbers in this model?  How does the delta compare to it? Is the delta very
small or trivial?
5) Is it possible to show how loose the convex relaxation is for a small toy
example? For example, the specification involving quadratic function is a
good candidate.
There are some small glitches in equations:
* In (4), k is undefined
* In (20), I am not sure if it is equivalent to the four inequalities after (22).
There are 4 inequalities after (22) but only 3 in (20).
Many papers uses convex relaxations for neural network verification. However
very few of them can deal with general non-linear units in neural networks.
ReLU activation is usually the only non-linear element than we can handle in
most neural network verification works. Currently the only works that can
handle other general non-linear elements are [1][2]. This paper uses more
general convex relaxations than these previous approaches, and it can handle
non-separable non-linear specifications. This is a unique contribution to this
field. I recommend accepting this paper as long as the minor issues mentioned
above can be fixed.
[1] "Efficient Neural Network Robustness Certification with General Activation
Functions" by Huan Zhang, Tsui-Wei Weng, Pin-Yu Chen, Cho-Jui Hsieh, Luca Daniel.
NIPS 2018
[2] "A dual approach to scalable verification of deep networks." by
Krishnamurthy Dvijotham, Robert Stanforth, Sven Gowal, Timothy Mann, and
Pushmeet Kohli. UAI 2018.


While many recent advances in deep reinforcement learning rely on model-free methods, model-based approaches remain an alluring prospect for their potential to exploit unsupervised data to learn environment dynamics. One prospect is to pursue hybrid approaches, as in AlphaGo, which combines Monte-Carlo Tree Search (MCTS)-a model-based method-with deep-Q networks (DQNs)-a model-free method. MCTS requires generating rollouts, which is computationally expensive. In this paper, we propose to simulate roll-outs, exploiting the latest breakthroughs in image-to-image transduction, namely Pix2Pix GANs, to predict the dynamics of the environment. Our proposed algorithm, generative adversarial tree search (GATS), simulates rollouts up to a specified depth using both a GANbased dynamics model and a reward predictor. GATS employs MCTS for planning over the simulated samples and uses DQN to estimate the Q-function at the leaf states. Our theoretical analysis establishes some favorable properties of GATS vis-a-vis the bias-variance trade-off and empirical results show that on 5 popular Atari games, the dynamics and reward predictors converge quickly to accurate solutions. However, GATS fails to outperform DQNs in 4 out of 5 games. Notably, in these experiments, MCTS has only short rollouts (up to tree depth 4), while previous successes of MCTS have involved tree depth in the hundreds. We present a hypothesis for why tree search with short rollouts can fail even given perfect modeling.

The submitted paper proposes GATS, a RL model combining model-free and model-bases reinforcement learning. Estimated models of rewards and dynamics are used to perform rollouts in MTCS without actually interacting with the true environment. The authors theoretically and empirically evaluate their approach for low depths rollouts. Empirically they show improved sample complexity on the Atari game Pong.
I think publishing negative research results is very important and should be done more often if we can learn from those results. But that is an aspect I feel this paper falls short at. I understand that the authors performed a great deal of effort trying to tune their model and evaluated many possible design choices, but they do not a provide a thorough investigation of the causes which make GATS "fail". I suggest that the authors try to understand the problems of MCTS with inaccurate models better with synthetic examples first. This could give insights into what the main sources of the problem are and how they might be circumvented. This would make the negative results much more insightful to the reader as each source of error is fully understood (e.g., beyond an error reate for predicting rewards which does not tell us about the distribution of errors which for example could have a big effect on the author's observations).
Another issue that needs further investigation is the author's "hypothesis on negative results". It would be great to experimentally underline the author's arguments. It is not trivial (at least to me) to fully see the "expected" interaction of learning dynamics and depths of rollouts. While MCTS should be optimal with any depths of rollouts given the true Q-function, the learning process seems more difficult to understand.
I would also like the authors to clarify one aspect of their theoretical analysis. e_Q is defined as the error in the Q-estimate for any single state and action in the main text. This appears to be inconsistent with the proof in the appendix, making the bound miss a factor which is exponential in H (all possible x_H that can be reached within H steps). This would change the properties of the bound quite significantly. Maybe I missed something, so please clarify.
Originality mainly comes from the use of GANs in MCTS and the theoretical analysis.
Strengths:
- Interesting research at the intersection of model-free and model-based research
- Lots of effort went into properly evaluating a wide range of possible design choices
- Mainly well written
Weaknesses:
- Missing depths in providing a deep understanding of why the author's expectations and empirical findings are inconsistent
- The authors use many tweaks and ideas to tune each component of their model making it difficult to draw conclusions about the exact contribution of each of these
- Error in the theoretical analysis (?)
Minor comment:
* The paper would benefit from improved and more self-contained figure captions.


EMERGING DISENTANGLEMENT IN AUTO-ENCODER BASED UNSUPERVISED IMAGE CONTENT TRANSFER
We study the problem of learning to map, in an unsupervised way, between domains A and B, such that the samples b ∈ B contain all the information that exists in samples a ∈ A and some additional information. For example, ignoring occlusions, B can be people with glasses, A people without, and the glasses, would be the added information. When mapping a sample a from the first domain to the other domain, the missing information is replicated from an independent reference sample b ∈ B. Thus, in the above example, we can create, for every person without glasses a version with the glasses observed in any face image. Our solution employs a single two-pathway encoder and a single decoder for both domains. The common part of the two domains and the separate part are encoded as two vectors, and the separate part is fixed at zero for domain A. The loss terms are minimal and involve reconstruction losses for the two domains and a domain confusion term. Our analysis shows that under mild assumptions, this architecture, which is much simpler than the literature guided-translation methods, is enough to ensure disentanglement between the two domains. We present convincing results in a few visual domains, such as no-glasses to glasses, adding facial hair based on a reference image, etc.

This paper tackles the task of content transfer. For a given type of images (frontal face shots), the goal is to transfer a particular localized property (e.g. glasses or facial hair) extracted from one image to another image of the same type (difference face). This is also known as the problem of guided image-to-image translation.
The problem is formalized as the one of learning to map two different domains, one domain being composed of images with the property/attribute of interest, the other one containing images without it. The problem is said to be ‘unsupervised’, i.e. there is no pairwise correspondences between images of the two domains (with/without attributes).
The novelty of the approach lies on
-	the loss, which is composed of three terms: two reconstruction losses and a domain confusion loss
-	the overall architecture and in particular the fact that images are represented as a combination of the output of two encoders: one encodes the face and the other encodes the property (e.g. glasses).
Overall comments:
+ a theoretical part discusses generalization bounds and the emergence of disentangled representations
+ visual results are appealing showing the suitability of the method to the considered task
- the discussion of the advantages of the proposed method could be improved
- the motivation for some of the experimental results is unclear (choice of experimental protocol and baselines).
- the scope of the method seems limited
Detailed comments:
I personally like the described model. The disentanglement mechanism is intuitive to understand, and seems well suited for this particular task, as qualitative evidence suggests. I am not sure if this approach would be applicable beyond the very specific scenario considered in the paper.
The paper emphasizes that the strength of the method lies on its simplicity w.r.t. competitors, and its better results. These two aspects could be better discussed.
Simplicity:
In several places the paper claims that the proposed approach is considerably simpler. Some parts hint to criteria for the ‘complexity’ comparison, such as Table 1 or a few sentences (e.g. “this allows us to train with many less parameters and without the need to applying excessive tuning”). It would be more convincing to have a dedicated discussion of the practical advantages of the simplicity claimed by this method, discussing e.g. training/testing time, memory footprint of the models, convergence properties, stability, etc.
Comparison:
The chosen baselines, i.e. MUNIT and DRIT are experimentally shown to perform poorly on the considered task. Yet although these methods were also developed for guided image translation, they were designed for a rather different application: style transfer. I am not sure these comparisons bring much insight on the performance of the method.
Experiments are conducted for a very specific task, on a single dataset. Would the method have broader application?
Experimental protocol:
I understand that such an approach is difficult to evaluate quantitatively but I am not sure what there is to learn from experiments reported in Table 3, as there is no point of comparison on this task. This could be clarified.
Additional comments:
-	The paper relies on the assumption that the distribution of persons with sunglasses and that of persons without them is the same, except for the sunglasses. This sounds like a strong requirement for the data used to train the network; it would be interesting to discuss the practical impact of this assumption, especially on the data requirement for the method to perform well
-	I found Figure 1 quite useful. A visual representation of the architecture and its associated description help follow the technical part.
-	I got confused with some of the claims in section 4.2. More generally, I found the technical part hard to follow.
-	The user study seems small: only 10 pairs of images are considered. How were those pair chosen? Is the set representative?


A long-held conventional wisdom states that larger models train more slowly when using gradient descent. This work challenges this widely-held belief, showing that larger models can potentially train faster despite the increasing computational requirements of each training step. In particular, we study the effect of network structure (depth and width) on halting time and show that larger models-wider models in particular-take fewer training steps to converge. We design simple experiments to quantitatively characterize the effect of overparametrization on weight space traversal. Results show that halting time improves when growing model's width for three different applications, and the improvement comes from each factor: The distance from initialized weights to converged weights shrinks with a power-law-like relationship, the average step size grows with a power-law-like relationship, and gradient vectors become more aligned with each other during traversal.

Understanding the effects of over-parametrization in neural network training has been a major challenge, albeit a lot of progress has been made in the past few years. The present paper is another attempt in this direction, with a slightly different point of view: the work characterizes the impact of over-parametrization in the number of iterations it takes an algorithm to converge. Along the way, it also presents further empirical observations such as the distance between the initial point and the final point and the angle between the gradients and the line that connects the initial and final points. Even though the observations presented are very interesting, unfortunately, the paper doesn't have the level of rigor required that would make it a solid reference.
The work presents its results somewhat clearly in the sense that one can simply reconstruct to probe in order to replicate the observations. This clarity is mainly due to the simplicity of the questions posed. There is nothing inherently wrong with simple questions, in fact, the kind of questions posed in the present paper are quite valuable, however, it lacks detailed study and rigor of a strong empirical work. Furthermore, the style of the exposition (anecdotal) and several obvious typos make the work look quite unfinished.
Here are some flaws and suggestions that would improve the work substantially:
- A deeper literature review would help guide the reader put the paper in a better context. Especially, the related work section is quite poor, how exactly do those papers appear related to the present work? Do they support similar ideas or do they propose different perspectives?
- The exposition should be made more to the point and concise (for instance 3rd paragraph of section 4.3 where it starts with Figure 5(a) What's meant by over-fitting regime, is it worse gen error, is it merely fitting tr data?.. How do we "know" from Figure 2, what's a strong evidence? Some concepts such as the capacity do not have precise and commonly agreed upon definitions, the paper uses those quite a bit and sometimes only later on the reader understands what it actually refers to... The misalignment section is also quite unclear.)
- The observations can be formalized and the curve fitting should be explained in further detail, the appendix touches upon simple cases but there is a strong literature behind those simple cases that could be quite useful for the purposes of the paper.
- The authors have a lot of data available at no point the power law decay and exponent fitting are discussed. For a paper whose main point is this precise scaling, this looks like a major omission unless there is a specific reason for it (other than the hardness of fitting exponents to power laws). Merely showing the observables in a log-log plot weakens the support of the main claims.
- The theoretical argument provided is just an elementary observation whose assumptions and conditions are not discussed. It is not a straightforward task, for instance, a suggestion for a theoretical result on the distance between the initial and final weights is presented here: Lemma 1 A.3 https://arxiv.org/abs/1806.07572 (distance shrink as the number of parameters increase consistent with the observations of the present paper) (note that this is in addition to the several early-2018 mean field approximations to NNs whose solutions are found in the limit where the number of  parameters tend to infinity)
- All the figures from 5 to 8 are presented very quantitatively such as looking at different layers and observing the percentage reductions. The message one can gain from such presentations are extremely limited and not systematic. I encourage the authors to formulate solid observables that can and should be tested in further detail.
Even though the paper is touching upon very interesting questions, at its current stage, it is not a good fit to be presented in a conference as it only presents anecdotal evidence. There is a lot of room to improve, but the good news is that most of the improvement should be straightforward.


THE UNUSUAL EFFECTIVENESS OF AVERAGING IN GAN TRAINING
We examine two different techniques for parameter averaging in GAN training. Moving Average (MA) computes the time-average of parameters, whereas Exponential Moving Average (EMA) computes an exponentially discounted sum. Whilst MA is known to lead to convergence in bilinear settings, we provide theto our knowledge -first theoretical arguments in support of EMA. We show that EMA converges to limit cycles around the equilibrium with vanishing amplitude as the discount parameter approaches one for simple bilinear games and also enhances the stability of general GAN training. We establish experimentally that both techniques are strikingly effective in the non-convex-concave GAN setting as well. Both improve inception and FID scores on different architectures and for different GAN objectives. We provide comprehensive experimental results across a range of datasets -mixture of Gaussians, CIFAR-10, STL-10, CelebA and ImageNet -to demonstrate its effectiveness. We achieve state-of-the-art results on CIFAR-10 and produce clean CelebA face images.

This paper tries to adapt the concept of averaging, well known is the game literature, to GAN training. In a simple min-max example the iterates obtained by gradient method do not converge to the equilibrium of the game but their average does. This work first provides intuitions on the potential benefits of exponential moving average (EMA) on a simple illustrative example and explore the effect of averaging on GAN.
In think that the approach of this paper is interesting. I particularly like the experiments on Celeb-A (Fig 6 and 7) that seem to show that the averaged iterates change more smoothly (with respect to the attributes of the faces) during the training procedure. Nevertheless, I have some concerns about the claims of the paper and the experimental process.
I'm surprised by the values of the inception score provided in Table 2 which do not seem to correlate with the sample quality in Fig. 3. Why did not you use the standard implementation of the inception score provided in Salimans et al. [2016]'s paper ?
I think that the effectiveness of EMA over uniform averaging is a bit overclaimed.
- From a theoretical point of view uniform averaging works better (at least in your example in 3.1): If you (uniformly) average the periodic orbit you get a converging iterate. Moreover, concerning to this toy example, note that this continuous analysis has been already introduced in [Goodfellow et al., 2016] and the Hamiltonian interpretation has been already provided in [Balduzzi et al. 2018].
However I think that the intuition on the vanishing magnitude of the oscillation provided by EMA is interesting.
- The continuous dynamics is actually different from the discrete one, I think that an analysis on the discrete case that is used in practice might be more insightful.
- The comparison with uniform averaging is not fair in the sense that uniform averaging has no hyperparameter to tune: In figure 6 uniform averaging performs better than a not well tuned EMA. A fair comparison would be for instance to propose a parametrized online averaging $\theta_{MA}^t = \frac{t - \alpha}{t} \theta_{MA}^{t-1} + \frac{\alpha}{t} \theta_t$ and to tune it the same way $\beta$ is tuned in EMA.
Refs:
Salimans, Tim, et al. "Improved techniques for training gans." Advances in Neural Information Processing Systems. 2016.
Goodfellow, I. (2016). NIPS 2016 tutorial: Generative adversarial networks. arXiv preprint arXiv:1701.00160.
Balduzzi, David, et al. "The Mechanics of n-Player Differentiable Games." ICML (2018).
Minor comments:
- In the introduction "gradient vector fields of the game may not be conservative (Mescheder et al. 2017)" and the related work "Mescheder et al. (2017) states that a reason for non-convergence is the non-conservative gradient
vector of the players.": the notion of conservative vs. non-conservative vector field is never mentioned in [Mescheder et al. 2017]. I think you are actually referring to the blog post on that paper https://www.inference.vc/my-notes-on-the-numerics-of-gans/ .
- In the Related work "can not"
- "In fact, it has recently been established that the smooth (continuous-time) analogues of first order methods such as online gradient descent (follow-the-regularized leader) in bilinear zero-sum games are recurrent (i.e. effectively periodic)
with trajectories cycling back into themselves. " can you provide a citation ?
- Some published papers are refereed as arxiv paper ( for instance (Mescheder et al. 2017) and (Mescheder et al. 2018)), you should cite the published version.


ABSTRACT Wilson et al. (2017) showed that, when the stepsize schedule is properly designed, stochastic gradient generalizes better than ADAM (Kingma & Ba, 2014). In light of recent work on hypergradient methods (Baydin et al., 2018), we revisit these claims to see if such methods close the gap between the most popular optimizers.
As a byproduct, we analyze the true benefit of these hypergradient methods compared to more classical schedules, such as the fixed decay of Wilson et al. (2017). In particular, we observe they are of marginal help since their performance varies significantly when tuning their hyperparameters. Finally, as robustness is a critical quality of an optimizer, we provide a sensitivity analysis of these gradient based optimizers to assess how challenging their tuning is.

Clarity: Below average
- The introduction would be easier to follow if you named Baydin's approach and your own approach, because in the 2-4 bullet points you say "this online scheme", and "the learning rate schedule", without being perfectly clear what you are talking about
- The last sentence of the introduction is meant to clearly state your hypothesis, so I was expecting "emphasize the value of *", i.e. either adaptive or non-adaptive methods, rather than just general 'tuning', which is self-apparently important.
Quality: Below average
This is a purely empirical study that does not go too deep. It is not quite a review paper, but only compares previous methods.
Pros:
I especially appreciate the sensitivity analysis, ie Fig 6. If only all ML papers had something like this to suggest the difficulty of setting hyperparameters for their proposed methods.
Cons:
- You should use mathematics to describe what you are talking about with adaptive stepsize in Sec 2.1. "these methods multiply the gradient with a matrix". Just giving one equation would be extremely helpful.
- If I understand correctly, you are interpreting the inverse-Hessian as used in Newton's method and other non-diagonal 'gradient conditioners' as types of stepsize. This is definitely interesting, but again it would be very simple to see what you are saying with an equation instead of starting with the phrase "stepsize" which is generally understood to be a scalar multiple on the gradient.
- I'm surprised you jump right into experiements after your background settings. It's apparent that this paper fundamentally relies on the Wilson (2017) hypergradient paper. Your paper should be more self-contained: 'hypergradient' is not even defined in this paper, is it?...
Especially:
How do you know that if you change the model architecture, data, and loss, that a similar result will occur? I imagine that it heavily relies on the data and model-- in other words, that the sensitivity is dependent on "how an algorithm reacts to a certain data/loss/model landscape". I'm trying to say that I'm not convinced these results generalize to any other situation than the one presented here (so does it really say anything about the different stepsize selection rules?)
Random side note:
Since your appendix is only a few lines, you could consider succinctly listing learning rates with set notation, for example {1e-n,5e-n : -5<n<1}.


DPSNET: END-TO-END DEEP PLANE SWEEP STEREO
Multiview stereo aims to reconstruct scene depth from images acquired by a camera under arbitrary motion. Recent methods address this problem through deep learning, which can utilize semantic cues to deal with challenges such as textureless and reflective regions. In this paper, we present a convolutional neural network called DPSNet (Deep Plane Sweep Network) whose design is inspired by best practices of traditional geometry-based approaches for dense depth reconstruction. Rather than directly estimating depth and/or optical flow correspondence from image pairs as done in many previous deep learning methods, DPSNet takes a plane sweep approach that involves building a cost volume from deep features using the plane sweep algorithm, regularizing the cost volume via a contextaware cost aggregation, and regressing the dense depth map from the cost volume. The cost volume is constructed using a differentiable warping process that allows for end-to-end training of the network. Through the effective incorporation of conventional multiview stereo concepts within a deep learning framework, DPSNet achieves state-of-the-art reconstruction results on a variety of challenging datasets.

The paper describes a method for learning a deep neural network for multi-view stereo. The overall network includes feature-extraction layers applied to all images, followed by a spatial-transformer network (which is differentiable, but with no learnable parameters) that is applied to warp these features from every matching image to the reference image's co-ordinate frame for a series of candidate depth planes, followed by concatenation of the reference and match image features and 3D convolution layers to form a cost volume. The cost volumes of different pairs are averaged, and additional layers are used to refine this cost volume while relying on the reference image's RGB features, followed by soft-max and an expectation over depth values to output the final depth at each pixel. The entire network is trained end-to-end and experiments show that it outperforms state-of-the-art methods for MVS by a significant margin on a number of datasets.
Overall, I have a positive view of the paper and believe it should be accepted to ICLR. However, I would like the authors to address the following issues:
- While the proposed network is complex, I do believe the description of the architecture could be a little better. It would be good to clarify that i indexes view (and N is the total number of views), and provide a few more definitions for the terms in equation (2): namely, are R and t the extrinsics of the reference camera or the i^th camera, etc. The overall approach is clear (for each plane, the method maps features from the paired camera to the reference camera  assuming all points in the the world lie on that plane), but it would be good to clarify the specifics. It might also be useful to emphasize that the cost-volume generation is per-pair (perhaps change the title of Sec 3.2) and that these volumes are averaged for all pairs.
- It might also be useful to apply the algorithm to the rectified binocular stereo case (where the warping and definition of planes by disparity are much simpler), and show comparisons to the many stereo algorithms on datasets like KITTI. At some level, the proposed algorithm can be thought of taking approaches proved to be successful for rectified binocular stereo and generalizing them (by generic warping + plane sweep) to the multi-view case. Hence, such comparisons could be illuminating. (Note: the method doesn't need to outperform the state-of-the-art there, but the results would be informative).
- I do believe the paper would significantly benefit from more discussion of DeepMVS since it's clearly the closest to this method (also solves MVS by deep networks + plane sweep). DeepMVS also learns the matching cost for cost volume generation, and the major difference seems to be that this method is learned end-to-end. It would be better to have a more detailed discussion of the differences (the current discussion at the end of Sec 2 is a little short on details)---architectures, super-vision at the end of the cost-volume vs end-to-end, etc.


Neural attention (NA) has become a key component of sequence-to-sequence models that yield state-of-the-art performance in as hard tasks as abstractive document summarization (ADS), machine translation (MT), and video captioning (VC). NA mechanisms perform inference of context vectors; these constitute weighted sums of deterministic input sequence encodings, adaptively sourced over long temporal horizons. Inspired from recent work in the field of amortized variational inference (AVI), in this work we consider treating the context vectors generated by softattention (SA) models as latent variables, with approximate finite mixture model posteriors inferred via AVI. We posit that this formulation may yield stronger generalization capacity, in line with the outcomes of existing applications of AVI to deep networks. To illustrate our method, we implement it and experimentally evaluate it considering challenging ADS, VC, and MT benchmarks. This way, we exhibit its improved effectiveness over state-of-the-art alternatives.

This paper presents a new way to construct the context vector in seq2seq networks. Specifically, the context vector in the proposed model is treated as a latent variable with a normal prior. The posterior is "defined" as a mixture of the latent presentations of the input sequence and the mixing weight is the attention. To do inference, maximising VAE-like "ELBO" is used. The proposed model is reported to achieve better performance in the seq2seq tasks including document summarisation and video caption.
My comments are as follows:
1. Although the authors claim a full statistical treatment of the context vector, the proposed model is not a proper statistical setting to me. I do not mean the setting is wrong, but it is really weird to me. The paper treats the context vector as a latent variable, which is used to generate the target sentence (take document summarisation task for example). The proper setting is to use the target sentence to do inference of the posterior of the context vector. However, the paper directly defines the format the "posterior", which is a mixture of the hidden representation of the source sentences, ignoring the target sentence. Therefore, I do not think this is the true posterior of the context vector. Moreover, seriously speaking, the "ELBO" of the proposed model is not a real ELBO. It is a training object function looked like an ELBO. Therefore, the difference of the proposed context vector with the standard one is instead of using a fixed hidden representation, the proposed one uses a stochastic one drawn from a Gaussian, the covariance of which is from a neural network.
2. To me, it is less intuitive to analyse why the performance gain comes from, given the proposed context vector works so better than the previous ones. I am expecting the authors to give some intuitive explanations and empirical results on this.
3. Minor comments on clarity: The abstract says both the mean and the covariance are modelled with neural networks, but in the paper, it seems that only the covariance is from a neural network.


SOFT Q-LEARNING WITH MUTUAL-INFORMATION REGULARIZATION
We propose a reinforcement learning (RL) algorithm that uses mutual-information regularization to optimize a prior action distribution for better performance and exploration. Entropy-based regularization has previously been shown to improve both exploration and robustness in challenging sequential decision-making tasks. It does so by encouraging policies to put probability mass on all actions. However, entropy regularization might be undesirable when actions have significantly different importance. In this paper, we propose a theoretically motivated framework that dynamically weights the importance of actions by using the mutualinformation. In particular, we express the RL problem as an inference problem where the prior probability distribution over actions is subject to optimization. We show that the prior optimization introduces a mutual-information regularizer in the RL objective. This regularizer encourages the policy to be close to a nonuniform distribution that assigns higher probability mass to more important actions. We empirically demonstrate that our method significantly improves over entropy regularization methods and unregularized methods.

The authors take the control-as-inference viewpoint and learn a state-independent prior (which is typically held fixed). They claim that this leads to better exploration when actions have different importance. They relate this objective to a mutual information constrained RL objective in a limiting case. They then propose a practical algorithm, MIRL and compare their algorithm against DQN and Soft Q-learning (SQL) on 19 Atari games and demonstrate improvements over both.
Generally I found the idea interesting and at a high level the deficiency of entropy regularization makes sense. However, I had great trouble understanding the reasoning behind their method and did not find the connection to mutual information helpful. Furthermore, I had a number of questions about the experiments. If the authors can clarify their motivation and reasoning and strengthen the experiments, I'd be happy to raise my score.
In Sec 3.1, why is it sensible to optimize the prior? Can the authors give intuition for maximizing \log p(R = 1) wrt to the prior? This is critical for justifying their approach. Currently, the authors provide a connection to MI, but don't explain why this matters. Does it justify the method? What insight are we supposed to take away from that?
The experiments could be strengthened by addressing the following:
* What was epsilon during training? Why was epsilon = 0.05 in evaluation? This is quite high compared to previous work, and it makes sense that this would degrade MIRLs performance less than DQN and SQL.
* What is the performance of SQL if we use \rho as the action selector in \epsilon-greedy. This would help understand if the performance gains are due to the impact on the policy or due to the changes in the behavior policy.
* Plotting beta over time
* Comparing the action distributions for SQL and MIRL to understand the impact of the penalty. In general, a deeper analysis of the impact on the policy is important.
* Are their environments we would expect MIRL to outperform SQL based on your theoretical understanding? Does it?
* How many seeds were run per game?
* How and why were the 19 games selected from the full set?
Comments:
The abstract claims state-of-the-art performance, however, what is actually shown is that MIRL outperforms DQN and SQL.
With a fixed prior, the action prior can be absorbed into the reward (e.g., Levine 2018), so it is of no loss of generality to assume a uniform prior.
Could state that the stationary distribution is assumed to exist and be unique.
In Sec 3.1, why is the prior state independent?
In Sec 3.1, p(R = 1|\tau) is defined to be proportional to exp(\beta \sum_t r_t). Is this well-specified? How would we compute the normalizing constant since p(R = 0 | \tau) is not defined?
Throughout, I suggest that the authors not use the phrases "closed form" and "analytic" for expressions that are in terms of intractable quantities.
It should be noted that Sec 3.2 Optimal policy for a fixed prior \rho follows from Levine 2018 and others by transforming the fixed prior into a reward bonus.
In Sec 3.2, the last statement does not appear to be necessary for the next subsection. Remove or clarify?
I believe that the connection to MI can be simplified. Plugging in the optimal \rho into Eq 3, we can see that Eq 3 simplifies to \max_\pi E_q[ \sum_t \gamma^t r_t] - (1 - gamma)/\beta MI_p(s, a) where p(s, a) = d^\pi(s) * \pi(a | s) and d^\pi is the discounted state visitation distribution. Thus Eq 3 can be thought of as a lower bound on the MI regularized objective.
In Sec 4, the authors state the main difference between their soft operator and the typical soft operator. What other differences are there? Is that the only one?
Sec 5 references the wrong Haarnoja reference in the first paragraph.
In Sec 5, alpha_beta = 3 * 10^5. Is that correct?
=====
11/26
At this time, the authors have not responded to the reviews. I have read the other reviews and comments, and I'm not inclined to change my score.
====
12/7
The authors have addressed most of my concerns, so I have raised my score. I'm still concerned that the exploration epsilon is quite different than existing work (e.g., https://github.com/google/dopamine/tree/master/baselines).


QUATERNION RECURRENT NEURAL NETWORKS
Recurrent neural networks (RNNs) are powerful architectures to model sequential data, due to their capability to learn short and long-term dependencies between the basic elements of a sequence. Nonetheless, popular tasks such as speech or images recognition, involve multi-dimensional input features that are characterized by strong internal dependencies between the dimensions of the input vector. We propose a novel quaternion recurrent neural network (QRNN), alongside with a quaternion long-short term memory neural network (QLSTM), that take into account both the external relations and these internal structural dependencies with the quaternion algebra. Similarly to capsules, quaternions allow the QRNN to code internal dependencies by composing and processing multidimensional features as single entities, while the recurrent operation reveals correlations between the elements composing the sequence. We show that both QRNN and QLSTM achieve better performances than RNN and LSTM in a realistic application of automatic speech recognition. Finally, we show that QRNN and QLSTM reduce by a maximum factor of 3.3x the number of free parameters needed, compared to real-valued RNNs and LSTMs to reach better results, leading to a more compact representation of the relevant information.

After the discussion with authors, I am happy to recommend acceptance.
————————————————————
1.	In “Consequently, for each input vector of size N, output vector of size M, dimensions are split into four parts: the first one equals to r, the second is xi, the third one equals to yj, and the last one to zk to compose a quaternion Q = r1 + xi + yj + zk”, are you splitting dimension M or M\times N? And if you split M \times N (I believe that’s what you are doing), in which order you are splitting (row major right?) Please explain.
2.	I did not understand why authors didn’t go in the negative direction of the gradient in Eq. (10-11)?
3.	In section 3.4, authors mentioned “Moreover, an hyper-complex parameter cannot be simply initialized randomly and component-wise, due to the interactions between components.” which I strongly agree. But in Eq. (7) and (9) why the update rules and activation function are applied component wise?
4.	I really like the elegance in the parameter initialization. Couple of minor things here: (1) It’s better to mention in Eq. (16) why E(|W|) is 0 because of symmetry. (2) Reference should be 6.1 instead of 5.1.
5.	Another reasonable baseline will be using a complex network like (https://openreview.net/forum?id=H1T2hmZAb) and use the first two terms in Eq. (19) for representation. This will also possibly justify the usefulness of using higher order partials.
6.	The authors mentioned multiple times about the achieved state-of-the-art results without giving any citation. As a reader not well versed in the acoustic domain, it will be nice to see some references to cross-validate the claim made.
General Comments:
1.	I understand the necessity of defining RNN/ LSTM model in the space of quaternions. But unit quaternions can be identified with other spaces where convolution is defined recently, e.g., with S^3 (https://arxiv.org/abs/1809.06211). I can see that this paper is contemporary, but at least can authors comment on the applicability of this general method in their case? Given that in NIPS’18 the following paper talked about RNN model on non-Euclidean spaces (https://arxiv.org/pdf/1805.11204.pdf), one can extend these ideas to develop an RNN model in the space of quaternions. Authors should look into it rigorously as future directions? But at least please comments on the applicability.
2.	The experimental results section is somewhat weak, the overall claim of using fewer parameters and achieving comparable results is only validated on TIMIT data. More experimentation is necessary.
3.	In terms of technical novelty, though quaternion algebra is well-known, I like the parameter initialization algorithm. I can see the merit of this in ML/ vision community.
Pros:
1. Nice well grounded methodological development on well-known algebra. (simple but elegant, so that's good).
2. Nicely written and all the maths check out (that's good).
3. Experimental result on TIMIT dataset shows usefulness in terms of using fewer parameters (but still can achieve SOA results).
Cons:
1. See my comments above. I expect the authors to rebut/ address the aforementioned comments. Overall though simple but nice (and necessary) development of RNN/ LSTM framework in the space of quaternions.
2. Lacks extensive experimental validation.
My reason for my rating is mainly because of (1) lack of experimental validation. (2) being aware of the recent development of general RNN model on non-Euclidean spaces, I want some comments in this direction (see detailed comment and reference above).


Learning-to-learn or meta-learning leverages data-driven inductive bias to increase the efficiency of learning on a novel task. This approach encounters difficulty when transfer is not mutually beneficial, for instance, when tasks are sufficiently dissimilar or change over time. Here, we use the connection between gradient-based meta-learning and hierarchical Bayes (Grant et al., 2018) to propose a mixture of hierarchical Bayesian models over the parameters of an arbitrary function approximator such as a neural network. Generalizing the model-agnostic metalearning (MAML) algorithm (Finn et al., 2017), we present a stochastic expectation maximization procedure to jointly estimate parameter initializations for gradient descent as well as a latent assignment of tasks to initializations. This approach better captures the diversity of training tasks as opposed to consolidating inductive biases into a single set of hyperparameters. Our experiments demonstrate better generalization performance on the standard miniImageNet benchmark for 1-shot classification. We further derive a novel and scalable non-parametric variant of our method that captures the evolution of a task distribution over time as demonstrated on a set of few-shot regression tasks.

Summary:
This work tackles few-shot (or meta) learning, providing an extension of the gradient-based MAML method to using a mixture over global hyperparameters. Each task stochastically picks a mixture component, giving rise to task clustering. Stochastic EM is used for end-to-end learning, an algorithm that is L times more expensive than MAML, where L is the number of mixture components. There is also a nonparametric version, based on Dirichlet process mixtures, but a large number of approximations render this somewhat heuristic.
Comparative results are presented on miniImageNet (5-way, 1-shot). These results are not near the state-of-the art anymore, and some of the state-of-art methods are simpler and faster than even MAML. If expensive gradient-based meta-learning methods are to be consider in the future, the authors have to provide compelling arguments why the additional computations pay off.
- Quality: Paper is technically complex, but based on simple ideas. In the case of
   infinite mixtures, it is not clear what is done in the end in the experiments.
   Experimental results are rather poor, given state-of-the-art.
- Clarity: The paper is not hard to understand. What is done, is done cleanly.
- Originality: The idea of putting a mixture model on the global parameters is not
   surprising. Important questions, such as how to make this faster, are not
   addressed.
- Significance: The only comparative results on miniImageNet are worse than the
   state-of-the-art by quite a margin (admittedly, the field moves fast here, but it
   is also likely these benchmarks are not all that hard). This is even though better
   performing methods, like Versa, are much cheaper to run
While the idea of task clustering is potentially useful, and may be important in practical use cases, I feel the proposed method is simply just too expensive to run in order to justify mild gains. The experiments do not show benefits of the idea.
State of the art results on miniImageNet 5-way, 1-shot, the only experiments here which compare to others, show accuracies better than 53:
- Versa: https://arxiv.org/abs/1805.09921.
   Importantly, this method uses a simpler model (logistic regression head models)
   and is quite a bit faster than MAML, so much faster than what is proposed here
- BMAML: https://arxiv.org/abs/1806.03836.
   This is also quite complex and expensive, compared to Versa, but provides good
   results.
Other points:
- You use a set of size N+M per task update. In your 5-way, 1-shot experiments,
   what is N and M? I'd guess N=5 (1 shot per class), but what is M? If N+M > 5,
   then I wonder why results are branded as 5-way, 1-shot, which to mean means
   that each update can use exactly 5 labeled points.
   Please just be exact in the main paper about what you do, and what main
   competitors do, in particular about the number of points to use in each task
   update.
- Nonparametric extension via Dirichlet process mixture. This is quite elaborate, and
   uses further approximations (ICM, instead of Gibbs sampling).
   Can be seen as a heuristic to evolve the number of components.
   What is given in Algorithm 2, is not compatible with Section 4. How do you merge
   your Section 4 algorithm with stochastic EM? In Algorithm 2, how do you avoid
   that there is always one more (L -> L+1) components? Some threshold must be
   applied somewhere.
   An alternative would be to use split&merge heuristics for EM.
- Results reported in Section 5 are potentially interesting, but entirely lack a
   reference point. The first is artificial, and surely does not need an algorithm of this
   complexity. The setup in Section 5.2 is potentially interesting, but needs more
   work, in particular a proper comparison to related work.
   This type of effort is needed to motivate an extension of MAML which makes
   everything quite a bit more expensive, and lacks behind the state-of-art, which
   uses amortized inference networks (Versa, neural processes) rather than
   gradient-based.


ROBUST ESTIMATION AND GENERATIVE ADVERSARIAL NETWORKS
Robust estimation under Huber's -contamination model has become an important topic in statistics and theoretical computer science. Statistically optimal procedures such as Tukey's median and other estimators based on depth functions are impractical because of their computational intractability. In this paper, we establish an intriguing connection between f -GANs and various depth functions through the lens of f -Learning. Similar to the derivation of fGANs, we show that these depth functions that lead to statistically optimal robust estimators can all be viewed as variational lower bounds of the total variation distance in the framework of f -Learning. This connection opens the door of computing robust estimators using tools developed for training GANs. In particular, we show in both theory and experiments that some appropriate structures of discriminator networks with hidden layers in GANs lead to statistically optimal robust location estimators for both Gaussian distribution and general elliptical distributions where first moment may not exist.

The paper considers the problem of robust high dimensional estimation in Huber’s contamination model. The algorithm is given samples from a distribution (1 - eps) * P + eps * Q, where P is a “nice” distribution (e.g. a Gaussian), eps is the fraction of contaminated points, and Q is some unconstrained noise distribution. The goal is then to estimate parameters of P as well as possible, given this noise. The settings they primarily consider in this paper are when P is a Gaussian with unknown mean and identity covariance, or when it is a Gaussian with unknown covariance. Classical estimators such as Tukey depth or matrix depth for these problems achieve optimal minimax rates, but are computationally expensive to compute. However, recent work of [1,2] propose efficient estimators for this problem that (nearly) achieve these rates.
This paper considers a different approach to this problem. They observe that in the case when P is a Gaussian, these classical depth functions (or minor variations thereof) can be written as the asymptotic limits of certain types of GANs. They then demonstrate that for specific choices for the architecture and regularization of the discriminator, the global optima of this GAN objective achieves minimax optimal error and rates in Huber’s contamination model. Unfortunately, they do not prove that their algorithm achieves these global minima. As a result they do not have any provable guarantees for their algorithms. However, they show experimentally that against many choices of noise distribution, their algorithms obtain good error, both for mean estimation and covariance estimation (at least, the JS-GAN seems to consistently succeed. They acknowledge that the TV-GAN seems to be unstable in certain regimes).
Pros:
- I think the question of finding algorithmic equivalents of Tukey median is a very interesting question, and this is an interesting attempt.
- I did not replicate their experiments on GANs, but the experimental numbers seem promising. However, I have some mixed feelings about this (see below).
Cons:
- A clear disadvantage of the approach to prior algorithmic work is that the algorithms proposed in the paper do not have provable guarantees. For settings such as secure machine learning, the lack of such guarantees is problematic. Given that previous works give efficient (i.e. practical) algorithms for these problems with provable guarantees, I am unclear how much impact this will have in practice.
- Given that TV-GAN is known to fail (as shown in Table 6), it is unclear how useful the numbers for it are in Table 1. Without these numbers, it then appears that JS-GAN and the filtering algorithm often achieve comparable results, although it is very interesting that JS-GAN is consistently slightly better.
- I feel that the authors fall short of their goal to make a good algorithmic analog of these depth-based estimators. This is a subtle but important point, so let me justify this. As the authors explain, the major advantage of such estimators would be that they are model-free: they should give robustness for a number of settings, not just Gaussians, but also elliptical distributions, sub-gaussian distributions, etc. However, the correspondence that the authors derive to their GAN formulation of depth heavily leverages the Gaussianity of the underlying distribution. Specifically, it leverages the fact that the Scheffe set between two Gaussians is a half-plane, which clearly fails for more general distributions. As a result, it appears to me that this variational formulation of depth succeeds only in a very model-specific setting. As a result, from a theoretical perspective it is unclear what advantage this formulation has.
Questions:
- How long does it take to train the GANs? Is it comparable to the runtime of the other algorithms?
- Can these algorithms work in the stronger notions of corruption considered in [1, 2]?
Overall conclusion:
The paper proposes a novel framework for robust estimation. However, in light of the previous provable and much simpler algorithms for robust estimation, in the end it seems to me that deep learning is an unnecessarily complicated approach to this problem. While the authors demonstrate some experimental improvement in the test cases they tried, the lack of provable guarantees for their approach limits the theoretical appeal of their paper. More conceptually, I am unconvinced that their approach is the correct approach to understanding algorithmic notions of depth, for the reasons described above.
[1] Kevin A Lai, Anup B Rao, and Santosh Vempala. Agnostic estimation of mean and covariance. In Foundations of Computer Science (FOCS), 2016 IEEE 57th Annual Symposium on, pp. 665–674. IEEE, 2016.
[2] Ilias Diakonikolas, Gautam Kamath, Daniel M Kane, Jerry Li, Ankur Moitra, and Alistair Stewart. Being robust (in high dimensions) can be practical. arXiv preprint arXiv:1703.00893, 2017.


The goal of network representation learning is to learn low-dimensional node embeddings that capture the graph structure and are useful for solving downstream tasks. However, despite the proliferation of such methods, there is currently no study of their robustness to adversarial attacks. We provide the first adversarial vulnerability analysis on the widely used family of methods based on random walks. We derive efficient adversarial perturbations that poison the network structure and have a negative effect on both the quality of the embeddings and the downstream tasks. We further show that our attacks are transferable since they generalize to many models, and are successful even when the attacker is restricted.

Brief Summary:
The authors present a novel adversarial attack on node embedding method based on random walks. They focus on perturbing the structure of the network. Because the bi-level optimization problem can be highly challenging, they refer to factorize a random walk matrix which is proved equivalent to DeepWalk. The experimental results show that their approach can effectively mislead the node embedding from multi-classification.
Quality:
This paper is well-written except for some minor spelling mistakes
Clarity:
This paper is quite clear and easy to follow.
Originality:
This work follow the proposal Qiu et al.(WSDM'18)'s proof and present a novel approach to calculate the loss when the network changes by A'.
Pros:
1. Detailed proofs presented in appendix
2. They present 6 questions and answer them with effective experiments.
3. They present a new way to attack node embedding by factorizing a equivalent matrix.
Cons:
1. I have noticed that Zügner et al.(KDD'18) present an adversarial attack method on GCN for graph data. I think it is reachable by the day the authors submitted this paper. This is opposite to the first sentence "Since this is the first work considering adversarial attacks on node embeddings there are no known
baselines" said in Section 4.
2. The author present the time analysis of their approach but the efficiency result of their approach is not presented.
3. To enforce misclassification of the target node t, the author set the candidate flip edges as edges around t. Does this mean only the node's local edges can mislead the target node from downstream tasks? I think the authors should consider more candidate edges but this may lead to larger time complexity.
4. Figure. 4 tells that low-degree nodes are easier to mis-classify. If the baseline method B_{rnd} randomly select edges to flip among the local area of node t, I think the result should be similar to the proposed approach on low-degree nodes because the flipped edges should be the same.
== I have read the rebuttal. Thanks for the response.


We explore a collaborative multi-agent reinforcement learning setting where a team of agents attempts to solve cooperative tasks in partially-observable environments. In this scenario, learning an effective communication protocol is key. We propose a communication architecture that allows for targeted communication, where agents learn both what messages to send and who to send them to, solely from downstream task-specific reward without any communication supervision. Additionally, we introduce a multi-stage communication approach where the agents co-ordinate via multiple rounds of communication before taking actions in the environment. We evaluate our approach on a diverse set of cooperative multi-agent tasks, of varying difficulties, with varying number of agents, in a variety of environments ranging from 2D grid layouts of shapes and simulated traffic junctions to complex 3D indoor environments. We demonstrate the benefits of targeted as well as multi-stage communication. Moreover, we show that the targeted communication strategies learned by agents are both interpretable and intuitive.

The authors present a study on multi-agent communication.
Specifically, they adapt communication to be targeted and multi-staged.
Experiments on  2 synthetic datasets and 1 3D visual dataset confirm that both additions are beneficial
Overall, this paper was somewhat clear and more importantly includes experiments on House3D, a more realistic dataset.
My main concern is the following: the method is not about targeting, but about selectively hearing.
If agents are sharing the reward then why should targeted communication be beneficial at all? Isn't the optimal strategy to just communicate everything to everyone? I understand that they should be selective at the listening side to properly integrate only the relevant information (so, attend over all received messages), but why should we expect the speaker to apriori know who this message should go to? Moreover, I don't really understand how targeted communication can even work (in the way the authors explain it) since the agents have partial information (e.g., in shapes they only see 5x5 around them), so they don't really know who is where --  but I could potentially see this working should the agents put information about their own identity and location.  So, given the positive results that the authors get, my understanding is that the signature doesn't have information about who should the recipient of the information be but more about what where the properties of the sender of this information.  So, based on my understanding, I don't feel that the flow of the story quite matches what is really happening and this might be very confusing for prospective readers. Can the authors elaborate on this, aim i getting things wrong?
There is literally no information about model size (or at least I wasn't able to find any). Is there any weight-sharing across agents? Do you obtain CommNets by using the implementations of the authors or by ablating the signature-part of your model? Moreover, why do agents have a limited view window on the SHAPES -- is (targeted) communication redundant when agents have full observability? The part about how multi-staged communication is implemented is quite cryptic at the moment -- is multi-staged the fact that the message is out-putted by processing with a recurrent unit? The messages is factorized into two parts k and u leading to a vector of size D -- what happens should we have one message of size D (rather than factorizing into 2), something like this would control for any improvements obtained from increases the parameters of the model.
Finally,  if the premises of the paper is to define more effective communication protocols, evident in the use of continuous communication, (rather than studying what form can multi-agent communication etc etc), a necessary baseline  (especially in cases where agents share reward), is to communicate the full observation (rather than a function of it).  This baseline is not presented here and it's absolutely necessary.


Continuously trainable models should be able to learn from a stream of data over an undefined period of time. This becomes even more difficult in a strictly incremental context, where data access to previously seen categories is not possible. To that end, we propose making use of a conditional generative adversarial model where the generator is used as a memory module through neural masking to emulate neural plasticity in the human brain. This memory module is further associated with a dynamic capacity expansion mechanism. Taken together, this method facilitates a resource efficient capacity adaption to accommodate new tasks, while retaining previously attained knowledge. The proposed approach outperforms state-of-the-art algorithms on publicly available datasets, overcoming catastrophic forgetting.

The proposed method tackles class-incremental continual learning, where new categories are incrementally exposed to the network but a classifier across all categories must be learned. The proposed method seems to be essentially a combination of generative replay (e.g. Deep Generative Replay) with AC-GAN as the model and attention (HAT), along with a growing mechanism to support saturating capacity. Quantitative results are shown on MNIST and SVHN while some analysis is provided on CIFAR.
Pros
 + The method combines the existing works in a way that makes sense, specifically AC-GAN to support a single generator network with attention-based methods to prevent forgetting in the generator.
 + The method results in good performance, although see caveats below.
 + Analysis of the evolution of mask values over time is interesting.
Cons

 - The method is very confusingly presented and requires both knowledge of HAT as well as more than one reading to understand. The fact that HAT-like masks are used for a generative replay approach is clear, but the actual mechanism of "growing capacity" is not made clear at all especially in the beginning of the paper. Further the contributions are not clear at all, since a large part of the detailed approach/equations relate to the masking which was taken from previous work. The authors should on the claimed contributions. Is it a combination of DGR and HAT with some capacity expansion?
 - It is not clear whether pushing the catastrophic forgetting problem into the generator is the best approach. Clearly, replaying data accurately from all tasks will work well, but why is it harder to guard against the generative forgetting problem than the discriminative one?
 - The approach also seems to add a lot of complexity and heuristics/hyper-parameters. It also adds capacity and it is not at all made clear whether the comparison is fair since no analysis on number of parameters are shown.
 - Relatedly, better baselines should be used; for example, if the memory used by the generative model is merely put to storing randomly chosen instances from the tasks, how will the results compare? Clearly storing instances bypasses the forgetting problem completely (as memory size approaches the dataset size it turns into the joint problem) and it's not clear how many instances are really needed per task, especially for these simpler problems. As such, I find it surprising that simply storing instances would do as poorly as stated in this paper which says cannot provide enough diversity.
 It also seems strange to say that storing instances "violates the strictly incremental setup" while generative models do not. Obviously there is a tradeoff in terms of memory usage, privacy, performance, etc. but since none of these methods currently achieve the best across all of these there is no reason to rule out any of the methods. Otherwise you are just defining the problem in a way that excludes other simple approaches which work.
 - There are several methodological issues: Why are CIFAR results not shown in a table as is done for the other dataset? How many times were the experiments run and what were the variances? How many parameters are used (since capacity can increase?) It is for example not clear that the comparison to joint training is fair, when stating: "Interestingly, DGM outperforms joint training on the MNIST dataset using the same architecture. This suggests that the strictly incremental training methodology indeed forced the network to learn better generalizations compared to what it would learn given all the data." Doesn't DGM grow the capacity, and therefore this isn't that surprising? This is true throughout; as stated before it is not clear how many parameters and how much memory these methods need, which makes it impossible to compare.
 Some other minor issues in the writing includes:
   1) The introduction makes it seem the generative replay is new, without citing approaches such as DGR (which are cited in the related work). The initial narrative mixes prior works' contributions and this paper's contributions; the contributions of the paper itself should be made clear,
   2) Using the word "task" in describing "joint training" of the generative, discriminative, and classification networks is very confusing (since "task" is used for the continual learning description too,
   3) There is no legend for CIFAR; what do the colors represent?
   4) There are several typos/grammar issues e.g. "believed to occurs", "important parameters sections", "capacity that if efficiently allocated", etc.).
 In summary, the paper presents what seems like an effective strategy for continual learning, by combining some existing methods together, but does not make it precise what the contributions are and the methodology/analysis make it hard to determine if the comparisons are fair or not. More rigorous experiments and analysis is needed to make this a good ICLR paper.