Reinforcement Learning (RL) problem can be solved in two different ways -the Value function-based approach and the policy optimization-based approach -to eventually arrive at an optimal policy for the given environment. One of the recent breakthroughs in reinforcement learning is the use of deep neural networks as function approximators to approximate the value function or q-function in a reinforcement learning scheme. This has led to results with agents automatically learning how to play games like alpha-go showing better-than-human performance. Deep Q-learning networks (DQN) and Deep Deterministic Policy Gradient (DDPG) are two such methods that have shown state-of-the-art results in recent times. Among the many variants of RL, an important class of problems is where the state and action spaces are continuous -autonomous robots, autonomous vehicles, optimal control are all examples of such problems that can lend themselves naturally to reinforcement based algorithms, and have continuous state and action spaces. In this paper, we adapt and combine approaches such as DQN and DDPG in novel ways to outperform the earlier results for continuous state and action space problems. We believe these results are a valuable addition to the fast-growing body of results on Reinforcement Learning, more so for continuous state and action space problems.

This paper combines elements of two existing reinforcement learning approaches, namely, Deep Q-learning Networks (DQN) with Prioritised Experience Replay (PER) and Deep Deterministic Policy Gradient (DDPG) to propose the Prioritized Deep Deterministic Policy Gradient (PDDPG) algorithm. The problem is interesting and there is a nice review of relevant work. The algorithm has a limited novelty with a simple modification of the DDPG algorithm to add the PER component. Experiment results show improvements in certain simulation environments. However, the paper lacks insight on how and why results are improved on some settings while performing worse than the others. Detailed comments are as follows:
1. Algorithm 1 is not self-contained. Yes, I understand that it is a slight modification to DDPG with changes being Line 11 and 16. But p_i^alpha is not defined anywhere in Algorithm 1. How the transition probabilties are updated on Line 16 is also not clear to me.
2. It would be better if multiple simulation runs on the same experiment can be performed to have a more reliable display of performance.
3. Section 6 is on Parameter Space Noise for Exploration. This is not the authors' proposed work so it is strange to have a separate section here. In the end of Section 1, the authors wrote that "We then use the concept of parameter space noise for exploration and show that this further improves the rewards achieved." This seems to be a bold claim from the varying performance displayed in Figure 2-4. Similar to Comment 2, more simulation runs and statistical tests need to be conducted to support this claim.


A deep reinforcement learning solution is developed for a collaborative multiagent system. Individual agents choose actions in response to the state of the environment, their own state, and possibly partial information about the state of other agents. Actions are chosen to maximize a collaborative long term discounted reward that encompasses the individual rewards collected by each agent. The paper focuses on developing a scalable approach that applies to large swarms of homogeneous agents. This is accomplished by forcing the policies of all agents to be the same resulting in a constrained formulation in which the experiences of each agent inform the learning process of the whole team, thereby enhancing the sample efficiency of the learning process. A projected coordinate policy gradient descent algorithm is derived to solve the constrained reinforcement learning problem. Experimental evaluations in collaborative navigation, a multi-predator-multi-prey game, and a multiagent survival game show marked improvements relative to methods that do not exploit the policy equivalence that naturally arises in homogeneous swarms.

This paper proposes a distributed policy gradient method for learning policies with large, collaborative, homogeneous swarms of agents.
Formalism / objective:
The setting is introduced as a "collaborative Markov team", so the objective is to maximise total team reward, as expressed in equation (3). This definition of the objective seems inconsistent with the one provided at line (14): Here the objective is stated as maximising the agent's return, L_n, after [k] steps of the agent updating their parameters with respect to L_n, assuming all other agents are static. I think the clearest presentation of the paper is to think about the algorithm in terms of meta-learning, so I will call this part the 'inner loop' from now on.
Note (14) is a very different objective: It is maximising the return of an agent optimising 'selfishly' for [k] steps, rather than the "collaborative objective" mentioned above. This seems to break with the entire premise of collaborative optimisation, as it was stated above.
My concern is that this also is reflected in the experimental results: In the food gathering game, since killing other agents incurs "a small negative reward", it is never in the interest of the team to kill other team-mates. However, when the return of individual agents is maximised both in the inner loop and the outer loop, it is unsurprising that this kind of behaviour can emerge. Please let me know if I am missing something here.
Other comments:
-The L_n(theta, theta_n) is defined and used inconsistently. Eg. compare line (9), L_n(theta_n, theta), with line below, L_n(theta, theta_n). This is rather confusing
-In equation (10) please specific which function dependencies are assumed to be kept? My understanding is that \theata_n is treated as a function of theta including all the dependencies on the policies of other agents in the environment?
-Related to above, log( pi_\theta_n ( \tan_n)) in line 16 is a function of all agents policies through the joint dependency on \theta. Doesn't that make this term extremely expensive to evaluate?
-Why were the TRPO_kitchensink and A3C_kitchensink set up to operate on the minimum reward rather than the team reward as it is defined in the original objective? It is entirely possible that the minimum reward is much harder to optimise, since feedback will be sparse.
-The survival game uses a discrete action space. I am entirely missing MARL baseline methods that are tailored to this setting, eg. VDN, QMIX, COMA etc to name a few. Even IQL has not been tried. Note that MADDPG assumes a continuous action space, with the gumble softmax being a common workaround for discrete action spaces which has not been shown to be competitive compared to the algorithms mentioned above.
-Algorithmically the method looks a lot like "Learning with Opponent Learning Awareness", with the caveat that the return is optimised after one step of 'self-learning' by each agent rather than after a step of 'Opponent-learning'. Can you please elaborate on the similarity / difference?
-Equation (6) and C1 are presented as contributions. This is the standard objective that's commonly optimised in MARL when using parameter sharing across agents.


Neural networks could misclassify inputs that are slightly different from their training data, which indicates a small margin between their decision boundaries and the training dataset. In this work, we study the binary classification of linearly separable datasets and show that linear classifiers could also have decision boundaries that lie close to their training dataset if cross-entropy loss is used for training. In particular, we show that if the features of the training dataset lie in a low-dimensional affine subspace and the cross-entropy loss is minimized by using a gradient method, the margin between the training points and the decision boundary could be much smaller than the optimal value. This result is contrary to the conclusions of recent related works such as , and we identify the reason for this contradiction. In order to improve the margin, we introduce differential training, which is a training paradigm that uses a loss function defined on pairs of points from each class. We show that the decision boundary of a linear classifier trained with differential training indeed achieves the maximum margin. The results reveal the use of cross-entropy loss as one of the hidden culprits of adversarial examples and introduces a new direction to make neural networks robust against them.

Due to the large variance in reviewer scores, I was asked to give this additional review.
Background: [Soudry et al. 2018] showed that the iterates of gradient descent, when optimizing logistic regression on separable data, converge to the L2 max-margin (SVM) solution for homogeneous linear separators (without bias). These results were later extended to other models and optimization methods.
This paper has two main results:
1)	It clarifies that the results of [Soudry et al. 2018] do not apply to logistic regression when the linear separator has a bias term (“b”). This is because the homogenous max margin solution in the extended [x,1] space is not the same as non-homogeneous max margin solution in the original space: the first has a penalty on the size of the bias term, i.e.
min_{w,b} ||w||^2 + b^2 s.t. y_n (w’x_n+b) >= 1
, while the latter does not:
min_{w,b} ||w||^2  s.t. y_n(w’x_n+b) >= 1
2)	It suggests using differential training to correct this issue.
However, I do not believe these contributions are enough for a publication in ICLR. First, (2) is simply a combination of two known results, as mentioned by Reviewer 2. Second, though I commend the authors for pointing out (1), I do not feel this by itself warrants a publication, for the following reasons:
a) It is very simple to explain (1) in only a few lines (as I did above). Therefore, it would be more informative just to write (1) as a comment on the original paper (the ICLR 2018 forum is still open), not as a completely new publication. For me, all the numerical demonstrations and examples of this simple issue did not add much.
b)	Regularizing the bias term usually does not make a significant difference to the sample complexity (see the end of section 15.1.1 in the textbook “Understanding Machine Learning: From Theory to Algorithms” by Shai Shalev Shwartz.). Furthermore, the main motivation behind [Soudry et al. 2018] was to explain implicit bias and generalization in deep networks, where there such max-margin results (which penalize all the parameters) could be used to derive generalization bounds (e.g., https://arxiv.org/abs/1810.05369).
c)	Lastly, the authors here say that “the solution obtained by cross-entropy minimization is different from the SVM solution”. This (as well as the title and abstract) may mislead the readers to think there is something wrong in the proofs of [Soudry et al. 2018] and later papers, and that logistic regression does not converge to the max-margin solution for homogeneous linear separators. However, the max-margin solution for homogeneous linear separators is also called the “max margin” or SVM solution (just for a different family). For example, see the previous paper on the topic [“Margin Maximizing Loss Functions”, Rosset et al. 2004] or section 15.1.1 in the textbook “Understanding Machine Learning: From Theory to Algorithms” by Shai Shalev Shwartz.  As I see it, the only issue in [Soudry et al. 2018] is the sentence “A bias term could be added in the usual way, extending x_n by an additional ’1’ component." which is confusing since it cannot be applied directly to the SVM solution. The authors should aim to pinpoint this issue, and clarify their phrasing to avoid such confusions.


Deep neural networks have demonstrated promising prediction and classification performance on many healthcare applications. However, the interpretability of those models are often lacking. On the other hand, classical interpretable models such as rule lists or decision trees do not lead to the same level of accuracy as deep neural networks and can often be too complex to interpret (due to the potentially large depth of rule lists). In this work, we present PEARL, Prototype lEArning via Rule Lists, which iteratively uses rule lists to guide a neural network to learn representative data prototypes. The resulting prototype neural network provides accurate prediction, and the prediction can be easily explained by prototype and its guiding rule lists. Thanks to the prediction power of neural networks, the rule lists from prototypes are more concise and hence provide better interpretability. On two real-world electronic healthcare records (EHR) datasets, PEARL consistently outperforms all baselines across both datasets, especially achieving performance improvement over conventional rule learning by up to 28% and over prototype learning by up to 3%. Experimental results also show the resulting interpretation of PEARL is simpler than the standard rule learning.

Review Summary
--------------
The paper presents a combination of rule lists, prototypes, and deep representation learning to fit classifiers that are said to be simultaneously "accurate" and "interpretable". While the topic is interesting and the direction seems novel, I don't think the work is quite polished or competitive enough to be accepted without significant revision. The major issues include non-competitive evaluation of what "interpretability" means, ROC AUC numbers that are indistinguishable from standard deep learning (RCNN) pipelines that use many fewer parameters, and many unjustified choices inside the method itself. The paper itself could also benefit from revision to improve flow and introduce technical ideas to be more accessible to readers.
Paper Summary
-------------
The paper presents a new method called "PEARL" (Prototype Learning via Rule Lists), which produces a rule list, a set of prototypes, and a deep feed-forward neural network that can embed any input data into a low-dimensional feature space. The primary intended application is classifying subjects into a finite set of possible disorders given longitudinal electronic health records with categorical features observed at T irregular time intervals.
The paper suggests learning a representation for each subject's data by feeding the EHR time series into a recurrent convolutional NN. The input data is a 2 x T array, with one row representing observed data and second row giving time delay between successive observations. The vector output of an initial convolutional RNN is then fed into a highway network to produce a final vector denoted "h".
Given an encoder to produce feature vectors, and a fixed rule list learned from data itself, the paper suggests obtaining a prototype for each rule by computing the average vector of all data that matches the given rule. The quality of these prototypes and related neural networks (for computing features and predicting labels from features) is then assessed via their loss function in Eq. 1: a weighted combination of how well the prototypes match the learned embeddings (distance to closest prototype) and how well the classifier predicts labels.  The core idea is that the embedding is learned to classify well while creating a latent space that looks like the prototypes of the rule list.
After training an embedding and NN classifier on a fixed rule list, it seems the data is reweighted according to some heuristic procedure to obtain better properties, then a new rule list is trained and the process repeats again. (I admit the reweight procedure's purpose was never clear to me).
Experiments are done on a proprietary heart failure EHR dataset and on a subset of MIMIC data.
Strengths
---------
* Seems original: I'm unaware of any other method connecting rule lists AND prototypes AND NNs
* Neat applications to healthcare
Limitations
-----------
* Interpretability evaluation seems weak: no human subject experiments, no quantiative metrics, unclear if rule-lists shown is an apples-to-apples comparison
* Prototypes themselves never evaluated
* Many design choices inside method not justified with experiments -- why highway networks + RCNNs?
Major Issues with Method
------------------------
## M1: Not clear that AUC difference between PEARL and baselines is significant
The major issue is that the presented approach does not seem significantly different in predictive performance than the baseline Recurrent CNN. Comparing ROC AUC, we have PEARL's 0.688 to RCNN'S 0.682 with stddev of 0.009 on the proprietary heart failure dataset, and PEARL's 0.769 to RCNN's 0.766 with stddev of 0.009. When AUCs match this closely, I struggle to believe one model is definitively better, especially given that the RCNN has 2x *fewer* parameters (8.4k to 18.4k).
If the counterargument is that the resulting "deep model" is not "interpretable", one should at least compare to a post-processing step where the decision boundary of the RCNN is the reference to which a rule list or decision tree is trained.
## M2: Interpretability evaluation not clear.
Isn't the maximum number of rules set in advance?
Additionally, prototypes are a key part of this work, but the learned prototypes are not evaluated at all in any figure (except to track avg. distance from prototype while training). If prototypes are so central to this work, I would like to see a formal evaluation of whether the learned prototypes are indeed better (in terms of distance, or inspection of values by an expert, or something else) than alternatives like Li et al.
## M3: Missing a good synthetic/small dataset experiment
Neither of the presented data tasks is particularly easy to understand for non-experts. I'd suggest creating an additional experiment where the audience of ML readers is likely to easily grasp whether a set of rule lists is "good" for the problem at hand... maybe create your own synthetic task or a UCI dataset or something, or even use the stop-and-frisk crime dataset from the Angelino et al. 2018 paper. Then you can compare against just a few relevant baselines (rule lists only or prototypes only). I think a better illustrative experiment will help readers grasp differences between methods.
## M4: How crucial is feature selection?
In each iteration, Algo. 1 performs feature selection before learning rules. Are any other baselines (trees, rule lists) allowed feature selection before the classifier is learned? What would happen to PEARL without feature selection? What method is used for selection? (A search of the document only has 'feature selection' occur once, in the Alg. itself, so it seems explanation is missing).
## M5: Why are multiple algorithm iterations needed?
Won't steps 3 and 4 of Alg. 1 result in the same rules every time? It's not clear then why on subsequent iterations the algorithm would improve. Perhaps it's just the reweighting of data that causes these steps to change?
Minor issues
------------
## Loss function notation confusing
Doesn't the rule list classifier s_R take the data itself X? Not the learned embedding h(X)? Please fix or clarify Eq. 1. I think you might clarify notation by just writing yhat(h(X)) if you mean the predicted label of some example as done by your NNs. Using "R" makes folks think the rule list is involved.
## Not clear why per-example reweighting is required
None of the experiments assess why per-example reweighting (lines 6-9 of Algo. 1) is required. Readers would like to see a comparison of performance with and without this step.
## Not clear or justified when "averaged" prototypes are acceptable
Are your "averaged" prototypes guaranteed to satisfy the rule they represent? Is taking the average of vectors that match a rule always guaranteed to also match the rule? I don't think this is necessarily true. Consider a rule that says "if x[0] == 0 or x[1] == 0, then ___".  Suppose the only matching vectors are x_A = [0 1] and x_B = [1 0]. The average vector is [0.5 0.5] which doesn't work.
## Several different measures of distance used without careful justification
Why use two different distances -- Euclidean distance to assess distance to prototypes for prototype assignment, and then cosine similarity when deciding which examples to upweight or downweight? Why not just use Euclidean distance for both (appropriately transformed to a similarity)?
Comments on Presentation
------------------------
Overall I think every section of the paper needs significant revision to improve a reader's ability to understand main ideas. Notation could be introduced slowly (explain purpose and dimension of every variable), assumptions could be clearly stated (e.g. each individual rule can have ANDs but not ORs), and design choices justified. You might try the test of giving the paper to a colleague and having them explain back the ideas of each section to you... currently I do not believe this version passes this test.
The introduction claims that "clinicians are often unwilling to accept algorithm recommendations without clarity as to the underlying reasoning", but I would be careful in blindly asserting this without evidence. For a nice argument about avoiding blind assumptions about what doctor's will and won't accept, see Lipton's 2017 paper "The Doctor Just Won't Accept That" (https://arxiv.org/abs/1711.08037)
Additionally, the authors should clarify more precisely what definition of interpretability is needed for their applications. Is it simplicity? Is it conceptual alignment with known medical facts? Is it the ability to transparently list the rules in plain English?
Line-by-line details
--------------------
## Sec. 2
When introducing p_j, should clarify this this is one prototype vector of many.
When defining p_j = f_j(X), can you clarify what dimensionality p_j has? Is it always the same size as each example's data vector x_i?
## Sec. 3
Fig. 2: I don't find this figure very easy-to-understand. It's clear that after embedding raw features to a new space, the learned rules are *different*, but it's not clear they are *better*.  None of the illustrated rules perfectly segments the different colors, for example. I guess the point is all the red dots are within one rule? But they aren't alone (there are blue and orange dots too), so it's still not clear this would be a better classifier.
For EHR datasets, are you assuming that events are always categorical? And that outcomes "y" are always discrete (one-of-L) variables? Or could y be real-valued?
Eq. 1: You should make notation clearly indicate which terms depend on \theta. Currently it seems that nothing is a function of \theta.
Eq. 1: Do you also find the prototype set P that minimizes this objective? Or is there another way to obtain P given parameters \theta? This is confusing just from reading the eqn.
What size is the learned representation h(X)? Is it a vector?
Eq. 6: Do you really need a "network" to compute the distance to each of the K prototypes? Can't you just compute these distances directly?
## Sec 4
"Mac OS 1.4" : Do you mean Mac OS version 10.4? Not clear this is relevant.
4.3 Case Study: How do I read these rules? Is this rule applied only if ALL conditions are true? or if any individual one is true ("or")? This is unclear.


MISGAN: LEARNING FROM INCOMPLETE DATA WITH GENERATIVE ADVERSARIAL NETWORKS
Generative adversarial networks (GANs) have been shown to provide an effective way to model complex distributions and have obtained impressive results on various challenging tasks. However, typical GANs require fully-observed data during training. In this paper, we present a GAN-based framework for learning from complex, high-dimensional incomplete data. The proposed framework learns a complete data generator along with a mask generator that models the missing data distribution. We further demonstrate how to impute missing data by equipping our framework with an adversarially trained imputer. We evaluate the proposed framework using a series of experiments with several types of missing data processes under the missing completely at random assumption.

This is a good paper, as we have good experimental evidence that the proposed method seems to have some advantage over baseline methods.
The authors measure the success of their algorithm by computing FID scores for the randomly inputed images. That is the authors use a metric which measures a distance between the distribution of the generated images and images in a dataset. This is fine and interesting to know, but people also care about the distance of the completed pixels from the ground truth (missing) values. (E.g. https://www.cs.rochester.edu/u/jliu/paper/Ji-ICCV09.pdf)
This is important, because in a real life application, one would pick the mode of the distribution of the missing samples, and not sample from that distribution as the authors seems to be doing in this paper.
I would therefore suggest adding experiments where authors pick the mode of the distribution and estimate an error metric such as root mean square error (RMSE or PSNR https://en.wikipedia.org/wiki/Peak_signal-to-noise_ratio )
I also found the 'marketing'/presentation of the algorithm little misleading, especially in the introduction, given that there exists another GAN based imputation algorithm. I think the authors should clearly state in the introduction that the other algorithm, abbreviated GAIN, exists as a GAN based missing data completion method. Then they should point out the differences of this algorithm from GAIN. Namely they should elaborate verbally on why learning the missing data distribution helps. Overall, what I am trying to say is, the key idea of this paper - that is learning the mask distribution - is not well motivated in this paper.
Despite my concerns above, I recommend an accept. The algorithm seems novel, and there is some experimental results to back it up.


Generating polyphonic music with coherent global structure is a major challenge for automatic composition algorithms. The primary difficulty arises due to the inefficiency of models to recognize underlying patterns beneath music notes across different levels of time scales and remain long-term consistency while composing. Hierarchical architectures can capture and represent learned patterns in different temporal scales and maintain consistency over long time spans, and this corresponds to the hierarchical structure in music. Motivated by this, focusing on leveraging the idea of hierarchical models and improve them to fit the sequence modeling problem, our paper proposes HAPPIER: a novel HierArchical PolyPhonic musIc gEnerative RNN. In HAPPIER, A higher 'measure level' learns correlations across measures and patterns for chord progressions, and a lower 'note level' learns a conditional distribution over the notes to generate within a measure. The two hierarchies operate at different clock rates: the higher one operates on a longer timescale and updates every measure, while the lower one operates on a shorter timescale and updates every unit duration. The two levels communicate with each other, and thus the entire architecture is trained jointly end-to-end by back-propagation. HAPPIER, profited from the strength of the hierarchical structure, generates polyphonic music with long-term dependencies compared to the state-of-the-art methods.

PRO's:
+good problem: generating polyphonic music with long-term structure
+reasonable approach: modification of SampleRNN: makes sense
CON's:
-doesn't work.
My fundamental critique of this paper is that, while the authors claim that their system " generates polyphonic music which maintains long-term dependencies", in fact what it generates it is not really polyphonic, nor-- more importantly-- does it demonstrate the kind of long-term structure present in the training set.
1) Polyphony: The model predicts a combination of monophonic melody plus chords (i.e. chord names such as "A+", "C7" etc). This is different from polyphony, in which the model would predict the actual voicing used for those chords. However, this could be seen as an error in terminology; if the authors claimed that they were predicting a monophonic musical voice plus chords, and they did that well, that would be absolutely fine. Generating a coherent melodic line that continues, along with the chords underneath it, would be a great achievement. However, that is not what happens here. For examples, in the provided examples, e.g. Measure 19 of Fig 4, Measures 1,3,4,5, ... of Fig 6, contain stylistically unusual combinations of chords and melodic lines. By "stylistically unusual", I simply mean that those examples are not consistent with the Nottingham dataset.  Furthermore, in my subjective opinion, the examples that I listed above also just don't really musically work. There is no question that in the right context, any of those particular combinations of chords and melody notes *could* be made to work: for example, the first measure of Fig 6 would be perfectly fine as the beginning of a different song. (E.g. it could be taken as a slight reharmonization of the opening of "lullaby of birdland", but that would require a coherent continuation. )
2) Long-term structure: It seems to me that one of the key things that this paper sets out to do is to get strong long-term dependencies. The motivation for the SampleRNN-inspired approach is to have generation at multiple time scales, for example. However, there is no evidence in the presented examples of long-term structure. Consider Fig 6, for example. Where is the long-term structure? A D major chord is frequently repeated with occasional A7. That is reasonable but it does not necessarily demonstrate long-term structure, anymore than learning that "q" is often followed by "u" demonstrates long-term structure. There is no melodic motif, there is no sense of 4-bar phrasing (or any other recurring such pattern that I can tell). In fact, all of the samples shown (Fig 4, 6, 7, 8) all end up with the chord D major played most of the time, after what appears to be a bit more variation in the first few chords.
The results of the listening test are strange to me (beyond some of the apples-to-oranges comparisons). I cannot comment on those without hearing the pairs of examples that were actually played. How were those pairs selected?
At the moment, it does not seem worthwhile for this review to get into details about exactly how the system works, in light of the problematic output. If there is reason for me to do so, I would gladly oblige. The authors do make a variety of choices that appear to be fairly sensible.
I would very much look forward to seeing a revised version of the system in future that produces the  kind of output that the system is intended to produce (and described as producing).


Antimalware products are a key component in detecting malware attacks, and their engines typically execute unknown programs in a sandbox prior to running them on the native operating system. Files cannot be scanned indefinitely so the engine employs heuristics to determine when to halt execution. Previous research has investigated analyzing the sequence of system calls generated during this emulation process to predict if an unknown file is malicious, but these models require the emulation to be stopped after executing a fixed number of events from the beginning of the file. Also, these classifiers are not accurate enough to halt emulation in the middle of the file on their own. In this paper, we propose a novel algorithm which overcomes this limitation and learns the best time to halt the file's execution based on deep reinforcement learning (DRL). Because the new DRL-based system continues to emulate the unknown file until it can make a confident decision to stop, it prevents attackers from avoiding detection by initiating malicious activity after a fixed number of system calls. Results show that the proposed malware execution control model automatically halts emulation for 91.3% of the files earlier than heuristics employed by the engine. Furthermore, classifying the files at that time improves the true positive rate by 61.5%, at a false positive rate of 1%, compared to a baseline classifier.

This paper attempts to train a predictor of whether software is malware. Previous studies have emulated potential malware for a fixed number of executed instructions, which risks both false negatives (haven’t yet reached the dangerous payload) and false positives (malware signal may be lost amidst too many other operations). This paper proposes using deep reinforcement learning over a limited action space: continue executing a program or halt, combined with an “event classifier” which predicts whether individual parts of the program consist of malware. The inputs at each time step are one of 114 high level “events” which correspond to related API invocations (e.g. multiple functions for creating a file). One limitation seems to be that their dataset is limited only to events considered by a "production malware engine", so their evaluation is limited only to the benefit of early stopping (rather than continuing longer than the baseline malware engine). They evaluate a variety of recurrent neural networks for classifying malware and show that all significantly underperform the “production antimalware engine”. Integrating the event classifier within an adaptive execution control, trained by DQN, improves significantly over the RNN methods.
It might be my lack of familiarity with the domain but I found this paper very confusing. The labeling procedure (the "production malware engine”) was left entirely unspecified, making it hard to understand whether it’s an appropriate ground-truth and also whether the DRL model’s performance is usable for real-world malware detection.
Also, the baseline models used an already fairly complicated architecture (Figure 3) and it would have been useful to see the performance of simple heuristics and simpler models.


HOW POWERFUL ARE GRAPH NEURAL NETWORKS?
Graph Neural Networks (GNNs) are an effective framework for representation learning of graphs. GNNs follow a neighborhood aggregation scheme, where the representation vector of a node is computed by recursively aggregating and transforming representation vectors of its neighboring nodes. Many GNN variants have been proposed and have achieved state-of-the-art results on both node and graph classification tasks. However, despite GNNs revolutionizing graph representation learning, there is limited understanding of their representational properties and limitations. Here, we present a theoretical framework for analyzing the expressive power of GNNs to capture different graph structures. Our results characterize the discriminative power of popular GNN variants, such as Graph Convolutional Networks and GraphSAGE, and show that they cannot learn to distinguish certain simple graph structures. We then develop a simple architecture that is provably the most expressive among the class of GNNs and is as powerful as the WeisfeilerLehman graph isomorphism test. We empirically validate our theoretical findings on a number of graph classification benchmarks, and demonstrate that our model achieves state-of-the-art performance.

This papers presents an interesting take on Weisfeiler-Lehman-type GNNs, where it shows that a WL-GNNs classification power is related to its ability to represent multisets. The authors show a few exemplar networks where the mean and the max aggregators are unable to distinguish different multisets, thus losing classification power. The paper also proposes averaging the node representation with its neighbors (foregoing the “concatenate” function) and using sum pooling rather than mean pooling as aggregator. All these observations are wrapped up in a GNN, called GIN. The experiments on Table 1 are inconclusive, unfortunately, as the average accuracies of the different methods are often close and there are no confidence intervals and statistical tests to help guide the reader to understand the significance of the results.
My chief concern is equating the Weisfeiler-Lehman test (WL-test) with Weisfeiler-Lehman-type GNNs (WL-GNNs). The WL-test relies on countable set inputs and injective hash functions. Here, the paper is oversimplifying the WL-GNN problem. After the first layer, a WL-GNN is operating on uncountable sets. On uncountable sets, saying that a function is injective does not tells us much about it; we need a measure of how closely packed we find the points in the function’s image (a measure in measure theory, a density in probability). On countable sets, saying a function is injective tells us much about the function. Moreover, the WL-test hash function does not even need to operate over sets with total or even partial orders. As a neural network, the WL-GNN “hash” ($f$ in the paper) must operate over a totally ordered set (\mathbb{R}^n, n > 0). Porting the WL-test argument of “convergence to unique isomorphic fingerprints” to a WL-GNN requires a measure-theoretic analysis of the output of the WL-GNN layers, and careful analysis if the total order of the set does not create attractors when they are applied recursively.
To illustrate the above *attractor* point, let’s consider the construct of Theorem 1 of (Xu et al., 2018), where the WL-GNN “hash” ($f$) is (roughly) described as the transition probability matrix of a random walk on the input graph. Under well-known conditions, the successive application of this operator ("hash" or transition probability matrix P in this case) can go towards an attractor (the steady state). Here, we need a measure-theoretic analysis of the “hash” even if it is bijective: random walk mixing. The random walk transition operator can be invertible (bijective), but we still say the random walker will mix, i.e., the walker forgets where it started, even if the transition operation can be perfectly undone by inversion (P^{-1}). In a WL-GNN that only uses the last layer for classification, this would manifest itself as poor performance in a WL-GNN with a large number of layers, and vanishing gradients. Of course, since (Xu et al., 2018) argued to revert back to the framework of (Duvenaud et al., 2015) of using the embeddings of all layers, one can argue that this mixing problem is just a problem of “wasted computation”.
The matrix analysis of the last paragraph also points to another potential problem with the sum aggregator. GIN needs to be shallow. With ReLU activations the reason is simple: for an adjacency matrix $A$, the value of $A^j$ grows very quickly with $j$ (diverges). With sigmoid activations, GIN would experience vanishing gradients in graphs with high variance in node degrees.
The paper should be careful with oversimplifications. Simplifications are useful for insight but can be dangerous if not prefaced by clear warnings and a good understanding of their limitations. I am not asking for a measure-theoretic analysis revision of the paper (it could be left to a follow-up paper). I am asking for a *relatively long* discussion of the limitations of the analysis.
Suggestions to strengthen the paper:
•	Please address the above concerns.
•	Table 1 should have confidence intervals (a statistical analysis of significance would be a welcome bonus).
•	Please mention the classes of graphs where the WL-test cannot distinguish two non-isomorphic graphs. See (Douglas, 2011), (Cai et al., 1992) and (Evdokimov and Ponomarenko, 1999) for the examples. It is important for the WL-GNN literature to keep track of the more fundamental limitations of the method.
•	(Hamilton et al, 2017) also uses the LSTM aggregator, besides max aggregator and mean aggregator, which outperforms both max and mean in some tasks. Does the LSTM aggregator also outperforms the sum aggregator in the tasks of Table 1? It is important for the community to know if unusual aggregators (such as the asymmetric LSTM) have some yet-to-be-discovered class-distinguishing power.
--------- Update -------
The counter-example in
https://openreview.net/forum?id=ryGs6iA5Km&noteId=rkl2Q1Qi6X
is indeed a problem for Theorem 3 if  \{h_v^{(k-1)}, h_u^{(k-1)} : u \in \mathcal{N}_v\} is not a typo for a set of tuples \{(h_v^{(k-1)}, h_u^{(k-1)}) : u \in \mathcal{N}_v\}. Unfortunately, in their proof, the submission states "difficulty in proving this form of aggregation mainly lies in the fact that it does not immediately distinguish the root or central node from its neighbors", which means \{h_v^{(k-1)}, h_u^{(k-1)} : u \in \mathcal{N}_v\} is actually \{h_v^{(k-1)}\} \cup \{ h_u^{(k-1)} : u \in \mathcal{N}_v\}, which is not as powerful as WL. Concatenating is more powerful than the summing the node's own embedding, but it results in a  simpler model and could be easier to learn in practice. And I am still concerned about the countable x uncountable domain/image issue I raised in my review.
Still, the reviewers seem to be doing all the discussion among themselves, with no input from the authors. I am now following Reviewer 2.
----
Reverting my score to my original score. The authors have addressed most of my concerns, thank you. The restricted theorems and propositions better describe the contribution.
I would like to note that while the proof of (Xu et al., 2018) is limited that does not mean it is not applicable to GIN or GraphSAGE or similar models. The paper uses 5 GNN layers, which in my experience is the maximum I could ever use with GNNs without seeing a degradation in performance. I don't think this should be a topic for this paper, though.
Xu, K., Li, C., Tian, Y., Sonobe, T., Kawarabayashi, K., & Jegelka, S. (2018). Representation Learning on Graphs with Jumping Knowledge Networks. In ICML.
Cai, J. Y., Fürer, M., & Immerman, N. (1992). An optimal lower bound on the number of variables for graph identification. Combinatorica, 12(4), 389-410.
Douglas, B. L. (2011). The Weisfeiler-Lehman method and graph isomorphism testing. arXiv preprint arXiv:1101.5211.
Evdokimov, S., & Ponomarenko, I. (1999). Isomorphism of coloured graphs with slowly increasing multiplicity of Jordan blocks. Combinatorica, 19(3), 321-333.


In this paper we introduce the Scratchpad Encoder, a novel addition to the sequence to sequence (seq2seq) framework and explore its effectiveness in generating natural language questions from a given logical form. The Scratchpad Encoder enables the decoder at each time step to modify all the encoder outputs, thus using the encoder as a "scratchpad" memory to keep track of what has been generated so far and to guide future generation. Experiments on a knowledge based question generation dataset show that our approach generates more fluent and expressive questions according to quantitative metrics and human judgments.

The paper studies the problem of question generation from sparql queries. The motivation is to generate more training data for knowledge base question answering systems to be trained on. However, this task is an instance of natural language generation: given a meaning representation (quite often a database record), generate the natural language text correspoding to it. And previous work on this topic has proposed very similar ideas to the scratchpad proposed here in order to keep track of what the neural decoder has already generated, here are two of them:
- Semantically Conditioned LSTM-based Natural Language Generation for Spoken Dialogue Systems
Tsung-Hsien Wen, Milica Gasic, Nikola Mrksic, Pei-Hao Su, David Vandyke, Steve Young, EMNLP 2015: https://arxiv.org/abs/1508.01745
- Globally Coherent Text Generation with Neural Checklist Models
Chloe Kiddon Luke Zettlemoyer Yejin Choi: https://aclweb.org/anthology/D16-1032
Thus the main novelty claim of the paper needs to be hedged appropriately. Also, to demonstrate the superiority of the proposed method an appropriate comparison against previous work is needed.
Some other points:
- How is the linearization of the inout done? It  typically matters
- Given the small size of the dataset, I would propose experimenting with non-neural approaches as well, which are also quite common in NLG.
- On the human evaluation: showing the gold standard reference to the judges introduces bias to the evaluation which is inappropriate as in language generation tasks there are multiple correct answers. See this paper for discussion in the context of machine translation: http://www.aclweb.org/anthology/P16-2013
- For the automatic evaluation measures there should be multiple references per SPARQL query since this is how BLEU et al are supposed to be used. Also, this would allow to compare the references against each other (filling in the missing number in Table 4) and this would allow an evaluation of the evaluation itself: while perfect scores are unlikely, the human references should be much better than the systems.
- In the outputs shown in Table 3, the questions generated by the scratchpad encoder often seem to be too general compared to the gold standard, or incorrect. E.g. "what job did jefferson have" is semntically related to his role in the declaration of independence but rather different. SImilarly, being married to someone is not the same as having a baby with someone. While I could imagine human judges preferring them as they are fluent, I think they are wrong as they express a different meaning than the SPARQL query they are supposed to express. What were the guidelines used?


In this work, we propose a goal-driven collaborative task that contains language, vision, and action in a virtual environment as its core components. Specifically, we develop a Collaborative image-Drawing game between two agents, called CoDraw. Our game is grounded in a virtual world that contains movable clip art objects. The game involves two players: a Teller and a Drawer. The Teller sees an abstract scene containing multiple clip art pieces in a semantically meaningful configuration, while the Drawer tries to reconstruct the scene on an empty canvas using available clip art pieces. The two players communicate via two-way communication using natural language. We collect the CoDraw dataset of ∼10K dialogs consisting of ∼138K messages exchanged between human agents. We define protocols and metrics to evaluate the effectiveness of learned agents on this testbed, highlighting the need for a novel crosstalk condition which pairs agents trained independently on disjoint subsets of the training data for evaluation. We present models for our task, including simple but effective nearest-neighbor techniques and neural network approaches trained using a combination of imitation learning and goal-driven training. All models are benchmarked using both fully automated evaluation and by playing the game with live human agents.

The paper proposes a game of collaborative drawing where a teller is
to communicate a picture to a drawer via natural language.  The picture
allows only a small number of components and a fixed and limited set
of detailed variations of such components.
Pros:
The work contributed a dataset where the task has relatively objective
criteria for success.  The dataset itself is a valuable contribution
to the community interested in the subject.   It may be useful for
purposes beyond those it was designed for.
The task is interesting and its visual nature allows for easy inspection
of the reasons for successes or failures.  It provides reasonable grounding
for the dialog.  By restricting the scope of variations through the options
and parameters, some detailed aspects of the conversation could be explored
with carefully controlled experiments.
The authors identified the need for and proposed a "crosstalk" protocol
that they believe can prevent leakage via common training data and
the development of non-language, shared codebooks that defeat the purpose
of focusing on the natural language dialog.
The set up allows for pairing of human and human, machine and machine,
and human and machine for the two roles, which enables comparison to
human performance baselines in several perspectives.
The figures give useful examples that are of great help to the readers.
Cons.:
Despite the restriction of the task context to creating a picture with
severely limited components, the scenario of the dialogs still has many
details to keep track of, and many important facets are missing in the
descriptions, especially on the data.
There is no analysis of the errors.  The presentation of
experimental results stops at the summary metrics, leaving many
doubts on why they are as such.
The work feels somewhat pre-mature in its exploration of the models
and the conclusions to warrant publication.  At times it feels like the
authors do not understand enough why the algorithms behave as they do.
However if this is considered as a dataset description paper and
the right expectation is set in the openings, it may still be acceptable.
The completed work warrants a longer report when more solid conclusions
can be drawn about the model behavior.
The writing is not organized enough and it takes many back-and-forth rounds
of checking during reading to find out about certain details that are given
long after their first references in other contexts.  Some examples are
included in the followings.
Misc.
Section 3.2, datasets of 9993 dialogs:
Are they done by humans?   Later it is understood from further descriptions.
It is useful to be more explicit at the first mention of this data collection effort.
The way they relate to the 10020 scenes is mentioned as "one per scene", with a footnote on some being removed.
Does it mean that no scene is described by two different people?  Does this
limit the usefulness of the data in understanding inter-personal differences?
Later in the descriptions (e.g. 4.1 on baseline methods) the notion of
training set is mentioned, but up to then there is no mentioning of how
training and testing (novel scenes) data are created.
It is also not clear what training data include: scenes only?
Dialogs associated with specific scenes?  Drawer actions?
Section 4.1, what is a drawer action?  How many possibilities are there?
From the description of "rule-based nearest-neighbor drawer" they seem to be
corresponding to "teller utterance".
However it is not clear where they come from.  What is an example of a drawer action?
Are the draw actions represented using the feature vectors discussed in the later sections?
Section 5.1, the need for the crosstalk protocol is an interesting observation,
however based on the description here, a reader may not be able to understand
the problem.  What do you mean by "only limited generalization has taken place"?  Any examples?
Section 5, near the end: the description of the dataset splits is too cryptic.
What are being split?  How is val used in this context?
All in all the data preparation and partitioning descriptions need substantial clarification.
Section 6:  Besides reporting averaged similarity scores, it will be useful to report some error analysis.
What are the very good or very bad cases?  Why did that happen?
Are the bad scenes constructed by humans the same as those bad scenes
constructed by machines?  Do humans and machines tend to make different errors?


Developing deep neural networks (DNNs) for manifold-valued data sets has gained much interest of late in the deep learning research community. Examples of manifold-valued data include data from omnidirectional cameras on automobiles, drones etc., diffusion magnetic resonance imaging, elastography and others. In this paper, we present a novel theoretical framework for DNNs to cope with manifold-valued data inputs. In doing this generalization, we draw parallels to the widely popular convolutional neural networks (CNNs). We call our network the ManifoldNet. As in vector spaces where convolutions are equivalent to computing the weighted mean of functions, an analogous definition for manifold-valued data can be constructed involving the computation of the weighted Fréchet Mean (wFM). To this end, we present a provably convergent recursive computation of the wFM of the given data, where the weights makeup the convolution mask, to be learned. Further, we prove that the proposed wFM layer achieves a contraction mapping and hence the ManifoldNet does not need the additional non-linear ReLU unit used in standard CNNs. Operations such as pooling in traditional CNN are no longer necessary in this setting since wFM is already a pooling type operation. Analogous to the equivariance of convolution in Euclidean space to translations, we prove that the wFM is equivariant to the action of the group of isometries admitted by the Riemannian manifold on which the data reside. This equivariance property facilitates weight sharing within the network. We present experiments, using the ManifoldNet framework, to achieve video classification and image reconstruction using an auto-encoder+decoder setting. Experimental results demonstrate the efficacy of ManifoldNet in the context of classification and reconstruction accuracy.

Brief summary:
The paper considers a generalization of convolutional neural networks (CNNs) to data residing on Riemannian manifolds. The idea is to replace convolutions with weighted averages, which are implemented intrinsically on the manifold. It is shown that this operator is equivariant to isometric group actions. A related approach for dimensionality reduction is also proposed, but I think this only applies to Euclidean data, so I am a bit confused about that part. Empirical performance is reported on toy data with weak baselines.
Good points about the paper:
+ It is a relevant point that the intrinsic average in manifolds is a way to generalized convolutions to the Riemannian domain.
+ The paper is generally fairly easy to read.
Concerns with the paper:
- A key point of CNNs is that you learn filters that are small, i.e. only have non-zero weight assigned to a few neighboring pixels. As far as I can tell, the here proposed "filters" would be one weight per data point.
- I am concerned about the stacking of multiple "convolution" (wFM) layers: since each layer computes the average of a set of points, then doesn't stacking multiple "convolution" (wFM) layers on top of each other correspond to computing the average of a set of averages? And can this not be computed by a single average? In other words, is a cascade of "convolution" (wFM) layers equivalent to a single layer? Seems like a complicated way of doing shallow learning unless I misunderstand.
- In section 2.2 it is argued that the weighted average (wFM) is a contraction mapping. While I think the proof is correct, I am concerned about the prerequisite assumption that that distance between a set of points X and Y is the *maximal* distance between points in the two sets. Usually, one would define this distance as the *minimal* distance (akin to the Hausdorff distance). It seems that under this more reasonable choice of distance, the proof no longer holds
- Large parts of section 2.1 is devoted to an efficient algorithm for computing weighted averages on manifolds.  Here the text is written such as to indicate that this is a novel contribution of the present paper, even if these results are readily available in the literature. I strongly encourage a re-writing to emphasize that this is a repetition of previous knowledge.
- Proposition 5 does not appear to come with a proof.
- Section 3.2 introduce a new dimensionality reducing layer based on the Grassmann average construction for subspace learning. I was quite confused when reading this. From what I can tell this layer is only applicable when the input data is Euclidean, and as such appears to be unrelated to the rest of the paper.
- At times the paper is rather sloppy written, e.g. fonts are way too small in figures, the dot(.) notation is not defined (e.g. in def. 7), and the citation style is very difficult to read (please use \citet and \citep instead of \cite).
Other comments:
*) The assumption (page 3) that the data reside within a geodesic ball of a certain radius seem quite strong. It would be good to comment on this in the paper.
*) It is not clear to me that the weighted average is a particularly good way to generalize convolution. Yes, I agree, it is *one
 way to generalize, but why should I pick this one in particular?
*) In the experiment in sec. 3.1 the manifold comes from a particular way to extract features from data. In a deep learning context, we would usually learn such features instead of manually designing them. This seem like a general issue, that most often manifold come into play through a manual feature-design, which seem to be at odds with the end-to-end learning mindset. It might be good to have examples where this is not the case.


Learning disentangled representations from visual data, where different high-level generative factors are independently encoded, is of importance for many computer vision tasks. Supervised approaches, however, require a significant annotation effort in order to label the factors of interest in a training set. To alleviate the annotation cost, we introduce a learning setting which we refer to as "referencebased disentangling". Given a pool of unlabelled images, the goal is to learn a representation where a set of target factors are disentangled from others. The only supervision comes from an auxiliary "reference set" that contains images where the factors of interest are constant. In order to address this problem, we propose reference-based variational autoencoders, a novel deep generative model designed to exploit the weak supervisory signal provided by the reference set. During training, we use the variational inference framework where adversarial learning is used to minimize the objective function. By addressing tasks such as feature learning, conditional image generation or attribute transfer, we validate the ability of the proposed model to learn disentangled representations from minimal supervision.

The authors address the problem of representation learning in which data-generative factors of variation are separated, or disentangled, from each other. Pointing out that unsupervised disentangling is hard despite recent breakthroughs, and that supervised disentangling needs a large number of carefully labeled data, they propose a “weakly supervised” approach that does not require explicit factor labels, but instead divides the training data in to two subsets. One set, the “reference set” is known to the learning algorithm to leave a set of generative “target factors” fixed at one specific value per factor, while the other set is known to the learning algorithm to vary across all generative factors. The problem setup posed by the authors is to separate the corresponding two sets of factors into two non-overlapping sets of latents.
Pros:
To address this problem, the authors propose an architecture that includes a reverse KL-term in the loss, and they show convincingly that this approach is indeed successful in separating the two sets of generative factors from each other. This is demonstrated in two different ways. First, quantitatively on an a modified MNIST dataset, showing that the information about the target factors is indeed (mostly) in the set of latents that are meant to capture them. Second, qualitatively on the modified MNIST and on a further dataset, AffectNet, which has been carefully curated by the authors to improve the quality of the reference set. The qualitative results are impressive and show that this approach can be used to transfer the target factors from one image, onto another image.
Technically, this work combines and extends a set of interesting techniques into a novel framework, applied to a new way of disentangling two sets of factors of variation with a VAE approach.
Cons:
The problem that this work solves seems somewhat artificial, and the training data, while less burdensome than having explicit labels, is still difficult to obtain in practice. More importantly, though, both the title and the start of the both the abstract and the introduction are somewhat misleading. That’s because this work does not actually address disentangling in the sense of “Learning disentangled representations from visual data, where high-level generative factors correspond to independent dimensions of feature vectors…” What it really addresses is separating two sets of factors into different parts of the representation, within each of which the factors can be, are very likely are, entangled with each other.
Related to the point that this work is not really about disentangling, the quantitative comparisons with completely unsupervised baselines are not really that meaningful, at least not in terms of what this work sets out to do. All it shows is whether information about the target factors is easily (linearly) decodable from the latents, which, while related to disentangling, says little about the quality of it. On the positive side, this kind of quantitative comparison (where the authors approach has to show that the information exists in the correct part of the space) is not pitted unfairly against the unsupervised baselines.
===
Update:
The authors have made a good effort to address the concerns raised, and I believe the paper should be accepted in its current form. I have increased my rating from 6 to 7, accordingly.


Building deep reinforcement learning agents that can generalize and adapt to unseen environments remains a fundamental challenge for AI. This paper describes progresses on this challenge in the context of man-made environments, which are visually diverse but contain intrinsic semantic regularities. We propose a hybrid model-based and model-free approach, LEArning and Planning with Semantics (LEAPS), consisting of a multi-target sub-policy that acts on visual inputs, and a Bayesian model over semantic structures. When placed in an unseen environment, the agent plans with the semantic model to make high-level decisions, proposes the next sub-target for the sub-policy to execute, and updates the semantic model based on new observations. We perform experiments in visual navigation tasks using House3D, a 3D environment that contains diverse human-designed indoor scenes with real-world objects. LEAPS outperforms strong baselines that do not explicitly plan using the semantic content.

The paper proposes a hybrid model-free and model-based RL agent for the task of navigation. Reaching the target is decomposed into a set of sub-goals, and the plan is updated as the agent explores the environment. The method has been tested in the House3D environment for the task of RoomNav, where the goal is to navigate towards a certain room.
The idea of integrating RL agents with semantic knowledge is interesting. However, the paper has several major issues that should be addressed in the rebuttal:
(1) The experiment results in Figure 3 and Figure 4 are based on groundtruth room information. The only experiment that is fully automatic is the one in Figure 5. However, there is no difference between the proposed method and the baselines in that case. So the proposed method is not effective without groundtruth information.
(2) The only evaluation metric that is used is "Success Rate". That metric is not sufficient for evaluation of navigation agents since it does not include episode length information. All of the results should be based on the protocol mentioned in "On Evaluation of Embodied Navigation Agents", arXiv 2018.
(3) There is no termination action according to Appendix B. So the agent does not know if it is at the target or not. It seems the agent will stop if it issues "stay still" three times. That is different from termination action. Also, it is confusing what 450 pixels means for a scene classifier that works on the image.
(4) The paper is written in a convoluted way:
   (a) It is not clear if the semantic model is trained along with the RL model end-to-end or not.
   (b) Regarding multi-target sub-policies, is there a separate policy for each pair of intermediate targets?
   (c) Regarding inference and planning on M, what is \tau exactly? How is the length of the plan determined?
   (d) Why is the model updated only after a fixed number of steps? That increases the episode length.
(5) The number of T_i's is manually set to 8. That causes serious generalization issues. How do we know how many T_i's exist in a new environment?
Minor comments:
- The paper mentions "An example of such environments is House3D which contains 45k real-world 3D scenes". House3D includes only synthetic scenes. They should not be called real-world scenes.
- How is the reward shaping done?
****
Final comments after reading the response and the reviews:
Regarding the fairness of the review, success rate is not sufficient to evaluate navigation agents. A random agent can achieve 100% success if it is given enough time. So it is totally fair to ask for a metric (such as SPL) that is a function of both success rate and episode length.
I am going to increase the rating to 5 since some of my concerns have been addressed. There are still a number of issues:
- The authors did not run the experiments with the termination action. I disagree that this is orthogonal to the focus of the paper. This is not just an additional action. It indicates whether the agent has learned anything or it is just a combination of better obstacle avoidance and luck. The SPL numbers are so low (maximum SPL is 6.19%) so adding the termination action will probably make the method similar to random.
- There is a huge gap between success rate and SPL numbers. For instance, success rate is 66.4, while SPL is 5.84 (note that for some reason the SPL numbers are multiplied by 10 in the table). I doubt that the agent has learned anything meaningful in comparison to the baseline. I understand that the task is hard, but this gap is so huge.
- A separate policy is trained for each sub-target. This doesn't scale. There should be one policy for all targets.


Modern deep neural networks are highly overparameterized, and often of huge sizes. A number of post-training model compression techniques, such as distillation, pruning and quantization, can reduce the size of network parameters by a substantial fraction with little loss in performance. However, training a small network of the post-compression size de novo typically fails to reach the same level of accuracy achieved by compression of a large network, leading to a widely-held belief that gross overparameterization is essential to effective learning. In this work, we argue that this is not necessarily true. We describe a dynamic sparse reparameterization technique that closed the performance gap between a model compressed through iterative pruning and a model of the post-compression size trained de novo. We applied our method to training deep residual networks and showed that it outperformed existing reparameterization techniques, yielding the best accuracy for a given parameter budget for training. Compared to existing dynamic reparameterization methods that reallocate non-zero parameters during training, our approach achieved better performance at lower computational cost. Our method is not only of practical value for training under stringent memory constraints, but also potentially informative to theoretical understanding of generalization properties of overparameterized deep neural networks.

This paper presents a method for training neural networks where an efficient sparse/compressed representation is enforced throughout the training process, as opposed to starting with are large model and pruning down to a smaller size.  For this purpose a dynamic sparse reparameterization heuristic is proposed and validated using data from MNIST, CIFAR-10, and ImageNet.
My concerns with this work in its present form are two-fold.  First, from a novelty standpoint, the proposed pipeline can largely be viewed as introducing a couple heuristic modifications to the SET procedure from reference (Mocanu, et al., 2018), e.g., substituting an approximate threshold instead of sorting for removing weights, changing how new weights are redistributed, etc.  The considerable similarity was pointed out by anonymous commenters and I believe somewhat understated by the submission.  Regardless, even if practically effective, these changes seem more like reasonable engineering decisions to improve the speed/performance rather than research contributions that provide any real insights.  Moreover, there is no attendant analysis regarding convergence and/or stability of what is otherwise a sequence of iterates untethered to a specific energy function being minimized.
Of course all of this could potentially be overcome with a compelling series of experiments demonstrating the unequivocal utility of the proposed modifications.  But it is here that unfortunately the paper falls well short.  Despite its close kinship with SET, there are surprisingly no comparisons presented whatsoever.  Likewise only a single footnote mentions comparative results with DeepR (Bellec et al., 2017), which represents another related dynamic reparameterization method.  In a follow up response to anonymous public comments, some new tests using CIFAR-10 data are presented, but to me, proper evaluation requires full experimental details/settings and another round of review.
Moreover, the improvement over SET in these new results, e.g., from a 93.42 to 93.68 accuracy rate at 0.9 sparsity level, seems quite modest.  Note that the proposed pipeline has a wide range of tuning hyperparameters (occupying a nearly page-sized Table 3 in the Appendix), and depending on these settings relative to SET, one could easily envision this sort of minor difference evaporating completely.  But again, this is why I strongly believe that another round of review with detailed comparisons to SET and DeepR is needed.
Beyond this, the paper repeatedly mentions significant improvement over "start-of-the-art sparse compression methods." But this claim is completely unsupported, because all the tables and figures only report results from a single existing compression baseline, namely, the pruning method from (Zhu and Gupta, 2017) which is ultimately based on (Han et al., 2015).  But just in the last year alone there have been countless compression papers published in the top ML and CV conferences, and it is by no means established that the pruning heuristic from (Zhu and Gupta, 2017) is state-of-the-art.
Note also that reported results can be quite deceiving on the surface, because unless the network structure, data augmentation, and other experimental design details are exactly the same, specific numbers cannot be directly transferred across papers.  Additionally, numerous published results involve pruning at the activation level rather than specific weights.  This definitively sacrifices the overall compression rate/model size to achieve structured pruning that is more naturally advantageous to implementation in practical hardware (e.g., reducing FLOPs, run-time memory, etc.).  One quick example is Luo et al., "ThiNet: A Filter Level Pruning Method for Deep Neural Network Compression," ICCV 2017, but there are many many others.
And as a final critique of the empirical section, why not report the full computational cost of training the proposed model relative to others?  For an engineered algorithmic proposal emphasizing training efficiency, this seems like an essential component.
In aggregate then, my feeling is that while the proposed pipeline may eventually prove to be practically useful, presently this paper does not contain a sufficient aggregation of novel research contribution and empirical validation.
Other comments:
- In Table 2, what is the baseline accuracy with no pruning?
- Can this method be easily extended to prune entire filters/activations?


Adaptive gradient methods have experienced great success in training deep neural networks (DNNs). The basic idea of the methods is to track and properly make use of the first and/or second moments of the gradient for model-parameter updates over iterations for the purpose of removing the need for manual interference. In this work, we propose a new adaptive gradient method, referred to as generalized adaptive moment estimation (Game). From a high level perspective, the new method introduces two more parameters w.r.t. AMSGrad (S. J. Reddi & Kumar (2018)) and one more parameter w.r.t. PAdam (Chen & Gu (2018)) to enlarge the parameterselection space for performance enhancement while reducing the memory cost per iteration compared to AMSGrad and PAdam. The saved memory space amounts to the number of model parameters, which is significant for large-scale DNNs. Our motivation for introducing additional parameters in Game is to provide algorithmic flexibility to facilitate a reduction of the performance gap between training and validation datasets when training a DNN. Convergence analysis is provided for applying Game to solve both convex optimization and smooth nonconvex optimization. Empirical studies for training four convolutional neural networks over MNIST and CIFAR10 show that under proper parameter selection, Game produces promising validation performance as compared to AMSGrad and PAdam.

Summary
------
The authors propose an adaptation of the Adam method, with the AMSGrad correction and an additional parameter to p to exponentiate the diagonal conditioning matrix V (Padam).
The proposed method changes two aspects: first, there is no need to retain two version of the rescaling matrix v, where amsgrad and Padam keeps the last monotone \hat v)t and non-monotone version v_t. Secondly, a new parameter q is introduced, that replaces the q=2 in the moment estimation phase of (P)Adam.
A regret analysis is proposed in the convex case, while a vanishing bound on the gradient is derived in the non-convex smooth case.
Review
------
Although improving optimization methods is certainly important for the machine learning community, the reviewer have strong concerns about this paper.
First of all, the paper is hard to read as it contains too many approximations. What does 'SGD is known to work reasonably well regardless of their problem structure' means ? Same thing for 'Its performance deteriorates when the gradients are dense due to a rapid decay of the learning rates.' The authors uses many times elliptical discourse to detail the course of their analysis, which is non informative: for instance, 'one can easily derive the upper bound expression', and 'It is not difficult to conclude that when G_t [...]'. This level of writing is not professional. Some completely irrelevant argument are proposed to justify the method: 'For instance the extension from l_2 norm to l_p norm and generalization from Cauchy-Schwart to Holder inequality'.
The reviewer has interrogations about the relevance of the proposed algorithm. The additional parameter q needs to be tuned, which carries only the promise of further overfitting. I would have been convinced by an sequence of experiment where q is set automatically by considering a validation set, and then tested on a left out test set. However, the authors report only the results for the best q, with non significant differences (and not quantified, there is no result tables). Using q=2 at least made sense from the point of view of empirical Fisher matrix approximation.
The review also have several concerns aout the correctness of the proposed arguments. First, the major argument of memory usage stems from 1) a miscalculation and 2) a misunderstanding of memory bottlenecks in deep learning. 1) adam models keeps in memory x_t (the model parameter), g_t (the model gradient), m_t (momentum) \hat v_t and v_{t-1} (the monotone and non monotone version of the second order moment estimation. In contrast, the proposed model do not track v_{t-1}: this amounts to a memory saving of 20% considering all model related parameters. 2) more importantly, the most important memory usage in deep leaning comes from the activations that need to be kept in memory during the forward pass to perform the backward pass. Even the biggest model are less than 1GB, and most of the memory used during training is dedicated to intermediary activations. This makes the major argument of the paper less convincing, and misleads the reader.
Second, even when disregarding the slightly abusive assumptions over the iterate sequences, that are common in the adaptive stochastic optimizers community, I think that the bound proposed in theorem 1 is non informative, as the second term behaves like T sqrt(T) assymptotically, due to the presence of 1 / \alpha_t. This does not show the convergence of averaged regret R_T / T.
Regarding the experiment section, I am afraid that testing a new optimizer over MNIST and CIFAR is not enough to show the relevance of the method for the whole deep learning community. An eperiment over a non-toy dataset (eg ImageNet), and on non computer-vision dataset (eg from NLP) would be a minimum, besides the overfitting concern described above.
In conclusion, it is the reviewer's opinion that significant rework in term of presentation and strong improvement of the experiment section to make the case for the Game optimizer.
Minor comments
------------
Table 1: what do you bound when you compare results ? I think there is a typo in Zhou et al. result: 1/2 should read p.
Eq (1): it is rather surprising to use x_t as the model parameters in the ICLR community.
p 7: the dimension d could be larger than T when training large-scale neural networks: how does it relate to comparing sqrt(dT) to (dT)^s ?


TRANSFER LEARNING FOR SEQUENCES VIA LEARN- ING TO COLLOCATE
Transfer learning aims to solve the data sparsity for a target domain by applying information of the source domain. Given a sequence (e.g. a natural language sentence), the transfer learning, usually enabled by recurrent neural network (RNN), represents the sequential information transfer. RNN uses a chain of repeating cells to model the sequence data. However, previous studies of neural network based transfer learning simply represents the whole sentence by a single vector, which is unfeasible for seq2seq and sequence labeling. Meanwhile, such layer-wise transfer learning mechanisms lose the fine-grained cell-level information from the source domain. In this paper, we proposed the aligned recurrent transfer, ART, to achieve celllevel information transfer. ART is under the pre-training framework. Each cell attentively accepts transferred information from a set of positions in the source domain. Therefore, ART learns the cross-domain word collocations in a more flexible way. We conducted extensive experiments on both sequence labeling tasks (POS tagging, NER) and sentence classification (sentiment analysis). ART outperforms the state-of-the-arts over all experiments.

This paper presents the following approach to domain adaptation. Train a source domain RNN. While doing inference on the target domain, first you run the source domain RNN on the sequence. Then while running the target domain RNN, set the hidden state at time step i, h^t_i, to be a function 'f' of  h^t_{i-1} and information from source domain \psi_i; \psi_i is computed as a convex combination of the state of the source domain RNN, h^s_{i}, and an attention-weighted average of all the states h^s{1...n}. So in effect, the paper transfers information from each of source domain cells -- the cell at time step i and all the "collocated" cells (collocation being defined in terms of attention). This idea is then extended in a straightforward way to LSTMs as well.

Doing "cell-level" transfer enables more information to be transferred according to the authors, but it comes at a higher computation since we need to do O(n^2) computations for each cell.
The authors show that this beats a variety of baselines for classification tasks (sentiment), and for sequence tagging task (POS tagging over twitter.)
Pros:
1. The idea makes sense and the experimental results show solid
Cons:
1. Some questions around generalization are not clearly answered. E.g. how are the transfer parameters of function 'f' (that controls how much source information is transferred to target) trained? If the function 'f' and the target RNN is trained on target data, why does 'f' not overfit to only selecting information from the target domain? Would something like dropping information from target domain help?
2. Why not also compare with a simple algorithm of transferring parameters from source to target domain? Another simple baseline is to just train final prediction function (softmax or sigmoid) on the concatenated source and target hidden states. Why are these not compared with? Also, including the performance of simple baselines like word2vec/bow is always a good idea, especially on the sentiment data which is very commonly used and widely cited.
3. Experiments: the authors cite the hierarchical attention transfer work of Li et al (https://www.aaai.org/ocs/index.php/AAAI/AAAI18/paper/download/16873/16149) and claim their approach is better, but do not compare with them in the experiments. Why?
Writing:
The writing is quite confusing at places and is the biggest problem with this paper. E.g.
1. The authors use the word "collocated" everywhere, but it is not clear at all what they mean. This makes the introduction quite confusing to understand. I assumed it to mean words in the target sentences that are strongly attended to. Is this correct? However, on page 4, they claim "The model needs to be evaluated O(n^2) times for each sentence pair." -- what is meant by sentence pair here? It almost leads me to think that they consider all source sentence and target sentences? This is quite confusing.
2. The authors keep claiming that "layer-wise transfer learning mechanisms lose the fine-grained cell-level information from the source domain", but it is not clear exactly what do they mean by layer-wise here. Do they mean transferring the information from source cell i to target cell i as it is? In the experiments section on LWT, the authors claim that "More specifically, only the last cell of the RNN layer transfers information. This cell works as in ART. LWT only works for sentence classification." Why is it not possible to train a softmax over both the source hidden state and the target hidden state for POS tagging?
nits:
page 4 line 1: "i'th cell in the source domain" -> "i'th cell in the target domain". "j'th cell in target" -> "j'th cell in sourcE".
Revised: increased score after author response.


Adversarial training has been recognized as a strong defense against adversarial attacks. In this paper, we propose a powerful second-order attack method that reduces the accuracy of the defense model by Madry et al. (2017). We demonstrate that adversarial training overfits to the choice of the norm in the sense that it is only robust to the attack used for adversarial training, thus suggesting it has not achieved universal robustness. The effectiveness of our attack method motivates an investigation of provable robustness of a defense model. To this end, we introduce a framework that allows one to obtain a certifiable lower bound on the prediction accuracy against adversarial examples. We conduct experiments to show the effectiveness of our attack method. At the same time, our defense model achieves significant improvements compared to previous works under our proposed attack.

This paper makes two different contributions in the field of adversarial training and robustness.
First the authors introduce a new type of attack that exploits second-order information while traditional attacks typically rely on first-order information.
Another contribution is a theorem that using the Renyi divergence certifies robustness of a classifier by adding Gaussian noise to pixels.
Overall, I find that the paper lacks clarity and does not properly contrast their work to existing results. They are also some issues with the evaluation results. I provide detailed feedback below.
1) Prior work
a) Connection between adversarial defense and robustness to random noise
This connection is established in Fawzi, A., Moosavi-Dezfooli, S. M., & Frossard, P. (2016). Robustness of classifiers: from adversarial to random noise. In Advances in Neural Information Processing Systems (pp. 1632-1640).
b) Connection between minimal perturbation required to confuse classifier and its confidence was discussed for the binary classification in Section 4 of
Fawzi, Alhussein, Omar Fawzi, and Pascal Frossard. "Analysis of classifiers’ robustness to adversarial perturbations." Machine Learning 107.3 (2018): 481-508.
c) The idea to compute the distribution of classifier outputs when the input is convolved with Gaussian noise was already “anticipated” in Section V of the following paper which relates the minimum perturbation needed to fool a model to it’s misclassification rate under Gaussian convolved input:
Lyu, Chunchuan, Kaizhu Huang, and Hai-Ning Liang. "A unified gradient regularization family for adversarial examples." Data Mining (ICDM), 2015 IEEE International Conference on. IEEE, 2015.
These papers should be discussed in the paper, please elaborate how you see your contribution regarding the results derived there.
2) Second-order attack introduced in the paper
I think they are a number of important details that are ignored in the presentation.
a) Regarding the assumption that the gradient vanishes in the difference of the loss, I think the authors should elaborate as to why this is a reasonable assumption to make. If we assume that the classifier has been trained to optimality then expanding the function at this (near-)optimum would perhaps indeed yield to a gradient term of small magnitude (assuming the function is smooth). However, nothing guarantees that the magnitude of the gradient term is negligible compared to the second-order information. The boundary of the classifier could very well be in a region of low-curvature.
b) The approximation of the second-order information is rather crude. However, the update derived is very similar to PGD with additional noise. In optimization, the use of noise is known to extract curvature, see e.g. (Xu & Yang, 2017) who showed that noisy gradient updates act as a noisy Power method that extracts negative curvature direction.
Xu, Y., & Yang, T. (2017). First-order Stochastic Algorithms for Escaping From Saddle Points in Almost Linear Time. arXiv preprint arXiv:1711.01944.
3) Issue of "degenerate global minimum": The authors argue that multistep attacks also suffer from this issue. However, the PGD attack of Madry is also initialized at a random point within the uncertainty ball around x, i.e. PGD attack first adds random noise to x before iteratively ascending the loss function. This PGD update + noise at first iteration seem rather similar to the update derived by the authors that uses random noise at every iteration. It could therefore be that the crude approximation of second-order information is not so different from previous work. This should be further investigated either theoretically or empirically.
4) Lack of details regarding some important aspects in the paper
a) “Note the evaluation requires adjustment and computing confidence intervals for p(1) and p(2), but we omit the details as it is a standard statistical procedure”
The authors seem to sweep this under the carpet but this estimation procedure gives only an estimate of the required quantities p(1) and p(2), which I think would require adjusting the result in the theorem to be a high probability bound (or an expectation bound) instead of a deterministic result.
b) “the noise is not necessarily added directly to the inputs but also to the first layer of a DNN. Given the Lipschitz constant of the first layer, one can still calculate an upper bound using our analysis. We omit the details here for simplicity”
What exactly changes here? How do you estimate the Lipschitz constant in practice?
5) Main theorem needs to be contrasted to previous results
The main Theorem uses the Renyi divergence certifies robustness of a classifier by adding Gaussian noise to pixels. There are already many results in the field of robust optimization that already derive similar results, see e.g.
Namkoong, H., & Duchi, J. C. (2017). Variance-based regularization with convex objectives. In Advances in Neural Information Processing Systems (pp. 2971-2980).
Gao, R., & Kleywegt, A. J. (2016). Distributionally robust stochastic optimization with Wasserstein distance. arXiv preprint arXiv:1604.02199.
Can you elaborate on the difference between your bounds and these ones? You do mention some of them require strong assumptions such as smoothness but this actually seems like a mild assumption (although some activation functions used in neural nets are indeed not smooth).
6) Adversarial Training Overfit to the Choice of norms
The main theorem derived in the paper uses the l_2 norm. What can be said regarding other norms?
7) Experiments:
a) the authors only report accuracies for attacks whose l2-norm is smaller than a fixed constant 0.8. However, this makes the results difficult to interpret and the authors should instead state the signal to noise ratio, i.e. dividing the l2-norm of the perturbation by the l2-norm of the image. Otherwise, it is not clear how strong or weak such perturbations are. (In particular, the norm depends on the dimension of the image, so l2-norms of perturbations for MNIST and CIFAR10 are not comparable).
b) In Section 6.2, the authors state that an l_infty trained model is vulnerable against l_2 perturbations. Why not training the model under both l_infty and l_2 perturbations?
c) Figure 1
Based on the results predicted in Theorem 2, it seems it would be more interesting to evaluate the largest L for which the classifier predictions are the same. Why did you report a different results?
8) Other comments
section 2.1: “Note this distribution is different from the one generated from softmax”. Why/How is this different?
connection to EOT attack’: authors claim: E_{d∼N(0,σ2I)} [∇_x L(θ, x, y)|x+d] = ∇_x E_{d∼N(0,σ2I)} [∇_x L(θ, x, y)|x+d]. There is a typo on the RHS where ∇_x is repeated twice. This is also the common reparametrization trick so could cite
Kingma, D. P., & Welling, M. (2013). Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114.


The open-domain question answering (OpenQA) task aims to extract answers that match specific questions from a distantly supervised corpus. Unlike supervised reading comprehension (RC) datasets where questions are designed for particular paragraphs, background sentences in OpenQA datasets are more prone to noise. We observe that most existing OpenQA approaches are vulnerable to noise since they simply regard those sentences that contain the answer span as ground truths and ignore the plausible correlation between the sentences and the question. To address this deficiency, we introduce a unified and collaborative model that leverages alignment information from query-sentence pairs in a small-scale supervised RC dataset and aggregates relevant evidence from distantly supervised corpus to answer open-domain questions. We evaluate our model on several real-world OpenQA datasets, and experimental results show that our collaborative learning methods outperform the existing baselines significantly.

This paper aims for open-domain question answering with distant supervision. First, the authors proposed an aggregation-based openQA model with sentence discriminator and sentence reader. Second, they use a semantic labeler to handle distant supervision problem by utilizing other span supervision tasks, and propose two different denoising methods. They run experiments on 3 open-domain QA datasets and achieve SOTA.
Strengths
1) Their semantic labeler and exploration of two different denoising methods are interesting and meaningful.
2) They conducted experiments on 3 widely-used open-domain datasets, and the performance gain is impressive.
Weakness
Although there is an impressive performance gain, the contribution of the paper seems to be marginal.
1) First of all, it is hard to say there is a contribution to the idea of sentence discriminator and sentence reader — people have used this framework for large-scale QA a lot. Also, the architecture of the models in this paper are almost identical to Chen et al (ACL 2017) and Lin et al (ACL 2018).
2) Thus, the contribution is more on semantic labeler and denoising method. However, this contribution is marginal as well since its role is almost the same as sentence discriminator plus pretraining methods which have widely used already.
Questions
1) What exactly is the difference between semantic labeler and sentence discriminator? For me, it seems like both of them label each sentence `yes` or `no`. My thought is sentence discriminator is only trained on the target dataset (distant supervision dataset) while semantic labeler is also trained (either jointly or separately) trained on the source dataset (span supervision dataset). (If my thought is wrong, please let me know, I would like to update my score.)
2) Chen et al (ACL 2017) have shown that pretraining QA model on span supervision dataset (SQuAD) is effective to train the model on distant supervision dataset. Similarly, Min et al (ACL 2018) have pretrained both QA model and sentence selector on SQuAD. While I think pretraining sentence selector on SQuAD is almost identical to sentence labeler with SSL method, could you give exact comparison of these different methods? For example, remove sentence labeler, and pretrain both sentence discriminator and reader on SQuAD, or jointly train them on SQuAD & target dataset.
Marginal comments
1) At the beginning of Section 2.4.1, it says the semantic labeler is able to transfer knowledge from the span supervised data — however, the authors should be careful since people usually refers to `knowledge` as an external knowledge. This method is more like better learning of accurate sentence selection, not transferring knowledge.
2) Please mention the TriviaQA data you used is Wikipedia domain, since there are two different domains (Wikipedia and Web).
3) In References section, the conference venues in many papers are omitted.
Overall comments
The paper explored several different methods to deal with distant supervision via sentence labeling, and I really appreciate their efforts. While the result is impressive, the idea in the paper is similar to the methods that have widely used already.