Probabilistic methods often need to draw samples from a nontrivial distribution. A * sampling is a nice algorithm by building upon a top-down construction of a Gumbel process, where a large state space is divided into subsets and at each round A * sampling selects a subset to process. However, the selection rule depends on a bound function, which can be intractable. Moreover, we show that such a selection criterion can be inefficient. This paper aims to improve A * sampling by addressing these issues. To design a suitable selection rule, we apply Probability Matching, a widely used method for decision making, to A * sampling. We provide insights into the relationship between A * sampling and probability matching by analyzing a nontrivial special case in which the state space is partitioned into two subsets. We show that in this case probability matching is optimal within a constant gap. Furthermore, as directly applying probability matching to A * sampling is time consuming, we design an approximate version based on Monte-Carlo estimators. We also present an efficient implementation by leveraging special properties of Gumbel distributions and well-designed balanced trees. Empirical results show that our method saves a significantly amount of computational resources on suboptimal regions compared with A * sampling.

Summary: This paper introduces a probability matching approach for optimizing Gumbel processes, i.e. the extension of the Gumbel-Max trick to more general measure spaces. The basic idea is to use a more refined subset selection mechanism as compared to A* Sampling, but at the cost of being able to guarantee an exact sample. Instead, the authors study the algorithm's best Gumbel so far as a function of the time budget.
Quality: This is an interesting and natural idea, and the experiments support the author's claim that it improves over A* Sampling in the regimes that they consider. The claims and proofs look correct to me.
Originality: This idea has not been explored in the literature, and is a sensible one to consider when having an exact sample is not necessary.
Clarity: The comparison to A* Sampling is a bit apples-to-oranges and the paper lacks some clarity overall. In particular, the authors should be much more clear that their termination condition necessarily differs from A* Sampling's. In A* Sampling the termination condition guarantees that the algorithm has found the true optimum, but in PM-A* the authors consider only a fixed budget of time --- terminating whether or not the algorithm has found the true optimum. The difference is a necessary feature, because otherwise I'm fairly sure A* Sampling is optimal (It has to push the upper bound of every subset below the true maximum, so if the term. cond. is not currently satisfied the subset with the current max. upper bound must eventually be visited before termination. Since it is the only subset where that is necessarily true, taking it is the optimal choice.) More comments below.
Significance: I think this is interesting work, but I would recommend the authors consider some examples that clarify where these ideas might impact that problems that the ICLR audience would be interested in.
Overall: The paper's clarity problems get in the way of its contribution. I think it is an interesting direction, but the authors are not clear enough about the correctness of the algorithm (it is not guaranteed to return an exact sample) to recommend acceptance. If this and other issues are fixed, it would be a good paper.
Pros:
- Solid contribution to the literature on Gumbels.
- Beginnings of a regret analysis.
- Uniform performance gains in comparison to A* Sampling (in the regime considered).
Cons:
- Lacks clarity in comparisons to A* Sampling.
- Writing could clarify significance as well.
Specific questions / suggestions:
- One of the important omissions of the paper is the following question. What is the distribution of PM-A*'s output at time T for a fixed T? It won't be the target distribution in general, but it may be close. This applicability of this work will be greatly improved by that analysis.
- In general the figures and experiments need more details. In particular,
(1) In figure 2a, why are there two sets of lines? Different values of T?
(2) You need to report T throughout.
(3) What did you take as Y? This is a critical part of the method, and you should report the estimator used.
- In A* Sampling for two subsets without splitting algorithm,  shouldn't you be removing ALL previous Xi from  be S_i? i.e. S_i \ {Xi_1, Xi_2, ... Xi_{k-1}} or something like that.


Although both our brain and deep neural networks (DNNs) can perform high-level sensory-perception tasks such as image or speech recognition, the inner mechanism of these hierarchical information-processing systems is poorly understood in both neuroscience and machine learning. Recently, Morcos et al. (2018) examined the effect of class-selective units in DNNs, i.e., units with high-level selectivity, on network generalization, concluding that hidden units that are selectively activated by specific input patterns may harm the network's performance. In this study, we revisit their hypothesis, considering units with selectivity for lowerlevel features, and argue that selective units are not always harmful to the network performance. Specifically, by using DNNs trained for image classification (7-layer CNNs and VGG16 trained on CIFAR-10 and ImageNet, respectively), we analyzed the orientation selectivity of individual units. Orientation selectivity is a low-level selectivity widely studied in visual neuroscience, in which, when images of bars with several orientations are presented to the eye, many neurons in the visual cortex respond selectively to a specific orientation. We found that orientation-selective units exist in both lower and higher layers of these DNNs, as in our brain. In particular, units in the lower layers become more orientationselective as the generalization performance improves during the course of training of the DNNs. Consistently, networks that generalize better are more orientationselective in the lower layers. We finally reveal that ablating these selective units in the lower layers substantially degrades the generalization performance, at least by disrupting the shift-invariance of the higher layers. These results suggest to the machine-learning community that, contrary to the triviality of units with highlevel selectivity, lower-layer units with selectivity for low-level features can be indispensable for generalization, and for neuroscientists, orientation selectivity can play a causally important role in object recognition.

COMMENTS RELATED TO REVISION:
The new analysis that has been added takes a step towards getting at the relationship to invariance. This is a positive. In general comments, the authors state as important contributions:
"1) to the debate on the harmfulness/importance of selectively activated units in DNNs [1-3] by presenting concrete examples where selectivity is important for generalization, and 2) to the neuroscience community, where, although orientation selectivity has been extensively analyzed for these 60 years since [4], its functional importance in a natural environment has remained unanswered."
On point 2, while this provides an example of an artificial network where orientation plays an important role, it's a stretch to generalize this to conclusions concerning functional importance in neuroscience.
One point 1, I agree that this paper takes steps in the right direction, but ultimately the overall conclusions still feel as though they are a natural and implied consequence of limiting orientation selectivity. It is noted that  orientation selectivity is "not just a superficial byproduct of object recognition, but is causally indispensable for object recognition". The idea that selectivity for oriented edges is indispensible for object recognition again is a conclusion that feels as though it would be shocking if this were not true. The notion of it being a superficial byproduct of object recognition presupposes that the purpose of the system is to recognize objects. Again, it would be surprising if there were vestigial features in the network that are learned, but play no important role - especially among early layers.
I think the paper might be strengthened by re-working this second point to more strongly establish the causality the authors claim.
ORIGINAL COMMENTS:
This paper presents interesting analysis and an ablation study on orientation selectivity in neural networks. This is analyzed with respect to generalization performance in decisions made. Overall, the paper is well written and interesting. However, I have a number of comments / concerns as described below:
Positives:
- The paper presents an in depth analysis of the role of orientation selectivity in neural hierarchies. This style of analysis is sorely lacking and fits the theme of learning representations
- The paper itself is well written and quite polished
- The authors have taken great care to rule out any possible confounds through experiments that are not identical to the main claims or objective (specifically the study of section 3.3)
To address:
- The only concern I have (and it is somewhat significant), is the notion of "generalization". This is a rather loaded term, and it is not clear on cursory inspection where this generalization comes from. Is it a function of invariance among higher layers to scene geometry? What are the fundamental underpinnings of these observations outside of correlation to orientation selectivity.
- It seems almost a tautology that removing orientation selectivity would impair performance. I would be more satisfied, and render a higher rating if I felt there was a "smoking gun" with respect to evidence. The conclusion is convincing, but the reasoning comes across as somewhat vague. With that said, it is also understood that this is a non-trivial matter to address and perhaps this paper is an important first step.


In this paper we developed a hierarchical network model, called Hierarchical Prediction Network (HPNet) to understand how spatiotemporal memories might be learned and encoded in a representational hierarchy for predicting future video frames. The model is inspired by the feedforward, feedback and lateral recurrent circuits in the mammalian hierarchical visual system. It assumes that spatiotemporal memories are encoded in the recurrent connections within each level and between different levels of the hierarchy. The model contains a feed-forward path that computes and encodes spatiotemporal features of successive complexity and a feedback path that projects interpretation from a higher level to the level below. Within each level, the feed-forward path and the feedback path intersect in a recurrent gated circuit that integrates their signals as well as the circuit's internal memory states to generate a prediction of the incoming signals. The network learns by comparing the incoming signals with its prediction, updating its internal model of the world by minimizing the prediction errors at each level of the hierarchy in the style of predictive self-supervised learning. The network processes data in blocks of video frames rather than a frame-to-frame basis. This allows it to learn relationships among movement patterns, yielding state-of-the-art performance in long range video sequence predictions in benchmark datasets. We observed that hierarchical interaction in the network introduces sensitivity to memories of global movement patterns even in the population representation of the units in the earliest level. Finally, we provided neurophysiological evidence, showing that neurons in the early visual cortex of awake monkeys exhibit very similar sensitivity and behaviors. These findings suggest that predictive self-supervised learning might be an important principle for representational learning in the visual cortex.

Summary:
The paper presents a novel architecture for video prediction consisting of a feed-forward path with sparse convolutions and an LSTM generating predictions of chunks of video based on the sequence of input chunks. A feedback path links the LSTMs of the different sparse prediction modules. Experiments in video prediction are performed on moving-MNIST and the KTH action recognition dataset and the model achieves state-of-the-art performance on both. Interestingly, the model is exhibits prediction suppressions effects as have been observed during neurophysiological experiments in the inferotemporal cortex of macaque monkeys. The proposed method exhibits prediction suppression effects also in the lower layers, motivating a neurophysiological experiment in the earlier V1/V2 regions, which yielded an observation similar to the model’s prediction.
Strengths:
The performance improvements over competing methods on Moving-MNIST and KTH presented in the experimental section are significant. The analysis seems fairly thorough.
Weaknesses and requests for clarification:
- The description of the sparse predictive module is difficult to follow, and I am not sure I understood it completely. I find it a bit unintuitive to start the description with the errors, instead of explaining what is computed from beginning to end. The section reads more like a loose description of isolated parts instead of an integrated whole. Maybe walking the reader step-by-step through one complete iteration of the computation helps to clarify this. Also, not every character in equations 1-5 and the algorithm has been defined. For example, what is L?
- The text makes it sound like the idea of using 3d convolutions in a convLSTM is novel. 3D convLSTMs have been previously used in 3d vision, see
Choy, C. B., Xu, D., Gwak, J., Chen, K., & Savarese, S. (2016, October). 3d-r2n2: A unified approach for single and multi-view 3d object reconstruction. In European conference on computer vision (pp. 628-644). Springer, Cham.
The application of 3d convLSTMs to video might be new, but the mentioned paper by Choy et al. (2016) should be cited.
- You mention that padding is used for rows and columns. Are you using padding on the temporal axis as well?
- The paper seems to be written in a rush, as it contains way too many typos and grammar mistakes, e.g. “a hierarchical of” (should be “a hierarchy of“ or just “hierarchical”), “feedforwad”, “Expriment” (section 4 heading), “achievedbetter”, “trained monkeys to image pairs”, “pervious”, “perserves”, “processure”, “sequnence” “viusal”. Many typos could have been caught by a spellcheck! This would improve readability a lot!
- The citations are not properly formatted: (1) If the author names are used as part of the sentence, use e.g. Lotter et al. (2016), else (2) If the author names are not part of the sentence, use (Lotter et al., 2016). These two styles are mixed randomly in the current draft. This makes the manuscript, which already contains a lot of language mistakes, difficult to read.
- Abbreviations that are used but not introduced: CNN, IT, PSTH, DCNN, LSTM.
- The related work section could benefit from referring to some of the related work in neuroscience.
- Adding a sentence explaining the intuition behind using SatLU in equation (1) might be helpful
To summarize my feedback: I think experimental results and analysis are strong, but the presentation is strongly lacking! The description of the approach definitely needs to be improved to make replication of the results easier. It might help to have someone who doesn’t know the model already read the description and explain it back to you while revising the draft. I hope I could provide some helpful suggestions. I would recommend the manuscript for acceptance, if the presentation is significantly improved!


LEARNING EXPLORATION POLICIES FOR NAVIGATION
Numerous past works have tackled the problem of task-driven navigation. But, how to effectively explore a new environment to enable a variety of down-stream tasks has received much less attention. In this work, we study how agents can autonomously explore realistic and complex 3D environments without the context of task-rewards. We propose a learning-based approach and investigate different policy architectures, reward functions, and training paradigms. We find that use of policies with spatial memory that are bootstrapped with imitation learning and finally finetuned with coverage rewards derived purely from on-board sensors can be effective at exploring novel environments. We show that our learned exploration policies can explore better than classical approaches based on geometry alone and generic learning-based exploration techniques. Finally, we also show how such task-agnostic exploration can be used for down-stream tasks. Videos are available at: https://sites.google.com/view/exploration-for-nav/.

This paper proposes a method for learning how to explore environments. The paper mentions that the “exploration task” that is defined in this paper can be used for improving the well-known navigation tasks. For solving this task, a reward function a network architecture that uses RGBD images + reconstructed map + imitation learning + PPO is designed.
<<Pros>>
-The paper is well-written (except for a few typos).
-The overall approach is simple and does not have much complications.
-The underling idea and motivation is clearly narrated in the intro and abstract and the paper has a easy-to-understand flow.
<<Cons>>
**The technical novelty is not significant**
-This paper does not provide significant technical novelty. It is a combination of known prior methods: imitation learning + ppo (prior RL work). The presented exploration task is not properly justified as to how it could be useful for the navigation task. The reconstruction of maps for solving the navigation problem is a well-explored problem in prior SLAM and 3D reconstruction methods. Overall the novelty of the approach and the proposed problem is incremental.
**The paper has major short comings in the experimental section. The presented experiments do not support the main claim of the paper which is improving the performance in the well-known navigation task. Major baselines are missing. Also, the provided results are not convincing in doing the right comparison with the baselines. **
-Experimental details are missing. The major experimental evaluations (Fig. 2 and Fig. 3) are based on the m^2 coverage after k steps and the plots are cut at 1000 steps. What are the statistical properties of the 3D houses used for training and testing? E.g what is their area in m^2? How big is each step in meters?  Why are the graphs cut at 1000 steps? How would different methods converge after more than 1000 steps, e.g. 2000 steps? I would like to see how would the different methods converge after larger number of steps? How long would each step take in terms of time? How could these numbers convey the significance of the proposed method in a real would problem settings?
-The experiments do not convey if learning has significantly resulted in improved exploration. Consider a simple baseline that follows a similar approach as explained in the paper for constructing the occupancy map using the depth sensor. A non-learning agent could use this map at each step to make a greedy choice about its next action which greedily maximizes the coverage gain based on its current belief of the map. While the performance of random policy is shown in Fig.2 the performance of this greedy baseline is a better representative of the lower bound of the performance on the proposed task and problem setup.
-What is the performance of a learning-based method that only performs collision avoidance? Collision avoidance methods tend to implicitly learn to do a good map coverage. This simple baseline can show a tangible lower bound of a learning-based approach that does not rely on map.
-The major promise of the paper is that the proposed exploration task can improve navigation. However, the navigation experiment does not compare the proposed method with any of prior works in navigation. There is a huge list of prior methods for navigation some of which are cited in the “learning for Navigation” section of the related works and the comparison provided in Fig. 4 is incomplete compared to the state-of-the-arts in navigation. For example, while the curiosity driven approach is compared for the exploration, the more related curiosity based navigation method which uses both “exploration strategy” and “imitation learning” : “Pathak, Deepak, et al. "Zero-shot visual imitation." International Conference on Learning Representations. 2018.
“ is missed in navigation comparison. The aforementioned paper is also missed in the references.
-Algorithmic-wise, it would make the argument of the paper clearer if results were conducted by running different exploration strategies for navigation to see if running RL with a good exploration strategy could solve the exploration challenge of the navigation problem without needing an explicit exploration stage (similar to the proposed method) which first explores and constructs the map and then does navigation by planning.
-The navigation problem as explained in section is solved based on planning approach that uses a reconstructed map. This is a fairly conventional approach that SLAM based methods use. Therefore, comparison with a SLAM method that constructs the map and then does navigation would be necessary.
** Technical details are missing or not explained clearly**
- Section 3.1 does not clearly explain the map construction. It seems that the constructed map is just a 2D reconstruction of the space (and not 3D) using the depth sensor which does not need transformation of the 3D point cloud. What is the exact 3D transformation that you have done using the intrinsic camera parameters? This section mentions that there can be error in such map reconstruction because of robot noise but alignment is not needed because the proposed learning method provides robustness against miss-alignment. How is this justified? Why not using the known loop closure techniques in SLAM?
-The technical details about the incorporated imitation learning method are missing. What imitation learning method is used? How is the policy trained during the imitation learning phase?
-Last paragraph of intro mentions that the proposed method uses 3D information efficiently for doing exploration. The point of this sentence is unclear. What 3D information is used efficiently in the paper? Isn’t it only 2.5D (information obtained by depth sensor) used in the proposed method?
**Presentation can be improved**
-The left and right plots of the Figure 3 contains lots of repetitions which brings in confusion in comparing the performance of runs with different settings. These two plots should be presented in a single plot.
- Interpretation of “green vs white vs black” in the reconstructed maps is left to the reader in Fig. 1.
- Last line in page 5: there is no need for reiteration. It is already clear.
**Missing references**
-Since the paper is about learning to explore, discussion about “exploration techniques in RL” is recommended to be added in at least the related work section.
-A big list papers for 3D map reconstruction is missing. Since the proposed method relies on a map reconstruction, those papers are relevant to this work and can potentially be used for comparison (as explained above). It is highly recommended that relevant prior 3D map reconstruction papers be added to the related work sections.


ADVERSARIAL ATTACKS ON GRAPH NEURAL NETWORKS VIA META LEARNING
Deep learning models for graphs have advanced the state of the art on many tasks. Despite their recent success, little is known about their robustness. We investigate training time attacks on graph neural networks for node classification that perturb the discrete graph structure. Our core principle is to use meta-gradients to solve the bilevel problem underlying training-time attacks, essentially treating the graph as a hyperparameter to optimize. Our experiments show that small graph perturbations consistently lead to a strong decrease in performance for graph convolutional networks, and even transfer to unsupervised embeddings. Remarkably, the perturbations created by our algorithm can misguide the graph neural networks such that they perform worse than a simple baseline that ignores all relational information. Our attacks do not assume any knowledge about or access to the target classifiers.

This paper studies the problem of learning a better poisoned graph parameters that can maximize the loss of a graph neural network. The proposed using meta-learning to compute the second-order derivatives to get the meta-gradients seems reasonable. The authors also proposed approximate methods to compute the graph as learning parameters, which could be more efficient since the second-order derivatives are no longer computed. The experimental results on three graph datasets show that the proposed model could improve the misclassification rate of the unlabeled nodes.
The paper is well-written. It would be good if the authors could address the following suggestions or concerns:
1) The proposed attack model assumes the only the graph structure are accessiable to the attackers, which might limit the proposed model in real applications. Joint study with the graph features would be useful to convince more audience and potentially have larger impacts.
2) In the self-learning setting, in order to define l_atk, l_self is used, however, l_self is using v_u, which is the ground truth label of the test nodes based on my understanding, so this approach is using labels of the unlabeled data, which might be not applicable in real world.
3) About the action space, based on the constraints of the attacker's capability, the possible attacks will be significantly smaller than O(N^2 delta), might be O(N^delta).
4) Change 'treat the graph structure as a hyperparameter' to 'treat the graph structure tensor/matrix as a hyperparameter' would be earier to understand. And is the graph structure tensor with shape (NXN)?
5) What's the relationship between T and S? Are T in theta_T is the same as the S in G_S?
6) The title of section 4.2 is misleading. It would be better to name it as 'Greedy Computing Meta-Gradients'.
7) It lacks intuition of why define S(u,v)=delta . (-2.a_uv+1). '(-2.a_uv+1)' looks lack of intuition. Please also change 'pair (i,j), we define S(u,v)' -> 'pair (u,v)'.
8) In the experiments, what's the definition of meta-train? l_atk=-l_train?
9) In the experiments, it would be interesting to study the impact of unnoticaability constraints on the model results.
10) In figure 1, it is not surprising that when increasing the number of edges changed, the misclassification rates will increase. A graph NN considers more graph features rather than the structure is expected to show the impact of the graph structure change.
I have read the authors' detailed rebuttal. Thanks.


Adversarial examples have been shown to be an effective way of assessing the robustness of neural sequence-to-sequence (seq2seq) models, by applying perturbations to the input of a model leading to large degradation in performance. However, these perturbations are only indicative of a weakness in the model if they do not change the semantics of the input in a way that would change the expected output. Using the example of machine translation (MT), we propose a new evaluation framework for adversarial attacks on seq2seq models taking meaning preservation into account and demonstrate that existing methods may not preserve meaning in general. Based on these findings, we propose new constraints for attacks on word-based MT systems and show, via human and automatic evaluation, that they produce more semantically similar adversarial inputs. Furthermore, we show that performing adversarial training with meaning-preserving attacks is beneficial to the model in terms of adversarial robustness without hurting test performance.

Summary: Proposes a framework for performing adversarial attacks on an NMT system in which perturbations to a source sentence aim to preserve its meaning, on the theory that an existing reference translation will remain valid if this is done. Given source and target metrics for measuring similarity, an attack is deemed successful if the source difference is smaller than the relative decrease in target similarity to the reference. A first experiment measures correlation with human judgements of similarity between original and perturbed sentences, and concludes that chrF is better than BLEU and METEOR for this purpose. Next, standard gradient-based adversarial attacks are carried out, replacing the three tokens that result in the biggest drop in (approximate) reference probability, either 1) with no constraints, 2) constrained to character swaps of the original token, or 3) constrained be among the 10 closest embeddings to the original token. In comparisons on three language pairs from IWSLT,  the constrained attacks are found to preserve meaning and yield more successful attacks according to the current framework. The Transformer architecture was also found to deal less well with attacks under the 10-closest embedding constraint. Finally, adversarial training with the character-swap constraint confers some robustness to this attack, without degrading performance on normal text.
I think it is a good idea to formalize a method for carrying out and assessing adversarial attacks, but the framework proposed here seems too narrow, as it excludes adversarial inputs that are sensible but not a close perturbation of an existing source/reference pair, or ones that contain varying amounts of noise. It is more difficult to measure output quality for such attacks, but that doesn’t seem like a good reason for excluding them from what is intended to be a general framework. Note also that “more difficult” doesn’t mean impossible, since good attacks can produce severely degraded output that is relatively easy to detect.
I found some of the methodology questionable. Limiting source perturbations to character swaps and neighbors in embedding space, then using automatic metrics to measure semantic distance seems both unnecessary and unlikely to succeed. Unnecessary because knowing the class of perturbation already gives you a lot of information about semantic distance. Unlikely to succeed because automatic metrics are too coarse to reliably distinguish among different perturbations. This is particularly obvious in the case of using character ngram distance (chrF) to determine which character swaps preserve meaning best. The experiments that support the viability of automatic metrics in 4.2 do so by measuring correlation with human judgment when the number of perturbed tokens varies from 1 to 3. I think the good correlation is likely due to the metrics being able to detect that, eg, changing 3 tokens makes things worse than changing only one. To be convincing, the experiments would have to be repeated with number of perturbations fixed at 3, to match the setting in the remaining experiments.
Apart from the interesting observation about the Transformer’s performance on embedding-neighbor attacks mentioned above, it is difficult to know what conclusions to draw from the experiments. In 4.3 it seems obvious a priori that perturbations intended to be relatively meaning preserving would indeed preserve meaning better than unconstrained ones. Similarly, it is not surprising that character swaps that by design produce an OOV token will cause more damage than choosing a near neighbor in embedding space. In 5.3, training with OOVs (resulting from character swaps) is of course not likely to hurt performance on test sets containing few OOVs, and, as is known from previous work, it will improve robustness to the same kind of noise. A final comment about the experiments is that word-based systems are not state of the art, and it isn’t clear how much we could expect any conclusions to carry over to sub-word models.
To conclude, although this is an interesting initiative, both the framework and the methodology need to be tightened up.
Details:
End of 2.1: this would be easier to interpret if you had previously specified the allowed range for s_src.
3.2 For kNN, being semantically related doesn’t imply that the relationship is synonymy, as would be required for meaning preservation. It also doesn’t imply that the substitution will be grammatical, which could jeopardize meaning preservation even if the words are synonyms.
CharSwap seems odd. If you’re just going to replace a work with an OOV symbol in any case, why go to the trouble of swapping characters? No matter what actual semantic shift is caused by the swap, the model will always see exactly the same representation.
4.1 “Following previous work on adversarial examples for seq2seq models (Belinkov & Bisk, 2018; Ebrahimi et al., 2018a)” - this is misleading: Ebrahimi et al only work with classification, and don’t use IWLST.
4.1 Should mention the size of the training sets in this section.
Table 1, first sentence, CharSwap example omits “faire”.
4.3, “Adding Constraints Helps Preserve…” last sentence: but here you need to reason in the opposite direction.
5.2 It would be good to also give absolute scores for table 6, so we can judge how much the systems actually benefited, and whether these gains were statistically significant.


OPTIMAL TRANSPORT MAPS FOR DISTRIBUTION PRESERVING OPERATIONS ON LATENT SPACES OF GENERATIVE MODELS
Generative models such as Variational Auto Encoders (VAEs) and Generative Adversarial Networks (GANs) are typically trained for a fixed prior distribution in the latent space, such as uniform or Gaussian. After a trained model is obtained, one can sample the Generator in various forms for exploration and understanding, such as interpolating between two samples, sampling in the vicinity of a sample or exploring differences between a pair of samples applied to a third sample. However, the latent space operations commonly used in the literature so far induce a distribution mismatch between the resulting outputs and the prior distribution the model was trained on. Previous works have attempted to reduce this mismatch with heuristic modification to the operations or by changing the latent distribution and re-training models. In this paper, we propose a framework for modifying the latent space operations such that the distribution mismatch is fully eliminated. Our approach is based on optimal transport maps, which adapt the latent space operations such that they fully match the prior distribution, while minimally modifying the original operation. Our matched operations are readily obtained for the commonly used operations and distributions and require no adjustment to the training procedure.

Noticing that widely used latent code interpolations for exploring the generative capabilities of VAEs and GANs have distribution mismatch problems, this paper proposes to utilize monotone transport map to exactly eliminate the distribution mismatch between modified interpolated codes and a prior distribution, assuming I.I.D. code components and a L1 code distance. More precisely, a transformation of the latent space operation is learnt with the objective that the distribution of the transformed variable match the prior distribution used in training the generative models. Optimal transport is used as a measure to minimize the two distributions. By restricting the class of cost functions used in the optimal transport formulation, the solution to the optimal transport problem (and hence the transformation function) has been shown to take a simple form (closed form in cases where cdf has a analytical form). Experiments on CIFAR-10, LLD-icon, LSUN, CelebA datasets show that, the minimally modified interpolated codes for several different interpolations produce samples with higher Inception Scores and better visual effects under an improved Wasserstein GAN than the original interpolated codes.
This paper is well written, the studied problem is highly important, and the approach presented has potentially wide applications.
However, there are some concerns about the experimental evaluations,
1. Although the quantitative evaluations for 2-point and 4-point interpolations are important, it is hard to assess these interpolations in a semantically meaningful way. Extensive quantitative (FID and IS) and qualitative evaluations should be conducted for analogy interpolations. For example, adding glasses, adding mustache, and many others. It is much easier to assess the quality of the generated images from the minimally modified interpolated code for this category in a meaningful way.
2. Another concern is that how big the effect of the transformation function inducing on the latent space operations will be. For example, a linear interpolation is no longer linear after getting transformed. So, are there transformations that drastically transform the original latent space operations? In that case, will the transformed variable make any sense with respect to the original latent space operations? Extensive experiments for analogy interpolations are required to answer these questions.
3. Experiments have been shown only on GAN architectures, however, the framework can be easily extended to VAEs. Experiments on VAEs will be informative.
Minor:
Section 1.1, in the second paragraph, (SLERP) should be moved a correct position.
Figure 2: it's better to use a different color for midpoint linear other than blue
Problem 1, f* ---> f*:


Recent studies have shown that reinforcement learning (RL) models can be vulnerable in various scenarios, where noises from different sources could appear. For instance, the observed reward channel is often subject to noise in practice (e.g., when observed rewards are collected through sensors), and thus observed rewards may not be credible. Also, in applications such as robotics, a deep reinforcement learning (DRL) algorithm can be manipulated to produce arbitrary errors. In this paper, we consider noisy RL problems where observed rewards by RL agents are generated with a reward confusion matrix. We call such observed rewards as perturbed rewards. We develop an unbiased reward estimator aided robust RL framework that enables RL agents to learn in noisy environments while observing only perturbed rewards. Our framework draws upon approaches for supervised learning with noisy data. The core ideas of our solution include estimating a reward confusion matrix and defining a set of unbiased surrogate rewards. We prove the convergence and sample complexity of our approach. Extensive experiments on different DRL platforms show that policies based on our estimated surrogate reward can achieve higher expected rewards, and converge faster than existing baselines. For instance, the state-of-the-art PPO algorithm is able to obtain 67.5% and 46.7% improvements in average on five Atari games, when the error rates are 10% and 30% respectively.

This paper investigates reinforcement learning with a perturbed reward signal. In particular, the paper proposes a particular model for adding noise to the reward function via a confusion matrix, which offers a nuanced notion of reward-noise that is not too complicated so-as to make learning impossible. I take this learning setting to be both novel and interesting for opening up areas for future work. The central contributions of the work are to 1) leverage a simple estimator to prove the convergence of Q-Learning under the reward-perturbed setting along with the sample-complexity of a variant of (Phased) Q-Learning which they call "Phrased" Q-Learning, and 2) An algorithmic scheme for learning in the reward-perturbed setting (Algorithm 1), and 3) An expansive set of experiments that explore the impact of various reward models on learning across different environment-algorithm combinations. The sample complexity term extends Phased Q-Learning to incorporate aspects of the reward confusion matrix, and to my knowledge is novel. Further, even though Theorem 1 is unsurprising (as the paper suggests), I take the collection of Theorem 1, 2, and 3 to be collectively novel.
Indeed, the paper focuses on an interesting and relatively unexplored direction for RL. Apart from the work cited by the paper (and perhaps work like Krueger et al. (2016), in which agents must pay some cost to observe true rewards), there is little work on learning settings of this kind. This paper represents a first step in gaining clarity on how to formalize and study this problem. I did, however, find the analysis and the experiments to be relatively disjointed -- the main sample complexity result presented by the paper (Theorem 2) was given for Phased Q-Learning, yet no experiments actually evaluate the performance of Phased Q-Learning. I think the paper could benefit from experiments focused on simple domains that showcase how traditional algorithms do in cases where it is easier to understand (and visualize) the impact of the reward perturbations (simple chain MDPs, grid worlds, etc.) -- and specifically experiments including Phased Q-Learning.
Pros:
	- General, interesting new learning setting to study.
	- Initial convergence and sample complexity results for this new setting.
	- Depth and breadth of experimentation (in terms of diversity of algorithms and environments), includes lots of detail about the experimental setup.
Cons:
	- Clarity of writing: lots of typos and bits of math that could be more clear (see detailed comments below) [Fixed]
	- The plots in Section 4 are all extremely jagged. More trials seem to be required. Moreover, I do think simpler domains might help offer insights into the reward perturbed setting. [Fixed]
	- The reward perturbation model is relatively simple.
Some high level questions/comments:
	- Why was Phrased Q-Learning not experimented with?
	- Why use majority voting as the rule? When this was introduced it sounded like any rule might be used. Have you tried/thought about others?
	- Your citation to Kakade's thesis needs fixing; it should read:
		"Kakade, Sham Machandranath. On the sample complexity of reinforcement learning. Ph.D Thesis. University of London, 2003."
		(right now it is cited as "(Gatsby 2003)" throughout the paper)
	- You might consider picking a new name for Phrased Q-Learning -- right now the name is too similar to Phased Q-Learning from [Kearns and Singh NIPS 1999].
        - As mentioned in the "cons" section, the confusion matrix is still a somewhat simple model of reward noise. I was left wondering: what might be the next most complicated form of adding reward noise? How might the proposed algorithm(s) respond to this slightly more complex model? That is, it's unclear how general the results are, or if they are honed too tightly to the specific proposed reward noise model. I was hoping the authors could respond to this point.

Section 0) Abstract:
	- Not immediately clear what is meant by "vulnerability" or "noisy settings". Might be better to pick a more clear initial sentence (same can be said of the "sources of noise..."")
Section 1) Introduction:
	- "adversaries in real-world" --> "adversaries in the real-world"
	- You might consider citing Loftin et al. (2014) regarding the bulleted point about "Application-Specific Noise".
	- "unbiased reward estimator aided reward robust reinforcement learning framework" --> this was a bit hard to parse. Consider making more concise, like: "unbiased reward estimator for use in reinforcement learning with perturbed rewards".
	- "Our solution framework builds on existing reinforcement learning algorithms, including the recently developed DRL ones" --> cite these up front So, cite: Q-Learning, CEM, SARSA, DQN, Dueling DQN, DDPG, NAF, and PPO, and spell out the acronym for each the first time you introduce them.
	- "layer of explorations" --> "layer of exploration"
Section 2) Problem Formulation
	- "as each shot of our" --> what is 'shot' in this context?
	- "In what follow," --> "In what follows,"
	- "where 0 < \gamma \leq 1" --> Usually, $\gamma \in [0,1)$, or $[0,1]$. Why can't $\gamma = 0$?
	- The transition notation changes between $\mathbb{P}_a(s_{t+1} | s_t)$ and $\mathbb{P}(s_{t+1} | s_t, a_t)$. I'd suggest picking one and sticking with it to improve clarity.
	- "to learn a state-action value function, for example the Q-function" --> Why is the Q-function just an example? Isn't is *the* state-action value function? That is, I'd suggest replacing "to learn a state-action value function, for example the Q-function" with "to learn a state-action value function, also called the Q-function"
	- "Q-function calculates" --> "The Q-function denotes"
	- "the reward feedbacks perfectly" --> "the reward feedback perfectly"
	- I prefer that the exposition of the perturbed reward MDP be done with C in the tuple. So: $\tilde{M} = \langle \mathcal{S}, \mathcal{A}, \mathcal{R}, C, \mathcal{P}, \gamma \rangle$. This seems the most appropriate definition, since the observed rewards will be generated by $C$.
	- The setup of the confusion matrix for reward noise over is very clean. It might be worth pointing out that $C$ need not be Markovian. There are cases where C is not just a function of $\mathcal{S}$ and $\mathcal{R}$, like the adversarial case you describe early on.
Section 3) Learning w/ Perturbed Rewards
	- Theorem 1 builds straightforwardly on Q-Learning convergence guarantee (it might be worth phrasing the result in those terms? That is: the addition of the perturbed reward does not destroy the convergence guarantees of Q-Learning.)
	- "we firstly" --> "we first"
	- "value iteration (using Q function)" --> "value iteration"
	- "Definition 2. Phased Q-Learning" --> "Definition 2. Phrased Q-Learning". I think? Unless you're talking about Phased Q from the Kearns and Singh '99 work.
	- "It uses collected m samples" --> "It uses the collected m samples"
	- Theorem 2: it would be helpful to define $T$ since it appears in the sample complexity term. Also, I would suggest specifying the domain of $\epsilon$, as you do with $\delta$.
	- "convergence to optimal policy" --> "convergence to the optimal policy"
	- "The idea of constructing MDP is similar to" --> this seems out of place. The idea of constructing which MDP? Similar to Kakade (2003) in what sense?
	- "the unbiasedness" --> "the use of unbiased estimators"
	- "number of state-action pair, which satisfies" --> "number of state-action pairs that satisfy"
	- "The above procedure continues with more observations arriving." --> "The above procedure continues indefinitely as more observation arrives." Also, which procedure? Updating $\tilde{c}_{i,j}$? If so, I would specify.
	- "is nothing different from Eqn. (2) but with replacing a known reward confusion" --> "replaces a known reward confusion"
4) Experiments:
	- Diverse experiments! That's great. Lots of algorithms, lots of environment types.
	- I expected to see Phrased Q-Learning in the experiments. Why was it not included?
	- The plots are pretty jagged, so I'm left feeling a bit skeptical about some of the results. The results would be strengthened if the experiments were repeated for more trials.
5) Conclusion:
	- "despite of the fact" --> "despite the fact"
	- "finite sample complexity of Q-Learning with estimated surrogate rewards are given" --> It's not really Q-Learning, though. It's a variant of Q-Learning. I'd suggest being explicit about that.
Appendix:
	- "It is easy to validate the unbiasedness of proposed estimator directly." --> "It is easy to verify that the proposed estimator is unbiased directly."
	- "For the simplicity of notations" --> "For simplicity"
	- "the Phrased Q-Learning could converge to near optimal policy" --> ""the algorithm Phrased Q-Learning can converge to the near optimal policy""
	- "Using union bound" --> "Using a union bound"
	- Same comment regarding $\gamma$: it's typically $0 \leq \gamma < 1$.
	- Bottom of page 16, the second equation from the bottom, far right term: $c.j$ --> $c,j$.
	- "Using CauchySchwarz Inequality" --> "Using the Cauchy-Schwarz Inequality"
References:
	Loftin, Robert, et al. "Learning something from nothing: Leveraging implicit human feedback strategies." Robot and Human Interactive Communication, 2014 RO-MAN: The 23rd IEEE International Symposium on. IEEE, 2014.
	Krueger, D., Leike, J., Evans, O., & Salvatier, J. (2016). Active reinforcement learning: Observing rewards at a cost. In Future of Interactive Learning Machines, NIPS Workshop.


LEARNING PROTEIN SEQUENCE EMBEDDINGS USING INFORMATION FROM STRUCTURE
Inferring the structural properties of a protein from its amino acid sequence is a challenging yet important problem in biology. Structures are not known for the vast majority of protein sequences, but structure is critical for understanding function. Existing approaches for detecting structural similarity between proteins from sequence are unable to recognize and exploit structural patterns when sequences have diverged too far, limiting our ability to transfer knowledge between structurally related proteins. We newly approach this problem through the lens of representation learning. We introduce a framework that maps any protein sequence to a sequence of vector embeddings -one per amino acid position -that encode structural information. We train bidirectional long short-term memory (LSTM) models on protein sequences with a two-part feedback mechanism that incorporates information from (i) global structural similarity between proteins and (ii) pairwise residue contact maps for individual proteins. To enable learning from structural similarity information, we define a novel similarity measure between arbitrarylength sequences of vector embeddings based on a soft symmetric alignment (SSA) between them. Our method is able to learn useful position-specific embeddings despite lacking direct observations of position-level correspondence between sequences. We show empirically that our multi-task framework outperforms other sequence-based methods and even a top-performing structure-based alignment method when predicting structural similarity, our goal. Finally, we demonstrate that our learned embeddings can be transferred to other protein sequence problems, improving the state-of-the-art in transmembrane domain prediction.

General comment
==============
The authors describe two loss functions for learning embeddings of protein amino acids based on i) predicting the global structural similarity of two proteins, and ii) predicting amino acid contacts within proteins. As far as I know, these loss functions are novel and the authors show clear improvements when using the learned embeddings in downstream tasks. The paper is well motivated and mostly clearly written. However, the evaluation must be strengthened and some aspects of it clarified. Provided that the authors address my comments below, I think it is a good ICLR application paper.
Major comments
=============
1. The authors should describe how they optimized hyperparameters such as the learning, lambda (loss section 3.3), or the smoothing factor (section 3.4). These should be optimized on an evaluation set, but the authors only mentioned that they split the dataset into training and holdout (test) set (section 4.1).
2. The way the authors present results in table 1 and table 2 is unclear. Both table 1 and table 2 contain results of the structural similarity tasks but with different baselines. ‘SSA w/ contact predictions’ is also undefined and can be interpreted as ‘with’ or ‘without’ contacts predictions. I therefore strongly recommend to show structural similarity results in table 1 and secondary structure results in table 2 and include in both tables i) ‘SSA full’, ‘SSA without contact predictions’, and ‘SSA without language model’.
3. The authors should compare SSA to the current state-of-the art in structure prediction in addition to baseline models.
4. The authors should evaluate how well their method predicts amino acid contact maps.
5. The authors should describe how they were dealing with variable-length protein sequences. Are sequences truncated and embedded to a fixed length? What is the mean and variance in protein sequence lengths in the considered datasets? The authors should point out that their method is limited to fixed length sequences.
6. The authors should briefly describe the training and inference time on a single GPU and CPU. How much memory is required for training with a certain sequence length, e.g. 400 amino acids per sequence? Does the model fit on a single GPU?
7. The authors should discuss limitations of their method, e.g. that it cannot handle variable length sequences and that the memory scales quadratically by the the sequence length.
8. CRF (SSA) (table 3) includes a biLSTM layer between SSA and the CRF. However, the biLSTM can learn a non-linear projection of embeddings learned by SSA such that it is unclear if improvements are due to the embeddings learned by SSA or the biLSTM+CRF architecture. The authors should therefore train a biLSTM+CRF model on one-hot encoded amino-acids and include it as baseline in table 3.
Minor comments
=============
9. The way the similarity score s’ is computed (section 3.2.1) should be motivated more clearly. Why do the authors compute the score s’ manually instead of predicting it, e.g. using a model that takes the embeddings z of both proteins as input and predicts a single scalar s’?
10. How does ordinal regression (section 3.2.2) perform compared with a softmax layer? Why do the authors compute s’ and then train logistic regression classifiers on s’ to predict the similarity level, instead of predicting the similarity level directly based on the embeddings z?
11. Why do the authors use a distance threshold of 8A (section 3.3)? Is this common practice in the field?
12. Why do the authors use the not product and the absolute difference as features instead of the embeddings z directly? Which activation function is used to predict contact probabilities (sigmoid, softmax, …)?
13. The authors should reference and describe the results presented in table 1 more clearly.
14. Optional: the authors should analyze if learned embeddings are correlated with amino acid and structural properties such as their size, charge, or solvent accessibility. Do embeddings clusters by certain properties? This can be analyzed, e.g., using a tSNE plot.
15. How does TMalign perform when using the maximum or geometric average instead of the arithmetic average of the two scores (section 4.1)


Giving provable guarantees for learning neural networks is a core challenge of machine learning theory. Most prior work gives parameter recovery guarantees for one hidden layer networks, however, the networks used in practice have multiple non-linear layers. In this work, we show how we can strengthen such results to deeper networks -we address the problem of uncovering the lowest layer in a deep neural network under the assumption that the lowest layer uses a high threshold before applying the activation, the upper network can be modeled as a well-behaved polynomial and the input distribution is gaussian.

This paper gives provable recovery guarantees for a class of neural networks which have high-threshold activation in the first layer, followed by a "well-behaved" polynomial, under Gaussian input. The algorithm is based on the approach by Ge et al. (2017), as well as an iterative refinement method.
While this could be an interesting result, I have several concerns regarding the assumptions, correctness, and writing.
1) It is required that the threshold is at least sqrt{log d} (Thm. 1), where d is the number of hidden neurons in the first layer. It seems that this essentially zeros out almost all the neurons, since the maximum among d Gaussian random variables is roughly sqrt{log d}. The authors should explain what exactly this model is doing, i.e., what kind of functions it can compute, in order to justify why this is an interesting model.
Furthermore, the authors claim that the studied model is a "deep" neural network, but I disagree. As I understand, the difference between this model and two-layer networks is that the second layer here is a polynomial instead of a linear function. This doesn't make it a deep network since the (polynomial) part above the first layer is not modeled in a layer-wise fashion, not to mention that under the setting considered in the paper the polynomial behaves similar to a linear function.
2) It is stated at the end of Section 2 that the angle can be reduced by a factor of 1-1/d ***with constant probability***. How does this ensure you can succeed after O(d log(1/nu)) iterations? As far as I see you need the success probability in one iteration to be at least something like 1-1/Omega(d) so that you can apply a union bound.
3) Even if the issues of motivation and correctness are clarified, I find it very difficult to understand the overall intuition and main technical contributions in this paper. The writing needs to be significantly improved to reach the level of a top conference.


Generative models have proven to be an outstanding tool for representing highdimensional probability distributions and generating realistic looking images. A fundamental characteristic of generative models is their ability to produce multimodal outputs. However, while training, they are often susceptible to mode collapse, which means that the model is limited in mapping the input noise to only a few modes of the true data distribution. In this paper, we draw inspiration from Determinantal Point Process (DPP) to devise a generative model that alleviates mode collapse while producing higher quality samples. DPP is an elegant probabilistic measure used to model negative correlations within a subset and hence quantify its diversity. We use DPP kernel to model the diversity in real data as well as in synthetic data. Then, we devise a generation penalty term that encourages the generator to synthesize data with a similar diversity to real data. In contrast to previous state-of-the-art generative models that tend to use additional trainable parameters or complex training paradigms, our method does not change the original training scheme. Embedded in an adversarial training and variational autoencoder, our Generative DPP approach shows a consistent resistance to modecollapse on a wide-variety of synthetic data and natural image datasets including MNIST, CIFAR10, and CelebA, while outperforming state-of-the-art methods for data-efficiency, convergence-time, and generation quality. Our code will be made publicly available.

The paper proposes to introduce DPP into the vanilla GAN loss and uses it as a way to regularize the generator to produce more diverse outputs, in order to combat the mode-collapse problem. Since the proposed method is added as a simple loss regularizer, the approach does not introduce additional parameters, therefore, less training difficulties. The results on synthetic data seems promising, but there is insufficient evaluation being performed on real and larger dataset where the mode collapse problems are more likely to happen.
- Method
The proposed methods seem sensible. But there are some critical details missing from the current text that prevents me from assessing this paper clearly.
* How are the \lambda and v in Eq.6 calculated? It seems to me that you need to estimate the eigenvalues and eigenvectors at every iteration of your training. I am aware of that many DPP-based models suffer from scalability issues. Could you discuss the potential overhead of this procedure? Also in experiments, you claim "DPP-GAN is cost-free, we observe that the training of GDPP-GAN has an indistinguishable running time than the DCGAN, which are the fastest models to finish training“, which is hard to believe.. could you give more details and analysis on the overhead here?
* In Eq.6 why there are both  "a diversity magnitude loss L_m", and "diversity structure loss L_s". What do they specifically try to capture respectively? Could you give a geometric interpretation on this part?
* what is the batch size used in your experiments on MNIST and CIFAR-10. It seems to me that the effectiveness of GDPP would rely on batch size used as per my understanding you will estimate the DPP kernel using the current batch of samples (generated or real)?
* Despite the fact that GDPP wants to reduce parameters introduced, it is not very intuitive to understand how it would work to use the features outputted by D as the DPP features as well. As D, as the discriminator itself, is trained to distinguish real from fake, while mimicking the eigenvalues/vectors of real data. How would these two goals be reconciled by the same set of parameters?
- Experiments
The results on the synthetic data seem promising, but the results on MNIST and CIFAR-10 are not impressive enough:
* The visual quality of Figure.9 does not look very appealing. I believe many simple variants DCGAN can produce better quality of images
* Why your DCGAN baseline on CIFAR-10 only reports inception score around 5 (with high variance, see Figure.5 and Table 3)? I believe vanilla DCGANs can easily attain an IS at 5.5 to 6, as reported in most recent GAN literature.
* More visual results on CIFAR10 should have been presented in order to demonstrate DPP does generate images with as many classes as existed in CIFAR-10 (which is 10)
* The results could be much more convincing if the authors could show the generation results and evaluation metrics on larger/more real datasets other than CIFAR-10 and MNIST. See GAN literature in 2018 about what dataset to use.
- Presentation
Most parts of this paper are well written. There are few typos and grammatical errors across the text which I believe are easily fixable. There are some missing details that hinder the understanding of some technical parts of the paper. See above for detailed comments.
- Other
Promoting diversity in (deep) generative models isn't a new topic. It would be good if the authors could established connections/differences between this work and this line of relevant  works.


Communication on heterogeneous edge networks is a fundamental bottleneck in Federated Learning (FL), restricting both model capacity and user participation. To address this issue, we introduce two novel strategies to reduce communication costs: (1) the use of lossy compression on the global model sent serverto-client; and (2) Federated Dropout, which allows users to efficiently train locally on smaller subsets of the global model and also provides a reduction in both client-to-server communication and local computation. We empirically show that these strategies, combined with existing compression approaches for client-toserver communication, collectively provide up to a 14× reduction in server-toclient communication, a 1.7× reduction in local computation, and a 28× reduction in upload communication, all without degrading the quality of the final model. We thus comprehensively reduce FL's impact on client device resources, allowing higher capacity models to be trained, and a more diverse set of users to be reached.

The paper presents some new approaches for communication efficient Federated Learning (FL) that allows for training of large models on heterogeneous edge devices. In FL, heterogeneous edge devices have access to potentially non-iid samples of data points and try to jointly learn a model by averaging their local models at a parameter server (the cloud). As the bandwidth of the up/downlink-link may be limited communication overheads may become the bottleneck during FL. Moreover, due to the heterogeneity of the hardware, large models may be hard to train on small devices. Due to that, there are several recent approaches that aim to minimize communication via methods of quantization, which also aim to allow for smaller models via methods of compression and model quantization.
In this paper, the authors suggest a combination of two methods to reduce communication and allow for large model training by 1) using a lossy compressed model when that is communicated from the cloud to the edge devices, and 2) subsampling the gradients, a form of dropout, at the edge device side that allows for an overall smaller model update. The novelty of either of those techniques is quite limited as individually they have been suggested before, but the combination of both of them is interesting.
The paper is overall well written, however there are two aspects that make the contribution lacking in novelty. First of all, the presented methods are a combination of existing techniques, that although interesting to combine together, are neither theoretically analyzed nor extensively tested. The model/update quantization technique has been used in the past extensively [eg 1-3]. Then, the “federated dropout” can be seen as a “coordinate descent” type of a technique, i.e., randomly zeroing out gradient elements per iteration.
Since this is a more experimental paper, the setup tested is quite limited in its comparisons. For example, one would expect to see extensive comparisons with methods for quantizing gradients, eg QSGD, or Terngrad, and combinations of that with DeepCompression. Although the authors do make an effort to experiment with a different set of hyperparameters (dropout probability, quantization levels, etc), a comparison with state of the art methods is lacking.
Overall, although the combination of the presented ideas has some merit, the lack of extensive experiments that would compare it with the state of the art is not convincing, and the overall effectiveness of this method is unclear at this point.
[1] https://arxiv.org/pdf/1510.00149.pdf
[2] https://arxiv.org/pdf/1803.03383.pdf
[4] https://arxiv.org/pdf/1610.05492.pdf


Neural network-based methods have recently demonstrated state-of-the-art results on image synthesis and super-resolution tasks, in particular by using variants of generative adversarial networks (GANs) with supervised feature losses. Nevertheless, previous feature loss formulations rely on the availability of large auxiliary classifier networks, and labeled datasets that enable such classifiers to be trained. Furthermore, there has been comparatively little work to explore the applicability of GAN-based methods to domains other than images and video. In this work we explore a GAN-based method for audio processing, and develop a convolutional neural network architecture to perform audio super-resolution. In addition to several new architectural building blocks for audio processing, a key component of our approach is the use of an autoencoder-based loss that enables training in the GAN framework, with feature losses derived from unlabeled data. We explore the impact of our architectural choices, and demonstrate significant improvements over previous works in terms of both objective and perceptual quality.

PRO’s:
+well-written
+nice overall system: GAN framework for super-sampling audio incorporating features from an autoencoder
+some good-sounding examples
CON’s:
-some confusing/weakly-presented parts (admittedly covering lots of material in short space)
-I am confused about the evaluation; would like additional qualitative/observational understanding of what works, including more on how the results differ from baseline
SUMMARY: The task addressed in this work is: given a low-resolution audio signal, generate corresponding high-quality audio. The approach is a generative neural network that operates on raw audio and train within a GAN framework.
Working in raw sample-space (e.g. pixels) is known to be challenging, so a stabilizing solution is to incorporate a feature loss. Feature loss, however, usually requires a network trained on a related task, and if such a net one does not already exist, then building one can have its own (possibly significant) challenges. In this work, the authors avoid this auxiliary challenge by using unsupervised feature losses, taking advantage of the fact that any audio signal can be downsampled, and therefore one has the corresponding upsampled signal as well.
The training framework is basically that of a GAN, but where, rather than providing the generator with a low-dimensional noise signal input, they provide the generator with the subsampled audio signal. The architecture includes a generator ( G(lo-fidelity)=high-fidelity ), a discriminator ( D(high-fidelity) = real or by super-sampled ? ), and an autoencoder ( \phi( signal x) = features of signal x at AE’s bottleneck).
COMMENTS:
The generator network appears to be nearly identical to that of Kuleshov et al (2017)-- which becomes the baseline-- and so the primary contribution differentiating this work is the insertion of that network into a GAN framework along with the additional feature-based loss term. This is overall a nice problem and a nice approach! In that light, I believe that there is a new focus in this work on the perceptual quality of the outputs, as compared to (Kuleshov et al 2017). I would therefore ideally like to see (a) some attempts at perceptually evaluating the resulting output (beyond PESQ, e.g. with human subjects and with the understanding that, e.g. not all AMT workers have the same aural discriminative abilities themselves), and/or (b) more detailed associated qualitative descriptions/visualization of the super-sampled signal, perhaps with a few more samples if that would help. That said, I understand that there are page/space limitations. (more on this next)
Given the similarity of the U-net architectures to (Kuleshov et al 2017), why not move some of those descriptions to the appendix?
For example, I found the description and figure illustrating the “superpixel layers” to be fairly uninformative: I see that the figure shows interleaving and de-interleaving, resulting in trading-off dimensionalities/ranks/etc, and we are told that this helps with well-known checkerboard artifacts, but I was confused about what the white elements represent, and the caption just reiterated that resolution was being increased and decreased. Overall, I didn’t really understand exactly the role that this plays in the system; I wondered if it either needed a lot more clarification (in an appendix?), or just less space spent on it, but keeping the pointers to the relevant references.  It seems that the subpixel layer was already implemented in Kuleshov 2017, with some explanation, yet in the present work a large table (Table 1(b)) is presented showing that there is no difference in quality metrics, and the text also mentions that there is no significant perceptual difference in audio. If the subpixel layer were explained in detail, and with justification, then I would potentially be OK with the negative results, but in this case it’s not clear why spend this time on it here. It’s possible that there is something simple about it that I am not understanding. I’m open to being convinced. Otherwise, why not just write: “Following (Kuleshov et al 2017), we use subpixel layers (Shi et al) [instead of ...] to speed up training, although we found that they make no significant perceptual effects.” or something along those lines, and leave it at that?
I did appreciate the descriptions of models’ sensitivity to size/structure of the conv filters, importance of the res connections, etc.
My biggest confusion was with the evaluation & results:
Since the most directly related work was (Kuleshov 2017), I compared the super resolution (U-net) samples on that website (https://kuleshov.github.io/audio-super-res/ ) to the samples provided for the present work ( https://sites.google.com/view/unsupervised-audiosr/home ) and I was a bit confused, because the quality of the U-net samples in (Kuleshov 2017) seemed to be perceptually significantly better than the quality of the Deep CNN (U-net) baseline in the present work. Perhaps I am in error about this, but as far as I can tell, the superresolution in (Kuleshov et al 2017) is significantly better than the Deep CNN examples here. Is this a result of careful selection of examples? I do believe what I hear, e.g. that the MU-GAN8 is clearly better on some examples than the U-net8. But then for non-identical samples, how come U-net4 actually generally sounds better than U-net8? That doesn’t make immediate sense either (assuming no overfitting etc). Is the benefit in moving from U-net4 to U-net8 within a GAN context but then stabilizing  it with the feature-based loss? If so, then how does MU-GAN8 compare to U-net4? Would there be any info for the reader by doing an ablation removing the feature loss from the GAN framework? etc. I guess I would like to get a better understanding of what is actually going on, even if qualitative. Is there any qualitative or anecdotal observation about which “types” of samples one system works better on than another? For example, in the provided examples for the present paper, it seemed to be the case that perhaps the MU-GAN8 was more helpful for supersampling female voices, which might have more high-frequency components that seem to get lost when downsampling, but maybe I’m overgeneralizing from the few examples I heard.
Some spectrograms might be helpful, since they do after all convey some useful information despite not telling much of the perceptual story. For example, are there visible but inaudible artifacts? Are such artifacts systematic?
Were individual audio samples represented as a one-hot encoding, or as floats? (I assume floats since there was no mention of sampling from a distribution to select the value).
A couple of typos:
descriminator → discriminator
pg 6 “Impact of superpixel layers” -- last sentence of 2nd par is actually not a sentence. “the reduction in convolutional kernels prior to the superpixel operation.”
Overall, interesting work, and I enjoyed reading it. If some of my questions around evaluation could be addressed-- either in a revision, or in a rebuttal (e.g. if I completely misunderstood something)-- I would gladly consider revising my rating (which is currently somewhere between 6 and 7).


SUPERVISED POLICY UPDATE FOR DEEP REINFORCE- MENT LEARNING
We propose a new sample-efficient methodology, called Supervised Policy Update (SPU), for deep reinforcement learning. Starting with data generated by the current policy, SPU formulates and solves a constrained optimization problem in the non-parameterized proximal policy space. Using supervised regression, it then converts the optimal non-parameterized policy to a parameterized policy, from which it draws new samples. The methodology is general in that it applies to both discrete and continuous action spaces, and can handle a wide variety of proximity constraints for the non-parameterized optimization problem. We show how the Natural Policy Gradient and Trust Region Policy Optimization (NPG/TRPO) problems, and the Proximal Policy Optimization (PPO) problem can be addressed by this methodology. The SPU implementation is much simpler than TRPO. In terms of sample efficiency, our extensive experiments show SPU outperforms TRPO in Mujoco simulated robotic tasks and outperforms PPO in Atari video game tasks.

The authors formulate policy optimization as a two step iterative procedure: 1) solving a constrained optimization problem in the non-parameterized policy space, 2) using supervised regression to project this onto a parameterized policy. This approach generally applies to both continuous and discrete action spaces and can handle a variety of constraints. Their primary claims is that this approach improves sample-efficiency over TRPO on Mujoco tasks and over PPO on Atari games.
The method proposed in the paper has strong similarities with existing methods, but lacks comparisons with these approaches. The authors have not clearly demonstrated that SPU provides novel insights beyond the existing literature. I'm happy to change my score if the authors can convince me otherwise.
Main comments:
The focus of the paper is sample-efficiency, but the intro restricts to the on-policy setting. The authors should justify this choice. It is well known that off-policy algorithms (e.g., SAC for continuous control and Implicit Quantile Networks for Atari) are much more sample-efficient.
In Sec 4, what is the advantage of breaking the problem up into these 3 steps versus directly trying to solve (9),(10)? In fact, if we convert (10) into a penalty and take the derivative, we arrive at nearly the same gradient as (17). As this is central to the SPU framework, this needs to be justified.
MPO (Abdolmaleki et al. 2018) is very closely related to SPU. It is unclear if SPU provides any additional insights or benefits over MPO. This needs to be discussed and compared.
The experimental section could be strengthened by:
* Given the similarity to SPU, comparisons to MPO and GAC should be made, or clear justification for why they are not comparable must be given.
* Why is the comparison on Mujoco to TRPO in the main text and the comparison to PPO relegated to the appendix? It would make more sense to compare to PPO, so the authors need to justify this decision.
* The results on Mujoco are quite poor compared to state-of-the-art methods (e.g., SAC). The authors should justify why we should care about their results.
Comments:
In Sec 2, the authors should be careful about the discounting. For example, they are almost surely not having A_{it} approximate \gamma^t A^{\pi_{\theta_k}}, rather A^{\pi_{\theta_k}}.
In Sec 2, the KL is denoted as KL(\pi || pi_k), but in the text is described as the KL from \pi_k to \pi (reversed). From the equations, it appears that is an error, and it should read KL from \pi to \pi_k.
In Sec 3, the description of NPG/TRPO is not accurate. The main goal of NPG/TRPO work was to establish monotonic improvement.
In Sec 3, computational speed is cited as a major deficit of GAC, especially the solving linear systems with the Hessian (wrt to the actions). This seems rather surprising. Inverting a 1000x1000 matrix on a modern computer takes <1 second, so it doesn't seem like this should be the limiting step for any of the problems encountered.
The KL penalty version of PPO seems closely related to SPU. Can the authors mention differences with that version of PPO in the related work?
In Sec 4, step iii is described as supervised learning. Can the authors elaborate on why? I would typically think of the other direction as supervised learning as that leads to MLE.
In Sec 5.1, what is the justification/reasoning for setting \tilde{\lambda_{s_i}} = \lambda and introducing the indicator functions?
Sec 5.2 is not evaluated and Sec 5.3 produces inferior results, so it may make sense to move these to the appendix. Otherwise, the authors should explain situations where we would expect these to be useful or provide some additional insight. It also should be noted that the proximity constraints in TRPO/PPO follow from a theoretical argument and are not arbitrary choices.
Sec 5.3 seems to deviate quite a bit from the SPU framework. In addition to the differences pointed out in the text, the "supervised" loss changes too. Can the authors justify/explain the reasoning for these changes?
=====
I appreciate the authors' efforts to improve the paper. However, there is still substantial room for improvement in writing clarity. For example, the authors optimize the reverse KL from typical supervised learning, which makes even the title of the method confusing. The method that was experimentally evaluated can be derived more simply without the two-step procedure by directly taking the gradient and add the heuristically motivated per state indicator. This in itself is interesting and the authors demonstrate that it works well experimentally. I think the paper would be substantially more useful to the community if the authors focused on that contribution alone. As it stands now, I find the paper difficult to read because most of the theoretical results have no bearing on the method.


We introduce "Talk The Walk", the first large-scale dialogue dataset grounded in action and perception. The task involves two agents (a "guide" and a "tourist") that communicate via natural language in order to achieve a common goal: having the tourist navigate to a given target location. The task and dataset, which are described in detail, are challenging and their full solution is an open problem that we pose to the community. We (i) focus on the task of tourist localization and develop the novel Masked Attention for Spatial Convolutions (MASC) mechanism that allows for grounding tourist utterances into the guide's map, (ii) show it yields significant improvements for both emergent and natural language communication, and (iii) using this method, we establish non-trivial baselines on the full task.

The paper introduces a new dataset "Talk the Walk" that are dialogs
between a guide and a tourist, where the guide is to help the tourist
navigate to a target location.  The guide has access to a map and the
target location, but he relies on the tourist to communicate her
state (location) by natural language.
Pros.:
The task represented in the dataset can be highly challenging, and
respectable effort went into creating the dataset based on real city
neighborhoods.   The description and analysis of the dataset are
detailed.  The paper is well written up to the end of page 3.
The dataset by itself is a good contribution to the scientific
community when it is shared.  There could be many topics open
for studying within the same data.
The list of references and related work is exceptionally thorough
and useful for researchers interested in the topic.
Cons.:
The description of the experiments done with the dataset, however,
suffers from being overly cryptic.  The methods are not sufficiently
motivated, very few alternatives are presented and argued against,
and the several sections give a dry report of the sequences of things
the authors did.  It is not clear how others may find value in the
results and conclusions.
While the paper opens with the emphasis of a real-world setting,
after a series of simplifications (e.g. landmark typing, perfect perception)
it seems that much of the full complexity of the natural task is taken out,
and the main goal of the study is no longer clear.
For example, since there are only 9 types of landmarks,
in a small neighborhood there are not many combinations to draw reference to.
Simple observation sequences of such can easily narrow down the location
uncertainty.  It is important to highlight what the remaining open
issues are that make the task still challenging.
Misc.:
Examples are missing in the discussion of the experimental tasks.
e.g. in the study of emergent language, what could be a message that a
tourist may generate to describe his location?  what makes it hard for
the guide to decode it?  Likewise, what could be an example state of
the tourist and the description of that state in natural language?
Without the examples, it is difficult for the reader to have a sense
of the challenges in each task.
Table 8 is the first place where (finally) some utterances are presented.
However the description in the table or in the text is not sufficient
to convey the point that is supposed to be explained by the example.
The descriptions of the landmarks are restricted to the type of business
at the location with 9 possibilities.  Is the list of 9 exhaustive?
Are there any exceptions  (e.g. schools)?  How are such exceptions
represented and treated in the dialogs?
To what extent the difficulty of the tasks depends on the variability
of the combination of landmarks visible at each location?
What could be a simplest way to do this without neural-modeling?
e.g. with the many limitations that are built into the task and its
representations, will a simple decision tree based instruction method suffice?
Or a traditional algorithm that relies on repeated exploration and evaluation?
It is surprising that such possibilities are not even mentioned.
A complex neural architecture does not seem to be well justified unless
it is motivated by the need to overcome limitations of a classical method.
Irrelevant to the research effort, a thought about the dataset is that,
in these days with popular uses of GPS, the reliance on such dialogs for
navigation feels a bit backwards.


LEARNING PARTICLE DYNAMICS FOR MANIPULATING RIGID BODIES, DEFORMABLE OBJECTS, AND FLUIDS
Real-life control tasks involve matters of various substances-rigid or soft bodies, liquid, gas-each with distinct physical behaviors. This poses challenges to traditional rigid-body physics engines. Particle-based simulators have been developed to model the dynamics of these complex scenes; however, relying on approximation techniques, their simulation often deviates from real-world physics, especially in the long term. In this paper, we propose to learn a particle-based simulator for complex control tasks. Combining learning with particle-based systems brings in two major benefits: first, the learned simulator, just like other particle-based systems, acts widely on objects of different materials; second, the particle-based representation poses strong inductive bias for learning: particles of the same type have the same dynamics within. This enables the model to quickly adapt to new environments of unknown dynamics within a few observations. We demonstrate robots achieving complex manipulation tasks using the learned simulator, such as manipulating fluids and deformable foam, with experiments both in simulation and in the real world. Our study helps lay the foundation for robot learning of dynamic scenes with particle-based representations.

The authors present an algorithm for learning the dynamics prediction of deformable and fluid bodies by modeling them as (potentially hierarchical) systems of many interacting particles.  This model applies a shared encoder to the particle states (positions and velocities), a shared relation network to nearby pairs, and a shared propagator network to the summed relation network outputs.  In some cases this process is applied in a multi-scale hierarchical fashion.  The authors demonstrate accurate rollouts of system dynamics and usefulness for manipulative control of deformable objects.
I find the motivation in the introduction persuasive and the algorithmic approach sound.  I also like the application to RL.  However, I do have some concerns, as follows:
1)  The novelty of the method is questionable.  Specifically, the hierarchical interaction network proposed here seems extremely similar to the prior (and cited) paper (Mrowca et al., 2018), which the authors do not directly compare against.  If there is a non-negligible difference between the two algorithms, then the authors should explicitly discuss the difference and empirically compare the two, in order to benefit others in the community who otherwise would not know which to use.
2)  The paper would benefit a lot from a diagram of the model.  Specifically, it would be good to have a diagram of the hierarchical interaction network demonstrating the multiscale propagation.  This could go in Figure 1, perhaps replacing elements (b) and (d) of the current Figure 1, which in my opinion are unnecessary and can be removed.
3)  The paper uses domain-specific hyperparameters, yet does not discuss or analyze the effects of these hyperparameters much.  Specifically, for this method to be useful to others, we would like to know how to choose (i) the propagation step L, (ii) the number of roots, and (iii) the neighborhood distance d.  In the paper, these numbers are chosen differently for the different environments without explanation.  Graphs showing performance on each task over a range of values of these parameters would be good (perhaps in the supplementary material).  Also, using the same hyperparameters for all environments (or at least a common generating function) would help support the generality of this model.
4)  The treatment of rigid bodies seems a bit hand-held.  Specifically, to determine the dynamics of rigid bodies, there is a ground-truth calculation which calculates computes the velocity and angular velocity of the body from the model predictions for its constituent particles.  Furthermore, if I understand correctly, there is a different motion predictor network for those particles in a rigid body than those in the surrounding fluid --- is this correct?  If so, this raises the questions:  (i) What happens if the same motion-predictor network is used for all particles, and (ii) What happens if the ground-truth rigid dynamics calculation is not done, so the model has to do all the work?  It would be interesting to have these as baselines.
5)  It would be nice to see more generalization results.  There is only one generalization experiment, testing for generalization over particle number in FluidShake.  However, the FluidShake model is not hierarchical.  The hierarchical models are a big emphasis in the paper, so showing generalization on BoxBath or RiceGrip would be much more meaningful.
6)  No confidence intervals for the quantitative results.  Confidence intervals would be good to see in the table in Figure 3-a.  Also, the bar graph in Figure 5 really would benefit from errorbars --- it is difficult to determine if the results are significant.
7)  While the text is generally clear and definitely understandable, I have a couple of comments about it:  (i) The last three paragraphs of the introduction are repetitive and I think they can be removed, or at least shortened a lot.  There are also quite a number of grammatical errors throughout the paper, though it is still comprehensible.
EDIT:  In their revision the authors addressed these concerns well and the paper is much more convincing (see longer comment below).  In light of this I have changed my rating from a 5 to an 8.


A2BCD: Asynchronous Acceleration with Optimal Complexity
In this paper, we propose the Asynchronous Accelerated Nonuniform Randomized Block Coordinate Descent algorithm (A2BCD). We prove A2BCD converges linearly to a solution of the convex minimization problem at the same rate as NU_ACDM, so long as the maximum delay is not too large. This is the first asynchronous Nesterov-accelerated algorithm that attains any provable speedup. Moreover, we then prove that these algorithms both have optimal complexity. Asynchronous algorithms complete much faster iterations, and A2BCD has optimal complexity. Hence we observe in experiments that A2BCD is the top-performing coordinate descent algorithm, converging up to 4 − 5× faster than NU_ACDM on some data sets in terms of wall-clock time. To motivate our theory and proof techniques, we also derive and analyze a continuous-time analogue of our algorithm and prove it converges at the same rate.

The authors design an accelerated, asynchronous block coordinate descent algorithm, which, for sufficiently small delays attains the iteration complexity of the current state of the art algorithm (which is not parallel/asynchronous). The authors prove a lower bound on the iteration complexity in order to show that their algorithm is near optimal. They also analyze an ODE which is the continuous time limit of A2BCD, which they use to motivate their approach.
I am a little bit confused about the guarantee of the algorithm, as it does not agree with my intuition. Perhaps I am simply mistaken in my intuition, but I am concerned that there may need to be additional premises to the Theorem.
My main confusion is with Theorem 1, which says that for $\psi < 3/7$ the iteration complexity is approximately the iteration complexity of NU_ACDM times a factor of $(1 + o(1))/(1-\psi)$, i.e. within that factor of the optimal *non-asynchronous/parallel* algorithm. In particular, since $\psi < 3/7$ this means that the algorithm is within a $7/4 + o(1)$ factor. As mentioned in Corollary 3, this applies for instance when $L_i = L$ for all i and $\tau = \Theta( n^{1/2}\kappa^{-1/4} )$. Therefore, in a regime where $n \approx \kappa$, and $n$ very large, this would indicate that the algorithm would be almost as good as the best synchronous algorithm even for delays $\tau \approx n^{1/4}$. Perhaps I am missing something, but this seems very surprising to me, in particular, I would expect more significant slowdown due to $\tau$.
I am also a little bit surprised that the maximum tolerable delay is proportional to the *minimum* smoothness parameter $\underbar{L}$. It seems like decreasing $\underbar{L}$ should make optimization easier and therefore more delay should be tolerated. Perhaps this is simply an artifact of the analysis.


Recurrent neural networks can be difficult to train on long sequence data due to the well-known vanishing gradient problem. Some architectures incorporate methods to reduce RNN state updates, thereby allowing the network to preserve memory over long temporal intervals. We propose a timing-gated LSTM RNN model, called the Gaussian-gated LSTM (g-LSTM) for reducing state updates. The time gate controls when a neuron can be updated during training, enabling longer memory persistence and better error-gradient flow. This model captures long temporal dependencies better than an LSTM on very long sequence tasks and the time gate parameters can be learned even from a non-optimal initialization. Because the time gate limits the updates of the neuron state, the number of computes needed for the network update is also reduced. By adding a computational budget term to the training loss, we obtain a network which further reduces the number of computes by at least 10×. Finally, we propose a temporal curriculum learning schedule for the g-LSTM that helps speed up the convergence time of the equivalent LSTM on long sequences.

In the vein of recent work on learning “ticking” behaviour for LSTMs such as Phased LSTM, this paper proposes to add additional data independent gates to LSTM units that are defined as Gaussian functions of time indices.
The performance of the modified g-LSTM is compared to LSTM on the Addition, sequential MNIST and sequential CIFAR-10 tasks. The authors argue that g-LSTM results in better performance and has faster convergence on these tasks.
Additionally, it is proposed that one can reduce the amount of computations performed by the network by adding a computation budget term to the optimized loss that encouraged the cells to update less often. Finally, a technique for gradually transitioning from a g-LSTM to an LSTM during training is proposed, with the objective of speeding up training over a regular LSTM.
The paper is well written and easy to understand in general. However, the main results of this paper are experimental, and I am not entirely convinced by the experiments that g-LSTM is an improvement over the LSTM baseline for certain scenarios.
One broad reason for my doubts is that the comparisons don’t seem to utilise proper hyperparameter tuning for the baseline LSTM. Network sizes, learning rates, decay schedules, initialisations etc. all appear to be fixed, so one can not be sure of the “real” performance or convergence behavior of the models. Biased gate initializations are not used, though they have been used successfully in past work to aid in long term memory.
I should note that for long term memory problems such as those proposed by Hochreiter and Schmidhuber (1997), the proposed LSTM did not use a forget gate (or even BPTT) and used biased gate initialisations. However, these features are useful for more realistic tasks, and popular LSTM designs are biased towards them instead of toy problems.
I would consider the addition problem and sequential MNIST and CIFAR-10 to be interesting and difficult toy tasks for initial validation of ideas (and more extensive hyperparameter searches). It is unclear if the proposed techniques will perform provide  improvements over a well-tuned baseline for some realistic tasks, or are they suitable only for toy problems.