HOW IMPORTANT IS A NEURON?
The problem of attributing a deep network's prediction to its input/base features is well-studied (cf. Simonyan et al. (2013)). We introduce the notion of conductance to extend the notion of attribution to understanding the importance of hidden units. Informally, the conductance of a hidden unit of a deep network is the flow of attribution via this hidden unit. We can use conductance to understand the importance of a hidden unit to the prediction for a specific input, or over a set of inputs. We justify conductance in multiple ways via a qualitative comparison with other methods, via some axiomatic results, and via an empirical evaluation based on a feature selection task. The empirical evaluations are done using the Inception network over ImageNet data, and a convolutional network over text data. In both cases, we demonstrate the effectiveness of conductance in identifying interesting insights about the internal workings of these networks.

This paper presents a new method to measure the importance of hidden neurons in deep neural networks. The method integrates notions of activation value, input influence to a neuron and neuron influence to the network's output. They provide results confirming that this measure is able to identify neurons that are important for specific tasks.
Quality
The experiments are well designed to verify their hypothesis, although there could be more to make sure those results are not particular to the few selected problems. Nevertheless, the results are consistent across those experiments.
Clarity
The text is well written in general, but the structure could be improved. The introduction contains too much related work, which should be divided in another section. Section 2 contains mostly high level explanations of the work, which should be integrated in the introduction, and thus before the related work section, to improve readability. See minor comments for more specific suggestions.
It is difficult to understand the goal of Section 4.2. Section 2 states that section 4.2 proves that a "path method" must be used in order to satisfy the axioms, but why such axioms are important is not stressed enough. Also, it is not clear why those are called axioms since they are not use to build anything else. It seems to me that those are rather "desirable properties" than axioms.
Originality
A important number of related works are cited and compared with the current work. Although the proposed measure is close to what is proposed by Datta et al., this paper makes the distinction clear and benchmarks its results properly against it.
Significance
There is an increasing need to interpretability of deep neural networks as they get more and more applied to real-world problems. Measures as the one proposed in this paper are a very important building block towards this.
Conclusion
For its original importance measure and the proper experiment benchmarks, I believe this paper should be accepted. There is however many minor issues that should be fixed for the camera-ready version. Although the recommended length is 8 pages, the strict limit is 10, so I would recommended to use a bit of the remaining extra space to conclude the paper properly with a discussion on the results and their consequences, as well as a conclusion to wrap up the paper.
***
Minor Comments
Introduction:
- The term flow is never defined precisely, we need to infer it based on the definition of conductance and attribution.
- First paragraph would be more clear with simple word explanation rather than maths. Also, second sentence is not a complete sentence
- Work on image indicators of importance could be compared better with current work. Indicators can be seen as a measure of importance.
- This sentence is not clear: "[...]; the nature of correlations in the two models may differ".
Section 2:
- Last paragraph of section 2 can be true for any well-performing importance measure. The statements should be put in perspective with others.
Section 3:
- Section 3 should be introduced by explaining the goal of the section otherwise it breaks the flow of reading.
- The role of the baseline x' should be better explained when it is presented (first paragraph of section 3).
- The interchangeable use of the term "conductance of neuron j" for equations 2 and 3 is confusing. Different terms should be use, even if the context makes it possible to infer which one is being referred to.
- Remark 1 seems trivial, but the selection of baseline x' seems less trivial. Some explanations should be devoted to it.
- Second paragraph of remark 1 is not clear. Why couldn't we take another layer's neuron as the neuron of interest, bounding the conductance measure on one layer as the input and the output of the model? If we make the input to be a neuron y rather than the true input x, we could take another neuron z in a subsequent layer to be the neuron of interest, resulting in conductance measure Cond^z_i(y).
Section 4:
- List of importance measure at beginning of Section 4 should probably have citations.
- Backward reference to section 3 seems to be a mistake, should it be subsection 4.2?
- Each of the justifications to get around the issue of distinguishing strange model behavior from bad feature importance technique should be explained briefly in paragraph before section 4.1.
Subsection 4.1:
- I do not understand the problem explained in fourth paragraph of section 4.1. g(f(1 - epsilon)) = 0, why would it be 1- epsilon?
- Problem explained in fifth Paragraph of section is not clear unless what the influence of the unit is clearly stated. Is it simply dg/df?
- A short explanation of what is tested in section 6 should be given at last paragraph of section 4.1. Although the results are favorable to the conductance metric, it is not clear how they precisely confirm the problem of incorrect signs presented in the caricature examples.
Subsection 4.2:
- As said in the my main comments, I am not convinced by the use of the term Axiom. They are not use as building blocks, and are rather used as desirable properties for which the authors prove that only "path methods" can satisfy.
- Footnote 2 on page 5 it difficult to read.
- Although the proof does not seem to use the axioms as a building block, which is fortunate since it would make it a circular argument otherwise, the text suggests so: "Given these three axioms, we can show that:".
- The importance of section 4.2 should be clarified. More emphasis on the importance of the axioms (desirable properties) should be made.
Section 5:
- Choices for experiments should be explained. Why choosing layers mixed** rather than others? Why choosing filters?
- Figures 1-4 are difficult to interpreted on a printed version. Since this is qualitative, I suggest to change the saturation of the images to make them easier to interpret. The absolute values are not important for a qualitative interpretation
- Figure 4 could be more interesting if compared with other classes, like other animal faces. Anyhow, I understand that those were chosen based on the subset of classes used for the experiments.
- Space should be added between figures to better divide the captions
Section 6:
- The difference between experiments of Figure 5 and 6 should be made more clear.
Section 7-8:
- Where are they? No discussion? No conclusion?


Adaptive regularization methods pre-multiply a descent direction by a preconditioning matrix. Due to the large number of parameters of machine learning problems, full-matrix preconditioning methods are prohibitively expensive. We show how to modify full-matrix adaptive regularization in order to make it practical and effective. We also provide novel theoretical analysis for adaptive regularization in non-convex optimization settings. The core of our algorithm, termed GGT, consists of efficient inverse computation of square roots of low-rank matrices. Our preliminary experiments underscore improved convergence rate of GGT across a variety of synthetic tasks and standard deep learning benchmarks.

The paper considers adaptive regularization, which has been popular in neural network learning.  Rather than adapting diagonal elements of the adaptivity matrix, the paper proposes to consider a low-rank approximation to the Gram/correlation matrix.
When you say that full-matrix computation "requires taking the inverse square root", I assume you know that is not really correct?  As a matter of good implementation, one never takes the inverse of anything.  Instead, on solves a linear system, via other means.  Of course, approximate linear system solvers then permit a wide tradeoff space to speed things up.
There are several issues convolved here: one is ``full-matrix,'' another is that this is really a low-rank approximation to a matrix and so not full matrix, another is that this may or may not be implementable on GPUs.  The latter may be important in practice, but it is orthogonal to the full matrix theory.
There is a great deal of discussion about full-matrix preconditioning, but there is no full matrix here.  Instead, it is a low-rank approximation to the full matrix.  If there were theory to be had here, then I would guess that the low-rank approximation may work even when full matrix did not, e.g., since the full matrix case would involve too may parameters.
The discussion of convergence to first order critical points is straightforward.
Adaptivity ratio is mentioned in the intro but not defined there.  Why mention it here, if it's not being defined.
You say that second order methods are outside the scope, but you say that your method is particularly relevant for ill-conditioned problems.  It would help to clarify the connection between the Gram/correlation matrix of gradients and the Hessian and what is being done to ill-conditioning, since second order methods are basically designed for ill-conditioned problems..
It is difficult to know what the theory says about the empirical results, given the tweaks discussed in Sec 2.2, and so it is difficult to know what is the benefit of the method versus the tweaks.
The results shown in Figure 4 are much more interesting than the usual training curves which are shown in the other figures.  If this method is to be useful, understanding how these spectral properties change during training for different types of networks is essential.  More papers should present this, and those that do should do it more systematically.
You say that you "informally state the main theorem."  The level of formality/informality makes it hard to know what is really being said.  You should remove it if it is not worth stating precisely, or state it precisely.  (It's fair to modularize the proof, but as it is it's hard to know what it's saying, except that your method comes with some guarantee that isn't stated.)


We first show that the commonly used dimensionality reduction (DR) methods such as t-SNE and LargeVis poorly capture the global structure of the data in the low dimensional embedding. We show this via a number of tests for the DR methods that can be easily applied by any practitioner to the dataset at hand. Surprisingly enough, t-SNE performs the best w.r.t. the commonly used measures that reward the local neighborhood accuracy such as precision-recall while having the worst performance in our tests for global structure. We then contrast the performance of these two DR method against our new method called TriMap. The main idea behind TriMap is to capture higher orders of structure with triplet information (instead of pairwise information used by t-SNE and LargeVis), and to minimize a robust loss function for satisfying the chosen triplets. We provide compelling experimental evidence on large natural datasets for the clear advantage of the TriMap DR results. As LargeVis, TriMap is fast and provides comparable runtime on large datasets.

In this paper, the authors present a novel dimensionality reduction method named TriMap. TriMap attempts to improve upon the widely-adopted t-SNE algorithm by incorporating global distances through the use of triplets, rather than pairwise comparisons. The authors compare to t-SNE, as well as a newer method called LargeVis which also claims to impose a global distances metric. The authors show that their method is more robust to the addition or removal of clusters and outliers and provides a more meaningful global distance relative to the methods against which they compare.
Technical Quality
The authors’ method is clear and well described and addresses a poignant issue in dimensionality reduction. However, the authors fail to compare their method to a number of relevant dimensionality reduction algorithms which also claim to provide solutions with globally meaningful distances. Such methods include force-directed graph drawing (Fruchterman & Reingold, 1991), diffusion maps (Coifman & Lafon, 2006) and PHATE (Moon et al., 2017).
Additionally, the handling of outliers is a concern. While the authors claim that the retention of outliers as disconnected from the manifold is a desirable quality of their technique, the presence of many outliers in a dataset (for example, in the Tabula Muris and lyrics datasets) has the potential to mask the interesting portion of the dimensionality reduction. It may be worth commenting on the desirability to identify and remove outliers, and the provision of such a technique in the software upon its release.
Finally, the runtime comparison is of concern. It is common to perform most DR methods on high-dimensional PCA representation of the data, particularly in single-cell genomics (e.g. the Tabula Muris dataset in Part 3.) In this context, both UMAP and PHATE successfully embed the Tabula Muris dataset in less than the reported TriMap time (3.5 and 5 minutes respectively, compared to 15 minutes reported for TriMap.)
Novelty
The authors’ method appears to provide improved results over the compared alternatives, however, it is worth noting that triplet-based embedding is not novel in its own right (van der Maaten & Weinberger, 2012), though one could argue novelty is warranted here due to claimed substantial improvements of results. In this case, the authors should include a comparison to competing triplet-based methods, at least in the appendix.
Potential impact
The authors’ method has the potential to be used widely across many fields, as a direct replacement for t-SNE. Its adoption is contingent on compelling evidence that it produces results substantially better than UMAP (which is currently heralded as an upcoming replacement for t-SNE in some fields) and other competing methods. The authors may find it worthwhile to provide such comparisons, if not in the main body of the paper at least in the appendix.
Clarity
The paper is easy to read and makes its point in a reasonably concise manner. Detailed explanation of experiments v) and vi) could be relegated to the appendix. More precise statement of the authors’ tests in Part 2 could be provided by quantifying the results of the tests in a more precise way; it is not clear what the authors seek to achieve by drawing the dotted lines between clusters in Figure 1a, or by providing AUC values in Figure 1.
Detailed Comments
•	In the definition of Equation 2, it is not until one paragraph later than q_{ij}^{~(t’)} is defined – this is confusing and hard to read.
•	The captions for Figures 1 and 3 would be substantially clearer with more detail on the dataset analyzed and in Figure 1, some discussion of the purpose of each subplot.
•	The Figure 3 caption needs a semicolon or period before introducing the bottom panel.
•	The claim that the authors’ heuristic triplet sampling (nearest-neighbor and random sampling) is sufficient to approximate full triplet sampling should be shown in the appendix.
•	The collaboration network analyzed in Part 3 is naturally a graph; it would make sense to cluster and visualize this using a graph-based clustering, rather than coercing it to Euclidean coordinates.
(Note: after reading the revised manuscript I have changed my recommendation from a 6 to a 5)
References
Coifman, R. R., & Lafon, S. (2006). Diffusion maps. Applied and computational harmonic analysis, 21(1), 5-30. https://doi.org/10.1016/j.acha.2006.04.006
Moon, K. R., van Dijk, D., Wang, Z., Burkhardt, D., Chen, W., van den Elzen, A., ... & Krishnaswamy, S. (2017). Visualizing transitions and structure for high dimensional data exploration. bioRxiv, 120378. https://doi.org/10.1101/120378
Fruchterman, T. M., & Reingold, E. M. (1991). Graph drawing by force‐directed placement. Software: Practice and experience, 21(11), 1129-1164. https://doi.org/10.1002/spe.4380211102
L. van der Maaten and K. Weinberger. Stochastic triplet embedding. In 2012 IEEE International Workshop on Machine Learning for Signal Processing, pp. 1–6, Sept 2012. doi: 10.1109/MLSP.2012.6349720.


We propose a novel approach to regularizing generative adversarial networks (GANs) leveraging learned structured Gibbs distributions. Our method consists of reparameterizing the discriminator to be an explicit function of two densities: the generator PDF q and a structured Gibbs distribution ν. Leveraging recent work on invertible pushforward density estimators, this reparameterization is made possible by assuming the generator is invertible, which enables the analytic evaluation of the generator PDF q. We further propose optimizing the Jeffrey divergence, which balances mode coverage with sample quality. The combination of this loss and reparameterization allows us to effectively regularize the generator by imposing structure from domain knowledge on ν, as in classical graphical models. Applying our method to a vehicle trajectory forecasting task, we observe that we are able to obtain quantitatively superior mode coverage as well as better-quality samples compared to traditional methods.

This paper combines a number of ideas to train generative models with (deep) structured constraints. The general idea is similar to Flow-GAN, which learns a normalizing flow-based generator by optimizing the negative loglikelihood with an augmented GAN loss. However, It’s difficult to impose prior structure information in the GAN framework. To address this problem, the authors proposed to minimize a so-called Gibbs-regularized variational bound of Jeffery divergence, which is the summation of KL and reverse KL divergence. The authors provide some justification that the Jeffery divergence works by yielding good mass-covering and mode-seeing properties.
It appears that the parameterization and adaptation of v throughout optimization is the key contribution of this work --- the technical details are not clear from the paper.
1.    Typo in the training objective (Eq .1):  the second (or the first) "sup" should be removed?
2.    Section 2.3 is very confusing. Particularly, how is the parameter \phi introduced? What’s the detailed update of \phi?
- "We now observe that our methods can also be interpreted as a way of learning v as a Gibbs distribution approximating p." If v_\phi(x) is a distribution, what’s the parametric form of v?
- "Generally, this is achieved by structuring the energy function V_\phi:=\log v_\phi." It seems that V_\phi(x) is a scalar-valued function that represents the negative energy of the distribution v_\phi(x), however, why the distribution is self-normalized? Specifically, why \int \exp(V_\phi) dx = 1? Otherwise, how the authors deal with the partition function \int \exp(V_\phi(x)).
- It is unclear to me why the inner loop optimization is connected with Itakura-Saito divergence minimization? The authors may consider including the detailed proofs?
3.    With the given description, the proposed algorithm is not easy to follow and implement by the reader. The paper would benefit from an Algorithm box with pseudocode.
If the authors can fully address the concerns above, I will consider changing the scores.
Other comments:
1. The empirical results are fairly weak. Similar datasets are used, the authors may consider evaluating their approach on various different tasks.
2. Duplicate citations – R2P2 [35] [36]
3. Other related papers:
 - Belanger et al., End-to-End Learning for Structured Prediction Energy Networks, ICML 17
- Tu et al., Learning Approximate Inference Networks for Structured Prediction, ICLR 18


VISUAL SEMANTIC NAVIGATION USING SCENE PRIORS
Coffee machine
Figure 1: Our goal is to use scene priors to improve navigation in unseen scenes and towards novel objects. (a) There is no mug in the field of view of the agent, but the likely location for finding a mug is the cabinet near the coffee machine. (b) The agent has not seen a mango before, but it infers that the most likely location for finding a mango is the fridge since similar objects such as apple appear there as well. The most likely locations are shown with the orange box.
How do humans navigate to target objects in novel scenes? Do we use the semantic/functional priors we have built over years to efficiently search and navigate? For example, to search for mugs, we search cabinets near the coffee machine and for fruits we try the fridge. In this work, we focus on incorporating semantic priors in the task of semantic navigation. We propose to use Graph Convolutional Networks for incorporating the prior knowledge into a deep reinforcement learning framework. The agent uses the features from the knowledge graph to predict the actions. For evaluation, we use the AI2-THOR framework. Our experiments show how semantic knowledge improves performance significantly. More importantly, we show improvement in generalization to unseen scenes and/or objects.

This paper tackles the problem of navigating scenes to find objects which are potentially not included in the training phase. To find an unseen object from a scene, the proposed model incorporates an external knowledge graph as an augmented input of the actor-critic model. To construct a knowledge graph, entities in a scene are identified by ResNet and then the link structure between entities are extracted from VIsual Genome dataset. Through the ablation study, it is shown that using the knowledge graph helps to track and identify unseen objects during training.
- The original knowledge graph (KG) has relation labels (such as next to, on in figure 3) between different objects, however, GCN does not take into account the relations between objects. Only co-occurrence patterns will be encoded into the KG constructed from an image. There are more complex graph convolutional models modelling relations between nodes such as [1]. Have you considered adding explicit relations between entities? will it increase the navigation performance? if not why?
- It is unclear how many objects are used to construct a KG from an image. For example, are top-k objects identified by ResNet used to construct a KG?
- Description of the reward is a bit unclear as well, especially when the model is trained without stop action. From the text, the agents receive a positive reward when it is close to the target (within a certain number of steps). Does this mean that the agent gets a positive reward on every step near the target while it's not in the final state?
- This might be a trivial question, but I couldn't find it from the text. Can you find all object from AI2-THOR in the categories of ImageNet and of Visual Genome? is there any information loss while constructing a KG from the classification result? What is the average number of nodes of a KG? and is there any correlation between the size of KG and the result?
- Why are the performances of the models is unstable with Bedroom dataset (in terms of variance)?
- The input feature of GCN is a combination of word feature and image feature. It is clear that there is a corresponding word embedding for each of the identified objects, but it is unclear what is the corresponding image feature. If two objects are identified in the same frame, do input features of these two objects share the same image features from Resnet?
[1] Schlichtkrull, Michael, et al. "Modeling relational data with graph convolutional networks." European Semantic Web Conference. Springer, Cham, 2018.


On the relation between the sharpest direc- tions of DNN loss and the SGD step length
Stochastic Gradient Descent (SGD) based training of neural networks with a large learning rate or a small batch-size typically ends in well-generalizing, flat regions of the weight space, as indicated by small eigenvalues of the Hessian of the training loss. However, the curvature along the SGD trajectory is poorly understood. An empirical investigation shows that initially SGD visits increasingly sharp regions, reaching a maximum sharpness determined by both the learning rate and the batch-size of SGD. When studying the SGD dynamics in relation to the sharpest directions in this initial phase, we find that the SGD step is large compared to the curvature and commonly fails to minimize the loss along the sharpest directions. Furthermore, using a reduced learning rate along these directions can improve training speed while leading to both sharper and better generalizing solutions compared to vanilla SGD. In summary, our analysis of the dynamics of SGD in the subspace of the sharpest directions shows that they influence the regions that SGD steers to (where larger learning rate or smaller batch size result in wider regions visited), the overall training speed, and the generalization ability of the final model.

Update after author response: I am changing my rating from 4 to 6 in light of the clarification and new experiments.
-------
In this paper the authors study the relationship between the SGD step size and the curvature of the loss surface, empirically showing that: 1) SGD is guided towards sharp regions of the loss surface at the start especially with a large learning rate or a small batch size. 2) Loss increases on average when taking a SGD step in the sharpest directions. 3) Modifying the SGD step size in the sharp directions (for example removing its component in the sharpest direction), can lead to substantial changes in both the quality and the local landscape of the minima (for the example mentioned, leading to a better and sharper minima). Motivated by these observations, the authors propose a variant of SGD that leads to better performance on the datasets considered.
Deep learning theory is a very important frontier for machine learning and one that’s needed to make the practice be guided more by the foundational principles than incessant tweaks. The paper makes some very interesting observations and uses those insights to improve the widely used SGD. However, I have a few concerns which leave me unconvinced about the impact of the contributions in the paper. My biggest problem is the use of second order information in the algorithm which makes the optimization process computationally cumbersome, and raises the question as to why might this approach be preferable to any other second order approach (the authors touch on Newton method in the appendix but the discussion far from settles the matter). Similar questions arise in considering the merit of the proposed methods in comparison to a host of other well-studied augmentations to SGD like momentum, Adam or AdaGrad. The quality of presentation is also a problem, and both the organization of the main matter as well as of the figures can use some polishing. The latter specifically sometimes lacked legends (Fig. 3 and 4), and some other times had legends covering a quarter of the plot (Fig. 5). Lastly, even though the claims sound theoretical, they are not derived from any set of first principles but come from observations on a few datasets. While this may after all be how SGD behaves in general, currently the paper doesn’t provide any evidence to believe that.
Minor issues: “withe” (page 2, spelling), “\alpha = 0.5, 1, 2 corresponding to red, green, and blue” (page 4, I believe it should be “blue, green and red”).
In summary, even though I liked what the paper set out to do, I am not convinced on the generalizability of these results and subsequently the rationale for using the proposed method over other competing options. A revised version of the paper with either validation on more datasets or sound theory generalizing the results to some extent would make for a much nicer contribution.


We introduce the deep abstaining classifier -a deep neural network trained with a novel loss function that provides an abstention option during training. This allows the DNN to abstain on confusing or difficult-to-learn examples while improving performance on the non-abstained samples. We show that such deep abstaining classifiers can: (i) learn representations for structured noise -where noisy training labels or confusing examples are correlated with underlying features -and then learn to abstain based on such features; (ii) enable robust learning in the presence of arbitrary or unstructured noise by identifying noisy samples; and (iii) be used as an effective out-of-category detector that learns to reliably abstain when presented with samples from unknown classes. We provide analytical results on loss function behavior that enable automatic tuning of accuracy and coverage, and demonstrate the utility of the deep abstaining classifier using multiple image benchmarks, Results indicate significant improvement in learning in the presence of label noise.

This paper formulates a new deep method called deep abstaining classifer. Their main idea is to introduce a new modified loss function that utilizes an absention output allowing the DNN to learn when abstention is a better option. The core idea resemble KWIK framework [1], which has been theoretical justified.
Pros:
1. The authors find a new direction for learning with noisy labels. Based on Eq. (1) (the modified loss), the propose \alpha auto-tuning algorithm, which is relatively novel.
2. The authors perform numerical experiments to demonstrate the efficacy of their framework. And their experimental result support their previous claims.
For example, they conduct experiments on CIFAR-10 and CIFAR-100. Besides, they conduct experiments on open-world detection dataset.
Cons:
We have three questions in the following.
1. Clarity: in Section 3, the author claim real-world data is corrupted in some non-arbitrary manner. However, in practice, it is really hard to reason the corrpution procedure for agnostic noisy dataset like Clothing1M [2]. The authors are encouraged to explain this point more.
2. Related works: In deep learning with noisy labels, there are three main directions, including small-loss trick [3], estimating noise transition matrix [4,5], and explicit and implicit regularization [6]. I would appreciate if the authors can survey and compare more baselines in their paper.
3. Experiment:
3.1 Baselines: For noisy labels, the author should compare with [7] directly, which is highly related to your work. Namely, designing new loss function can overcome the issue of noisy labels. Without this comparison, the reported result has less impact. Moreover, the authors should add MentorNet [2] as a baseline https://github.com/google/mentornet
3.2 Datasets: For datasets, I think the author should first compare their methods on symmetric and aysmmetric noisy data. Besides, the authors are encouraged to conduct 1 NLP dataset.
References:
[1] L. Li, M. Littman, and T. Walsh. Knows what it knows: a framework for self-aware learning. In ICML, 2008.
[2] T. Xiao, T. Xia, Y. Yang, C. Huang, and X. Wang. Learning from massive noisy labeled data for image classification. In CVPR, 2015.
[3] L. Jiang, Z. Zhou, T. Leung, L. Li, and L. Fei-Fei. Mentornet: Learning data-driven curriculum for very deep neural networks on corrupted labels. In ICML, 2018.
[4] G. Patrini, A. Rozza, A. Menon, R. Nock, and L. Qu. Making deep neural networks robust to label noise: A loss correction approach. In CVPR, 2017.
[5] J. Goldberger and E. Ben-Reuven. Training deep neural-networks using a noise adaptation layer. In ICLR, 2017.
[6] T. Miyato, S. Maeda, M. Koyama, and S. Ishii. Virtual adversarial training: A regularization method for supervised and semi-supervised learning. ICLR, 2016.
[7] Z. Zhang and M. Sabuncu. Generalized cross entropy loss for training deep neural networks with noisy labels. In NIPS, 2018.


One approach to deal with the statistical inefficiency of neural networks is to rely on auxiliary losses that help to build useful representations. However, it is not always trivial to know if an auxiliary task will be helpful for the main task and when it could start hurting. We propose to use the cosine similarity between gradients of tasks as an adaptive weight to detect when an auxiliary loss is helpful to the main loss. We show that our approach is guaranteed to converge to critical points of the main task and demonstrate the practical usefulness of the proposed algorithm in a few domains: multi-task supervised learning on subsets of ImageNet, reinforcement learning on gridworld, and reinforcement learning on Atari games.

The paper is addressing the problem of a specific multi-task learning setup such that there are two tasks namely main task and auxiliary task. Auxiliary task is used for the sole purpose of helping the main one. In other words, auxiliary task performance is not of interest. The simple and sensible approach proposed in the paper is using cosine similarity between the gradients of two loss functions and incorporating the auxiliary one if it is positively aligned with the main gradient. Authors suggest to further scale loss functions using the cosine similarity but it only experiments with the simpler case of binary decision of using both gradients or only the main one. Authors provide a convergence guarantee (without any convergence rate) by simply extending the convergence of gradient method.
The paper is definitely addressing an important problem as the authors cite many previous work which uses the setup of set of auxiliary tasks helping a main one. The method is simple and easy to implement. Hence, it has a potential to be useful for the community.
One major issue for me is the experimental setup. The authors cite many interesting, realistic and practical setups (Zhang et al., 2016; Jaderberg et al., 2017; Mirowski et al., 2017; Papoudakis et al., 2018), but do not use any of these setups in their experiments. Instead, paper uses set of toy experiments. This is very puzzling to me as all these papers set existing baselines for interesting problems which authors can easily compare. I think the paper needs to be experimented and compared with these established methods.
Another major issue is the weak multi-task learning baseline used in the paper. There have been many interesting developments in adaptive scaling of multiple loss functions in the literature. However, paper does not compare with them. Example of these methods are: [GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networks, ICML 2018] and [Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semantics, CVPR 2018]. Although these methods addresses the case of all tasks being important, it is a valid baseline and need to be compared. Similar to my first points, these papers also use very realistic and interesting experiments which would fit better than the toy experiments in the paper.
Final major issue is the fact that experimental results are suggesting the method is not effective. In ImageNet experiment, auxiliary tasks actually hurt the final performance as the single task is better than all methods including the proposed one. Proposed method does not guarantee that auxiliary tasks will have no harm. The GridWorld experiment is sort of a sanity check to me as it is very hand-crafted. For Breakout experiment, single task actually outperforms all baselines and this means the proposed method results in a harm similar to ImageNet case. For Breakout+MSPacMan experiment, multi task and the proposed method performs almost exactly same. I do not get why the performance on Breakout is relevant for this case since it is not a main task. The paper clearly states that only performance of an interest is the main one which is MSPacMan in this case. Also, in this experiment clearly all methods are still learning as the curve did not plateau yet. I am curious, why the learning is stop there. I do not think we need the method to be effective to be published; but, the negative result should be explained properly.
MINOR NITPICKS
- Algorithm 1&2 are crucial to understand the paper, they should be in main text
- ImageNet class IDs change between years. So, actual wordnet IDs or class names is a better thing to state
- What happens if there are multiple auxiliary tasks?
- Does the theory still hold for loss functions which are not Lipschitz as the Cauchy's gradient method requires that for convergence
In summary, the paper is proposing a sensible method for an important problem. However, it is only tested for toy problems although there are interesting existing setups which would be ideal for the method to be tested. Moreover, it is only compared with the most-naive multi task learning baselines. Even this limited experimental setup does not confirm what the paper is claiming (using auxiliary tasks only when they help). And the paper fails to explain this failure cases. The method needs to be experimented with a more realistic setup with more realistic baselines.
------
After rebuttal:
I gave detailed responses to each part of the rebuttal below. Here is the summary:
Although the response addresses some of my concerns. There are still major issues with the experimental study. 1) there are existing, relevant and well-studied multi-task setups with negative interference. Method should be experimented with some of those setups. 2) Multi-task baseline in the paper is naive and far from state-of-the-art. Paper need strong baselines as discussed. Hence, I am keeping my score. Paper needs to be improved with a stronger experimental study and need to be re-submitted.


FROM HARD TO SOFT: UNDERSTANDING DEEP NETWORK NONLINEARITIES VIA VECTOR QUANTIZATION AND STATISTICAL INFERENCE
Nonlinearity is crucial to the performance of a deep (neural) network (DN). To date there has been little progress understanding the menagerie of available nonlinearities, but recently progress has been made on understanding the rôle played by piecewise affine and convex nonlinearities like the ReLU and absolute value activation functions and max-pooling. In particular, DN layers constructed from these operations can be interpreted as max-affine spline operators (MASOs) that have an elegant link to vector quantization (VQ) and K-means. While this is good theoretical progress, the entire MASO approach is predicated on the requirement that the nonlinearities be piecewise affine and convex, which precludes important activation functions like the sigmoid, hyperbolic tangent, and softmax. This paper extends the MASO framework to these and an infinitely large class of new nonlinearities by linking deterministic MASOs with probabilistic Gaussian Mixture Models (GMMs). We show that, under a GMM, piecewise affine, convex nonlinearities like ReLU, absolute value, and max-pooling can be interpreted as solutions to certain natural "hard" VQ inference problems, while sigmoid, hyperbolic tangent, and softmax can be interpreted as solutions to corresponding "soft" VQ inference problems. We further extend the framework by hybridizing the hard and soft VQ optimizations to create a β-VQ inference that interpolates between hard, soft, and linear VQ inference. A prime example of a β-VQ DN nonlinearity is the swish nonlinearity, which offers state-of-the-art performance in a range of computer vision tasks but was developed ad hoc by experimentation. Finally, we validate with experiments an important assertion of our theory, namely that DN performance can be significantly improved by enforcing orthogonality in its linear filters.

This work extends the applicability of the spline theory of deep networks explored in previous works of Balestriero/ Baraniuk. The previous works setup DNs as layer-wise max-affine spline operators (MASOs) and recovers several non-linearities practically used as special cases of these MASOs. The previous works already recover RELU variants and some downsampling operators that the current submission characterizes as "hard" quantization.
The major contribution of this work is extending the application to "soft" quantization that recovers several new non-linear activations such as soft-max. It is well-known that the k-means algorithm can be considered as a run of an EM algorithm to recover the mean parameters of a gaussian mixture model. The "hard" to "soft" transformation, and any interpolation in between follows from combining this insight with the previous works. As such there isnt a major technical contribution imho in this work. Furthermore, the presented orthogonalization for easier inference has been used before in many works, some of which this submission also cites, most importantly in the previous work of Balestriero/ Baraniuk that this submission extends.
Nevertheless there is value in novel results that may follow from previous works in a straightforward but non-trivial fashion, as long as it is well-presented and thoroughly researched and implication well-highlighted. This paper does that adequately, so I will suggest weak accept. Furthermore, this work could spark interesting future works and fruitful discussions at the ICLR. It is well-written and the experimental evaluation is adequate.
I would suggest a couple of ways to possibly improve the exposition. The paper is somewhat notation heavy. When considering single layers, the superscript for the layer could be dropped in favor of clarity. I would suggest moving the definition of MASOs to the main text, and present Proposition 8 in some form in the main text as well. To a reader not familiar with previous works, or with splines, this could be helpful. Use of orthogonalization could be highlighted not just a tool for tractability but also regularization. For inference on GMMs, it corresponds to a type of variational inference, which could be mentioned.


EXEMPLAR GUIDED UNSUPERVISED IMAGE-TO- IMAGE TRANSLATION WITH SEMANTIC CONSISTENCY
Image-to-image translation has recently received significant attention due to advances in deep learning. Most works focus on learning either a one-to-one mapping in an unsupervised way or a many-to-many mapping in a supervised way. However, a more practical setting is many-to-many mapping in an unsupervised way, which is harder due to the lack of supervision and the complex inner-and cross-domain variations. To alleviate these issues, we propose the Exemplar Guided & Semantically Consistent Image-to-image Translation (EGSC-IT) network which conditions the translation process on an exemplar image in the target domain. We assume that an image comprises of a content component which is shared across domains, and a style component specific to each domain. Under the guidance of an exemplar from the target domain we apply Adaptive Instance Normalization to the shared content component, which allows us to transfer the style information of the target domain to the source domain. To avoid semantic inconsistencies during translation that naturally appear due to the large inner-and cross-domain variations, we introduce the concept of feature masks that provide coarse semantic guidance without requiring the use of any semantic labels. Experimental results on various datasets show that EGSC-IT does not only translate the source image to diverse instances in the target domain, but also preserves the semantic consistency during the process.

The introduction is written to perfection. The paper discusses a core failing and need for I2I translation models. The one-to-one mapping assumption does not apply to most tasks. While the approach seems novel the analysis of the results are insufficient to convince me that the method is really working. This should be a workshop paper.
For the motivation of the approach I am not convinced how the conditioned style is being used. It would be nice to see some analysis of how the latent space changes given different input images. Why would style information be propagated through the network? Why wouldn't noise work just as well? Although an abiliation study is performed there is no standard deviation reported so it is unclear if this number is fair.
In Figure 5 the t-sne doesn't look correct. The points all seems to be projected on walls which could indicate some sort of overflow error. The text only devotes 3 lines to discuss this figure. It is not mentioned what part of the model the t-sne is computed from. To me this experiment that studies the internal representation is critical to convincing a reader to use this method.
The segmentation results sound good. Where is the improvement coming from? The experimental section is cut short. The experiment section is really squeezed in the last two pages while the other sections are overly descriptive and could be reduced.
The figures should be changed to be visible without color (put a texture on each block).


PAY LESS ATTENTION WITH LIGHTWEIGHT AND DYNAMIC CONVOLUTIONS
Self-attention is a useful mechanism to build generative models for language and images. It determines the importance of context elements by comparing each element to the current time step. In this paper, we show that a very lightweight convolution can perform competitively to the best reported self-attention results. Next, we introduce dynamic convolutions which are simpler and more efficient than self-attention. We predict separate convolution kernels based solely on the current time-step in order to determine the importance of context elements. The number of operations required by this approach scales linearly in the input length, whereas self-attention is quadratic. Experiments on large-scale machine translation, language modeling and abstractive summarization show that dynamic convolutions improve over strong self-attention models. On the WMT'14 English-German test set dynamic convolutions achieve a new state of the art of 29.7 BLEU.

The authors present lightweight convolutions and dynamic convolutions, two significant advances over existing depthwise convolution sequence models, and demonstrate very strong results on machine translation, language modeling, and summarization. Their results go even further than those of the Transformer paper in countering the conventional wisdom that recurrence (or another way of directly modeling long-distance dependencies) is crucial for sequence-to-sequence tasks. Some things that I noticed:
- While you do cite "Depthwise Separable Convolutions for Neural Machine Translation" from Kaiser et al. (ICLR 2018), there are some missed opportunities to compare more directly to that paper (e.g., by comparing to their super-separable convolutions). Kaiser et al. somewhat slipped under the community's radar after the same group released the Transformer on arXiv a week later, but it is in some ways a more direct inspiration for your work than the Transformer paper itself.
- I'd like to see more analysis of the local self-attention ablation. It's fantastic to see such a well-executed ablation study, especially one that includes this important comparison, but I'd like to understand more about the advantages and drawbacks of local self-attention compared to dynamic convolutions. (For instance, dynamic convolutions are somewhat faster at inference time in your results, but I'm unsure if this is contingent on implementation choices or if it's inherent to the architecture.)
- From a systems and implementation perspective, it would be great to see some algorithm-level comparisons of parallelism and critical path length between dynamic convolutions and self-attention. My gut feeling is that dynamic convolutions significantly more amenable to parallelization on certain kinds of hardware, especially at train time, but that the caching that's possible in self-attention inference might make the approaches more comparable in terms of critical path latency at inference time; this doesn't necessarily line up with your results so far though.
- You mostly focus on inference time, but you're not always as clear about that as you could be; I'd also like to see train time numbers. Fairseq is incredibly fast on both sides (perhaps instead of just saying "highly optimized" you can point to a paper or blog post?)
- The nomenclature in this space makes me sad (not your fault). Other papers (particularly a series of three papers from Tao Shen at University of Technology Sydney and Tianyi Zhou at UW) have proposed architectures that are similarly intermediate between self-attention and (in their case 1x1) convolution, but have decided to call them variants of self-attention. I could easily imagine a world where one of these groups proposed exactly your approach but called it "Dynamic Local Self-Attention," or even a world where they've already done so but we can't find it among the zillions of self-attention variants proposed in the past year. Not sure if there's anything anyone can do about that, but perhaps it would be helpful to briefly cite/compare to some of the Shen/Zhou work.
- I think you should have tried a language modeling dataset with longer-term dependencies, like WikiText-103. Especially if the results were slightly weaker than Transformer, that would help place dynamic convolutions in the architecture trade-off space.
That last one is probably my most significant concern, and one that should be fairly easy to address. But it's already a great paper.


It is unclear whether the extensively applied long-short term memory (LSTM) is an optimised architecture for recurrent neural networks. Its complicated design makes the network hard to analyse and non-immediately clear for its utilities in real-world data. This paper studies LSTMs as systems of difference equations, and takes a theoretical mathematical approach to study consecutive transitions in network variables. Our study shows that the cell state propagation is predominantly controlled by the forget gate. Hence, we introduce DecayNets, LSTMs with monotonically decreasing forget gates, to calibrate cell state dynamics. With recurrent batch normalisation, DecayNet outperforms the previous state of the art for permuted sequential MNIST. The Decay mechanism is also beneficial for LSTMbased optimisers, and decrease optimisee neural network losses more rapidly.

This paper analyses the internal dynamics of an LSTM, focusing on the cell state as being the most important component, and analyses what directly influences the contents of the cell state using difference equations. The authors note that at any timestep, the output cell state is the sum of (previous cell state * forget) and (input gate * input). The former can only shrink or maintain the cell value, which the authors label 'catch' and the latter can increase the magnitude, labelled 'release'.
The authors show that for a single neuron, with chosen values for the forget gate and inputs, consistent growth or consistent shrinking of the cell state can be observed. When the forget is large, say 0.9, the input gate can be anywhere in the range 0.5, 1.0] and still produce growth in the cell value. For forget gate = 0.25, an even larger range of input gate values all produce a shrinking cell value.
Due to the forget gate seemingly having a stronger effect than the (input gate * input) component, the authors propose to hard wire the forget gate to produce a continuous and monotonic decrease, producing the DecayNet. The rate of this decay is controlled by a learned function of the input and previous hidden state, with some shifting in order to maintain a monotonic decrease. Each forget neuron will decrease at a different rate through the processing of a sequence, leading to sections of the cell state which will decay slowly and sections which will decay quickly.
The authors perform two sets of experiments. The second is sequence classification with the standard 'pixel by pixel' permuted sequential MNIST, in which they show a new SOTA with using Recurrent Batch Norm combined with DecayNet. They also demonstrate a DecayNet with fewer parameters producing roughy the same median performance as a normal LSTM but with lower variance.
The first experiment is described as "image classification", with MNIST and Fashion-MNIST. This section is unclear to me, I had initally assumed that the data would be fed in one pixel at a time, but due to the presence of the other experiments I presume this is not the case. It is not clear what the 'time' dimension is in how the RNNs are applied here, if not through some ordering of pixels. If the entire image is presented as a flattened input, and the time dimension is iterating through the dataset, then there is no reason to use an RNN here. More detail must be added here to make it clear exactly how these RNNs are being applied to images - the text says the softmax layer is produced from the final hidden state, but without the information about how the different hidden states are produced for a given training example this not meaningful. I can imagine that both tasks are pixel by pixel, and the only difference is whether to apply the permutation.. but that is my guesswork.
In general I find this paper an interesting idea, reasonably well communicated but some parts are not clear. All the experiments (as far as I can tell) work on fixed length sequences. One advantage of an LSTM is that can run onnline on arbitrary length data, for example when used in a RL Agent. In those circumstances, does learning a fixed monotonic delay on the forget gate make sense? I would guess not, and therefore I think the paper could be more explicit in indicating when a DecayNet is a good idea.
There are definitely tasks in which you want to have the forget gate drop to zero, to reset the state, and then go back up to 1 in subsequent timesteps to remember some new information. Presumably the monotonic delay would perform poorly.
Is DecayNet appropriate only when you have fixed length sequences, where the distribution of 'when does relevant information appear in the input' is fixed? These questions make me doubt the generality of this approach, whereas "this reformulation increases LSTM modelling power ... and also yields more consistent results" from the abstract reads like this is a strictly better LSTM. A much wider variety of experiments would be required to justify this sentence.
It would be interesting to see some diagrams of forget gate / cell state changes throughout a real task, ie a graph with `k` on the x axis. The presentation of the new forget gate in "System 2" is clear in terms of being able to implement it, but it's not intuitive to me what this actually looks like. The graphs I suggest might go a long way to providing intuition for readers.
Overall while I like the spirit of trying to understand and manipulate LSTM learning dynamics I am recommending reject. I do not think the paper sufficiently motivates why a monotonic decay should be good, and while the new SOTA on permuted MNIST is great, I'm concerned that the first experiments are not reproducable, as detailed previously in this review. All hyperparameters appear to be present, so this paper would be reproducable, except for the NIST experiments.
General recommendations for a future resubmission:
* Clarify description of first MNIST experiments, and how they are different from permuted MNIST.
* Experiments on a wider variety of canonical RNN tasks - Penn Treebank is an obvious contender.
* Some mention of in what situations this is obviously not a good model to use (RL?)
* More intuition / visualisations as to what the internal dynamics inside DecayNet look like, vs normal LSTM.
* Devote less space to the initial dynamics analysis, or modify to be representative of a real task. This part was interesting on first read, but the only thing I think it really proves is 'when we artificially choose the input values things get bigger or smaller'. The important thing is, what actually happens when training on a task that we care about - if the same catch and release dynamics are observable, then that makes the idea more compelling.
Notes and suggestions:
I feel the notation would be clearer if instead of k = 1 .. D, this index was t = 1 ... T. This makes it cleare that s_k is not the k'th item in the array, but rather than whole activation array at a specific time. The notation \sigma_t could then be replace with \tanh.
"We replace \sigma_t with sin for an ergodic delay over time": as this is a new gate for the forget gate, should this be \sigma_s?
One DL rule of thumb heard relatively often is to simply initialise LSTM forget gate biases to 1, to "remember more by default". As this is a (much simpler) way of trying to influence the behaviour of the gate, and it anecdotally improves data efficiency, it is worth mentioning in the paper.


ON THE CONVERGENCE OF A CLASS OF ADAM-TYPE ALGORITHMS FOR NON-CONVEX OPTIMIZATION
This paper studies a class of adaptive gradient based momentum algorithms that update the search directions and learning rates simultaneously using past gradients. This class, which we refer to as the "Adam-type", includes the popular algorithms such as Adam (Kingma & Ba, 2014)  , AMSGrad (Reddi et al., 2018)  , AdaGrad (Duchi et al., 2011). Despite their popularity in training deep neural networks (DNNs), the convergence of these algorithms for solving non-convex problems remains an open question. In this paper, we develop an analysis framework and a set of mild sufficient conditions that guarantee the convergence of the Adam-type methods, with a convergence rate of order O(log T / √ T ) for non-convex stochastic optimization. Our convergence analysis applies to a new algorithm called AdaFom (AdaGrad with First Order Momentum). We show that the conditions are essential, by identifying concrete examples in which violating the conditions makes an algorithm diverge. Besides providing one of the first comprehensive analysis for Adam-type methods in the non-convex setting, our results can also help the practitioners to easily monitor the progress of algorithms and determine their convergence behavior.

Summary:
This paper presents a convergence analysis in the non-convex setting for a family of optimization algorithms, which the authors call the "Adam-type". This family incorporates popular existing methods like Adam, AdaGrad and AMSGrad. The analysis relies only on standard assumptions like Lipschitz smoothness and bounded gradients.
Individual Comments/Questions:
- In Table 1, the characterization of Adam ignores the fact that, in practice, Adam adds a positive epsilon to the $\hat{v}_t$ in the denominator. I would like the authors to at least comment on that in the paper. I assume that AdaFom and AMSGrad also need such an epsilon in practice. Could the author comment on whether (and how) this would affect their analysis of those methods? In particular, in Theorem 3.1, can we really assume the term $\Vert \alpha_t m_t / \sqrt{\hat{v}_t} \Vert$ to be bounded by a constant without such an epsilon?
- In the first bullet point in Section 3.1, the authors relate the term
$$ \sum_t \Vert \alpha_t g_t / \sqrt{\hat{v}_t} \Vert^2 $$ (*)
to the term $\sum_t \alpha_t^2$ in the analysis of SGD. I don't think this is a fair analogy. The effective step size of Adam-type methods is $\alpha_t / \sqrt{\hat{v}_t}$, while the aformentioned term also contains the magnitude of the stochastic gradient $g_t$. So, while the SGD analysis only poses a condition on the step sizes, bounding (1) also poses a condition on the magnitude of the stochastic gradient.
- In the experiments of Section 3.2.1, the authors use a step size of 0.01 for SGD (which really is gradient descent, since this is a non-stochastic problem). Existing theory tells us that GD only converges for step sizes smaller than 2/L where L is the Lipschitz constant of the gradient, which is L=200 in this example. So this is literally setting the method up for failure and I don't really see any merit in that experiment.
- The experiments in Section 4 are of course very limited, but this paper makes a significant theoretical contribution, so I don't really see the need for extensive experiments.
- To my knowledge, under similar assumptions, plain SGD has been show to converge at a rate of O(1/sqrt(T)). The convergence analysis presented here has an additional log(T) factor, so it is not really suitable to explain any possible benefits of these adaptive methods over SGD. This is totally fine in and of itself; after all the analysis of theses methods is hard and this is a great first step. The issue I have is that this is not mentioned in the paper at all.
Originality:
To the best of my knowledge, the convergence analysis of the Adam-type methods (including established methods AdaGrad, RMSprop, Adam, AMSGrad) in the _non-convex_ setting is a novel, original contribution. The authors also propose a new algorithm, AdaFom. This exact algorithm is proposed in [1], which was uploaded to arXiv before the ICLR deadline. However, this can be considered concurrent work.
Significance:
The convergence properties of popular optimization methods in machine learning (e.g., Adam) are generally very poorly understood in "realistic" settings. The analysis presented in this paper is an important step to better theoretical understanding of these methods which, in my opinion, is highly significant.
Correctness:
This was a short-notice emergency review and I did not check any of the proofs in the appendix. I will try to verify at least parts of the proofs in the coming days.
Conclusion:
This is an original paper making a significant theoretical contribution. I can't comment on the correctness of the mathematical analysis (yet). I'm cautiously recommending acceptance for now, but would be willing to upgrade my rating if the authors respond to my comments/questions.
[1] Zou and Shen. On the Convergence of Weighted AdaGrad with Momentum for Training Deep Neural Networks. https://arxiv.org/abs/1808.03408.
--------------------------------
Update
--------------------------------
The authors have provided a detailed response to my concerns and have fixed many of them in their revised version. I verified parts of the proofs in the appendix (Theorem 3.1 and its Corollaries). I congratulate the authors on their work and recommend acceptance!


MULTI-STEP RETRIEVER-READER INTERACTION FOR SCALABLE OPEN-DOMAIN QUESTION ANSWERING
This paper introduces a new framework for open-domain question answering in which the retriever and the reader iteratively interact with each other. The framework is agnostic to the architecture of the machine reading model, only requiring access to the token-level hidden representations of the reader. The retriever uses fast nearest neighbor search to scale to corpora containing millions of paragraphs. A gated recurrent unit updates the query at each step conditioned on the state of the reader and the reformulated query is used to re-rank the paragraphs by the retriever. We conduct analysis and show that iterative interaction helps in retrieving informative paragraphs from the corpus. Finally, we show that our multistep-reasoning framework brings consistent improvement when applied to two widely used reader architectures (DR. QA and BIDAF)  on various large open-domain datasets -TRIVIAQA-unfiltered, QUASAR-T, SEARCHQA, and SQUAD-open 1 .

This paper introduces a new framework to interactively interact document retriever and reader for open-domain question answering. While retriever-reader framework was often used for open-domain QA, this bi-directional interaction between the retriever and the reader is novel and effective because
1) If the retriever fails to retrieve the right document at the first step, the reader can give a signal to the retriever so that the retriever can recover its mistake at the next step
2) The idea of `reader state` from the reader to the retriever is new
3) The retriever use question-independent representation of paragraphs, which does not require different representation depending on the question and makes the framework easily scalable.
Strengths
1) The idea of multi-step & bi-directional interaction between the retriever and the reader is novel enough (as mentioned above). The paper contains enough literature studies on existing retriever-reader framework in open-domain setting, and clearly demonstrates how their framework is different from them.
2) The authors run the experiments on 4 different dataset, which supports the argument about the framework’s effectiveness.
Weakness
1) The authors seem to highlight multi-step `reasoning`, while it is not `reasoning` in my opinion. Multi-step reasoning refers to the task which you need evidence from different documents, and/or you need to find first evident to find the second evidence from a different document. I don’t think the dataset here are not multi-step reasoning dataset, and the authors seem not to claim it either. Therefore, I recommend using another term (maybe `multi-step interaction`?) instead of `multi-step reasoning`.
2) While the idea of multi-step interaction and how it benefits the overall performance is interesting, the analysis is not enough. Figure 3 in the paper does not have enough description — for example, I got the left example means step 2 recovers the mistake from step 1, but what does the right example mean?
Questions on result comparison
1) On TriviaQA (both open and full), the authors mentioned the result is on hidden test set — did you submit it to the leaderboard? I don’t see the same numbers on the TriviaQA leaderboard. Also, the authors claim they are SOTA on TriviaQA, but there are higher numbers on the leaderboard (which are submitted prior to the ICLR deadline).
2) There are other published papers with higher result on Quasar-T, SearchQA and TriviaQA (such as https://aclanthology.info/papers/P18-1161/p18-1161 and https://arxiv.org/abs/1805.08092) which the authors did not compare with.
3) In Section 4.2, is there a reason for the specific comparison to AQA (5th line), though AQA is not SOTA on SearchQA? I don’t think it means latent space is better than natural language space. They are totally different model and the only intersection is they contains interaction between two submodules.
4) In Section 5, the authors mentioned their framework outperforms previous SOTA by 15% margin on TriviaQA, but what is that? I don’t see 15% margin in Table 2.
Marginal comments:
1) If I understood correctly, `TriviaQA-open` and `TriviaQA-full` in the paper are officially called `TriviaQA-full` and `open-domain TriviaQA`. How about changing the term for readers to better understand the task? Also, in Section 4, the authors said TriviaQA-open is larger than web/wiki setting, but to my knowledge, this setting is part of the wiki setting.
2) It would be great if the authors make the capitalization consistent. e.g. EM, Quasar-T, BiDAF. Also, the authors can use EM instead of `exact match` after they mentioned EM refers to exact match in Section 4.2.
Overall comment
The idea in the paper is interesting, and their model and experiments are concrete. My only worries is that the terms in the paper are confusing and performance comparison are weak. I would like to update the score when the authors update the paper.
Update 11/27/2018
Thanks for the authors for updating the paper. The updated paper have more clear comparisons with other models, with more & stronger experiments with the additional dataset. Also, the model is claimed to perform multi-step interaction rather than multi-step reasoning, which clearly resolves my initial concern. The analysis, especially ablations in varying number of iterations, was helpful to understand how their framework benefits. I believe these make the paper stronger along with its initial novelty in the framework. In this regard, I vote for acceptance.


We propose a simple and robust non-parameterized approach for building sentence representations. Inspired by the Gram-Schmidt Process in geometric theory, we build an orthogonal basis of the subspace spanned by a word and its surrounding context in a sentence. We model the semantic meaning of a word in a sentence based on two aspects. One is its relatedness to the word vector subspace already spanned by its contextual words. The other is the word's novel semantic meaning which shall be introduced as a new basis vector perpendicular to this existing subspace. Following this motivation, we develop an innovative method based on orthogonal basis to combine pre-trained word embeddings into sentence representations. This approach requires zero training and zero parameters, along with efficient inference performance. We evaluate our approach on 11 downstream NLP tasks. Experimental results show that our model outperforms all existing zero-training alternatives in all the tasks and it is competitive to other approaches relying on either large amounts of labelled data or prolonged training time.

Paper overview: This paper proposes a new geometry-based method for sentence embedding from word embedding vectors, inspired by Arora et al (2017). The idea is to quantify the novelty,significance and corpus-wise uniqueness of each word. In order to do so, they analyze geometrically how the word vector of the target word relates to 1) the subspace created by the word-vectors in its context 2) its alignment with the meanings in its context (using SVD) 3) its presence in the all the corpus. For each of these aspects, they output a score or weight. The final sentence representation is a weighted average, using these scores, of the word vectors of the sentence.
Remarks and questions:
     1) In table 1, Glove and word2vec are word representations, how is the sentence representation computed here?
     2) The authors are not comparing to what is now considered the state of the art methods, such as Quick thoughts vectors (ICLR 2018, 'an efficient framework for learning sentence representations' by Logeswaran et al.), Transformer (Attention is all you need by Vaswani et al.) and ELMo (Deep contextualized word representations, by Peters et al.).
Points in favor:
    1) Results: The method gives the best performance for non-training methods with an +2 point improvement on average, although it cannot beat training methods (see Table 3, for instance).
   2) On the result tables, it should be reported also the std, not just the average, so the reader can evaluate if the difference between the methods is statistically significant.
    3) Inference speed: the method is fast (see table 5)
    4) stability of the results: The method is robust to slight changes in the hyperparameters such as the size of the window, number of principal components used, etc (see Fig 2)
Points against:
     The methods presented in the paper are not novel. The main novelties are the geometrical analysis on the contribution of each word of the sentence to the sentence overall semantic meaning, and the definition of the scores (eqs 4,6,8) that allow to improve the weighted average sentence representation (eq 9), an idea already present in Arora et al.'s paper.
 Conclusion:
     Although the geometric analysis of the paper is interesting, I dont think it is sufficient to justify a paper at ICLR, unless, after comparison with the other methods proposed previously, the proposed model is still competitive and the difference is statistically significant.


AIN'T NOBODY GOT TIME FOR CODING: STRUCTURE-AWARE PROGRAM SYNTHESIS FROM NATURAL LANGUAGE
Program synthesis from natural language (NL) is practical for humans and, once technically feasible, would significantly facilitate software development and revolutionize end-user programming. We present SAPS, an end-to-end neural network capable of mapping relatively complex, multi-sentence NL specifications to snippets of executable code. The proposed architecture relies exclusively on neural components, and is trained on abstract syntax trees, combined with a pretrained word embedding and a bi-directional multi-layer LSTM for processing of word sequences. The decoder features a doubly-recurrent LSTM, for which we propose novel signal propagation schemes and soft attention mechanism. When applied to a large dataset of problems proposed in a previous study, SAPS performs on par with or better than the method proposed there, producing correct programs in over 92% of cases. In contrast to other methods, it does not require post-processing of the resulting programs, and uses a fixed-dimensional latent representation as the only interface between the NL analyzer and the source code generator.

# Summary
This paper introduces a model called SAPS for the task of mapping natural language descriptions of programs to the AST tree of the corresponding program. The model consists of a variation of a double recurrent neural network (DRNN) which is pre-trained using an autoencoder. The natural language description is turned into a latent vector using pretrained word embeddings and a bidirectional stacked LSTM. The final model consists of training this sentence embedding model jointly with the decoder of the autoencoder.
# Quality
The authors introduce a reasonable model which achieves good performance on a relevant task. The results section contains a fair amount of numbers which give some insight into the performance of the model. However, the results make it hard to compare the proposed model with other models, or with other training procedures.
For example, the Seq2Tree model that is shown in table 2 was not necessarily intended to be used without a search algorithm. It is also not mentioned how many parameters both models have which makes it hard to judge how fair of a comparison it is. (I couldn't find the dimensionality of the encoder in the text, and the decoder dimensionality is only shown in figure 2.)
The model proposed in this work uses decoder pretrained in an autoencoder setting. No results are shown for how the model performs without pretraining. Pretraining using autoencoders is a technique that fell out of favor years ago, so it seems worthwhile to investigate whether or not this pretraining is necessary, and if so, why and how it aids the final performance.
It is unclear to me what type of robustness the authors are trying to show in table 5. The use of robustness here is not clear (robustness is often used to refer to a network's susceptability to adversarial attacks or perturbations of the weights). It also seems that the type of "simple replacements" mentioned are very similar to the way the examples were generated in the first place (section 4 of Polosukhin). If the goal is to measure generalization, why do the authors believe that performance on the test set alone is not a sufficient measure of generalization?
Some smaller comments and questions:
* In section 4 you mention training the sentence-to-tree and the sentence-to-vector mappings. Isn't the sentence-to-vector model a subset of the sentence-to-tree model? Should I interpret this as saying that, given the pretrained decoder and the glove embeddings, you now train the entire model jointly? Or do you first learn the mapping from sentences to the latent space, and only then finetune the entire model?
* The attention mechanism is not actually an attention mechanism: Attention mechanisms are used to reduce a variable number of elements to a single element by learning a weighting function and taking a weighted sum. My understanding is that in this case, the input (the latent representation) is of fixed size. The term "gating function" would be more appropriate.
* You specify that the hidden states of the decoder are initialized to zero, but don't specify what the cell states are initialized to.
# Clarity
The writing in the paper is passable. It lacks a bit in structure (e.g., I would introduce the problem and dataset before introducing the model) and sometimes fails to explain what insights the authors draw from certain results, or why certain results are reported. Take table 3 as an example: As a reader, I was confused at first why I should care about the reconstruction performance of the autoencoder alone, considering its only purpose is pretraining. Then, when looking at the numbers, I am even more confused since it is counterintuitive that it is harder to copy a program than it is to infer it. At the end of the paragraph the authors propose an explanation (the encoder isn't as powerful as the decoder) but leave it unclear as to why these numbers were being reported in the first place.
In general, the paper would do well to restructure the text so that the reader is explained what the goal of the different experiments is, and what insights should be drawn from them.
A variety of smaller concerns and comments:
* Please reduce and harmonize the terminology in the paper: the terms latent-to-AST, NLP2Tree, NLP2Vec, tree2tree/tree-to-tree, sentence-to-tree, sentence-to-vector, NL-to-latent, and spec-to-latent all appear in the paper and several of them are redundant, making it significantly harder to follow along with the text.
* Avoid citing the same work multiple times within a paragraph; cite only the first use and use prose to make clear that future references are to the same work.
* Formula 14 has h_i^{(pred)} on both sides of the quation, and is used in the definition of A as well. I am assuming these two terms are actually the h_i^{(pred)} from equation 8, but this should be made clear in the notation.
* Why does figure 1 have boxes for "NL specification", "NL spec.", and "NL query"? In general, the boxes inconsistently seem to represent both values and operations.
* It is never explicitly stated that Seq2Tree is the model from the Polosukhin et al. paper, which is a bit confusing.
* Parameters is missing an -s in the first paragraph of section 4.
* It is said that regularization is applied to layer normalization, which I assume means that the regularization is applied to the gain parameters of the layer normalization.
* It says "like a in the above example" when table 1 is rendered at the bottom of the page by Latex.
# Originality and significance
The paper introduces model variations that the authors claim improve performance on this particular program synthesis problem. In particular, in the DRNN decoder the hidden state is never reset for the "horizontal" (breadth-first order) decoder, and each node is only allowed to attend over the latent representation of the program. The authors claim this means their model "significantly diverges from [other] works", which seems hyperbolical.
The main contribution of this work is then the performance on the program synthesis task of Polosukhin and Skidanov. However, the model fails to improve on the Seq2Tree-guided search approach, so its main claimed benefit is that it is trained end-to-end. Although there is a strong trend in ML research to prefer end-to-end systems, it is worthwhile to ask when and why end-to-end systems are preferred. It is often clear that they are better than having separately learned components that are combined later. However, this does not apply to the model from Polosukhin et al., which consists of a single learned model being used in a search, which is a perfectly acceptable technique. In comparison, translations in neural machine translation are also produced by performing a beam search, guided by the probabilities under the model.
The paper would be significantly stronger if it could show that some alternative/existing method (e.g., a standard DRNN, or a Seq2Tree model with the same number of parameters, or a non-pretrained network) would fail to solve the problem where the authors' proposed method does not. However, the single comparison with the Seq2Tree model does not show this.
# Summary
Pros:
* Reasonable model, extensive results reported
* Decently written
Cons:
* Unclear how the performance compares to other models
* Not well justified why end-to-end methods would be better than guided-search based methods
* Model architectural differences seem relatively minor compared to original DRNN
* The pretraining using an autoencoder and the use of pretrained word embeddings seems arbitrary and is not critically evaluated
* Lack of coherent story to several results (the autoencoder performance, robustness analysis)


Reward Constrained Policy Optimization
Solving tasks in Reinforcement Learning is no easy feat. As the goal of the agent is to maximize the accumulated reward, it often learns to exploit loopholes and misspecifications in the reward signal resulting in unwanted behavior. While constraints may solve this issue, there is no closed form solution for general constraints. In this work, we present a novel multi-timescale approach for constrained policy optimization, called 'Reward Constrained Policy Optimization' (RCPO), which uses an alternative penalty signal to guide the policy towards a constraint satisfying one. We prove the convergence of our approach and provide empirical evidence of its ability to train constraint satisfying policies.

The paper introduces RCPO, a model-free deep RL algorithm for learning optimal policies that satisfy some per-state constraint on expectation. The derivation of the algorithm is quite straightforward, starts from the definition of constrained optimization problem, and proceed by forming and optimizing the Lagrangian. Additionally, a value function for the constraint is learned. The algorithm is only compared to a baseline optimizing the Lagrangian directly using Monte-Carlo sampling.
The paper has two major problems. First, while the derivation of the method makes intuitively sense, it is supported by vaguely stated theorems, which mixes rigorous guarantees with practical approximations. For example, Equation 4 assumes strong duality. How would the result change if weak duality was used instead? The main result in Theorem 1 makes the assumption that dual variable is constant with respect the policy, which might be true in practice, but it is not obvious how the approximation affects the theory. Further, instead of simply referring to prior convergence results, I would strongly suggest including the exact prior theorems and assumptions in the appendix.
The second problem is the empirical validation, which is incomplete and misleading. Constrained policy optimization is not a new topic (e.g. work by Achiam et al.), so it is important to compare to the prior works. It is stated in the paper that the prior methods cannot be used to handle mean value constraints. However, it would be important to include experiments that can be solved with prior methods too, for example the experiments in Achiam at al. for proper comparison. The results in Table 2 are confusing: what makes the bolded results better than the others? If the criterion is highest return and torque < 25%, then \lambda=0.1 should be chosen for Hopper-v2. Also, The results seem to have high variance, and judging based on Table 2 and Figure 3, it is not obvious how well RCPO actually works.
To summarize, while the topic is undoubtedly important, the paper would need be improved in terms of better differentiating the theory from practice, and by including a rigorous comparison to prior work.
Minor points:
- What is exactly the difference between discounted sum constraint and mean value constraint?
- Could consider use colors in Table 1.
- Section 4.1.: What does “... enables training using a finite number of samples” exactly mean in this case?
- Table 2: The grid for \lambda is too sparse.
- Proof of Theorem 1: What does it mean \theta to be stable?
- Proof of Theorem 2: “Theorem C” -> “Theorem 1”


TOWARD UNDERSTANDING THE IMPACT OF STALENESS IN DISTRIBUTED MACHINE LEARNING
Most distributed machine learning (ML) systems store a copy of the model parameters locally on each machine to minimize network communication. In practice, in order to reduce synchronization waiting time, these copies of the model are not necessarily updated in lock-step, and can become stale. Despite much development in large-scale ML, the effect of staleness on the learning efficiency is inconclusive, mainly because it is challenging to control or monitor the staleness in complex distributed environments. In this work, we study the convergence behaviors of a wide array of ML models and algorithms under delayed updates. Our extensive experiments reveal the rich diversity of the effects of staleness on the convergence of ML algorithms and offer insights into seemingly contradictory reports in the literature. The empirical findings also inspire a new convergence analysis of SGD in non-convex optimization under staleness, matching the best-known convergence rate of O(1/ √ T ).

This paper presents and empirical and theoretical study of the convergence of asynchronous stochastic gradient descent training if there are delays due to the asynchronous part of it. The paper can be neatly split in two parts: a simulation study and a theoretical analysis.
The simulation study compares, under fixed hyperparameters, the behavior of distributed training under different simulated levels of delay on different problems and different model architectures. Overall the results are very interesting, but the simulation could have been more thorough. Specifically, the same hyperparameter values were used across batch sizes and across different values of the distributed delay. Some algorithms failed to converge under some settings and others experienced dramatic slowdowns, but without careful study of hyperparameters it's hard to tell whether these behaviors are normal or outliers. Also it would have been interesting to see a recurrent architecture there, as I've heard much anecdotal evidence about the robustness of RNNs and LSTMs to asynchronous training. I strongly advise the authors to redo the experiments with some hyperparameter tuning for different levels of staleness to make these results more believable.
The theoretical analysis identifies a quantity called gradient coherence and proves that a learning rate based on the coherence can lead to an optimal convergence rate even under asynchronous training. The proof is correct (I checked the major steps but not all details), and it's sufficiently different from the analysis of hogwild algorithms to be of independent interest. The paper also shows the empirical behavior of the gradient coherence statistic during model training; interestingly this seems to also explain the heuristic commonly believed that to make asynchronous training work one needs to slowly anneal the number of workers (coherence is much worse in the earlier than later phases of training). This quantity is interesting also because it's somewhat independent of the variance of the stochastic gradient across minibatches (it's the time variance, in a way), and further analysis might also show interesting results.


This paper propose a cumulative saliency based Globally Soft Filter Pruning (GSFP) scheme to prune redundant filters of Convolutional Neural Networks (CNNs). Specifically, the GSFP adopts a robust pruning method, which measures the global redundancy of the filter in the whole model by using the soft pruning strategy. In addition, in the model recovery process after pruning, we use the cumulative saliency strategy to improve the accuracy of pruning. GSFP has two advantages over previous works: (1) More accurate pruning guidance. For a pre-trained CNN model, the saliency of the filter varies with different input data. Therefore, accumulating the saliency of the filter over the entire data set can provide more accurate guidance for pruning. On the other hand, pruning from a global perspective is more accurate than local pruning. (2) More robust pruning strategy. We propose a reasonable normalization formula to prevent certain layers of filters in the network from being completely clipped due to excessive pruning rate. Experiment results show that GSFP is effective on many classic CNN architectures and different data sets. Within my knowledge, GSFP is the first algorithm to combine global and soft pruning strategies. Notably, on MNIST and CIFAR-10, it achieves a much higher compression ratio compared with prior work while maintaining the same test accuracy.

This paper proposes new heuristics to prune and compactify neural networks. The heuristics try to consider 1) filter weight and gradient normalization by their size, 2) saliency normalization across layers, 3) saliency accumulation across batch. The author claims that these can address problems previous studies had and experimental results show that the proposed method achieve higher compression ration with less loss of accuracy.
This paper discusses how to determine the importance of filters. As cited in the paper, there have been various attempts to tackle the same problem and the paper contributes to the series of efforts. The paper introduces a new way to compute such importance values based on their observations. The method is tested on a few dataset and a various models and compared with some previous studies. I like the simple but yet effective method, however, I think it is not good enough for ICLR.
1. What is effective is not very clear.
The paper pointed out issues of previous studies and proposed the new method based on the observations. However, only the final method is compared with other work and it did not examine which part of the method was essential. The paper needs more detailed analyses on the proposed method. For example, the readers would want to know if the normalization in Eq. (2) is really important or not. The readers would be also interested in a visualization like Fig. 2 without saliency normalization.
2. The numbers of previous studies come only from their papers.
It is very difficult to know if the proposed method is actually better than the previous methods if the numbers just come from their papers. We want to compare the ideas, but not numbers. The essential ideas of other papers need to be abstracted and tested in the paper by itself. It relates to the first item above. "Baseline" should be a baseline method but not models without pruning.
Numbers from other papers are still useful to show that the numbers in the paper are good in an absolute manner.
3. Weak theoretical reasoning
Eq. (1) in the paper is not actually used for optimization while some previous methods do. If the proposed method is better than other methods which directly optimizes the loss, should we think that the formulation itself is bad?
The paper discusses imbalanced pruned pruning results. It needs to show that it is actually bad.
* minor things
** Table 1: Should the first row of "Wen et al. (2016)" have "5-19" and "1-5" or "4-19" and "1-4" for "Filters" and "Channels", respectively?
** I'd recommend another proofreading.


RMSProp and ADAM continue to be extremely popular algorithms for training neural nets but their theoretical convergence properties have remained unclear. Further, recent work has seemed to suggest that these algorithms have worse generalization properties when compared to carefully tuned stochastic gradient descent or its momentum variants. In this work, we make progress towards a deeper understanding of ADAM and RMSProp in two ways. First, we provide proofs that these adaptive gradient algorithms are guaranteed to reach criticality for smooth non-convex objectives, and we give bounds on the running time.
Next we design experiments to empirically study the convergence and generalization properties of RMSProp and ADAM against Nesterov's Accelerated Gradient method on a variety of common autoencoder setups and on VGG-9 with CIFAR-10. Through these experiments we demonstrate the interesting sensitivity that ADAM has to its momentum parameter β 1 . We show that at very high values of the momentum parameter (β 1 = 0.99) ADAM outperforms a carefully tuned NAG on most of our experiments, in terms of getting lower training and test losses. On the other hand, NAG can sometimes do better when ADAM's β 1 is set to the most commonly used value: β 1 = 0.9, indicating the importance of tuning the hyperparameters of ADAM to get better generalization performance.
We also report experiments on different autoencoders to demonstrate that NAG has better abilities in terms of reducing the gradient norms, and it also produces iterates which exhibit an increasing trend for the minimum eigenvalue of the Hessian of the loss function at the iterates.

Summary:
This paper present a convergence analysis of the popular methods RMSProp and ADAM in the case of smooth non-convex functions. In particular it was shown that the above adaptive gradient algorithms are guaranteed to reach critical points for smooth non-convex objectives and bounds on the running time are provided. An empirical investigation is also presented with main focus on the comparison of the adaptive gradient methods and the Nesterov accelerated gradient algorithm (NAG).
Comments:
Although the results are promising, I found the reading (mainly because of the not defined notation) of this paper really hard.
In terms of presentation, the motivation in introduction is fine, but the following section named "Notations and Pseudocodes" is confusing and has many undefined notations which makes the paper very hard to read. It gives the impression that the section was added the last minute. For example what is fundtion "g" in the definition 1? What is support(v) and the diag(v) in the definition 2. the diag(v) is more obvious to me but then why at page 18 the diag(v)at the top of the page is bold (are these two things different)?
In the presentation of RMSProp what the $g_t^2$ means? Please have a look to last year's ICLR paper [Reddi, Sashank J., Satyen Kale, and Sanjiv Kumar. "On the convergence of adam and beyond." (2018).] for a more appropriate introduction of the notation.
In the introduction the authors refer to NAG as a stochastic variant of the Nesterov's acceleration and they informally present the algorithm in the end of the first paragraph. There the update rule includes stochastic gradients \nable f_i(.) while in the formal presentation in the update rule there is \nabla f(x) which is the full gradient of the objective function of the original problem. I expect this difference is somehow justified from the mentioning in the algorithm of the possibly noisy oracle but this is never mention in the main text.
If the above statements in terms of presentation, are ignored the convergence results and numerical experiments are interesting.
However, the numerical evaluation does not correspond to the theoretical results. It is a comparison of NAG ,ADAM and RMSPROP with interesting conclusions  that can be beneficial for practitioners that they use these methods.
Some missing references:
On Adam methods:
1) Chen, Xiangyi, et al. "On the convergence of a class of adam-type algorithms for non-convex optimization." arXiv preprint arXiv:1808.02941 (2018).
2) Zhou, Dongruo, et al. "On the convergence of adaptive gradient methods for nonconvex optimization." arXiv preprint arXiv:1808.05671 (2018).
On momentum (heavy ball) methods:
3) Loizou, Nicolas, and Peter Richtárik. "Momentum and stochastic momentum for stochastic gradient, Newton, proximal point and subspace descent methods." arXiv preprint arXiv:1712.09677 (2017).


Attention mechanisms in sequence to sequence models have shown great ability and wonderful performance in various natural language processing (NLP) tasks, such as sentence embedding, text generation, machine translation, machine reading comprehension, etc. Unfortunately, existing attention mechanisms only learn either high-level or low-level features. In this paper, we think that the lack of hierarchical mechanisms is a bottleneck in improving the performance of the attention mechanisms, and propose a novel Hierarchical Attention Mechanism (Ham) based on the weighted sum of different layers of a multi-level attention.
Ham achieves a state-of-the-art BLEU score of 0.26 on Chinese poem generation task and a nearly 6.5% averaged improvement compared with the existing machine reading comprehension models such as BIDAF and Match-LSTM. Furthermore, our experiments and theorems reveal that Ham has greater generalization and representation ability than existing attention mechanisms.

The paper proposes to enhance existing multi-level attention (self-attention) mechanism by obtaining query and key vectors (= value vectors) from all levels after weighted-averaging them. The paper claims that this is also theoretically beneficial because the loss function will converge to zero as the number of layers increase. It claims that the proposed architecture outperforms existing attention-based models in English MRC test (SQuAD), Chinese MRC test, and Chinese poem generation task.
I find three major issues in the paper.
1.  I think the proposed hypothesis lacks the novelty that ICLR audience seeks for. Through many existing architectures (ResNet, ELMo), we already know that skip connection between CNN layers or weighted average of multiple LSTM layers could improve model significantly. Perhaps this could be an application paper that brings existing methods to a slightly different (attention) domain, but not only such paper is less suitable for ICLR, but also it would require strong experimental results. But as I will detail in the second point, I also have some worries about the experiments.
2. The experimental results have problems. For English MRC experiment (SQuAD), the reproduced match-LSTM score is ~10% below the reported number in its original paper. Furthermore, it is not clear whether the improvement comes from having multiple attention layers (which is not novel) or weighted-averaging the attention layers (the proposed method). BiDAF and match-LSTM have single attention layers, so it is not fair to compare them with multi-layer attention.
3. Lastly, I am not sure I understood the theoretical section correctly, but it is not much interesting that having multiple layers allow one to approach closer to zero loss. In fact, any sufficiently large model can obtain close-to-zero loss on the training data. This is not a sufficient condition for a good model. We cannot guarantee if the model has generalized well; it might have just overfit to the training data.
A few minor issues and typos  on the paper:
- First para second sentence: In -> in
- First para second sentence: sequence to sequence -> sequence-to-sequence
- Second last para of intro: sentence fragment
- Figure 3: would be good to have English translation.