We present a deep recurrent neural network architecture to solve a class of stochastic optimal control problems described by fully nonlinear Hamilton Jacobi Bellman partial differential equations. Such PDEs arise when one considers stochastic dynamics characterized by uncertainties that are additive, state dependent and control multiplicative. Stochastic models with the aforementioned characteristics have been used in computational neuroscience, biology, finance and aerospace systems and provide a more accurate representation of actuation than models with only additive uncertainty. Previous literature has established the inadequacy of the linear HJB theory and instead relies on a non-linear Feynman-Kac lemma resulting in a second order Forward-Backward Stochastic Differential Equations representation. However, the proposed solutions that use this representation suffer from compounding errors and computational complexity resulting in lack of scalability. In this paper, we propose a deep learning based algorithm that leverages the second order Forward-Backward SDE representation and LSTM based recurrent neural networks to not only solve such Stochastic Optimal Control problems but also overcome the problems faced by traditional approaches and shows promising scalability.
The resulting control algorithm is tested on nonlinear systems from robotics and biomechanics in simulation to demonstrate feasibility and out-performance against previous methods.

######### Rebuttal Response:
Thanks for the clarifications and especially for updating the formatting. The current state does not convince me to rate the paper as weak accept but I increased my rating to weak reject.
"Pereira et. al. has shown that a recurrent network architecture using LSTM outperforms the fully connected networks at every time step proposed by Han et. al. in task completion, space and time complexity. Therefore, in this paper we choose to use an LSTM-based network architecture."
-> Yes it might be true that a recurrent function approximator does in practice perform better than a feed-forward function approximator. However, in theory a feed-forward network should be sufficient as the value function does not depend on the previous states. Therefore, the question is, why does the LSTM perform better? Does the recurrent nature of the LSTM make the predictions smoother compared to a feed-forward network?  Can any other regularizing scheme be introduced s.t. the feed-forward networks performs equally well?
######### Review:
Summary:
The paper builds on the work of Pereira et. al. and uses forward backward stochastic differential equations to learn the Hessian of the Value function Vxx and \partial _t V_x + 1/2 tr(\partial_{xx} V_x CC^T). In contrast to the prior work, this paper introduces multiplicative noise for the control and uses second order optimization. The performance is evaluated on different control tasks, e.g., linear system, cartpole, quadcopter & human arm actuated by tendons.
Conclusion:
All in all, I like the proposed research of combining theoretical approaches and deep learning to perform trajectory optimization and I would like to see much more of this research like this within the ICLR community. Furthermore, I think that the paper has a contribution and that the paper was improved compared to the initial Neurips submission (i.e., adding ILQG as baseline). However, the writeup and formatting is still very much sub-standard and must be improved to make this paper worth publishing. The current write-up is not accessible for the ICLR community and the understandability must be significantly improved (Details are provided below). Therefore, I currently rate this paper as a clear rejection but I am happy to improve the score to 7-8 if the write up is improved during the rebuttal.
Theoretical Structure:
I like the introduction, which covers the topic but might be a bit too long. Maybe you want to shorten the introduction and add an additional related work section at the end. The stochastic control introduction is nice and has the correct level of abstraction for the reader. However, the paper introduces many complex concepts which are not essential for understanding the paper (e.g., filtered probability space etc.). One might want to trade off understandability vs. mathematical rigor especially, if the paper does not rely on these concepts. Furthermore, you might want to make eq 1 more explicit as the multiplicative action noise is not visible from eq 1. Section 3 'A FBSDE Solution to the HJB PDE' is the most problematic section of this paper, which is not understandable for the common ICLR reader. Eq. 6, which just appears without any derivation, is not understandable and the reader has no intuition how to derive this eq. Furthermore, Eq 6 (page. 4) uses notations which is only clearly introduced later within the paper or even the appendix (e.g., Y being the propagated value function, Z being the propagated gradient of the value function is only mentioned in the appendix, i.e., page 12. \Gamma is only introduced in page 5. Yes, Eq. 7 defines these variables but the style of definition is not standard and one does not expect the variables to be defined in this style.). Could the authors please provide an intuitive derivation of these equation and use clearer notation (Why would one want to abstract V, V_x, V_x, \mathcal|{H}(V_x) in the first place as these are intuitive for the ICLR community and sufficiently short?) Especially, as this section highlights the difference to the prior works of Pereira et. al., this section should be very clear. Section 4 is clear but should include the loss function as the loss is not trivial and essential for the optimization. Currently, the loss description is buried in the appendix. All in all, the theoretical explanation and the bloated notation should be simplified and every equation should be embedded into an intuitive derivation. Currently these explanations are not understandable without reading the appendix and prior work.
Experiments:
The experiments apply 2FBSDE to 4 different control tasks (Linear system, quadcopter, cartpole & human arm) and compare the performance to the prior work of FBSDE and iLQG. The number of baselines and systems is sufficient. However, the paper should provide more evaluations:
(1) Plot the histogram of the obtained cost distributions.
(2) Plot a single state- and action trajectory (and the action distribution). Using these plots, the level of noise and smoothness and hence the applicability to physical systems can be evaluated.
(3) Plot the noise free trajectories and show that these mean trajectories reach the desired solution.
(4) Specify the exact cost function for every experiment
Further Comments to the individual experiments:
Cartpole:
The Cartpole iLQG seems to perform much better (swing-up the pendulum faster, don't deviate so much from x=0, much more coherent velocity compared to 2FBSDE, FBSDE). Could the authors please discuss these aspects in more detail and present experiments with longer time-horizons to check whether the proposed method can stabilize the cart at [0, 0, 0, 0]. The current plots don't reach this target state. Your plots also hint that the cartpole does not need to pre-swing the pendulum, which is most likely due to the very low action cost. This selection of action cost significantly simplifies the problem. Could the authors please include a cartpole with higher action cost and show that 2FBSDE can learn to pre-swing the pendulum.
The quadcopter:
Could the authors please specify the exact quadcopter dynamics. What kind of abstraction did you model? What are the control inputs? Furthermore, the citation for the dynamics is wrong and puts the supervisor of the master thesis as first author. Furthermore, can the authors please provide longer plots to highlight, which method can stabilize the system.
Human Arm:
For the human arm neither iLQG or 2FBSDE reach the desired target location. Can you explain why no trajectory optimization does reach the desired position.
Formatting:
Please rework the formatting such that the inline math does not cause the formatting issues of different line spacings (e.g., sec. 2.1, sec. 5) and irregular whitespaces (e.g., last line of paragraph 2.1 Preliminaries). Please remove the color coding of text for the experiments and make sure that the legends are sufficiently large and include all lines. Currently the legends are missing the target state. You can also extend the figure captions to highlight the conclusion of the plots. Please rework the figures such that the figures do not cause so much whitespace (e.g., Figure 3, 4 & 5). When reconfiguring the plots, you gain space, which can be used for further explanation of the theory. Furthermore, you might want add dotted lines to the confidence intervals as the confidence intervals are important but the differences are not clearly visible from the plots. Also, the labelling in figure 3 seems wrong, the axis labeled cart velocity should be pendulum angle and the pendulum angle axis should be cart velocity. All in all, the formatting can be significantly improved, which is especially bad as this paper is most likely a resubmission from Neurips.
Minor Comments / Questions:
- 'where l :Rnx×Rnu→R+ is the running cost and C1,23 \phi :Rnx→R+ is the terminal state cost.' Is l and \phi of class C^{1,2} or only one of them? The notation is confusing and should be simplified.
- Can you comment on how important this multiplicative noise in physical system? ¬¬¬
- Why are you using a LSTM instead of a simple feed-forward neural network as the ff-nn should be sufficient to model V(x) as the value function is not recurrent. Have you tried using a simple ff-nn?


Natural question generation (QG) aims to generate questions from a passage and an answer. Previous works on QG either (i) ignore the rich structure information hidden in text, (ii) solely rely on cross-entropy loss that leads to issues like exposure bias and inconsistency between train/test measurement, or (iii) fail to fully exploit the answer information. To address these limitations, in this paper, we propose a reinforcement learning (RL) based graph-to-sequence (Graph2Seq) model for QG. Our model consists of a Graph2Seq generator with a novel Bidirectional Gated Graph Neural Network based encoder to embed the passage, and a hybrid evaluator with a mixed objective function that combines both the crossentropy and RL loss to ensure the generation of syntactically and semantically valid text. We also introduce an effective Deep Alignment Network for incorporating the answer information into the passage at both the word and contextual level. Our model is end-to-end trainable and achieves new state-of-the-art scores, outperforming existing methods by a significant margin on the standard SQuAD benchmark for QG.

This paper focuses on improving the performance on the task of natural language generation. To this end, they propose a  graph-to-sequence (Graph2Seq) model for the task of question generation which exploits the rich structure information in the text as well as use reinforcement learning based policy gradient approach to address the exposure bias and inconsistency between test/train distributions in cross-entropy optimization setup.
The Graph2Seq model has a bidirectional gated graph neural network on the encoder side, which is an extension of traditional gated graph neural network. To exploit the rich hidden structure information in the input text, they explore two different methods: (1) syntax-based static graph; (2) semantics-aware dynamics graph.
The proposed model achieves state-of-the-art results on question generation, which are further validated with human evaluations.
Overall, The paper should be rejected because the paper have minor extensions to each of their modules but lacking any major important contribution.
Some major concerns:
1) The bidirectional gated GNN doesn’t seem novel enough in comparison to previous work
2) I believe RL to Graph2Seq is a minor extension from Seq2Seq, since RL mostly deals with the decoder part which is common in across both Graph2Seq and Seq2Seq
Arguments:
1) Adding the structure information to the encoder via the GNNs is an interesting angle for question generation. Compared to previous work, this paper proposes an additional deep alignment network on the encoder side to align paragraph and answer. However, the importance of this module is not well studied in the experiments section. I see that there is an ablation with/without this module but its not fairly compared with other aligning or simple techniques like in Zhao et al. (2018).
2) The addition of RL component to Graph2Seq is a minor extension from the Seq2Seq model, because both of these models have similar decoder and RL mainly deals with it. Also the importance of each reward component or the effect of each phrase-matching automatic metrics is missing.
3) Open part I am unclear about the dataset is which dataset version did you use sentence-level or paragraph level? I see that the baselines correspond to sentence-level, but the Figure-1 alignment module has input paragraph. Also I couldn’t find the SeqCopyNet (Zhou et al., 2018) split-2 BLEU4 score=13.02 in the original paper!
4) Some of the latest papers which use BERT based models are not discussed in the paper which achieve state-of-the-art-results: “Addressing Semantic Drift in Question Generation for Semi-Supervised Question Answering”
5) For Table-2 results are the differences in the scores for the two models statistically significant?
6) Table-3: First of all, evaluating only one metric is no sufficient. Please see latest papers that have also introduced new metrics that are good for QG evaluation, e.g., Q-BLEU. The gap between G2Ssta+BERT vs. G2Ssta+BERT+RL seems negligible, and missing statistical significance.
7) Minor comments: BLUE -> BLEU; please cite KNN-style graph sparsification; the color choices in Figure-3 are creating confusion in understanding the model.


Modern neural network architectures often generalize well despite containing many more parameters than the size of the training dataset. This paper explores the generalization capabilities of neural networks trained via gradient descent. We develop a data-dependent optimization and generalization theory which leverages the low-rank structure of the Jacobian matrix associated with the network. Our results help demystify why training and generalization is easier on clean and structured datasets and harder on noisy and unstructured datasets as well as how the network size affects the evolution of the train and test errors during training. Specifically, we use a control knob to split the Jacobian spectum into "information" and "nuisance" spaces associated with the large and small singular values. We show that over the information space learning is fast and one can quickly train a model with zero training loss that can also generalize well. Over the nuisance space training is slower and early stopping can help with generalization at the expense of some bias. We also show that the overall generalization capability of the network is controlled by how well the labels are aligned with the information space. A key feature of our results is that even constant width neural nets can provably generalize for sufficiently nice datasets. We conduct various numerical experiments on deep networks that corroborate our theoretical findings and demonstrate that: (i) the Jacobian of typical neural networks exhibit low-rank structure with a few large singular values and many small ones leading to a low-dimensional information space, (ii) over the information space learning is fast and most of the labels falls on this space, and (iii) label noise falls on the nuisance space and impedes optimization/generalization.

The authors use the empirically supported assumption of the low-rank nature of the neural network Jacobian to provide new elements of data-dependent optimization and generalization theory. By modelling the data as a low-rank object itself, they analytically study the evolution of the train and test errors. The paper divides the space of weights and biases into the “information” and “nuisance” subspaces, spanned by the top largest and the remaining singular vectors of the Jacobian respectively. They use this division and its alignment with the low-rank structure of the data to talk about convergence speed. Finally, they provide numerical experiments to back their claims.
I enjoyed the paper, however, there were many points where I was unclear on the precise nature of the assumptions used / the strength of the results.
Disclaimer: I didn’t manage to read through the proofs in the appendix and cannot therefore vouch for its correctness.
-- Point 1 --
Leveraging the data structure
I am unclear on how exactly you were modelling the structure of the data. From your proofs, it seems that you have been dealing with the matrix X comprising the concatenated flatted vectors of the raw input features (e.g. pixels) of the input data [x1,x2,...,x_datasetsize]^T. In particular, the only place where I see data explicitly enter is in Definition 2.1, where you look at the X X^T and fi’(w X) fi’(w X)^T.
If the data is linearly separable in the raw input space on its own, then I see that the matrix X X^T will be low-rank (related to the number of classes). I also see your point about the connection of the y to the relevant (semantic) clusters. The same argument could by applied to fi’(w X) fi’(w X)^T -- provided that the features produced are again linearly separable, we will observe this object to have a low (number of classes - 1) rank.
What is unclear to me is whether these assumptions are warranted. I understand that some simple datasets, e.g. MNIST, are essentially linearly separable in the raw pixels, and therefore the X X^T indeed is low rank. However, I doubt anything like that is true for big datasets, such as ImageNet. For deep networks that are used on these big datasets, such a modelling assumptions would likely not be true. I wonder how this relates to your results, since the low-rank nature of the Hessian is observed even for those networks, which is in turn related to the low rank nature of the Jacobian.
-- Point 2 --
Data implicitly present in the Jacobian tensor.
I wonder how you modelled the Jacobian tensor that you started using on page 3. Since the Jacobian -- the derivative of the output logits with respect to the weights, has to be evaluated at a particular input X, the assumptions you make on the data are in turn having an effect on the Jacobian, and vice versa.
I am unclear on exactly what assumptions you make about the object, and whether you are actually saying that its low rank structure comes from the data, is empirical observed and therefore assumed, or due to the network regardless of the data.
I recently saw a new arXiv submission that seems to be looking into this on real networks: https://arxiv.org/pdf/1910.05929.pdf Their model explicitly assumes that logit gradients cluster in the weight space in a particular way.
-- Point 3 --
Square loss vs softmax
You are using the square loss |f(X) - y|^2 throughout your work. Many of the empirical low-rank observations of the Hessian (related to the JJ^T) are performed on real networks with the cross-entropy loss. While the Hessian with the square loss is of the form JJ^T + terms, the softmax in the cross-entropy loss introduces an additional cross-term (let us call it P for now), which in turn makes it JP(PJ)^T + terms. Do you know how this relates to your results?
More generally, does the square loss you use make the results significantly different from what we would get for a softmax?
-- Point 4 --
Neural Tangent Kernel (NTK) -- assumptions
Under the NTK assumption, you still need to model the derivatives of each logit with respect to each weight on each input in order to obtain the Jacobian matrix and in turn JJ^T. I am therefore very confused by “Based on our simulations the M-NTK indeed haslow-rank structure with a few large eigenvalues and many smaller ones. “ on page 5. What assumptions exactly do you use in your model?
-- Point 5 --
Neural Tangent Kernel (NTK) -- validity
By assuming the NTK holding, do you limit the validity of your results? I think it is believed that NTK might not, generally speaking, be enough to capture the complexity of DNNs, and therefore assuming it might limit the range of applicability of any results derived assuming it.
-- Conclusion --
In general, my points of confusion often stem from being unsure as to what parts of the argument were assumed and based on what empirical / theoretical evidence, and what parts were generically true. While the paper seems interesting, I am not sure what its novel contribution is and how broad the claims made actually are in their applicability.
Appendix: I was not able to judge the proofs in the appendix.


Image classification problems today are often solved by first collecting examples along with candidate labels, second obtaining clean labels from workers, and third training a large, overparameterized deep neural network on the clean examples. The second, manual labeling step is often the most expensive one as it requires manually going through all examples. In this paper we propose to i) skip the manual labeling step entirely, ii) directly train the deep neural network on the noisy candidate labels, and iii) early stop the training to avoid overfitting. With this procedure we exploit an intriguing property of overparameterized neural networks: While they are capable of perfectly fitting the noisy data, gradient descent fits clean labels faster than noisy ones. Thus, training and early stopping on noisy labels resembles training on clean labels only. Our results show that early stopping the training of standard deep networks (such as ResNet-18) on a subset of the Tiny Images dataset (which is obtained without any explicit human labels and only about half of the labels are correct), gives a significantly higher test performance than when trained on the clean CIFAR-10 training dataset (which is obtained by labeling a subset of the Tiny Images dataset). In addition, our results show that the noise generated through the label collection process is not nearly as adversarial for learning as the noise generated by randomly flipping labels, which is the noise most prevalent in works demonstrating noise robustness of neural networks.

The paper reads well, the topic is interesting, and the connection between early stopping and fitting true label distributions before noisy ones, if generally true, is of potential impact. I do not feel the paper is ready for publication, though:
Missing context/references:
	⁃	Noisily collected labels are standard elsewhere, e.g., in sentiment analysis (self-ratings), in discourse parsing (explicit discourse markers), in weakly supervised POS tagging (crowdsourced dictionaries), in NER (Wikidata links), and in machine translation from mined parallel corpora or comparable corpora.
	⁃	Learning a ground truth from a population of turkers (MACE and subsequent work). It is well-known that the noise in such silver standard or non-adjudicated annotations is also “more structured” and to a large extent predictable (Plank et al., EACL 2014 and subsequent work on learning/predicting inter-annotator disagreements).
	⁃	Connection between L2 and early stopping. Can you replicate your results with simple L2 regularisation?
	⁃	Cost-sensitive learning of systematic disagreements (use class coherence scores for cost-sensitive learning).
	⁃	The observation that “clean examples are fitted faster than noise” is obviously related to baby steps training regimes (training on easy examples first), including active learning.
Experimental details and flaws:
	⁃	Using a black box search engine to collect labels is problematic for a few reasons: (a) Search engines change, so results are hard to reproduce. (b) Search engines are biased toward certain types of categories. Such biases will be reinforced by any model trained on this data.
	⁃	I’m always a little worried about papers that only report results for a single dataset.
	⁃	I would have liked to see some more error analysis. Are the improvements on classes with more or less support in the original dataset, for example?
	⁃	It seems to be that it would have been relatively straight-forward to construct synthetic datasets that would more directly evaluate the hypothesis that the true labels are learned first. I realize you’re interested in “real” noise rather than “random” noise - but synthetic noise doesn’t have to be random.
	⁃	That “the noise generated through the label collection process is not nearly as adversarial for learning as the noise generated by randomly flipping labels” is no surprise, and I would not present this as a finding. It would be interesting to describe the bias, e.g., by presenting confusion matrices (see Plank et al., EACL 2014).


We introduce SLM Lab, a software framework for reproducible reinforcement learning (RL) research. SLM Lab implements a number of popular RL algorithms, provides synchronous and asynchronous parallel experiment execution, hyperparameter search, and result analysis. RL algorithms in SLM Lab are implemented in a modular way such that differences in algorithm performance can be confidently ascribed to differences between algorithms, not between implementations. In this work we present the design choices behind SLM Lab and use it to produce a comprehensive single-codebase RL algorithm benchmark. In addition, as a consequence of SLM Lab's modular design, we introduce and evaluate a discrete-action variant of the Soft Actor-Critic algorithm (Haarnoja et al., 2018)   and a hybrid synchronous/asynchronous training method for RL agents.

This paper presents a new RL library called « SLM Lab ». Its most relevant features for RL research are: (1) modularity to help re-use existing components (thus reducing the risk of subtle implementation differences when comparing algorithms), (2) implementations of most popular algorithms like DQN & variants, A3C, PPO, SAC, (3) ability to parallelize both actors (through vectorized environments) and the learner (through distributed gradient descent), and (4) utilities for hyper-parameter optimization, reproducible experiments and reporting. The paper also reports performance over Atari games, Roboschool environments as well as some Unity ML-Agents tasks. Finally, it provides a high-level overview of SLM Lab’s capabilities compared to 23 other RL open source libraries, showing that it is the only one that combines: reporting of the performance of the implemented algorithms, ability to specify hyper-parameters in the config file, parallelization, hyper-parameter optimization, and visualization of the results.
Overall this looks like a solid RL library, but I am not convinced that it brings enough novelty to the RL software landscape for a published ICLR paper — it would better fit in a workhop dedicated to ML libraries for instance, thus the weak reject.
Table 3 shows that SLM Lab is the only RL library with such a broad offering of features, and this is definitely impressive, but I would argue that many of these features can typically be added to other libraries by plugging in other open source software. For instance there are several tools for experiment management and hyper-parameter optimization (and for RLLib in particular, hyper-parameter optimization is not checked but is straightforward with Ray Tune). TensorBoard can also often be easily used for visualization.
The parallelization capabilities of SLM Lab also seem limited: if I understand correctly, actor parallelization can only occur on a single machine, and thus an algorithm like Ape-X or R2D2 could not be implemented. If this is correct then it is a major limitation of the framework, since such parallelization across multiple computers can be extremely useful when environments are slow and costly to run (in CPU / RAM).
It is not clear to me to which extent multi-agent is supported. It seems like it is possible to have multiple agents in one environment, but is that enough for general multi-agent RL? (ex: how to specify individual / team rewards? share information between agents? deal with agents not acting all at the same timestep? centralize part of the training / execution?…)
I appreciate that the paper presents a discrete version of SAC, mentioning how easy it was to implement thanks to the modular design of SLM Lab, but results from Table 1 do not look very good (especially since it did not work on some of the environments). Relying on the Gumbel-softmax might not be the most robust & stable way to train a discrete SAC — see e.g. the recent « Soft Actor-Critic for Discrete Action Settings » for a different approach.
Finally, it is also great to have some benchmarks of the algorithms being implemented, but at least for Atari, I am not aware of previous work using the exact same evaluation setting, so it is hard to tell how they compare to other implementations.
In spite of the above, I do not mean to criticize SLM Lab too heavily as from what I can tell it seems to be a a solid library with many useful features, and I am sure many researchers will find it useful in their day-to-day work.
Minor points:
- Anonymity was clearly violated with the two github links
- Having some synthetic result on Atari (like the typically reported median human-normalized score) would be good
- I am personally not a fan of large config JSON files due to the lack of comments in JSON
- A.5 (« Key Implementation Lessons ») is great!
Review update after author feedback: I am on the fence for this paper, but still leaning towards rejection due to the fact I am still not convinced that this library brings that much novelty compared to existing other libraries (although it seems like a nice RL library, I am not sure ICLR is the right venue for talking about it). The authors argue that their benchmark results are a key contribution of the paper, but I do not find these results particularly insightful, especially the Atari ones that are not comparable to previous results (due to using a different evaluation method) and the lack of state-of-the-art algorithms like Rainbow or IQN.


We show that a variety of modern deep learning tasks exhibit a "double-descent" phenomenon where, as we increase model size, performance first gets worse and then gets better. Moreover, we show that double descent occurs not just as a function of model size, but also as a function of the number of training epochs. We unify the above phenomena by defining a new complexity measure we call the effective model complexity, and conjecture a generalized double descent with respect to this measure. Furthermore, our notion of model complexity allows us to identify certain regimes where increasing (even quadrupling) the number of train samples actually hurts test performance. Figure 1: Left: Train and test error as a function of model size, for ResNet18s of varying width on CIFAR-10 with 15% label noise. Right: Test error, shown for varying train epochs. All models trained using Adam for 4K epochs. The largest model (width 64) corresponds to standard ResNet18.

This paper provides a valuable and detailed empirical study of the double descent behaviour in neural networks. It investigates presence of this behaviour in a range of neural network architectures and apart of identifying it as a function of the model size it also identifies it as a function of training time which I believe is novel. Overall I think the paper presents results valuable to the community. At the same time it has several issues that need to be addressed. If the issues are addressed in a satisfactory manner I will recommend acceptance of the paper.
Issues and questions:
** "Such a phenomenon was first postulated by Belkin et al. (2018) who named it “double descent” and demonstrated it on MNIST with decision trees, random features, and 2-layer neural networks with `2 loss." This is not a correct statement. The authors cite works (Advani & Saxe 2017) and there is also "Stefano Spigler, Mario Geiger, Stephane d’Ascoli, Levent Sagun, Giulio Biroli, and Matthieu Wyart. A jamming transition from under-to over-parametrization affects loss landscape and generalization. arXiv preprint arXiv:1810.09665, 2018." that both identified this behaviour prior to the Belkin et al. (2018) work. There is even a much older line of work identifying the peak in the generalization error by Opper's group:
http://www.ki.tu-berlin.de/fileadmin/fg135/publikationen/opper/Op03b.pdf (1995)
Siegfried B¨os and Manfred Opper. Dynamics of training. In Advances in Neural Information
Processing Systems, pages 141–147, 1997.
http://www.ki.tu-berlin.de/fileadmin/fg135/publikationen/opper/Op01.pdf
Moreover, as is only said in the supplementary material, Geiger et al. also tested on the CIFAR dataset, while the introduction of the present article only mentions previous experiments on MNIST. The fact that previous work also observed this in CIFAR should be moved in the introduction.
** The authors claim to define the "effective model complexity (EMC)" and claim this as one of the main results of the paper.
Presenting this as a new measure is strange, this exactly is what Geiger et all call jamming transition/threshold in Stefano Spigler, Mario Geiger, St´ephane d’Ascoli, Levent Sagun, Giulio Biroli, and Matthieu Wyart. A jamming transition from under-to over-parametrization affects loss landscape and generalization. arXiv preprint arXiv:1810.09665, 2018. And very closely related to what Belkin calls "interpolation threshold" (to define interpolation threshold we could fix the model and vary the number of samples, Belkin et all fix number of samples and vary size of the model, but one is just the dual of the other). Giving it a yet completely new name seems to create more noise than value.
The relation between EMC and the jamming transition is discussed in the supplement. However, not in a accurate way. The authors say "a ”jamming transition” when the number of parameters exceeds a threshold that allows training to near-zero train loss." but jamming is inspired by the physical phenomena when spheres are added into a finite volume box and the more spheres we have the harder it gets to fit them all in until it is not possible anymore. The analogy here is that fitting sphere in = reaching training error zero. Then number of spheres = number of samples. Thus the more samples the harder it is to get the training error to zero, leading to the jamming transition. Both in training and jamming the number of samples at which this happened depends on the details of the protocol/training, thus it does depend on things such as regularization.
The only aspect that I have not seen covered in the jamming analogy is the epoch-wise double descent.
In any case the discussion in the paper needs to be adjusted and these relevant relations to previous work corrected and moved from the supplement to the introduction.
** "Informally, our intuition is that for model-sizes at the interpolation threshold, there is effectively only one model that fits the train data and this interpolating model is very sensitive to noise in the train set and/or model mis-specification." This intuition is correct I believe. However, it should not be called "our intuition" as it already appeared in the line of work by Opper cited above.
** The authors present as another main result the fact that under comparable training conditions training with more data samples provides worse generalization, examples of this is also included already in the papers by Opper et al. cited above.


While classic work in convex-concave min-max optimization relies on averageiterate convergence results, the emergence of nonconvex applications such as training Generative Adversarial Networks has led to renewed interest in last-iterate convergence guarantees. Proving last-iterate convergence is challenging because many natural algorithms, such as Simultaneous Gradient Descent/Ascent, provably diverge or cycle even in simple convex-concave min-max settings, and previous work on global last-iterate convergence rates has been limited to the bilinear and convex-strongly concave settings. In this work, we show that the HAMILTONIAN GRADIENT DESCENT (HGD) algorithm achieves linear convergence in a variety of more general settings, including convex-concave problems that satisfy a novel "sufficiently bilinear" condition. We also prove convergence rates for stochastic HGD and for some parameter settings of the Consensus Optimization algorithm of Mescheder et al. (2017).

Summary:
The paper, considers methods for solving smooth unconstrained min-max optimization problems.  In particular, the authors prove that the Hamiltonian Gradient Descent (HGD) algorithm converges with linear convergence rate to the min-max solution. One of the main contributions of this work is that the proposed analysis is focusing on last iterate convergence guarantees for the HGD. This result, as the authors claim can be particularly useful in the future for analyzing more general settings (nonconvex-nonconcave min-max problems).
In addition, two preliminary convergence theorems were provided for two extensions of HGD: (i) a stochastic variant of HGD and (ii)  Consensus Optimization Algorithm (CO) (by establishing connections of CO and HGD).
Main Comments:
The paper is well written and the main contributions are clear. I believe that the idea of the paper is interesting and the convergence analysis seems correct, however i have some concerns regarding  the presentation and the combination of different assumptions used in the theory.
1) I think definition 2.5 of Higher order Lipschitz is very strong assumption to have. What exactly means? Essentially the authors upper bounded any difficult term appear in the theorems. Is it possible to avoid having something so strong? Please elaborate.
2) In assumption 3.1 is not clear what $L_H$ is. This quantity never mentioned before. Reading the Lemmas of Section 4 (Lemma 4.4) you can see that it is the smoothness parameter of function $H$. Thus, is not necessary to have it there (not important for the definition).
3) What is the main difference on the combination of assumptions on Theorems 3.2, 3.2 and 3.4. Which one is stronger. Is there a reason for the existence of Theorem 3.3?
4) All the results heavily depend on the PL condition. I think having this in mind, showing the convergence of Theorems 3.2-3.4 is somehow trivial. In particular, one can propose several combinations of assumptions in order for the function H to satisfy the PL condition. Can we avoid having the PL condition? The authors need to elaborate more on this.
5) In Theorem 5.2, the term 1/sqrt(2) is missing from the final bound.
Minor Suggestions:
In first paragraph of page 5 where the authors divide the existing literature into the three particular cases, I am suggesting to add the refereed papers inside each one of this cases (which papers assumed function g bilinear , which papers strongly convex-concave etc.)
I understand that the main contribution of the work is the theoretical analysis of the proposed method but would like to see some numerical evaluation in the main paper. There are some preliminary results in the appendix but it will be useful for the reader if there are are some plots showing the benefit of the method in comparison with existing methods that guarantee convergence (which method is faster?). In the current experiments there is a comparison only with CO algorithm and SGDA.
In general i find the paper interesting, with nice ideas and I believe that will be appreciated from researchers that are interested on smooth games and their connections to machine learning applications.
I suggest weak accept but I am open to reconsider in case that my above concerns are answered.
**********after rebuttal********
I would like to thank the authors for their reply and for the further clarification.
I will keep my score the same but I highly encourage the authors to add some clarification related to my last comment on the globally bounded gradient.
In their response they mentioned that the analysis only requires that  H is smooth and that $\|\xi(x^{(0)})\|$ is sufficient bound. This needs to be clear in the paper (add clear arguments and related references).
In addition, in their response they highlight the non-increasing nature of function H over the course of the algorithm which is important for their argument. Having this in mind note that the theoretical results on stochastic variant presented in the paper are wrong. In SGD,  function H does not necessarily decrease over the course of the algorithm. The authors either need to remove these results or restate them in a different way in order to satisfy the assumed conditions.


Learning diverse and natural behaviors is one of the longstanding goal for creating intelligent characters in the animated world. In this paper, we propose "COmposable Semi-parametric MOdelling" (COSMO), a method for generating long range diverse and distinctive behaviors to achieve a specific goal location. Our proposed method learns to model the motion of human by combining the complementary strengths of both non-parametric techniques and parametric ones. Given the starting and ending state, a memory bank is used to retrieve motion references that are provided as source material to a deep network. The synthesis is performed by the deep network that controls the style of the provided motion material and modifies it to become natural. On skeleton datasets with diverse motion, we show that the proposed method outperforms existing parametric and non-parametric baselines. We also demonstrate the generated sequences are useful as subgoals for actual physical execution in the animated world. Please refer to our project page 1 for more synthesised results.

This paper aims at taking techniques from motion interpolation into the regime where one is able to generate longer range motion sequences, in the domain of physically plausible computer animation of characters. In the way that the authors have set up the problem, an initial database seeds the search for plausible transitions between two given poses. So, the technique being proposed must address how to keep physical realism (the long-standing question of "dynamics filtering" along the lines of Yamane, Katsu, and Yoshihiko Nakamura. "Dynamics filter-concept and implementation of online motion generator for human figures." IEEE transactions on robotics and automation 19.3 (2003): 421-432. Of course the problem here also needs to address "style" which needs different models).
The authors propose an approach, building on recent neural network architectures including GANs and VAEs, which combines an embedding into a latent space to capture the style+content and then a Bi-LSTM model to make P-step predictions to extract the interpolating poses. The architecture is not very well explained and could be presented much better. However, the constituent elements are fairly standard ones. The learning of the embedding space is based on minimising a reconstruction loss and adapting parameters of a network that takes convolutions over time windows and stacks different channels. The LSTM training is also based on prior work in that domain.
The authors try to show with experiments that the proposed model is doing better both on diversity and accuracy in standard motion capture datasets. However, the baselines used, especially for the accuracy comparison, are more along the lines of an ablation study than a genuine comparison with alternate methods.
This is coupled with the fact that the authors do not seem to have engaged with a fairly established literature on modelling such sequences, e.g.,
Brand, M., & Hertzmann, A. (2000, July). Style machines. In Proceedings of the 27th annual conference on Computer graphics and interactive techniques (pp. 183-192). ACM Press/Addison-Wesley Publishing Co..
Li, Y., Wang, T., & Shum, H. Y. (2002, July). Motion texture: a two-level statistical model for character motion synthesis. In ACM transactions on graphics (ToG) (Vol. 21, No. 3, pp. 465-472). ACM.
These papers also solve the same problem the authors have set out to solve and arguably do pretty well. How well do they compare to the authors' approach and in what ways have they built further? It would be helpful to understand this with quantitative evidence.
As it stands, this comes across as a report on a preliminary experiment. Indeed, figure 10 is just a single learning curve without much more interpretation and analysis. The paper would be much stronger if situated better with respect to other established work and also supported by more systematic empirical experiments.


Domain adaptation is an open problem in deep reinforcement learning (RL). Often, agents are asked to perform in environments where data is difficult to obtain. In such settings, agents are trained in similar environments, such as simulators, and are then transferred to the original environment. The gap between visual observations of the source and target environments often causes the agent to fail in the target environment. We present a new RL agent, SADALA (Soft Attention DisentAngled representation Learning Agent). SADALA first learns a compressed state representation. It then jointly learns to ignore distracting features and solve the task presented. SADALA's separation of important and unimportant visual features leads to robust domain transfer. SADALA outperforms both prior disentangled-representation based RL and domain randomization approaches across RL environments (Visual Cartpole and DeepMind Lab).

Pros:
This paper proposed a new method for zero-shot transfer learning under the reinforcement learning setting. The use of attention weights to regularize the latent states was fairly interesting.
Cons:
Limited applicability of the proposed methods
- The paper was restricted in a setting where rewards, actions, and true states were identical between source and target environments, and only the observed states differed due to differing renderers. Working under such a restricted setting was interesting in its own right, but it might also lead to limited applicability of the proposed method in the real-world setting.
- The proposed method focused on solving a very specific problem: learning a dis-entangled latent representation for images. As a result, the potential impact of the proposed methods could be minimal.
Limited technical novelty
- The proposed method, SADALA, was built on top of Higgins et al., 2017 (DARLA). The only difference was an added attention layer to the learning of latent states. As a result, the novelty of the proposed method was very incremental and limited from a technology perspective.
- Even with additional attention layer, the paper could have performed a more thorough study to help the readers understand and appreciate the idea. For example, this paper didn’t discuss the tradeoff between training SADALA over separate stages, versus training it from end to end. For example, why the weights of the pre-trained beta-VAE had to be frozen and used as weights in the state representation stage.
Insufficient experiments
-More thorough discussion of the qualitative results should be helpful to understand whether the attention weights helped the model to focus on the right thing. For example, this paper did study the quality of reconstruction in Figure 3-5 of the proposed method. When comparing Figure 3 and Figure 5, it appeared to me that the reconstructed the angle of the pole was different from the original one. And it seemed like attention weights did successfully ignored the color of the cart and pole, but it ignored the angle of the pole, which should be important to the learning task. Unfortunately, the paper didn't further explain the implication of such misrepresentation.
-Quantitative results
* It would be interesting to all compare the proposed methods against model-agonistic methods like MAML
* It would be useful to include confidence intervals over different tasks.
* It would be useful to compare different methods with different parameter settings
* The authors mentioned “Visual Pendulum tasks” but didn’t include them in the paper
Reproducibility
- It's unclear to me how reproducible the research conducted in this paper was, and it would be useful to open source the code used to conduct the experiments.


The recent success of the lottery ticket hypothesis by Frankle & Carbin (2018) suggests that small, sparsified neural networks can be trained as long as the network is initialized properly. Several followup discussions on the initialization of the sparsified model have discovered interesting characteristics such as the necessity of rewinding (Frankle et al. (2019)), importance of sign of the initial weights (Zhou et al. (2019)), and the transferability of the winning lottery tickets (S. Morcos et al. (2019)). In contrast, another essential aspect of the winning ticket, structure of the sparsified model, has been little discussed. To find the lottery ticket, unfortunately, all the prior work still relies on computationally expensive iterative pruning. In this work, we conduct an in-depth investigation of the structure of winning lottery tickets. Interestingly, we discover that there exist many lottery tickets that can achieve equally good accuracy much before the regular training schedule even finishes. We provide insights into the structure of these early winning tickets with supporting evidence. 1) Under stochastic gradient descent optimization, lottery ticket emerges when weight magnitude of a model saturates; 2) Pruning before the saturation of a model causes the loss of capability in learning complex patterns, resulting in the accuracy degradation. We employ the memorization capacity analysis to quantitatively confirm it, and further explain why gradual pruning can achieve better accuracy over the one-shot pruning. Based on these insights, we discover the early winning tickets for various ResNet architectures on both CIFAR10 and ImageNet, achieving state-of-the-art accuracy at a high pruning rate without expensive iterative pruning. In the case of ResNet50 on ImageNet, this comes to the winning ticket of 75.31% Top-1 accuracy at 80% pruning rate in only 22% of the total epochs for iterative pruning.

Overview:
The paper is dedicated to conducting an in-depth investigation of the structure of winning lottery tickets. The author provides supporting evidence for the structure of the early winning tickets: 1) lottery tickets emerge when the weight magnitude of a model saturates with SGD optimization. 2) pruning before model saturation may result in accuracy degradation. In the experiment part, they employ the memorization capacity analysis and discover the early wining tickets without expensive iterative pruning. The author also conducts extensive experiments with various ResNet architectures on both CIFAR 10 and ImageNet, achieving state-of-the-art results with only 1/5 of the total epochs for iterative pruning.
Strength Bullets:
1. The experiment organization is complete and convincing. Especially for the figure, it not only clearly shows that lottery tickets emerge much earlier before full training ends, but also shows the effect of rewinding.
2. The author not reveal interesting observations, but also provides useful guidance. It is a complete logic chain that is also aligned with my intuition. For example, first, the author discusses the memorization capacity of different pruned models at different epochs. Then, they introduce a reasonable gradual pruning technique. Finally, they conduct experiments to confirm it.
3.  The early winning tickets in this paper achieve state-of-the-art results with only 1/5 of the total epochs for iterative pruning.
Weakness Bullets:
1. For lottery tickets, especially for early winning tickets, I think there is a lot of randomnesses. Thus, for the plot like figure 2, figure 5, they need to contain an error bar and the curve should be the average of tens of experiments. It will be more convincing if it decouples the randomness from the real patterns.
2. The description and organization of section 4 need to be more clear. For example, an algorithm pseudo code will definitely give readers a much more clear understanding of the early winning tickets finding strategy.
Question the authors don't answer which confuses me more:
Need more convincing analysis about the indicator - Hamming Distance
Just as the comment I posted after Review 1, we would like to see more analysis about Hamming Distance between different winning tickets. The author mentioned in the following way:
''However, as we demonstrate in Fig. 3 of our paper, the mask-distance does not well characterize the winning tickets. E.g., the lottery tickets drawn at Epoch 120 and 200 have a mask distance of 0.082 in Fig. 3, which is much larger than the mask distance between Epoch 180 and 200. Whereas, all three tickets achieve comparably high accuracy as shown in Fig. 1, implying a shallow correlation between the accuracy and the mask distance''
As far as I know, this observation only tells us that the structure of the lottery tickets changes, which are drawn from 120 epochs to 180 epochs (although the maintain a similar accuracy). However, we can not conclude that the mask distance is not a reliable measure. As mention by the [EB] paper provided by the authors, you can use the rate of the distance change to indicate the early winning tickets. In this way, we can find winning tickets much earlier than authors' work.
To better address this question, I suppose the authors need to provide more analysis of the mask distance indicator. I think all three reviewers would like to see the results. A good indicator for early winning tickets is very important, otherwise authors' notion of early still involves quite a lot of training.
Recommendation:
Due to the unsolved important question, here is a weak reject.


Traditional synchronous distributed training is performed inside a cluster since it requires high bandwidth and low latency network (e.g. 25Gb Ethernet or Infiniband). However, in many application scenarios, training data are often distributed across many geographic locations, where physical distance is long and latency is high. Traditional synchronous distributed training cannot scale well under such limited network conditions. In this work, we aim to scale distributed learning under high-latency network. To achieve this, we propose Delayed and Temporally Sparse (DTS) update that enables synchronous training to tolerate extreme network conditions without compromising accuracy. We benchmark our algorithms on servers deployed across three continents in the world: London (Europe), Tokyo (Asia), Oregon (North America) and Ohio (North America). Under such challenging settings, DTS achieves 90× speedup over traditional methods without loss of accuracy on ImageNet.

This paper tackles the issue of scalability in distributed SGD over a high-latency network.
Along with experiments, this paper contributes two ideas: delayed updates and temporally sparse updates.
- strengths:
· Clear-looking overall presentation.
  Good efforts to explain the network problems (latency, congestion) and how they are tackled by delayed updates and temporally sparse updates.
- weaknesses:
· While the overall presentation looks clean, the paper does not talk about the setting (one parameter server and many workers, only workers, etc).
  This is arguably basic information, and the readers is left to understand by themselves that the setting consists in many workers without a parameter server.
· There is no theoretical analysis of the soundness of the proposed algorithm.
  For instance in ASGD (that the authors cite), stale gradients are dampened before being used, which in turn is used to guarantee convergence.
  In the proposed algorithm, there is no dampening nor apparent limit of the maximum value of t; such a difference with prior art should entail a serious (theoretical) analysis.
· Finally, using SSGD for comparison is not very "fair", as communication-efficient algorithms have already been published for quite some time [1, 2, and follow-ups (e.g. searching for "local sgd")].
  At the very least a comparison with ASGD (cited) is necessary, as in a realistic setting latency is indeed a problem but arguably not bandwidth
  (plus, orthogonal gradient compression techniques do exist, e.g. as in "Federated Learning: Strategies for Improving Communication Efficiency").
questions to the authors:
- Can you clarify the setting?
- Can you give at least an intuition why accepting stale gradient is correct (i.e. does not impede convergence)?
  There is no theoretical limit on the value of t; can workers take any arbitrary old gradient?
  So when the training is close to convergence (if it reaches it), i.e., when the norm of the gradients are close to 0,  the algorithm could then use very old gradients (which norms could be orders of magnitude larger) to "correct the mismatch" caused by the local training; is this correct?
  To solve that issue, ASGD introduces a simple dampening mechanism (which is necessary in the convergence proof), which your algorithm does not have.
The related work section focuses on gradient compression techniques (which tackle low bandwidth, not latency) and asynchronous SGD (which is more prone to congestion, with a single parameter server),
but seems to overlook that sparse communication techniques already exist (this fact should at least be mentioned).
The idea of sparse communication for SGD exists since at least 2012 [1, Algorithm 3].
A first mention of the use of such techniques for "communication efficiency" dates from (at least) 2015 [2].
[1] Ofer Dekel, Ran Gilad-Bachrach, Ohad Shamir, and Lin Xiao.
    Optimal distributed online prediction using mini-batches.
    J. Mach. Learn. Res., 13(1):165–202, January 2012.
[2] Sixin Zhang, Anna E. Choromanska, Yann LeCun.
    Deep learning with Elastic Averaging SGD.
    NeurIPS, 2015.
I do not see a clear novelty, nor a proof (or even intuitions) that the proposed algorithm is theoretically sound.
The comparison with SSGD is arguably unfair, since SSGD is arguably not at all the state-of-the-art in the proposed setting (hence the claimed speedup of x90 can be very misleading).


The performance of deep neural networks is significantly affected by how well mini-batches are constructed. In this paper, we propose a novel adaptive batch selection algorithm called Recency Bias that exploits the uncertain samples predicted inconsistently in recent iterations. The historical label predictions of each sample are used to evaluate its predictive uncertainty within a sliding window. By taking advantage of this design, Recency Bias not only accelerates the training step but also achieves a more accurate network. We demonstrate the superiority of Recency Bias by extensive evaluation on two independent tasks. Compared with existing batch selection methods, the results showed that Recency Bias reduced the test error by up to 20.5% in a fixed wall-clock training time. At the same time, it improved the training time by up to 59.3% to reach the same test error.

This paper proposes Recency Bias, an adaptive mini batch selection method for training deep neural networks. To select informative minibatches for training, the proposed method maintains a fixed size sliding window of past model predictions for each data sample. At a given iteration, samples which have highly inconsistent predictions within the sliding window are added to the minibatch. The main contribution of this paper is the introduction of sliding window to remember past model predictions, as an improvement over the SOTA approach: Active Bias, which maintains a growing window of model predictions. Empirical studies are performed to show the superiority of Recency Bias over two SOTA  approaches. Results are shown on the task of (1) image classification from scratch and (2) image classification by fine-tuning pretrained networks.
+ves:
+ The idea of using a sliding window over a growing window in active batch selection is  interesting.
+ Overall, the paper is well written. In particular, the Related Work section has a nice flow and puts the proposed method into context. Despite the method having limited novelty (sliding window instead of a growing window), the method has been well motivated by pointing out the limitations in SOTA methods.
+ The results section is well structured. It's nice to see hyperparameter tuning results; and loss convergence graphs in various learning settings for each dataset.
Concerns:
- The key concern about the paper is the lack of rigorous experimentation to study the usefulness of the proposed method. Despite the paper stating that there have been earlier work (Joseph et al, 2019 and Wang et al, 2019) that attempt mini-batch selection, the paper does not compare with them. This is limiting. Further, since the proposed method is not specific to the domain of images, evaluating it on tasks other than image classification, such as text classification for instance, would have helped validate its applicability across domains.
- Considering the limited results, a deeper analysis of the proposed method would have been nice. The idea of a sliding window over a growing window is a generic one, and there have been many efforts to theoretically analyze active learning over the last two decades. How does the proposed method fit in there? (For e.g., how does the expected model variance change in this setting?) Some form of theoretical/analytical reasoning behind the effectiveness of recency bias (which is missing) would provide greater insights to the community and facilitate further research in this direction.
- The claim of 20.5% reduction in test error mentioned in the abstract has not been clearly addressed and pointed out in the results section of the paper.
- On the same note, the results are not conclusively in favor of the proposed method, and only is marginally better than the competitors. Why does online batch perform consistently than the proposed method? There is no discussion of these 	inferences from the results.
- The results would have been more complete if results were shown in a setting where just recency bias is used without the use of the selection pressure parameter. In other words, an ablation study on the effect of the selection pressure parameter would have been very useful.
- How important is the warm-up phase to the proposed method? Considering the paper states that this is required to get good estimates of the quantization index of the samples, some ablation studies on reducing/increasing the warm-up phase and showing the results would have been useful to understand this.
- Fig 4: Why are there sharp dips periodically in all the graphs? What do these correspond to?
- The intuition behind the method is described well, however, the proposed method would have been really solidified if it were analysed in the context of a simple machine learning problem (such as logistic regression). As an example, verifying if the chosen minibatch samples are actually close to the decision boundary of a model (even if the model is very simple) would have helped analyze the proposed method well.
Minor comments:
* It would have been nice to see the relation between the effect of using recency bias and the difficulty of the task/dataset.
* In the 2nd line in Introduction, it should be "deep networks" instead of "deep networks netowrks".
* Since both tasks in the experiments are about image classification, it would be a little misleading to present them as "image classification" and "finetuning". A more informative way of titling them would be "image classification from scratch" and "image classification by finetuning".
* In Section 3.1, in the LHS of equation 3, it would be appropriate to use P(y_i/x_i; q) instead of P(y/x_i; q) since the former term was used in the paragraph.
=====POST-REBUTTAL COMMENTS========
I thank the authors for the response and the efforts in the updated draft. Some of my queries were clarified. However, unfortunately, I still think more needs to be done to explain the consistency of the results and to study the generalizability of this work across datasets.  I retain my original decision for these reasons.


Deep generative models can emulate the perceptual properties of complex image datasets, providing a latent representation of the data. However, manipulating such representation to perform meaningful and controllable transformations in the data space remains challenging without some form of supervision. While previous work has focused on exploiting statistical independence to disentangle latent factors, we argue that such requirement can be advantageously relaxed and propose instead a non-statistical framework that relies on identifying a modular organization of the network, based on counterfactual manipulations. Our experiments support that modularity between groups of channels is achieved to a certain degree on a variety of generative models. This allowed the design of targeted interventions on complex image datasets, opening the way to applications such as computationally efficient style transfer and the automated assessment of robustness to contextual changes in pattern recognition systems.

Summary:
    Authors propose a method to analyze a trained generator (a neural network) from a trained generative model for a distribution of images and produce a family of modules (subsets of neurons) that control certain aspects of the image (upon intervention) such that the different modules identified are "distentangled" in the sense of a lot of work in this area attempting to find latent controllable factors.
This paper has many original ideas and substantial novelty. One of the prime ones is to focus on existing generators and identify subsets of neurons "inside" the generator box that can be manipulated producing meaningful changes in the image. All existing works attempt to constrain the latent space in an appropriate way during training so as to produce different latent factors that manipulate different aspects of the image. Authors point out that disentangled latents cannot be statistically independent. This is very important (and motivates them to group neurons in the model that are statistically dependent) and I cannot agree more with the authors on this point.
Transformation of the image is defined to be intrinsically disentangled if the same transformation can be produced in the image by intervening (transforming) only a small subset of neurons.
Authors assume that the map from z (latents) to image space (g_M) and map from neurons space to image space induced by the generator (\tilde{g}_M) are invertible .
Authors define a class of manipulations of certain subsets of neurons through counterfactuals. Setting the latents to be the same (unique) z that gave rise to the image Y, just intervene on a susbet E of neurons and set them to h (from the valid range of values it can take) - the result of this intervention for this unit (z) produces a counterfactual image corresponding to y. Now, if the counterfactuals corresponding to the manifold of images map back into the manifold, then it is called a faithful counterfactual.
Authors show a very interesting result - Transformation is disentangled if and only if it is produced by a faithful counterfactual mapping. There is an additional result relating modularity of a subset of neurons to disentanglement.
Then authors produce a concrete algorithm inspired by faithful counterfactual mapping as follows. The authors take two images and two latents z_1 and z_2 that produced them. Then they identify a specific neuron under z_1 , then swap that neuron with the value of the neuron from z_2 with everything else remaining the same under z_1. Then they look at the difference in the images and average across all channels. Then they average over all pairs z_1 and z_2 - this representative "image with one channel" is the average effect of manipulating that neuron.
Then, using nonnegative matrix factorization on the thresholded versions of the average effects of all neurons in a layer, they cluster neurons and then they actually form modules. So now each module can be manipulated together when two images are taken together and neurons in the modules are swapped from one into the other.
Pros:
  I really like the key idea of this paper - counterfactually manipulating neurons using its values from that of some other image and observing differences and clustering them to find similar neurons which could be manipulated as a bunch. Some of the results (although seems handpicked) seem pretty good given this is the first work to group neurons inside a pre-trained generator. I have not seen any work on "intrinsic disentanglement" before. This also has additional implications for people interested in GANs. The fact that pre-trained GANs can lead to counterfactually "realistic" images when "concepts" (clustered neurons of this method)  are swapped shows that GANs really do learn something non-trivial about images.
I highly recommend this paper for acceptance. However I would like the authors to address some of my concerns.
Cons:
a)  Typo in Proposition 2 - "then and transformation.." - and should not be there.
b) v_0 is used in some cases - h(z) is used in some places and h (being a constant vector) is used in some places to define counterfactual mapping. Its pretty confusing to read some of the proofs. Is it possible to uniformly define it throughout with constant h and then of course only during experiments and hybridization replace h by v(z_2) (the v from image to be swapped with).
c) Section 3.4 - first paragraph - last line. "Influence maps are grouped by similarity to define modules.." - This is vague. Does this refer to the procedure in the third paragraph of the same section? If so this line could be make to point to this more precise description that comes later.
d) So there are many synthetic experiments where the same pair of images is used to produce many counterfactual images by hybridizing under manipulation of different modules (returned by the clustering algorithm)
What will be useful is for one VAE or GAN ,demonstrate the changes on multiple pairs of images. It seems the image pair is handpicked along with the clustered modules. It would be good to fix the modules and illustrate changes for multiple images using the same set of modules.
e) It seems that the clustered influence maps are simplistic - representing face region, hair region and background. However, ideally we would like to find out - makeup or not, gender , hair color, bangs or not - these are various attributes of the celebA dataset that u get annotated.
Is it possible to correlate obtained modules/influence maps with these annotations already available to find finer concepts that are represented by these annotations ?
One way to find out is - take a module or a neuron - manipulate it by hybridizing and produce a hybrid image from a pair of original images, build a classifier for one of these annotated labels (attributes) on the CelebA dataset .Then do inference on the hybridized image versus the original two images - if the prediction confidence of the attribute classifier changed across original images and also between the hybrid and one of the original- we know that module/neuron is highly correlated with this attribute. One could build a confusion matrix (sort of) between annotated attributes and all elementary (neurons or modules). This could easily tell us if indeed modules/neurons are capturing all attributes we would like.


Auto-regressive sequence-to-sequence models with attention mechanism have achieved state-of-the-art performance in many tasks such as machine translation and speech synthesis. These models can be difficult to train. The standard approach, teacher forcing, guides a model with reference output history during training. The problem is that the model is unlikely to recover from its mistakes during inference, where the reference output is replaced by generated output. Several approaches deal with this problem, largely by guiding the model with generated output history. To make training stable, these approaches often require a heuristic schedule or an auxiliary classifier. This paper introduces attention forcing, which guides the model with generated output history and reference attention. This approach can train the model to recover from its mistakes, in a stable fashion, without the need for a schedule or a classifier. In addition, it allows the model to generate output sequences aligned with the references, which can be important for cascaded systems like many speech synthesis systems. Experiments on speech synthesis show that attention forcing yields significant performance gain. Experiments on machine translation show that for tasks where various re-orderings of the output are valid, guiding the model with generated output history is challenging, while guiding the model with reference attention is beneficial.

This paper proposes an alternative mechanism of training the attention values of a sequence to sequence learning model as applied to tasks like speech synthesis and translation.  During training they compute two forms of attention: (1) the standard soft-attention from a decoder fed with teacher forced output, and (2) the inference-time attention from a decoder fed with predicted outputs.  Their training objective consists of two terms: The first is the token-wise cross entropy loss but by conditioning on the predicted output  but with teacher-forced attention.  The second is a KL distance between the above two types of attention distributions.   Experiments with mechanical  turks indicate that their attention forcing mechanism is strongly preferred over the existing teacher forced output and attention model.  On translation their method provides little or no improvement.
I am inclined towards rejecting the paper because the experiment and related work section still requires a lot of work before 1. The claimed utility of the idea is established, and 2. The novelty over the many existing attention architectures is established.   I elaborate on each of these next.
Related work: Recently, many papers have directly or indirectly handled the problem of exposure bias that this paper attempts to address.  The paper does not discuss most of these.  Here are some that are missed from the paper:
1.   Sequence level training with recurrent neural networks
MA Ranzato, S Chopra, M Auli, W Zaremba, 2015.
This paper shows that the scheduled sampling method (discussed in the paper) is much worse than a reinforce-based training mechanism of handling exposure bias.
2. An actor-critic algorithm for sequence prediction
D Bahdanau, P Brakel, K Xu, A Goyal, R Lowe
3.  Posterior Attention Models for Sequence to Sequence Learning
S Shankar, S Sarawagi - 2019
4. Latent Alignment and Variational Attention
Yuntian Deng, Yoon Kim, Justin Chiu, Demi Guo, Alexander M. Rush 2018
5. Attention Focusing for Neural Machine Translation by Bridging Source and Target Embeddings
Shaohui Kuang, Junhui Li, António Branco, Weihua Luo, Deyi Xiong
Experiments:  Their experiments are rather sketchy and limited.
The TTS experiments are only on one dataset.  Their method is compared only with the standard seq2seq learning approach.  Even the scheduled sampling or professor forcing methods are not compared with.  In addition, state of the art TTS methods have gained significantly from hierarchical attention.  As such as far as the TTS task is concerned the significance of the improved quality over a baseline seq2seq method is limited.
For translation they consider only the English-Vietnamese task whereas there are tens of other translation tasks that are used in recent literature.
Overall, the idea proposed seems quite incremental, experiments are limited, and related work discussion incomplete.
*********
I read the author response but I do not think the paper is ready for publication yet without the thorough comparison with related work.


We propose a novel technique for improving the stochastic gradient descent (SGD) method to train deep networks, which we term PoweredSGD. The proposed PoweredSGD method simply raises the stochastic gradient to a certain power γ ∈ [0, 1] elementwise during iterations and introduces only one additional parameter, namely, the power exponent γ (when γ = 1, PoweredSGD reduces to SGD). We further propose PoweredSGD with momentum, which we term PoweredSGDM, and provide convergence rate analysis on both PoweredSGD and PoweredSGDM methods. Experiments are conducted on popular deep learning models and benchmark datasets. Empirical results show that the proposed PoweredSGD and PoweredSGDM obtain faster initial training speed than adaptive gradient methods, comparable generalization ability with SGD, and improved robustness to hyper-parameter selection and vanishing gradients. PoweredSGD is essentially a gradient modifier via a nonlinear transformation. As such, it is orthogonal and complementary to other techniques for accelerating gradient-based optimization.

This paper investigates an SGD variant (PowerSGD) where the stochastic gradient is raised to a power of $\gamma \in [0,1]$.  The authors introduce PowerSGD and PowerSGD with momentum (PowerSGDM). The theoretical proof of  the convergence is given and experimental results show that the proposed algorithm converges faster than some of the existing popular adaptive SGD techniques.  Intuitively, the proposed PowerSGD can boost the gradient (since $\gamma \in [0,1]$) so it may be helpful for the gradient of the lower layers of a deep network which may be hit by the vanishing gradient issue. This may give rise to a faster convergence.   So overall the idea makes sense but I have the following concerns.
1. The major issue I have with this paper is Theorem 3.1 on the ergodic convergence rate of the proposed PowerSGD.  At the first glance, it is $O(\frac{1}{T})$ which is faster than the conventional SGD convergence rate $O(\frac{1}{\sqrt{T}})$.  But after a closer look, this rate is obtained by a very strong assumption on the batch size $B_{t}=T$.  In other words, when the number of iterations is large, the batch size will be large too.  I consider this assumption unrealistic.  Given that $T$ is typically very large (it is iterations, not epochs),  it will require a huge batch size, probably close to the whole training set. In this case, it is basically a GD, not SGD any more. That's why the rate is $O(\frac{1}{T})$, which is the convergence rate of GD.   I would like to see a convergence proof where the batch size $B_{t}$ is treated as a small constant like other SGD proofs assume.  Actually in the experiments the authors never use an increasing batch size. Instead, a constant batch size 128 is used. Therefore,  the faster convergence demonstrated in the experiments can not be explained by Theorem 3.1 or Theorem 3.2.
2. There are numerous inaccuracies in the proof given the supplementary material.  For instance, in Eq.7,  $\nabla f(x) \sigma(\nabla f(x))$  should be $\nabla f(x)^{T} \sigma(\nabla f(x))$   The random variable $\xi_{t}$ should be a scalar on training samples, not a vector, etc..  The authors should clean it up.
3. It would be helpful to show the $\gamma$ value on each experiment with different tasks. It would be good to know how $\gamma$ varies across tasks.
4. I think in the comparative experiments, the plain SGD should be added as another reference algorithm.
5. The term "PowerSGD" seems to have been used by other papers.


Responding with knowledge has been recognized as an important capability for an intelligent conversational agent. Yet knowledge-grounded dialogues, as training data for learning such a response generation model, are difficult to obtain. Motivated by the challenge in practice, we consider knowledge-grounded dialogue generation under a natural assumption that only limited training examples are available. In such a low-resource setting, we devise a disentangled response decoder in order to isolate parameters that depend on knowledge-grounded dialogues from the entire generation model. By this means, the major part of the model can be learned from a large number of ungrounded dialogues and unstructured documents, while the remaining small parameters can be well fitted using the limited training examples. Evaluation results on two benchmarks indicate that with only 1/8 training data, our model can achieve the state-of-the-art performance and generalize well on out-of-domain knowledge.

This paper studies knowledge-grounded dialogue response generation in the low-resource setting. More precisely, it proposes a disentangled decoder consisting of three components: language model, context processor, and document reader. Disentangled decoder architecture provides a flexibility to train (or pre-train) different components on different data, making it convenient for low-resource setting. Overall, it is a sound idea for low-resource setting with generally positive experimental results, but limited in novelty in terms of the proposed architecture (similar to [1*, 2*]) and disentangling language and knowledge idea (similar to [3*]), lacks comparison with baselines for low-resource setting, and lacks discussion/reference of a few closely related works.
Here are some of my questions and concerns for the paper:
Great to have some ablations in terms of pre-training to see its effect on different components. However, it would be quite useful to also have ablations over components by completely dropping a component (like LM) while training decoding manager. It would also be useful to see some statistics/discussion on the effect of different components in the actual generation of responses. It might also be useful to include qualitative examples of the generated responses annotated with predictions of different components for each generated word.
In Figure-2 (c) and (d), some ablation results are reported for when pre-training is removed for each of the three components independently. It is interesting, though, to see that removing pre-training does not hurt (might even improve) the performance (esp. for Test Unseen) much for FULL training data case. Is there a particular reason for this observation? Also, it makes me curious how the proposed model would perform without pre-training any of the components. Would it already outperform the baselines discussed in the paper? If so, are these baselines strong enough (SOTA or close to SOTA) to help draw a meaningful conclusion from comparison with them? For example, how would fine-tuning a pre-trained MASS  [4*] perform and compare as a baseline? Can authors comment on this?
The proposed approach is very similar in architecture to [1*, 2*, 3*], which are not discussed/referenced in the paper. Except for pre-training, the only difference from [2] is that copying and generation distributions are softly combined into separate distribution each, independently. Inducing a single output distribution is done instead by deciding which source to use by an MLP layer on decoder state as in Eq. 11. So, I think the authors need to better isolate what the core contribution of this paper is: disentangled decoder or pre-training strategy? If it is the proposed disentangled decoder architecture, then authors should compare with similar architectures [2, 3] in the low-resource setting by initializing encoder and decoder from pre-trained weights on the same Reddit corpus. If it is the pre-training strategy, then it should be compared with various pre-training strategies for sequence generation (e.g., [4*]) proposed recently. For example, it would be useful to include a comparison with fine-tuning a pre-trained MASS  [4*] with the same amount of WoW training data (changing from 1/16 to 1/1).
Presentation of the paper can be improved by 1) changing the name of “document reader” (maybe to “knowledge processor” similar to context) as it essentially attends on the document rather than reading, 2) using abstraction in the technical section to help simplify notation and make it more interpretable.
REFERENCES:
[1*] Get To The Point: Summarization with Pointer-Generator Networks, See et al.
[2*] DeepCopy: Grounded Response Generation with Hierarchical Pointer Networks, Yavuz et al.
[3*] Disentangling Language and Knowledge in Task-Oriented Dialogs, Raghu et al.
[4*] MASS: Masked Sequence to Sequence Pre-training for Language Generation, Song et al.


The recently developed link between strongly overparametrized neural networks (NNs) and kernel methods has opened a new way to understand puzzling features of NNs, such as their convergence and generalization behaviors. In this paper, we make the bias of initialization on strongly overparametrized NNs under gradient descent explicit. We prove that fully-connected wide ReLU-NNs trained with squared loss are essentially a sum of two parts: The first is the minimum complexity solution of an interpolating kernel method, while the second contributes to the test error only and depends heavily on the initialization. This decomposition has two consequences: (a) the second part becomes negligible in the regime of small initialization variance, which allows us to transfer generalization bounds from minimum complexity interpolating kernel methods to NNs; (b) in the opposite regime, the test error of wide NNs increases significantly with the initialization variance, while still interpolating the training data perfectly. Our work shows that -contrary to common belief -the initialization scheme has a strong effect on generalization performance, providing a novel criterion to identify good initialization strategies.

This paper studies overparameterized fully-connected neural networks trained with squared loss. The authors show that the resulting network can be decomposed as a sum of the solution of a certain interpolating kernel regression and a term that only depends on initialization. Based on this, the authors also derive a generalization bound of deep neural networks by transferring it to a kernel method. My major concern about this paper is the novelty and significance of its results:
In terms of connection to NTK, It seems that the connection between neural networks trained with squared loss and the result of NTK-based kernel regression has already been well-studied by
Arora, Sanjeev, Simon S. Du, Wei Hu, Zhiyuan Li, Ruslan Salakhutdinov, and Ruosong Wang. "On exact computation with an infinitely wide neural net." arXiv preprint arXiv:1904.11955 (2019).
which is a missed citation. Without a clear explanation on the difference between the submission and this paper above, I don’t think this paper is ready for publication.
In terms of generalization, it is also very difficult to judge whether this paper's result is novel. In fact this paper misses almost all citations on generalization bounds for neural networks. Moreover, the generalization bound given in this paper does not seem to be very complete and significant, since the authors do not show when can L_{test}^{int} be small. To demonstrate the novelty and significance of the result, the authors should at least compare their generalization result with the following generalization bounds for over-parameterized neural networks in Section 4:
Allen-Zhu, Zeyuan, Yuanzhi Li, and Yingyu Liang. "Learning and generalization in overparameterized neural networks, going beyond two layers." arXiv preprint arXiv:1811.04918 (2018).
Cao, Yuan, and Quanquan Gu. "A generalization theory of gradient descent for learning over-parameterized deep relu networks." arXiv preprint arXiv:1902.01384 (2019).
Arora, Sanjeev, Simon S. Du, Wei Hu, Zhiyuan Li, and Ruosong Wang. "Fine-grained analysis of optimization and generalization for overparameterized two-layer neural networks." arXiv preprint arXiv:1901.08584 (2019).
Cao, Yuan, and Quanquan Gu. "Generalization Bounds of Stochastic Gradient Descent for Wide and Deep Neural Networks." arXiv preprint arXiv:1905.13210 (2019).
Overall, I suggest that the authors should make a clear discussion on the relation of this paper to many existing works mentioned above. As long as the authors can give a convincing demonstration of the novelty and significance of their results, I will be happy to increase my score.
A minor comment: how can the bound in Theorem 3 be derived based on Theorem 2? Should there be a constant factor in the bound?


We demonstrate how machine learning is able to model experiments in quantum physics. Quantum entanglement is a cornerstone for upcoming quantum technologies such as quantum computation and quantum cryptography. Of particular interest are complex quantum states with more than two particles and a large number of entangled quantum levels. Given such a multiparticle high-dimensional quantum state, it is usually impossible to reconstruct an experimental setup that produces it. To search for interesting experiments, one thus has to randomly create millions of setups on a computer and calculate the respective output states. In this work, we show that machine learning models can provide significant improvement over random search. We demonstrate that a long short-term memory (LSTM) neural network can successfully learn to model quantum experiments by correctly predicting output state characteristics for given setups without the necessity of computing the states themselves. This approach not only allows for faster search but is also an essential step towards automated design of multiparticle high-dimensional quantum experiments using generative machine learning models.

This paper looks at the problem of predicting 2 properties of quantum states produced by an optical table consisting of a sequence of physical elements that modify a quantum state in a particular way, given the sequence of optical elements applied. The authors train 2 separate recurrent networks (LSTMs). The first one takes a sequence of optical elements and predicts a binary boolean label that answers the question of whether the resulting quantum state would be maximally entangled. The second one takes a sequence of optical elements and predicts the Schmid rank vector of the resulting state -- a linear algebraic quantity that is used in quantum information science. The authors explore whether the trained networks generalize to longer sequences of optical elements than trained on, and conduct other checks on the result.
I really appreciate that the authors checked the applicability to sequences of elements longer than trained on. Another great point is the separation of the problem into the two LSTM, so that spurious correlations between the Schmid rank vector and entanglement cannot be used by the NN.
While overall I find the problem interesting and the authors’ approach reasonable, I have a large number of questions / points of confusion that I will detail below.
--------------- Point 1 ----------------
It is unclear what is actually fed into the optical table. My interpretation of the paper is that the input state to the optical table *must be fixed*, since it is not an input to the LSTM. This severely limits the applicability of the framework presented.
I was unable to find what quantum state is actually being input into the sequence of optical elements whose effect you are trying to model. Considering that the properties of the quantum state /after/ the application of the set of N optical elements depend very strongly on the *input quantum state*, I find this omission significant. For an input state rho_0, each element L (which in this particular case is the input to the LSTM), will transform the state as rho -> L rho -> rho’. A series of N elements can be viewed as an application of a sequence of N operations L_1, …. , L_N. Therefore the resulting state is rho_final = L_N L_{N-1} …. L_1 rho_0. As it is clear here, the output state rho_final depends *both* on a) the sequence of applied operations / optical elements (L1,L2, …, LN) as modeled in this paper, and b) the actual input state rho_0. Since what is actually fed into the LSTM is only the sequence of optical elements  (L1,L2, …, LN), while the loss depends on the properties of rho_final, I believe rho_0 must have been assumed fixed and constant throughout the whole paper (i.e. that every data point in the train and test set actually had the same rho_0). If that is the case, the problem that the paper addresses is not /predicting the properties of a quantum state produced by a particular sequence of optical elements/, but rather /predicting the properties of a quantum state produced by a particular sequence of optical elements *given a fixed, constant input state, and not other input state*/. That is a much more restricted regime. I think authors should be clearer about the actual problem their paper is addressing.
--------------- Point 2 ----------------
Lack of baselines. The problem setup and metrics used are quite specific (understandably) to the problem at hand. However, I do not have intuitions for what a “baseline” performance should be and therefore cannot validate whether the approach presented improves in it. I would like to see something equivalent to the 10% chance result on a 10-class classification task. To be more precise, while the binary classification of the maximally entangled nature of the quantum state is clear, the L_2 distance on the sorted 3-tuple of the Schmid rank vector is less so. The explicitly encoded sorted nature of the output 3-tuple (it is parameterized as such) makes it hard for me to make sense of the L_2 distance threshold r that the authors use (which is in fact at some sections r=3, i.e. quite large). I think that adding a baseline performance of some sort would greatly improve my ability to see how much better the LSTM approach performs.
--------------- Point 3 ----------------
The datatype of the LSTM inputs is unclear to me. As I discussed in point 1), the resulting quantum properties of the output state after the application of N optical elements depends on the input state as well as the sequence of elements applied. I am not clear on how the authors encode the optical elements themselves -- i.e. what datatype are the inputs x1, … , xN to the LSTM. If they model them categorically as one-hot vectors, i.e. a 0 degree orientation polarization plate as (1,0,0,0,...0), a beam splitter as (0,1,0,0,....,0), a 90 degrees polarization place as (0,0,1,0,....,0) etc, they are restricting the applicability of the trained models to that particular set of optical elements. For example, if I wanted to know what adding a polarization plate with orientation alpha will do, I would not be able to do that. If it indeed is true that the inputs are drawn from a small number of possible optical elements that has to be specified prior to training, this would again limit the scope of applicability of the results presented.
--------------- Point 4 ----------------
States as state vectors or density matrices? It wasn’t clear to me what the actual state that is being fed into the optical table is. This might be a minor point, but Equation 1 suggests it is in fact a state vector, while I thought quantum optical experiments work with density matrices. This is important for the dimensionality considerations involved, since e.g. for a set of M qubits, the state vector would have 2^M elements (-constraints), while the density matrix would have 2^(2M) elements (-constraints). Solving the latter problem would therefore be more impressive than the former.
--------------- Point 5 ----------------
Are you always working with 3 subsystems/photons? From reading the paper I didn’t understand whether you were restricting your setup to always work in the 3 photon regime that you mentioned in Equation 1. What is the range of the values that can appear in the Schmid rank vector? If they are small, then setting the threshold r=3 would be very generous and making it easy for the LSTM.
--------------- Conclusion ----------------
While I like the problem and the attempted solution, the results presented (as I understood them) are more restricted and therefore weaker than I originally expected. I believe the results are promising, but more clarity and more work on generalizing beyond the restricted regime presented would greatly improve the paper.