Differently from the popular Deep Q-Network (DQN) learning, Alternating Qlearning (AltQ) does not fully fit a target Q-function at each iteration, and is generally known to be unstable and inefficient. Limited applications of AltQ mostly rely on substantially altering the algorithm architecture in order to improve its performance. Although Adam appears to be a natural solution, its performance in AltQ has rarely been studied before. In this paper, we first provide a solid exploration on how well AltQ performs with Adam. We then take a further step to improve the implementation by adopting the technique of parameter restart. More specifically, the proposed algorithms are tested on a batch of Atari 2600 games and exhibit superior performance than the DQN learning method. The convergence rate of the slightly modified version of the proposed algorithms is characterized under the linear function approximation. To the best of our knowledge, this is the first theoretical study on the Adam-type algorithms in Q-learning. Q-learning (Watkins & Dayan, 1992) is one of the most important model-free reinforcement learning (RL) problems, which has received considerable research attention in recent years (Bertsekas & Tsitsiklis, 1996;Even-Dar & Mansour, 2003;Hasselt, 2010; Lu et al., 2018;Achiam et al., 2019). When the state-action space is large or continuous, parametric approximation of the Q-function is often necessary. One remarkable success of parametric Q-learning in practice is its combination with deep learning, known as the Deep Q-Network (DQN) learning (Mnih et al., 2013;2015). It has been applied to various applications in computer games (Bhatti et al., 2016), traffic control (Arel et al., 2010), recommendation systems (Zheng et al., 2018;Zhao et al., 2018), chemistry research (Zhou et al., 2017), etc. Its on-policy continuous variant (Silver et al., 2014)  has also led to great achievements in robotics locomotion (Lillicrap et al., 2016).

This paper is well-written and it provides a convergence result for traditional Q-learning, with linear function approximation, when using an Adam-like update (AMSGrad). It does the same for a variation of this algorithm where the momentum-like term is reset every now and then. This second result is not that exciting as it ends up concluding that the “best way” to converge with such an approach is by resetting the momentum-like term rarely. That being said, it is still interesting to have such theoretical result. On the empirical side, this paper evaluates the traditional Q-learning algorithm with non-linear function approximation (through a neural network), using Adam (and AdamR) while not using a target network, in both an LQR problem and a subset of the Atari games. The empirical results are not necessarily that convincing and there are important details missing. I’m willing to increase my score if my concerns w.r.t. the empirical validation are addressed since this paper presents a potentially interesting theoretical result with Adam, which is so prominent in the literature nowadays.
With respect to the empirical analysis, it is not clear to me why only 23 Atari games were used. The traditional answer often revolves around limited computational resources, but for this set of experiments it is particularly concerning. *How were these 23 games chosen?* The reason I ask is that DQN, without a target network, does work in approximately half of the Atari games. In the other games DQN presents instability and it doesn’t succeed. I wonder if the proposed approach would actually be stable throughout the whole set of Atari games. Moreover, it is not clear to me *why such a small buffer size was used*. As acknowledged in the paper: “Considering we use a smaller buffer size than common practice, DQN is not consistently showing improved return over all tested games”. It seems to me that DQN was evaluated in a suboptimal parameter setting, so it is not clear to me how relevant the claims that the evaluated algorithm outperforms DQN are.
The results itself, which are summarized in Figure 2 and in the Appendix, in Figure 3, are not that convincing (I would also like to see a table with the raw numbers at the end, as it is often done). In Figure 2, as acknowledged in the paper, “AltQ-Adam” has a huge variance at the end (raising concerns about its stability, as I previously mentioned). “AltQ-AdamR” does seem to be more stable but the confidence intervals (or standard deviation?) overlap (red and blue). Is there a significant difference? Looking at Figure 3 in the Appendix, the individual learning curves are not that convincing as well. In 12 out of 23 games DQN outperforms the other methods or there is pretty much no difference in performance between them (i.e., Alien, Assault, Asteroids, Bowling, CrazyClimber, FishingDerby, JamesBond, Pitfall, Pong, QBert, Robotank, Tutankham). In 11 out of 23 games there’s a clear advantage of AltQ-Adam or AltQ-AdamR (i.e., Amidar, Asterix, BeamRider, Enduro, DemonAttack(?), DoubleDunk, Gopher, Gravitar, Seaquest, SpaceInvaders, Tennis). Consequently, although AltQ seems better than DQN, the difference is not that big. Importantly, some crucial details are missing (at least I couldn’t find them). *How many seeds were used in each game?* One seed is way too little since the performance difference in Tennis could be explained by “luck” in the random action selection process. Which Atari version was used? The deterministic one or the stochastic one (Machado et al., 2018)?
With respect to presentation, I don’t understand why Q-learning is being called AltQ. AltQ, with linear function approximation, is exactly what Q-learning does. *Introducing a new name for an old algorithm is very distracting* and it actually might reduce the impact of the paper, since more people can relate to Q-learning than to AltQ, which sounds as a newly introduced algorithm. Some details are not clear in Algorithms 1 and 2. *Is there a minibatch? That’s not represented in these algorithms, it it? Would it be fair to say that AltQ is _online_ Q-learning? I don’t think so since it uses a minibatch uniformly sampled from the experience replay buffer. It would be beneficial to have a paragraph explicitly discussing what are the differences between AltQ and DQN. Is it only dropping the target network?* Finally, there are some typos (e.g., “Nestrov”) and the references could be improved. Specifically, I’d recommend the authors to not use citations as nouns (e.g., “which is justified in (Duchi et al., 2011).”) and to cite Bellemare et al. (2013) instead of (or with) Brockman et al. (2016) when referring to the Atari games.
References:
Marc G. Bellemare, Yavar Naddaf, Joel Veness, Michael Bowling: The Arcade Learning Environment: An Evaluation Platform for General Agents. J. Artif. Intell. Res. 47: 253-279 (2013)
Marlos C. Machado, Marc G. Bellemare, Erik Talvitie, Joel Veness, Matthew J. Hausknecht, Michael Bowling: Revisiting the Arcade Learning Environment: Evaluation Protocols and Open Problems for General Agents. J. Artif. Intell. Res. 61: 523-562 (2018)
---
>>> Update after rebuttal: I stand by my score after the rebuttal.
This paper has several presentation issues that need to be addressed before publication. The theoretical result does make it an interesting contribution, but the empirical section weakens the paper quite a lot. The paper is trying to show that the proposed approach is more stable but the smaller replay buffer size is not satisfying, since it is a setting which no one uses. Moreover, it is not possible to claim that a method presents lower variance when all one has are to 2 seeds to back up those claims. I don't think "limited computational capability and cost" is a good justification in that case.
Finally, the term AltQ is distracting and misleading. The authors didn't demonstrate any intention of changing this name, but if they decide to submit this paper to a next conference, with stronger empirical evidence to back up the paper claims, I recommend them to drop the AltQ learning name as well.


Semantic sentence embedding models encode natural language sentences into vectors, such that closeness in embedding space indicates closeness in the semantics between the sentences. Bilingual data offers a useful signal for learning such embeddings: properties shared by both sentences in a translation pair are likely semantic, while divergent properties are likely stylistic or language-specific. We propose a deep latent variable model that attempts to perform source separation on parallel sentences, isolating what they have in common in a latent semantic vector, and explaining what is left over with language-specific latent vectors. Our proposed approach differs from past work on semantic sentence encoding in two ways. First, by using a variational probabilistic framework, we introduce priors that encourage source separation, and can use our model's posterior to predict sentence embeddings for monolingual data at test time. Second, we use highcapacity transformers as both data generating distributions and inference networks -contrasting with most past work on sentence embeddings. In experiments, our approach substantially outperforms the state-of-the-art on a standard suite of unsupervised semantic similarity evaluations. Further, we demonstrate that our approach yields the largest gains on more difficult subsets of these evaluations where simple word overlap is not a good indicator of similarity.

This paper addresses the problem of constructing a sentence embedding using a generative transformer model which encodes semantic aspects and language-specific aspect separately. They use transformers to encode and decode sentence embedding, and the objective reconstructs input with a latent variables (language variables for each language and semantic language).  These latent variables are sampled from multivariate Gaussian prior, and the learning uses evidence lower bound (ELBO) for variational approximation of the joint distribution of latent variables and input.
The method is evaluated on two tasks: sentence similarity task and machine translation evaluation metric tasks. Both tasks evaluates how similar are two sequences, and the metric is correlation score with score’s from human judge. The model shows promising results on the first task, but weaker results on the second task, especially when compared against pretty naively built sentence embedding from BERT model. I’m not expert in sentence embedding literature, so a bit tricky to evaluate, but baselines seem strong and experimental results on semantic textual similarity task.
In terms of evaluation, I appreciated how they defined harder subset of the evaluation dataset and showed a larger improvements on those portions of the dataset. The The paper also includes analysis on what is captured by their language-specific latent vector and semantic latent vector. While I’m not totally convinced this distinction between language-specific characteristics and semantics of the sentence, it makes it easier to understand what’s going on in the model.
One of my question is, why not test this method in more popular benchmark such as MNLI or other classification tasks? MNLI evaluates how each sentence pair relates to one another, thus would be a good benchmark for sentence embeddings as well. Having to encode all the information about a sentence into a single vector will make these sentence embedding model weaker than other models which can do cross sentence attentions and etc, but I think that’s the genuine limitation of sentence embedding research and has to be clarified as such. I recommend discussing and clarifying these points.
I’m a bit unclear how these sentence embeddings are translated into a score that decides the degree to which sentences have the same meaning. Is it just cosine similarity of two sentence embedding vectors?
While the purpose of these references is to generate sentences instead of building a sentence embedding, the method is related and comparison and discussion would be worthwhile.
Generating Sentences from a Continuous Space
Samuel R. Bowman, Luke Vilnis, Oriol Vinyals, Andrew M. Dai, Rafal Jozefowicz, Samy Bengio
https://arxiv.org/abs/1511.06349
Toward Controlled Generation of Text
Zhiting Hu, Zichao Yang, Xiaodan Liang, Ruslan Salakhutdinov, Eric P. Xing
https://arxiv.org/abs/1703.00955
Comments & Questions:
- Methods using a large amount of unsupervised monolingual data shows very strong performance in a panoply of NLP tasks these days. If I understand correctly, this model is constrained by the amount of bitext — some analysis on this would be interesting.
- Figure 1 mentions about “Section 3, 4” but I don’t think they are correct references?
- BERT baseline seemed not to allow fine-tuning of the LM parameters. I think this makes the baseline significantly weaker?
- It seems odd that only English semantic encoder is used to downstream application.
- Does table 3 covers all the data? What proportion of the data is covered by each row?
- Given the similarity of English and French, I’m not sure how “language-specific” such latent vectors are. It would be much more interesting analysis if it studies distant language pairs.


To collect large scale annotated data, it is inevitable to introduce label noise, i.e., incorrect class labels. A major challenge is to develop robust deep learning models that achieve high test performance despite training set label noise. We introduce a novel approach that directly cleans labels in order to train a high quality model. Our method leverages statistical principles to correct data labels and has a theoretical guarantee of the correctness. In particular, we use a likelihood ratio test (LRT) to flip the labels of training data. We prove that our LRT label correction algorithm is guaranteed to flip the label so it is consistent with the true Bayesian optimal classifier with high probability. We incorporate our label correction algorithm into the training of deep neural networks and train models that achieve superior testing performance on multiple public datasets.

This paper proposes a new method for correcting training label noise, using a likelihood ratio test on the predicted probability of the classifier trained on the noisy labels, and then proposes an algorithm for iteratively cleaning training labels and retraining a model.  The paper also provides a theoretical guarantee for the probability of correctly re-labeling the training set, and provides empirical results showing that the proposed method significantly outperforms existing approaches for handling noisy training labels.
Overall, I think the empirical results appear very strong, but think this paper is below the acceptance threshold due to three factors, in ranked order:
- (1) The theoretical guarantee, which is positioned as a core contribution of the paper (and in fact claims it as "the first to correct labels with theoretical guarantees", which is not true), is based on assumptions that seem overly strong; these are somewhat relaxed in a "Remark", but this seems unproven and is a confusing presentation regardless.
- (2) The theoretical bound itself is somewhat vacuous as it contains several constant factors that seem very material to the bound, but totally opaque to the reader.
- (3) The experiments are very strong overall, which is a major plus for the paper; however, there are some questions about the hyperparameter tuning and some other points where more clarity could improve the strength of the empirical results
Regarding (1):
As a reader, my first natural reaction was to worry about circularity/degeneracy in the proposed method: basically, we are using the confidence (as a ratio of predicted cond. probabilities) of the model trained on the corrupted labels to correct those labels... if the labels are so corrupted that the model is also way off, then intuitively, this method should not work.  I was wondering about how this situation would be bounded / handled.
It turns out in Thm 1 that an incredibly strong assumption is made, namely that the model trained on the corrupted labels, f, is a linear function of the true model, with constants a, b known to some small degree of error epsilon (note that the theorem statement says that these constants are unknown- but it then assumes that \Delta, which is set based on a and b, is known up to \epsilon error).  This seems like an incredibly strong assumption- and no context / motivation is given about why it should be taken as reasonable.
Then, immediately after the Theorem, "Remark 2" states that this condition is not actually needed at all- but (a) then why not just strike it from Thm 1 statement, and (b) there does not seem to be any proof of this Remark in the appendix (where the proof of the main theorem itself is closer to a sketch than a standard proof...).
Regarding (2):
The bound produced in Thm 1 seems somewhat vacuous: letting \tau_{01} = \tau_{10}, then the probability of the label correction being erroneous is bounded by 8C(O(\epsilon))^\lambda.  This quantity is presumably in (0,1/2], so it's a small range to start... but it seems hard to get anything from this bound without some idea of what the constant factors (C, and those hidden in O(\epsilon)) and \lambda are.  In particular, as presented, it seems implausible that \epsilon- the error in specifying the \Delta threshold- gets that small, in which case these constants become very important to know!  Another way of phrasing this remark: many theoretical bounds have lots of unknown constant factors, but are ultimately just trying to expose some scaling with respect to one parameter, e.g. number of data points, and therefore the constants don't need to be known that well for the statement to have some value.  This doesn't exactly seem to be the case here- therefore it seems hard to extract something from this statement (even ignoring the strong assumptions it is predicated on).
Regarding (3):
Overall, I think the empirical performance reported in Table 2, and overall thoroughness of the ablation in Section 4, are major strong points for the paper- the performance is very impressive!  However I have a few questions, clarification of which would be very helpful in my mind:
- (i) A major issue that seems to be raised in the earlier sections is that there is a hyperparameter \Delta--the threshold for the likelihood ratio test--that everything depends on, and must be chosen empirically.  Table 5 shows that the effect of choosing it is not crazy, but also clearly not insignificant.  My question is: how is it chosen?  On the validation dataset?  And is this validation dataset also corrupted in the same way as the training dataset?  If not, that seems like a major whole in the setup.
- (ii) I also have a high level question for understanding: how is it possible for the various approaches to do so well with 0.6 and 0.8 noise level of uniform flipping?  In the p=0.8 noise model, for example, the probability of a data point getting flipped to *any individual wrong label* is *greater than that of it being the correct label*.  How is it possible to learn a model based on such a dataset?  Was there some kind of pre-training?  Was the validation set not corrupted?  I don't conceptually understand how the results shown are possible...?
Overall, I think points in (1) and (3) could be helped with additional clarification and contextualization, and possibly (2) as well.


With the success of modern machine learning, it is becoming increasingly important to understand and control how learning algorithms interact. Unfortunately, negative results from game theory show there is little hope of understanding or controlling general n-player games. We therefore introduce smooth markets (SM-games), a class of n-player games with pairwise zero sum interactions. SM-games codify a common design pattern in machine learning that includes (some) GANs, adversarial training, and other recent algorithms. We show that SM-games are amenable to analysis and optimization using first-order methods.
"I began to see legibility as a central problem in modern statecraft. The premodern state was, in many respects, partially blind [. . .] It lacked anything like a detailed 'map' of its terrain and its people. It lacked, for the most part, a measure, a metric that would allow it to 'translate' what it knew into a common standard necessary for a synoptic view. As a result, its interventions were often crude and self-defeating." -from Seeing like a State by Scott (1999)

This paper draws from concepts and patterns of game theory and economics to re-interpret common machine learning algorithms, and introduces a paradigm of n-player games such that there exists a simple way to study such games where each player is individually optimized by machine learning algorithms.
Overall, this is a very interesting and inspiring paper with its interdisciplinary touch, but at the same time doesn’t lose readability for audience with mostly machine learning background. The paper is clear, concise, and well-written, and hence I do not have any overarching comments. There are several suggestions and questions, though, that I’d like to propose.
1. Since many concepts are not from the machine learning domain, further and more detailed touch on related work are very beneficial. The current section of related work (Section 1.2) is more succinct than laying the right background information. Authors could consider giving some examples of “market mechanism”, and similarly for “design pattern”. This can give readers some ideas on how this present work’s use of “existing design pattern” is different from prior works’ proposals of “market mechanism”. Again for the second paragraph, although these concepts are more familiar to the audience, a short description for each mentioned concept is good to include (e.g. monotone games, etc)
2. Move Lemma 1 before Definition 2? Since Lemma 1 is most related to Definition 1 and not related to Definition 2.
3. For Definition 2, authors could consider adding “the classic definition [of Nash equilibrium]” to the list to clarify the difference. This classic Nash equilibrium definition will be referred to again in later sections.
4. In Section 2.1, consider giving a formal definition of “potential game”.
5. In Section 2.3, it is not immediately clear after the text why stop_gradient becomes a problem.
6. In Section 3.1, could we elucidate a bit on what is “near zero sum”?
7. In Section 5, why the equation following the text “firm i’s forecast” is always positive? Since by definition, forecast is meant to represent the change of profit, which can be negative?


Transfer reinforcement learning (RL) aims at improving learning efficiency of an agent by exploiting knowledge from other source agents trained on relevant tasks. However, it remains challenging to transfer knowledge between different environmental dynamics without having access to the source environments. In this work, we explore a new challenge in transfer RL, where only a set of source policies collected under unknown diverse dynamics is available for learning a target task efficiently. To address this problem, the proposed approach, MULTI-source POLicy AggRegation (MULTIPOLAR), comprises two key techniques. We learn to aggregate the actions provided by the source policies adaptively to maximize the target task performance. Meanwhile, we learn an auxiliary network that predicts residuals around the aggregated actions, which ensures the target policy's expressiveness even when some of the source policies perform poorly. We demonstrated the effectiveness of MULTIPOLAR through an extensive experimental evaluation across six simulated environments ranging from classic control problems to challenging robotics simulations, under both continuous and discrete action spaces.

The paper introduces a novel multi-source policy transfer problem, where we want to utilize policies from multiple source domains with different dynamics to improve the performance of the policy on our target dynamics.
The paper addresses the problem by adaptively aggregating the deterministic actions produced by source policies to maximize the expected return in the target environment. The method further trains an auxiliary network to predict a residual to revise the predicted action when some source policies are not useful or even adversarial.
 In my understanding, the paper assumes that the source and target domain shared the same task (reward structure) but only differs in dynamics. Also, the two domains share similar state and action spaces since the policy accepts the target states and predict actions in the target environment. This may limit the usage of the method.
The paper proposes to use residual learning as an auxiliary to compensate for the sub-optimal expressiveness of the source policies, which is novel and interesting.
The paper performs experiments on multiple environments. But the source and target domains only vary in some parameters of the agents. The domain gap seems small for these experiments. The paper needs a metric to measure the domain gap between source and target dynamics and report how the domain gap influences the proposed method and the baselines according to the metric.
One of my major concerns is that the limitation of the method with the same state and action space of the source and target domains. Also, there is no theoretical or intuitive analysis of how large the domain gap can be. This problem can be impractical for real-world applications with restrict limitations.
Post-rebuttal:
My major concern is that the method is a naive combination of previous works and the paper is more like an engineering work. The method is also a weighted sum of source policies. There is no insight why the combination can work.
No assumption on source policies is given. That means I can get any random policy to learn a combination. This is like learning a policy from scratch by reinforcement learning. With better source policies, we can achieve better initialization for RL.
The paper is also related to hierarchical reinforcement learning, where the target reinforcement learning step is like building a high-level policy.
The work still requires lots of steps to train in the target domain, which does not fit to the real application of transfer RL. We hope transfer learning can adapt to the target environment fast.


Word embedding plays a key role in various tasks of natural language processing. is carried with the resulting embeddings. To generate interpretable word embed-3 dings we intend to replace the word vector with a probability density distribution.

The paper addresses the problem the problem in word embeddings where "word-word" or "context-context" inner products are relied upon in practice when the embeddings are usually only optimized for good properties of "word-context" inner products. The new approach to training word embeddings addresses interpretability in the sense of having a theoretically grounded interpretation for the good properties that the inner products should possess -- namely that they capture the PMI between words. By changing the embedding training procedure (to include a step that samples vectors from each word's distribution representation) this interpretation is made possible without problematic theoretical assumptions. Now trained for the purpose they are intended to be used for, the approach yields strong SOTA results on the non-small challenge datasets.
The word is well exceptionally well motivated (we should directly optimize for the properties we want!) and situated in the literature. The opposition of Arora 2016 (getting at "interpretable" word embeddings with a latent variable model) and Mimno & Thompson 2017 (problematic assumptions in skip-gram embeddings) is particularly convincing. The vMF distribution is underexamined as tool for analyzing high-dimensional semantic vector distributions, and it is an excellent fit for the purposes of this project.
To back up claims of theoretical interpretability, a derivation (building on Ma 2017) proceeds without problematic leap thanks to a property introduced by a carefully selected (if straightforward) regularization term. This reviewer did not read the appendix (not sure where this would be located -- first time reviewing with OpenReview here), but the intuition doesn't seem to be much different than Ma.
To back up claims of applied performance, appropriate experiments are conducted on multiple evaluation datasets. The results seem to be fairly interpreted.
This reviewer decides to accept this paper for its balance of theoretical and empirical contributions and for the role it might play in reducing dependence on mysticisms in word embeddings (relying on accidental / uncharacterized properties of previous embedding strategies).
Suggestions for improvement:
- Run a spell checker. It will catch a large number of problems that weren't ever big enough to hurt my appreciation of the paper.
- Consider the creation of an adversarial dataset (of ~3 words in ~3 contexts) where techniques that optimize for the wrong thing will catastrophically fail but the proposed approach succeeds.
- Write one or two more sentences about the fate of \kappa_c -- is it ever updated or am I just missing something? Making a bigger point about leaving \kappa un-optimized shows there is room for additional depth in this line of thinking. (If you start learning concentration parameters, check out the Kent distribution for an even more expressive distribution on the unit hypersphere: https://en.wikipedia.org/wiki/Kent_distribution)
- Figure out what happened to your appendix.
- Watch out that readers/reviewers of similar work may try to read the word "interpretable" in the title as a reference to the much broader topic of interpretability in ML related to explaining a model's behavior. Is there a synonym for "interpretable" that won't raise this false link?


Neural network robustness has recently been highlighted by the existence of adversarial examples. Many previous works show that the learned networks do not perform well on perturbed test data, and significantly more labeled data is required to achieve adversarially robust generalization. In this paper, we theoretically and empirically show that with just more unlabeled data, we can learn a model with better adversarially robust generalization. The key insight of our results is based on a risk decomposition theorem, in which the expected robust risk is separated into two parts: the stability part which measures the prediction stability in the presence of perturbations, and the accuracy part which evaluates the standard classification accuracy. As the stability part does not depend on any label information, we can optimize this part using unlabeled data. We further prove that for a specific Gaussian mixture problem illustrated by Schmidt et al. (2018), adversarially robust generalization can be almost as easy as the standard generalization in supervised learning if a sufficiently large amount of unlabeled data is provided. Inspired by the theoretical findings, we further show that a practical adversarial training algorithm that leverages unlabeled data can improve adversarial robust generalization on MNIST and Cifar-10.

Paper summary: This paper seeks to improve robust generalization performance with the help of unlabeled data. The authors first consider the toy model presented in Schmidt et al. and show how the labeled sample complexity in the robust setting can be lowered to match the standard setting if sufficient unlabeled data is available. They then propose a practical algorithm to improve robust test accuracy and evaluate it on the MNIST and CIFAR datasets.
Comments: The problem the paper seeks to address (bridging the generalization gap in the adversarial setting) is an important one, and the paper is clear and well-written.
As the authors discuss, there have been three (other) independent papers that tackle the same problem (which were accepted at NeurIPS). Even though I tried to evaluate this paper keeping in mind that it was written concurrently, I think it falls short in a couple of important aspects which make it hard to recommend acceptance. In particular:
1. Unlike the other papers, the algorithm discussed in the theoretical section (which is able to reduce sample complexity by leveraging unlabeled data) is entirely different from the one used in practice on MNIST/CIFAR. It would make for a more compelling case if the algorithm used experimentally could also work on the toy model or vice versa (which is the case for Carmon et al. and Uesato et al.).
2 . The empirical evaluation is not detailed enough and there is some inconsistency in the baselines.
- In particular, the authors report that VAT attains poor robustness (<2.5% for both 5k and 10k labeled). However, Uesato et al. also benchmark against VAT in a very similar setting of 4k labelled CIFAR data points (with the same eps=8/255) and get ~32% accuracy (cf. Figure 1 from their paper). I could not find any difference between the two baselines except for the fact that Uesato et al. implement VAT with a KL divergence penalty (as suggested in the VAT paper) instead of cross entropy (as is used in this paper). This is somewhat concerning because based on the baselines reported in Uesato et al., the improvement of the approach proposed in this paper (which comes from doing 7 steps instead of 1 to find the adversarial example) are marginal. (Additionally, in this setting the approach of Uesato et al. gets robustness of about ~45% which is significantly better than ~33% reported in this paper.)
- Moreover, this paper evaluates on much fewer benchmarks (only MNIST/CIFAR with few labeled examples) compared to the other papers (which also study for example SVHN and the impact of using unlabeled ImageNet on CIFAR robustness).
The overlap with concurrent work is unfortunate, and it makes it hard to evaluate this paper. However, my two main concerns are (1) inconsistency in baselines which cast some doubt on the improvements offered by the proposed approach, and (2) the fact that both the algorithm and the experimental evaluation seem to be a subset of that in concurrent work (especially Uesato et al.). Thus, I have to recommend rejection.


We provide a framework for incorporating robustness -to perturbations in the transition dynamics which we refer to as model misspecification -into continuous control Reinforcement Learning (RL) algorithms. We specifically focus on incorporating robustness into a state-of-the-art continuous control RL algorithm called Maximum a-posteriori Policy Optimization (MPO). We achieve this by learning a policy that optimizes for a worst case expected return objective and derive a corresponding robust entropy-regularized Bellman contraction operator. In addition, we introduce a less conservative, soft-robust, entropy-regularized objective with a corresponding Bellman operator. We show that both, robust and soft-robust policies, outperform their non-robust counterparts in nine Mujoco domains with environment perturbations. In addition, we show improved robust performance on a high-dimensional, simulated, dexterous robotic hand. Finally, we present multiple investigative experiments that provide a deeper insight into the robustness framework. This includes an adaptation to another continuous control RL algorithm as well as learning the uncertainty set from offline data. Performance videos can be found online at https://sites.google.com/view/robust-rl.

AFTER REBUTTAL:
The authors answered all my questions and (significantly) modified the paper according to all the comments from all reviewers. The paper has significant value given the extensive experimental results and I believe could provide insight to other researchers in the field. I still have (like other reviewers) some concerns about scalability regarding the space of possible perturbations, but, as it is, the paper is worth for publication. For these reasons, *I increase my score to 7*, since I believe it's better than a weak accept (ignore the system rating, since it doesn't allow to put 7).
======================================================
Summary:
The authors provide a framework to incorporate robustness against perturbed dynamics to RL agents. They build upon the MPO algorithm and incorporate a few modifications that enable such robustness and entropy regularization. They show in multiple experiments that the robust versions of MPO outperform the non-robust ones when testing on environments with novel perturbations.
Decision:
I am happy with the experiments conducted on this paper and I think they would be a nice contribution to the community. However, I have serious concerns about the novelty of the proposed approach. The authors claim many novel contributions, but most of them are special cases of existing literature (not cited), or are simple modifications to existing approaches. For this reason my score is a reject. If the authors clearly cite the existing literature, state their contributions accordingly, and redirect the paper more towards their good experimental results (which they have plenty) I would be willing to substantially upgrade my score. Find below my supporting arguments.
About novelty
=============
The authors claim to have a novel framework and novel MDP formulations  for robust MDPs disregarding a substantial volume of the literature on robustness in the context of planning, control, reinforcement learning and MDPs. Since the authors really consider two types of robustness (robustness to dynamics and robustness to action stochasticity) I will expose in the following the relevant literature that is missing on both types, and when combining both.
Robustness to Dynamics:
Bart van den Broek, Wim Wiegerinck, and Hilbert J. Kappen.  Risk sensitive path integral control. In UAI, 2010.
Arnab Nilim and Laurent El Ghaoui.   Robust control of markov decision processes with uncertain transition matrices.Operations Research, 53(5):780–798, 2005
Wolfram Wiesemann, Daniel Kuhn, and Berc  Rustem.  Robust markov decision processes. Mathematics of Operations Research, 38(1):153–183, 2013.
Lars Peter Hansen and Thomas J Sargent.Robustness. Princeton university press, 2008
Yun Shen, Michael J Tobia, Tobias Sommer, and Klaus Obermayer. Risk-sensitive reinforcement learning. Neural computation, 26(7):1298–1328, 2014.
Yinlam  Chow,  Aviv  Tamar,  Shie  Mannor,  and  Marco  Pavone.   Risk-sensitive  and  robust decision-making: a cvar optimization approach.   In Advances in Neural Information Pro-cessing Systems, pages 1522–1530, 2015.
Robustness to Action Stochasticity:
Roy Fox, Ari Pakman, and Naftali Tishby.  G-learning: Taming the noise in reinforcement learning via soft updates.arXiv preprint arXiv:1512.08562, 2015.
Jonathan Rubin, Ohad Shamir, and Naftali Tishby.  Trading value and information in mdps.In Decision Making with Imperfect Decision Makers, pages 57–74. Springer, 2012.
Daniel A Braun, Pedro A Ortega, Evangelos Theodorou, and Stefan Schaal.  Path integral control  and bounded  rationality.   In Adaptive  Dynamic  Programming  And  ReinforcementLearning (ADPRL), 2011 IEEE Symposium on, pages 202–209. IEEE, 2011.
Combination of both:
[1] Grau-Moya, Jordi, et al. "Planning with information-processing constraints and model uncertainty in Markov decision processes." Joint European Conference on Machine Learning and Knowledge Discovery in Databases. Springer, Cham, 2016.
Importantly, in [1] above, it is shown a very similar line of research as the authors are proposing. In [1] it is combined entropy regularization with robustness in dynamics. In particular, using the formulation in [1] one can see how the formulations in the present paper can be recovered by setting the parameters in various ways. For example,
1- letting the entropy regularization coefficient ($\alpha$ in [1]) to a finite value and the robustness coefficient ($\beta$ in [1]) to $-\infty$ one recovers the traditional Robust MDP formulation.
2- Selecting a uniform prior over discrete theta (where each component in theta would correspond to a particular perturbation)  and setting $\beta \rightarrow 0$ one can also recover Soft-RE formulation (although a non-MPO version of it, i.e. just the MDP formulation).
3- Similarly, one can get rid of the entropy regularization by setting $\alpha \rightarrow \infty$.
Note that in [1] the uncertainty set is determined by a KL constraint whereas here it is determined by the chosen perturbations. However, in [1] setting the proper prior and $\beta$, one can obtain the same behaviour.
Importantly, in all previous references multiple contraction proofs (for entropy regularization Bellman operators, for robust operators alone, and for robust and entropy regularized operators) have already been discovered.
I hope all of the previous points can convince the authors about my decision above regarding novelty. Note, please, that I am aware that a robust version of MPO is slightly novel, however, a robust MDP formulation is more general than that and it could have perfectly been applied to any other RL optimization algorithm. Therefore, I don't think adding robustness to MPO alone is sufficient for enough novelty (other wise we would publish 1 paper per method converting one by one all known algorithms into their robust version of it).
About scalability
=================
I liked the experimental section. However, I have a question about practicality of the approach. As the authors show in the paper, the proposed method scales well in terms of  high-dimensional state and action spaces. But what about the scalability in terms of possible perturbations? I can imagine a plausible situation where we are dealing with a high-dimensional robot, let's say a humanoid robot, which allows for many possible "perturbations" e.g. longer left leg, longer right leg, longer fingers, shorter thumbs, and so on. The space of possible perturbations can be potentially very big. Could the authors elaborate on scalability issues of this type? Have they already thought about how one would solve this? I imagine that perturbing, let's say in one dimension, might change the dynamics quite in a different way compared to perturbing in another dimension. It would be interesting to see how their method generalizes when testing on unseen "dimension" perturbations.
Although this might present problems in terms of generalization, I understand that it does not invalidate the results presented here.
Good experimental results:
I have carefully checked all experimental results and I must say that they are very well executed e.g. multiple ablation studies and sufficient environments to support their approach.  Maybe a small comment: In figures 6 and 8 in the appendix there are multiple plots that only differ in the data (the titles are the same, e.g. two hopper figures with the same title). Maybe the authors can correct for this.


In this paper, we propose a framework, called MaskConvNet, for ConvNets filter pruning. MaskConvNet provides elegant support for training budget-aware pruned networks from scratch, by adding a simple mask module to a ConvNet architecture. MaskConvNet enjoys several advantages -(1) Flexible, the mask module can be integrated with any ConvNets in a plug-and-play manner. (2) Simple, the mask module is implemented by a hard Sigmoid function with small number of trainable mask variables, adding negligible memory and computational overheads to the networks during training. (3) Effective, it is able to achieve competitive pruning rate while maintaining comparable accuracy with the baseline ConvNets without pruning, regardless of the datasets and ConvNet architectures used. (4) Fast, it is observed that the number of training epochs required by MaskConvNet is close to training a baseline without pruning. (5) Budget-aware, with a sparsity budget on target metric (e.g. model size and FLOP), MaskConvNet is able to train in a way that the optimizer can adaptively sparsify the network and automatically maintain sparsity level, till the pruned network produces good accuracy and fulfill the budget constraint simultaneously. Results on CIFAR-10 and ImageNet with several ConvNet architectures show that MaskConvNet works competitively well compared to previous pruning methods, with budgetconstraint well respected. Code is available at https://www.dropbox.com/ s/c4zi3n7h1bexl12/maskconv-iclr-code.zip?dl=0. We hope MaskConvNet, as a simple and general pruning framework, can address the gaps in existing literate and advance future studies to push the boundaries of neural network pruning.

This paper proposes a masking process to improve the pruning of DNN. In addition, the algorithm proposes to automatically allocate the pruning rates over layers.
On the positive side:
- I do believe the main contribution is automatically allocating the pruning rates over the network.

Related works:
- How does this relate to methods using gating to prune in the presence of residual layers or BN?
- Paper claims multiple times that related works need a train-prune-retrain process which is only valid for post-processing works.
- I missed sparsity promoting works related to training from scratch where similar masks are implicitly used during training (even not explicitly stated) to maintain the zeros in the network. At least in [1,2] the training time is the same as the time used for training a network from scratch (same claim as in this paper).
Method:
- The mask is trained using a sigmoid function and claims this will be representative of the relevance of the 'neuron' within the layer. How is this really related?
- Why the initialization of the mask is to the mean? I think I am confused there. For initialization, I would argue all the neurons/parameters are relevant, right?
- The claim that this method is 'much simpler' is a bit subjective. I do not see why. Please elaborate.
- I am not sure if the claim of pruning filters and not considering the bn layer is correct. I would tend to think that, if pruning a neuron, the corresponding BN module should be modified (that is, propagating the zero to subsequent layers).
- I do like the extension to residual connections. It would be great to have more and clearer details on how is this done. Would this also apply to the UNET type of architecture?
- What is the intuition behind Eq 4 and how it is related to the relevance of a parameter?
- The extension to multiple metrics is of interest, however, there is little detailed there. How is the automatic allocation done? This is repeated in section 3.3 but details are missing.
- My understanding is that the regularization multiplier is affected by the learning rate, therefore their effect is lower as the training progresses. In this case, seems the opposite, right? (page 6 before 3.4).
- The part with the sparsity budget is interesting. What guarantees are that the newly enabled neurons are actually useful? Could it be possible that the budget suggested is not the right for the task at hand and, therefore, the additional parameters are not really relevant / needed?

Experiments:
- There is loss with little sparsity when it comes to Imagenet (15 and 17% pruning) does not seem very promising even the training time is similar.
- Seems like the experiments and comparisons to L1-pruning are not very surprising. It would be nice to have more comprehensive numbers. For imagenet, if using Resnet-50, would be easier to compare to other numbers.
- In the imagenet comparison, L1-pruning is the version-A of the paper. What about the others or why that particular model?
- How are the actual groups made?
Minor details:
- Please improve figure 1. It is not easy to understand. Same with Figure 3. What is seen in Figure 4.
- I do believe the WideResNet-28-10 number of parameters for BAR is not correct.
- Section 4.2 is a bit overselling. I do not see 'much-higher' parameter sparsity. The claims are mostly valid for VGG type of networks in this particular setting.

[1] Learning the Number of Neurons in Deep Networks, NeurIps 2016
[2] Compression-aware training of DNN, NeurIps 2017


Anomaly detection in time series data is an important topic in many domains. However, time series are known to be particularly hard to analyze. Based on the recent developments in adversarially learned models, we propose a new approach for anomaly detection in time series data. We build upon the idea to use a combination of a reconstruction error and the output of a Critic network. To this end we propose a cycle-consistent GAN architecture for sequential data and a new way of measuring the reconstruction error. We then show in a detailed evaluation how the different parts of our model contribute to the final anomaly score and demonstrate how the method improves the results on several data sets. We also compare our model to other baseline anomaly detection methods to verify its performance.

The paper trains a GAN on univariate time series data and uses reconstruction errors in combination with the critic's output to predict anomalous subsequences. The method is applied on two real-world data sets and compared to three simple baselines.
I have several reservations about this manuscript:
- The methodology isn't very original: The GAN architecture is essentially a slightly modified CycleGAN trained with Wasserstein loss.
- Many design choices appear to be ad-hoc: there has been no principled selection of the GAN's hyperparameters, nor has their effect on the experimental results been studied. For computing the reconstruction error the authors use the integral over the difference between two time series without taking the absolute value, which is a very unusual choice. (Standard choices would be e.g. L1 or L2 norm.) While this turns out to have worked "surprisingly well" on the studied datasets, it is not difficult to construct scenarios where this choice will fail.
- Another design choice that should be discussed in more detail is the smoothing of the time series data using a moving average filter; in the context of anomaly detection this can have a significant impact, and it may not work equally well for different data sets, so a principled approach for determining the level of filtering is paramount. Same goes for the de-trending, and the actual parameters both of the moving averages and the de-trending functions should be reported.
- It is not clear to me how exactly the anomaly scores (supposedly for different sequence lengths l?) are used to predict the subsequence(s) containing an anomaly. It seems there is no incentive to keep the predicted subsequences as short as possible, i.e. if the anomaly score indicates there might be an anomaly present, then the safe approach is to just flag the entire sequence as anomalous (it doesn't hurt the sensitivity they way it's computed)?
- The description of the experiments requires more detail. Are both datasets labelled? After dividing the datasets into rolling chunks of length 100, how many samples do the training and test sets contain? How many of the test set samples contain anomalies?
- The baselines need to be described in more detail. What method was used, e.g. to select and fit the ARIMA models? What type of reconstruction loss was used? How were the anomalous subsequences predicted?
- Is there any way to compare the proposed method with Li et al's (who also used a GAN for anomaly detection in time series data)?
Detailed comments:
- abstract: "particular hard" -> "particularly hard"
- p.3, paragraph starting with "To support...": shouldn't this be "E(x) with x ~ P_X"?
- p.4 and p.6: broken citations "?"
------------------
I acknowledge that I have read the authors' response, but it doesn't change my assessment that several major revisions are needed:
- further motivating the design choices and comparing them to the state-of-the-art;
- formalizing (ideally providing a model for) the sort of anomalies that the proposed method aims to detect;
- discussing limitations of the proposed approach.


Due to the significant costs of data generation, many prediction tasks within drug algorithm that is better suited to these settings. Our algorithm consists of learning 7 a deep network in combination with a kernel function and a differentiable kernel 8 algorithm. As the choice of kernel is critical, our algorithm learns to find the 9 appropriate kernel for each task during inference. It thus performs more effectively 10 with complex task distributions, outperforming current state-of-the-art algorithms 11 on both toy and novel, real-world benchmarks that we introduce herein. By 12 introducing novel benchmarks derived from biological assays, we hope that the 13 community will progress towards the development of FSR algorithms suitable for 14 use in noisy and uncertain environments such as drug discovery.

The submission is at the intersection of few-shot learning, kernel regression methods, and computational biology.
The main contributions are:
  - A few-shot learning algorithm combining several ideas and features:
    - Combining metric learning (shared across tasks) and kernel regression within each task
    - Learning a task representation by maximizing an estimate of the mutual information between the train (support) and valid (query) sets of a task
    - The addition of learned, synthetic "pseudo-examples"
  - Two datasets for few-shot regression, from real-life biological assays
The proposed algorithm outperforms (or is competitive with) mainstream few-shot learning methods on a synthetic 1D regression dataset, as well as the two proposed datasets.
This paper should be accepted, because it significantly expands the field of few-shot learning by proposing both a novel problem to tackle (few-shot regression from a high-dimensional, noisy input), with public datasets, and novel algorithm to solve it (combining several recent advances in different sub-fields of machine learning).
Overview
The overall problem is clearly stated, as well as its main challenges: noisiness of the data, different behaviors of the same input across tasks.
Despite the complexity of the proposed algorithm, its different pieces and their motivations are clearly motivated, introduced, and tested in ablation studies. I liked the clarity of section 2.
The "related work" section is clear and presents a good overall picture of the field. Additional papers that may be of interest:
  - Learned hallucination (Low-shot learning from imaginary data, Wang et al., CVPR 2018) seems to relate to the "pseudo-representations"
  - TADAM (Task dependent adaptive metric for improved few-shot learning, Oreshkin et al., NeurIPS 2018) is another example of task representation used in conjunction with metric learning (in the context of classification, not regression, though)
The proposed datasets are an interesting new benchmarking task, that naturally requires learning a task description, I hope it will get traction.
Questions:
  - The backpropagation through the kernel regressor (in order to train the parameters of the embedding function and other networks) seems unexpectedly straightforward. Is that only because there is a closed-form solution for the optimal regressor? Were the specific algorithms (KRR, GP) chosen because of that? Or could something like MetaOptNet (Meta-Learning with differentiable convex optimization, Lee et al., CVPR 2019) be used to relax that constraint?
  - What "generalization guarantees of kernel-based models" (l. 122) would be applicable here? Do they hold even when the kernel is applied on top of a learned embedding ($\phi_\theta$), or even when it depends on a trained model itself ($C_t$)?
  - What does "correlations > 0.8" mean exactly? (l. 258, 263)
  - In the "Binding" dataset, is it possible that the same protein is used in different bio-assays? In that case, are the different experiments "merged" into the same task, or be considered different tasks? Could it be possible that tasks involving the same protein would be in different (meta-)splits, or has that been taken care of during the data collection?
  - In the meta-test splits, how are examples split between the train/support and valid/query parts of each task?
  - How were the specific architectures of the different neural networks designed, or selected? Between $\phi$, v, r, $MLP_\rho$, the space of hyper-parameters seems huge, and the effects of these choices might be drastic.
Additional feedback
  - The caption of Figure 1 could (re-)introduce a definition of the notation. U and C_t for instance have not been introduced yet.
  - I'm not sure I agree that FSDKL only "share[s] characteristics with the metric learning framework", I see it more as being in that framework, but incorporating other elements as well (like other methods do, e.g., TADAM or RelationNet).
  - The horizontal axis of Figure 4 (a) and (c) are not clear until we see Table 4 of the appendix, and suggest an ordering of the different configurations, rather than 10 different categories. I'm not sure how to improve it though, maybe letters instead of numbers?
On notation:
  - On l. 155, is $\phi$ the same as $\phi_\theta$, or a different embedding function for x?
  - In Eq. 11, should $D^t_{trn}$ be $D^{t_j}_{trn}$ instead in both terms? Similarly, $D^{t_j}_{val}$ in the first term?
  - In Eq. 12, and l. 173, $\phi_x'$ suggests it is a different embedding $\phi'$ of the same $x$, if we want to convey that there are two inputs x and x' instead, $\phi_{x'}$ may be clearer.
  - In Fig. 4 (a) and 5, the vertical axis is labeled $\gamma_{mine}$ instead of (I assume) $\gamma_{task}$.


Value-based methods constitute a fundamental methodology in planning and deep reinforcement learning (RL). In this paper, we propose to exploit the underlying structures of the state-action value function, i.e., Q function, for both planning and deep RL. In particular, if the underlying system dynamics lead to some global structures of the Q function, one should be capable of inferring the function better by leveraging such structures. Specifically, we investigate the lowrank structure, which widely exists for big data matrices. We verify empirically the existence of low-rank Q functions in the context of control and deep RL tasks (Atari games). As our key contribution, by leveraging Matrix Estimation (ME) techniques, we propose a general framework to exploit the underlying low-rank structure in Q functions, leading to a more efficient planning procedure for classical control, and additionally, a simple scheme that can be applied to any value-based RL techniques to consistently achieve better performance on "low-rank" tasks. Extensive experiments on control tasks and Atari games confirm the efficacy of our approach.

Summary:
	This paper develops a method for taking advantage of structure in the value function to facilitate faster planning and learning. The key insight is that MDPs with low rank Q^* matrices can be solved more expediently using matrix estimation methods, both for classical dynamic programming methods (value iteration) and for learning in rich environments using recent model-free deep RL techniques. Thorough empirical analysis is conducted both for value iteration in tabular MDPs and for deep RL in rich environments. These experiments highlight new findings about the role the rank of the Q matrix plays in planning convergence and learning rates.
	I view this paper as containing several key contributions: first, the analysis on the role of Q rank in planning and learning---experiments conducted indicate that even complicated environments tend to have low rank Q matrices (when approximated). Highlighting the role of this rank and the corresponding empirical analysis estimating it in benchmark RL and control tasks is, to my knowledge, novel. Second, and perhaps the most significant contribution, is "Structured Value RL" (SV-RL), an easy-to-apply method that can be incorporated into many Q-based deep RL methods with little overhead. The empirical results are compelling: across three different variations of DQN-like architectures, the SV RL augmentation tends to improve learning. Presentation of results is rigorous, too, and provide strong evidence that the method works.
	As the paper mentions, theoretical analysis on the impact of Q-rank on dynamic programming (and perhaps learning) would be of great interest to the community. I take this analysis to be out of scope for this paper, but could see the work motivating future investigation into these questions.
Verdict: Overall, I take this paper to present many novel insights, establish solid motivation with good writing and examples, and offers compelling evidence about the strength of SV RL. I recommend accepting the paper.
Comments:
	C1: The visuals throughout the paper are helpful!
	C2: The paper is well written: the use of examples was effective in developing the motivation.
	C3: Section 2 is helpful for understanding the ideas developed in the paper. However, there are many well developed planning frameworks for MDPs that trade-off optimality with computational efficiency. It might be worth discussing some of these methods up front. For instance, Bounded Real-Time Dynamic Programming (McMahan et al. 2005) explicitly uses value function structure to improve planning speed, with performance guarantees, as does Focused RTDP (Smith and Simmons 2006). I don't take the computational complexity improvements of the proposed method to be the primary contribution, so just a brief discussion to contextualize the work against other planning literature would be helpful.
	C4: While the "rank" studied here is of a different form, some discussion of the Bellman Rank work (Jiang et al. 2017) might be useful for differentiating the two notions of "rank" at play, and how they are each used to expedite learning. The Bellman Rank is used as a measure of complexity of an MDP---Jiang et al. develop an RL algorithm that has sample complexity that depends on this measure. It is not strictly necessary, but I could see multiple uses of "rank" appearing in the RL literature as a means of exploiting structure for faster learning being confusing. If space (perhaps in the appendix if not), a sentence or two differentiating the two ideas might be helpful to readers. Additionally, the study of sparsity in value function representation was studied by Calandriello et al. 2014. If space permits, the paper might benefit from some discussion of the relation to this work.
Questions:
	Q1: In the inverted pendulum results, I am curious about the effect of the discretization on plan quality. Specifically: how were 2500 states and 1000 actions chosen? Were different orders of magnitude (for both values) considered? How did this impact SVP? Does the rank change as the discretization becomes more or less coarse? I don't think this is strictly critical for the paper, but a few sentences clarifying this point would be informative.
	Q2: Figure 4 provides nice insights into how to scale these ideas to deep RL. How were the four games chosen? Is there anything special that motivated their selection?
	Q3: Additionally, I am curious about whether the results from Figure 4 are the consequence of algorithmic decisions, rather than the environment. Is it possible to determine whether different value based methods (or different choices of hyperparameters) lead to different outcomes? For instance, I could imagine a more shallow network, or a tighter bottleneck, leading to Q evaluations that produce higher rank.
Typos and Writing Suggestions:
[Abstract]
	- This sentence is quite long, and I had a hard time following it as a result: "As our key contribution, by leveraging...". Consider dividing into two sentences.
[Intro]
	- Oxford comma: "control, planning and reinforcement learning"::"control, planning, and reinforcement learning
	- "the structured dynamic"::"the structure in the dynamics"
	- "where much fewer samples"::"where fewer samples"
	- Consider rewording: "almost the same policy as the optimal one". Is it that the policies are in fact the same? Or that their values are close? Perhaps: "a policy with near optimal value".
	- When introducing Double DQN and Dueling DQN for the first time it would be appropriate to cite each (end of Section 1).
[Sec. 2: Warm Up]
	- "understand the structures"::"understand the structure"
	- "give a strong evidence for"::"provide evidence that"
	- "exploit the structures for"::"exploit structure in the value function for"
	- I think the italicized statement at the top of page 3 could be sharpened. The antecedent currently stating "why not" is quite a soft statement compared to the motivation the section develops. Consider changing: "...why not enforcing such a structure throughout the iterations?"::"...then enforcing such a structure throughout planning can improve the rate of convergence".
[Sec. 3: Structured ... Planning]
	- "even non-convex optimization approaches (..."::"even non-convex optimization approaches to solving this problem (..."
	- "offer a sounding foundation for future"::"offer a sound foundation for future"
[Sec. 4: Structured ... RL]
	- "Previously, we start by"::"Previously, we started by"
	- "which in deep scenarios"::"which in scenarios with large state spaces", or perhaps: "which in deep scenarios"::"which in scenarios where value function approximation is used"
References:
Calandriello, Daniele, Alessandro Lazaric, and Marcello Restelli. "Sparse multi-task reinforcement learning." Advances in Neural Information Processing Systems. 2014.
Jiang, Nan, et al. "Contextual decision processes with low Bellman rank are PAC-learnable." Proceedings of the 34th International Conference on Machine Learning-Volume 70. JMLR. org, 2017.
McMahan, H. Brendan, Maxim Likhachev, and Geoffrey J. Gordon. "Bounded real-time dynamic programming: RTDP with monotone upper bounds and performance guarantees." Proceedings of the 22nd international conference on Machine learning. ACM, 2005.
Smith, Trey, and Reid Simmons. "Focused real-time dynamic programming for MDPs: Squeezing more out of a heuristic." AAAI. 2006.


The use of deep pre-trained transformers has led to remarkable progress in a number of applications . For tasks that make pairwise comparisons between sequences, matching a given input with a corresponding label, two approaches are common: Cross-encoders performing full self-attention over the pair and Bi-encoders encoding the pair separately. The former often performs better, but is too slow for practical use. In this work, we develop a new transformer architecture, the Poly-encoder, that learns global rather than token level self-attention features. We perform a detailed comparison of all three approaches, including what pre-training and fine-tuning strategies work best. We show our models achieve state-of-the-art results on four tasks; that Poly-encoders are faster than Cross-encoders and more accurate than Bi-encoders; and that the best results are obtained by pre-training on large datasets similar to the downstream tasks.

Summary: This work proposes a new transformer architecture for tasks that involve a query sequence and multiple candidate sequences. The proposed architecture, called poly-encoder, strikes a balance between a dual encoder which independently encodes the query and candidate and combines representations at the top, and a more expressive architecture which does full joint attention over the concatenated query and candidate sequences. Experiments on utterance retrieval tasks for dialog and an information retrieval task show that poly-encoders strike a good trade-off between the inference speed of the dual encoder model and the performance of the full attention model.
Pros:
- Strong results compared to baselines on multiple dialog and retrieval tasks.
- Detailed discussion of hyperparameter choices and good ablations.
- Paper is well written and easy to follow.
Cons:
- Limited novelty of methods. Ideas similar to the model variants discussed in this work have been considered in other work (Eg: [1]). It is also known that in-domain pre-training (i.e, pre-training on data close to the downstream task’s data distribution) helps (Eg: [2]). So this work can be considered as an application of existing ideas to dialog tasks.
- In terms of impact, utterance retrieval has fairly limited applicability in dialog. The dialog tasks considered in this work have a maximum of 100 candidate utterances, whereas in practice, the space of possible responses is much larger. While retrieval models are useful, I am skeptical about the practical value of the improvements shown in the paper (especially the improvements over bi-encoder, which is already a decent model).
Suggestions:
One way to get around the inefficiency of the cross-encoder architecture is to first use an inexpensive scoring mechanism such as TFIDF or bi-encoder to identify a small number of promising candidates from all the possible candidates. We can then use the cross-encoder to do more precise scoring of only the promising candidates. I am curious where a pipelined model such as this compares against the variants discussed in the paper in terms of speed and performance.
While the paper presents strong results on several dialog utterance retrieval tasks, the methods presented have limited novelty and impact. I am hence leaning towards borderline.
References
[1] Logeswaran Lajanugen, Ming-Wei Chang, Kenton Lee, Kristina Toutanova, Jacob Devlin, and Honglak Lee. 2019. Zero-Shot Entity Linking by Reading Entity Descriptions. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics.
[2] Jeremy Howard and Sebastian Ruder. 2018. Universal language model fine-tuning for text classification. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics.
Edit: I have read the author response. Based on the rebuttal, I am more convinced about the practical impact of the approach. I am raising my score and recommending accept.


Attention based Transformer architecture has enabled significant advances in the field of natural language processing. In addition to new pre-training techniques, recent improvements crucially rely on working with a relatively larger embedding dimension for tokens. Unfortunately, this leads to models that are prohibitively large to be employed in the downstream tasks. In this paper we identify one of the important factors contributing to the large embedding size requirement. In particular, our analysis highlights that the scaling between the number of heads and the size of each head in the existing architectures gives rise to this limitation, which we further validate with our experiments. As a solution, we propose a new way to set the projection size in attention heads that allows us to train models with a relatively smaller embedding dimension, without sacrificing the performance.

This work studies the head size <--> head number tradeoff in multihead attention. It argues and formally establishes that (1) the expressivity of an attention head is determined by its dimension and (b) fixing the head dimension, one gains additional expressive power by using more heads. In response to such observations, the paper proposes Fixed Multihead Attention, where the constraint that `head_size * number_of_heads = embedding_size` in standard multihead attention is lifted; and it allows for using more attention heads without making each head smaller. One can control the total amount of parameters by using smaller embedding sizes, making it comparable (in terms of #parameters) to standard multihead attention. Empirical results on language modeling and NLI tasks confirms the arguments.
Pros:
- The arguments on head size and head number tradeoff could be inspiring to future works.
- A simple approach that proves strong in several NLP tasks.
Cons:
- The theoretical discussion imposes too strong assumptions that might make it less interesting in practice.
- No NMT experiments.
- The takeaway seems a bit trivial.
Details:
- Theorem 1 presents a rank-based view of each attention head's capacity, which is nice. Yet it is still unclear whether it is the case that the more expressive are the heads the better. For example, several recent works argues for specialized attention heads, i.e., each head has specific "job," which may not require it being very expressive [1, 2]. Further, other works shows that a low-rank P matrix could be beneficial [3, 4, 5], which contradicts the argument in this work. It would be nice if the authors and discuss this in the revision
(To be clear, I do believe this is still an open question, and do not think presenting a different view from previous works hurts the contribution of this work in any way.)
- Theorem 2. I didn't carefully check the proof. Why is it required that the V matrices for each head have the same product. For both Thm.1 and 2, it would be nice to see some discussion on how they translate into the models in practice.
- Can the authors compare the training/inference speed? It probably will be the same as standard transformers, but it would be nice to confirm.
- Figure 1: the caption says trying out embedding sizes from 256 to 512. But it seems that only 4 values are tried. Can the authors comment on this? Also, it is a bit awkward to plot a line chart out of 4 points. Same for Figure 2.
- It would be nice to see some NMT experiments.
- The proposed method is so straightforward that I'm actually very surprised that this paper is the first trying this. The authors might need justify the technical contribution more.
(I'm on the fence for this one, but the system doesn't allow me to. I'm happy to revise the score if the authors can address my concerns.)
[1] Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned. https://arxiv.org/abs/1905.09418
[2] Are Sixteen Heads Really Better than One? https://arxiv.org/abs/1905.10650
[3] Generating Long Sequences with Sparse Transformers. https://arxiv.org/abs/1904.10509
[4] Generating Long Sequences with Sparse Transformers. https://arxiv.org/pdf/1904.10509.pdf.
[5] Adaptively Sparse Transformers. https://arxiv.org/abs/1909.00015.


It is well known that neural networks with rectified linear units (ReLU) activation functions are positively scale-invariant (i.e., the neural network is invariant to positive rescaling of weights). Optimization algorithms like stochastic gradient descent that optimize the neural networks in the vector space of weights, which are not positively scale-invariant. To solve this mismatch, a new parameter space, called path space, has been proposed for feedforward and convolutional neural networks. The path space is positively scale-invariant and optimization algorithms operating in path space have been shown to be superior than that in the original weight space. However, the theory of path space and the corresponding optimization algorithm cannot be naturally extended to more complex neural networks, like Recurrent Neural Networks (RNN) due to the recurrent structure and the parameter sharing scheme over time. In this work, we aim to construct path space for RNN with ReLU activations so that we can employ optimization algorithms in path space. To achieve the goal, we propose leveraging the reduction graph of RNN which removes the influence of time-steps, and prove that all the values of paths in the reduction graph can serve as a sufficient representation of the RNN with ReLU activations. We then prove that the path space for RNN is composed by the basis paths in reduction graph, and design a Skeleton Method to identify the basis paths efficiently. With the identified basis paths, we develop the optimization algorithm in path space for RNN models. Our experiments on several benchmark datasets show that we can obtain more effective RNN models in this way than using optimization methods in the weight space.

This paper aims to improve the training of recurrent neural networks (RNN) with ReLU activations by optimizing in the path space instead of the weight space. Studies on multi-layered perceptrons (MLP) and convolutional neural networks (CNN) have shown that these architectures are positively scale invariant (PSI), however traditional SGD optimizes in the weight space that does not have this property. It has been shown in prior work that this mismatch can slow down the optimization process for SGD and it has been demonstrated that optimizing in the so called path space, which has the PSI property, can be faster and more efficient.  The authors of this paper aim to extend an existing path-space framework (G-SGD) that is used for ReLU networks to facilitate RNNs.  To tackle the challenge posed by the time-dependency of RNNs, they use a static representation, called reduction graph, to define the path space. First, they prove that any path in the original RNN graph can be easily obtained from paths in its reduction graph by simple operations. Then, they define the basis for the path space in the reduction graph, and show that basis paths are sufficient to represent the output of the RNN. They propose a method (Skeleton method) to generate a basis path in the reduction graph and specify an equivalent of the G-SGD algorithm for RNNs.  Through numerical simulations they demonstrate that (1) G-SGD for RNNs has a fairly low additional computational cost compared to SGD and (2) G-SGD for RNNs achieves better test accuracy compared with SGD and an additional path space based approach.
Even though the paper has some original contribution, due to issues with the delivery, clarity and execution of the paper I would lean to reject it at this point. I would be willing to change my decision if the authors made significant changes to the paper in aspects detailed below. The paper also has many grammatical and typographical mistakes that hinders the reader in understanding key ideas.
First,  Section 2 could be much better explained. It seems like the notation introduced for a node and its value are used interchangeably, the edges are not clearly defined and Definition 1 is extremely confusing (what is s’ an arbitrary time step or all time steps between t and t+s, where is the list of weights mentioned in the first sentence and so on).  The definition for a key concept in the paper, the value of the path, is also not clear due to the lack of sufficient definition of the graph itself.
Second, in Section 3 the definition of the reduction graph and how it is obtained from the directed graph is not clear, which is problematic since it is one of the key ideas in the paper. Clear explanation of a recurrent edge is also missing.
Third, in Section 4 terms such as ‘without recurrent edges parts’  that are used in defining the Skeleton method are very lax and needs to be defined more rigorously. It is hard to understand what the boxed steps are saying. Moreover,  in Algorithm 2 division by a matrix occurs at several points. Does this mean entry-wise division? What does the function BP(.) do? I couldn’t find it defined anywhere.
Lastly, in Section 5 it would be fair to make a comparison with adaptive optimization methods such as Adam or AdaGrad as it is done in [Neyshabur et al.: Path SGD].
I would like to provide some additional feedback that do not impact my decision, but could potentially improve the paper. In Section 2: I cannot find an explanation what an ‘unfold directed graph’ means. In eq. (3) x is used without any prior explanation/definition. In Section 3, the message around eq. (4) is not clear, how we pick p2…p5? Moreover, in the definition of basis paths P is used without explaining what it denotes.  Comments on Appendix: in Table 4 it is confusing to highlight the results corresponding to the papers method, because it suggests that these are the best results across the table, however there is a lower value in the second row. In B.1. taking derivative w.r.t p doesn’t seem to make sense, it should probably be v_p (this holds for the whole derivation). In Table 6, it would be a clearer to present relative (%) results than absolute time cost.  Comment on the proofs: they are extremely verbose and could potentially be explained with more math and less words. It is difficult to follow what is going on. There is also some notation not introduced before in eq. (28). In general, I would recommend only numbering equations that are referred to in the text.


Reinforcement learning algorithms, though successful, tend to over-fit to training environments, thereby hampering their application to the real-world. This paper proposes WR 2 L -a robust reinforcement learning algorithm with significant robust performance on low and high-dimensional control tasks. Our method formalises robust reinforcement learning as a max-min game with a constraint derived from Wasserstein distance, and we propose an efficient and scalable algorithm to solve it. Our algorithm applies zero-order optimisation method that we believe can be useful to numerical optimisation in general. Apart from the formulation, we also propose an efficient and scalable solver following a novel zero-order optimisation method that we believe can be useful to numerical optimisation in general. We empirically demonstrate significant gains compared to standard and robust state-of-the-art algorithms on high-dimensional MuJuCo environments.

The paper proposes a novel solution to a *restricted* reinforcement learning problem. I am not quite sure if I understand the restriction from the text or thus what relevance this has in applications. The method factorizes the transition dynamics (where they rely on a parametrized solver) and a policy modeled by a NN and trained with policy gradient with monte-carlo returns. The ultimate goal is robust control so they care about the maximin objective of highest return along the worst trajectory. The robustness is implemented as a constraint in the dynamics optimization: a wasserstein distance between the distribution of samples obtained applying the learnt policy and learnt dynamics parameters and those obtained by random policy and some reference dynamics model. They solve the maximin by alternating minimization on the dynamics and policy. The contribution is in the intricacies of computing the wasserstein efficiently in particular they obtain a perturbation based estimate of the gradient of the dynamics parameters. The experiments seem sensible but I  have some suggestions for improvement.
I have published in RL but my familiarity with Wasserstein and robustness is limited. I have not checked the derivations in the appendix in detail. If there is a high score I'll have a closer look.
Major points:
1) The restriction imposed is that only part of the transition model is really learnt. So it is more like a system identification setup ?! Why is that necessary ?
2) The explanation in on page 4 about how the wasserstein distance is computed is a bit unclear. Is it sampling data from the P_\phi and \pi on one hand and P_0 and \pi_{Uniform} ? Then there is a bucketing step (what does it mean ? how is it computed ?). Is it then a discrete wasserstein between the two empirical distributions ?! What is Monte-Carlo in this case ?
3) All the approximations done in the method should be more clearly explained because they are relevant both to people who might apply the method in practice as well as other researchers that might want to build on/ improve on it.
4) The experiments seem interesting but I am not sure how to interpret the results. If a method has access to a simulator and a representation of the relevant dynamics parameter space it seems clear that it will be more robust. This is impressive if one has a simulator and can infer some state components e.g. in pendulum starting with an empty simulator and adding a pendulum. But that seems as hard or maybe even harder than trying to learn an implicit transition model just from observations.
5) Imagine the method only gets observations and an imperfect simulator, which will be the case in an application, what are the caveats we should have in mind ?
6) Can you have a learnt model baseline and compare your learnt and simulation ? Maybe it would have to be a latent variable model so you can do the same inference for internal parameters and still do an adaptation in the same way as now. This seems easy since it is a supervised problem.


Standard deep neural networks (DNNs) used for classification are trained in an end-to-end fashion for very specific tasks -object recognition, face identification, character recognition, etc. This specificity often leads to overconfident models that generalize poorly to samples that are not from the original training distribution. Moreover, they do not allow to leverage information from heterogeneously annotated data, where for example, labels may be provided with different levels of granularity. Finally, standard DNNs do not produce results with simultaneous different levels of confidence for different levels of detail, they are most commonly an all or nothing approach. To address these challenges, we introduce the problem of nested learning: how to obtain a hierarchical representation of the input such that a coarse label can be extracted first, and sequentially refine this representation to obtain successively refined predictions, all of them with the corresponding confidence. We explicitly enforce this behaviour by creating a sequence of nested information bottlenecks. Looking at the problem of nested learning from an information theory perspective, we design a network topology with two important properties. First, a sequence of low dimensional (nested) feature embeddings are enforced. Then we show how the explicit combination of nested outputs can improve both robustness and finer predictions. Experimental results on CIFAR-10, MNIST, and FASHION-MNIST demonstrate that nested learning outperforms the same network trained in the standard end-to-end fashion. Since the network can be naturally trained with mixed data labeled at different levels of nested details, we also study what is the most efficient way of annotating data, when a fixed training budget is given and the cost of labels increases with the levels in the nested hierarchy.

I read the author response, thank you for responding to my questions.
Original review:
This paper presents a hierarchical learning approach that trains neural network classifiers on a known hierarchy of labels.  The experiments show that the approach makes a network more robust to distortion compared to standard end-to-end learning.  The approach in this paper and its formalization of the task are interesting, but the significance of the techniques is somewhat unclear and the experiments could be more thorough in terms of the baselines and data sets considered.
From the related work section, the relationship between this paper and the previous work is somewhat complicated -- it is hard for a reader not deeply familiar with these previous works to understand the unique contribution made in the submission, and assess why it is significant.  For example, in the last paragraph of that section, many of the distinctions drawn between the submitted work and previous methods seem minor (including a confidence measure for a certain prediction, for example) or seemingly subjective (about whether an operation with a previous method was “natural” or could be done “transparently”).  Making crisper, less ambiguous distinctions between this work and previous work would help.
Likewise, the experimental results here do not compare against any of the hierarchical learning approaches discussed in the related work section.  The results show that the paper’s approach is more robust to distortion compared to standard end-to-end learning.  However, I was unclear on why it was not appropriate to compare against the other hierarchical learning methods from previous work.  Also, if the claim is improved robustness, I feel that evaluating against adversarial training (Madry et al., ICLR 2017) or similar approaches is necessary to understand the practical relevance of these improvements.
Finally, experiments that consider larger hierarchies (here, the number of target classes tends to be small, CIFAR-10 and MNIST each have ten classes, meaning the hierarchies are not very rich) would help illustrate the potential power of the techniques.
Minor
I didn’t understand the following statement, and given that it’s a fairly bold claim I would rephrase it or explain it better in the paper body rather than referring the reader to the appendix:
“A standard DNN unknowingly uses low quality data also to train higher layers, even if there is no high level information in the data.”
I don’t understand what the right arrow operator on the top of page 5 means.  From earlier statements it seems that I(f_i(X), Y_i) -> I(X, Y_i) means that the left quantity is approximately equal to the right.  But I’m not sure why to use an arrow for that rather than an \approx symbol.  The arrow would seem to imply that the left approaches the right in the limit, but if that is what you mean you should tell us in the limit of what.  Later the arrow is used to represent links in a Markov chain, adding further confusion for me.
I think it would be helpful if before Equation 1, you mentioned this holds for strictly nested Y_i’s (since earlier in the paper, Y_i referred to more general things).

I assume the ECE is computed over held-out validation data (i.e., not training data)?  The paper should say this.
Page 7: lineal -> linear


Knowledge graph is a kind of valuable knowledge base which would benefit lots of AI-related applications. Up to now, lots of large-scale knowledge graphs have been built. However, most of them are non-Chinese and designed for general purpose. In this work, we introduce TechKG, a large scale Chinese knowledge graph that is technology-oriented. It is built automatically from massive technical papers that are published in Chinese academic journals of different research domains. Some carefully designed heuristic rules are used to extract high quality entities and relations. Totally, it comprises of over 260 million triplets that are built upon more than 52 million entities which come from 38 research domains. Our preliminary experiments indicate that TechKG has high adaptability and can be used as a dataset for many diverse AI-related applications.

I have read the author response, thank you for responding.
Original review:
This paper presents the extraction of a bibliographic database of Chinese technical papers.  This database could potentially be a valuable resource for the community.  However, the paper is mis-targeted to the ICLR conference, as it does not discuss learning representations or using deep learning (the 36-page appendix does include some material on knowledge graph embedding, but this is not covered in the main body of the paper).  Also, for important tasks like name deduplication, the paper does not discuss or compare against techniques from previous work, and instead proposes a small set of heuristics, which is a sensible approach for building a resource but will not be of interest to the ICLR audience.
The paper refers to their resource as a knowledge graph, but I would say it is more accurate to call it a bibliographic database (it consists primarily of paper titles, authors, and keywords).  This is very different from the broader KBs and KGs discussed in the related work.  YAGO, Freebase, Cyc, and so on capture a much wider variety of semantic relationships and entity types.  I think re-positioning this submission as being aimed at building a bibliographic database, rather than a Knowledge Graph, would help ensure it reaches the right audience.  Also, targeting a different venue like a digital libraries conference and making a strong case that this set of bibliographic data offers advantages in coverage or accuracy over other comparable resources, if any, would help.
Finally, one of the most valuable features of a bibliographic database is the citation graph, it would be exciting if TechKG could be extended to include citation information.
Minor:
Section 2, most of these citations should not be shortcites but should be regular full cites.  For example in the first paragraph, Miller (1995) should be (Miller, 1995).


With the prevalence of machine learning services, crowdsourced data containing sensitive information poses substantial privacy challenges. Existing work focusing on protecting against membership inference attacks under the rigorous framework of differential privacy are vulnerable to attribute inference attacks. In light of the current gap between theory and practice, we develop a novel theoretical framework for privacy-preservation under the attack of attribute inference. Under our framework, we propose a minimax optimization formulation to protect the given attribute and analyze its privacy guarantees against arbitrary adversaries. On the other hand, it is clear that privacy constraint may cripple utility when the protected attribute is correlated with the target variable. To this end, we also prove an informationtheoretic lower bound to precisely characterize the fundamental trade-off between utility and privacy. Empirically, we extensively conduct experiments to corroborate our privacy guarantee and validate the inherent trade-offs in different privacy preservation algorithms. Our experimental results indicate that the adversarial representation learning approaches achieve the best trade-off in terms of privacy preservation and utility maximization.

Overall, this paper provides valuable insight into the trade-off between privacy preservation and utility when training a representation to fight against attribute inference attack.
Detail comments:
Strength:
+ The whole paper is well organized with logic. Notations are well defined and distinguished.
+ The final results have a good intuitive explanation.
+ The most impressive part of this paper is the analysis of the trade-off between privacy and utility from which the upper bound is quantified.
Weakness:
+ The minimax method looks trivial. The difficulty of using such an objective should be emphasized for practical implementation.
+ The major weakness is the experiments. The experiments only on two datasets may not be convincing for me. And the repetition times, 5 or 3, for each dataset are pretty small. Considering the experiments are conducted with random noise, e.g., DP methods, such a small repetition time is not fair since there a large chance these results could be selected.
+ Which DP Laplacian mechanism is used is not specified. Since there are already many improvements on the DP Laplacian mechanism, e.g., [A], it is necessary to make sure the baseline should be state-of-the-art for a fair comparison.
+ The result is not intuitively surprising that the privacy loss and utility loss will be balanced toward an upper bound (Theorem 4.2). I am not sure how the Jensen-Shannon entropy between D^Y_0 and D^Y_1 can be calculated in practice since the true conditional distribution is not observed. For example, when the data distribution is heavily biased, then the conditional distribution might show less correlation between Y and A. Then, the privacy protection will be a pretty simple task with a very high upper bound. Based on this, it is worth to ask what the upper bound looks like in the dataset used in the experiments.
+ How efficient this algorithm could be? In comparison to other baselines, does this method provide a more efficient solution?
[A] Phan, N., Wu, X., Hu, H., & Dou, D. (2017). Adaptive Laplace Mechanism: Differential Privacy Preservation in Deep Learning. 2017 IEEE International Conference on Data Mining (ICDM), 385–394. https://doi.org/10.1109/ICDM.2017.48