Published as a conference paper at ICLR 2018 LEAVE NO TRACE: LEARNING TO RESET FOR SAFE AND AUTONOMOUS REINFORCEMENT LEARNING
Deep reinforcement learning algorithms can learn complex behavioral skills, but real-world application of these methods requires a large amount of experience to be collected by the agent. In practical settings, such as robotics, this involves repeatedly attempting a task, resetting the environment between each attempt. However, not all tasks are easily or automatically reversible. In practice, this learning process requires extensive human intervention. In this work, we propose an autonomous method for safe and efficient reinforcement learning that simultaneously learns a forward and reset policy, with the reset policy resetting the environment for a subsequent attempt. By learning a value function for the reset policy, we can automatically determine when the forward policy is about to enter a non-reversible state, providing for uncertainty-aware safety aborts. Our experiments illustrate that proper use of the reset policy can greatly reduce the number of manual resets required to learn a task, can reduce the number of unsafe actions that lead to non-reversible states, and can automatically induce a curriculum.

This paper proposes the idea of having an agent learning a policy that resets the agent's state to one of the states drawn from the distribution of starting states. The agent learns such policy while also learning how to solve the actual task. This approach generates more autonomous agents that require fewer human interventions in the learning process. This is a very elegant and general idea, where the value function learned in the reset task also encodes some measure of safety in the environment.
All that being said, I gave this paper a score of 6 because two aspects that seem fundamental to me are not clear in the paper. If clarified, I'd happily increase my score.
1) *Defining state visitation/equality in the function approximation setting:* The main idea behind the proposed algorithm is to ensure that "when the reset policy is executed from any state, the distribution over final states matches the initial state distribution p_0". This is formally described, for example, in line 13 of Algorithm 1.
The authors "define a set of safe states S_{reset} \subseteq S, and say that we are in an irreversible state if the set of states visited by the reset policy over the past N episodes is disjoint from S_{reset}." However, it is not clear to me how one can uniquely identify a state in the function approximation case. Obviously, it is straightforward to apply such definition in the tabular case, where counting state visitation is easy. However, how do we count state visitation in continuous domains? Did the authors manually define the range of each joint/torque/angle that characterizes the start state? In a control task from pixels, for example, would the exact configuration of pixels seen at the beginning be the start state? Defining state visitation in the function approximation setting is not trivial and it seems to me the authors just glossed over it, despite being essential to your work.
2) *Experimental design for Figure 5*: This setup is not clear to me at all and in fact, my first reaction is to say it is wrong. An episodic task is generally defined as: the agent starts in a state drawn from the distribution of starting states and at the moment it reaches the goal state, the task is reset and the agent starts again. It doesn't seem to be what the authors did, is that right? The sentence: "our method learns to solve this task by automatically resetting the environment after each episode, so the forward policy can practice catching the ball when initialized below the cup" is confusion. When is the task reset to the "status quo" approach? Also, let's say an agent takes 50 time steps to reach the goal and then it decides to do a soft-reset. Are the time steps it is spending on its soft-reset being taken into account when generating the reported results?
Some other minor points are:
- The authors should standardize their use of citations in the paper. Sometimes there are way too many parentheses in a reference. For example: "manual resets are necessary when the robot or environment breaks (e.g. Gandhi et al. (2017))", or "Our methods can also be used directly with any other Q-learning methods ((Watkins & Dayan, 1992; Mnih et al., 2013; Gu et al., 2017; Amos et al., 2016; Metz et al., 2017))"
- There is a whole line of work in safe RL that is not acknowledged in the related work section. Representative papers are:
    [1] Philip S. Thomas, Georgios Theocharous, Mohammad Ghavamzadeh: High-Confidence Off-Policy Evaluation. AAAI 2015: 3000-3006
    [2] Philip S. Thomas, Georgios Theocharous, Mohammad Ghavamzadeh: High Confidence Policy Improvement. ICML 2015: 2380-2388
- In the Preliminaries Section the next state is said to be drawn from s_{t+1} ~ P(s'| s, a). However, this hides the fact the next state is dependent on the environment dynamics and on the policy being followed. I think it would be clearer if written: s_{t+1} ~ P(s'| s, \pi(a|s)).
- It seems to me that, in Algorithm 1, the name 'Act' is misleading. Shouldn't it be 'ChooseAction' or 'EpsilonGreedy'? If I understand correctly, the function 'Act' just returns the action to be executed, while the function 'Step' is the one that actually executes the action.
- It is absolutely essential to depict the confidence intervals in the plots in Figure 3. Ideally we should have confidence intervals in all the plots in the paper.


Published as a conference paper at ICLR 2018 DEEP GRADIENT COMPRESSION: REDUCING THE COMMUNICATION BANDWIDTH FOR DISTRIBUTED TRAINING
Large-scale distributed training requires significant communication bandwidth for gradient exchange that limits the scalability of multi-node training, and requires expensive high-bandwidth network infrastructure. The situation gets even worse with distributed training on mobile devices (federated learning), which suffers from higher latency, lower throughput, and intermittent poor connections. In this paper, we find 99.9% of the gradient exchange in distributed SGD are redundant, and propose Deep Gradient Compression (DGC) to greatly reduce the communication bandwidth. To preserve accuracy during this compression, DGC employs four methods: momentum correction, local gradient clipping, momentum factor masking, and warm-up training. We have applied Deep Gradient Compression to image classification, speech recognition, and language modeling with multiple datasets including Cifar10, ImageNet, Penn Treebank, and Librispeech Corpus. On these scenarios, Deep Gradient Compression achieves a gradient compression ratio from 270× to 600× without losing accuracy, cutting the gradient size of ResNet-50 from 97MB to 0.35MB, and for DeepSpeech from 488MB to 0.74MB. Deep gradient compression enables large-scale distributed training on inexpensive commodity 1Gbps Ethernet and facilitates distributed training on mobile.

I think this is a good work that I am sure will have some influence in the near future. I think it should be accepted and my comments are mostly suggestions for improvement or requests for additional information that would be interesting to have.
Generally, my feeling is that this work is a little bit too dense, and would like to encourage the authors in this case to make use of the non-strict ICLR page limit, or move some details to appendix and focus more on more thorough explanations. With increased clarity, I think my rating (7) would be higher.
Several Figures and Tables are never referenced in the text, making it a little harder to properly follow text. Pointing to them from appropriate places would improve clarity I think.
Algorithm 1 line 14: You never seem to explain what is sparse(G). Sec 3.1: What is it exactly that gets communicated? How do you later calculate the Compression Ratio? This should surely be explained somewhere.
Sec 3.2 you mention 1% loss of accuracy. A pointer here would be good, at that point it is not clear if it is in your work later, or in another paper. The efficient momentum correction is great!
As I was reading the paper, I got to the experiments and realized I still don't understand what is it that you refer to as "deep gradient compression". Pointer to Table 1 at the end of Sec 3 would probably be ideal along with some summary comments.
I feel the presentation of experimental results is somewhat disorganized. It is not clear what is immediately clear what is the baseline, that should be somewhere stressed. I find it really confusing why you sometimes compare against Gradient Dropping, sometimes against TernGrad, sometimes against neither, sometimes include Gradient Sparsification with momentum correction (not clear again what is the difference from DGC). I recommend reorganizing this and make it more consistent for sake of clarity. Perhaps show here only some highlights, and point to more in the Appendix.
Sec 5: Here I feel would be good to comment on several other things not mentioned earlier.
Why do you only work with 99.9% sparsity? Does 99% with 64 training nodes lead to almost dense total updates, making it inefficient in your communication model? If yes, does that suggest a scaling limit in terms of number of training nodes? If not, how important is the 99.9% sparsity if you care about communication cost dominating the total runtime? I would really like to better understand how does this change and what is the point beyond which more sparsity is not practically useful. Put differently, is DGC with 600x size reduction in total runtime any better than DGC with 60x reduction?
Finally, a side remark:
Under eq. (2) you point to something that I think could be more discussed. When you say what you do has the effect of increasing stepsize, why don't you just increase the stepsize?
There has recently been this works on training ImageNet in 1 hour, then in 24 minutes, latest in 15 minutes... You cite the former, but highlight different part of their work. Broader idea is that this is trend that potentially makes this kind of work less relevant. While I don't think that makes your work bad or misplaced, I think mentioning this would be useful as an alternative approach to the problems you mention in the introduction and use to motivate your contribution.
...what would be your reason for using DGC as opposed to just increasing the batch size?


ON THE CONVERGENCE OF ADAM AND BEYOND
Several recently proposed stochastic optimization methods that have been successfully used in training deep networks such as RMSPROP, ADAM, ADADELTA, NADAM are based on using gradient updates scaled by square roots of exponential moving averages of squared past gradients. In many applications, e.g. learning with large output spaces, it has been empirically observed that these algorithms fail to converge to an optimal solution (or a critical point in nonconvex settings). We show that one cause for such failures is the exponential moving average used in the algorithms. We provide an explicit example of a simple convex optimization setting where ADAM does not converge to the optimal solution, and describe the precise problems with the previous analysis of ADAM algorithm. Our analysis suggests that the convergence issues can be fixed by endowing such algorithms with "long-term memory" of past gradients, and propose new variants of the ADAM algorithm which not only fix the convergence issues but often also lead to improved empirical performance.

This work identifies a mistake in the existing proof of convergence of
Adam, which is among the most popular optimization methods in deep
learning. Moreover, it gives a simple 1-dimensional counterexample with
linear losses on which Adam does not converge. The same issue also
affects RMSprop, which may be viewed as a special case of Adam without
momentum. The problem with Adam is that the "learning rate" matrices
V_t^{1/2}/alpha_t are not monotonically decreasing. A new method, called
AMSGrad is therefore proposed, which modifies Adam by forcing these
matrices to be decreasing. It is then shown that AMSGrad does satisfy
essentially the same convergence bound as the one previously claimed for
Adam. Experiments and simulations are provided that support the
theoretical analysis.
Apart from some issues with the technical presentation (see below), the
paper is well-written.
Given the popularity of Adam, I consider this paper to make a very
interesting observation. I further believe all issues with the technical
presentation can be readily addressed.
Issues with Technical Presentation:
- All theorems should explicitly state the conditions they require
  instead of referring to "all the conditions in (Kingma & Ba, 2015)".
- Theorem 2 is a repetition of Theorem 1 (except for additional
  conditions).
- The proof of Theorem 3 assumes there are no projections, so this
  should be stated as part of its conditions. (The claim in footnote 2
  that they can be handled seems highly plausible, but you should be up
  front about the limitations of your results.)
- The regret bound Theorem 4 establishes convergence of the optimization
  method, so it plays the role of a sanity check. However, it is
  strictly worse than the regret bound O(sqrt{T}) for online gradient
  descent [Zinkevich,2003], so it cannot explain why the proposed
  AMSgrad method might be adaptive. (The method may indeed be adaptive
  in some sense; I am just saying the *bound* does not express that.
  This is also not a criticism of the current paper; the same remark
  also applies to the previously claimed regret bound for Adam.)
- The discussion following Corollary 1 suggests that sum_i
  hat{v}_{T,i}^{1/2} might be much smaller than d G_infty. This is true,
  but we should always expect it to be at least a constant, because
  hat{v}_{t,i} is monotonically increasing by definition of the
  algorithm, so the bound does not get better than O(sqrt(T)).
  It is also suggested that sum_i ||g_{1:T,i}|| = sqrt{sum_{t=1}^T
  g_{t,i}^2} might be much smaller than dG_infty, but this is very
  unlikely, because this term will typically grow like O(sqrt{T}),
  unless the data are extremely sparse, so we should at least expect
  some dependence on T.
- In the proof of Theorem 1, the initial point is taken to be x_1 = 1,
  which is perfectly fine, but it is not "without loss of generality",
  as claimed. This should be stated in the statement of the Theorem.
- The proof of Theorem 6 in appendix B only covers epsilon=1. If it is
  "easy to show" that the same construction also works for other
  epsilon, as claimed, then please provide the proof for general
  epsilon.
Other remarks:
- Theoretically, nonconvergence of Adam seems a severe problem. Can you
  speculate on why this issue has not prevented its widespread adoption?
  Which factors might mitigate the issue in practice?
- Please define g_t \circ g_t and g_{1:T,i}
- I would recommend sticking with standard linear algebra notation for
  the sqrt and the inverse of a matrix and simply using A^{-1} and
  A^{1/2} instead of 1/A and sqrt{A}.
- In theorems 1,2,3, I would recommend stating the dimension (d=1) of
  your counterexamples, which makes them very nice!
Minor issues:
- Check accent on Nicol\`o Cesa-Bianchi in bibliography.
- Near the end of the proof of Theorem 6: I believe you mean Adam
  suffers a "regret" instead of a "loss" of at least 2C-4.
  Also 2C-4=2C-4 is trivial in the second but last display.


Social dilemmas are situations where individuals face a temptation to increase their payoffs at a cost to total welfare. Building artificially intelligent agents that achieve good outcomes in these situations is important because many real world interactions include a tension between selfish interests and the welfare of others. We show how to modify modern reinforcement learning methods to construct agents that act in ways that are simple to understand, nice (begin by cooperating), provokable (try to avoid being exploited), and forgiving (try to return to mutual cooperation). We show both theoretically and experimentally that such agents can maintain cooperation in Markov social dilemmas. Our construction does not require training methods beyond a modification of self-play, thus if an environment is such that good strategies can be constructed in the zero-sum case (eg. Atari) then we can construct agents that solve social dilemmas in this environment.

This paper addresses multiagent learning problems in which there is a social dilemma: settings where there are no 'cooperative polices' that form an equilibrium. The paper proposes a way of dealing with these problems via amTFT, a variation of the well-known tit-for-that strategy, and presents some empirical results.
My main problem with this paper is clarity and I am afraid that not everything might be technically correct. Let me just list my main concerns in the below.
The definition of social dilemma, is unclear:
"A social dilemma is a game where there are no cooperative policies which form equilibria. In other
words, if one player commits to play a cooperative policy at every state, there is a way for their
partner to exploit them and earn higher rewards at their expense."
does this mean to say "there are no cooperative *Markov* policies" ? It seems to me that the paper precisely intents to show that by resorting to history-dependent policies (such as both using amTFT), there is a cooperative equilibrium.
I don't understand:
"Note that in a social dilemma there may be policies which achieve the payoffs of cooperative policies because they cooperate on the trajectory of play and prevent exploitation by threatening non-cooperation on states which are never reached by the trajectory. If such policies exist, we call the social dilemma solvable."
is this now talking about non-Markov policies? If not, there seems to be a contradiction?
The work focuses on TFT-like policies, motivated by
"if one can commit to them, create incentives for a partner to behave cooperatively"
however it seems that, as made clear below definition 4, we can only create such incentives for sufficiently powerful agents, that remember and learn from their failures to cooperate in the past?
Why is the method called "approximate Markov"? As soon as one introduces history dependence, the Markov property stops to hold?
On page 4, I have problems following the text due to inconsistent use of notation: subscripts and superscripts seem random, it is not clear which symbols denote strategy profiles (rather than individual strategies), there seems mix-ups between 'i' and '1' / '2', there is sudden use of \hat{}, and other undefined symbols (Q_CC?).
For all practical purposes, it seems that the made assumptions imply uniqueness of the cooperative joint strategy. I fully appreciate that the coordination question is difficult and important, so if the proposed method is not compatible with dealing with that important question, that strikes me as a large drawback.
I have problems understanding how it is possible to guarantee "If they start in a D phase, they eventually return to a C phase." without making more assumptions on the domain. The clear example being the typical 'heaven or hell' type of problems: what if after one defect, we are trapped in the 'hell' state where no cooperation is even possible?
"If policies converge with this training then πˆ is a Markov equilibrium (up to function approximation)." There are two problems here:
1) A problem is that very typically things will not converge... E.g.,
Wunder, Michael, Michael L. Littman, and Monica Babes. "Classes of multiagent q-learning dynamics with epsilon-greedy exploration." Proceedings of the 27th International Conference on Machine Learning (ICML-10). 2010.
2) "Up to function approximation" could be arbitrary large?
Another significant problem seems to be with this statement:
"while in the cooperative reward schedule the standard RL convergence guarantees apply. The latter is because cooperative training is equivalent to one super-agent controlling both players and trying to optimize for a single scalar reward." The training of individual learners is quite different from "joint action learners" [Claus & Boutilier 98], and this in turn is different from a 'super-agent' which would also control the exploration. In absence of the super-agent, I believe that the only guarantee is that one will, in the limit, converge to a Nash equilibrum, which might be arbitrary far from the optimal joint policy. And this only holds for the tabular case. See the discussion in
A concise introduction to multiagent systems and distributed artificial intelligence. N Vlassis. Synthesis Lectures on Artificial Intelligence and Machine Learning 1 (1), 1-71
Also, the approach used in the experiments "Cooperative (self play with both agents receiving sum of rewards) training for both games", would be insufficient for many settings where a cooperative joint policy would be asymmetric.
The entire approach hinges on using rollouts (the commented lines in Algo. 1). However, it is completely not clear to me how this works. The one paragraph is insufficient to get across these crucial parts of the proposed approach.
It is not clear why the tables in Figure 1 are not symmetric; this strikes me as extremely problematic. It is not clear what the colors encode either.
It also seems that "grim" is better against all, except against amTFT, why should we not use that? In general, the explanation of this closely related paper by De Cote & Littman (which was published at UAI'08), is insufficient. It is not quite clear to me what the proposed approach offers over the previous method.


To solve the text-based question and answering task that requires relational reasoning, it is necessary to memorize a large amount of information and find out the question relevant information from the memory. Most approaches were based on external memory and four components proposed by Memory Network. The distinctive component among them was the way of finding the necessary information and it contributes to the performance. Recently, a simple but powerful neural network module for reasoning called Relation Network (RN) has been introduced. We analyzed RN from the view of Memory Network, and realized that its MLP component is able to reveal the complicate relation between question and object pair. Motivated from it, we introduce Relation Memory Network (RMN) which uses MLP to find out relevant information on Memory Network architecture. It shows new state-of-the-art results in jointly trained bAbI-10k story-based question answering tasks and bAbI dialog-based question answering tasks.

This paper introduces Related Memory Network (RMN), an improvement over Relationship Networks (RN). RMN avoids growing the relationship time complexity as suffered by RN (Santoro et. Al 2017). RMN reduces the complexity to linear time for the bAbi dataset. RN constructs pair-wise interactions between objects in RN to solve complex tasks such as transitive reasoning. RMN instead uses a multi-hop attention over objects followed by an MLP to learn relationships in linear time.
Comments for the author:
The paper addresses an important problem since understanding object interactions are crucial for reasoning. However, how widespread is this problem across other models or are you simply addressing a point problem for RN? For example, Entnet is able to reason as the input is fed in and the decoding costs are low. Likewise, other graph-based networks (which although may require strong supervision) are able to decode quite cheaply.
The relationship network considers all pair-wise interactions that are replaced by a two-hop attention mechanism (and an MLP). It would not be fair to claim superiority over RN since you only evaluate on bABi while RN also demonstrated results on other tasks. For more complex tasks (even over just text), it is necessary to show that you outperform RN w/o considering all objects in a pairwise fashion. More specifically, RN uses an MLP over pair-wise interactions, does that allow it to model more complex interactions than just selecting two hops to generate attention weights. Showing results with multiple hops (1,2,..) would be useful here.
More details are needed about Figure 3. Is this on bAbi as well? How did you generate these stories with so many sentences? Another clarification is the bAbi performance over Entnet which claims to solve all tasks. Your results show 4 failed tasks, is this your reproduction of Entnet?
Finally, what are the savings from reducing this time complexity? Some wall clock time results or FLOPs of train/test time should be provided since you use multiple hops.
Overall, this paper feels like a small improvement over RN. Without experiments over other datasets and wall clock time results, it is hard to appreciate the significance of this improvement. One direction to strengthen this paper is to examine if RMN can do better than pair-wise interactions (and other baselines) for more complex reasoning tasks.


In inductive transfer learning, fine-tuning pre-trained convolutional networks substantially outperforms training from scratch. When using fine-tuning, the underlying assumption is that the pre-trained model extracts generic features, which are at least partially relevant for solving the target task, but would be difficult to extract from the limited amount of data available on the target task. However, besides the initialization with the pre-trained model and the early stopping, there is no mechanism in fine-tuning for retaining the features learned on the source task. In this paper, we investigate several regularization schemes that explicitly promote the similarity of the final solution with the initial model. We eventually recommend a simple L 2 penalty using the pre-trained model as a reference, and we show that this approach behaves much better than the standard scheme using weight decay on a partially frozen network.

This work addresses the scenario of fine-tuning a pre-trained network for new data/tasks and empirically studies various regularization techniques. Overall, the evaluation concludes with recommending that all layers of a network whose weights are directly transferred during fine-tuning should be regularized against the initial net with an L2 penalty during further training.
Relationship to prior work:
Regularizing a target model against a source model is not a new idea. The authors miss key connections to A-SVM [1] and PMT-SVM [2] -- two proposed transfer learning models applied to SVM weights, but otherwise very much the same as the proposed solution in this paper. Though the study here may offer new insights for deep nets, it is critical to mention prior work which also does analysis of these regularization techniques.
Significance:
As the majority of visual recognition problems are currently solved using variants of fine-tuning, if the findings reported in this paper generalize, then it could present a simple new regularization which improves the training of new models. The change is both conceptually simple and easy to implement so could be quickly integrated by many people.
Clarity and Questions:
The purpose of the paper is clear, however, some questions remain unanswered.
1) How is the regularization weight of 0.01 chosen? This is likely a critical parameter. In an experimental paper, I would expect to see a plot of performance for at least one experiment as this regularization weighting parameter is varied.
2) How does the use of L2 regularization on the last layer effect the regularization choice of other layers? What happens if you use no regularization on the last layer? L1 regularization?
3) Figure 1 is difficult to read. Please at least label the test sets on each sub-graph.
4) There seems to be some issue with the freezing experiment in Figure 2. Why does performance of L2 regularization improve as you freeze more and more layers, but is outperformed by un-freezing all.
5) Figure 3 and the discussion of linear dependence with the original model in general seems does not add much to the paper. It is clear that regularizing against the source model weights instead of 0 should result in final weights that are more similar to the initial source weights. I would rather the authors use this space to provide a deeper analysis of why this property should help performance.
6) Initializing with a source model offers a strong starting point so full from scratch learning isn’t necessary -- meaning fewer examples are needed for the continued learning (fine-tuning) phase. In a similar line of reasoning, does regularizing against the source further reduce the number of labeled points needed for fine-tuning? Can you recover L2 fine-tuning performance with fewer examples when you use L2-SP?
[1] J. Yang, R. Yan, and A. Hauptmann. Adapting svm classifiers to data with shifted distributions. In ICDM Workshops, 2007.
[2]  Y. Aytar and A. Zisserman. Tabula rasa: Model transfer for object category detection. In Proc. ICCV, 2011.
------------------
Post rebuttal
------------------
The changes made to the paper draft as well as the answers to the questions posed above have convinced me to upgrade my recommendation to a weak accept. The experiments are now clear and thorough enough to provide a convincing argument for using this regularization in deep nets. Since it is simple and well validated it should be easily adopted.


LEARNING A GENERATIVE MODEL FOR VALIDITY IN COMPLEX DISCRETE STRUCTURES
Deep generative models have been successfully used to learn representations for high-dimensional discrete spaces by representing discrete objects as sequences and employing powerful sequence-based deep models. Unfortunately, these sequencebased models often produce invalid sequences: sequences which do not represent any underlying discrete structure; invalid sequences hinder the utility of such models. As a step towards solving this problem, we propose to learn a deep recurrent validator model, which can estimate whether a partial sequence can function as the beginning of a full, valid sequence. This validator provides insight as to how individual sequence elements influence the validity of the overall sequence, and can be used to constrain sequence based models to generate valid sequences -and thus faithfully model discrete objects. Our approach is inspired by reinforcement learning, where an oracle which can evaluate validity of complete sequences provides a sparse reward signal. We demonstrate its effectiveness as a generative model of Python 3 source code for mathematical expressions, and in improving the ability of a variational autoencoder trained on SMILES strings to decode valid molecular structures.

The authors use a recurrent neural network to build generative models of sequences in domains where the vast majority of sequences is invalid. The basic idea, outlined in Eq. 2, is moderately straightforward: at each step, use an approximation of the Q function for subsequences of the appropriate length to pick a valid extension. There are numerous details to get right. The writing is mostly clear, and the examples are moderately convincing. I wish the paper had more detailed arguments and discussions.
I question the appropriateness of Eq. 2 as a target. A correctly learned model will put positive weight on valid sequences, but it may be an arbitrarily slow way to generate diverse sequences, depending on the domain. For instance, imagine a domain of binary strings where the valid sequences are the all 1 sequence, or any sequence beginning with a 0. Half the generated sequences would be all 1's in this situation, right? And it's easy to construct further examples that are much worse than this?
The use of Bayesian active learning to generate the training set feels like an elegant idea. However, I wish there were more clarity about what was ad hoc and what wasn't. For instance, I think the use of  dropout to get q is suspect (see for instance https://arxiv.org/abs/1711.02989), and I'd prefer a little more detail on statements like "The nonlinearity of g(·) means that our Monte
Carlo approximation is biased, but still consistent." Do we have any way of quantifying the bias? Is the statement about K=16 being reasonable a statement about bias, variance, or both?
For Python strings:
- Should we view the fact that high values of tau give a validity of 1.0 as indicative that the domain's constraints are fairly easy to learn?
- "The use of a Boltzmann policy allows us to tune the temperature parameter to identify policies
which hit high levels of accuracy for any learned Q-function approximation." This is only true to the extent the domain is sufficiently "easy" right? Is the argument that even in very hard domains, you might get this by just having an RNN which memorized a single valid sequence (assuming at least one could be found)?
- What's the best explanation for *why* the active model has much higher diversity? I understand that the active model is picking examples that tell us more about the uncertainty in w, but it's not obvious to me that means higher diversity. Do we think this is a universal property of domains?
- The highest temperature active model is exploring about half of valid sequences (modulo the non-tightness of the bound)? Have you tried gaining some insight by generating thousands of valid sequences manually and seeing which ones the model is rejecting?
- The coverage bound is used only for for Python expressions, right? Why not just randomly sample a few thousand positives and use that to get a better estimate of coverage? Since you can sample from the true positive set, it seems that your argument from the appendix about the validation set being "too similar to the training set" doesn't apply?
- It would be better to see a comparison to a strong non-NN baseline. For instance, I could easily make a PCFG over Python math expressions, and use rejection sampling to get rid of those that aren't exactly length 25, etc.?
I question how easy the Python strings example is. In particular, it might be that it's quite an easy example (compared to the SMILES) example. For SMILES, it seems like the Bayesian active learning technique is not by itself sufficient to create a good model? It is interesting that in the solubility domain the active model outperforms, but it would be nice to see more discussion / explanation.
Minor note: The incidence of valid strings in the Python expressions domain is (I believe) > 1/5000, although I guess 1 in 10,000 is still the right order of magnitude.
If I could score between "marginal accept" and "accept" I would.


Skip connections are increasingly utilized by deep neural networks to improve accuracy and cost-efficiency. In particular, the recent DenseNet is efficient in computation and parameters, and achieves state-of-the-art predictions by directly connecting each feature layer to all previous ones. However, DenseNet's extreme connectivity pattern may hinder its scalability to high depths, and in applications like fully convolutional networks, full DenseNet connections are prohibitively expensive. This work first experimentally shows that one key advantage of skip connections is to have short distances among feature layers during backpropagation. Specifically, using a fixed number of skip connections, the connection patterns with shorter backpropagation distance among layers have more accurate predictions. Following this insight, we propose a connection template, Log-DenseNet, which, in comparison to DenseNet, only slightly increases the backpropagation distances among layers from 1 to (1+log 2 L), but uses only L log 2 L total connections instead of O(L 2 ). Hence, Log-DenseNets are easier to scale than DenseNets, and no longer require careful GPU memory management. We demonstrate the effectiveness of our design principle through ablation studies and by showing better performance than DenseNets on tabula rasa semantic segmentation, and competitive results on visual recognition.

This paper investigates how to impose layer-wise connections in DenseNets most efficiently. The authors propose a connection-pattern, which connects layer i to layer i-2^k, k=0,1,2... The authors also propose maximum backpropgation distance (MBD) for measuring the fluency of gradient flow in the network, and justify the Log-DenseNet's advantage in this framework. Empirically, the author demonstrates the effectiveness of Log-DenseNet by comparing it with two other intuitive connection patterns on CIFAR datasets. Log-DenseNet also improves on FC-DenseNet, where the connection budget is the bottleneck because the feature maps are of high resolutions.
Strengths:
1. Generally, DenseNet is memory-hungry if the connection is dense, and it is worth studying how to sparsify a DenseNet. By showing the improvements on FC-DenseNet, Log-DenseNet demonstrates good potential on tasks which require upsampling of feature maps.
2. The ablation experiments are well-designed and the visualizations of connectivity pattern are clear.
Weakness:
1. Adding a comparison with Log-DenseNet and vanilla DenseNet in the Table 2 experiment would make the paper stronger. Also, the NearestHalfAndLog pattern is not used in any latter visual recognition experiments, so I think it's better to just compare LogDenseNet with the two baselines instead. Despite there are CIFAR experiments on Log-DenseNet in latter sections, including results here would be easier to follow.
2. I would like to see the a comparison with the DenseNet-BC in the segmentation and CIFAR classification tasks, which uses 1x1 conv layers to reduce the number of channels. It should be interesting to study whether it is possible to further sparsify DenseNet-BC, as it has much higher efficiency.
3. The improvement of efficiency on classifications task is not that significant.


ENSEMBLE ADVERSARIAL TRAINING: ATTACKS AND DEFENSES
Adversarial examples are perturbed inputs designed to fool machine learning models. Adversarial training injects such examples into training data to increase robustness. To scale this technique to large datasets, perturbations are crafted using fast single-step methods that maximize a linear approximation of the model's loss. We show that this form of adversarial training converges to a degenerate global minimum, wherein small curvature artifacts near the data points obfuscate a linear approximation of the loss. The model thus learns to generate weak perturbations, rather than defend against strong ones. As a result, we find that adversarial training remains vulnerable to black-box attacks, where we transfer perturbations computed on undefended models, as well as to a powerful novel single-step attack that escapes the non-smooth vicinity of the input data via a small random step. We further introduce Ensemble Adversarial Training, a technique that augments training data with perturbations transferred from other models. On ImageNet, Ensemble Adversarial Training yields models with strong robustness to black-box attacks. In particular, our most robust model won the first round of the NIPS 2017 competition on Defenses against Adversarial Attacks (Kurakin et al., 2017c).

This paper describes computationally efficient methods for training adversarially robust deep neural networks for image classification. (These methods may extend to other machine learning models and domains as well, but that's beyond the scope of this paper.)
The former standard method for generating adversarially images quickly and using them in training was to do a single gradient step to increase the loss of the true label or decrease the loss of an alternate label. This paper shows that such training methods only lead to robustness against these "weak" adversarial examples, leaving the adversarially-trained models vulnerable to multi-step white-box attacks and black-box attacks (adversarial examples generated to attack alternate models).
There are two proposed solutions. The first is to generate additional adversarial examples from other models and use them in training. This seems to yield robustness against black-box attacks from held-out models as well.  Of course, it requires that you have a somewhat diverse group of models to choose from. If that's the case, why not directly build an ensemble of all the models? An ensemble of neural networks can still be represented as a neural network, although a more computationally costly one. Thus, while this heuristic appears to be useful with current models against current attacks, I don't know how well it will hold up in the future.
The second solution is to add random noise before taking the gradient step.  This yields more effective adversarial examples, both for attacking models and for training, because it relies less on the local gradient. This is another simple idea that appears to be effective. However, I would be interested to see a comparison to a 2-step gradient-based attack.  R+Step-LL can be viewed as a 2-step attack: a random step followed by a gradient step. What if both steps were gradient steps instead? This interpolates between Step-LL and I-Step-LL, with an intermediate computational cost. It would be very interesting to know if R+Step-LL is more or less effective than 2+Step-LL, and how large the difference is.
I like that this paper demonstrates the weakness of previous methods, including extensive experiments and a very nice visualization of the loss landscape in two adversarial dimensions. The proposed heuristics seem effective in practice, but they're somewhat ad hoc and there is no analysis of how these heuristics might or might not be vulnerable to future attacks.


We propose an unsupervised method for building dynamic representations of sequential data, particularly of observed interactions. The method simultaneously acquires representations of input data and its dynamics. It is based on a hierarchical generative model composed of two levels. In the first level, a model learns representations to generate observed data. In the second level, representational states encode the dynamics of the lower one. The model is designed as a Bayesian network with switching variables represented in the higher level, and which generates transition models. The method actively explores the latent space guided by its knowledge and the uncertainty about it. That is achieved by updating the latent variables from prediction error signals backpropagated to the latent space. So, no encoder or inference models are used since the generators also serve as their inverse transformations. The method is evaluated in two scenarios, with static images and with videos. The results show that the adaptation over time leads to better performance than with similar architectures without temporal dependencies, e.g., variational autoencoders. With videos, it is shown that the system extracts the dynamics of the data in states that highly correlate with the ground truth of the actions observed.

The authors propose an architecture and generative model for static images and video sequences, with the purpose of generating an image that looks as similar as possible to the one that is supplied. This is useful for for example frame prediction in video and detection of changes in video as a consequence to changes in the dynamics of objects in the scene.
The architecture minimizes the error between the generated image(s) and the supplied image(s) by refining the generated image over time when the same image is shown and by adapting when the image is changed. The model consists of three neural networks (F_Zµ, F_Zsigma, f_X|Z) and three multivariate Gaussian distributions P(S_t), P(Z_t) and P(X_t,Z_t) with diagonal covariances. The NNs do not change over time but they relate the three Gaussian distributions in different ways and these distributions are changing over time in order to minimize the error of the generated image(s).
The paper took a while to understand due to its structure and how it is written. A short overview of the different components of Figure 1 giving the general idea and explaining
* what nodes are stochastic variables and NNs
* what is trained offline/online
It would also help the structure if the links/arrows/nodes had numbers corresponding to the relevant equations defining the relations/computations. Some of these relations are defined with explicit equation numbers, others are baked into the text which makes it difficult to jump around in the paper when reading it and trying to understand the architecture.
There are also numerous language errors in the paper and many of them are grammatical. For example:
  Page 5, second paragraph: "osculations" -> "oscillations"
  Page 5, fourth paragraph: "Defining .. is defined as.."
The results seem impressive and the problem under consideration is important and have several applications. There is however not much in terms of discussion nor analysis of the two experiments.
I find the contribution fairly significant but I lack some clarity in the presentation as well as in the experiments section.
I do not find the paper clearly written. The presentation can be improved in several chapters, such as the introduction and the method section.
The paper seem to be technically correct. I did not spot any errors.
General comments:
- Why is 2 a suitable size of S?
- Why use two by two encoding instead of stride for the VAE baseline? How does this affect the experiments?
- How is S_0 and the Z_0 prior set (initialized) in the experiments?
- It would improve the readability of the paper, not least for a broader audience, if more details are added on how the VAE baseline architecture differ from the proposed architecture.
- In the first experiment:
-- How many iterations does it take for your method to beat VAE?
-- What is the difference between the VAE basline and your approach that make VAE perform better than your approach initially (or even after a few iterations)?
-- What affect does the momentum formulation have on the convergence rate (number of iterations necessary to reach VAE and your methods result at t=10)?
- In the second experiment and Figure 5 in particular it was observed that some actions are clearly detected while others are not. It is mentioned that those that are not detected by the approach are more similar. In what sense are the actions more similar, which are the most prominent (from a humans perspective) such actions, what is making the model not detecting them and what can be done (within your approach) in order to improve or adjust the detection fidelity?
Please add more time labels under the time axis in Figure 4 and Figure 5. Also please annotate the figures at the time points where the action transitions are according to the ground truth.


Long Short-Term Memory (LSTM) is one of the most powerful sequence models. Despite the strong performance, however, it lacks the nice interpretability as in state space models. In this paper, we present a way to combine the best of both worlds by introducing State Space LSTM (SSL) models that generalizes the earlier work Zaheer et al. (2017) of combining topic models with LSTM. However, unlike Zaheer et al. (2017), we do not make any factorization assumptions in our inference algorithm. We present an efficient sampler based on sequential Monte Carlo (SMC) method that draws from the joint posterior directly. Experimental results confirms the superiority and stability of this SMC inference algorithm on a variety of domains.

This paper introduces a novel extension of the LSTM which incorporates stochastic inputs at each timestep. These stochastic inputs are themselves dependent on the LSTM state at the previous timestep. Considering the stochastic dependencies, this then yields a highly flexible non-Markov state space model, where the latent variable transitions are partially parameterized by an LSTM update.
Naturally, the challenges are then efficiently estimating parameters and performing inference over the latent states. Here, SMC (and conditional SMC / particle Gibbs) are used for inference over the latent states z. A particularly nice touch is that even when the LSTM model is used for the transitions in the latent space, so long as the conditional distributions p(z_t | z_{1:t-1}) are conjugate with the emission distribution then it is possible to compute the optimal forward filtering proposal distribution in closed form, as done for the conditionally Gaussian (with affine Gaussian observations) and conditionally multinomial models considered here. Note that this really is a special feature of the models under consideration, though: for example, if the emission distribution p(x_t | z_t) is instead a *nonlinear* Gaussian, then one would have to fall back to bootstrap proposals. This probably deserves some mention: equations (13) are not, generally, tractable to integrate or normalize.
I think this paper is missing a few necessary details on how the overall optimization algorithm proceeds, which I would like to see in an update. I understand that particle Gibbs updates (or SMC) are used to approximate the posterior distribution in a Monte Carlo EM algorithm. However, this does leave some questions:
1. For the M step, how are the \omega parameters (of the LSTM) handled in equation (8)? I understand that due to the particular models considered, maximum likelihood estimates of \phi can be found in closed form. However, that’s not the case for \omega. Is a gradient descent algorithm run to convergence? Or is a single gradient step taken, interleaved with a single PG update? Or something else?
2. How reliably does the algorithm as a whole converge? Monte Carlo EM does not in general have convergence guarantees of “standard” EM (i.e. each step is not guaranteed to monotonically improve the lower bound). This might be fine! But, I think requires a bit of discussion.
3. Is it necessary to include a replenishing operation (or independent MCMC steps) in the particle Gibbs algorithm? A known issue when running an iterated conditional SMC algorithm like this is that path degeneracy can make it very difficult for the PG kernel to mix well over the early time steps in the LSTM. Does this issue appear here? How many particles P are needed to efficiently mix, when considering time series of length T?


Humans rely on episodic memory constantly, in remembering the name of someone they met 10 minutes ago, the plot of a movie as it unfolds, or where they parked the car. Endowing reinforcement learning agents with episodic memory is a key step on the path toward replicating human-like general intelligence. We analyze why standard RL agents lack episodic memory today, and why existing RL tasks do not require it. We design a new form of external memory called Masked Experience Memory, or MEM, modeled after key features of human episodic memory. To evaluate episodic memory we define an RL task based on the common children's game of Concentration. We find that a MEM RL agent leverages episodic memory effectively to master Concentration, unlike the baseline agents we tested.

There are a number of attempts to add episodic memory to RL agents. A common approach is to use some sort of recurrent model with a model-free agent. This work follows this approach using what could be considered a memory network with a identity embedding function and tests on 'Concentration', a game which requires matching pairs of cards. They find their model outperforms a DNC and LSTM baselines.
The primary novelty is the use of an explicitly masked similarity function (with learned mask) and the concentration task, which requires more memory than, for example, common tasks adapted from the psychology literature such as the Morris watermaze or T-maze (although in the supervised setting tasks such as Omniglot are quite similar).
This work is well-communicated and cites relevant prior work. The author's should also be commended for agreeing to release their code on publication.
The primary weakness of this work its lack of novelty and lack of evidence of generalization of the approach, which limits its significance. The model introduced is a slight variant of memory networks. Additionally, the single task the model is tested on appears custom-designed to favor the model (see next paragraph). While the analysis of the weakness of cosine similarity is interesting, memory networks which compute separate embeddings for the 'label' (content-based label for retrieval) and memory content don't appear to suffer from the same issue as the DNC. They can store only retrieval-relevant content in the label and thus avoid issues with normalization.
The observation vector is stored directly in memory without passing through an embedding function, which in general seems quite limiting. However, in the constructed task the labels are low-dimensional, random vectors and there is no noise in the labels (i.e. two cards with the same label are labelled identically, rather the similarly). The author's mention avoiding naturalistic labels such as omniglot characters (closer to the real version of concentration) due to the possibility the agent might memorise the finite set of labels, however by choosing a large dataset and using a non-overlapping set of examples for the test set this probably could be avoided and would provide a more naturalistic test set.
The comparison with the DNC also seems designed to favor their model. DNC has write-gates, which might be relevant in a task with many irrelevant observations, but in this task are clearly going to impair learning. A memory network seems the more appropriate comparison. Its not clear why the DNC model used two different DNCs for computing the policy and value.
To demonstrate their model is of more general interest it would be necessary to try on a wider range of more naturalistic tasks and a comparison with model-free agents augmented with memory networks. Simply showing that a customized model can outperform on a single custom, synthetic task is insufficient to demonstrate that these changes are of wider interest.
Minor issues:
- colorblind seems an odd description for agents which cannot perceive the card face. Why not just 'blind'? colorblind would seem to imply partial perception of the card face.
- the observations of the environment are defined explicitly, but not the action space.


The digitization of data has resulted in making datasets available to millions of users in the form of relational databases and spreadsheet tables. However, a majority of these users come from diverse backgrounds and lack the programming expertise to query and analyze such tables. We present a system that allows for querying data tables using natural language questions, where the system translates the question into an executable SQL query. We use a deep sequence to sequence model in wich the decoder uses a simple type system of SQL expressions to structure the output prediction. Based on the type, the decoder either copies an output token from the input question using an attention-based copying mechanism or generates it from a fixed vocabulary. We also introduce a value-based loss function that transforms a distribution over locations to copy from into a distribution over the set of input tokens to improve training of our model. We evaluate our model on the recently released WikiSQL dataset and show that our model trained using only supervised learning significantly outperforms the current state-of-the-art Seq2SQL model that uses reinforcement learning.

This paper proposes a model for solving the WikiSQL dataset that was released recently.
The main issues with the paper is that its contributions are not new.
* The first claimed contribution is to use typing at decoding time (they don't say why but this helps search and learning). Restricting the type of the decoded tokens based on the programming language has already been done by the Neural Symbolic Machines of Liang et al. 2017. Then Krishnamurthy et al. expanded that in EMNLP 2017 and used typing in a grammar at decoding time. I don't really see why the authors say their approach is simpler, it is only simpler because the sub-language of sql used in wikisql makes doing this in an encoder-decoder framework very simple, but in general sql is not regular. Of course even for CFG this is possible using post-fix notation or fixed-arity pre-fix notation of the language as has been done by Guu et al. 2017 for the SCONE dataset, and more recently for CNLVR by Goldman et al., 2017.
So at least 4 papers have done that in the last year on 4 different datasets, and it is now close to being common practice so I don't really see this as a contribution.
* The authors explain that they use a novel loss function that is better than an RL based function used by Zhong et al., 2017. If I understand correctly they did not implement Zhong et al. only compared to their numbers which is a problem because it is hard to judge the role of optimization in the results.
Moreover, it seems that the problem they are trying to address is standard - they would like to use cross-entropy loss when there are multiple tokens that could be gold. the standard solution to this is to just have uniform distribution over all gold tokens and minimize the cross-entropy between the predicted distribution and the gold distribution which is uniform over all tokens. The authors re-invent this and find it works better than randomly choosing a gold token or taking the max. But again, this is something that has been done already in the context of pointer networks and other work like See  et al. 2017 for summarization and Jia et al., 2016 for semantic parsing.
* As for the good results - the data is new, so it is probable that numbers are not very fine-tuned yet so it is hard to say what is important and what not for final performance. In general I tend to agree that using RL for this task is probably unnecessary when you have the full program as supervision.


Model pruning seeks to induce sparsity in a deep neural network's various connection matrices, thereby reducing the number of nonzero-valued parameters in the model. Recent reports (Han et al., 2015a;Narang et al., 2017) prune deep networks at the cost of only a marginal loss in accuracy and achieve a sizable reduction in model size. This hints at the possibility that the baseline models in these experiments are perhaps severely over-parameterized at the outset and a viable alternative for model compression might be to simply reduce the number of hidden units while maintaining the model's dense connection structure, exposing a similar trade-off in model size and accuracy. We investigate these two distinct paths for model compression within the context of energy-efficient inference in resource-constrained environments and propose a new gradual pruning technique that is simple and straightforward to apply across a variety of models/datasets with minimal tuning and can be seamlessly incorporated within the training process. We compare the accuracy of large, but pruned models (large-sparse) and their smaller, but dense (small-dense) counterparts with identical memory footprint. Across a broad range of neural network architectures (deep CNNs, stacked LSTM, and seq2seq LSTM models), we find large-sparse models to consistently outperform small-dense models and achieve up to 10x reduction in number of non-zero parameters with minimal loss in accuracy.

Summary:
This paper presents a thorough examination of the effects of pruning on model performance.  Importantly, they compare the performance of "large-sparse" models (large models that underwent pruning in order to reduce memory footprint of model) and "small-dense" models, showing that "large-sparse" models typically perform better than the "small-dense" models of comparable size (in terms of number of non-zero parameters, and/or memory footprint).  They present results across a number of domains (computer vision, language modelling, and neural machine translation) and model types (CNNs, LSTMs).  They also propose a way of performing pruning with a pre-defined sparsity schedule, simplifying the pruning process in a way which works across domains.  They are able to show convincingly that pruning is an effective way of trading off accuracy for model size (more effective than simply reducing the size of model architecture), although there does come a point where too much sparsity degrades the model performance considerably; this suggests that pruning a medium size model to 80%-90% sparsity is likely better than pruning a larger model to >= 95% sparsity.
Review:
Quality: The quality of the work is high --- the experiments are extensive and thorough.  I would have liked to see "small-dense" vs. "large-sparse" comparisons on Inception (only large-sparse results are reported).
Clarity: The paper is clearly written, though there is room for improvement. For example, many of the results are presented in a redundant manner (in both tables and figures, where the table and figure are often not next to each other in the document).  Also, it is not clear in several cases exactly which training/heldout/test sets are used, and on which partition of the data the accuracies/BLEU scores/perplexities presented correspond to. A small section (before "Methods") describing the datasets/features in detail would be helpful.  Also, it would have probably been nice to explain all of the tasks and datasets early on, and then present all the results at once (NIT: include the plots in paper, and move the tables to an appendix).
Originality: Although the experiments are informative, the work as a whole is not very original.  The method proposed of using a sparsity schedule to perform pruning is simple and effective, but is a rather incremental contribution. The primary contribution of this paper is its experiments, which for the most part compare known methods.
Significance: The paper makes a nice contribution, though it is not particularly significant or surprising.  The primary observations are:
(1) large-sparse is typically better than small-dense, for a fixed number of non-zero parameters and/or memory footprint.
(2) There is a point at which increasing the sparsity percentage severely degrades the performance of the model, which suggests that there is a "sweet-spot" when it comes to choosing the model architecture and sparsity percentage which give the best performance (for a fixed memory footprint).
Result #1 is not very surprising, given that Han et al (2016) were able to show significant compression without loss in accuracy; thus, because one would expect a smaller dense model to perform worse than the large dense model, it would also perform worse than the large sparse model.
Result #2 had already been seen in Han et al (2016) (for example, in Figure 6).
Pros:
- Very thorough experiments across a number of domains
Cons:
- Methodological contributions are minor.
- Results are not surprising, and are in line with previous papers.


Published as a conference paper at ICLR 2018 VARIATIONAL CONTINUAL LEARNING
This paper develops variational continual learning (VCL), a simple but general framework for continual learning that fuses online variational inference (VI) and recent advances in Monte Carlo VI for neural networks. The framework can successfully train both deep discriminative models and deep generative models in complex continual learning settings where existing tasks evolve over time and entirely new tasks emerge. Experimental results show that VCL outperforms stateof-the-art continual learning methods on a variety of tasks, avoiding catastrophic forgetting in a fully automatic way.

This paper proposes a new method, called VCL, for continual learning. This method is a combination of the online variational inference for streaming environment with Monte Carlo method. The authors further propose to maintain a coreset which consists of representative data points from the past tasks. Such a coreset is used for the main aim of avoiding the catastrophic forgetting problem in continual learning. Extensive experiments shows that VCL performs very well, compared with some state-of-the-art methods.
The authors present two ideas for continual learning in this paper: (1) Combination of online variational inference and sampling method, (2) Use of coreset to deal with the catastrophic forgetting problem. Both ideas have been investigated in Bayesian literature, while (2) has been recently investigated in continual learning. Therefore, the authors seems to be the first to investigate the effectiveness of (1) for continual learning. From extensive experiments, the authors find that the first idea results in VCL which can outperform other state-of-the-art approaches, while the second idea plays little role.
The finding of the effectiveness of idea (1) seems to be significant. The authors did a good job when providing a clear presentation, a detailed analysis about related work, an employment to deep discriminative models and deep generative models, and a thorough investigation of empirical performance.
There are some concerns the authors should consider:
- Since the coreset plays little role in the superior performance of VCL, it might be better if the authors rephrase the title of the paper. When the coreset is empty, VCL turns out to be online variational inference [Broderich et al., 2013; Ghahramani & Attias, 2000]. Their finding of the effectiveness of online variational inference for continual learning should be reflected in the writing of the paper as well.
- It is unclear about the sensitivity of VCL with respect to the size of the coreset. The authors should investigate this aspect.
- What is the trade-off when the size of the coreset increases?


Reinforcement learning typically requires carefully designed reward functions in order to learn the desired behavior. We present a novel reward estimation method that is based on a finite sample of optimal state trajectories from expert demonstrations and can be used for guiding an agent to mimic the expert behavior. The optimal state trajectories are used to learn a generative or predictive model of the "good" states distribution. The reward signal is computed by a function of the difference between the actual next state acquired by the agent and the predicted next state given by the learned generative or predictive model. With this inferred reward function, we perform standard reinforcement learning in the inner loop to guide the agent to learn the given task. Experimental evaluations across a range of tasks demonstrate that the proposed method produces superior performance compared to standard reinforcement learning with both complete or sparse hand engineered rewards. Furthermore, we show that our method successfully enables an agent to learn good actions directly from expert player video of games such as the Super Mario Bros and Flappy Bird.

To speed up RL algorithms, the authors propose a simple method based on utilizing expert demonstrations. The proposed method consists in explicitly learning a prediction function that maps each time-step into a state. This function is learned from expert demonstrations. The cost of visiting a state is then defined as the distance between that state and the predicted state according to the learned function. This reward is then used in standard RL algorithms to learn to stick close to the expert's demonstrations. An on-loop variante of this method consists of learning a function that maps each state into a next state according to the expert, instead of the off-loop function that maps time-steps into states.
While the experiments clearly show the advantage of this method, this is hardly surprising or novel. The concept of encoding the demonstration explicitly in the form of a reward has been around for over a decade. This is the most basic form of teaching by demonstration. Previous works had used other models for generalizing demonstrations (GMMs, GPs, Kernel methods, neural nets etc..). This paper uses a three layered fully connected auto-encoder (which is not that deep of a model, btw) for the same purpose. The idea of using this model as a reward instead of directly cloning the demonstrations is pretty straightforward.
Other comments:
- Most IRL methods would work just fine by defining rewards on states only and ignoring actions all together. If you know the transition function, you can choose actions that lead to highly rewarding states, so you don't need to know the expert's executed actions.
- "We assume that maximizing likelihood of next step prediction in equation 1 will be globally optimized in RL". Could you elaborate more on this assumption? Your model finds rewards based on local state features, where a greedy (one-step planning) policy would reproduce the expert's demonstrations (if the system is deterministic). It does not compare the global performance of the expert to alternative policies (as is typically done in IRL).
- Related to the previous point: a reward function that makes every step of the expert optimal may not be always exist. The expert may choose to go to terrible states with the hope of getting to a highly rewarding state in the future. Therefore, the objective functions set in this paper may not be the right ones, unless your state description contains features related to future states so that you can incorporate future rewards in the current state (like in the reacher task, where a single image contains all the information about the problem). What you need is actually features that can capture the value function (like in DQN) and not just the immediate reward (as is done in IRL methods).
- What if in two different trajectories, the expert chooses opposite actions for the same state appearing in both trajectories? For example, there are two shortest paths to a goal, one starts with going left and another starts with going right. If you try to generate a state that minimizes the sum of distances to the two states (left and right ones), then you may choose to remain in the middle, which is suboptimal. You wouldn't have this issue with regular IRL techniques, because you can explain both behaviors with future rewards instead of trying to explain every action of the expert using only local state description.


THE ROLE OF MINIMAL COMPLEXITY FUNCTIONS IN UNSUPERVISED LEARNING OF SEMANTIC MAPPINGS
We discuss the feasibility of the following learning problem: given unmatched samples from two domains and nothing else, learn a mapping between the two, which preserves semantics. Due to the lack of paired samples and without any definition of the semantic information, the problem might seem ill-posed. Specifically, in typical cases, it seems possible to build infinitely many alternative mappings from every target mapping. This apparent ambiguity stands in sharp contrast to the recent empirical success in solving this problem. We identify the abstract notion of aligning two domains in a semantic way with concrete terms of minimal relative complexity. A theoretical framework for measuring the complexity of compositions of functions is developed in order to show that it is reasonable to expect the minimal complexity mapping to be unique. The measured complexity used is directly related to the depth of the neural networks being learned and a semantically aligned mapping could then be captured simply by learning using architectures that are not much bigger than the minimal architecture. Various predictions are made based on the hypothesis that semantic alignment can be captured by the minimal mapping. These are verified extensively. In addition, a new mapping algorithm is proposed and shown to lead to better mapping results.

The paper addresses the problem of learning mappings between different domains without any supervision. It belongs to the recent family of papers based on GANs.
The paper states three conjectures (predictions in the paper):
1. GAN are sufficient to learn « semantic mappings » in an unsupervised way, if the considered networks are small enough
2. Controlling the complexity of the network, i.e. the number of the layers, is crucial to come up with what is called « semantic » mappings when learning in an unsupervised way.
More precisely there is tradeoff to achieve between the complexity of the model and its simplicity. A rich model is required in order to minimize the discrepancy between the distributions of the domains, while a  not too complex model is necessary to avoid mappings that are not « meaningful ».
 To this aim, the authors  introduce a new notion of function complexity which can be seen as a proxy of Kolmogorov complexity. The introduced notion is very simple and intuitive and is defined as  the depth of a network  which is necessary to  implement the considered function.
Based on this definition, and assuming identifiability (i.e. uniqueness up to invariants), and for networks with Leaky ReLU activations,  the authors prove that if the number of mappings which preserve a degree of discrepancy (density preserving in the text) is small, then the  set of « minimal » mappings  of complexity C   that achieve the same degree of  discrepancy is also small.
This result is related to the third conjecture of the paper that is :
3. the number of the number of mappings which preserve a degree of discrepancy  is small.
The authors also prove a byproduct result stating that identifiability holds for Leaky ReLU networks with one hidden layer.
The paper  comes with a series of experiments to empirically « demonstrate » the conjectures.
The paper is well written. The different ideas are clearly stated and discussed, and hence open interesting questions and debates.
Some of these questions that need to be addressed IMHO:
- A critical general question: if the addressed problem is the alignment between e.g. images and not image generation, why not formalizing the problem as a similarity search one (using e.g. EMD or any other transport metric). The alignment task  hence reduces to computing a ranking from this similarity. I have the impression that we use a jackhammer to break a small brick here (no offence). But maybe that I’m missing something here.
- Several works consider the size and the depth of the network as hyper-parameters to optimize, and this is not new. What is the actual contribution of the paper w.r.t. to this body of work?
- It is considered that the GAN are trained without any problem, and therefore work in an optimal regime. But the training of the GAN is in itself a problem. How does this affect the paper statements and results?
- Are the results still valid for another measure of discrepancy based for instance on another measure, e.g. Wasserstein?
Some minor remarks :
- p3: the following sentence is not clear  «  Our hypothesis is that the lowest complexity small discrepancy mapping approximates the alignment of the target semantic function. »
- p6: $C^{\epsilon_0}_{A,B}$ is used (after Def. 2) before being defined.
- p7: build->built
Section II :
A diagram explaining  the different mappings (h_A, h_B, h_AB, etc.) and their spaces (D_A, D_B, D_Z) would greatly help the understanding.
Papers 's pros :
- clarity
- technical results
cons:
- doubts about the interest and originality
The authors provided detailed and convincing answers to my questions. I thank them for that.  My scores were changed accrodingly.


Here we study the problem of learning labels for large text corpora where each document can be assigned a variable number of labels. The problem is trivial when the label dimensionality is small and can be easily solved by a series of one-vs-all classifiers. However, as the label dimensionality increases, the parameter space of such one-vs-all classifiers becomes extremely large and outstrips the memory. Here we propose a latent variable model to reduce the size of the parameter space, but still efficiently learn the labels. We learn the model using spectral learning and show how to extract the parameters using only three passes through the training dataset. Further, we analyse the sample complexity of our model using PAC learning theory and then demonstrate the performance of our algorithm on several benchmark datasets in comparison with existing algorithms.

The paper addresses the problem of multi-label learning for text corpora and proposes to tackle the problem using tensor factorization methods. Some analysis and experimental results for the proposed algorithm are presented.
QUALITY: I find the quality of the results in this paper rather low. The proposed probabilistic model is defined ambiguously. The authors then look at joint probability distributions of co-occurence of two and three words, which gives a matrix and a tensor, respectively. They propose to match these matrix and tensor to their sample estimates and refer to such procedure as the moment matching method, which it is not. They then apply a standard two step technique from the moment matching literature consisting of whitening and orthogonal tensor factorization. However, in their case this does not have much statistical meaning. Indeed, whitening of the covariance matrix is usually justified by the scaling unidentifiability of the problem. In their case, the mathematics works because of the orthogonal unidentifiability of the square root of a matrix. Furthermore, the proposed sample estimators do not actually estimate densities they are dealing with (see, e.g., Eq. (16) and (17)). Their theoretical analysis seems like a straightforward extension of the analysis by Anandkumar, et al. (2012, 2014), however, I find it difficult to assess this analysis due to numerous ambiguities in the problem formulation and method development. This justifies my statement in the beginning of the paragraph.
CLARITY: The paper is not well written and, therefore, is difficult to assess. Many important details are omitted, the formulation of the model is self contradicting, the standard concepts and notations are sometimes abused, some statements are wrong. I provide some examples in the detailed comment below.
ORIGINALITY AND SIGNIFICANCE: The idea to apply tensor factorization approaches to the multi-label learning is novel up to my knowledge and is a pro of the paper. However, I have problems to find other pros in this submission because the clarity is quite low and in the present form there is no novelty in the proposed procedure. Moreover, the authors claim to work with densities, but end up estimating other quantities, which are not guaranteed to have the desirable form. They also emphasize the fact that there is the simplex constraint on the estimated parameters, but this constraint is completely ignored by the algorithm and, in general, won't be satisfied in practice. If think the authors should do some more work before this paper can be published.
DETAILED COMMENTS: Since I am quite critical about the paper, I point out some examples of drawbacks or flaws of this paper:
  - The proposed model (Section 2) is not well defined. In particular, the description in Section 2 is not sufficient to understand the proposed model; the plate diagram in Figure 2 is not consistent with the text. It is not mentioned how at least some conditional distributions behave (e.g., tokens given labels or states). The diagram in Fig. 1 does not help since it isn't consistent with the text (e.g. the elements of labels or states are not conditionally / independent). The model is very close to latent Dirichlet allocation by Blei, et al. (2003), but differences are not discussed.
  - The standard terminology is often abused. For example, the proposed approach is referred to as the method of moments when it is not. In Section 2.1, the authors aim to match joint distributions (not the moments) to their empirical approximations (which are also wrong; see below). The usage of tokes and documents is interchanged without any explanations.
  - The use of the whitening approach is not justified in their setting working with joint distributions of couples and triples and it has no statistical meaning. No explanation is provided. I would definitely not call this whitening.
  - In Section 2.2, the notation is not defined and is different from what is usually used in the literature. For example, Eq. (15) does not make much sense as is. One could guess from the context that they are talking about the eigenvectors of an orthogonal tensor as defined in, e.g. Anandkumar, et al. (2014).
  - In Section 3, the authors emphasize the fact that their parameters are constrained to the probability simplex, but this constraint is not ensured in the proposed algorithm (Alg. 1).
  - Importantly, the estimators of the matrix M_2 and tensor M_3 do not make much sense to me. For example, for estimating M_2 it would be reasonable to average over all word pairs, i.e. something like [M_2]_{ij} = 1/L \sum_{w_k \not = w_l} P(w_k = v_i, w_l = v_j), where L is the number of pairs. This is different from the expression in Eq. (16), which is just a rescaled non-central second moment. Similar issue is true for the order-3 estimator.
  - The factorization procedure does not ensure non-negativity of the obtained parameters and, therefore, the rescaling is not guaranteed to belong to the probability simplex. I could not find any explanations of this issue.
  - I explain good plots in the experimental section, potentially, by the fact that the authors do algorithmically something different from what they aim to do, because the estimators do not estimate the desired entities (i.e. are not consistent). The procedure looks to me quite similar to the procedure for LDA, hence the reasonable results. However, the authors do not justify their proposed method.


ALTERNATING MULTI-BIT QUANTIZATION FOR RECURRENT NEURAL NETWORKS
Recurrent neural networks have achieved excellent performance in many applications. However, on portable devices with limited resources, the models are often too large to deploy. For applications on the server with large scale concurrent requests, the latency during inference can also be very critical for costly computing resources. In this work, we address these problems by quantizing the network, both weights and activations, into multiple binary codes {−1, +1}. We formulate the quantization as an optimization problem. Under the key observation that once the quantization coefficients are fixed the binary codes can be derived efficiently by binary search tree, alternating minimization is then applied. We test the quantization for two well-known RNNs, i.e., long short term memory (LSTM) and gated recurrent unit (GRU), on the language models. Compared with the full-precision counter part, by 2-bit quantization we can achieve ∼16× memory saving and ∼6× real inference acceleration on CPUs, with only a reasonable loss in the accuracy. By 3-bit quantization, we can achieve almost no loss in the accuracy or even surpass the original model, with ∼10.5× memory saving and ∼3× real inference acceleration. Both results beat the exiting quantization works with large margins. We extend our alternating quantization to image classification tasks. In both RNNs and feedforward neural networks, the method also achieves excellent performance.

Revision:
The authors have addressed my concerns around the achievable speedup. I am increasing my score to 7.
Original Review:
The paper proposes a technique for quantizing neural network weight matrices by representing columns of weight matrices as linear combinations of binary (+1/-1) vectors. Given a weight vector, the paper proposes an alternating optimization procedure to estimate the set of k binary vectors and coefficients that best represent (in terms of MSE) the original vector. This yields a k-bit quantization. First, the coefficients/binary weights are initialized using a greedy procedure proposed in prior work. Then, the binary weights are updated using a clever binary search procedure, followed by updates to the coefficients. Experiments are conducted in an RNN context for some language modeling tasks.
The paper is relatively easy to read, and the technique is clearly explained. The technique is as far as I can tell novel, and does seem to represent an improvement over existing approaches for similar multi-bit quantization strategies.
I have a few questions/concerns. First, I am quite skeptical of many of the speedup calculations: These are rather delicate to do properly, and depend on the specific instructions available, SIMD widths, the number of ALUs present in a core, etc. All of these can easily shift numbers around by a factor of 2-8x. Without an implementation in hand, comparing against a well-optimized reference GEMM for full floating point, it's not clear how much faster this approach really would be in practice. Also, the online quantization of activations doesn't seem to be factored into the speedup calculations, and no benchmarks are provided demonstrating how fast the quantization is (unless I'm missing something). This is concerning since the claimed speedups aren't possible without the online quantization of actiations.
It would have been nice to have more discussion of/comparison with other approaches capable of 2-4 bit quantization, such as some of the recent work on ternary quantization, product quantization approaches, or at least scalar (per-dimension) k-means (non-uniform quantization).
Finally, the experiments are reasonable, but the choice of RNN setting isn't clear to me. It would have been easier to compare to prior work if the experiments also included some standard image classification tasks (e.g., CIFAR10).
Overall though, I think the paper does just enough to warrant acceptance.


GENERATIVE MODELS OF VISUALLY GROUNDED IMAGINATION
It is easy for people to imagine what a man with pink hair looks like, even if they have never seen such a person before. We call the ability to create images of novel semantic concepts visually grounded imagination. In this paper, we show how we can modify variational auto-encoders to perform this task. Our method uses a novel training objective, and a novel product-of-experts inference network, which can handle partially specified (abstract) concepts in a principled and efficient way. We also propose a set of easy-to-compute evaluation metrics that capture our intuitive notions of what it means to have good visual imagination, namely correctness, coverage, and compositionality (the 3 C's). Finally, we perform a detailed comparison of our method with two existing joint image-attribute VAE methods (the JMVAE method of Suzuki et al. (2017) and the BiVCCA method of Wang et al. (2016b)) by applying them to two datasets: the MNIST-with-attributes dataset (which we introduce here), and the CelebA dataset (Liu et al., 2015).

This paper presented a multi-modal extension of variational autoencoder (VAE) for the task "visually grounded imagination."  In this task,  the model learns a joint embedding of the images and the attributes. The proposed model is novel but incremental comparing to existing frameworks.  The author also introduced new evaluation metrics to evaluate the model performance concerning correctness, coverage, and compositionality.
Pros:
1. The paper is well-written, and the contribution (both the model and the evaluation metric) potentially can to be very useful in the community.
2. The discussion comparing the related work/baseline methods is insightful.
3. The proposed model addresses many important problems, such as attribute learning, disentanged representation learning, learning with missing values, and proper evaluation methods.
Cons/questions:
1. The motivation of the model choice of q is not clear.
Comparing to BiVCCA, apart from the differences that the author discussed, a big difference is the choice of q.  BiVCCA uses two inference networks q(z|x) and q(z|y), while the proposed method uses three. q(z|x), q(z|y), and q(z|x,y).  How does such model choice affect the final performance?
2. Baselines are not necessarily sufficient.
The paper compared the vanilla version of BiVCCA but not the one with factorized representation version. In the original VAECCA paper, the extension of using factorized representation (private and shared) improved the performance]. The author should also compare this extension of VAECCA.
3. Some details are not clear.
a) How to set/learn the scaling parameter \lambda_y and \beta_y? If it is set as hyper-parameter, how does the performance change concerning them?
b) Discussion of the experimental results is not sufficient. For example, why JMVAE performs much better than the proposed model when all attributes are given.  What is the conclusion from Figure 4(b)? The JMVAE seems to generate more diverse (better coverage) results which are not consistent with the claims in the related work.  The same applies to figure 5.


We propose a novel method that makes use of deep neural networks and gradient decent to perform automated design on complex real world engineering tasks. Our approach works by training a neural network to mimic the fitness function of a design optimization task and then, using the differential nature of the neural network, perform gradient decent to maximize the fitness. We demonstrate this methods effectiveness by designing an optimized heat sink and both 2D and 3D airfoils that maximize the lift drag ratio under steady state flow conditions. We highlight that our method has two distinct benefits over other automated design approaches. First, evaluating the neural networks prediction of fitness can be orders of magnitude faster then simulating the system of interest. Second, using gradient decent allows the design space to be searched much more efficiently then other gradient free methods. These two strengths work together to overcome some of the current shortcomings of automated design.

This paper introduces an appealing application of deep learning: use a deep network to approximate the behavior of a complex physical system, and then design optimal devices (eg airfoil shapes) by optimizing this network with respect to its inputs. Overall, this research direction seems fruitful, both in terms of different applications and in terms of extra machine learning that could be done to improve performance, such as ensuring that the optimization doesn't leave the manifold of reasonable designs.
 On one hand, I would suggest that this work would be better placed in an engineering venue focused on fluid dynamics. On the other hand, I think the ICLR community would benefit from about the opportunities to work on problems of this nature.
 =Quality=
The authors seem to be experts in their field. They could have done a better job explaining the quality of their final results, though. It is unclear if they are comparing to strong baselines.
=Clarity=
The overall setup and motivation is clear.
=Originality=
This is an interesting problem that will be novel to most member of the ICLR community. I think that this general approach deserves further attention from the community.
=Major Comments=
* It's hard for me to understand if the performance of your method is actually good. You show that it outperforms simulated annealing. Is this the state of the art? How would an experienced engineer perform if he or she just sat down and drew the shape of an airfoil, without relying on any computational simulation at all?
* You can afford to spend lots of time interacting with the deep network in order to optimize it really well with respect to the inputs. Why not do lots of random initializations for the optimization? Isn't that a good way to help avoid local optima?
* I'd like to see more analysis of the reliability of your deep-network-based approximation to the physics simulator. For example, you could evaluate the deep-net-predicted drag ratio vs. the simulator-predicted drag ratio at the value of the parameters corresponding to the final optimized airfoil shape. If there's a gap, it suggests that your NN approximation might have not been that accurate.
=Minor Comments=
* "We also found that adding a small amount of noise too the parameters when computing gradients helped jump out of local optima"
Generally, people add noise to the gradients, not the values of the parameters. See, for example, uses of Langevin dynamics as a non-convex optimization method.
* You have a complicated method for constraining the parameters to be in [-0.5,0.5]. Why not just enforce this constraint by doing projected gradient descent? For the constraint structure you have, projection is trivial (just clip the values).
 * "The gradient decent approach required roughly 150 iterations to converge where as the simulated annealing approach needed at least 800."
This is of course confounded by the necessary cost to construct the training set, which is necessary for the gradient descent approach. I'd point out that this construction can be done in parallel, so it's less of a computational burden.
* I'd like to hear more about the effects of different parametrizations of the airfoil surface. You optimize the coefficients of a polynomial. Did you try anything else?
* Fig 6: What does 'clean gradients' mean? Can you make this more precise?
* The caption for Fig 5 should explain what each of the sub figures is.


Data structured in form of overlapping or non-overlapping sets is found in a variety of domains, sometimes explicitly but often subtly. For example, teams, which are of prime importance in social science studies are "sets of individuals"; "item sets" in pattern mining are sets; and for various types of analysis in language studies a sentence can be considered as a "set or bag of words". Although building models and inference algorithms for structured data has been an important task in the fields of machine learning and statistics, research on "set-like" data still remains less explored. Relationships between pairs of elements can be modeled as edges in a graph. However, modeling relationships that involve all members of a set, a hyperedge is a more natural representation for the set. In this work, we focus on the problem of embedding hyperedges in a hypergraph (a network of overlapping sets) to a low dimensional vector space. We propose a probabilistic deep-learning based method as well as a tensor-based algebraic model, both of which capture the hypergraph structure in a principled manner without loosing set-level information. Our central focus is to highlight the connection between hypergraphs (topology), tensors (algebra) and probabilistic models. We present a number of interesting baselines, some of which adapt existing node-level embedding models to the hyperedge-level, as well as sequence based language techniques which are adapted for set structured hypergraph topology. The performance is evaluated with a network of social groups and a network of word phrases. Our experiments show that accuracy wise our methods perform similar to those of baselines which are not designed for hypergraphs. Moreover, our tensor based method is quiet efficient as compared to deep-learning based auto-encoder method. We therefore, argue that we have proposed more general methods which are suited for hypergraphs (and therefore also for graphs) while maintaining accuracy and efficiency.

The paper studies different methods for defining hypergraph embeddings, i.e. defining vectorial representations of the set of hyperedges of a given hypergraph. It should be noted that the framework does not allow to compute a vectorial representation of a set of nodes not already given as an hyperedge. A set of methods is presented : the first one is based on an auto-encoder technique ; the second one is based on tensor decomposition ; the third one derives from sentence embedding methods. The fourth one extends over node embedding techniques and the last one use spectral methods. The two first methods use plainly the set structure of hyperedges. Experimental results are provided on semi-supervised regression tasks. They show very similar performance for all methods and variants. Also run-times are compared and the results are expected. In conclusion, the paper gives an overview of methods for computing hypernode embeddings. This is interesting in its own. Nevertheless, as the target problem on hypergraphs is left unspecified, it is difficult to infer conclusions from the study. Therefore, I am not convinced that the paper should be published in ICLR'18.
* typos
* Recent surveys on graph embeddings have been published in 2017 and should be cited as "A comprehensive survey of graph embedding ..." by Cai et al
* Preliminaries. The occurrence number R(g_i) are not modeled in the hypergraphs. A graph N_a is defined but not used in the paper.
* Section 3.1. the procedure for sampling hyperedges in the lattice shoud be given. At least, you should explain how it is made efficient when the number of nodes is large.
* Section 3.2. The method seems to be restricted to cases where the cardinality of hyperedges can take a small number of values. This is discussed in Section 3.6 but the discussion is not convincing enough.
* Section 3.3 The term Sen2vec is not common knowledge
* Section 3.3 The length of the sentences depends on the number of permutations of $k$ elements. How can you deal with large k ?
* Section 3.4 and Section 3.5. The methods proposed in these two sections should be related with previous works on hypergraph kernels. I.e. there should be mentions on the clique expansion and star expansion of hypergraphs. This leads to the question why graph embeddings methods on these expansions have not be considered in the paper.
* Section 4.1. Only hyperedeges of cardinality in [2,6] are considered. This seems a rather strong limitation and this hypothesis does not seem pertinent in many applications.
* Section 4. For online multi-player games, hypernode embeddings only allow to evaluate existing teams, i.e. already existing as hyperedges in the input hypergraph. One of the most important problem for multi-player games is team making where team evaluation should be made for all possible teams.
* Section 5. Seems redundant with the Introduction.