The problem of exploration in reinforcement learning is well-understood in the tabular case and many sample-efficient algorithms are known. Nevertheless, it is often unclear how the algorithms in the tabular setting can be extended to tasks with large state-spaces where generalization is required. Recent promising developments generally depend on problem-specific density models or handcrafted features. In this paper we introduce a simple approach for exploration that allows us to develop theoretically justified algorithms in the tabular case but that also give us intuitions for new algorithms applicable to settings where function approximation is required. Our approach and its underlying theory is based on the substochastic successor representation, a concept we develop here. While the traditional successor representation is a representation that defines state generalization by the similarity of successor states, the substochastic successor representation is also able to implicitly count the number of times each state (or feature) has been observed. This extension connects two until now disjoint areas of research. We show in traditional tabular domains (RiverSwim and SixArms) that our algorithm empirically performs as well as other sample-efficient algorithms. We then describe a deep reinforcement learning algorithm inspired by these ideas and show that it matches the performance of recent pseudo-count-based methods in hard exploration Atari 2600 games.

Being familiar but not an expert in reinforcement learning, my review will focus on the overall soundness of the proposed method
Summary:
The authors are interested in the problem of sample efficiency in reinforcement learning, i.e. how to learn a policy achieving good performance (discounted reward) in a RL setting using as little interaction with the environment as possible.
To do this the authors propose to learn a policy in a new environment where the reward has changed: an exploration bonus is added to the reward that should bias the agent towards the least frequently visited states.
The algorithms proposed throughout the manuscript are extensions of a two-part algorithm of the following flavour: 1) An estimate of visitation count is done in an online fashion using a modified version of the successor representation (SR). 2) This estimates parametrizes the exploration bonus of the environment . Both learning algorithms are optimized together.
This initial algorithm is fairly simple in its description and builds on well established ideas in RL. The authors then ‘evaluate the effectiveness of the proposed exploration bonus in a standard model-based algorithm’ against other baselines. They do explain how the model is learned, but not how the policy is optimized.
The remainder of the manuscript applies the same idea to different settings.
For large state spaces, The SR expected visits are learned using TD along with state action value functions. The counts of visitations are replaced by features that are also learned.
Overall, the manuscript is rather confusing.
The SSR theorem is stated (with no real intuition and the actual bounds on n(s) left for the reader to derive). It is not well motivated. Why would we want expected counts and not the discounted version?
Then the remainder of the paper actually makes no use of this theorem, but only use it as a distant inspiration. Tentative connections are made such as TD underestimating SR thus leading to a result more akin to SSR, which is highly speculative. It is also irrelevant since features are learned anyway.
The final proposed architecture has many additions to a simple DQN (the reconstruction + the exploration bonus + the MMC). This makes it difficult to understand what the contribution of the exploration bonus is.
It does not help that results are manually extracted from histograms found in  papers.
Overall, although the intuition is interesting (though not so new).
The overall motivation and structure of this manuscript makes think it does not match the standards of ICLR for publication


SAMPLE EFFICIENT IMITATION LEARNING FOR CON- TINUOUS CONTROL
The goal of imitation learning (IL) is to enable a learner to imitate expert behavior given expert demonstrations. Recently, generative adversarial imitation learning (GAIL) has shown significant progress on IL for complex continuous tasks. However, GAIL and its extensions require a large number of environment interactions during training. In real-world environments, the more an IL method requires the learner to interact with the environment for better imitation, the more training time it requires, and the more damage it causes to the environments and the learner itself. We believe that IL algorithms could be more applicable to real-world problems if the number of interactions could be reduced. In this paper, we propose a model-free IL algorithm for continuous control. Our algorithm is made up mainly three changes to the existing adversarial imitation learning (AIL) methods -(a) adopting off-policy actor-critic (Off-PAC) algorithm to optimize the learner policy, (b) estimating the state-action value using off-policy samples without learning reward functions, and (c) representing the stochastic policy function so that its outputs are bounded. Experimental results show that our algorithm achieves competitive results with GAIL while significantly reducing the environment interactions.

The paper proposes a method for imitation learning via inverse reinforcement learning based on a specific modeling of the reward. It is modeled as the log probability of a state action pair to belong to the expert policy. It models this distribution as a Bernoulli one and thus it reduces the IRL problem to a classification task. The global method also uses an off-policy algorithm to learn the value function of the current agent policy to improve sample efficiency. The method is tested on a set of continuous control tasks such as walker, hopper or humanoid.
I think the paper has several flaws. First, I found the paper not very well written and organized. It is hard to read. It uses some terminology in a way that is different from the rest of the littérature (such as Q-learning as learning the Q-function of the expert policy instead of using the  optimal Bellman operator (even if the expert is supposed to be optimal)). I also think that the related work section is missing a lot of important refs because it really focuses on recent papers while imitation learning has a long history.
Yet, my main concern is that the proposed method seems to reduce to a classification problem to me and is likely to suffer from the same issues than the supervised learning method (AKA behavior cloning). It probably overfits a lot and there is nothing in the experiments that shows how robust is the method to perturbations. In a discrete world, this method would ideally place a reward of 1 in every state visited by the expert and 0 elsewhere which is very likely to overfit and result in unstable behaviors in the presence of noise etc. I would like to see experiments showing robustness.
The experiments are also a bit strange since the learning is stopped early for the proposed method. Is it because the learning is unstable ?


A noisy and diverse demonstration set may hinder the performances of an agent aiming to acquire certain skills via imitation learning. However, state-of-the-art imitation learning algorithms often assume the optimality of the given demonstration set. In this paper, we address such optimal assumption by learning only from the most suitable demonstrations in a given set. Suitability of a demonstration is estimated by whether imitating it produce desirable outcomes for achieving the goals of the tasks. For more efficient demonstration suitability assessments, the learning agent should be capable of imitating a demonstration as quick as possible, which shares similar spirit with fast adaptation in the meta-learning regime. Our framework, thus built on top of Model-Agnostic Meta-Learning, evaluates how desirable the imitated outcomes are, after adaptation to each demonstration in the set. The resulting assessments hence enable us to select suitable demonstration subsets for acquiring better imitated skills. The videos related to our experiments are available at https://sites.google.com/view/deepdj.

The problem is described as doing imitation learning from a set of demonstrations that includes useless behavior. Authors propose a method that is an extension of MAML which selects the useful demonstrations by their provided performance gains at the meta-training time.
Paper clearly demonstrates significant amount of work. Pieces from different modern method implementations (like MAML, TRPO, GAIL, multiple custom loss functions) are combined to work together. Also four custom task domains are implemented with MuJoCo. Finally decent amount of experiments are run.
Unfortunately, all that hard work can't be justified by the motivations that are very artificial in details and by the final task performance.
First of all, the setup includes small number of demonstrations where almost none of them are seemingly successful (judging by the videos). This is a very artificial setting that does not reflect the actual imitation learning problems like demonstrations provided by humans. There, normally the problem is either dealing with small number of demonstrations that are all typically successful but similarly suboptimal or dealing with small number of distinct demonstrators which are again successful but have significantly different styles. In the summary video, authors motivate the case by learning from sources like internet videos, but that setting is also very far away from the case here, because such video collections are much larger but more importantly the main problem is dealing with the third person perspective. All the experiments here is done from first person demonstrations (in one case with a slightly different body).
Biggest caveat of the paper is that it is promoted as a purely imitation learning method. Yet everything hinges on the existence of a "task heuristic" which is nothing but a reward function. If such function exists, all these first person demonstrations can be judged and selected based on that function. There would be no need for a complicated meta-learning scheme. Also the task could be trained directly on that reward by reinforcement learning. Also computation of this heuristic function is not specified. As far as I understand, it is a different quantity than the sparse "Task Success Reward".
Finally, the final performance of the imitating agents are far from accomplishing the task, though they show some resemblance to the imitation behavior. This is not all that surprising, given small number of demonstrations and high dimensional control problems.
Overall, the details of the setup makes the problem very artificial, the final performance is not impressive. Method is an amalgamation of bunch other recent work, which gives the impression of creating complexity for its own sake. I do not think that this method will be useful for moving the field forward and produce any impact.


THE ROLE OF OVER-PARAMETRIZATION IN GENERALIZATION OF NEURAL NETWORKS
Despite existing work on ensuring generalization of neural networks in terms of scale sensitive complexity measures, such as norms, margin and sharpness, these complexity measures do not offer an explanation of why neural networks generalize better with over-parametrization. In this work we suggest a novel complexity measure based on unit-wise capacities resulting in a tighter generalization bound for two layer ReLU networks. Our capacity bound correlates with the behavior of test error with increasing network sizes (within the range reported in the experiments), and could partly explain the improvement in generalization with over-parametrization. We further present a matching lower bound for the Rademacher complexity that improves over previous capacity lower bounds for neural networks.

Let me start by apologizing for the delayed review - in fact I was asked today to replace an earlier assigned reviewer. Hopefully the clarifications I request won't be too time consuming to meet the deadline coming up.
###
First of all, the problem which the authors are attempting to answer is quite important: the effect of over-parametrization is not well understood on a theoretical level. As the paper illustrate, 2-layer networks are already capable of generalizing while being over-parameterized, therefore justifying their setting.
Next this paper motivates the study of complexity quantities that tend to decrease with the number of parameters, in particular figure 3 motivates the conjecture that the complexity measure in Theorem 2 can control generalization error. The paper also does a great job comparing related work, motivating their results.
###
At this point, I would like to request a couple of clarifications in the proofs. Perhaps it's due to the fact that I only spent a day reading, but at least I think we could improve on its readability. Regardless, I currently do not yet trust a couple of the proofs, and I believe the acceptance of this paper should be conditioned on confirming the correctness of these proofs.
(1) Let's start with Lemma 10. In the middle equation block, we obtain a bound
  \| alpha^prime \|_p^p <= beta^p ( 1 + D/K )
and the proof concludes alpha^prime is in Q. However this cannot be the case for all alpha^prime.
Consider x=0 which is in S_{p, beta}^D, then we have alpha^prime = 0 as well. In the definition of Q, we require all the j's to sum up to K+D, which is not met here.
At the same time, the next claim
  \| alpha \|_2 <= D^{1/2 - 1/p} \| alpha^prime \|_p
does not seem to follow from the above calculations. In particular, alpha^prime seems to be defined with respect to an x in S_{p, beta}, however in this case we did not specify such an x. Perhaps did you mean there exist such an alpha^prime?
(2) In the proof of Theorem 3, there is an important inequality needed to complete the proof
  max{ <s, f_i> , <s, -f_i> } >= 1/2 * ( <s, [f_i]_+> + <s, [-f_i]_+> )
Perhaps I am missing something obvious, but I believe this inequality fails when we choose s as a constant vector, and f_i to have the same number of positive and negative signs (which is possible in a Hadamard matrix). In this case, the left hand side should be equal to zero, where as the right hand side will be positive.
###
To summarize, if these proofs can be confirmed, I believe this paper would have made significant contribution to the problem of over-parametrization in deep learning, and of course should be accepted.
###
I corrected several typos and found minor issues as I read, perhaps this will be useful to improve readability as well.
Page 13, proof of Lemma 8
  - after the V_0 term is separated, there is a sup over \|V_0\|_F <= r in the expectation, which should be \|V-V_0\|_F <= r instead.
Page 14, Lemma 9
  - the lemma did not define rho_{ij} in the statement
Page 15, proof of Lemma 9
  - in equation (12), there is an x_y vector that should x_t
Page 15, proof of Theorem 1
  - while I eventually figured it out, it's unclear how Lemma 8 is applied here. Perhaps one more step identifying the exact matrices in the statement of Lemma 8 will be helpful to future readers, and maybe explain where the sqrt(2) factor come from as well.
Page 16, proof of Lemma 10
  - in the beginning of the proof, to stay consistent with the notation, we should replace S_{p, beta} with S_{p, beta}^D
  - I believe the cardinality of Q should be (K + D - 1) choose (D - 1), as we need to choose positive j's to sum up to (K+D) in the definition of Q. This reduces down to the problem of choosing natural numbers j's summing K, which is (K+D-1) choose (D-1). Consider the stack exchange post here:
https://math.stackexchange.com/questions/919676/the-number-of-integer-solutions-of-equations
Page 16, proof and statement of Lemma 11
  - I believe in the first term, the factor should be m instead of sqrt(m). I think the mistake happened when applying the union bound, as it should only affect the term containing delta
Page 17, Lemma 12
  - same as Lemma 11, we should have m instead of sqrt(m)
Page 18, proof of Theorem 3
  - at the bottom the statement "F is orthogonal" does not imply the norm is less than 1, but rather we should say "F is orthonormal"
Page 19, proof of Theorem 3
  - at the top, "we will omit the index epsilon" should be "xi" instead
  - in the final equation block, we have the Rademacher complexity of F_{W_2}, instead it should be F_{W^prime}


TEMPORAL DIFFERENCE VARIATIONAL AUTO-ENCODER
To act and plan in complex environments, we posit that agents should have a mental simulator of the world with three characteristics: (a) it should build an abstract state representing the condition of the world; (b) it should form a belief which represents uncertainty on the world; (c) it should go beyond simple step-by-step simulation, and exhibit temporal abstraction. Motivated by the absence of a model satisfying all these requirements, we propose TD-VAE, a generative sequence model that learns representations containing explicit beliefs about states several steps into the future, and that can be rolled out directly without single-step transitions. TD-VAE is trained on pairs of temporally separated time points, using an analogue of temporal difference learning used in reinforcement learning.

There are several ingredients in this paper that I really liked. For example, (1) the notion that an agent should build a deterministic function of the past which implicitly captures the belief (the uncertainty or probability distribution about the state), by opposition for example to sampling trajectories to capture uncertainty, (2) modelling the world's dynamic in a learned encoded state-space (by opposition to the sensor space), (3) instead of modeling next-step probabilities p(z(t+1)|z(t)), model 'jumpy transitions' p(z(t+delta)|z(t)) to avoid unrolling at the finest time scale.
Now for the weak points:
(a) the justification for the training loss was not completely clear to me, although I can see that it has a variational flavor
(b) there is no discussion of the issue that we can't get a straightforward decomposition of the joint probability over the data sequence according to next-step probabilities via the chain rule of probabilities, so we don't have a clear way to compare the TD-VAE models with jumpy predictions against other more traditional models
(c) none of the experiments make comparisons against previously published models and quantitative results (admittedly because of (b) this may not be easy).
So I believe that the authors are onto a great direction of investigation, but the execution of the paper could be improved.


M 3 RL: MIND-AWARE MULTI-AGENT MANAGEMENT REINFORCEMENT LEARNING
Most of the prior work on multi-agent reinforcement learning (MARL) achieves optimal collaboration by directly learning a policy for each agent to maximize a common reward. In this paper, we aim to address this from a different angle. In particular, we consider scenarios where there are self-interested agents (i.e., worker agents) which have their own minds (preferences, intentions, skills, etc.) and can not be dictated to perform tasks they do not want to do. For achieving optimal coordination among these agents, we train a super agent (i.e., the manager) to manage them by first inferring their minds based on both current and past observations and then initiating contracts to assign suitable tasks to workers and promise to reward them with corresponding bonuses so that they will agree to work together. The objective of the manager is to maximize the overall productivity as well as minimize payments made to the workers for ad-hoc worker teaming. To train the manager, we propose Mind-aware Multi-agent Management Reinforcement Learning (M 3 RL), which consists of agent modeling and policy learning. We have evaluated our approach in two environments, Resource Collection and Crafting, to simulate multi-agent management problems with various task settings and multiple designs for the worker agents. The experimental results have validated the effectiveness of our approach in modeling worker agents' minds online, and in achieving optimal ad-hoc teaming with good generalization and fast adaptation.

This paper studies the problem of coordinating many strategic agents with private valuation to perform a series of common goals. The algorithm designer is a manager who can assign goals to various agents but cannot see their valuation or control them explicitly. The manager has a utility function for various goals and wants to maximize the total revenue. The abstract problem is well-motivated and significant and is an entire branch of study called algorithmic mechanism design. However often many assumptions have to be made to make the problem mathematically tractable. In this paper, the authors take an empirical approach by designing an RL framework that efficiently maximizes rewards across many episodes. Overall I find the problem interesting, well-motivated. The paper is well-written and contains significant experiments to support its point. However, I do not have the necessary background in the related literature to assess the significance of the methods proposed compared to prior work and thus would refrain from making a judgment on the novelty of this paper in terms of methodology. Here are some of my comments/questions to the author on this paper.
(1) I want to clarify how the skills of the agents play a role in the problem setup. Does it show up in the expression for the manager's reward? In particular, does it affect the Indicator for whether a goal is completed Eq. (2) via a process that need not be explicitly modeled but can be observed via a feedback of whether or not the goal is completed? So in the case of resource collection example, the skill set is a binary value for each resource, whether it can be collected or not?
(2) Related to the first point, the motivation for modeling the agents as maximizing their utility is the assumption that agents do not know their skills. I am wondering, is this really justified? Over the course of episodes, can the agents learn their skills based on the relationship between their intention and the goals they achieve? In the resource collection example, when they reach a resource and are not able to collect it, they understand that they do not have the corresponding skill. Is there a way to extrapolate the results from this paper to such a setting?
(3) I am slightly concerned about the sample complexity of keeping track of the probability of worker i finishing goal g within t steps with a bonus b. This scales linearly in parameters which usually would be large (such as the number of time-steps). Are there alternate ways to overcome maintaining the UCB explicitly, especially for the number of time-steps?
Some minor comments on the presentation.
(1) What are the units for rewards in the plots? Is it the average per episode reward? It would be good to mention this in the caption.
(2) There are a few typos in the paper. Some I could catch was,
- Last line in Page 5: "quantitative" -> "quantity"
- Page 8: skills nad preferences -> skills and preferences
- Page 8: For which we combining -> for which we combine


UNDERSTANDING COMPOSITION OF WORD EMBED- DINGS VIA TENSOR DECOMPOSITION
Word embedding is a powerful tool in natural language processing. In this paper we consider the problem of word embedding composition -given vector representations of two words, compute a vector for the entire phrase. We give a generative model that can capture specific syntactic relations between words. Under our model, we prove that the correlations between three words (measured by their PMI) form a tensor that has an approximate low rank Tucker decomposition. The result of the Tucker decomposition gives the word embeddings as well as a core tensor, which can be used to produce better compositions of the word embeddings. We also complement our theoretical results with experiments that verify our assumptions, and demonstrate the effectiveness of the new composition method.

The authors consider the use of tensor approximations to more accurately capture syntactical aspects of compositionality for word embeddings. Given two words a and b, when your goal is to find a word whose meaning is roughly that of the phrase (a,b), a standard approach to to find the word whose embedding is close to the sum of the embeddings, a + b. The authors point out that others have observed that this form of compositionality does not leverage any information on the syntax of the pair (a,b), and the propose using a tensor contraction to model an additional multiplicative interaction between a and b, so they propose finding the word whose embedding is closest to a + b + T*a*b, where T is a tensor, and T*a*b denotes the vector obtained by contracting a and b with T. They test this idea specifically on the use-case where (a,b) is an adjective,noun pair, and show that their form of compositionality outperforms weighted versions of additive compositionality in terms of spearman and pearson correlation with human judgements. In their model, the word embeddings are learned separately, then the tensor T is learned by minimizing an objective whose goal is to minimize the error in predicting observed trigram statistics. The specific objective comes from a nontrivial tensorial extension of the original matricial RAND-WALK model for learning word embeddings.
The topic is fitting with ICLR, and some attendees will find the results interesting. As in the original RAND-WALK paper, the theory is interesting, but not the main attraction, as it relies on strong generative modeling assumptions that essentially bake in the desired results. The main appeal is the idea of using T to model syntactic interactions, and the algorithm for learning T. Given that the main attraction of the paper is the potential for more performant word embeddings, I do not believe the work will have wide appeal to ICLR attendees, because no evidence is provided that the features from the learned tensor, say [a, b, T*a*b], are more useful in downstream applications than [a,b] (one experiment in sentiment analysis is tried in the supplementary material with no compelling difference shown).
Pros:
- theoretical justification is given for their assumption that the higher-order interactions can be modeled by a tensor
- the tensor model does deliver some improvement over linear composition on noun-adjective pairs when measured against human judgement
Cons:
- no downstream applications are given which show that these higher order interactions can be useful for downstream tasks.
- the higher-order features T*a*b are useful only when a is noun and b is an adjective: why not investigate using T to model higher-order interaction for all (a,b) pairs regardless of the syntactic relationships between a and b?
- comparison should be made to the linear composition method in the Arora, Liang, Ma ICLR 2017 paper
Some additional citations:
- the above-mentioned ICLR paper provides a performant alternative to unweighted linear composition
- the 2017 Gittens, Achlioptas, Drineas ACL paper provides theory on the linear composition of some word embeddings


The architecture search methods for convolutional neural networks (CNNs) have shown promising results. These methods require significant computational resources, as they repeat the neural network training many times to evaluate and search the architectures. Developing the computationally efficient architecture search method is an important research topic. In this paper, we assume that the structure parameters of CNNs are categorical variables, such as types and connectivities of layers, and they are regarded as the learnable parameters. Introducing the multivariate categorical distribution as the underlying distribution for the structure parameters, we formulate a differentiable loss for the training task, where the training of the weights and the optimization of the parameters of the distribution for the structure parameters are coupled. They are trained using the stochastic gradient descent, leading to the optimization of the structure parameters within a single training. We apply the proposed method to search the architecture for two computer vision tasks: image classification and inpainting. The experimental results show that the proposed architecture search method is fast and can achieve comparable performance to the existing methods.

The authors propose to formulate the neural network architecture as a collection of multivariate categorical distributions. They further derive sample-based gradient estimators for both the stochastic architecture and the deterministic parameters, which leads to a simple alternating algorithm for architecture search.
Pros:
+ Intuitions and formulations are easy to comprehend.
+ Simpler to implement than most prior methods.
+ Appealing results (on CIFAR-10) as compared to the state-of-the-art.
Cons:
- Limited technical novelty. The approach is a straightforward extension of Shirakawa et al. 2018. The main algorithm is essentially the same except minor differences in gradient derivations.
- Lack of theoretical justifications. It seems all the derivations at the beginning of Section 2 assume the architecture is optimized wrt the training set. However, the authors ended up splitting the dataset into two parts in the experiments and optimize the architecture wrt a separate validation set instead. This would invalidate all the previous derivations.
- The method is a degenerated version of ENAS. A closer look at eq (2) and (3) suggests the resulting iterative algorithm is almost the same as that in ENAS, where the weights are optimized using GD wrt the training set and the architecture is optimized using the log-derivative trick wrt the validation set. The only distinction are (i) using a degenerated controller/policy formulated as categorical distributions (ii) using the validation loss instead of the validation accuracy as the reward (according to eq. (3)). This is also empirically reflected in Table 1, which shows the proposed PDAS is similar to ENAS both in terms of efficiency and performance. The mathematical resemblance with ENAS is not necessarily bad, but the authors need to make it more explicit in the paper.
Minor issues:
* I'm not sure whether it's a good practice to report the "best" test error among multiple runs in Table 1.
* The method is not really "parameterless" as claimed in the introduction. For example, a suitable learning rate adaptation rule can be task-specific thus requires manual tuning/design. The method also consists of some additional hyperparameters like the \lambda in the utility transform.


HUMAN-LEVEL PROTEIN LOCALIZATION WITH CON- VOLUTIONAL NEURAL NETWORKS
Localizing a specific protein in a human cell is essential for understanding cellular functions and biological processes of underlying diseases. A promising, low-cost, and time-efficient biotechnology for localizing proteins is high-throughput fluorescence microscopy imaging (HTI). This imaging technique stains the protein of interest in a cell with fluorescent antibodies and subsequently takes a microscopic image. Together with images of other stained proteins or cell organelles and the annotation by the Human Protein Atlas project, these images provide a rich source of information on the protein location which can be utilized by computational methods. It is yet unclear how precise such methods are and whether they can compete with human experts. We here focus on deep learning image analysis methods and, in particular, on Convolutional Neural Networks (CNNs) since they showed overwhelming success across different imaging tasks. We propose a novel CNN architecture "GapNet-PL" that has been designed to tackle the characteristics of HTI data and uses global averages of filters at different abstraction levels. We present the largest comparison of CNN architectures including GapNet-PL for protein localization in HTI images of human cells. GapNet-PL outperforms all other competing methods and reaches close to perfect localization in all 13 tasks with an average AUC of 98% and F1 score of 78%. On a separate test set the performance of GapNet-PL was compared with three human experts and 25 scholars. GapNet-PL achieved an accuracy of 91%, significantly (p-value 1.1e −6 ) outperforming the best human expert with an accuracy of 72%.

This manuscript describes a deep convolutional neural network for
assigning proteins to subcellular compartments on the basis of
microscopy images.
Positive points:
- This is an important, well-studied problem.
- The results appear to improve significantly on the state of the art.
- The experimental comparison is quite extensive, including
  reimplementations of four, competing state-of-the-art methods, and
  lots of details about how the comparisons were carried out.
- The manuscript also includes a human-computer competition, which the
  computer soundly wins.
- The manuscript is written very clearly.
Concerns:
There is not much here in the way of new machine learning methods.
The authors describe a particular neural network architecture
("GapNet-PL") and show empirical evidence that it performs well on a
particular dataset.  No claims are made about the generalizability of
the particular model architecture used here to other datasets or other
tasks.
A significant concern is one that is common to much of the deep
learning literature these days, namely, that the manuscript fails to
separate model development from model validation. We are told only
about the final model that the authors propose here, with no
discussion of how the model was arrived at.  The concern here is that,
in all likelihood, the authors had to try various model topologies,
training strategies, etc., before settling on this particular setup.
If all of this was done on the same train/validation/test split, then
there is a risk of overfitting.
The dataset used here is not new; it was the basis for a competition
carried out previously.  It is therefore somewhat strange that the
authors chose to report only the results from their reimplementations
of competing methods.  There is a risk that the authors'
reimplementations involve some suboptimal choices, relative to the
methods used by the originators of those methods.
Another concern is the potential circularity of the labels.  At one
point, we are told that "Most importantly, these labels have not been
derived from the given microscopy images, but from other
biotechnologies such as microarrays or from literature."  However,
earlier we are told that the labels come from "a large battery of
biotechnologies and approaches, such as microarrays, confocal
microscopy, knowledge from literature, bioinformatics predictions and
additional experimental evidence, such as western blots, or small
interfering RNA knockdowns."  The concern is that, to the extent that
the labels are due to bioinformatics predictions, then we may simply
be learning to re-create some other image processing tool.
The manuscript contains a fair amount of biology jargon (western
blots, small interfering RNA knockdowns, antibodies, Hoechst staining,
etc.) that will not be understandable to a typical ICLR reader.
At the end, I think it would be instructive to show some examples
where the human expert and the network disagreed.
Minor:
p. 2: "automatic detection of malaria" -- from images of what?
p. 2: Put a semicolon before "however" and a comma after.
p. 2: Change "Linear Discriminant" to "linear discriminant." Also, remove
the abbreviations (SVM and LDA), since they are never used again in
this manuscript.
p. 5: Delete comma in "assumption, that."
p. 8: "nearly perfect" -> "nearly perfectly"
The confusion matrices in Figure 5 should not be row normalized --
just report raw counts.  Also, it would be better to order the classes
so that confusable ones are nearby in the list.


PLAN ONLINE, LEARN OFFLINE: EFFICIENT LEARNING AND EXPLORATION VIA MODEL-BASED CONTROL
We propose a "plan online and learn offline" framework for the setting where an agent, with an internal model, needs to continually act and learn in the world. Our work builds on the synergistic relationship between local model-based control, global value function learning, and exploration. We study how local trajectory optimization can cope with approximation errors in the value function, and can stabilize and accelerate value function learning. Conversely, we also study how approximate value functions can help reduce the planning horizon and allow for better policies beyond local solutions. Finally, we also demonstrate how trajectory optimization can be used to perform temporally coordinated exploration in conjunction with estimating uncertainty in value function approximation. This exploration is critical for fast and stable learning of the value function. Combining these components enable solutions to complex control tasks, like humanoid locomotion and dexterous in-hand manipulation, in the equivalent of a few minutes of experience in the real world.

This paper was a joy to read.   The description and motivation of the POLO framework was clear, smart, and sensible.  The fundamental idea is to explore the interplay between value-function estimation and model-predictive control and demonstrate how they benefit one another.  None of these ideas is fundamentally new, but the descriptions and their combination is very nice.
As I finished the paper, though, I was left with a lingering lack of understanding of the exact problem setting that is being addressed. The name is cute but didn't help clarify.    As I understand it:
- we have a correct dynamics model (I'm assuming that's what "nominal dynamics model" means) and a good trajectory optimization algorithm
- the agent has limited online cognitive capacity
- there is no opportunity for offline computation
If offline computation time were available, then we could run this algorithm (or your favorite other RL algorithm) in the agent's head before taking any actions in the actual world.   That does not seem to be the setting here, although it does seem to me that you might be able to show that POLO is a good algorithm for finding a value function, offline, with no actual interaction with the world.
So, fundamentally, this paper is about action under computational time constraints.   One strategy would be for the robot to use 7 of its cores to run your favorite approximate DP / RL algorithm in parallel with 1 core that's used for action selection.  Why is that worse than your algorithm 1?
Setting this question aside, I had some other comments:
- It is better *not* to use "trajectory optimization" and "model-predictive control" interchangeably.  I can use traj opt in other circumstances (e.g. with open loop trajectory following) and could use other planners for MPC.
- Some version of lemma 2 probably (almost certainly) already exists somewhere in the literature;  I'm sorry, though, that I can't point you to a concrete reference.
- The argument about MPC letting us approximate H Bellman backups is plausible, but seems somewhat subtle;  it would be good to elaborate it in some more detail.
- The set of assertions and experiments is very nice.
- Why are no variances shown in figure 3?   Why does performance seem to degrade after a certain horizon.
This paper doesn't seem really to be about learning representations.  I don't know if that's important to the ICLR decision-making.


DETERMINISTIC PAC-BAYESIAN GENERALIZATION BOUNDS FOR DEEP NETWORKS VIA GENERALIZING NOISE-RESILIENCE
The ability of overparameterized deep networks to generalize well has been linked to the fact that stochastic gradient descent (SGD) finds solutions that lie in flat, wide minima in the training loss -minima where the output of the network is resilient to small random noise added to its parameters. So far this observation has been used to provide generalization guarantees only for neural networks whose parameters are either stochastic or compressed. In this work, we present a general PAC-Bayesian framework that leverages this observation to provide a bound on the original network learned -a network that is deterministic and uncompressed. What enables us to do this is a key novelty in our approach: our framework allows us to show that if on training data, the interactions between the weight matrices satisfy certain conditions that imply a wide training loss minimum, these conditions themselves generalize to the interactions between the matrices on test data, thereby implying a wide test loss minimum. We then apply our general framework in a setup where we assume that the pre-activation values of the network are not too small (although we assume this only on the training data). In this setup, we provide a generalization guarantee for the original (deterministic, uncompressed) network, that does not scale with product of the spectral norms of the weight matrices -a guarantee that would not have been possible with prior approaches.

The authors demonstrate the generalization bound for deep neural networks using the PAC-Bayesian approach. They adopt the idea of noise resilience in the analysis and obtain a result that has improved dependence in terms of the network dimensions, but involves parameters (e.g., pre-activation) that may be large potentially.
My major concern is also regarding the dependence on the pre-activation that can be very large in practice. This is also shown in the numerical experiments. Therefore, the overall generalization bound can be larger than existing results, though the later have stronger dependence on the network sizes. By examining the analysis for the main result, it seems to me that the reason the authors can induce weaker dependence on network sizes is essentially they involved the pre-activation parameters. This can be viewed as a trade-off how strong the generalization bound depend on the network sizes and other related parameters (like the pre-activation here) rather than strictly tighten the error bound from a more refined/structured way. I also suggest that the authors provide the comparison of their bound and existing ones to see the quantitative difference of the results.
Regarding the noise resilience, it is not clear to where the noise resilience shows up from the analysis or the result. From the proof of the main result, the analysis seems to be standard as in the PAC-Bayesian analysis, which is based on bounding the difference of the network before and after injecting randomness into the parameters. The difference with respect to the previous result due to the different way of bounding such a gap, where the Jacobian, the pre-activation and function output pop up. But this does not explain how well a network can tolerate the noise, either in the parameter space of the data space. This is different with the previous analysis based on the noise resilience, such as [1]. So, the title and the way the authors explain as noise resilience is somewhat misleading. More detailed explanation will help.
[1] Arora et al. Stronger generalization bounds for deep nets via a compression approach.


A large amount of parallel data is needed to train a strong neural machine translation (NMT) system. This is a major challenge for low-resource languages. Building on recent work on unsupervised and semi-supervised methods, we propose a multidual learning framework to improve the performance of NMT by using an almost infinite amount of available monolingual data and some parallel data of other languages. Since our framework involves multiple languages and components, we further propose a timing optimization method that uses reinforcement learning (RL) to optimally schedule the different components in order to avoid imbalanced training. Experimental results demonstrate the validity of our model, and confirm its superiority to existing dual learning methods.

[Summary]
This paper proposes an extension of the dual learning framework, with a guider network and multiple languages included: (1) Each language $i$ has a guider network $GN_i$, that can be used to reconstruct the source sentence from either the output of the encoder or the output of the decoder. (2) Multiple languages are used in this framework, where each language also has a $GN_i$  for guiding the training according to the reconstruction error. The authors work on MultiUN dataset to verify their algorithms.

[Clarity]
This work is not easy to follow. My suggestions to revise the paper are shown as follows:
(1) Please use the \begin{equation}…\end{equation} environment to clearly describe your framework and training objectives, with each notation, function and hyper-parameter clearly defined. Actually, I do not find the training objective function in this paper.
Besides, currently, in this paper, there are many undefined notations and typos, for example, (1) in section 3.1, first paragraph, what is the $n$? Then in Eqn.(1) ,what is $N$ and $M$? Also, it is very confusing to use subscripts $i$ and $j$ to distinguish the hidden states from the encoder and decoder. (2) What is the mathematical definition of $ISE_i$? (3) In page 5, 3rd line, “then ISD_i is used to reconstruct Si = GNi(ISE_i , \theta)…” Should the ISE_i be ISD_i?
(2) Please use \begin{algorithm}…\end{algorithm} to tell the readers how your framework works.

[Details]
1. The first question is “why this problem”. In the 3rd paragraph of page 1, you mentioned that “However, the best direction to update parameters heavily relies on the quality of sampled translations ... which may be far from real translations Y due to inaccurate translations existing in the sampled ones……” But in practice, dual learning as well as back-translation [ref1] works well for many language pairs. In particular, the dual learning and back-translation works for the unsupervised NMT [ref2], where no labeled data is available. Therefore, I am not fully convinced by this claim and then, the motivation of this work. What’s more, this paper does not work on standard WMT dataset, while previous dual learning and back-translation work on that most commonly used dataset. Therefore, the comparison between the guider network and dual learning are not fair.
2. I am not sure how the BDE in Eqn. (1) is related to the NMT translation quality. Any reference or theoretical/empirical proofs?
3. It is hard to reproduce such a complex NMT system with NMT, GN and an RL scheduler. Any open-source code or any simple solutions?
4. Do you use a single-layer LSTM or a deep LSTM? Transformer [ref3] is the state-of-the-art NMT system. Why don’t you choose this system? Also, you do not work on WMT dataset to verify your GLF-2L (Table 1). Therefore, I cannot justify whether the proposed algorithm is efficient compared to the current NMT algorithms. I am not convinced by the experimental results.
5. The connection/difference between this work and (Tu et al 2017) should be discussed clearly, and you should implement (Tu et al 2017) as your baseline.  Besides, for the 3-language setting, no multilingual baseline is implemented.

[Pros & Cons]
(+) This paper tries to extend dual learning from word level to hidden state level;
(+) Multiple languages are involved in this framework;
(-) Experiments are not convincing; the models are weak; many important baselines are missing; no results on widely used WMT datasets;
(-) The paper is not easy to follow. (See [clarify] part for details);
(-) Training process is a little complex; not easy to implement;

References
[ref1] Edunov, Sergey, et al. "Understanding back-translation at scale." EMNLP 2018
[ref2] Lample, Guillaume, et al. "Phrase-Based & Neural Unsupervised Machine Translation." EMNLP 2018
[ref3] Vaswani, Ashish, et al. "Attention is all you need." Advances in Neural Information Processing Systems. 2017.


SPREADING VECTORS FOR SIMILARITY SEARCH
Discretizing multi-dimensional data distributions is a fundamental step of modern indexing methods. State-of-the-art techniques learn parameters of quantizers on training data for optimal performance, thus adapting quantizers to the data. In this work, we propose to reverse this paradigm and adapt the data to the quantizer: we train a neural net which last layer forms a fixed parameter-free quantizer, such as pre-defined points of a hyper-sphere. As a proxy objective, we design and train a neural network that favors uniformity in the spherical latent space, while preserving the neighborhood structure after the mapping. We propose a new regularizer derived from the Kozachenko-Leonenko differential entropy estimator to enforce uniformity and combine it with a locality-aware triplet loss. Experiments show that our end-to-end approach outperforms most learned quantization methods, and is competitive with the state of the art on widely adopted benchmarks. Furthermore, we show that training without the quantization step results in almost no difference in accuracy, but yields a generic catalyzer that can be applied with any subsequent quantizer. The code is available online 1 .

Pros
----
[Originality]
The authors propose a novel idea of learning representations that improves the performance of the subsequent fixed discretization method.
[Clarity]
The authors clearly motivate their solution and explain the different ideas and enhancements introduced. The manuscript is fairly easy to follow. The different terms in the optimization problem are clearly explained and their individual behaviour are presented for the better understanding.
[Significance]
The empirical results for the proposed scheme are compared against various baselines under various scenarios and the results demonstrate the significant utility of the proposed scheme.
Limitations
-----------
[Clarity]
The training times for the catalyzer is never discussed in this manuscript (even relative to the training times of the considered baselines). Moreover, it is not clear if the inference time of the catalyzer is included in the results such as Table 1. Even if, PQ and the catalyzer+lattice might have comparable search recalls, it would be good to understand the relative search times to get similar accuracy especially since the inference time for the catalyzer (which is part of the search time) can be fairly significant.
[Clarity/Significance]
One important point not discussed in this manuscript is the choice of the structure (architechture) of the catalyzer. Is the catalyzer architecture dependent on the data?
  - If yes, how to find an appropriate architecture?
  - If no, what is it about the proposed architecture that makes it sufficient for all data sets?
In my opinion, this is extremely important since this drives the applicability of the proposed scheme beyond the presented examples.
[Minor question]
- Is the parameter r in the rank loss same as the norm r in the lattice quantizer? This is a bit confusing.


In learning a predictive model over multivariate time series consisting of target and exogenous variables, the forecasting performance and interpretability of the model are both essential for deployment and uncovering knowledge behind the data. To this end, we propose the interpretable multi-variable LSTM recurrent neural network (IMV-LSTM) capable of providing accurate forecasting as well as both temporal and variable level importance interpretation. In particular, IMV-LSTM is equipped with hidden state matrix and update process, so as to learn variableswise hidden states. On top of it, we develop a mixture attention mechanism and associated summarization methods to quantify the temporal and variable importance in data. Extensive experiments using real datasets demonstrate the prediction performance and interpretability of IMV-LSTM in comparison to a variety of baselines. It also exhibits the prospect as an end-to-end framework for both forecasting and knowledge extraction over multi-variate data.

This paper describes a recurrent model (LSTM specifically, but generalizable) which can produce variable-wise hidden states that can be further used for two types of attentions: 1) variable importance for the importance of each variable (not accounting for time), and 2) temporal importance of each variable for the importance of each variable over time. The proposed NN model (IMV-LSTM) does not seem to directly provide such importance. Rather, the outputs are “decomposed” for each variable/time that allows probabilistic inference on top of this.
One of my main concerns (described in Cons/Comments below) is how it is not straightforward to grasp the quality of variable importance and temporal variable importance results despite this is the key strength of this paper. If this comes from my lack of understanding, I would appreciate if the authors could provide a little more explanation.
Pros:
1.	The overall quality of the paper is decent and mostly clear.
2.	The experiments are quite extensive.
3.	The fact that each variable should have different level of importance is interesting and practical.
Cons/Comments:
1.	The term “tensor” is used throughout the paper to describe the stacked matrices. While this is not technically wrong to describe 2>-dimensional structures, this term could potentially imply (and make the readers to expect) tensor-based schemes such as tensor decomposition. This is not necessarily bad, but to me, “tensor” and “variable-wise correspondence” do not seems to be associated too deeply since the “tensor” used in IMV-LSTM is a stack of matrices that are also independently used with respect to each other.
2.	The variable importance experiments seem quite extensive and thorough, especially the lists of variable-wise temporal importance matrices provided in the appendix. However, the authors could provide some significance or relevance of the findings with respect to any domain knowledge or literature, it may help further appreciate and interpret the quality of the variable importance which is quite subjective to non-experts. Such information may not even need to be in the main paper; including a short description in the appendix.
3.	Related to comments (2), the difference between IMV-Full and IMV-Tensor is hard to interpret since neither one is always better than the other (i.e., IMV-Full > IMV-Tensor in some experiments, vice versa). While the key difference is speculated to be from how the LSTM handles the variables, I am curious how this related to the differences in the results and how the differences variable importance results (i.e., Fig.3) can be in at least speculated.
Questions:
1.	Should \tilde{h}_t in Figure 1 (a) be \tilde{h}__{t-1} since this hidden state is from t-1? The figure itself currently implies that the hidden state for t is used, but this is computed from x_t using U_j. With \tilde{h}__{t-1}, it follows Eq.(1).
2.	In Equation set 2 for IMV-Tensor, are W and U (not W_j and U_j) also in tensor forms so each variable and hidden state get transformed correspondingly (i.e., W_1 for h^1_{t-1}, U_1 for x^1_t).
3.	The IMV-Tensor version of IMV-LSTM (related to the question above) can be considered as a set of parallel LSTMs, one for each variable. Such independence could also be inferred from Figure 1. If that’s the case, where do the variables “interact” with each other? Is this happening in the later stage where the hidden states across variable/time are aggregated in the attention stage (Eq.(8) and on)?
4.	Up until Eq.(8), n was used for the variable index where n = 1,…,N. In Eq.(8), it seems to be still used as the variable index (i.e., h_T^n and g^n), but it is also a set of possible values for a random variable z_{T+1}. Is n used the same way for z_{T+1} as well? I am slightly confused on how z is used. Also, (just to clarify), if we use N variables, we are using y_t as well (i.e., [x_t^1,…,x_t^{N-1}, y_t])?
5.	f_agg: Is this for aggregating over instances? For \bar{\alpha}^n, I’m guessing this is aggregated over instances for variable n for t=1,…,T_1.
6.	I am not too familiar with the notion of “time-lag” in the experiments. If the authors could explain this a little bit, I would appreciate it.


Deep neural networks on graph structured data have shown increasing success in various applications. However, due to recent studies about vulnerabilities of machine learning models, researchers are encouraged to explore the robustness of graph neural networks (GNNs). So far there are two works targeting to attack GNNs by adding/deleting edges to fool graph based classification tasks. Such attacks are challenging to be detected since the manipulation is very subtle compared with traditional graph attacks. In this paper we propose the first detection mechanism against these two proposed attacks. Given a perturbed graph, we propose novel graph generation method together with link prediction as preprocessing to detect potential malicious edges. We also propose novel features which can be leveraged to perform outlier detection when the number of added malicious edges are large. Different detection components are proposed and tested, and we also evaluate the performance of the final detection pipeline. Extensive experiments are conducted to show that the proposed detection mechanism can achieve AUC above 90% against the two attack strategies on both Cora and Citeseer datasets. We also provide in-depth analysis for different attack strategies and corresponding suitable detection methods. Our results shed light on several principles for detecting different types of attacks.

The authors address the interesting problem about the attack on the graph convolutional network model. The proposed method is developed under the assumptions about the attacking models sound simple but reasonable with proper references.
However, the proposed approaches mostly include the ideas about the detecting mechanisms instead of being formulated in some novel form. Given that the proposed algorithms do not leverage the underlying model structure very much, why the proposed algorithms are special to the graphical neural network is not very clear.
Also, the evaluations need to be further improved. It seems that victim nodes are carefully selected and fixed throughout all the experiments, but it limits the generalization about the performance of the proposed algorithms. Particularly, since the different detection algorithms perform differently on different datasets, more extensive evaluations are required along with the guideline of what detection algorithm we have to choose for the unsupervised setting.
*Details
- It will be great if the authors clearly describe what the proposed methods aim to defend. Basically, the values by protecting some victim nodes, regardless of what attacking models are assumed, will help the audience with better understanding. Some of content in Section 2.2 can be brought up in the introduction.
- In Section 3.1 and some other subsections, it seems to assume that the links in a given network are very clean but in reality there are a lot of noisy connections. How can we distinguish some random connections from malicious connections? Evaluation along with this question will be also useful.
- In Section 3.2, eventually, the ratio of malicious edges remains the same if the authors use random sampling. In that case, how SubGraphLinkPred helps is not very convincing.
- The detection algorithm seems to exist to detect malicious edges without supervision. In that case, how can we determine which method we should use given that detection performance differs in different dataset?
- It would be useful to compare with some existing malicious node/graph pattern mining algorithms such as Graph-Based Fraud Detection in the Face of Camouflage, Hooi et. al. even if the baseline method does not aim to directly solve the addressed problem. And also that literature needs to be cited.


Sequentially learning of tasks arriving in a continuous stream is a complex problem and becomes more challenging when the model has a fixed capacity. Lifelong learning aims at learning new tasks without forgetting previously learnt ones as well as freeing up capacity for learning future tasks. We argue that identifying the most influential parameters in a representation learned for one task plays a critical role to decide on what to remember for continual learning. Motivated by the statistically-grounded uncertainty defined in Bayesian neural networks, we propose to formulate a Bayesian lifelong learning framework, BLLL, that addresses two lifelong learning directions: 1) completely eliminating catastrophic forgetting using weight pruning, where a hard selection mask freezes the most certain parameters (BLLL-PRN) and 2) reducing catastrophic forgetting by adaptively regularizing the learning rates using the parameter uncertainty (BLLL-REG). While BLLL-PRN is by definition a zero-forgetting guaranteed method, BLLL-REG, despite exhibiting some small forgetting, is a task-agnostic lifelong learner, which does not require to know when a new task arrives. This feature makes BLLL-REG a more convenient candidate for applications such as robotics or on-line learning in which such information is not available. We evaluate our Bayesian learning approaches extensively on diverse object classification datasets in short and long sequences of tasks and perform superior or marginally better than the existing approaches.

The paper addresses the problem of lifelong learning of neural networks - a setting where learning is performed on a continuously arriving new tasks without having access to previously encountered data.
Authors propose a method that prevents catastrophic forgetting typical for naive application of stochastic gradient descent by preventing supposedly important weights to change (in either soft of hard manner), where the weight importance is assessed by its signal to noise ratio estimated from the corresponding (approximate) posterior distribution.
Authors evaluate their approach on a set of image classification datasets and find it superior to the PackNet baseline as well as few simpler ones.
The idea of using uncertainty estimates obtained from Bayesian training to adjust weight updates is natural and potentially very promising.
However, to me this paper does not seem to investigate the idea sufficiently deep.
The weight pruning or hard masking variant of the method depends on a very important hyperparameter p (size of the mask) which is unclear how to set beforehand.
I also struggle with understanding the weight regularisation or soft masking variant.
Authors seem to get their inspiration in the idea of assumed density filtering, where the posterior for 1:T-1 is approximated and used a prior for task T (last sentence on page 5).
At the same time, in Algorithm 2, line 6 the prior is defined as the standard BBB mixture prior and not the approximate posterior from the previous task.
Quite oddly, parameters of the _approximate posterior_ are being quadratically regularalized to not deviate from parameters of the _approximate posterior_ from the previous task.
This deviates from the original idea and requires additional justification.
Besides that, I find the way this regularisation is applied potentially problematic for the variance parameter (last term in eq. 6).
Here authors apply the regularisation to the parameter of the softplus transformation they use, but scale it with the inverse std deviation which is the “classical” parametrisation. The choice of parametrisation was not discussed, however, clearly different parametrizations may lead to very different results.
On the experimental side, I have two major issues:
1. The datasets considered are very small, authors could consider using ImageNet, especially given that they already work with 224x224 images.
2. The only prior work used as a baseline is PackNet, while there is no reason why other established methods such as EWC are not applicable.
Minor comments:
The middle expression in eq. 5 seems to miss the -log p(D_T | D_{1:T-1}) term which does not change the latter expression (since it does not depend on parameters theta).
Page 3: “citestochastic methods”, a citation seems to be missing.


A plain well-trained deep learning model often does not have the ability to learn new knowledge without forgetting the previously learned knowledge, which is known as catastrophic forgetting. Here we propose a novel method, SupportNet, to efficiently and effectively solve the catastrophic forgetting problem in the class incremental learning scenario. SupportNet combines the strength of deep learning and support vector machine (SVM), where SVM is used to identify the support data from the old data, which are fed to the deep learning model together with the new data for further training so that the model can review the essential information of the old data when learning the new information. Two powerful consolidation regularizers are applied to stabilize the learned representation and ensure the robustness of the learned model. We validate our method with comprehensive experiments on various tasks, which show that SupportNet drastically outperforms the state-of-the-art incremental learning methods and even reaches similar performance as the deep learning model trained from scratch on both old and new data.

Summary:
The authors offer a novel incremental learning method called SupportNet to combat catastrophic forgetting that can be seen in standard deep learning models. Catastrophic forgetting is the phenomenon where the networks don’t retain old knowledge when they learn new knowledge.  SupportNet uses resnet network with 32 layers, trains an SVM on the last layer and the support vector points from this SVM are given to the network along with the new data. Furthermore, two regularizers, feature and EWC regularizer, are added to the network. The feature regularizer forces the network to produce fixed representation for the old data, since if the feature representation for the old data changes when the network is fine-tuned on the new data then the support vectors generated from the old feature representation of the old data would become invalid.  The EWC regularizer works by constraining parameters crucial for the classification of the old data, making it harder for the network to change them. SupportNet is compared to five methods (all data: network is re-trained with new and old data, upper bound for performance, iCaRL: state-of-the-art method for incremental learning, EWC: Only EWC regularizer added, Fine tune: Only new data, Random guessing: Random guess to assign labels) on six datasets (MNIST, CIFAR-10, CIFAR-100, Enzyme Function Prediction, HeLa Subcellular Structure Classification, Breast Tumor Classification). It shows some improvement in overall accuracy with each newly added class when compared to iCaRL, EWC, Fine Tune and Random guessing.  Additionally, they show that overfitting for the real training data (a chosen subset of old data and the new data) is a problem for the competition iCaRL and affects SupportNet to a much lesser degree.
Pros:
(1)
The authors propose a sensible approach, which is also novel to be best of our knowledge, using SVM to select support data from old data to be fed to the network along with the new data in the incremental learning framework to avoid catastrophic forgetting. Additionally, they offer a feature regularizer that penalizes the network for changing the feature representation of the support data when training the network on new data and an EWC regularizer that constrains the parameters that are crucial for the classification of the old data and makes it harder to change them.
(2)
The authors use six different datasets and several other approaches (subsets of their method’s components, other competing methods) to show these three components alleviate catastrophic forgetting and show improvement in overall accuracy.
(3)
The paper is well written and easy to follow.
Cons:
Major Points:
(1)
To show that the method proposed in the paper addresses catastrophic forgetting, in addition to the overall accuracy shown in Figure 3, it is also necessary to show the accuracy of different models on old classes when new classes are added to the network. This will strengthen the argument that the improvement in accuracy is indeed due to correct classification on old data.
 (2)
The authors claim that iCaRL suffers from overfitting on real training data (section 4.1) however Table 2 shows iCaRL only on the enzyme function prediction which is also the dataset where the difference in performance between iCaRL and SupportNet is the largest. To support the general overfitting claim made in section 4.1, the authors should repeat this analysis on any of the other five datasets where the performance difference between the two methods is much smaller.  SupportNet also suffers from overfitting (Table 3, Accuracy: test data: 83.9%, real training data: 98.7%) although to a lesser extent than iCaRL.
(3)
The individual impact of the support points and the joint impact of support points with feature regularizer on accuracy is not assessed. To prove their usefulness, add two methods to Figure 3:
(a)A method that uses support points without any regularizer.
(b) A method that uses support points with just the feature regularizer.
Other points:
(1)
In section 2.3.2, EWC regularizer, Eq. 9: We think F(theta_new) should be F(theta_old) since we want to constrain parameters crucial for classification of old data and should be computing Fisher Information for the old parameters.
(2)
In section 2.1 Deep Learning and SVM: additional steps are needed to show how Eq. 3 is derived from Eq. 2.
(3)
In section 2.1 Deep Learning and SVM: In the line before Eq. 4. “t represtent” instead of “t represents”.
(4)
Figures are small and hard to read. Please increase the size and resolution of the figures.


Computing universal distributed representations of sentences is a fundamental task in natural language processing. We propose ConsSent, a simple yet surprisingly powerful unsupervised method to learn such representations by enforcing consistency constraints on sequences of tokens. We consider two classes of such constraints -sequences that form a sentence and between two sequences that form a sentence when merged. We learn sentence encoders by training them to distinguish between consistent and inconsistent examples, the latter being generated by randomly perturbing consistent examples in six different ways. Extensive evaluation on several transfer learning and linguistic probing tasks shows improved performance over strong unsupervised and supervised baselines, substantially surpassing them in several cases. Our best results are achieved by training sentence encoders in a multitask setting and by an ensemble of encoders trained on the individual tasks.

== Clarity ==
The primary strength of this paper is the simplicity of the approach.
Main idea #1: corrupt sentences (via random insertions/deletions/permutations), and train a sentence encoder to determine whether a sentence has been corrupted or not.
Main idea #2: split a sentence into two parts (two different ways to do this were proposed). Train a sequence encoder to encode each part such that we can tell whether the two parts came from the same sentence or not.
I can see that this would be very easy for others to implement, perhaps encouraging its adoption.
== Quality of results ==
The proposed approach is evaluated on the well-known SentEval benchmark.
It generally does not outperform supervised approaches such as InferSent and MultiTask. However, this is fine because the proposed approach uses no supervised data, and can be applied in domains/languages where supervised data is not available.
The approach is competitive with existing state-of-the-art sentence representations such as QuickThoughts. However, it is not definitively better:
Out of the 9 tasks with results for QuickThoughts, this approach (ConsSent) performs better on 3 (MPQA +0.1%, TREC +0.4%, MRPC +0.4%). For the other 6 tasks, ConsSent performs worse (MR -1.8%, CR -1.7%, SUBJ -1%, SST -3.8%, SK-R, -2.4%). Taken together, the losses seem to be larger than the gains.
Furthermore, the QuickThoughts results were obtained with a single model across all SentEval tasks. In contrast, the ConsSent approach requires a different hyperparameter setting for each task in order to achieve comparable results -- there is no single hyperparameter setting that would give state-of-the-art results across all tasks.
The authors also evaluate on the newly-released linguistic probing tasks in SentEval. They strongly outperform several existing methods on this benchmark. However, it is unclear why they did not compare against QuickThoughts, which was the strongest baseline on the original SentEval tasks.
== Originality ==
The proposed approach is simple and straightforward. This is on the whole a great thing, but perhaps not especially surprising from an originality/novelty perspective.
Therefore, the significance and impact of this approach really needs to be carried by the quality of the empirical results.
The sentence pair based approaches (ConsSent-N and C) are conceptually interesting, but don't seem to be responsible for the best results on the linguistic probing tasks.
== Conclusion ==
Pros:
- conceptual simplicity
- competitive results (better than many previous unsup. sentence representation methods, excluding QuickThoughts)
- strong results on SentEval's linguistic probing task
Cons:
- no single hyperparameter value (perturbation method and value for k) gets great results across all tasks
- some important baselines possibly missing for linguistic probing tasks


Representation learning is typically applied to only one mode of a data matrix, either its rows or columns. Yet in many applications, there is an underlying geometry to both the rows and the columns. We propose utilizing this coupled structure to perform co-manifold learning: uncovering the underlying geometry of both the rows and the columns of a given matrix, where we focus on a missing data setting. Our unsupervised approach consists of three components. We first solve a family of optimization problems to estimate a complete matrix at multiple scales of smoothness. We then use this collection of smooth matrix estimates to compute pairwise distances on the rows and columns based on a new multi-scale metric that implicitly introduces a coupling between the rows and the columns. Finally, we construct row and column representations from these multi-scale metrics. We demonstrate that our approach outperforms competing methods in both data visualization and clustering.

Review for CO-MANIFOLD LEARNING WITH MISSING DATA
Summary:
This paper proposes a two-stage method to recovering the underlying structure of a data manifold using both the rows and columns of an incomplete data matrix. In the first stage they impute the missing values using their proposed co-clustering algorithm and in the second stage they propose a new metric for dimension reduction.
The overall motivation for how they construct the algorithm and the intuition behind how all the pieces of the algorithm work together are not great. The paper also has significant specific clarity issues (listed below). Currently these issues seem to imply the proposed algorithm has significant logic issues (mainly on the convex/concave confusions); however depending on how they are addressed, this may end up not being an issue. The experimental results for the two simulated datasets look very good. However for the lung dataset, the results are less promising and it is less clear of the advantage of the proposed algorithm to the two competing ones.
Novelty/Significance:
The overall idea of the algorithm is sufficiently novel. It is very interesting to consider both rows and column correlations. Each piece of the algorithm seems to draw heavily on previous work; bi-clustering, diffusion maps, but overall the idea is novel enough. The algorithm is significant in that it addresses a relatively open problem that currently doesn’t have a well established solution.
Questions/Clarity:
Smooth is not clearly defined and not an obvious measure for a matrix. Figure 1 shows smooth matrices at various levels, but still doesn’t define explicitly what smoothness is. Does smoothness imply all entries are closer to the same value?
 “Replacing Jr(U) and Jc(U) by quadratic row and column Laplacian penalties” – The sentence is kind of strange as Laplacian penalties is not a thing. Graph Laplacian can be used as an empirical estimate for the Laplace Beltrami operator which gives a measure of smoothness in terms of divergence of the gradient of a function on a manifold; however the penalty is one on a function’s complexity in the intrinsic geometry of a manifold. It is not clear how the proposed penalty is an estimator for the intrinsic geometry penalty. It seems like the equation that is listed is just the function map Omega(x) = x^2, which also is not a concave function (it is convex), so it does not fit the requirements of Assumption 2.2.
Proposition 1 is kind of strangely presented. At first glance, it is not clear where the proof is, and it takes some looking to figure out it is Appendix B because it is reference before, not after the proposition. Or it might be more helpful if it is clearly stated at the beginning of Appendix B that this is the proof for Proposition 1.
The authors write: “Missing values can sabotage efforts to learn the low dimensional manifold underlying the data. … As the number of missing entries grows, the distances between points are increasingly distorted, resulting in poor representation of the data in the low-dimensional space.” However, they use the observed values to build the knn graph used for the row/column penalties, which is counter-intuitive because this knn graph is essentially estimating a property of a manifold and the distances have the same distortion issue.
Why do the author’s want Omega to be concave functions as this makes the objective not convex. Additionally the penalty sqrt(|| ||_2) is approximately doing a square root twice because the l2-norm already is the square root of the sum of squares. Also what is the point of approximating the square root function instead of just using the square root function? It is overall not clear what the nature of the penalty term g2 is; Appendix A, implies it must be overall a convex function because of the upper bound.
Equation 5 is not clear that it is the first order taylor approximation. Omega’ is the derivative of the Omega function? Do the other terms cancel out? Also what is the derivative with respect to; each Ui. for all Uj. ?
 “first-order Taylor approximation of a differentiable concave function provides a tight bound on the function” – Tight bound is not an appropriate term and requires being provable. Unless the function is close to linear, a first order Taylor approximation won’t be anything close to tight.
The authors state the objective in 1 is not convex. Do they mean it is not strictly convex? In which case, by stationary points, they are specifically referring to local minima? Otherwise, what benefits does the MM algorithm have on an indefinite objective i.e. couldn’t you end up converging to a saddle point or a local maxima instead of a local minima, as these are all fixed points.
It is not clear what the sub/super scripts l, k mean. Maybe with these defined, the proposed multi-scale metric would have obvious advantages, but currently it is not clear what the point of this metric is.
Figure 4 appears before it is mentioned and is displayed as part of the previous section.
For the Lung data, it does not look like the proposed algorithm is better than the other two. None of the algorithms seem to do great at capturing any of the underlying structure, especially in the rows. It also is not super clear that the normal patients are significantly further from the cancer patients. Additionally are the linkage results from figure 3 from one trial? Without multiple trials it is hard to argue that this not just trial noise.
How big are N1 and N2 in the linkage simulations. The Lung dataset is not very large, and it seems like the proposed algorithm has large computation complexity (it is not clear). Will the algorithm work on even medium-large sized matrices (10^4 x 10^4)?


The celebrated Sequence to Sequence learning (Seq2Seq) technique and its numerous variants achieve excellent performance on many tasks. However, many machine learning tasks have inputs naturally represented as graphs; existing Seq2Seq models face a significant challenge in achieving accurate conversion from graph form to the appropriate sequence. To address this challenge, we introduce a general end-to-end graph-to-sequence neural encoder-decoder architecture that maps an input graph to a sequence of vectors and uses an attention-based LSTM method to decode the target sequence from these vectors. Our method first generates the node and graph embeddings using an improved graph-based neural network with a novel aggregation strategy to incorporate edge direction information in the node embeddings. We further introduce an attention mechanism that aligns node embeddings and the decoding sequence to better cope with large graphs. Experimental results on bAbI, Shortest Path, and Natural Language Generation tasks demonstrate that our model achieves state-of-the-art performance and significantly outperforms existing graph neural networks, Seq2Seq, and Tree2Seq models; using the proposed bi-directional node embedding aggregation strategy, the model can converge rapidly to the optimal performance.

This work proposes an end-to-end graph encoder to sequence decoder model with an attention mechanism in between.
Pros (+) :
+ Overall, the paper provides a good first step towards flexible end-to-end graph-to-seq models.
+ Experiments show promising results for the model to be tested in further domains.
Cons (-) :
- The paper would benefit more motivation and organization.
Further details below (+ for pros / ~ for suggestions / - for cons):
The paper could benefit a little more motivation:
- Mentioning a few tasks in the introduction may not be enough. Explaining why these tasks are important may help. What is the greater problem the authors are trying to solve?
- Same thing in the experiments, not well motivated, why these three? What characteristics are the authors trying to analyze with each of these tasks?
Rephrase the novelty argument:
- The authors argue to present a “novel attention mechanism” but the attention mechanism used is not new (Bahdanau 2014 a & b). The fact that it is applied between a sequence decoder and graph node embeddings makes the paper interesting but maybe not novel.
~ The novelty added by this paper is the “bi-edge-direction“ aggregation technique with the exploration of various pooling techniques. This could be emphasized more.
Previous work:
~ The Related Work section could mention Graph Attention Networks (https://arxiv.org/abs/1710.10903) as an alternative to the node aggregation strategy.
Aggregation variations:
+ The exploration between the three aggregator architectures is well presented and well reported in experiments.
~ The two Graph Embedding methods are also well presented, however, I didn’t see them in experiments. Actually, it isn’t clear at all if these are even used since the decoder is attending over node embeddings, not graph embedding… Could benefit a little more explanation
Experiments:
+ Experiments show some improvement on the proposed tasks compared to a few baselines.
- The change of baselines between table 1 for the first two tasks and table 2 for the third task is not explained and thus confusing.
~ There are multiple references to the advantage of using “bi-directional” node embeddings, but it is not clear from the description of each task where the edge direction comes from. A better explanation of each task could help.
Results:
- Page 9, the “Impact of Attention Mechanism” is discussed but no experimental result is shown to support these claims.
Some editing notes:
(1) Page 1, in the intro, when saying “seq2seq are excellent for NMT, NLG, Speech Reco, and drug discovery”: this last example breaks the logical structure of the sentence because it has nothing to do with NLP.
(2) Page 1, in the intro, when saying that “<...> a network can only be applied to sequential inputs”: replace network by seq2seq models to be exact.
(3) Typo on page 3, in paragraph “Neural Networks on Graphs”, on 8th line “usig” -> “using”
(4) Page 3, in paragraph “Neural Networks on Graphs”, the following sentence: “An extension of GCN can be shown to be mathematically related to one variant of our graph encoder on undirected graphs.” is missing some information, like a reference, or a proof in Appendix, or something else…
(5) Page 9, the last section of the “Impact of Hop Size” paragraph talks about the impact of the attention strategy. This should be moved to the next paragraph which discusses attention.
(6) Some references are duplicates:
|_ Hamilton 2017 a & c
|_ Bahdanau 2014 a & b


HOW TO TRAIN YOUR MAML
The field of few-shot learning has recently seen substantial advancements. Most of these advancements came from casting few-shot learning as a meta-learning problem. Model Agnostic Meta Learning or MAML is currently one of the best approaches for few-shot learning via meta-learning. MAML is simple, elegant and very powerful, however, it has a variety of issues, such as being very sensitive to neural network architectures, often leading to instability during training, requiring arduous hyperparameter searches to stabilize training and achieve high generalization and being very computationally expensive at both training and inference times. In this paper, we propose various modifications to MAML that not only stabilize the system, but also substantially improve the generalization performance, convergence speed and computational overhead of MAML, which we call MAML++.

Paper summary - This paper provides a bag of sensible tricks for making MAML more stable, faster to learn, and better in final performance.
Quality - The quality of the work is strong: the results demonstrate that tweaks to MAML produce significant improvements in performance. However, I have some concern that certain portions of the text overclaim (see concerns section below).
Clarity - The paper is reasonably clear, with some exceptions (see concerns section).
Originality - The techniques described in the paper range from only mildly novel (e.g. MSL, DA), to very obvious (e.g. CA). Additionally, the paper's contributions amount to tweaks to a previously existing algorithm.
Significance - The quality of the results make this a significant contribution in my view.
Pros - Good results on a problem/algorithm of great current interest.
Cons - Only presents (in some cases obvious) tweaks to a previous algorithm; clarity and overclaiming issues in the writeup.
Concerns (please address in author response)
- The paper says  "we … propose multiple ways to automate most of the hyperparameter searching required". I'm not sure that this is true. The only technique that arguably removes a hyperparameter is LSLR. Even in this case, you still have to initialize the inner loop learning rates, so I'm not convinced that even this reduces hyperparameters. Perhaps I've missed something, please clarify.
- Section 4's paragraph on LSLR seems to say that you have a single alpha for each layer of the network. If this is right, then saying your method has a "per layer gradient direction" is very confusing. Each layer's alpha modulates the magnitude of that layer's update vector, but not its direction. The per-layer alphas together modify the direction of the global update vector. Perhaps I've misunderstood; equations describing exactly what LSLR does would be helpful. In any case, this should be clarified in the text.
Suggestions (less essential than the concerns above)
- The write-up is redundant and carries unnecessary content. The paper would be better shorter (8 pages is not a minimum :)
Section 1 covers a lot of background on the basics of meta-learning background that could be skipped. Other papers you cite (e.g. the MAML paper cover this).
    - Section 2 goes into more detail about e.g. matching nets than is necessary.
    - Section 2 explains MAML, which is then covered in much more detail in Section 3; better to leave out the Section 2 MAML paragraph.
    - Sections 3 and 4 are very redundant. Combine them for a shorter (i.e., better!) paper.
- The paper says, "Furthermore, for each learning rate learned, there will be N instances of that learning rate, one for each step to be taken. By doing this, the parameters are free to learn to decrease the learning rates at each step which may help alleviate overfitting." Does this happen empirically? Space could be freed up (see above) to have a figure showing whether or not this happens.
- The paper says, "we propose MAML++, an improved meta-learning framework" -- it's a little too far to call this a new framework. it's still MAML, with improvements.
Typos
- "4) increase the system’s computational overheads" -> overhead
- "composed by" -> composed of
- "Santurkar et al. (2018).", "Krizhevsky et al. (2012),",  "Finn et al. (2017) " -> misplaced citation parens
- "a method that reduce" -> reduces
- "An evaluation ran consisted" -> evaluation consisted
- The Loshchlikov and Hutter citation in the bibliography isn't right. It should be "Sgdr: Stochastic gradient descent with restarts." (2016) instead of "Fixing weight decay regularization in adam" (2017).


Boolean satisfiability (SAT) is one of the most well-known NP-complete problems and has been extensively studied. State-of-the-art solvers exist and have found a wide range of applications. However, they still do not scale well to formulas with hundreds of variables for uniform 3-SAT problems. To tackle this fundamental scalability challenge, we introduce CNNSAT, a fast and accurate statistical decision procedure for SAT based on convolutional neural networks. CNNSAT's effectiveness is due to a precise and compact representation of Boolean formulas. On both real and synthetic formulas, CNNSAT is highly accurate and orders of magnitude faster than the state-of-the-art solver Z3. We also describe how to extend CNNSAT to predict satisfying assignments when it predicts a formula to be satisfiable.

[Second Update] I still find the method proposed in this paper appealing, and think that it may have practical applications in addition to providing significant research contributions. A key question that was raised by the other two reviewers was whether the proposed approach was fairly evaluated against existing state-of-the-art solvers. The authors have responded to these concerns by adding clarifications and new baselines to their paper. However, based on the discussions to date, I feel that I am not sufficiently familiar with related work on SAT solvers to say whether the other reviewers' concerns have been fully addressed. If they have been, I'd strongly lean towards accepting the paper. As for the concerns from my original review: the transferability experiments reported in the author comments below are quite informative, and I'd encourage the authors to incorporate them into the paper (or an appendix if space is an issue). I'd also encourage the author to incorporate the full comparisons against Z3, PicoSAT, MiniSAT, Glucose, Dimetheus, and CaDiCaL from Section 5.1. (I've updated my rating for the paper from 5 to 6, and my confidence score from 3 to 2.)
[First Update] Based on the feedback of the other two reviewers, I believe that I was missing some important context about SAT solvers when I wrote my initial review. Reviewer 1 and Reviewer 2 both raised serious concerns about the types of SAT instances that were used to evaluate the experimental setup, as well as about the use of Z3 as a baseline for solving random SAT instances. (No author response was provided.) Given this additional information, I've lowered my score for the paper from an 8 to a 5. I do think that the approach is interesting, but have reservations about the experimental evaluation and the claims made by the current submission. (Note: As the paper authors point out in the comment below, this update was mistakenly submitted a few days before the end of the rebuttal period.)
[Original Title] Interesting idea, impressive results for a first paper
[Summary] The authors propose a method of using convolutional neural networks to determine whether large boolean formulas (containing hundreds of variables and thousands of clauses) are satisfiable. The resulting models are accurate (correctly distinguishing between satisfiable and unsatisfiable formulas more between 90% and 99% of the time, depending on the dataset) while taking 10x - 100x less time than an off-the-shelf solver (Z3), offering slightly better quality on some problems and slightly worse quality on others. In addition to determining whether formulas are satisfiable, the authors propose and evaluate a method for finding satisfying assignments. They also evaluate their system on SMT benchmarks, where it also shows 10x-100x speed-ups, albeit with somewhat lower accuracy (e.g., 73% - 92% accuracy; I couldn't find baselines for these experiments).
[Key Comments] Unless I'm missing something major, I'd prefer to accept this paper, since the problem appears novel and the experimental results seem very promising for a first paper on a new problem.
[Pro 1] The paper seems polished and well-written. I generally found it well-motivated and easy to follow.
[Pro 2] To the best of my knowledge, the problem domain (machine learning for satisfiability problems that are so large that they are difficult to solve using conventional methods) is both novel and well-motivated.
[Pro 3] Algorithms seem conceptually straightforward (but might be a bit challenging to implement in practice due to the large input size), and yield excellent results. The magnitude of speed-ups reported in the paper (10x - 100x) is large enough to be exciting from a research perspective, and also seems like it should be large enough to have significant practical applications.
[Pro 4] Results are evaluated on a variety of different boolean satisfiability and SMT problems.
[Con 1] To improve reproducibility, it would be helpful if the authors could provide more details about their model training setup. Figure 2 is a good start, but adding details about the layer sizes, types of pooling layers used, and the model training setup would help clarify the experiments.
[Con 2] It seems like a significant number of labeled training examples (i.e., examples that are already known to be satisfiable or unsatisfiable) are needed in order to train a neural network. This seems like it could present a bootstrapping problem for certain domains: it may be computationally expensive to generate ground-truth labels for training examples, but a significant number of labels are needed to train a good prediction model. I'd be very interested to see a study of how well trained models transfer across domains: how well do models trained on one domain (e.g., a synthetic problem where labeled training data is cheap to generate) transfer to a different domain (e.g., a real-world problem where training labels are expensive to compute)? However, this is a minor point for a first paper on a new problem, and I think the paper is interesting enough to merit acceptance without such an analysis.