RIGOROUS AGENT EVALUATION: AN ADVERSARIAL APPROACH TO UNCOVER CATASTROPHIC FAILURES
This paper addresses the problem of evaluating learning systems in safety critical domains such as autonomous driving, where failures can have catastrophic consequences. We focus on two problems: searching for scenarios when learned agents fail and assessing their probability of failure. The standard method for agent evaluation in reinforcement learning, Vanilla Monte Carlo, can miss failures entirely, leading to the deployment of unsafe agents. We demonstrate this is an issue for current agents, where even matching the compute used for training is sometimes insufficient for evaluation. To address this shortcoming, we draw upon the rare event probability estimation literature and propose an adversarial evaluation approach. Our approach focuses evaluation on adversarially chosen situations, while still providing unbiased estimates of failure probabilities. The key difficulty is in identifying these adversarial situations -since failures are rare there is little signal to drive optimization. To solve this we propose a continuation approach that learns failure modes in related but less robust agents. Our approach also allows reuse of data already collected for training the agent. We demonstrate the efficacy of adversarial evaluation on two standard domains: humanoid control and simulated driving. Experimental results show that our methods can find catastrophic failures and estimate failures rates of agents multiple orders of magnitude faster than standard evaluation schemes, in minutes to hours rather than days.

PAPER SUMMARY
-------------
The paper proposes a method for evaluating the failure probability of a learned agent, which is important in safety critical domains.
Using plain Monte Carlo for this evaluation can be too expensive, since discovering a failure probability of epsilon requires on the order of 1/epsilon samples. Therefore the authors propose an adversarial approach, which focuses on scenarios which are difficult for the agent, while still yielding unbiased estimates of failure probabilities.
The key idea of the proposed approach is to learn a failure probability predictor (FPP). This function attempts to predict at which initial states the system will fail. This function is then used in an importance sampling scheme to sample the regions with higher failure probability more often, which leads to higher statistical efficiency.
Finding the FPP is itself a problem which is just as hard as the original problem of estimating the overall failure probability. However, the FPP can be trained using data from different agents, not just the final agent to be evaluated (for instance the data from agent training, containing typically many failure cases). The approach hinges on the assumption that these agents tend to fail in the same states as the final agent, but with higher probability.
The paper shows that the proposed method finds failure cases orders of magnitude faster than standard MC in simulated driving as well as a simulated humanoid task. Since the proposed approach uses data acquired during the training of the agent, it has more information at its disposal than standard MC. However, the paper shows that the proposed method is also orders of magnitudes more efficient than a naive approach using the failure cases during training.
REVIEW SUMMARY
--------------
I believe that this paper addresses an important problem in a novel manner (as far as I can tell) and the experiments are quite convincing.
The main negative point is that I believe that the proposed method has some flaws which may actually decrease statistical efficiency in some cases (please see details below).
DETAILED COMMENTS
-----------------
- It seems to me that a weak point of the method is that it may also severly reduce the efficiency compared to a standard MC method. If the function f underestimates the probability of failure at certain x, it would take a very long time to correct itself because these points would hardly ever be evaluated. It seems that the paper heuristically addresses this to some extent using the exponent alpha of the function. However, I think there should be a more in-depth discussion of this issue. An upper-confidence-bound type of algorithm may be a principled way of addressing this problem.
- The proposed method relies on the ability to initialize the system in any desired state. However, on a physical system, where finding failure cases is particularly important, this is usually not possible. It would be interesting if the paper would discuss how the proposed approach would be used on such real systems.
- On page 6, in the first paragraph, the state is called s instead of x as before. Furthermore, the arguments of f are switched.


In this work, we study the connection between regularization and robustness of deep neural networks by viewing them as elements of a reproducing kernel Hilbert space (RKHS) of functions and by regularizing them using the RKHS norm. Even though this norm cannot be computed, we consider various approximations based on upper and lower bounds. These approximations lead to new strategies for regularization, but also to existing ones such as spectral norm penalties or constraints, gradient penalties, or adversarial training. Besides, the kernel framework allows us to obtain margin-based bounds on adversarial generalization. We show that our new algorithms lead to empirical benefits for learning on small datasets and learning adversarially robust models. We also discuss implications of our regularization framework for learning implicit generative models.

This paper looks at adversarial examples from the context of RKHS norms for neural networks.  The work builds conceptually on the work of Bietti and Mairal (2018), who investigate approximate RKHS norms for neural networks (including computation via a specialized convolutional kernel), and Xu et al., (2009) which looks at robustness properties of kernel classifiers.  The authors discuss how the RKHS norm of neural network functions provide robustness guarantees for the resulting classifier, both in terms of a straightforward robustness property for a given example, as well as in terms of generalization guarantees about robustness.
Overall, I think there are some interesting ideas in this work, but ultimately not enough to make a compelling independent paper.  The core issue here is that the RKHS properties are used only in a very minimal manner to actually provide much analysis or insight into the robustness properties of the network.  For example, the upper bound in (8) seems to be central here to illustrating how a bound on the RKHS norm can be upper bounded as a function of the operator l2 norm of the inner weight matrices (though the actual form of the bound isn't mentioned), and the latter term could thus provide a certified bound on the robustness loss of a classifier.  However, there are two big issues here: 1) it's trivial to directly bound the l2 robustness of a classifier by the product of the weight spectral norms and 2) the actual regularization term the authors proposed to use (the sum of spectral norms) is notably _not_ an upper bound on either the robust loss or the RKHS norm; naturally, this penalty along with the constrained version will still provide some degree of control over the actual robustness, but the authors don't connect this to any real bound.  I also think the authors aren't properly acknowledging just how similar this is to past work: the Parseval networks paper (Cisse et al., 2017), for instance, presents a lot of similar discussion of how to bound generalization error based based upon terms involving operator norms of the matrices, and the actual spectral normalization penalty that the authors advocate for has been studied by Miyato et al. (2018).  To be clear, both of these past works (and several similar ones) are of course cited by the current paper, but from a practical standpoint it's just not clear to me what the takeaways should be here above and beyond this past work, other than the fact that these quantities _also_ bound the relevant RKHS norms.  Likewise the generalization bound in the paper is a fairly straightforward application of existing bounds given the mechanics of the RKHS norm defined by previous work.
To be clear, I think the RKHS perspective that the authors advocate for here is actually quite interesting.  I wasn't particularly familiar with the Bietti and Mairal (2018) work, and going through it in some detail for reviewing this paper, I think it's an important directly for analysis of deep networks, including from a perspective of robustness.  But the results here seem more like a brief follow-on note to the past work, not a complete set of results in and of themselves.  Indeed, because the robustness perspective here can largely be derived completely independently of the RKHS framework, and because the resulting training procedures seem to be essentially identical to previously-proposed approaches, the mere contribution of connecting these works to the RKHS norm doesn't seem independently to be enough of a contribution in my mind.
One final, though more minor, point: It's worth pointing out that (globally) bounding the Lipschitz constant seems top stringent a condition for most networks, and most papers on certifiable robustness seem to instead focus on some kind of local Lipschitz bound around the training or test examples.  Thus, it's debatable whether even the lower bound on the RKHS norm is really reasonable to consider for the purposes of adversarial robustness.


COMPETITIVE EXPERIENCE REPLAY
Deep learning has achieved remarkable successes in solving challenging reinforcement learning (RL) problems when dense reward function is provided. However, in sparse reward environment it still often suffers from the need to carefully shape reward function to guide policy optimization. This limits the applicability of RL in the real world since both reinforcement learning and domain-specific knowledge are required. It is therefore of great practical importance to develop algorithms which can learn from a binary signal indicating successful task completion or other unshaped, sparse reward signals. We propose a novel method called competitive experience replay, which efficiently supplements a sparse reward by placing learning in the context of an exploration competition between a pair of agents. Our method complements the recently proposed hindsight experience replay (HER) by inducing an automatic exploratory curriculum. We evaluate our approach on the tasks of reaching various goal locations in an ant maze and manipulating objects with a robotic arm. Each task provides only binary rewards indicating whether or not the goal is achieved. Our method asymmetrically augments these sparse rewards for a pair of agents each learning the same task, creating a competitive game designed to drive exploration. Extensive experiments demonstrate that this method leads to faster converge and improved task performance.

The authors propose a new method for learning from sparse rewards in model-free reinforcement learning settings. This is a challenging and important problem in model-free RL, mainly due to the lack of effective exploration. They propose a new way of densifying the reward by encouraging a pair of agents to explore different states (using competitive self-play) while trying to learn the same task. One of the agents (A) receives a penalty for visiting states that the other agent (B) also visits, while B is rewarded for visiting states found by A. They evaluate their method on a few tasks with continuous action spaces such as ant navigation in a maze and object manipulation by a simulated robotic arm.  Their method shows faster convergence (in some cases) and better performance than comparable algorithms.

Strengths:
Attempts to solve a long-standing problem in model-free RL (effective exploration in sparse reward environments)
Clear writing and structure, easy to understand (except for some minor details)
Novel, intuitive, and simple method building on ideas from previous works
Good empirical results (better than state of the art, in terms of performance) on some challenging tasks
Weaknesses:
Not very clear why (and when) the method works -- more insight from experiments in less complex environments or some theoretical analysis would be helpful
It would also be useful to better understand the conditions under which we can expect this to bring significant gains and when we can expect this to fail (or not help more than other methods)
Not clear how stable (to train) and robust (to different environment dynamics) the method is
Main Comments / Questions:
The paper makes the claim that their technique “automatically generates a curriculum of exploration” which seems to be based more on intuition rather than clear experiments or analysis. I would suggest to either avoid making such claims or include stronger evidence for that. For example, you could consider visualizing the visited states by A and B (for a fixed goal and initial state) at different training epochs. Other such experiments and analysis would be very helpful.
It is known that certain reward shaping approaches can have negative consequences and lead to undesired behaviors (Ng et al., 1999; Clark & Amodei, 2016). Why can we expect that this particular type of reward shaping doesn’t have such side effects? Can it be the case that due to this adversarial reward structure, A learns a policy that takes it to some bad states from which it will be difficult to recover or that A & B get stuck in a cyclic behavior? Have you observed such behaviors in any of your experiments?
Do you train the agents with using the shaped reward (from the exploration competition between A and B) for the entire training duration? Have you tried to continue training from sparse reward only (e.g. after the effect ratio has stabilized)? One problem I see with this approach is the fact that you never directly optimize the true sparse reward of the tasks, so in the late stages of training your performance might suffer because the agent A is still trying to explore different parts of the state space.
Can you comment on how stable this method is to train (given its adversarial nature) and what potential tricks can help in practice (except for the discussion on batch size)?
Please make clear the way you are generating the result plots (i.e. is A evaluated on the full task with sparse reward and initial goal distribution with no relabelling?).
In Algorithm 1, can you include the initialization of the goals for A and B? Does B receive identical goals as A?
It would also be helpful to more clearly state the limitations and advantages of this method compared to other algorithms designed for more efficient exploration (e.g. the need for a resettable environment for int-CER but not for ind-CER etc.).
Minor Comments / Questions:
You might consider including more references in the Related Work section that initializing from different state distributions such as Hosu & Rebedea (2016), Zhu et al. (2016), and Kakade & Langford (2002), and perhaps more papers tackling the exploration problem.
Can you provide some intuition on why int-CER performs better than ind-CER (on most tasks) and why in Figure 1, HER + int-CER takes longer to converge than the other methods on the S maze?
In Figure 4, why are you not including ind-CER (without HER)?
Have you considered training a pool of agents with self-play (for the competitive exploration) instead of two agents? Is there any intuition on expecting one or the other to perform better?
Plots:
What is the x-axis of the plots? Number of samples, episodes, epochs? Please label it.
Please be explicit about the variance shown in the plots. Is that the std?
It would be helpful if to have larger numbers on the xy-axes. It is difficult to read when on paper.
Can you explain how you smoothed the curves -- whether before or after taking the average and perhaps include the min and max as well. I believe this could go in the Appendix.
Notation:
I don’t understand the need for calling the reward r_g instead of r. I believe this introduces confusion since the framework already has r taking as argument the goal g (eq. 1) while the g in the subscript doesn’t seem to refer to a particular g but rather to a general fact (that this is a reward for a goal-oriented task with sparse reward, where the goals are a subset of the states) (eq. 4)
Please use a consistent notation for Q. In sections 2.1 and 2.2, at times you use Q(s,a,g), Q(a,s,g) or Q(s,a).
Typos:
Page 6, last paragraph of section 4.1: Interestingly, even the … , is enough to support …
Page 7, last paragraph of section 4.3: Interestingly, … adversely affects both ...


Being able to automatically discover synonymous entities from a large free-text corpus has transformative effects on structured knowledge discovery. Existing works either require structured annotations, or fail to incorporate context information effectively, which lower the efficiency of information usage. In this paper, we propose a framework for synonym discovery from free-text corpus without structured annotation. As one of the key components in synonym discovery, we introduce a novel neural network model SYNONYMNET to determine whether or not two given entities are synonym with each other. Instead of using entities features, SYNONYMNET makes use of multiple pieces of contexts in which the entity is mentioned, and compares the context-level similarity via a bilateral matching schema to determine synonymity. Experimental results demonstrate that the proposed model achieves state-of-the-art results on both generic and domain-specific synonym datasets: Wiki+Freebase, PubMed+UMLS and MedBook+MKG, with up to 4.16% improvement in terms of Area Under the Curve (AUC) and 3.19% in terms of Mean Average Precision (MAP) compare to the best baseline method.

This paper studies the problem of identifying (discovering) synonymous entities. The paper proposes using the "contexts" of the entities as they occur in associated text corpora (e.g. Wiki) in the proposed neural-network based embedding approach for this task. The key novelties of the approach lie in the "matching" system used, where contexts of one entity are matched with that for the other entity to see how well they align with each other (which effectively determines the similarity of the two entities). Experiments are conducted on three different datasets to show the efficacy of the proposed approach.
Overall I found the paper to be an interesting read with some nice ideas mixed in. However I also had some concerns which are highlighted later down below, which I believe if addressed would lead to a very strong work.
Quality: Above average
In general the method seems to work somewhat better than the baselines and the method does have a couple of interesting ideas.
Clarity: Average
I found a few key details to be missing and also felt the paper could have been better written.
Originality: Average
The matching approach and use of the leaky units was interesting tidbits. Outside of that the work is largely about the application of such Siamese RNNs based networks to this specific problem. (The use of context of entities has already been looked at in previous works albeit in a slightly more limited manner)
Significance: Slightly below average
I am not entirely sold on the use of this approach for this problem given its complexity and unclear empirical gains vs more sophisticated baselines. The matching aspect may have some use in other problems but nothing immediately jumps out as an obvious application.
----
Strengths / Things I liked about the paper:
- In general the method is fairly intuitive and simple to follow which I liked.
- The matching approach was an interesting touch.
- Similarly for the "leaky" unit.
- Experiments conducted on multiple datasets.
- The results indicate improvements over the baselines considered on all the three datasets.
Weaknesses / Things that concerned me:
-  (W1) Slightly unfair baselines? One of the first things that struck me in the experimental results was how competitive word2vec by itself was across all three datasets. This made me wonder what would happen if we were to use a more powerful embedding approach say FastText, Elmo, Cove or the recently proposed BERT? (The proposed method itself uses bidirectional LSTMs)
Furthermore all of them are equally capable of capturing the contexts as well. An even more competitive (and fair) set of baselines could have taken the contexts as well and use their embeddings as well. Currently the word2vec baseline is only using the embedding of the entity (text), whereas the proposed approach is also provided the different contexts at inference time. The paper says using the semantic structure and the diverse contexts are weaknesses of approaches using the contexts, but I don't see any method that uses the context in an embedding manner -- say the Cove context vectors. If the claim is that they won't add any additional value above what is already captured by the entity it would be good to empirically demonstrate this.
- (W2) Significance testing: On the topic of experimentation, I was concerned that significance testing / error estimates weren't provided for the main emprical results. The performance gaps seem to be quite small and to me it is unclear how significant these gaps are. Given how important significance testing is as an empirical practice this seems like a notable oversight which I would urge the authors to address.
- (W3) Missing key details: There were some key aspects of the work that I thought were not detailed. Chief among these was the selection of the contexts for the entities. How was this? How were the 20 contexts identified? Some of these entities are likely far more common than just 20 sentences and hence I wonder how these were selected?
Another key aspect I did not see addressed: How were the entities identified in the text (to be able to find the contexts for them)? The paper claims that they would like to learn from minimal human annotations but I don't understand how these entity annotations in the text were obtained. This again seems like a notable oversight.
- (W4) Concerns about the method: I had two major concerns about the method:
(a) Complexity of method :  I don't see an analysis of the computational cost of the proposed method (which scales quadratically with P the number of contexts);
(b) Effect of redundant "informative" contexts: Imagine you have a number of highly informative contexts for an entity but they are all very similar to each other. Due to the way the matching scores are aggregated, these scores are made to sum to 1 and hence no individual score would be very high. Given that this is the final coefficient for the associated context, this seems like a significant issue right?
Unless the contexts are selected to be maximally diverse, it seems like this can essentially end up hurting an entity which occurs in similar contexts repeatedly. I would like to see have seen the rationale for this better explained.
(c) A smaller concern was understanding the reasoning behind the different loss functions in the siamese loss function with a different loss for the positive and the negative, one using a margin and one which doesn't. One which scales to 1/4, the other scaling to (1-m)^2. This seems pretty arbitrary and I'd like to understand this.
-(W5) Eval setting : My last concern was with the overall evaluation setup. Knowledge bases like Freebase are optimized for precision rather than recall, which is why "discovery" of new relations is important. However if you treat all missing relationships as negative examples then how exactly are you measuring the true ability of a method? Thus overall I'm pretty skeptical about all the given numbers simply because we know the KBs are incomplete, but are penalizing methods that may potentially discover relations not in the KB.


Similar to models of brain-like computation, artificial deep neural networks rely on distributed coding, parallel processing and plastic synaptic weights. Training deep neural networks with the error-backpropagation algorithm, however, is considered bio-implausible. An appealing alternative to training deep neural networks is to use one or a few hidden layers with fixed random weights or trained with an unsupervised, local learning rule and train a single readout layer with a supervised, local learning rule. We find that a network of leaky-integrate-andfire neurons with fixed random, localized receptive fields in the hidden layer and spike timing dependent plasticity to train the readout layer achieves 98.1% test accuracy on MNIST, which is close to the optimal result achievable with errorbackpropagation in non-convolutional networks of rate neurons with one hidden layer. To support the design choices of the spiking network, we systematically compare the classification performance of rate networks with a single hidden layer, where the weights of this layer are either random and fixed, trained with unsupervised Principal Component Analysis or Sparse Coding, or trained with the backpropagation algorithm. This comparison revealed, first, that unsupervised learning does not lead to better performance than fixed random projections for large hidden layers on digit classification (MNIST) and object recognition (CI-FAR10); second, networks with random projections and localized receptive fields perform significantly better than networks with all-to-all connectivity and almost reach the performance of networks trained with the backpropagation algorithm. The performance of these simple random projection networks is comparable to most current models of bio-plausible deep learning and thus provides an interesting benchmark for future approaches.

This article compares different methods to train a two-layer spiking neural network (SNN) in a bio-plausible way on the MNIST dataset, showing that fixed localized random connections that form the hidden layer, in combination with a supervised local learning rule on the output layer can achieve close to state-of-the-art accuracy compared to other SNN architectures. The authors investigate three methods to train the first layer in an unsupervised way: principal component analysis (PCA) on the rates, sparse coding of activations, and fixed random local receptive fields.  Each of the methods is evaluated on the one hand in a time-stepped simulator, using LIF neurons and on the other hand using a rate-approximated model which allows for faster simulations. Results are compared between each other and as reference with  standard backpropagation and feedback alignment.  The main finding is that localized random projections outperform other unsupervised ways of computing first layer features, and with many hidden neurons approaches backpropagation results. These results are summarized in Table 8, which compares results of the paper and other state-of-the-art and bio-plausible SNNs. PCA and sparse coding work worse on MNIST than local random projections, regardless if the network is rate-based, spike-based or a regular ANN trained with the delta rule. Feedback Alignment, although only meant for comparison, performs best of the algorithms investigated in this paper.
In general the question how to train multi-layer spiking neural networks in a bio-plausible way is very relevant for computational neuroscience, and has attracted some attention from the machine learning community in recent years (e.g. Bengio et al. 2015, Scellier & Bengio 2016, Sacramento et al. 2018). It is therefore a suitable topic for ICLR. Of course the good performance of single-layer random projections is not surprising, because it is essentially the idea of the Extreme Learning Machine, and this concept has been well studied also for neuromorphic approaches (e.g. Yao & Basu, 2017), and versions with local receptive fields exist as well (Huang et al. 2015 "Local Receptive Fields Based Extreme Learning Machine"). While the comparison of different unsupervised methods on MNIST is somehow interesting, it fails to show any deeper insights because MNIST is a particularly simple task, and already the CIFAR 10 results are far away from the state-of-the-art (which is >96% using CNNs). Another interesting comparison that is missing is with clustering weights, which has shown good performance for CNNs e.g. in (Coates & Ng, 2012) or (Dundar et al. 2015), and is also unsupervised.
The motivation is not 100% clear because the first experiment uses spikes, and shows a non-negligible difference to rate models (the authors claim it's almost the same, but for MNIST differences of 0.5% are significant). All later results are purely about rate models. The authors apparently did not explore e.g. conversion techniques as in (Diehl et al. 2015) to make the spiking results match the rate versions better e.g. by weight normalization.
I would rate the significance to the SNN community as average, and to the entire ICLR community as low. The significance would be higher if it was shown that this method scales to deeper networks or at least can be utilized in deeper architectures. Scrutinizing the possibilitites with random projections on the other hand could lead to more interesting results. But the best results here are obtained with 5000 neurons with 10x10 receptive fields on images of size 28x28, thus the representation is more than overcomplete, and of higher complexity than a convolution layer with 3x3 kernels and many input maps.
Because the results provide only limited insights beyond MNIST I can therefore not support acceptance at ICLR.
Pros:
+ interesting comparison of unsupervised feature learning techniques
+ interesting topic of bio-plausible deep learning
Cons:
- only MNIST, no indications if method will scale
- results are not better than state-of-the-art
Minor comments:
The paper is generally well-written and structured, although some of the design choices could have been explained in more detail. Generally, it is not discussed if random connections have any advantage over other spiking models in terms of accuracy, efficiency or speed, besides the obvious fact that one does not have to train this layer.
The title is a bit confusing. While it's not wrong, I had to read it multiple times to understand what was meant.
The first sentence in the caption for Fig. 2 is also confusing, mixing the descriptions of panel A and B. Also, in A membrane potentials are shown, but the post-membrane potential seems to integrate a constant current instead of individual spikes. Is this already the rate approximation of Eq. 2? Or is it because of the statement in the caption that they both receive very high external inputs. In general, the figures in panel A and B do not make the dynamics of the network or the supervised STDP much clearer.
Principal Component Analysis and Sparse Coding are done algorithmically instead of using a sort of nonlinear Hebbian Learning as in Lillicrap 2016. It would have been interesting to see if this changes the comparatively bad results for PCA and SC.
In Fig. 3, the curve in the random projections case is not saturated, maybe it would have been interesting to go above n_h = 5000. As there are 784 input neurons, a convolutional neural network with 7 filter banks already would have around 5000 neurons, but in this case each filter would be convolved over the whole image, while with random projections the filter only exists locally.
In Eq. 1, the notation is a bit ambigous: The first delta-function seems to be the Dirac-delta for continuous t, while the second delta is a Kronecker-delta with discrete t.
In A.1 and A.4.2 it is stated that the output of a layer is u_{t+1} = W u_t + b but I think in both cases it should be W a_t + b where a_t = phi(u_t). Otherwise, you just have a linear model and no activations.
In Table 3, a typo: "eq. equation"


Imitation learning is an effective alternative approach to learn a policy when the reward function is sparse. In this paper, we consider a challenging setting where an agent has access to a sparse reward function and state-only expert observations. We propose a method which gradually balances between the imitation learning cost and the reinforcement learning objective. Built upon an existing imitation learning method, our approach works with state-only observations. We show, through navigation scenarios, that (i) an agent is able to efficiently leverage sparse rewards to outperform standard state-only imitation learning, (ii) it can learn a policy even when learner's actions are different from the expert, and (iii) the performance of the agent is not bounded by that of the expert due to the optimized usage of sparse rewards.

The draft proposes a heuristic combining environment rewards with an IRL-style rewards recovered from expert demonstrations, seeking to extend the GAIL approach to IRL to the case of mismatching action spaces between the expert and the learner. The interesting contribution is, in my opinion, the self-exploration parameter that reduces the reliance of learning on demonstrations once they have been learned sufficiently well.
Questions:
- In general, it's known that behavioral cloning, of which this work seem to be an example in so much it learns state distributions that are indistinguishable from the expert ones, can fail spectacularly because of the distribution shift (Kaariainen@ALW06, Ross&Bagnell@AISTATS10, Ross&Bagnell@AISTATS11). Can you comment if GAN-based methods are immune or susceptible to this?

- Would this work for tasks where the state-space has to be learned together with the policy? E.g. image captioning tasks or Atari games.
- Is it possible to quantify the ease of learning or the frequency of use of the "new" actions, i.e. $A^l \setminus A^e$?. Won't learning these actions effectively be as difficult as RL with sparse rewards? Say, in a grid world where 4-way diagonal moves allow reaching the goal faster, learner is king 8-way, demonstrations come from a 4-way expert, rewards are sparse and each step receives a -1 reward and the final goal is large positive -- does the learner's final policy actually use the diagonals and when?
Related work:

- Is it possible to make a connection to (data or policy) aggregation methods in IL. Such methods (e.g. Chang et al.@ICML15) can also sometimes learn policies better than the expert.
Experiments:
- why GAIL wasn't evaluated in Fig. 3 and Fig. 4?
Minor:
- what's BCE in algorithm 1?
- Fig.1: "the the"
- sec 3.2: but avoid -> but avoids
- sec 3.2: be to considers -> be to consider
- sec 3.2: any hyperparameter -> any hyperparameters
- colors in Fig 2 are indistinguishable
- Table 1: headers saying which method is prior work and which is contribution would be helpful
- Fig. 3: if possible try to find a way of communicating the relation of action spaces between expert and learner (e.g. a subset of/superset of). Using the same figure to depict self-exploration make it complicated to analyse.
- sec 3.2: wording in the last paragraph on p.4 (positive scaling won't _make_ anything positive if it wasn't before)


IMPROVING DIFFERENTIABLE NEURAL COMPUTERS THROUGH MEMORY MASKING, DE-ALLOCATION, AND LINK DISTRIBUTION SHARPNESS CONTROL
The Differentiable Neural Computer (DNC) can learn algorithmic and question answering tasks. An analysis of its internal activation patterns reveals three problems: Most importantly, the lack of key-value separation makes the address distribution resulting from content-based look-up noisy and flat, since the value influences the score calculation, although only the key should. Second, DNC's deallocation of memory results in aliasing, which is a problem for content-based look-up. Thirdly, chaining memory reads with the temporal linkage matrix exponentially degrades the quality of the address distribution. Our proposed fixes of these problems yield improved performance on arithmetic tasks, and also improve the mean error rate on the bAbI question answering dataset by 43%.

Overview:
This paper proposes modifications to the original Differentiable Neural Computer architecture in three ways. First by introducing a masked content-based addressing which dynamically induces a key-value separation. Second, by modifying the de-allocation system by also multiplying the memory contents by a retention vector before an update. Finally, the authors propose a modification in the link distribution, through renormalization. They provide some theoretical motivation and empirical evidence that it helps avoiding memory aliasing.
The authors test their approach in the some algorithm task from the DNC paper (Copy, Associative Recall and Key-Value Retrieval), and also in the bAbi dataset.
Strengths: Overall I think the paper is well-written, and proposes simple adaptions to the DNC architecture which are theoretically grounded and could be effective for improving general performance. Although the experimental results seem promising when comparing the modified architecture to the original DNC, in my opinion there are a few fundamental problems in the empirical session (see weakness discussion bellow).
Weaknesses: Not all model modifications are studied in all the algorithmic tasks. For example, in the associative recall and key-value retrieval only DNC and DNC + masking are studied.
For the bAbi task, although there is a significant improvement (43%) in the mean error rate compared to the original DNC, it's important to note that performance in this task has improved a lot since the DNC paper was release. Since this is the only non-toy task in the paper, in my opinion, the authors have to discuss current SOTA on it, and have to cite, for example the universal transformer[1], entnet[2], relational nets [3], among others architectures that shown recent advances on this benchmark.
Moreover, the sparse DNC (Rae el at., 2016) is already a much better performant in this task. (mean error DNC: 16.7 \pm 7.6, DNC-MD (this paper) 9.5 \pm 1.6, sparse DNC 6.4 \pm 2.5). Although the authors mention in the conclusion that it's future work to merge their proposed changes into the sparse DNC, it is hard to know how relevant the improvements are, knowing that there are much better baselines for this task.
It would also be good if besides the mean error rates, they reported best runs chosen by performance on the validation task, and number of the tasks solve (with < 5% error) as it is standard in this dataset.
Smaller Notes.
1) In the abstract, I find the message for motivating the masking from the sentence  "content based look-up results... which is not present in the key and need to be retrieved."  hard to understand by itself. When I first read the abstract, I couldn't understand what the authors wanted to communicate with it. Later in 3.1 it became clear.
2) page 3, beta in that equation is not defined
3) First paragraph in page 5 uses definition of acronyms DNC-MS and DNC-MDS before they are defined.
4) Table 1 difference between DNC and DNC (DM) is not clear. I am assuming it's the numbers reported in the paper, vs the author's implementation?
5)In session 3.1-3.3, for completeness. I think it would be helpful to explicitly compare the equations from the original DNC paper with the new proposed ones.
--------------
Post rebuttal update: I think the authors have addressed my main concern points and I am updating my score accordingly.


Learning disentangling representations of the independent factors of variations that explain the data in an unsupervised setting is still a major challenge. In the following paper we address the task of disentanglement and introduce a new state-ofthe-art approach called Non-synergistic variational Autoencoder (Non-Syn VAE). Our model draws inspiration from population coding, where the notion of synergy arises when we describe the encoded information by neurons in the form of responses from the stimuli. If those responses convey more information together than separate as independent sources of encoding information,they are acting synergetically. By penalizing the synergistic mutual information within the latents we encourage information independence and by doing that disentangle the latent factors. Notably, our approach could be added to the VAE framework easily, where the new ELBO function is still a lower bound on the log likelihood p x . In addition, we qualitatively compare our model with Factor VAE and show that this one implicitly minimises the synergy of the latents.

This paper proposes a new approach to enforcing disentanglement in VAEs using a term that penalizes the synergistic mutual information between the latent variables, encouraging representations where any given piece of information about a datapoint can be garnered from a single latent.  In other words, representations where there is no information conveyed by combinations of latents that is not conveyed by considering each latent in isolation.  As the resultant target is intractable to evaluate, a number of approximations are employed for practical training.
The high-level idea is quite interesting, but the paper itself is quite a long way of convincing me that this is actually a good approach.  Moreover, the paper is a long way of the level of completeness, rigor, clarity, and polish that is required to seriously consider it for publication.  In short, the work is still at a relatively early stage and a lot more would need to be done for it to attain various minimum standards for acceptance.  A non-exhaustive list of specific examples of its shortfalls are given below.
1. The paper is over a page and a half under length, despite wasting large amounts of space (e.g. figures 3 and 4 should be two lines on the same plot)
2. The experimental evaluation is woefully inadequate.  The only quantitative assessment is to compare to a single different approach on a single toy dataset, and even then the metric being used is the one the new method uses to train for making it somewhat meaningless.
3. The introduction is completely generic and says nothing about the method itself, just providing a (not especially compelling) motivation for disentanglement in general.  In fact, the motivation of the introduction is somewhat at odds with the work -- correctly talking about the need for hierarchical representations which the approach actually actively discourages.
4. There are insufficient details on the algorithm itself in terms of the approximations that are made to estimate the synergistic mutual information.  These are mostly glossed over with only a very short explanation in the paragraph after equation 15.  Yes there are algorithm blocks, but these are pretty incomprehensible and lack accompanying text.  In particular, I cannot understand what A_w is supposed to be.  This is very important as I suspect the behavior of the approximation is very different to the true target.  Similarly, it would be good to provide more insight into the desired target (i.e. Eq 15).  For example, I suspect that it will encourage a mismatch between the aggregate posterior and prior by encouraging higher entropy on the former, in turn causing samples from the generative model to provide a poor match to the data.
5. The repeated claims of the approach and results being "state-of-the-art" are cringe-worthy bordering on amusing.  Writing like this serves no purpose even when it justified, and it certainly is not here.
6. There are a lot of typos throughout and the production values are rather poor.  For example, the algorithm blocks which are extremely messy to the point where they are difficult to follow, citep/citet mistakes occur almost every other citation, there is a sign error in Equation 16.
This is a piece of work in an exciting research area that,  with substantial extra work, could potentially result in a decent paper due to fact that the core idea is simple and original.  However, it is a long way short of this in its current state.  Along with addressing the specific issues above and improving the clarity of the work more generally, one thing in particular that would need to address in a resubmission is a more careful motivation for the method (ideally in the form of a proper introduction).
Though I appreciate this is a somewhat subjective opinion, for me, penalizing the synergistic information is probably actually a bad thing to do when taking a more long-term view on disentanglement.  Forcing simplistic representations where no information is conveyed through the composition of latents beyond that they provide in isolation is all well and good for highly artificial and simplistic datasets like dsprites, but is clearly not a generalizable approach for larger datasets where no such simplistic representation exists.  As you say in the first line of your own introduction, hierarchy and composition are key parts of learning effective and interpretable representations and this is exactly what you are discouraging.  A lot of the issue here is one of the disentanglement literature at large rather than this paper (though I do find it to be a particularly egregious offender) and it is fine to have different opinions.  However, it is necessary to at least make a sensible case for why your approach is actually useful.
Namely, is there actually any real applications where such a simplistic disentanglement is actually useful?  Is there are anyway the current works helps in the longer vision of achieving interpretable representations?  When and why is the synergistic information a better regularizer than, for example, the total correlation?  The experiments you have do not make any inroads to answering these questions and there are no written arguments of note to address them.  I am not trying to argue here that there isn't a good case to be made for the suggested approach in the context of these questions (though I am suspicious), just that if the work is going to have any lasting impact on the community then it needs to at least consider them.


Domain specific goal-oriented dialogue systems typically require modeling three types of inputs, viz., (i) the knowledge-base associated with the domain, (ii) the history of the conversation, which is a sequence of utterances and (iii) the current utterance for which the response needs to be generated. While modeling these inputs, current state-of-the-art models such as Mem2Seq typically ignore the rich structure inherent in the knowledge graph and the sentences in the conversation context. Inspired by the recent success of structure-aware Graph Convolutional Networks (GCNs) for various NLP tasks such as machine translation, semantic role labeling and document dating, we propose a memory augmented GCN for goal-oriented dialogues. Our model exploits (i) the entity relation graph in a knowledge-base and (ii) the dependency graph associated with an utterance to compute richer representations for words and entities. Further, we take cognizance of the fact that in certain situations, such as, when the conversation is in a code-mixed language, dependency parsers may not be available. We show that in such situations we could use the global word co-occurrence graph and use it to enrich the representations of utterances. We experiment with the modified DSTC2 dataset and its recently released code-mixed versions in four languages and show that our method outperforms existing state-of-the-art methods, using a wide range of evaluation metrics.

The current paper proposes using Graph Convolutional Networks (GCN) to explicitly represent and use relational data in dialog modeling, as well an attention mechanism for combining information from multiple sources (dialog history, knowledge base, current utterance). The work assumes that the knowledge base associated with the dialog task has en entity-to-entity-relationship format and can be naturally expressed as a graph. The dependency tree of dialog utterances can also be expressed as a graph, and the dialog history as a set of graphs. To utilize this structure, the proposed method uses GCNs whose lowest layer embeddings are initialized with the entity embeddings or via outputs of standard RNN-like models. The main claim is that the proposed model outperforms the current state-of-the-art on a goal-oriented dialog task.
The idea of explicitly modeling the relational structure via GCNs is interesting. However, the use of GCNs independently per sentence and per knowledge-base is a bit disappointing, since it does not couple these sources of information in a structured way. Instead, from my current understanding, the approach merely obtains better representations for each of these sources of information, in the same way it is done in the related language tasks. For instance, have you considered passing information across the trees in the history as well? Or aligning the parsed query elements with the KB elements?
The results are very good. That said, a source of concern is that the model is only evaluated as a whole, without showing which modification brought the improvements. The comparison between using/not using RNNs to initiate the first GCN layer is promising, but why not compare to using only RNN also? Why not compare the various encoders within an established framework (e.g. without the newly introduced attention mechanism)? Finally, the attention mechanism, stated as a contribution, is not motivated well.
Clarity:
The notation is described well, but it's not terribly intuitive (the query embedding is denoted by c, the history embedding by a, etc.), making section 4.4. hard to follow. A figure would have made things easier to follow, esp. due to the complexity of the model. A clearer parallel with previous methods would also improve the paper: is the proposed approach adding GCN on top of an established pipeline? Why not?
More discussion on code-mixed language, e.g. in section 4.6, would also improve clarity a bit (make the paper more self-contained). While the concept is clear from the context, it would be helpful to describe the level of structure in the mixed language. For instance, can dependency trees not be obtained code-mixed languages? Is there any research in this direction? (or is the concept very new?) Maybe I am just missing the background here, but it seems helpful in order to asses how appropriate the selected heuristic (based on the co-occurence matrix) is.
Relevant Reference:
Learning Graphical State Transitions, Johnson, ICLR 2017 also uses graph representations in question answering, though in a somewhat different setting.
Typos:
Section 4: "a model with following components"
Section 5: "the various hyperparameters that we conisdered"


Different kinds of representation learning techniques on graph have shown significant effect in downstream machine learning tasks. Recently, in order to inductively learn representations for graph structures that is unobservable during training, a general framework with sampling and aggregating (GraphSAGE) was proposed by Hamilton and Ying and had been proved more efficient than transductive methods on fileds like transfer learning or evolving dataset. However, GraphSAGE is uncapable of selective neighbor sampling and lack of memory of known nodes that've been trained. To address these problems, we present an unsupervised method that samples neighborhood information attended by co-occurring structures and optimizes a trainable global bias as a representation expectation for each node in the given graph. Experiments show that our approach outperforms the state-of-the-art inductive and unsupervised methods for representation learning on graphs.

This paper modifies the GraphSAGE on unsupervised inductive node embedding.
The authors propose to use the bi-attention architecture to sample
interesting nodes (instead of the uniform sampler in GraphSAGE), and to use a
global embedding bias matrix in the local aggregating functions. The method
showed improvements over GraphSAGE and other baselines on unsupervised
graph embeddings.
The proposition makes sense and the performance improvements are expected.
A major comment, however, is that that the proposed method is useful in very
restricted settings, and it is not clear how to generalize to
other applications which GraphSAGE can be applied on.
The overall technical contribution is incremental and
may not have enough novelty to be published in ICLR.
The technical representation is very poor, unorganized and not self-contained.
The paper cannot pass the threshold merely based on the way it is presented.
In the algorithms, please give the output besides the input. After the
algorithms, please remark on the computational and memory cost.
In algorithm 1, what is this function BIATT()?
After algorithm 1, please describe this function as well as SAGE().
In the beginning of section 3, please describe the meaning of the
global bias matrix. In algorithm 1, if B is zero-initialized, why
does one need it as input?
Some of the equations are poor formatted (e.g. reduce_sum in page 5).
Please try to use rigorous mathematical formulations instead of "pseudo equations".
For example, re-write "One_hot(i)". In section 3.2, explain A_{gg}, etc.
use $\langle \rangle$ instead of $\alpha$.
There are many typos.


Reasoning over text and Knowledge Bases (KBs) is a major challenge for Artificial Intelligence, with applications in machine reading, dialogue, and question answering. Transducing text to logical forms which can be operated on is a brittle and error-prone process. Operating directly on text by jointly learning representations and transformations thereof by means of neural architectures that lack the ability to learn and exploit general rules can be very data-inefficient and not generalise correctly. These issues are addressed by Neural Theorem Provers (NTPs) , neuro-symbolic systems based on a continuous relaxation of Prolog's backward chaining algorithm, where symbolic unification between atoms is replaced by a differentiable operator computing the similarity between their embedding representations. In this paper, we first propose Neighbourhoodapproximated Neural Theorem Provers (NaNTPs) consisting of two extensions to NTPs, namely a) a method for drastically reducing the previously prohibitive time and space complexity during inference and learning, and b) an attention mechanism for improving the rule learning process, deeming them usable on real-world datasets. Then, we propose a novel approach for jointly reasoning over KB facts and textual mentions, by jointly embedding them in a shared embedding space. The proposed method is able to extract rules and provide explanations-involving both textual patterns and KB relations-from large KBs and text corpora. We show that NaNTPs perform on par with NTPs at a fraction of a cost, and can achieve competitive link prediction results on challenging large-scale datasets, including WN18, WN18RR, and FB15k-237 (with and without textual mentions) while being able to provide explanations for each prediction and extract interpretable rules.

[Summary]
This paper scales NTPs by using approximate nearest neighbour search over facts and rules during unification. Additionally, the paper incorporates mentions as additional facts where the predicate is the text that the entities of the mention are contained in. The paper also suggests parameterizing predicates using attention over known predicates. The increments presented are reasonable and justified, but the experimental results, specifically on the larger datasets, warrant further investigation.
[Pros]
- Reasonable and interesting increments on top of NTP.
- Scaling the approach to larger datasets is well motivated.
- Utilizing text is an interesting direction for NTP in terms of integrating it with past work on KG completion.
[Cons]
- Empirical performance on larger datasets needs further investigation.
- No ablation study is performed so the effect of incorporating mentions and attention are unclear.
- Baseline performance on FB15k-237 seems weak compared to the original papers as well as more recent papers re-examining baselines for KG completion (http://aclweb.org/anthology/W17-2609). Is this due to the d=100 restriction, or were pretrained embeddings not used? Without further explanation, the claim that scores are competitive with SOTA seems unjustified, at least for FB15k-237 since the model performs significantly worse than the baselines which seem to be worse than previously reported.
[Comments]
- For reproducibility: it is unclear whether evaluation in FB15k-237 is carried out on the KB+Text, KB, or Text portions of the dataset.
[Overall]
It’s great that NTP was scaled up to handle larger datasets, however further analysis is needed. The argument that performance is given up for interpretability needs more discussion, and the effect of each addition to the system should be discussed as well.


Neural networks have been shown to be an effective tool for learning algorithms over graph-structured data. However, graph representation techniques-that convert graphs to real-valued vectors for use with neural networks-are still in their infancy. Recent works have proposed several approaches (e.g., graph convolutional networks), but these methods have difficulty scaling and generalizing to graphs with different sizes and shapes. We present GRAPH2SEQ, a new technique that represents vertices of graphs as infinite time-series. By not limiting the representation to a fixed dimension, GRAPH2SEQ scales naturally to graphs of arbitrary sizes and shapes. GRAPH2SEQ is also reversible, allowing full recovery of the graph structure from the sequences. By analyzing a formal computational model for graph representation, we show that an unbounded sequence is necessary for scalability. Our experimental results with GRAPH2SEQ show strong generalization and new state-of-the-art performance on a variety of graph combinatorial optimization problems.

This paper proposes a new representation learning model for graph optimization, Graph2Seq. The novelty of Graph2Seq lies in utilizing intermediate vector representation of vertices in the final representation. Theoretically, the authors show that an infinite sequence of such intermediate representations is much more powerful than existing models, which do not maintain intermediate representations. Experimentally, Graph2Seq results in greedy heuristics that generalize very well from small training graphs (e.g. 15 nodes) to large testing graphs (e.g. 3200 nodes).
Overall, the current version of the paper raises a number of crucial questions that I would like the authors to address before I make my decision.
First, some strengths of the paper:
- Theory: although I have not reviewed the proofs in details, the theorems are very interesting. If correct, the theorems provide a strong basis for Graph2Seq. In contrast, this aspect is missing from other work on ML for optimization.
- Experiments: the experiments are generally thorough and well-presented. The performance of Graph2Seq is remarkable, especially in terms of generalization to significantly larger graphs.
- Writing: the paper is very well-written and complex ideas are neatly articulated. I also liked the Appendix trying to interpret the trained model. Good job!
That being said, I have some serious concerns. Please clarify if I misunderstood anything and update the paper otherwise.
- Graph2Seq at test time: in section Testing, you explain how multiple solutions are output by G2S-RNN at intermediate "states" of the model, and the best w.r.t. the objective value is returned. If I understand all this correctly, you take the output of the T-th LSTM unit, run it through the Q-network, then select the next node (e.g. in a vertex cover solution). Then, the complexity should be O((E+V)*T_max*V), since the Graph2Seq operations are linear in the size of the graph O(E+V), a single G2S-RNN(i) takes O(V) times if you want to construct a cover of size O(V), and you repeat that process exactly T_max times, for each i between 1 and T_max. What's wrong in my understanding of G2S-RNN here? Or is your complexity incorrect?
- Local-Gather definition: in your definition of the Local-Gather model, do you assume that computations are performed for a single iteration, i.e. a single local step followed by a gather step? If so, then how is Graph2Seq infinity-local-gather? What does that even mean? I understand how some of the other GCNN-based models like Khalil et al.'s is 4-local-gather (assuming 4 embedding iterations of structure2vec), but how is Graph2Seq infinity-local-gather?
- Comparison to Structure2Vec: for fair comparison, why not apply Algorithm 2 to that method? Just run more embedding iterations up to T_max, and use the best among the solutions constructed between 1 and T_max.
Minor:
- Section 4: Vinyals et al. (2015) does not do any RL.


In this paper, we advocate the use of explicit memory for efficient exploration in reinforcement learning. This memory records structured trajectories that have led to interesting states in the past, and can be used by the agent to revisit those states more effectively. In high-dimensional decision making problems, where deep reinforcement learning is considered crucial, our approach provides a simple, transparent and effective way that can be naturally combined with complex, deep learning models. We show how such explicit memory may be used to enhance existing exploration algorithms such as intrinsically motivated ones and count-based ones, and demonstrate our method's advantages in various simulated environments.

In this paper, the authors propose an exploration strategy based on the explicit storage and recall of trajectories leading to novel states. A pool of such trajectories is managed over time, and a method is proposed so that the agent can learn how to follow a path corresponding to these trajectories so as to explore novel states. The idea is demonstrated in a set of room experiments, and quickly shown efficient in Montezuma's Revenge and PrivateEye Atari games.
Overall, the idea has some merits, but the empirical study is weak and the paper suffers from unsufficient writing effort (or more probably time).
What I like most in the paper is the split of exploration methods into 3 categories: adding some "intrinsic reward" bonuses to novel states (curiosity-driven exploration) , trying to reach various goals (goal-driven exploration) and using memory to reach again novel states (memory-driven exploration). Actually, this split may be debated. For instance, some frameworks based on goals have been labelled curiosity-driven, e.g. "Curiosity-Driven Exploration of Learned Disentangled Goal Spaces" (Laversanne-Finot, Péré and Oudeyer, CoRL 2018), but anyways I find it useful. That said, this aspect of the introduction is reiterated in the "Related Work" section in a quite redundant way, whereas both parts could have been better integrated. Furthermore, the related work section is hardly a draft, I'll come to that later.
The presentation of the method in Section 2 is rather clear and convincing. My only concern is about the assumption that the agent is always starting in the same state. This assumption may not hold in many settings, and the approach appears to be quite dependent on it. A discussion of how it could be extended to a less restricting assumption would be welcome.
The experimental section is weaker. A few concerns:
- I could not find much about the number of seeds, trials, the way to depict some variance, the statistical significance of the differences between results presented in Figure 1. The same is true about Figs. 2, 3 and 5.
- In Fig.2, the claim that the author's method learns better models is hardly supported by the left-hand-side plot, and significance is not assessed.
- I'm puzzled about the very low performance of baselines in the plots of Fig. 3. Could the author explain why these performances are null.
- The Atari games section helps figuring out that the framework is not too specific of the rooms environment, but the lack of analysis does not help making sure that this is just the explicit recall mechanism that is responsible for superior performance and why.
Another point about this section is that poor writing does not help understanding some points.
- to me, the first sentence of Section 3.2.2 does not make sense at all.
- in the caption of Fig. 4, "The second row is the heatmaps for states that the number of times being selected as a target state.": I don't get what it means, thus I don't understand what that row shows.
- Fig.5 comes with no caption
About the related work:
- The comparison to other methods using memory needs to be expanded. In particular, I would put HER-like mechanisms here rather than in 4.1, as "explicit recall" shares some importan ideas with "experience replay"
- Section 4.4 (HRL) is not useful as is.
Finally, in the conclusion, the claim that the method can be combined with "many sota exploration methods" is not supported, as the authors have only tried two and did not analyse the results in much details.
typos:
- p4:
we can easily counting
(include borders) => including
is provide => provided
are less less-visited states: quite inelegant
- p7:
In Montezuma's Revenge, Comparing => comparing
Where they encourage => remove "Where"
- p8:
recallcan => recall can
the problem of reach goals => reaching
it succesfully reproduce => reproduces
The last paragraph of Section 4.2 needs a careful rewriting, as long sentences with parenthese in the middle appear to be some draft version.
control(Pritzel => Missing space
Our method use memory => uses
Although ..., but => remove but
The path function can be seen as a form of skills => skill?
Besides, the "can be seen" needs to be further explained...
Appendix
Finally, we provided => provide
is around (math formula) => cannot you be more specific?


Prior researches suggest that attentional neural machine translation (NMT) is able to capture word alignment by attention, however, to our surprise, it almost fails for NMT models with multiple attentional layers except for those with a single layer. This paper introduce two methods to induce word alignment from general neural machine translation models. Experiments verify that both methods obtain much better word alignment than the method by attention. Furthermore, based on one of the proposed method, we design a criterion to divide target words into two categories (i.e. those mostly contributed from source "CFS" words and the other words mostly contributed from target "CFT" words), and analyze word alignment under these two categories in depth. We find that although NMT models are difficult to capture word alignment for CFT words but these words do not sacrifice translation quality significantly, which provides an explanation why NMT is more successful for translation yet worse for word alignment compared to statistical machine translation. We further demonstrate that word alignment errors for CFS words are responsible for translation errors in some extent by measuring the correlation between word alignment and translation for several NMT systems.

This paper sets out to build good bilingual word alignments from the information in an NMT system (both Transformer and RNN), where the goal is to match human-generated word-alignments as measured by AER. At least that’s how it starts. They contribute two aligners: one supervised aligner that uses NMT source and target representations as features and is trained on silver data generated by FastAlign, and one interpretability-based aligner that scores the affinity of a source-target word-pair by deleting the source word (replacing its embedding with a 0-vector) and measuring the impact on the probability of the target word. These are both shown to outperform directly extracting alignments from attention matrices by large margins. Despite the supervised aligner getting better AER, the authors proceed to quickly discard it as they dive deep on the interpretability approach, applying it also to target-target word pairs, and drawing somewhat interesting conclusions about two classes of target words: those that depend most of source context and those that depend most on target context.
Ultimately, this paper’s main contribution is its subtraction-based method for doing model interpretation. Its secondary contributions are the idea of evaluating this interpretation method empirically using human-aligned sentence pairs, and the idea of using the subtraction method on target-target pairs. The conclusion does a good job of emphasizing these contributions, but the abstract and front-matter do not. Much of the rest of the paper feels like a distraction. Overall, I believe the contributions listed above are valuable, novel and worth publishing. I can imagine using this paper’s techniques and ideas in my own research.
Specific concerns:
The front-matter mentions ‘multiple attention layers’. It would probably be a good idea to define this term carefully, as there are lots of things that could fit: multiple decoder layers with distinct attentions, multi-headed attention, etc.
In contrast to what is said in the introduction, GNMT as described in the Wu et al. 2016 paper only calculates attention once, based on the top encoder layer and the bottom decoder layer, so it doesn’t fit any definition of multiple attention layers.
Equation (1) and the following text use the variable L without defining it.
‘dominative’ -> ‘dominant’
Is there any way to generate a null alignment with Equation 3? That is, a target word that has no aligned source words? If not, that is a major advantage for FastAlign.
Similarly, what exactly are you evaluating when you evaluate FastAlign? Are you doing the standard tricks from the phrase-based days, and generating source->target and target->source models, and combining their alignments with grow-diag-final? If so, you could apply the same tricks to the NMT system to help even the playing field. Maybe this isn’t that important since the paper didn’t win up being about how to build the best possible word aligner from NMT (which I think is for the best).
I found Equations (7) and (8) to be confusing and distracting. I understand that you were inspired by Zintgraf’s method, but the subtraction-based method you landed on doesn’t seem to have much to do with the original Zintgraf et al. approach (and your method is much easier to the understand in the context of NMT than theirs). Likewise, I do not understand why you state, “we take the uniform distribution as P(x) regarding equation 8 for simplicity” - equation 9 completely redefines the LHS of equation 8, with no sum over x and no uniform distribution in sight.
The Data section of 4.1 never describes the NIST 2005 hand-aligned dataset.
The conclusions drawn at the end of 4.4 based on ‘translation recall’ are too strong. What we see is that the Transformer outperforms Moses by 2.8 onCFS, and by 3.7 on CFT. This hardly seems to support a claim that CFT words are the reason why Transformer yields better translation.
4.5 paragraph 1: there is no way to sample 12000 datasets without replacement from NIST 2005 and have the samples be the same size as NIST 2005. You must mean “with replacement”?


EFFICIENT MULTI-OBJECTIVE NEURAL ARCHITEC- TURE SEARCH VIA LAMARCKIAN EVOLUTION
Neural Architecture Search aims at automatically finding neural network architectures that are competitive with architectures designed by human experts. While recent approaches have achieved state-of-the-art predictive performance for, e.g., image recognition, they are problematic under resource constraints for two reasons:
(1) the neural architectures found are solely optimized for high predictive performance, without penalizing excessive resource consumption; (2) most architecture search methods require vast computational resources. We address the first shortcoming by proposing LEMONADE, an evolutionary algorithm for multi-objective architecture search that allows approximating the entire Pareto front of architectures under multiple objectives, such as predictive performance and number of parameters, in a single run of the method. We address the second shortcoming by proposing a Lamarckian inheritance mechanism for LEMONADE which generates child networks that are warm started with the predictive performance of their trained parents. This is accomplished by using (approximate) network morphism operators for generating children. The combination of these two contributions allows finding models that are on par or even outperform both hand-crafted as well as automatically-designed networks.

Summary:
The paper proposes LEMONADE, an evolutionary-based algorithm the searches for neural network architectures under multiple constraints. I will say it first that experiments in the paper only actually address to constraints, namely: log(#params) and (accuracy on CIFAR-10), and the method as currently presented does not show possible generalization beyond these two objectives, which is a weakness of the paper.
Anyhow, for the sake of summary, let’s say the method can actually address multiple, i.e. more than 2, objectives. The method works as follows.
1. Start with an architecture.
2. Apply network morphisms, i.e. operators that change a network’s architecture but also select some weights that do not strongly alter the function that the network represents. Which operations to apply are sampled according to log(#params). Details are in the paper.
3. From those sampled networks, the good ones are kept, and the evolutionary process is repeated.
The authors propose to use operations such as “Net2WiderNet” and “Net2DeeperNet” from Chen et al (2015), which enlarge the network but also choose a set of appropriate weights that do not alter the function represented by the network. The authors also propose operations that reduce the network’s size, whilst only slightly change the function that the network represented.
Experiments in the paper show that LEMONADE finds architecture that are Pareto-optimal compared to existing model. While this seems like a big claim, in the context of this paper, this claim means that the networks found by LEMONADE are not both slower and more wrong than existing networks, hand-crafted or automatically designed.
Strengths:
1. The method solves a real and important problem: efficiently search for neural networks that satisfy multiple properties.
2. Pareto optimality is a good indicator of whether a proposed algorithm works on this domain, and the experiments in the paper demonstrate that this is the case.
Weaknesses:
1. How would LEMONADE handle situations when there are more than one $f_{cheap}$, especially when different $f_{cheap}$ may have different value ranges? Eqn (8) and Eqn (9) does not seem to handle these cases.
2. Same question with $f_{exp}$. In the paper the only $f_{exp}$ refers to the networks’ accuracy on CIFAR-10. What happens if there are multiple objectives, such as (accuracy on CIFAR-10, accuracy on ImageNet) or (accuracy on CIFAR-10, accuracy on Flowers, image segmentation on VOC), etc.
I thus think the “Multi-Objective” is a bit overclaimed, and I strongly recommend that the authors adjust their claim to be more specific to what their method is doing.
3. What value of $\epsilon$ in Eqn (1) is used? Frankly, I think that if the authors train their newly generated children networks using some gradient descent methods (SGD, Momentum, Adam, etc.), then how can they guarantee the \epsilon-ANM condition? Can you clarify and/or change the presentation regarding to this part?


STABLE OPPONENT SHAPING IN DIFFERENTIABLE GAMES
A growing number of learning methods are actually differentiable games whose players optimise multiple, interdependent objectives in parallel -from GANs and intrinsic curiosity to multi-agent RL. Opponent shaping is a powerful approach to improve learning dynamics in these games, accounting for player influence on others' updates. Learning with Opponent-Learning Awareness (LOLA) is a recent algorithm that exploits this response and leads to cooperation in settings like the Iterated Prisoner's Dilemma. Although experimentally successful, we show that LOLA agents can exhibit 'arrogant' behaviour directly at odds with convergence. In fact, remarkably few algorithms have theoretical guarantees applying across all (n-player, non-convex) games. In this paper we present Stable Opponent Shaping (SOS), a new method that interpolates between LOLA and a stable variant named LookAhead. We prove that LookAhead converges locally to equilibria and avoids strict saddles in all differentiable games. SOS inherits these essential guarantees, while also shaping the learning of opponents and consistently either matching or outperforming LOLA experimentally.

This paper focuses on the problem of convergence in multi-objective optimisation with differentiable losses. This topic is timely and relevant, given the increasing amount of recent work on multi-objective architectures, e.g. GANs, adversarial learning, multi-agent reinforcement learning. The authors focus on stable fixed points (SFP), rather than Nash equilibria, as the solution concept in the entirety of their analysis. Casting the recently proposed LOLA gradient adjustment into a general matrix form, they diagnose an example where the shaping term in LOLA prevents convergence to SFP. They also find that discarding the shaping term leads to an earlier method (which they name ''LA'') with convergence guarantees in two-player two-action games. However, this also loses the opponent shaping ability of LOLA. To address these limitation, the authors propose SOS, which interpolates between LA and LOLA, and dynamically chooses the interpolation coefficient $p$ so that their adjusted gradient preserves LOLA's shaping ability only to the extent allowed by the constraint of moving in LA's direction. The main goal of the paper is to show that SOS converges locally to SFP, and to fixed points only, while avoiding strict saddles. Experiments on synthetic games show that SOS preserves the benefit of LOLA while avoiding its theoretically-predicted issues, and a more complex Gaussian mixture GAN experiment shows SOS is empirically competitive with other gradient adjustment methods.
The main conceptual novelty consists of the dynamic interpolation term to combine advantages of LOLA and LA while avoiding pitfalls of both. The major strength of the paper lies in the clear justification for this interpolation approach. The paper contains strong theoretical results for general differentiable games, and deserves the notice of the ICLR community if valid. However, I have major concerns with the proof of Theorem 2 (i.e. Theorem D.4 in the appendix), which affects the validity of Corollary 3 and Theorem 4.
In the proof of Theorem D.4:
1. How does the expression $u^T M^{-1}GMu$ have conformable dimensions, when $G \in R^{d \times d}$ while $u \in R^{d-1}$? Was any assumption made about the matrix $M = (I + \alpha H_d)^{1/2}$?
2. In the middle of page 14, a unit vector $u \in S^m$ is defined, but it is not clear what vector space is meant by $S^m$.
3. In the second-to-last line of page 14, a quantity $S$ is used but not defined clearly in any preceding part of the proof. Remark D.5 refers to $S$ as the symmetric part of $G$, and asserts that S is not positive definite. If the quantity $S$ used in the proof is the same non-PD quantity, then $S$ does not have a Cholesky factorisation. So how is Cholesky decomposition conducted at end of page 14?
4. In the first line of page 15, a quantity $A$ is used but not defined anywhere else in the entire paper.
5. From the subsequent line, it appears to be the anti-symmetric part of H. Is it correct assumption? If so, $H^2$ is not $(S^T - A^T)(S + A)$. If you replace it with correct form, whole quantity does not compute to be positive or becomes meaningless.
As Theorem 2 is the crux for all the theoretical advancement presented in the paper, clarifications on above correctness questions is very important for clear acceptance of this work.
While Definition 1 precisely defines differentiable games to have *twice* differentiable losses, why do the authors assume *thrice* differentiable losses at the start of Section 4?
In Section 2.2, the authors make a broad statement that ''Nash equilibria cannot be the right solution concept for multi-agent learning.'' They provide one example where Nash is undesirable (L^1 = L^2 = xy). However, since this example can be viewed as a fully cooperative game with joint loss L = 2xy, it does not support the broader statement that Nash is undesirable in all games. Because this statement directly motivates the authors to focus on stable fixed points, rather than Nash, as the solution concept in their subsequent analysis, it is very important to provide better justification for the claim.
Minor comments:
1. Under Proposition 1, the authors suddenly speak of ''...the policy being optimal''. Since the author's work pertains to general multi-objective settings, not solely multi-agent reinforcement learning, the word ''policy'' sounds strange in context.
2. The statement of Proposition B.1, and the concluding line of the derivation, left out a coefficient $\alpha$ that is present in Proposition 1 in the main text.
3. While the authors claim and prove independence of theoretical results from choice of a and b, are there any practical implications in terms of performance or convergence?


In this paper, we present a neural network based representation for addressing the open set recognition problem. In this representation instances from the same class are close to each other while instances from different classes are further apart, resulting in statistically significant improvement when compared to other approaches on three datasets from two different domains.

This paper deals with the open set classification problem, where in addition to the known classes, the method should also be able to recognize the unknown class. The main idea is based on two parts: learning a discriminative representation, and a threshold based detection rule. To learn the embedding, the authors propose to minimize the inner class distance (between each instance to its center) and enlarge the distance between centers. The outlier score of an instance is computed as the minimum distance between known class prototypes. Experiments on various datasets show the ability of the learned method.
I'm not completely sure whether the whole approach is novel or not in the open set recognition domain, but both parts are not novel enough. Pulling similar instances together and pushing dissimilar ones away is the main idea in embedding learning. The ii-loss is similar to the triplet-center loss in the paper "He et al. Triplet-Center Loss for Multi-View 3D Object Retrieval. CVPR18".
Although in the experiments the proposed method achieves good results in most cases, the reviewer suggests the authors comparing with more baselines to make the work solid.
1. Comparing with other embedding learning methods with the same outlier detection score.
The authors should prove that the proposed embedding is important enough in the open set case. For example, using the center loss (Wen et al. A discriminative feature learning approach for deep face recognition. ECCV16), triplet-center loss, triplet loss (computing class centers after embedding).
2. Discuss more on the outlier score part.
How to differentiate the known class outlier and new class? Will the problem be more difficult when the unknown class contains more heterogeneous classes? The authors can also apply existing open set recognition rule on the learned embedding.
Some detailed questions:
1. What's the difference between "the network weights are first updated to minimize on ii-loss and then in a separate step updated to minimize cross entropy loss" and optimize both loss terms simultaneously?
2. "We assume that a certain percent of the training set to be noise/outliers", how to determine the concrete value? Is 1% the helpful one for all cases?
3. Since there is not optimize over the unknown classes in training, could the reason for "the unknown class instances fully occupy the open space between the known classes" is the unknown classes are randomly sampled from the whole class set? For example, if classes about animals are known classes and classes about scene compose the unknown class, will the unknown class also occupy the whole space in this case?
4. What is the motivation of making "the unknown class instances fully occupy the open space between the known classes"?


SELF-TUNING NETWORKS: BILEVEL OPTIMIZATION OF HYPERPARAMETERS US- ING STRUCTURED BEST-RESPONSE FUNCTIONS
Hyperparameter optimization can be formulated as a bilevel optimization problem, where the optimal parameters on the training set depend on the hyperparameters. We aim to adapt regularization hyperparameters for neural networks by fitting compact approximations to the best-response function, which maps hyperparameters to optimal weights and biases. We show how to construct scalable best-response approximations for neural networks by modeling the best-response as a single network whose hidden units are gated conditionally on the regularizer. We justify this approximation by showing the exact best-response for a shallow linear network with L 2 -regularized Jacobian can be represented by a similar gating mechanism. We fit this model using a gradient-based hyperparameter optimization algorithm which alternates between approximating the best-response around the current hyperparameters and optimizing the hyperparameters using the approximate best-response function. Unlike other gradient-based approaches, we do not require differentiating the training loss with respect to the hyperparameters, allowing us to tune discrete hyperparameters, data augmentation hyperparameters, and dropout probabilities. Because the hyperparameters are adapted online, our approach discovers hyperparameter schedules that can outperform fixed hyperparameter values. Empirically, our approach outperforms competing hyperparameter optimization methods on large-scale deep learning problems. We call our networks, which update their own hyperparameters online during training, SelfTuning Networks (STNs).

========\\
Summary\\
========\\
The paper deals with hyper-parameter optimization of neural networks. The authors formulate the problem as a bilevel optimization problem: minimizing the validation loss over the hyperparameters, subject to the parameters being at the minimum of the training loss. The authors propose an approximation of the so-called best-response function, that maps the hyperparameters to the corresponding optimal parameters (w.r.t the minimization of the training loss), allowing a formulate as a single-level optimization problem and the use gradient descent algorithm. The proposed
approximation is based on shifting and scaling the weights and biases of the network. There are no guarantee on its quality except in some very simple cases. The approach assumes a distribution on the hyperparameters, governed by a parameter, which is adapted during the course of the training to achieve a compromise between the flexibility of the best-response function and the quality of its local approximation around the current hyperparameters. The authors show
that their approach beats grid-search, random search and Bayesian optimization on the CIFAR-10 and PTB datasets. They point out that the dynamic update of the hyperparameters during the training allows to reach a better performance than any fixed hyperparameter. \\
======================\\
Comments and questions\\
======================\\
Can cross-validation be adapted to this approach? \\
Can this be used to optimize the learning rate? Which is of course a crucial hyperparameter and that needs an update schedule during the training. \\
Section 3.2:\\
"If the entries are too large, then θ̂ φ will not be flexible enough to capture the best- response over the sampled neighborhood. However, its entries must remain sufficiently large so that θ̂ φ captures the local shape around the current hyperparameter values." Not clear why -- more explanations would be helpful. \\
"minimizing the first term eventually moves all probability mass towards an optimum λ∗ ,resulting in σ = 0". I can't see how minimizing the first term w.r.t \phi (as in section "2.2.Local approximation") would alter \sigma. \\
"τ must be set carefully to ensure...". The authors still do not explain how to set \tau. \\
Section 3.3: \\
If the hyperparameter is discrete and falls in Case 2, then REINFORCE gradient estimator is used. What about the quality of this gradient? \\
Section 5, paragraph Gradient-Based HO: "differentiating gradient descent" needs reformulation -- an algorithm cannot be differentiated. \\
Pros \\
- The paper is pretty clear \\
- Generalizes a previous idea and makes it handle discrete hyperparameters and scale better. \\
- I like the idea of hyperparameters changing dynamically during the training which allows to explore a much larger space than one value \\
- Although limited, the experimental results are convincing \\
Cons \\
- The method itself depends on some parameters and it is not clear how to choose them. Therefore it might be tricky to make it work in practice. I feel like there is a lot of literature around HO but very often people still use the very simple grid/random search, because the alternative methods are often quite complex to implement and make really work. So the fact that the method depends on "crucial" parameters but that are not transparently managed may be a big drawback to its applicability. \\
- No theoretical guarantee on the quality of the used approximation for neural networks \\
- Does not handle the learning rate which is a crucial hyperparameter (but maybe it could) \\


Representation learning becomes especially important for complex systems with multimodal data sources such as cameras or sensors. Recent advances in reinforcement learning and optimal control make it possible to design control algorithms on these latent representations, but the field still lacks a large-scale standard dataset for unified comparison. In this work, we present a large-scale dataset and evaluation framework for representation learning for the complex task of landing an airplane. We implement and compare several approaches to representation learning on this dataset in terms of the quality of simple supervised learning tasks and disentanglement scores. The resulting representations can be used for further tasks such as anomaly detection, optimal control, model-based reinforcement learning, and other applications.

Overview and contributions:
The authors present a newly collected dataset and evaluation framework for learning representations for landing an airplane. The dataset is collected from the X-Plane simulation environment and consists of 8011 landings, each landing consists of time series data from 1090 sensors. Their evaluation metric is a combination of disentanglement score, regression tasks, and failure classification. The authors test a combination of baseline models from basic autoencoders to dynamic actions-aware encoders. The writing is generally clear but I have doubts about the proposed evaluation metrics, experiments, and significance of the dataset (details below).
Strengths:
1. The task seems to be novel and complex. The authors have done a good job of collecting the dataset and ensuring that the data is clean and comprehensive.
2. The authors have performed a comprehensive job of evaluating the combinations of baseline models for their proposed task.
Weaknesses:
1. Table 4 on evaluation results, while comprehensive, lacks some explanation. The issue with MSE is that it is hard to interpret what these values mean. Specifically, how difficult is this task? How well can a human perform on this task? How well are the baselines doing relative to human-level performance, and is there room for improvement? The answers to these questions are important towards whether this new dataset will be a strong benchmark for representation learning.
2. There is less novelty in terms of the models presented for evaluation since they are composed of existing models. What are some state-of-the-art models for similar tasks, and do they constitute fair comparison?
Questions to authors:
1. Refer to weakness points 1 and 2.
2. What biases do you think might exist in the dataset during the collection process? How might these biases affect what the models learn, and how can they be mitigated?
3. How do you ensure that all sensors are active at all times and that all sensors provide useful information for predicting the label? Are there cases where the multisensor data is noisy in certain modalities or missing in other modalities? If so, what are some models that can remain robustness to noisy or missing modalities?
4. Why do you think disentangled representations will help? Sure, they have been generally shown to help learn more interpretable representations, and help in flexible generation from disentangled factors. But in terms of discriminative or generative performance on your newly proposed dataset, does learning disentangled representations help? What are some models that can learn effectively learn such disentangled representations?
Presentation improvements, typos, edits, style, missing references:
Section 3, line 7, 'with with a frequency' -> 'with'
I would suggest referring to some recent work on multimodal temporal fusion, such as "Memory Fusion Network for Multi-view Sequential Learning. Amir Zadeh, Paul Pu Liang, Navonil Mazumder, Soujanya Poria, Erik Cambria, Louis-Philippe Morency, AAAI 2018"


RELAXED QUANTIZATION FOR DISCRETIZED NEURAL NETWORKS
Neural network quantization has become an important research area due to its great impact on deployment of large models on resource constrained devices. In order to train networks that can be effectively discretized without loss of performance, we introduce a differentiable quantization procedure. Differentiability can be achieved by transforming continuous distributions over the weights and activations of the network to categorical distributions over the quantization grid. These are subsequently relaxed to continuous surrogates that can allow for efficient gradient-based optimization. We further show that stochastic rounding can be seen as a special case of the proposed approach and that under this formulation the quantization grid itself can also be optimized with gradient descent. We experimentally validate the performance of our method on MNIST, CIFAR 10 and Imagenet classification.

Summary
=======
This paper introduces a method for learning neural networks with quantized weights and activations. The main idea is to stochastically – rather than deterministically – quantize values, and to replace the resulting categorical distribution over quantized values with a continuous relaxation (the "concrete distribution" or "Gumbel-Softax distribution"; Maddison et al., 2016; Jang et al., 2016). Good empirical performance is demonstrated for LeNet-5 applied to MNIST, VGG applied to CIFAR-10, and MobileNet and ResNet-18 applied to ImageNet.
Review
======
Relevance:
Training non-differentiable neural networks is a challenging and important problem for several applications and a frequent topic at ICLR.
Novelty:
Conceptually, the proposed approach seems like a straight-forward application/extension of existing methods, but I'm unaware of any paper which uses the concrete distribution for the express purpose of improved efficiency as in this paper. There is a thorough discussion of related work, although I was missing Williams (1992), who used stochastic rounding before Gupta et al. (2015), and Soudry et al. (2014), who introduced a Bayesian approach to deal with discrete weights and activations.
Results:
The empirical work is thorough, achieving state-of-the-art results in several classification benchmarks. It would be interesting to see how well these methods perform in other tasks (e.g., compression or even regression), even though the literature on quantization seems to focus on classification.
Clarity:
The paper is well written and clear.