Published as a conference paper at ICLR 2018 GENERATING NATURAL ADVERSARIAL EXAMPLES
Due to their complex nature, it is hard to characterize the ways in which machine learning models can misbehave or be exploited when deployed. Recent work on adversarial examples, i.e. inputs with minor perturbations that result in substantially different model predictions, is helpful in evaluating the robustness of these models by exposing the adversarial scenarios where they fail. However, these malicious perturbations are often unnatural, not semantically meaningful, and not applicable to complicated domains such as language. In this paper, we propose a framework to generate natural and legible adversarial examples that lie on the data manifold, by searching in semantic space of dense and continuous data representation, utilizing the recent advances in generative adversarial networks. We present generated adversaries to demonstrate the potential of the proposed approach for black-box classifiers for a wide range of applications such as image classification, textual entailment, and machine translation. We include experiments to show that the generated adversaries are natural, legible to humans, and useful in evaluating and analyzing black-box classifiers.

Summary:
 A method for creation of semantical adversary examples in suggested. The ‘semantic’ property is measured by building a latent space with mapping from this space to the observable (generator) and back (inverter). The generator is trained with a WGAN optimization. Semantic adversarials examples are them searched for by inverting an example to its sematic encoding and running local search around it in that space. The method is tested for generation of images on MNist and part of LSUM data and for creation of text examples which are adversarial in some sense to inference and translation sentences. It is shown that the distance between adversarial example and the original example in the latent space is proportional to the accuracy of the classifier inspected.
Page 3: It seems that the search algorithm has a additional parameter: r_0, the size of the area in which search is initiated. This should be explicitly said and the parameter value should be stated.
Page 4:
-	the implementation details of the generator, critic and invertor networks are not given in enough details, and instead the reader is referred to other papers. This makes this paper non-clear as a stand alone document, and is a problem for a paper which is mostly based on experiments and their results: the main networks used are not described.
-	the visual examples are interesting, but it seems that they are able to find good natural adversary examples only for a weak classifier. In the MNist case, the examples for thr random forest are nautral and surprising, but those for the LE-Net are often not: they often look as if they indeed belong to the other class (the one pointed by the classifier). In the churce-vs. tower case, a  relatively weak MLP classifier was used. It would be more instructive to see the results for a better, convolutional classifier.
Page 5:
-	the description of the various networks used for text generation is insufficient for understanding:
o	The AREA is described in two sentences. It is not clear how this module is built, was loss was it used to optimize in the first place, and what elements of it are re0used for the current task
o	 ‘inverter’ here is used in a sense which is different than in previous sections of the paper: earlier it denoted the mapping from output (images) to the underlying latent space. Here it denote  a mapping between two latent spaces.
o	 It is not clear what the ‘four-layers strided CNN’ is: its structure, its role in the system. How is it optimized?
o	In general: a block diagram showing the relation between all the system’s components may be useful, plus the details about the structure and optimization of the various modules. It seems that the system here contains 5 modules instead of the three used before (critic, generator and inverter), but this is not clear enough. Also which modules are pre-trained, which are optimized together,a nd which are optimized separately is not clear.
o	SNLI data should be described: content, size, the task it is used for
Pro:
-	A novel idea of producing natural adversary examples with a GAN
-	The generated examples are in some cases useful for interpretation and network understanding
-	The method enables creation of adversarial examples for block box classifiers
Cons
-	The idea implementation is basic. Specifically search algorithm presented is quite simplistic, and no variations other than plain local search were developed and tested
-	The generated adversarial examples created for successful complex classifiers are often not impressive and useful (they are either not semantical, or semantical but correctly classified by the classifier). Hence It is not clear if the latent space used by the method enables finding of interesting adversarial examples for accurate classifiers.


EVIDENCE AGGREGATION FOR ANSWER RE-RANKING IN OPEN-DOMAIN QUESTION ANSWERING
A popular recent approach to answering open-domain questions is to first search for question-related passages and then apply reading comprehension models to extract answers. Existing methods usually extract answers from single passages independently. But some questions require a combination of evidence from across different sources to answer correctly. In this paper, we propose two models which make use of multiple passages to generate their answers. Both use an answerreranking approach which reorders the answer candidates generated by an existing state-of-the-art QA model. We propose two methods, namely, strengthbased re-ranking and coverage-based re-ranking, to make use of the aggregated evidence from different passages to better determine the answer.

Traditional open-domain QA systems typically have two steps: passage retrieval and aggregating answers extracted from the retrieved passages.  This paper essentially follows the same paradigm, but leverages the state-of-the-art reading comprehension models for answer extraction, and develops the neural network models for the aggregating component.  Although the idea seems incremental, the experimental results do seem solid.  The paper is generally easy to follow, but in several places the presentation can be further improved.
Detailed comments/questions:
  1. In Sec. 2.2, the justification for adding H^{aq} and \bar{H}^{aq} is to downweigh the impact of stop word matching.  I feel this is a somewhat indirect and less effective design, if avoiding stop words is really the reason.  A standard preprocessing step may be better.
  2. In Sec. 2.3, it seems that the final score is just the sum of three individual normalized scores. It's not truly a "weighted" combination, where the weights are typically assumed to be tuned.
  3. Figure 3: Connecting the dots in the two subfigures on the right does not make sense.  Bar charts should be used instead.
  4. The end of Sec. 4.2: I feel it's a bad example, as the passage does not really support the answer. The fact that "Sesame Street" got picked is probably just because it's more famous.
  5. It'd be interesting to see how traditional IR answer aggregation methods perform, such as simple classifiers or heuristics by word matching (or weighted by TFIDF) and counting.  This will demonstrates the true advantages of leveraging modern NN models.
Pros:
  1. Updating a traditional open-domain QA approach with neural models
  2. Experiments demonstrate solid positive results
Cons:
  1. The idea seems incremental
  2. Presentation could be improved


Grasping an object and precisely stacking it on another is a difficult task for traditional robotic control or hand-engineered approaches. Here we examine the problem in simulation and provide techniques aimed at solving it via deep reinforcement learning. We introduce two straightforward extensions to the Deep Deterministic Policy Gradient algorithm (DDPG), which make it significantly more data-efficient and scalable. Our results show that by making extensive use of offpolicy data and replay, it is possible to find high-performance control policies that successfully achieve precise stacking behaviour in > 95% of 1000 randomly initialized configurations. Further, our results on data efficiency hint that it may soon be feasible to train successful stacking policies by collecting interactions on real robots.

The title is too generic and even a bit misleading. Dexterous manipulation usually refers to more complex skills, like in-hand manipulation or using the fingers to turn an object, and not simple pick and place tasks. Reinforcement learning methods are generally aiming to be data-efficient, and the method does not seem designed specifically for dexterous manipulation (which is actually a positive point, as it is more general).
The paper presents two extensions for DDPG: multiple network updates per physical interactions, and asynchronous updates from multiple robots. As the authors themselves state, these contributions are fairly straightforward, and the contributions are largely based on prior works. The  authors do evaluate the methods with different parameter settings to see the effects on learning performance.
The simulation environment is fairly basic and seems unrealistic. The hand always starts close to the blocks, which are close together, so the inverse kinematics will be close to linear. The blocks are always oriented in the same direction and they can connect easily with no need to squeeze or wiggle them together. The task seems more difficult from the description in the paper, and the authors should describe the environment in more detail.
Does the robot learn to flip the blocks over such that they can be stacked? The videos show the
blocks turning over accidentally, but then the robot seems to give up. Having the robot learn to turn the blocks  would make for a more challenging task and a better policy.
The paper’s third contribution is a recipe for constructing shaped reward functions for composite tasks. The method relies on a predefined task structure (reach-grasp-stack) and is very similar to reward shaping already used in many other reinforcement learning for manipulation papers. A comparison of different methods for defining the rewards and a more formal description of the reward generation procedure would improve the impact of this section.  The authors should also consider using tasks with longer sequences of actions, e.g., stacking four blocks.
The fourth and final listed contribution is learning from demonstrated states. Providing the robot with prior knowledge and easier partial tasks will result in faster learning. This result is not surprising. It is not clear though how applicable this approach is for a real robot system. It effectively assumes that the robot can grasp the block and pick it up, such that it can learn the stacking part, while simultaneously still learning how to grasp the block and pick it up. For testing the real robot applicability, the authors should try having the robot learn the task without simulation resets.
What are the actual benefits of using deep learning in this scenario? The authors mention skill representations, such as dynamic motor primitives, which employ significantly more prior knowledge than a deep network. However, as demonstrations of the task are provided, the task is divided into steps, the locations of the objects and finger tips are given, a suitable reward function is provided, and the generalization is only over the object positions, why not train a set of DMPs and optimize them with some additional reinforcement learning? The authors should consider adding a Cartesian DMP policy as a benchmark, as well as discussing the benefits of the proposed approach given the prior knowledge.


We exploit a recently derived inversion scheme for arbitrary deep neural networks to develop a new semi-supervised learning framework that applies to a wide range of systems and problems. The approach reaches current state-of-the-art methods on MNIST and provides reasonable performances on SVHN and CIFAR10. Through the introduced method, residual networks are for the first time applied to semi-supervised tasks. Experiments with one-dimensional signals highlight the generality of the method. Importantly, our approach is simple, efficient, and requires no change in the deep network architecture.

In summary, the paper is based on a recent work Balestriero & Baraniuk 2017 to do semi-supervised learning. In Balestriero & Baraniuk, it is shown that any DNN can be approximated via a linear spline and hence can be inverted to produce the "reconstruction" of the input, which can be naturally used to do unsupervised or semi-supervised learning. This paper proposes to use automatic differentiation to compute the inverse function efficiently. The idea seems interesting. However, I think there are several main drawbacks, detailed as follows:
1. The paper lacks a coherent and complete review of the semi-supervised deep learning. Herewith some important missing papers, which are the previous or current state-of-the-art.
[1] Laine S, Aila T. Temporal Ensembling for Semi-Supervised Learning[J]. arXiv preprint arXiv:1610.02242, ICLR 2016.
[2] Li C, Xu K, Zhu J, et al. Triple Generative Adversarial Nets[J]. arXiv preprint arXiv:1703.02291, NIPS 2017.
[3] Dai Z, Yang Z, Yang F, et al. Good Semi-supervised Learning that Requires a Bad GAN[J]. arXiv preprint arXiv:1705.09783, NIPS 2017.
Besides, some papers should be mentioned in the related work such as Kingma et. al. 2014. I'm not an expert of the network inversion and not sure whether the related work of this part is sufficient or not.
2. The motivation is not sufficient and not well supported.
As stated in the introduction, the authors think there are several drawbacks of existing methods including "training instability, lack of topology generalization and computational complexity." Based on my knowledge, there are two main families of semi-supervised deep learning methods, classified by depending on deep generative models or not.  The generative approaches based on VAEs and GANs are time consuming, but according to my experience, the training of VAE-based methods are stable and the topology generalization ability of such methods are good. Besides, the feed-forward approaches including [1] mentioned above are efficient and not too sensitive with respect to the network architectures.  Overall, I think the drawbacks mentioned in the paper are not common in existing methods and I do not see clear benefits of the proposed method. Again, I strongly suggest the authors to provide a complete review of the literature.
Further, please explicitly support your claim via experiments. For instance, the proposed method should be compared  with the discriminative approaches including VAT and [1] in terms of the training efficiency. It's not fair to say GAN-based methods require more training time because these methods can do generation and style-class disentanglement while the proposed method cannot.
3. The experimental results are not so convincing.
First, please systematically compare your methods with existing methods on the widely adopted benchmarks including MNIST with 20, 100 labels and SVHN with 500, 1000 labels and CIFAR10 with 4000 labels. It is not safe to say the proposed method is the state-of-the-art by only showing the results in one setting.
Second, please report the results of the proposed method with comparable architectures used in previous methods and state clearly the number of parameters in each model. Resnet is powerful but previous methods did not use that.
Last, show the sensitive results of the proposed method by tuning alpha and beta. For instance, please show what is the actual contribution of the proposed reconstruction loss to the classification accuracy with the other losses existing or not?
I think the quality of the paper should be further improved by addressing these problems and currently it should be rejected.


ACTIVE NEURAL LOCALIZATION
Localization is the problem of estimating the location of an autonomous agent from an observation and a map of the environment. Traditional methods of localization, which filter the belief based on the observations, are sub-optimal in the number of steps required, as they do not decide the actions taken by the agent. We propose "Active Neural Localizer", a fully differentiable neural network that learns to localize accurately and efficiently. The proposed model incorporates ideas of traditional filtering-based localization methods, by using a structured belief of the state with multiplicative interactions to propagate belief, and combines it with a policy model to localize accurately while minimizing the number of steps required for localization. Active Neural Localizer is trained end-to-end with reinforcement learning. We use a variety of simulation environments for our experiments which include random 2D mazes, random mazes in the Doom game engine and a photo-realistic environment in the Unreal game engine. The results on the 2D environments show the effectiveness of the learned policy in an idealistic setting while results on the 3D environments demonstrate the model's capability of learning the policy and perceptual model jointly from raw-pixel based RGB observations. We also show that a model trained on random textures in the Doom environment generalizes well to a photo-realistic office space environment in the Unreal engine.

The paper describes a neural network-based approach to active localization based upon RGB images. The framework employs Bayesian filtering to maintain an estimate of the agent's pose using a convolutional network model for the measurement (perception) function. A convolutional network models the policy that governs the action of the agent. The architecture is trained in an end-to-end manner via reinforcement learning. The architecture is evaluated in 2D and 3D simulated environments of varying complexity and compared favorably to traditional (structured) approaches to passive and active localization.
As the paper correctly points out, there is large body of work on map-based localization, but relatively little attention has been paid to decision theoretic formulations to localization, whereby the agent's actions are chosen in order to improve localization accuracy. More recent work instead focuses on the higher level objective of navigation, whereby any effort act in an effort to improve localization are secondary to the navigation objective. The idea of incorporating learned representations with a structured Bayesian filtering approach is interesting, but it's utility could be better motivated. What are the practical benefits to learning the measurement and policy model beyond (i) the temptation to apply neural networks to this problem and (ii) the ability to learn these in an end-to-end fashion? That's not to say that there aren't benefits, but rather that they aren't clearly demonstrated here. Further, the paper seems to assume (as noted below) that there is no measurement uncertainty and, with the exception of the 3D evaluations, no process noise.
The evaluation demonstrates that the proposed method yields estimates that are more accurate according to the proposed metric than the baseline methods, with a significant reduction in computational cost. However, the environments considered are rather small by today's standards and the baseline methods almost 20 years old. Further, the evaluation makes a number of simplifying assumptions, the largest being that the measurements are not subject to noise (the only noise that is present is in the motion for the 3D experiments). This assumption is clearly not valid in practice. Further, it is not clear from the evaluation whether the resulting distribution that is maintained is consistent (e.g., are the estimates over-/under-confident?). This has important implications if the system were to actually be used on a physical system. Further, while the computational requirements at test time are significantly lower than the baselines, the time required for training is likely very large. While this is less of an issue in simulation, it is important for physical deployments. Ideally, the paper would demonstrate performance when transferring a policy trained in simulation to a physical environment (e.g., using diversification, which has proven effective at simulation-to-real transfer).
Comments/Questions:
* The nature of the observation space is not clear.
* Recent related work has focused on learning neural policies for navigation, and any localization-specific actions are secondary to the objective of reaching the goal. It would be interesting to discuss how one would balance the advantages of choosing actions that improve localization with those in the context of a higher-level task (or at least including a cost on actions as with the baseline method of Fox et al.).
* The evaluation that assigns different textures to each wall is unrealistic.
* It is not clear why the space over which the belief is maintained flips as the robot turns and shifts as it moves.
* The 3D evaluation states that a 360 deg view is available. What happens when the agent can only see in one (forward) direction?
* AML includes a cost term in the objective. Did the author(s) experiment with setting this cost to zero?
* The 3D environments rely upon a particular belief size (70 x 70) being suitable for all environments. What would happen if the test environment was larger than those encountered in training?
* The comment that the PoseNet and VidLoc methods "lack a strainghtforward method to utilize past map data to do localization in a new environment" is unclear.
* The environments that are considered are quite small compared to the domains currently considered for
* Minor: It might be better to move Section 3 into Section 4 after introducing notation (to avoid redundancy).
* The paper should be proofread for grammatical errors (e.g., "bayesian" --> "Bayesian", "gaussian" --> "Gaussian")
UPDATES FOLLOWING AUTHORS' RESPONSE
(Apologies if this is a duplicate. I added a comment in light of the authors' response, but don't see it and so I am updating my review for completeness).
I appreciate the authors's response to the initial reviews and thank them for addressing several of my comments.
RE: Consistency
My concerns regarding consistency remain. For principled ways of evaluating the consistency of an estimator, see Bar-Shalom "Estimation with Applications to Tracking and Navigation".
RE: Measurement/Process Noise
The fact that the method assumes perfect measurements and, with the exception of the 3D experiments, no process noise is concerning as neither assumptions are valid for physical systems. Indeed, it is this noise in particular that makes localization (and its variants) challenging.
RE: Motivation
The response didn't address my comments about the lack motivation for the proposed method. Is it largely the temptation of applying an end-to-end neural method to a new problem? The paper should be updated to make the advantages over traditional approaches to active localization.


Log-linear models models are widely used in machine learning, and in particular are ubiquitous in deep learning architectures in the form of the softmax. While exact inference and learning of these requires linear time, it can be done approximately in sub-linear time with strong concentrations guarantees. In this work, we present LSH Softmax, a method to perform sub-linear learning and inference of the softmax layer in the deep learning setting. Our method relies on the popular Locality-Sensitive Hashing to build a well-concentrated gradient estimator, using nearest neighbors and uniform samples. We also present an inference scheme in sub-linear time for LSH Softmax using the Gumbel distribution. On language modeling, we show that Recurrent Neural Networks trained with LSH Softmax perform on-par with computing the exact softmax while requiring sub-linear computations.

In this paper, the authors propose a new approximation of the softmax, based on approximate nearest neighbors search and sampling.
More precisely, they propose to approximate to partition function (which is the bottleneck to compute the softmax and its gradient), by using:
- the top-k classes (retrieved using LSH) ;
- uniform samples (to account for the tail of the distribution).
They describe how this technique can be used for learning, by performing sparse updates for the gradient (corresponding to the elements used to compute the partition function), and re-hashing the updated element of the softmax layers.
In section 5, they show how this method can be implemented on GPU, using standard operations available in neural networks framework such as TensorFlow or PyTorch.
Finally, they compare their approach to importance sampling and negative sampling, using language modeling as a benchmark.
They use 3 standards datasets to perform the evaluations: penn treebank, text8 and wikitext-2.
Pros:
 - well written and easy to read paper
 - interesting theoretical guarantees of the approximation
Cons:
 - a bit incremental
 - weak empirical evaluations
 - no support for the claim of efficient GPU implementation
== Incremental ==
While the theoretical justification of the methods are interesting, these are not a contribution of the paper (but of previous work by Mussmann et al.).
In fact, the main contribution of this paper is to show how to apply the technique of Mussmann et al. in the setup of neural network.
The main difference with Mussmann et al. is the necessity of re-hashing the updated elements of the softmax at each step.
Other previous works have also proposed to use LSH to speed up computations in neural network, but are not discussed in the paper (see list of references).
== Weak evaluations ==
I believe that the empirical evaluation of section 6 are a bit weak.
First, there is a large gap between the perplexity obtained using the proposed method and the exact softmax (e.g. 97 v.s. 83 on ptb, 115 v.s. 95 on wikitext-2).
Thus, I do not believe that the experiments support the claim that the proposed method "perform on-par with computing the exact softmax".
Moreover, these numbers are pretty far from what other papers have reported on these datasets with similar models (I am wondering if the gap would be even larger with SOTA models).
Second, the authors do not report any runtime numbers for their method and the baselines on GPUs.
I believe that it would be more fair to plot the learning curves (Fig. 1) using the runtime instead of the number of epochs.
== Efficient implementation ==
In section 5, the authors claims that their approach can be efficiently implemented on GPUs.
However, several of the operations used by their approach are inefficient, especially when using mini-batches.
The authors state that only step 2 is inefficient, but I also believe that step 3 is (compared to sampling approaches).
Indeed, for their method, each example of a mini-batch uses a different set of elements to approximate the partition function (while for other sampling methods, the same set is used for the whole batch).
Thus a matrix-matrix multiplication is replaced by n matrix-vector multiplication (n is the batch size).
While these can be performed in parallel, it is much less efficient than a matrix-matrix multiplication.
Finally, the only runtime numbers provided by the authors comparing their approach to sampling is for a CPU implementation with a batch of size 1.
This setting is super favorable to their approach, but a bit unrealistic for most practical settings.
== Missing references ==
Scalable and Sustainable Deep Learning via Randomized Hashing
Ryan Spring, Anshumali Shrivastava
A New Unbiased and Efficient Class of LSH-Based Samplers and Estimators for Partition Function Computation in Log-Linear Models
Ryan Spring, Anshumali Shrivastava
Deep networks with large output spaces
Sudheendra Vijayanarasimhan, Jonathon Shlens, Rajat Monga & Jay Yagnik


It is becoming increasingly clear that many machine learning classifiers are vulnerable to adversarial examples. In attempting to explain the origin of adversarial examples, previous studies have typically focused on the fact that neural networks operate on high dimensional data, they overfit, or they are too linear. Here we show that distributions of logit differences have a universal functional form. This functional form is independent of architecture, dataset, and training protocol; nor does it change during training. This leads to adversarial error having a universal scaling, as a power-law, with respect to the size of the adversarial perturbation. We show that this universality holds for a broad range of datasets (MNIST, CIFAR10, ImageNet, and random data), models (including state-of-the-art deep networks, linear models, adversarially trained networks, and networks trained on randomly shuffled labels), and attacks (FGSM, step l.l., PGD). Motivated by these results, we study the effects of reducing prediction entropy on adversarial robustness. Finally, we study the effect of network architectures on adversarial sensitivity. To do this, we use neural architecture search with reinforcement learning to find adversarially robust architectures on CIFAR10. Our resulting architecture is more robust to white and black box attacks compared to previous attempts.

This work presents an empirical study aiming at improving the understanding of the vulnerability of neural networks to adversarial examples. Paraphrasing the authors, the main observation of the study is that the vulnerability is due to an inherent uncertainty that neural networks have about their predictions ( the difference between the logits). This is consistent across architectures, datasets. Further, the authors note that "the universality is not a result of the specific content of these datasets nor the ability of the model to generalize."
While this empirical study contains valuable information, its above conclusions are factually wrong. It can be theoretically proven at least using two routes. They are also in contradiction with other empirical observations consistent across several previous studies.
1-Constructive counter-argument: Consider a neural network that always outputs a constant prediction. It (1) is by definition independent of any dataset (2) generalizes perfectly (3) has zero adversarial error, hence contradicting the central statement of the paper.
2- Analysis-based counter-argument: Consider a neural network with one hidden layer and two classes. It is easy to show that the difference between the scores (logits) of the two classes is linear in the operator norm of the hidden weight matrix and linear in the L2-norm of the last weight vector. Therefore, the robustness of the model indeed depends on its capability to generalize because the latter is essentially governed by the geometric margin of the linear separator and the spectral norm of the weight matrix (see [1,2,3]). QED.
3- Further, the lack of calibration of neural networks and its causes are well known. Among other things, it is due to the use of building blocks (such as batch-norm [4]), regularization (e.g., weight decay) or the use of softmax+cross-entropy during training. While this is convenient for optimization reasons, it indeed hurts the calibration. The authors should try to train a neural network with a large margin criteria and see if the same phenomenon still holds when they measure the geometric margin. Another alternative is to use a temperature with the softmax[4]. Therefore, the observations of the empirical study cannot be generalized to neural networks and should be explicitly restricted to neural networks using softmax with cross-entropy as criteria.
I believe the conclusions of this study are misleading, hence I recommend to reject the paper.
[1] Spectrally Normalized Margin-bounds Margin bounds for neural networks (Bartlett et al., 2017)
[2] Parseval Networks: Improving Robustness to Adversarial Examples (Cisse et al., 2017)
[3] Formal Guarantees on the Robustness of a classifier against adversarial examples (Hein et al., 2017)
[4] On the Calibration of Modern Neural Networks (Guo et al., 2017)


Asynchronous Advantage Actor Critic (A3C) is an effective Reinforcement Learning (RL) algorithm for a wide range of tasks, such as Atari games and robot control. The agent learns policies and value function through trial-and-error interactions with the environment until converging to an optimal policy. Robustness and stability are critical in RL; however, neural network can be vulnerable to noise from unexpected sources and is not likely to withstand very slight disturbances. We note that agents generated from mild environment using A3C are not able to handle challenging environments. Learning from adversarial examples, we proposed an algorithm called Adversary Robust A3C (AR-A3C) to improve the agent's performance under noisy environments. In this algorithm, an adversarial agent is introduced to the learning process to make it more robust against adversarial disturbances, thereby making it more adaptive to noisy environments. Both simulations and real-world experiments are carried out to illustrate the stability of the proposed algorithm. The AR-A3C algorithm outperforms A3C in both clean and noisy environments.

The authors propose an extension of adversarial reinforcement learning to A3C. The proposed technique is of modest contribution and the experimental results do not provide sufficient validation of the approach.
The authors propose extending A3C to produce more robust policies by training a zero-sum game with two agents: a protagonist and an antagonist. The protagonist is attempting to achieve the given task while the antagonist's goal is for the task to fail.
The contribution of this work, AR-A3C, is extending adversarial reinforcement learning, namely robust RL (RRL) and robust adversarial RL (RARL), to A3C. In the context of this prior work the novelty is extending the family of adversarial RL methods. However, the proposed method is still within the same family methods as demonstrated by RARL.
The authors state that AR-A3C requires half as many rollouts as compared to RARL. However, no empirical comparison between the two methods is performed. The paper only performs analysis against the A3C and no other adversarial baseline and on only one environment: cartpole.  While they show transfer to the real world cartpole with this technique, there is not sufficient analysis to satisfactorily demonstrate the benefits of the proposed technique.
The paper reads well. There are a few notational issues in the paper that should be addressed. The authors mislabel the value function V as the  action value, or Q function. The action value function is action dependent where the value function is not.  As a much more minor issue, the authors introduce y as the discount factor, which deviates from the standard notation of \gamma without any obvious reason to do so.
Double blind was likely compromised with the youtube video, which was linked to a real name account instead of an anonymous account.
Overall, the proposed technique is of modest contribution and the experimental results do not provide sufficient validation of the approach.


Computing distances between examples is at the core of many learning algorithms for time series. Consequently, a great deal of work has gone into designing effective time series distance measures. We present Jiffy, a simple and scalable distance metric for multivariate time series. Our approach is to reframe the task as a representation learning problem-rather than design an elaborate distance function, we use a CNN to learn an embedding such that the Euclidean distance is effective. By aggressively max-pooling and downsampling, we are able to construct this embedding using a highly compact neural network. Experiments on a diverse set of multivariate time series datasets show that our approach consistently outperforms existing methods.

This paper presents a solid empirical analysis of a simple idea for learning embeddings of time series: training a convolutional network with a custom pooling layer that generates a fixed size representation to classify time series, then use the fixed size representation for other tasks. The primary innovation is a custom pooling operation that looks at a fraction of a sequence, rather than a fixed window. The experiments are fairly thorough (albeit with some sizable gaps) and show that the proposed approach outperforms DTW, as well as embeddings learned using Siamese networks. On the whole, I like the line of inquiry and the elegant simplicity of the proposed approach, but the paper has some flaws (and there are some gaps in both motivation and the experiments) that led me to assign a lower score. I encourage the authors to address these flaws as much as possible during the review period. If they succeed in doing so, I am willing to raise my score.
QUALITY
I appreciate this line of research in general, but there are some flaws in its motivation and in the design of the experiments. Below I list strengths (+) and weaknesses (-):
+ Time series representation learning is an important problem with a large number of real world applications. Existing solutions are often computationally expensive and complex and fail to generalize to new problems (particularly with irregular sampling, missing values, heterogeneous data types, etc.). The proposed approach is conceptually simple and easy to implement, faster to train than alternative metric learning approaches, and learns representations that admit fast comparisons, e.g., Euclidean distance.
+ The experiments are pretty thorough (albeit with some noteworthy gaps) -- they use multiple benchmark data sets and compare against strong baselines, both traditional (DTW) and deep learning (Siamese networks).
+ The proposed approach performs best on average!
- The custom pooling layer is the most interesting part and warrants additional discussion. In particular, the "naive" approach would be to use global pooling over the full sequence [4]. The authors should advance an argument to motivate %-length pooling and perhaps add a global pooling baseline to the experiments.
- Likewise, the authors need to fully justify the use of channel-wise (vs. multi-channel) convolutions and perhaps include a multi-channel convolution baseline.
- There is something incoherent about training a convolutional network to classify time series, then discarding the classification layer and using the internal representation as input to a 1NN classifier. While this yields an apples-to-apples comparison in the experiments, I am skeptical anyone would do this in practice. Why not simply use the classifier (I am dubious the 1NN would outperform it)? To address this, I recommend the authors do two things: (1) report the accuracy of the learned classifier; (2) discuss the dynamic above -- either admit to the reader that this is a contrived comparison OR provide a convincing argument that someone might use embeddings + KNN classifier instead of the learned classifier. If embeddings + KNN outperforms the learned classifier, that would surprise me, so that would warrant some discussion.
- On a related note, are the learned representations useful for tasks other than the original classification task? This would strengthen the value proposition of this approach. If, however, the learned representations are "overfit" to the classification task (I suspect they are), and if the learned classifier outperforms embeddings + 1NN, then what would I use these representations for?
- I am modestly surprised that this approach outperformed Siamese networks. The authors should report the Siamese architectures -- and how hyperparameters were tuned on all neural nets -- to help convince the reader that the comparison is fair.
- To that end, did the Siamese convolutional network use the same base architecture as the proposed classification network (some convolutions, custom pooling, etc.)? If not, then that experiment should be run to help determine the relative contributions of the custom pooling layer and the loss function.
- Same notes above re: triplet network -- the authors should report results in Table 2 and disclose architecture details.
- A stronger baseline would be a center loss [1] network (which often outperforms triplets).
- The authors might consider adding at least one standard unsupervised baseline, e.g., a sequence-to-sequence autoencoder [2,3].
CLARITY
The paper is clearly written for the most part, but there is room for improvement:
- The %-length pooling requires a more detailed explanation, particularly of its motivation. There appears to be a connection to other time series representations that downsample while preserving shape information -- the authors could explore this. Also, they should add a figure with a visual illustration of how it works (and maybe how it differs from global pooling), perhaps using a contrived example.
- How was the %-length pooling implemented? Most deep learning frameworks only provide pooling layers with fixed length windows, though I suspect it is probably straightforward to implement variable-width pooling layers in an imperative framework like PyTorch.
- Figure 1 is not well executed and probably unnecessary. The solid colored volumes do not convey useful information about the structure of the time series or the neural net layers, filters, etc. Apart from the custom pooling layer, the architecture is common and well understood by the community -- thus, the figure can probably be removed.
- The paper needs to fully describe neural net architectures and how hyperparameters were tuned.
ORIGINALITY
The paper scores low on originality. As the authors themselves point out, time series metric learning -- even using deep learning -- is an active area of research. The proposed approach is refreshing in its simplicity (rather than adding additional complexity on top of existing approaches), but it is straightforward -- and I suspect it has been used previously by others in practice, even if it has not been formally studied. Likewise, the proposed %-length pooling is uncommon, but it is not novel per se (dynamic pooling has been used in NLP [5]). Channel-wise convolutional networks have been used for time series classification previously [6].
SIGNIFICANCE
Although I identified several flaws in the paper's motivation and experimental setup, I think it has some very useful findings, at least for machine learning practitioners. Within NLP, there appears to be gradual shift toward using convolutional, instead of recurrent, architectures. I wonder if papers like this one will contribute toward a similar shift in time series analysis. Convolutional architectures are typically much easier and faster to train than RNNs, and the main motivation for RNNs is their ability to deal with variable length sequences. Convolutional architectures that can effectively deal with variable length sequences, as the proposed one appears to do, would be a welcome innovation.
REFERENCES
[1] Wen, et al. A Discriminative Feature Learning Approach for Deep Face Recognition. ECCV 2016.
[2] Fabius and van Amersfoort. Variational Recurrent Auto-Encoders. ICLR 2015 Workshop Track.
[3] Tikhonov and Yamshchikov. Music generation with variational recurrent autoencoder supported by history. arXiv.
[4] Hertel, Phan, and Mertins. Classifying Variable-Length Audio Files with All-Convolutional Networks and Masked Global Pooling.
[5] Kalchbrenner, Grefenstette, and Blunsom. A Convolutional Neural Network for Modelling Sentences. ACL 2014.
[6] Razavian and Sontag. Temporal Convolutional Neural Networks for Diagnosis from Lab Tests. arXiv.


We develop a reinforcement learning based search assistant which can assist users through a set of actions and sequence of interactions to enable them realize their intent. Our approach caters to subjective search where the user is seeking digital assets such as images which is fundamentally different from the tasks which have objective and limited search modalities. Labeled conversational data is generally not available in such search tasks and training the agent through human interactions can be time consuming. We propose a stochastic virtual user which impersonates a real user and can be used to sample user behavior efficiently to train the agent which accelerates the bootstrapping of the agent. We develop A3C algorithm based context preserving architecture which enables the agent to provide contextual assistance to the user. We compare the A3C agent with Q-learning and evaluate its performance on average rewards and state values it obtains with the virtual user in validation episodes. Our experiments show that the agent learns to achieve higher rewards and better states.

The paper "IMPROVING SEARCH THROUGH A3C REINFORCEMENT LEARNING BASED CONVERSATIONAL AGENT" proposes to define an agent to guide users in information retrieval tasks. By proposing refinements of the query, categorizations of the results or some other bookmarking actions, the agent is supposed to help the user in achieving his search. The proposed agent is learned via reinforcement learning.
My concern with this paper is about the experiments that are only based on simulated agents, as it is the case for learning. While it can be questionable for learning (but we understand why it is difficult to overcome), it is very problematic for the experiments to not have anything that demonstrates the usability of the approach in a real-world scenario. I have serious doubts about the performances of such an artificially learned approach for achieving real-world search tasks. Also, for me the experimental section is not sufficiently detailed, which lead to not reproducible results. Moreover, authors should have considered baselines (only the two proposed agents are compared which is clearly not sufficient).
Also, both models have some issues from my point of view. First, the Q-learning methods looks very complex: how could we expect to get an accurate model with 10^7 states ? No generalization about the situations is done here, examples of trajectories have to be collected for each individual considered state, which looks very huge (especially if we think about the number of possible trajectories in such an MDP). The second model is able to generalize from similar situations thanks to the neural architecture that is proposed. However, I have some concerns about it: why keeping the history of actions in the inputs since it is captured by the LSTM cell ? It is a redondant information that might disturb the process. Secondly, the proposed loss looks very heuristic for me, it is difficult to understand what is really optimized here. Particularly, the loss entropy function looks strange to me. Is it classical ? Are there some references of such a method to maintain some exploration ability. I understand the need of exploration, but including it in the loss function reduces the interpretability of the objective (wouldn't it be preferable to use a more classical loss but with an epsilon greedy policy?).
Other remarks:
   - In the begining of "varying memory capacity" section, what is "100, 150 and 250" ? Time steps ? What is the unit ? Seconds ?
   - I did not understand the "Capturing seach context at local and global level" at all
   - In the loss entropy formula, the two negation signs could be removed


Deep reinforcement learning has achieved many recent successes, but our understanding of its strengths and limitations is hampered by the lack of rich environments in which we can fully characterize optimal behavior, and correspondingly diagnose individual actions against such a characterization. Here we consider a family of combinatorial games, arising from work of Erdos, Selfridge, and Spencer, and we propose their use as environments for evaluating and comparing different approaches to reinforcement learning. These games have a number of appealing features: they are challenging for current learning approaches, but they form (i) a low-dimensional, simply parametrized environment where (ii) there is a linear closed form solution for optimal behavior from any state, and (iii) the difficulty of the game can be tuned by changing environment parameters in an interpretable way. We use these Erdos-Selfridge-Spencer games not only to compare different algorithms, but also to compare approaches based on supervised and reinforcement learning, to analyze the power of multi-agent approaches in improving performance, and to evaluate generalization to environments outside the training set.

The paper presents Erdos-Selfridge-Spencer games as environments for investigating
deep reinforcement learning algorithms. The proposed games are interesting and clearly challenging, but I am not sure what they tell us about the algorithms chosen to test them. There are some clarity issues with the justification and evaluation which undermine the message the authors are trying to make.
In particular, I have the following concerns:
  • these games have optimal policies that are expressible as a linear model, meaning that if the architecture or updating of the learning algorithm is such that there is a bias towards exploring these parts of policy space, then they will perform better than more general algorithms. What does this tell us about the relative merits of each approach? The authors could do more to formally motivate these games as "difficult" for any deep learning architecture if possible.
  • the authors compare linear models with non-linear models at some point for attacker policies, but it is unclear whether these linear models are able to express the optimal policy. In fact, there is a level of non-determinism in how the attacker policies are encoded which means that an optimal policy cannot be (even up to soft-max) expressed by the agent (as I read things the number of pieces chosen in level l is always chosen uniformly randomly).
  • As the authors state, this paper is an empirical evaluation, and the theorems presented are derived from earlier work. There is possibly too much focus on the proofs of these theorems.
  • There are a number of ambiguities and errors which places difficulties on the interpretation (and potential replication) of the experiments. As this is an empirical study, this is the yardstick by which the paper should be judged. In particular, this relates to:
    ◦ The architecture of each of the tested Deep RL methods.
    ◦ What is done to select appropriate tuning parameters of the tested Deep RL methods, if anything.
    ◦ It is unclear whether 'incorrect actions' in the supervised learning evaluations, refer to non-optimal actions, or simply actions that do not preserve the dominance of the defender, e.g. both partitions may have potential >0.5
    ◦ Fig 4. right looks like a reward signal, but is labelled Proportion correct. The text is not clear enough to be sure which it is.
    ◦ Fig 4. left and right has 4 methods: rl rewards, rl correct actions, sup rewards, and sup correct actions. The specifics of how these methods are constructed is unclear from the paper.
    ◦ What parts of the evaluation explores how well these methods are able to represent the states (feature/representation learning) and what parts are evaluating the propagation of sparse rewards (the reinforcment learning core)? The authors could be clearer and more targetted with respect to this question.
There is value in this work, but in its current state I do not think it is ready for publicaiton.
# Detailed notes
[p4, end of sec 3] The authors say that the difficulty of the games can be varied with "continuous changes in potential", but the potential is derived from the discrete initial game state, so these values are not continuously varying (even though it is possible to adjust them by non-integer amounts).
[p4, sec 4.1]
"strategy unevenly partitions the occupied levels...with the proportional difference between the two sets being sampled randomly"
What is meant by this? The proportional difference between the two sets is discussed as if it is a continuous property, but must be chosen from the discrete set of all available partitions. If one partition one is chosen uniformly randomly from all possibly sets A, B (and the potential proportion calculated) then I don't know why it would be written in this way. That suggests that proportions that are closer to 1:1 are chosen more often than "extreme" partitions, but how? This feels a little under-justified.
"very different states A, B (uneven potential, disjoint occupied levels)"
Are these states really "very different", or at least for the reasons indicated. Later on (Theorem 3) we see how an optimal partition is generated. This chooses a partition where one part contains all pieces in layer (l+1) and above and one part with all pieces in layer (l-1) and below, with layer l being distributed between the two parts. The first part will typically have a slightly lower potential than the other and all layers other than layer l will be disjoint.
[p6, Fig 4] The right plot y-limits vary between -1 and 1 so it cannot represent a proportion of correct actions. Also, in the text the authors say:
  >> The results, shown in Figure 4 are surprising. Reinforcement learning
  >> is better at playing the game, but does worse at predicting optimal moves.
I am not sure which plot shows the playing of the game. Is this the right hand plot? In which case are we looking at rewards? In fact, I am a little confused as to what is being shown here. Is "sup rewards" a supervised learning method trained on rewards, or evaluated on rewards, or both? And how is this done. The text is just not clear enough.
[p7 Fig 6 and text] Here the authors are comparing how well agents select the optimal actions as compared to how close they are to the end of the game. This relates to the "surprising" fact that "Reinforcement learning is better at playing the game, but does worse at predicting optimal moves.". I think an important point here is how many training/test examples there are in each bin. If there are more in the range 3-7 moves from the end of the game, than there are outside this range, then the supervised learner will
[p8 proof of theorem 3]
"φ(A l+1 ) < 0.5 and φ(A l ) > 0.5."
Is it true that both these inequalities are strict?
"Since A l only contains pieces from levels K to l + 1"
In fact this should read from levels K to l.
"we can move k < m − n pieces from A l+1 to A l"
Do the authors mean that we can define a partition A, B where A = A_{l+1} plus some (but not all) elements in level l (A_{l}\setminus A_{l+1})?
"...such that the potential of the new set equals 0.5"
It will equal exactly 0.5 as suggested, but the authors could make it more precise as to why (there is a value n+k < l (maybe <=l) such that (n+k)*2^{-(K-l+1)}=0.5 (guaranteed). They should also indicate why this then justifies their proof (namely that phi(S0)-0.5 >= 0.5).
[p8 paramterising action space] A comment: this doesn't give as much control as the authors suggest. Perhaps the agent should also chose the proportion of elements in layer l to set A. For instance, if there are a large number of elements in l, and or phi(A_{l+1}) is very close to 0.5 (or phi(A_l) is very close to 0.5) then this doesn't give the attacker the opportunity to fine tune the policy to select very good partitions.  It is unclear expected level of control that agents have under various conditions (K and starting states).
[p9 Fig 8] As the defender's score is functionally determined by the attackers score, it doesn't help to include this on the plot. It just distracts from the signal.


Workshop track -ICLR 2018 DLVM: A MODERN COMPILER INFRASTRUCTURE FOR DEEP LEARNING SYSTEMS
Deep learning software demands reliability and performance. However, many of the existing deep learning frameworks are software libraries that act as an unsafe DSL in Python and a computation graph interpreter. We present DLVM, a design and implementation of a compiler infrastructure with a linear algebra intermediate representation, algorithmic differentiation by adjoint code generation, domainspecific optimizations and a code generator targeting GPU via LLVM. Designed as a modern compiler infrastructure inspired by LLVM, DLVM is more modular and more generic than existing deep learning compiler frameworks, and supports tensor DSLs with high expressivity. With our prototypical staged DSL embedded in Swift, we argue that the DLVM system enables a form of modular, safe and performant frameworks for deep learning.

The success of Deep Learning is, in no small part, due the development of libraries and frameworks which have made building novel models much easier, faster and less error prone and also make taking advantage of modern hardware (such as GPUs) more accessible. This is still a vital area of work, as new types of models and hardware are developed.
This work argues that prior solutions do not take advantage of the fact that a tensor compiler is, essentially, just a compiler. They introduce  DLVM (and NNKit) which comprises LLVM based compiler infrastructure and a DSL allowing the use of Swift to describe a typed tensor graph. Unusually, compared to most frameworks, gradients are calculated using source code transformation, which is argued to allow for easier optimization.
This paper is not well-adapted for an ICLR audience, many of which are not experts in compilers or LLVM. For example, the Figure 3, table 1 would be benefit from being shorter with more exposition on what the reader should understand and take away from them.
The primary weakness of this work is the lack of careful comparison with existing framework. The authors mention several philosophical arguments in favor of their approach, but is there a concrete example of an model which is cumbersome to write in an existing framework but easy here? (e.g. recent libraries pytorch, TF eager can express conditional logic much more simply than previous approaches, its easy to communicate why you might use them). Because of this work seems likely to be of limited interest to the ICLR audience, most of which are potentially interested users rather than compiler experts. There is also no benchmarking, which is at odds with the claims the compiler approaches allows easier optimization.
One aspect that seemed under-addressed and which often a crucial aspect of a good framework, is how general purpose code e.g. for loading data or logging interacts with the accelerated tensor code.


Unsupervised learning of time series data, also known as temporal clustering, is a challenging problem in machine learning. Here we propose a novel algorithm, Deep Temporal Clustering (DTC), to naturally integrate dimensionality reduction and temporal clustering into a single end-to-end learning framework, fully unsupervised. The algorithm utilizes an autoencoder for temporal dimensionality reduction and a novel temporal clustering layer for cluster assignment. Then it jointly optimizes the clustering objective and the dimensionality reduction objective. Based on requirement and application, the temporal clustering layer can be customized with any temporal similarity metric. Several similarity metrics and state-of-the-art algorithms are considered and compared. To gain insight into temporal features that the network has learned for its clustering, we apply a visualization method that generates a region of interest heatmap for the time series. The viability of the algorithm is demonstrated using time series data from diverse domains, ranging from earthquakes to spacecraft sensor data. In each case, we show that the proposed algorithm outperforms traditional methods. The superior performance is attributed to the fully integrated temporal dimensionality reduction and clustering criterion.

Summary:
The authors proposed an unsupervised time series clustering methods built with deep neural networks. The proposed model is equipped with an encoder-decoder and a clustering model. First, the encoder employs CNN to shorten the time series and extract local temporal features, and the CNN is followed by bidirectional LSTMs to get the encoded representations. A temporal clustering model and a DCNN decoder are applied on the encoded representations and jointly trained. An additional heatmap generator component can be further included in the clustering model. The authors compared the proposed method with hierarchical clustering with 4 different temporal similarity methods on several univariate time series datasets.
Detailed comments:
The problem of unsupervised time series clustering is important and challenging. The idea of utilizing deep learning models to learn encoded representations for clustering is interesting and could be a promising solution.
One potential limitation of the proposed method is that it is only designed for univariate time series of the same temporal length, which limits the usage of this model in practice. In addition, given that the input has fixed length, clustering baselines for static data can be easily applied and should be compared to demonstrate the necessity of temporal clustering.
Some important details are missing or lack of explanations. For example, what is the size of each layer and the dimension of the encoded space? How much does the model shorten the input time series and how is this be determined?
How does the model combine the heatmap output (which is a sequence of the same length as the time series) and the clustering output (which is a vector of size K) in Figure 1? The heatmap shown in Figure 3 looks like the negation of the decoded output (i.e., lower value in time series -> higher value in heatmap). How do we interpret the generated heatmap?
From the experimental results, it is difficult to judge which method/metric is the best. For example, in Figure 4, all 4 DTC-methods achieved the best performance on one or two datasets. Though several datasets are evaluated in experiments, they are relatively small. Even the largest dataset (Phalanges OutlinesCorrect) has only 2 thousand samples, and the best performance is achieved by one of the baseline, with AUC score only 0.586 for binary classification.
Minor suggestion:
In Figure 3, instead of showing the decoded output (reconstruction), it may be more helpful to visualize the encoded time series since the clustering method is applied directly on those encoded representations.


Published as a conference paper at ICLR 2018 BACKPROPAGATION THROUGH THE VOID: OPTIMIZING CONTROL VARIATES FOR BLACK-BOX GRADIENT ESTIMATION
Gradient-based optimization is the foundation of deep learning and reinforcement learning, but is difficult to apply when the mechanism being optimized is unknown or not differentiable. We introduce a general framework for learning low-variance, unbiased gradient estimators, applicable to black-box functions of discrete or continuous random variables. Our method uses gradients of a surrogate neural network to construct a control variate, which is optimized jointly with the original parameters. We demonstrate this framework for training discrete latent-variable models. We also give an unbiased, action-conditional extension of the advantage actor-critic reinforcement learning algorithm.

This paper suggests a new approach to performing gradient descent for blackbox optimization or training discrete latent variable models. The paper gives a very clear account of existing gradient estimators and finds a way to combine them so as to construct and optimize a differentiable surrogate function. The resulting new gradient estimator is then studied both theoretically and empirically. The empirical study shows the benefits of the new estimator for training discrete variational autoencoders and for performing deep reinforcement learning.
To me, the main strengths of the paper is the very clear account of existing gradient estimators (among other things it helped me understand obscurities of the Q-prop paper) and a nice conceptual idea. The empirical study itself is more limited and the paper suffers from a few mistakes and missing information, but to me the good points are enough to warrant publication of the paper in a good conference like ICLR.
Below are my comments for the authors.
---------------------------------
General, conceptual comments:
When reading (6), it is clear that the framework performs regression of $c_\phi$ towards the unknown $f$ simultaneously with optimization over $c_\phi$.
Taking this perspective, I would be glad to see how the regression part performs with respect to standard least square regression,
i.e. just using $||f(b)-c_\phi(b)||^2$ as loss function. You may compare the speed of convergence of $c_\phi$ towards $f$ using (6) and the least squared error.
You may also investigate the role of this regression part into the global g_LAX optimization by studying the evolution of the components of (6).
Related to the above comment, in Algo. 1, you mention "f(.)" as given to the algo. Actually, the algo does not know f itself, otherwise it would not be blackbox optimization. So you may mean different things. In a batch setting, you may give a batch of [x,f(x) (,cost(x)?)] points to the algo. You more probably mean here that you have an "oracle" that, given some x, tells you f(x) on demand. But the way you are sampling x is not specified clearly.
This becomes more striking when you move to reinforcement learning problems, which is my main interest. The RL algorithm itself is not much specified. Does it use a replay buffer (probably not)? Is it on-policy or off-policy (probably on-policy)? What about the exploration policy? I want to know more... Probably you just replace (10) with (11) in A2C, but this is not clearly specified.
In Section 4, can you explain why, in the RL case, you must introduce stochasticity to the inputs? Is this related to the exploration issue (see above)?
Last sentence of conclusion: you are too allusive about the relationship between your learned control variate and the Q-function. I don't get it, and I want to know more...
-----------------------------------
Local comments:
Backpropagation through the void: I don't understand why this title. I'm not a native english speaker, I'm probably missing a reference to something, I would be glad to get it.
Figure 1 right. Caption states variance, but it is log variance. Why does it oscillate so much with RELAX?
Beginning of 3.1: you may state more clearly that optimizing $c_\phi$ the way you do it will also "minimize" the variance, and explain better why ("we require the gradient of the variance of our gradient estimator"...). It took me a while to get it.
In 3.1.1 a weighting based on $\d/\d\theta log p(b)$ => shouldn't you write $... log p(b|\theta)$ as before?
Figure 2 is mentioned in p.3, it should appear much sooner than p6.
In Figure 2, there is nothing about the REINFORCE PART. Why?
In 3.4 you alternate sums over an infinite horizon and sums over T time steps. You should stick to the T horizon case, as you mention the case T=1 later.
p6 Related work
The link to the work of Salimans 2017 is far from obvious, I would be glad to know more...
Q-prop (Haarnoja et al.,2017): this is not the adequate reference to Q-prop, it should be (Gu et al. 2016), you have it correct later ;)
Figure 3: why do you stop after so few epochs? I wondered how expensive is the computation of your estimator, but since in the RL case you go up to 50 millions (or 4 millions?), it's probably not the issue. I would be glad to see another horizontal lowest validation error for your RELAX estimator (so you need to run more epochs).
"ELBO" should be explained here (it is only explained in the appendices).
6.2, Table 1: Best obtained training objective: what does this mean? Should it be small or large? You need to explain better. How much is the modest improvement (rather give relative improvement in the text?)? To me, you should not defer Table 3 to an appendix (nor Table 4).
Figure 4: Any idea why A2C oscillates so much on inverted pendulum? Any idea why variance starts to decrease after 500 episodes using RELAX? Isn't related to the combination of regression and optimization, as suggested above?
About Double Inverted Pendulum, Appendix E3 mentions 50 million frames, but the figure shows 4 millions steps. Where is the truth?
Why do you give steps for the reward, and episodes for log-variance? The caption mentions "variance (log-scale)", but saying "log-variance" would be more adequate.
p9: the optimal control variate: what is this exactly? How do you compare a control variate over another? This may be explained in Section 2.
GAE (Kimura, 2000). I'm glad you refer to former work (there is a very annoying tendency those days to refer only to very recent papers from a small set of people who do not correctly refer themselves to previous work), but you may nevertheless refer to John Schulman's paper about GAEs anyways... ;)
Appendix E.1 could be reorganized, with a common hat and then E.1.1 for one layer model(s?) and E.1.2 for the two layer model(s?)
A sensitivity analysis wrt to your hyper-parameters would be welcome, this is true for all empirical studies.
In E2, is the output layer linear? You just say it is not ReLU...
The networks used in E2 are very small (a standard would be 300 and 400 neurons in hidden layers). Do you have a constraint on this?
"As our control variate does not have the same interpretation as the value function of A2C, it was not directly clear how to add reward bootstrapping and other variance reduction techniques common in RL into our model. We leave the task of incorporating these and other variance reduction techniques to future work."
First, this is important, so if this is true I would move this to the main text (not in appendix).
But also, it seems to me that the first sentence of E3 contradicts this, so where is the truth?
{0.01,0.003,0.001} I don't believe you just tried these values. Most probably, you played with other values before deciding to perform grid search on these, right?
The same for 25 in E3.
Globally, you experimental part is rather weak, we would expect a stronger methodology, more experiments also with more difficult benchmarks (half-cheetah and the whole gym zoo ;)), more detailed analyses of the results, but to me the value of your paper is more didactical and conceptual than experimental, which I really appreciate, so I will support your paper despite these weaknesses.
Good luck! :)
---------------------------------------
Typos:
p5
monte-carlo => Monte(-)Carlo (no - later...)
taylor => Taylor
you should always capitalize Section, equation, table, figure, appendix, ...
gradient decent => descent (twice)
p11: probabalistic
p15 ELU(Djork-... => missing space


Autonomous vehicles are becoming more common in city transportation. Companies will begin to find a need to teach these vehicles smart city fleet coordination. Currently, simulation based modeling along with hand coded rules dictate the decision making of these autonomous vehicles. We believe that complex intelligent behavior can be learned by these agents through Reinforcement Learning. In this paper, we discuss our work for solving this system by adapting the Deep Q-Learning (DQN) model to the multi-agent setting. Our approach applies deep reinforcement learning by combining convolutional neural networks with DQN to teach agents to fulfill customer demand in an environment that is partially observable to them. We also demonstrate how to utilize transfer learning to teach agents to balance multiple objectives such as navigating to a charging station when its energy level is low. The two evaluations presented show that our solution has shown that we are successfully able to teach agents cooperation policies while balancing multiple objectives.

The main contribution of the paper seems to be the application to this problem, plus minor algorithmic/problem-setting contributions that consist in considering partial observability and to balance multiple objectives. On one hand, fleet management is an interesting and important problem. On the other hand, although the experiments are well designed and illustrative, the approach is only tested in a small 7x7 grid and 2 agents and in a 10x10 grid with 4 agents. In spirit, these simulations are similar to those in the original paper by M. Egorov. Since the main contribution is to use an existing algorithm to tackle a practical application, it would be more interesting to tweak the approach until it is able to tackle a more realistic scenario (mainly larger scale, but also more realistic dynamics with traffic models, real data, etc.).
Simulation results compare MADQN with Dijkstra's algorithm as a baseline, which offers a myopic solution where each agent picks up the closest customer. Again, since the main contribution is to solve a specific problem, it would be worthy to compare with a more extensive benchmark, including state of the art algorithms used for this problem (e.g., heuristics and metaheuristics).
The paper is clear and well written. There are several minor typos and formatting errors (e.g., at the end of Sec. 3.3, the authors mention Figure 3, which seems to be missing, also references [Egorov, Maxim] and [Palmer, Gregory] are bad formatted).
-- Comments and questions to the authors:
1. In the introduction, please, could you add references to what is called "traditional solutions"?
2. Regarding the partial observability, each agent knows the location of all agents, including itself, and the location of all obstacles and charging locations; but it only knows the location of customers that are in its vision range. This assumption seems reasonable if a central station broadcasts all agents' positions and customers are only allowed to stop vehicles in the street, without ever contacting the central station; otherwise if agents order vehicles in advance (e.g., by calling or using an app) the central station should be able to communicate customers locations too. On the other hand, if no communication with the central station is allowed, then positions of other agents may be also partial observable. In other words, the proposed partial observability assumption requires some further motivation. Moreover, in Sec. 4.3, it is said that agents can see around them +10 spaces away; however, experiments are run in 7x7 and 10x10 grid worlds, meaning that the agents are able to observe the grid completely.
3. The fact that partial observability helped to alleviate the credit-assignment noise caused by the missing customer penalty might be an artefact of the setting. For instance, since the reward has been designed arbitrarily, it could have been defined as giving a penalty for those missing customers that are at some distance of an agent.
4. Please, could you explain the last sentence of Sec. 4.3 that says "The drawback here is that the agents will not be able to generalize to other unseen maps that may have very different geographies." In particular, how is this sentence related to partial observability?


We present a novel approach for training neural machines which incorporates additional supervision on the machine's interpretable components (e.g., neural memory). To cleanly capture the kind of neural machines to which our method applies, we introduce the concept of a differential neural computational machine (∂NCM) and show that several existing architectures (e.g., NTMs, NRAMs) can be instantiated as a ∂NCM and can thus benefit from any amount of additional supervision over their interpretable components. Based on our method, we performed a detailed experimental evaluation with NTM and NRAM machines, showing the approach leads to significantly better convergence and generalization capabilities of the learning phase than standard training using only input-output examples.

Summary
This paper presents differentiable Neural Computational Machines (∂NCM), an abstraction of existing neural abstract machines such as Neural Turing Machines (NTMs) and Neural Random Access Machines (NRAMs). Using this abstraction, the paper proposes loss terms for incorporating supervision on execution traces. Adding supervision on execution traces in ∂NCM improves performance over NTM and NRAM which are trained end-to-end from input/output examples only. The observation that adding additional forms of supervision through execution traces improves generalization may be unsurprising, but from what I understand the main contribution of this paper lies in the abstraction of existing neural abstract machines to ∂NCM. However, this abstraction does not seem to be particularly useful for defining additional losses based on trace information. Despite the generic subtrace loss (Eq 8), there is no shared interface between ∂NCM versions of NTM and NRAM that would allow one to reuse the same subtrace loss in both cases. The different subtrace losses used for NTM and NRAM (Eq 9-11) require detailed knowledge of the underlying components of NTM and NRAM (write vector, tape, register etc.), which questions the value of ∂NCM as an abstraction.
Weaknesses
As explained in the summary, it is not clear to me why the abstraction to NCM is useful if one still needs to define specific subtrace losses for different neural abstract machines.
The approach seems to be very susceptible to the weight of the subtrace loss λ, at least when training NTMs. In my understanding each of the trace supervision information (hints, e.g. the ones listed in Appendix F) provides a sensible inductive bias we would the NTM to incorporate. Are there instances where these biases are noisy, and if not, could we incorporate all of them at the same time despite the susceptibility w.r.t λ?
NTMs and other recent neural abstract machines are often tested on rather toyish algorithmic tasks. I have the impression providing extra supervision in form of execution traces makes these tasks even more toyish. For instance, when providing input-output examples as well as the auxiliary loss in Eq6, what exactly is left to learn? What I like about Neural-Programmer Interpreters and Neural Programmer [1] is that they are tested on less toyish tasks (a computer vision and a question answering task respectively), and I believe the presented method would be more convincing for a more realistic downstream task where hints are noisy (as mentioned on page 5).
Minor Comments
p1: Why is Grefenstette et al. (2015) an extension of NTMs or NRAMs? While they took inspiration from NTMs, their Neural Stack has not much resemblance with this architecture.
p2: What is B exactly? It would be good to give a concrete example at this point. I have the feeling it might even be better to explain NCMs in terms of the communication between κ, π and M first, so starting with what I, O, C, B, Q are before explaining what κ and π are (this is done well for NTM as ∂NCM in the table on page 4). In addition, I think it might be better to explain the Controller before the Processor. Furthermore, Figure 2a should be referenced in the text here.
p4 Eq3: There are two things confusing in these equations. First, w is used as the write vector here, whereas on page 3 this is a weight of the neural network. Secondly, π and κ are defined on page 2 as having an element from W as first argument, which are suddenly omitted on page 4.
p4: The table for NRAM as ∂NCM needs a bit more explanation. Where does {1}=I come from? This is not obvious from Appendix B either.
p3 Fig2/p4 Eq4: Related to the concern regarding the usefulness of the ∂NCM abstraction: While I see how NTMs fit into the NCM abstraction, this is not obvious at all for NRAMs, particularly since in Fig 2c modules are introduced that do not follow the color scheme of κ and π in Fig 2a (ct, at, bt and the registers).
p5: There is related work for incorporating trace supervision into a neural abstract machine that is otherwise trained end-to-end from input-output examples [2].
p5: "loss on example of difficulties" -> "loss on examples of the same difficulty"
p5: Do you have an example for a task and hints from a noisy source?
Citation style: sometimes citation should be in brackets, for example "(Graves et al. 2016)" instead of "Graves et al. (2016)" in the first paragraph of the introduction.
[1] Neelakantan et al. Neural programmer: Inducing latent programs with gradient descent. ICLR. 2015.
[2] Bosnjak et al. Programming with a Differentiable Forth Interpreter. ICML. 2017.


TRUST-PCL: AN OFF-POLICY TRUST REGION METHOD FOR CONTINUOUS CONTROL
Trust region methods, such as TRPO, are often used to stabilize policy optimization algorithms in reinforcement learning (RL). While current trust region strategies are effective for continuous control, they typically require a large amount of on-policy interaction with the environment. To address this problem, we propose an off-policy trust region method, Trust-PCL, which exploits an observation that the optimal policy and state values of a maximum reward objective with a relative-entropy regularizer satisfy a set of multi-step pathwise consistencies along any path. The introduction of relative entropy regularization allows Trust-PCL to maintain optimization stability while exploiting off-policy data to improve sample efficiency. When evaluated on a number of continuous control tasks, Trust-PCL significantly improves the solution quality and sample efficiency of TRPO.

This paper presents a policy gradient method that employs entropy regularization and entropy constraint at the same time. The entropy regularization on action probability is to encourage the exploration of the policy, while the entropy constraint is to stabilize the gradient.
The major weakness of this paper is the unclear presentation. For example, the algorithm is never fully described, though a handful variants are discussed. How the off-policy version is implemented is missing.
In experiments, why the off-policy version of TRPO is not compared. Comparing the on-policy results, PCL does not show a significant advantage over TRPO. Moreover, the curves of TRPO is so unstable, which is a bit uncommon.
What is the exploration strategy in the experiments? I guess it was softmax probability. However, in many cases, softmax does not perform a good exploration, even if the entropy regularization is added.
Another issue is the discussion of the entropy regularization in the objective function. This regularization, while helping exploration, do changes the original objective. When a policy is required to pass through a very narrow tunnel of states, the regularization that forces a wide action distribution could not have a good performance. Thus it would be more interesting to see experiments on more complex benchmark problems like humanoids.


Inspired by the combination of feedforward and iterative computations in the visual cortex, and taking advantage of the ability of denoising autoencoders to estimate the score of a joint distribution, we propose a novel approach to iterative inference for capturing and exploiting the complex joint distribution of output variables conditioned on some input variables. This approach is applied to image pixel-wise segmentation, with the estimated conditional score used to perform gradient ascent towards a mode of the estimated conditional distribution. This extends previous work on score estimation by denoising autoencoders to the case of a conditional distribution, with a novel use of a corrupted feedforward predictor replacing Gaussian corruption. An advantage of this approach over more classical ways to perform iterative inference for structured outputs, like conditional random fields (CRFs), is that it is not any more necessary to define an explicit energy function linking the output variables. To keep computations tractable, such energy function parametrizations are typically fairly constrained, involving only a few neighbors of each of the output variables in each clique. We experimentally find that the proposed iterative inference from conditional score estimation by conditional denoising autoencoders performs better than comparable models based on CRFs or those not using any explicit modeling of the conditional joint distribution of outputs.

The paper proposes an image segmentation method which iteratively refines the semantic segmentation mask obtained from a deep net. To this end the authors investigate a denoising auto-encoder (DAE). Its purpose is to provide a semantic segmentation which improves upon its input in terms of the log-likelihood.
More specifically, the authors `propose to condition the autoencoder with an additional input’ (page 1). To this end they use features obtained from the deep net. Instead of training the DAE with ground truth y, the authors found usage of the deep net prediction to yield better results.
The proposed approach is evaluated on the CamVid dataset.
Summary:
——
I think the paper discusses a very interesting topic and presents an elegant approach. A few points are missing which would provide significantly more value to a reader. Specifically, an evaluation on the classical Pascal VOC dataset, details regarding the training protocol of the baseline (which are omitted right now), an assessment regarding stability of the proposed approach (not discussed right now), and a clear focus of the paper on segmentation or conditioning. See comments below for details and other points.
Comments:
——
1. When training the DAE, a combination of squared loss and categorical cross-entropy loss is used. What’s the effect of the squared error loss and would the categorical cross-entropy on its own be sufficient? This question remains open when reading the submission.
2. The proposed approach is evaluated on the CamVid dataset which is used less compared to the standard and larger Pascal VOC dataset. I conjecture that the proposed approach wouldn’t work too well on Pascal VOC. On Pascal VOC, images are distinctly different from each other whereas subsequent frames are similar in CamVid, i.e., the road is always located at the bottom center of the image. The proposed architecture is able to take advantage of this dataset bias, but would fail to do so on Pascal VOC, which has a much more intricate bias. It would be great if the authors could check this hypothesis and report quantitative results similar to Tab. 1 and Fig. 4 for Pascal VOC.
3. The authors mention a grid-search for the stepsize and the number of iterations. What values were selected in the end on the CamVid and hopefully the Pascal VOC dataset?
4. Was the dense CRF applied out of the box, or were its parameters adjusted for good performance on the CamVid validation dataset? While parameters such as the number of iterations and epsilon are tuned for the proposed approach on the CamVid validation set, the submission doesn’t specify whether a similar procedure was performed for the CRF baseline.
5. Fig. 4 seems to indicate that the proposed approach doesn’t converge. Hence an appropriate stepsize and a reasonable number of iterations need to be chosen on a validation set. Choosing those parameters guarantees that the method performs well on average, but individual results could potentially be entirely wrong, particularly if large step sizes are chosen. I suspect this effect to be more pronounced on the Pascal VOC dataset (hence my conjecture in point 2). To further investigate this property, as a reader, I’d be curious to get to know the standard deviation/variance of the accuracy in addition to the mean IoU. Again, it would be great if the authors could check this hypothesis and report those results.
6. I find the experimental section to be slightly disconnected from the initial description. Specifically, the paper `proposes to condition the autoencoder with an additional input’ (page 1). No experiments are conducted to validate this proposal. Hence the main focus of the paper (image segmentation or DAE conditioning) remains vague. If the authors choose to focus on image segmentation, a comparison to state-of-the-art should be provided on classical datasets such as Pascal VOC, if DAE conditioning is the focus, some experiments in this direction should be included in addition to the Pascal VOC results.
Minor comment:
——
- I find it surprising that the authors choose not to cite some related work on combining deep nets with structured prediction.


GRAPH ATTENTION NETWORKS
We present graph attention networks (GATs), novel neural network architectures that operate on graph-structured data, leveraging masked self-attentional layers to address the shortcomings of prior methods based on graph convolutions or their approximations. By stacking layers in which nodes are able to attend over their neighborhoods' features, we enable (implicitly) specifying different weights to different nodes in a neighborhood, without requiring any kind of computationally intensive matrix operation (such as inversion) or depending on knowing the graph structure upfront. In this way, we address several key challenges of spectral-based graph neural networks simultaneously, and make our model readily applicable to inductive as well as transductive problems. Our GAT models have achieved or matched state-of-the-art results across four established transductive and inductive graph benchmarks: the Cora, Citeseer and Pubmed citation network datasets, as well as a protein-protein interaction dataset (wherein test graphs remain unseen during training).

The paper introduces a neural network architecture to operate on graph-structured
data named Graph Attention Networks.
Key components are an attention layer and the possibility to learn how to
weight different nodes in the neighborhood without requiring spectral decompositions
which are costly to be computed.
I found the paper clearly written and very well presented. I want to thank
the author for actively participating in the discussions and in clarifying already
many of the details that I was missing.
As also reported in the comments by T. Kipf I found the lack of comparison to previous
works on attention and on constructions of NN for graph data are missing.
In particular MoNet seems a more general framework, using features to compute node
similarity is another way to specify the "coordinate system" for convolution.
I would argue that in many cases the graph is given and that one would have
to exploit its structure rather than the simple first order neighbors structure.
I feel, in fact, that the paper deals mainly with "localized metric-learning" rather than
using the information in the graph itself. There is no
explicit usage of the graph beyond the selection of the local neighborhood.
In many ways when I first read it I though it would be a modified version of
memory networks (which have not been cited). Sec. 2.1 is basically describing
a way to learn a matrix W so that the attention layer produces the weights to be
used for convolution, or the relative coordinate system, which is to me a
memory network like construction, where the memory is given by the neighborhood.
I find the idea to use the multi-head attention very interesting, but one should
consider the increase in number of parameters in the experimental section.
I agree that the proposed method is computationally efficient but the authors
should keep in mind that parallelizing across all edges involves lot of redundant
copies (e.g. in a distributed system) as the neighborhoods highly overlap, at
least for interesting graphs.
The advantage with respect to methods that try to use LSTM in this domain
in a naive manner is clear, however the similarity function (attention) in this
work could be interpreted as the variable dictating the visit ordering.
The authors seem to emphasize the use of GPU as the best way to scale their work
but I tend to think that when nodes have varying degrees they would be highly
unused. Main reason why they are widely used now is due to the structure in the
representation of convolutional operations.
Also in case of sparse data GPUs are not the best alternative.
Experiments are very well described and performed, however as explained earlier
some comparisons are needed.
An interesting experiment could be to use the attention weights as adjacency
matrix for GCN.
Overall I liked the paper and the presentation, I think it is a simple yet
effective way of dealing with graph structure data. However, I think that in
many interesting cases the graph structure is relevant and cannot be used
just to get the neighboring nodes (e.g. in social network analysis).


VARIATIONAL INFERENCE OF DISENTANGLED LATENT CONCEPTS FROM UNLABELED OBSERVATIONS
Disentangled representations, where the higher level data generative factors are reflected in disjoint latent dimensions, offer several benefits such as ease of deriving invariant representations, transferability to other tasks, interpretability, etc. We consider the problem of unsupervised learning of disentangled representations from large pool of unlabeled observations, and propose a variational inference based approach to infer disentangled latent factors. We introduce a regularizer on the expectation of the approximate posterior over observed data that encourages the disentanglement. We also propose a new disentanglement metric which is better aligned with the qualitative disentanglement observed in the decoder's output. We empirically observe significant improvement over existing methods in terms of both disentanglement and data likelihood (reconstruction quality).

########## UPDATED AFTER AUTHOR RESPONSE ##########
Thanks for the good revision and response that addressed most of my concerns. I am bumping up my score.
###############################################
This paper presents a Disentangled Inferred Prior (DIP-VAE) method for learning disentangled features from unlabeled observations following the VAE framework. The basic idea of DIP-VAE is to enforce the aggregated posterior q(z) = E_x [q(z | x)] to be close to an identity matrix as implied by the commonly chosen standard normal prior p(z). The authors propose to moment-match q(z) given it is hard to minimize the KL-divergence between q(z) and p(z). This leads to one additional term to the regular VAE objective (in two parts, on- and off-diagonal). It has the similar property as beta-VAE (Higgins et al. 2017) but without sacrificing the reconstruction quality. Empirically the authors demonstrate that DIP-VAE can effectively learn disentangled features, perform comparably better than beta-VAE and at the same time retain the reconstruction quality close to regular VAE (beta-VAE with beta = 1).
The paper is overall well-written with minor issues (listed below). I think the idea of enforcing an aggregated (marginalized) posterior q(z) to be close to the standard normal prior p(z) makes sense, as opposed to enforcing each individual posterior q(z|x) to be close to p(z) as (beta-)VAE objective suggests. I would like to make some connection to some work on understanding VAE objective (Hoffman & Johnson 2016, ELBO surgery: yet another way to carve up the variational evidence lower bound) where they derived something along the same line of an aggregated posterior q(z). In Hoffman & Johnson, it is shown that KL(q(z) | p(z)) is in fact buried in ELBO, and the inequality gap in Eq (3) is basically a mutual information term between z and n (the index of the data point). Similar observations have led to the development of VAMP-prior (Tomczak & Welling 2017, VAE with a VampPrior). Following the derivation in Hoffman & Johnson, DIP-VAE is basically adding a regularization parameter to the KL(q(z) | p(z)) term in standard ELBO. I think this interpretation is complementary to (and in my opinion, more clear than) the one that’s described in the paper.
My concerns are mostly regarding the empirical studies:
1. One of my main concern is on the empirical results in Table 1. The disentanglement metric score for beta-VAE is suspiciously lower than what’s reported in Higgins et al., where they reported a 99.23% disentanglement metric score on 2D shape dataset. I understand the linear classier is different, but still the difference is too large to ignore. Hence my current more neutral review rating.
2. Regarding the correlational plots (the bottom row of Table 3 and 4), I don’t think I can see any clear patterns (especially on CelebA). I wonder what’s the point of including them here and if there is a point, please explain them clearly in the paper.
3. Figure 2 is also a little confusing to me. If I understand the procedure correctly, a good disentangled feature would imply smaller correlations to other features (i.e., the numbers in Figure 2 should be smaller for better disentangled features). However, looking at Figure 2 and many other plots in the appendix, I don’t think DIP-VAE has a clear win here. Is my understanding correct? If so, what exactly are you trying to convey in Figure 2?
Minor comments:
1. In Eq (6) I think there are typos in terms of the definition of Cov_q(z)(z)? It appears as only the second term in Eq (5).
2. Hyperparameter subsection in section 3: Shouldn’t \lambda_od be larger if the entanglement is mainly reflected in the off-diagonal entries? Why the opposite?
3. Can you elaborate on how a running estimate of Cov_p(x)(\mu(x)) is maintained (following Eq (6)). It’s not very clear at the current state of the paper.
4. Can we have error bars in Table 2? Some of the numbers are possibly hitting the error floor.
5. Table 5 and 6 are not very necessary, unless there is a clear point.