Published as a conference paper at ICLR 2018 SEARNN: TRAINING RNNS WITH GLOBAL-LOCAL LOSSES
We propose SEARNN, a novel training algorithm for recurrent neural networks (RNNs) inspired by the "learning to search" (L2S) approach to structured prediction. RNNs have been widely successful in structured prediction applications such as machine translation or parsing, and are commonly trained using maximum likelihood estimation (MLE). Unfortunately, this training loss is not always an appropriate surrogate for the test error: by only maximizing the ground truth probability, it fails to exploit the wealth of information offered by structured losses. Further, it introduces discrepancies between training and predicting (such as exposure bias) that may hurt test performance. Instead, SEARNN leverages test-alike search space exploration to introduce global-local losses that are closer to the test error. We first demonstrate improved performance over MLE on two different tasks: OCR and spelling correction. Then, we propose a subsampling strategy to enable SEARNN to scale to large vocabulary sizes. This allows us to validate the benefits of our approach on a machine translation task.

This paper proposes an adaptation of the SEARN algorithm to RNNs for generating text. In order to do so, they discuss various issues on how to scale the approach to large output vocabularies by sampling which actions the algorithm to explore.
Pros:
- Good literature review. But the future work on bandits is already happening:
Paper accepted at ACL 2017: Bandit Structured Prediction for Neural Sequence-to-Sequence Learning. Julia Kreutzer, Artem Sokolov, Stefan Riezler.
Cons:
- The key argument of the paper is that SEARNN is a better IL-inspired algorithm than the previously proposed ones. However there is no direct comparison either theoretical or empirical against them. In the examples on spelling using the dataset of Bahdanau et al. 2017, no comparison is made against their actor-critic method. Furthermore, given its simplicity, I would expect a comparison against scheduled sampling.
- A lot of important experimental details are in the appendices and they differ among experiments. For example, while mixed rollins are used in most experiments, reference rollins are used in MT, which is odd since it is a bad option theoretically. Also,  no details are given on how the mixing in the rollouts was tuned. Finally, in the NMT comparison while it is stated that similar architecture is used in order to compare fairly against previous work, this is not the case eventually, as it is acknowledged at least in the case of MIXER. I would have expected the same encoder-decoder architecture to have been used for all the methods considered.

- the two losses introduced are not really new. The log-loss is just MLE, only assuming that instead of a fixed expert that always returns the same target, we have a dynamic one. Note that the notion of dynamic expert is present in the SEARN paper too. Goldberg and Nivre just adapted it to transition-based dependency parsing. Similarly, since the KL loss is the same as XENT, why give it a new name?
- the top-k sampling method is essentially the same as the targeted exploration of Goodman et al. (2016) which the authors cite. Thus it is not a novel contribution.

- Not sure I see the difference between the stochastic nature of SEARNN and the online one of LOLS mentioned in section 7. They both could be mini-batched similarly. Also, not sure I see why SEARNN can be used on any task, in comparison to other methods. They all seem to be equally capable.
Minor comments:
- Figure 1: what is the difference between "cost-sensitive loss" and just "loss"?
- local vs sequence-level losses: the point in Ranzato et al and Wiseman & Rush is that the loss they optimizise (BLEU/ROUGE) do not decompose over the the predictions of the RNNs.
- Can't see why SEARNN can help with the vanishing gradient problem. Seem to be rather orthogonal.


Published as a conference paper at ICLR 2018 LEARNING PARAMETRIC CLOSED-LOOP POLICIES FOR MARKOV POTENTIAL GAMES
Multiagent systems where the agents interact among themselves and with an stochastic environment can be formalized as stochastic games. We study a subclass of these games, named Markov potential games (MPGs), that appear often in economic and engineering applications when the agents share some common resource. We consider MPGs with continuous state-action variables, coupled constraints and nonconvex rewards. Previous analysis followed a variational approach that is only valid for very simple cases (convex rewards, invertible dynamics, and no coupled constraints); or considered deterministic dynamics and provided open-loop (OL) analysis, studying strategies that consist in predefined action sequences, which are not optimal for stochastic environments. We present a closed-loop (CL) analysis for MPGs and consider parametric policies that depend on the current state and where agents adapt to stochastic transitions. We provide easily verifiable, sufficient and necessary conditions for a stochastic game to be an MPG, even for complex parametric functions (e.g., deep neural networks); and show that a closed-loop Nash equilibrium (NE) can be found (or at least approximated) by solving a related optimal control problem (OCP). This is useful since solving an OCP-which is a single-objective problem-is usually much simpler than solving the original set of coupled OCPs that form the game-which is a multiobjective control problem. This is a considerable improvement over the previously standard approach for the CL analysis of MPGs, which gives no approximate solution if no NE belongs to the chosen parametric family, and which is practical only for simple parametric forms. We illustrate the theoretical contributions with an example by applying our approach to a noncooperative communications engineering game. We then solve the game with a deep reinforcement learning algorithm that learns policies that closely approximates an exact variational NE of the game.

Summary:
This paper studies multi-agent sequential decision making problems that belong to the class of games called Markov Potential Games (MPG). It considers finding the optimal policy within a parametric space of policies, which can be represented by a function approximator such as a DNN.
A main contribution of this work is that it shows that for MPG, instead of solving a multi-objective optimization problem (Eq. 8), which is difficult, it is sufficient to solve a scalar-valued optimization problem (Eq. 16).  Theorem 1 shows that under certain conditions on the reward function, the game is MPG. It also shows how one might find the potential function J, which is used in the single objective optimization problem.
Finding J can be computationally expensive in general. So the paper provides some properties that lead to finding J easier. For example, obtaining J is easy if we have a cooperative game (Corollary 1) or the reward can be decomposed/decoupled in a certain way (Theorem 2).
Evaluation:
This is a well-written paper that studies an important problem, but I don’t think ICLR is the right venue for it. There is not much about (representation) learning in this work. The use of TRPO as an RL algorithm in the Experiment does not play a critical role in this work either. Aside this general comment, I have several other more specific comments.
- There is a significant literature on the use of RL for multi-agent systems. The paper does not do a good job comparing and positioning with respect to them. For example, refer to the following recent paper and references therein:
Perolat, Strub, et al., “Learning Nash Equilibrium for General-Sum Markov Games from Batch Data,” AISTATS, 2017.
- If I understand correctly, the policies are considered to be functions from the state of the system to a continuous action. So it is a function, and not a probability distribution. This means that the space of considered policies correspond to the space of pure strategies. We know that for some games, the Nash equilibrium is a mixed strategy. Isn’t this a big limitation of this approach?
- I am unclear how this approach can handle stochastic dynamics. For example, the optimization (P1) depends on the realization of (theta_i)_i. But this is not available. The dependence is not only in the objective, but also in the constraints, which makes things more difficult.
I understand that in the experiments the authors used two models (either the average of random realization, or solving a different optimization for each realization), but none of them is an appropriate solution for a stochastic system.
- How large is the MPG class? Is there any structural result that positions them compared to other Markov Games? For example, is the class of zero-sum games an example of MPG?
- There is a comment close to the end of Section 5 that when there is no prior knowledge of the dynamics and the reward, one can use the proposed approach to learn PCL-NE by using any DRL.
This is questionable because if the reward is not known, the conditions of Theorems 1 or 2 cannot be verifies, so it is not possible to use (P1) instead of (G2).
- What comments can you make about the computational complexity? It seems that depending on the dynamics, the optimization problem P1 can be non-convex, hence computationally difficult to solve.
- How is the work related to the following paper?
Macua, Zazo, Zazo, “Learning in Constrained Stochastic Dynamic Potential Games,” ICASSP, 2016
======
I updated the score based on the authors' rebuttal.


ACTION-DEPENDENT CONTROL VARIATES FOR POL- ICY OPTIMIZATION VIA STEIN'S IDENTITY
Policy gradient methods have achieved remarkable successes in solving challenging reinforcement learning problems. However, it still often suffers from the large variance issue on policy gradient estimation, which leads to poor sample efficiency during training. In this work, we propose a control variate method to effectively reduce variance for policy gradient methods. Motivated by the Stein's identity, our method extends the previous control variate methods used in REINFORCE and advantage actor-critic by introducing more general action-dependent baseline functions. Empirical studies show that our method significantly improves the sample efficiency of the state-of-the-art policy gradient approaches.

The paper proposes action-dependent baselines for reducing variance in policy gradient, through the derivation based on Stein’s identity and control functionals. The method relates closely to prior work on action-dependent baselines, but explores in particular on-policy fitting and a few other design choices that empirically improve the performance.
A criticism of the paper is that it does not require Stein’s identity/control functionals literature to derive Eq. 8, since it can be derived similarly to linear control variate and it has also previously been discussed in IPG [Gu et. al., 2017] as reparameterizable control variate. The derivation through Stein’s identity does not seem to provide additional insights/algorithm designs beyond direct derivation through reparameterization trick.
The empirical results appear promising, and in particular in comparison with Q-Prop, which fits Q-function using off-policy TD learning. However, the discussion on the causes of the difference should be elaborated much more, as it appears there are substantial differences besides on-policy/off-policy fitting of the Q, such as:
-FitLinear fits linear Q (through parameterization based on linearization of Q) using on-policy learning, rather than fitting nonlinear Q and then at application time linearize around the mean action. A closer comparison would be to use same locally linear Q function for off-policy learning in Q-Prop.
-The use of on-policy fitted value baseline within Q-function parameterization during on-policy fitting is nice. Similar comparison should be done with off-policy fitting in Q-Prop.
I wonder if on-policy fitting of Q can be elaborated more. Specifically, on-policy fitting of V seems to require a few design details to have best performance [GAE, Schulman et. al., 2016]: fitting on previous batch instead of current batch to avoid overfitting  (this is expected for your method as well, since by fitting to current batch the control variate then depends nontrivially on samples that are being applied), and possible use of trust-region regularization to prevent V from changing too much across iterations.
The paper presents promising results with direct on-policy fitting of action-dependent baseline, which is promising since it does not require long training iterations as in off-policy fitting in Q-Prop. As discussed above, it is encouraged to elaborate other potential causes that led to performance differences. The experimental results are presented well for a range of Mujoco tasks.
Pros:
-Simple, effective method that appears readily available to be incorporated to any on-policy PG methods without significantly increase in computational time
-Good empirical evaluation
Cons:
-The name Stein control variate seems misleading since the algorithm/method does not rely on derivation through Stein’s identity etc. and does not inherit novel insights due to this derivation.


Published as a conference paper at ICLR 2018 GLOBAL OPTIMALITY CONDITIONS FOR DEEP NEURAL NETWORKS
We study the error landscape of deep linear and nonlinear neural networks with the squared error loss. Minimizing the loss of a deep linear neural network is a nonconvex problem, and despite recent progress, our understanding of this loss surface is still incomplete. For deep linear networks, we present necessary and sufficient conditions for a critical point of the risk function to be a global minimum. Surprisingly, our conditions provide an efficiently checkable test for global optimality, while such tests are typically intractable in nonconvex optimization. We further extend these results to deep nonlinear neural networks and prove similar sufficient conditions for global optimality, albeit in a more limited function space setting.

Summary:
The paper gives theoretical results regarding the existence of local minima in the objective function of deep neural networks. In particular:
- in the case of deep linear networks, they characterize whether a critical point is a global optimum or a saddle point by a simple criterion. This improves over recent work by Kawaguchi who showed that each critical point is either a global minimum or a saddle point (i.e., none is a local minimum), by relaxing some hypotheses and adding a simple criterion to know in which case we are.
- in the case of nonlinear network, they provide a sufficient condition for a solution to be a global optimum, using a function space approach.
Quality:
The quality is very good. The paper is technically correct and nontrivial. All proofs are provided and easy to follow.
Clarity:
The paper is very clear. Related work is clearly cited, and the novelty of the paper well explained. The technical proofs of the paper are in appendices, making the main text very smooth.
Originality:
The originality is weak. It extends a series of recent papers correctly cited. There is some originality in the proof which differs from recent related papers.
Significance:
The result is not completely surprising, but it is significant given the lack of theory and understanding of deep learning. Although the model is not really relevant for deep networks used in practice, the main result closes a question about characterization of critical points in simplified models if neural network, which is certainly interesting for many people.


Published as a conference paper at ICLR 2018 WRPN: WIDE REDUCED-PRECISION NETWORKS
For computer vision applications, prior works have shown the efficacy of reducing numeric precision of model parameters (network weights) in deep neural networks. Activation maps, however, occupy a large memory footprint during both the training and inference step when using mini-batches of inputs. One way to reduce this large memory footprint is to reduce the precision of activations. However, past works have shown that reducing the precision of activations hurts model accuracy. We study schemes to train networks from scratch using reduced-precision activations without hurting accuracy. We reduce the precision of activation maps (along with model parameters) and increase the number of filter maps in a layer, and find that this scheme matches or surpasses the accuracy of the baseline full-precision network. As a result, one can significantly improve the execution efficiency (e.g. reduce dynamic memory footprint, memory bandwidth and computational energy) and speed up the training and inference process with appropriate hardware support. We call our scheme WRPN -wide reducedprecision networks. We report results and show that WRPN scheme is better than previously reported accuracies on ILSVRC-12 dataset while being computationally less expensive compared to previously reported reduced-precision networks.

This is a well-written paper with good comparisons to a number of earlier approaches. It focuses on an approach to get similar accuracy at lower precision, in addition to cutting down the compute costs. Results with 2-bit activations and 4-bit weights seem to match baseline accuracy across the models listed in the paper.
Originality
This seems to be first paper that consistently matches baseline results below int-8 accuracy, and shows a promising future direction.
Significance
Going down to below 8-bits and potentially all the way down to binary (1-bit weights and activations) is a promising direction for future hardware design. It has the potential to give good results at lower compute and more significantly in providing a lower power option, which is the biggest constraint for higher compute today.
Pros:
- Positive results with low precision (4-bit, 2-bit and even 1-bit)
- Moving the state of the art in low precision forward
- Strong potential impact, especially on constrained power environments (but not limited to them)
- Uses same hyperparameters as original training, making the process of using this much simpler.
Cons/Questions
- They mention not quantizing the first and last layer of every network. How much does that impact the overall compute?
- Is there a certain width where 1-bit activation and weights would match the accuracy of the baseline model? This could be interesting for low power case, even if the "effective compute" is larger than the baseline.


Published as a conference paper at ICLR 2018 LATENT SPACE ODDITY: ON THE CURVATURE OF DEEP GENERATIVE MODELS
Deep generative models provide a systematic way to learn nonlinear data distributions through a set of latent variables and a nonlinear "generator" function that maps latent points into the input space. The nonlinearity of the generator implies that the latent space gives a distorted view of the input space. Under mild conditions, we show that this distortion can be characterized by a stochastic Riemannian metric, and we demonstrate that distances and interpolants are significantly improved under this metric. This in turn improves probability distributions, sampling algorithms and clustering in the latent space. Our geometric analysis further reveals that current generators provide poor variance estimates and we propose a new generator architecture with vastly improved variance estimates. Results are demonstrated on convolutional and fully connected variational autoencoders, but the formalism easily generalizes to other deep generative models.

The paper makes an important observation: the generating function of a generative model (deep or not) induces a (stochastic) Riemannian metric tensor on the latent space. This metric might be the correct way to measure distances in the latent space, as opposed to the Euclidean distance.
While this seems obvious, I had actually always thought of the latent space as "unfolding" the data manifold as it exists in the output space. The authors propose a different view which is intriguing; however, they do not, to the best of my understand, give a definitive theoretical reason why the induced Riemannian metric is the correct choice over the Euclidean metric.
The paper correctly identifies an important problem with the way most deep generative models evaluate variance. However the solution proposed seems ad-hoc and not particularly related to the other parts of the paper. While the proposed variance estimation (using RBF networks) might work in some cases, I would love to see (perhaps in future work) a much more rigorous treatment of the subject.
Pros:
1. Interesting observation and mathematical development of a Riemannian metric on the latent space.
2. Good observation about the different roles of the mean and the variance in determining the geodesics: they tend to avoid areas of high variance.
3. Intriguing experiments and a good effort at visualizing and explaining them. I especially appreciate the interpolation and random walk experiments. These are hard to evaluate objectively, but the results to hint at the phenomena the authors describe when comparing Euclidean to Riemannian metrics in the latent space.
Cons:
1. The part of the paper proposing new variance estimators is ad-hoc and is not experimented with rigorously, comparing it to other methods in terms of calibration for example.
Specific comments:
1. To the best of my understanding eq. (2) does not imply that the natural distance in Z is locally adaptive. I think of eq (2) as *defining* a type of distance on Z, that may or may not be natural. One could equally argue that the Euclidean distance on z is natural, and that this distance is then pushed forward by f to some induced distance over X.
2. In the definition of paths \gamma, shouldn't they be parametrized by arc-length (also known as unit-speed)? How should we think of the curve \gamma(t^2) for example?
3. In Theorem 2, is the term "input dimension" appropriate? Perhaps "data dimension" is better?
4. I did not fully understand the role of the LAND model. Is this a model fit AFTER fitting the generative model, and is used to cluster Z like a GMM ? I would appreciate a clarification about the context of this model.


Published as a conference paper at ICLR 2018 LEARNING INTRINSIC SPARSE STRUCTURES WITHIN LONG SHORT-TERM MEMORY
Model compression is significant for the wide adoption of Recurrent Neural Networks (RNNs) in both user devices possessing limited resources and business clusters requiring quick responses to large-scale service requests. This work aims to learn structurally-sparse Long Short-Term Memory (LSTM) by reducing the sizes of basic structures within LSTM units, including input updates, gates, hidden states, cell states and outputs. Independently reducing the sizes of basic structures can result in inconsistent dimensions among them, and consequently, end up with invalid LSTM units. To overcome the problem, we propose Intrinsic Sparse Structures (ISS) in LSTMs. Removing a component of ISS will simultaneously decrease the sizes of all basic structures by one and thereby always maintain the dimension consistency. By learning ISS within LSTM units, the obtained LSTMs remain regular while having much smaller basic structures. Based on group Lasso regularization, our method achieves 10.59× speedup without losing any perplexity of a language modeling of Penn TreeBank dataset. It is also successfully evaluated through a compact model with only 2.69M weights for machine Question Answering of SQuAD dataset. Our approach is successfully extended to non-LSTM RNNs, like Recurrent Highway Networks (RHNs). Our source code is available 1 .

The authors propose a technique to compress LSTMs in RNNs by using a group Lasso regularizer which results in structured sparsity, by eliminating individual hidden layer inputs at a particular layer. The authors present experiments on unidirectional and bidirectional LSTM models which demonstrate the effectiveness of this method. The proposed techniques are evaluated on two models: a fairly large LSTM with ~66.0M parameters, as well as a more compact LSTM with ~2.7M parameters, which can be sped up significantly through compression.
Overall this is a clearly written paper that is easy to follow, with experiments that are well motivated. To the best of my knowledge most previous papers in the area of RNN compression focus on pruning or compression of the node outputs/connections, but do not focus as much on reducing the computation/parameters within an RNN cell. I only have a few minor comments/suggestions which are listed below:
1. It is interesting that the model structure where the number of parameters is reduced to the number of ISSs chosen from the proposed procedure does not attain the same performance as when training with a larger number of nodes, with the group lasso regularizer. It would be interesting to conduct experiments for a range of \lambda values: i.e., to allow for different degrees of compression, and then examine whether the model trained from scratch with the “optimal” structure achieves performance closer to the ISS-based strategy, for example, for smaller amounts of compression, this might be the case?
2. In the experiment, the authors use a weaker dropout when training with ISS. Could the authors also report performance for the baseline model if trained with the same dropout (but without the group LASSO regularizer)?
3. The colors in the figures: especially the blue vs. green contrast is really hard to see. It might be nicer to use lighter colors, which are more distinct.
4. The authors mention that the thresholding operation to zero-out weights based on the hyperparameter \tau is applied “after each iteration”. What is an iteration in this context? An epoch, a few mini-batch updates, per mini-batch? Could the authors please clarify.
5. Clarification about the hyperparameter \tau used for sparsification: Is \tau determined purely based on the converged weight values in the model when trained without the group LASSO constraint? It would be interesting to plot a histogram of weight values in the baseline model, and perhaps also after the group LASSO regularized training.
6. Is the same value of \lambda used for all groups in the model? It would be interesting to consider the effect of using stronger sparsification in the earlier layers, for example.
7. Section 4.2: Please explain what the exact match (EM) and F1 metrics used to measure performance of the BIDAF model are, in the text.
Minor Typographical/Grammatical errors:
- Sec 1: “... in LSTMs meanwhile maintains the dimension consistency.” → “... in LSTMs while maintaining the dimension consistency.”
- Sec 1: “... is public available” → “is publically available”
- Sec 2: Please rephrase: “After learning those structures, compact LSTM units remain original structural schematic but have the sizes reduced.”
- Sec 4.1: “The exactly same training scheme of the baseline ...” → “The same training scheme as the baseline ...”


Generative image models have made significant progress in the last few years, and are now able to generate low-resolution images which sometimes look realistic. However the state-of-theart models utilize fully entangled latent representations where small changes to a single neuron can effect every output pixel in relatively arbitrary ways, and different neurons have possibly arbitrary relationships with each other. This limits the ability of such models to generalize to new combinations or orientations of objects as well as their ability to connect with more structured representations such as natural language, without explicit strong supervision.
In this work explore the synergistic effect of using partial natural language scene descriptions to help disentangle the latent entities visible an image. We present a novel neural network architecture called Generative Entity Networks, which jointly generates both the natural language descriptions and the images from a set of latent entities. Our model is based on the variational autoencoder framework and makes use of visual attention to identify and characterise the visual attributes of each entity. Using the Shapeworld dataset, we show that our representation both enables a better generative model of images, leading to higher quality image samples, as well as creating more semantically useful representations that improve performance over purely dicriminative models on a simple natural language yes/no question answering task.

**Summary**
The paper proposes an extension of the attend, infer, repeat generative model of Eslami, 2016 and extends it to handle ``visual attribute descriptions. This straightforward extension is claimed to improve image quality and shown to improve performance on a previously introduced image caption ranking task. In general, the paper shows improvements on an image caption agreement task introduced in Kuhnle and Copestake, 2017.  The paper seems to have weaknesses pertaining to the approach taken, clarity of presentation and comparison to baselines which mean that the paper does not seem to meet the acceptance threshold for ICLR. See more detailed points below in Weaknesses.
**Strengths**
I like the high-level motivation of the work, that one needs to understand and establish that language or semantics can help learn better representations for images. I buy the premise and think the work addresses an important issue.
**Weakness**
Approach:
* A major limitation of the model seems to be that one needs access to both images and attribute vectors at inference time to compute representations which is a highly restrictive assumption (since inference networks are discriminative). The paper should explain how/if one can compute representations given just the image, for instance, say by not using amortized inference. The paper does propose to use an image-only encoder but that is intended in general as a modeling choice to explain statistics which are not captured by the attributes (in this case location and orientation as explained in the Introduction of the paper).
Clarity:
* Eqn. 5, LHS can be written more clearly as \hat{a}_k.
* It would also be good to cite the following related work, which closely ties into the model of Eslami 2016, and is prior work:
Efficient inference in occlusion-aware generative models of images,
Jonathan Huang, Kevin Murphy.
ICLR Workshops, 2016
* It would be good to clarify that the paper is focusing on the image caption agreement task from Kuhnle and Copestake, as opposed to generic visual question answering.
* The claim that the paper works with natural language should be toned down and clarified. This is not natural language, firstly because the language in the dataset is synthetically generated and not “natural”. Secondly, the approach parses this “synthetic” language into structured tuples which makes it even less natural. Also, Page. 3. What does “partial descriptions” mean?
* Section 3: It would be good to explicitly draw out the graphical model for the proposed approach and clarify how it differs from prior work (Eslami, 2016).
* Sec. 3. 4 mentions that the “only image” encoder is used to obtain the representation for the image, but the “only image” encoder is expected to capture the “indescribable component” from the image, then how is the attribute information from the image captured in this framework? One cannot hope to do image caption association prediction without capturing the image attributes...
*, In general, the writing and presentation of the model seem highly fragmented, and it is not clear what the specifics of the overall model are. For instance, in the decoder, the paper mentions for the first time that there are variables “z”, but does not mention in the encoder how the variables “z” were obtained in the first place (Sec. 3.1). For instance, it is also not clear if the paper is modeling variable length sequences in a similar manner to Eslami, 2016 or not, and if this work also has a latent variable [z, z_pres] at every timestep which is used in a similar manner to Eqn. 2 in Eslami, 2016. Sec. 3.4 “GEN Image Encoder” has some typo, it is not clear what the conditioning is within q(z) term.
* Comparison to baselines:
  1. How well does this model do against a baseline discriminative image caption ranking approach, similar to [D]? This seems like an important baseline to report for the image caption ranking task.
  2. Another crucial baseline is to train the Attend, Infer, Repeat model on the ShapeWorld images, and then take the latent state inferred at every step by that model, and use those features instead of the features described in Sec. 3.4 “Gen Image Encoder” and repeat the rest of the proposed pipeline. Does the proposed approach still show gains over Attend Infer Repeat?
  3. The results shown in Fig. 7 are surprising -- in general, it does not seem like a regular VAE would do so poorly. Are the number of parameters in the proposed approach and the baseline VAE similar? Are the choices of decoder etc. similar? Did the model used for drawing Fig. 7 converge? Would be good to provide its training curve. Also, it would be good to evaluate the AIR model from Eslami, 2016 on the same simple shapes dataset and show unconditional samples. If the claim from the work is true, that model should be just as bad as a regular VAE and would clearly establish that using language is helping get better image samples.
* Page 2: In general the notion of separating the latent space into content and style, where we have labels for the “content” is an old idea that has appeared in the literature and should be cited accordingly. See [B] for an earlier treatment, and an extension by [A]. See also the Bivcca-private model of [C] which has “private” latent variables for vision similar to this work (this is relevant to Sec. 3.2.)
References:
[A]: Upchurch, Paul, Noah Snavely, and Kavita Bala. 2016. “From A to Z: Supervised Transfer of Style and Content Using Deep Neural Network Generators.” arXiv [cs.CV]. arXiv. http://arxiv.org/abs/1603.02003.
[B]: Kingma, Diederik P., Danilo J. Rezende, Shakir Mohamed, and Max Welling. 2014. “Semi-Supervised Learning with Deep Generative Models.” arXiv [cs.LG]. arXiv. http://arxiv.org/abs/1406.5298.
[C]: Wang, Weiran, Xinchen Yan, Honglak Lee, and Karen Livescu. 2016. “Deep Variational Canonical Correlation Analysis.” arXiv [cs.LG]. arXiv. http://arxiv.org/abs/1610.03454.
[D]: Kiros, Ryan, Ruslan Salakhutdinov, and Richard S. Zemel. 2014. “Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models.” arXiv [cs.LG]. arXiv. http://arxiv.org/abs/1411.2539.


Many anomaly detection methods exist that perform well on low-dimensional problems however there is a notable lack of effective methods for highdimensional spaces, such as images. Inspired by recent successes in deep learning we propose a novel approach to anomaly detection using generative adversarial networks. Given a sample under consideration, our method is based on searching for a good representation of that sample in the latent space of the generator; if such a representation is not found, the sample is deemed anomalous. We achieve stateof-the-art performance on standard image benchmark datasets and visual inspection of the most anomalous samples reveals that our method does indeed return anomalies.

In the paper, the authors proposed using GAN for anomaly detection.
In the method, we first train generator g_\theta from a dataset consisting of only healthy data points.
For evaluating whether the data point x is anomalous or not, we search for a latent representation z such that x \approx g_\theta(z).
If such a representation z could be found, x is deemed to be healthy, and anomalous otherwise.
For searching z, the authors proposed a gradient-descent based method that iteratively update z.
Moreover, the authors proposed updating the parameter \theta of the generator g_\theta.
The authors claimed that this parameter update is one of the novelty of their method, making it different from the method of Schlegl et al. (2017).
In the experiments, the authors showed that the proposed method attained the best AUC on MNIST and CIFAR-10.
In my first reading of the paper, I felt that the baselines in the experiments are too primitive.
Specifically, for KDE and OC-SVM, a naive PCA is used to reduce the data dimension.
Nowadays, there are several publicly available CNNs that are trained on large image datasets such as ImageNet.
Then, one can use such CNNs as feature extractor, that will give better low dimensional expression of the data than the naive PCA.
I believe that the performances of KDE and OC-SVM can be improved by using such feature extractors.
Additionally, I found that some well-known anomaly detection methods are excluded from the comparison.
In Emmott et al. (2013), which the authors referred as a related work, it was reported that Isolation Forest and Ensemble of GMMs performed well on several datasets (better than KDE and OC-SVM).
It would be essential to add these methods as baselines to be compared with the proposed method.
Overall, I think the experimental results are far from satisfactory.
### Response to Revision ###
It is interesting to see that the features extracted from AlexNet are not helpful for anomaly detection.
It would be interesting to see whether features extracted from middle layers are helpful or they are still useless.
I greatly appreciate the authors for their extensive experiments as a response to my comments.
However, I have decided to keep my score unchanged, as the additional experiments have shown that the performance of the proposed method is not significantly better than the other methods.
In particular, in MNIST, GMM performed better.


Published as a conference paper at ICLR 2018 MITIGATING ADVERSARIAL EFFECTS THROUGH RAN- DOMIZATION
Convolutional neural networks have demonstrated high accuracy on various tasks in recent years. However, they are extremely vulnerable to adversarial examples. For example, imperceptible perturbations added to clean images can cause convolutional neural networks to fail. In this paper, we propose to utilize randomization at inference time to mitigate adversarial effects. Specifically, we use two randomization operations: random resizing, which resizes the input images to a random size, and random padding, which pads zeros around the input images in a random manner. Extensive experiments demonstrate that the proposed randomization method is very effective at defending against both single-step and iterative attacks. Our method provides the following advantages: 1) no additional training or fine-tuning, 2) very few additional computations, 3) compatible with other adversarial defense methods. By combining the proposed randomization method with an adversarially trained model, it achieves a normalized score of 0.924 (ranked No.2 among 107 defense teams) in the NIPS 2017 adversarial examples defense challenge, which is far better than using adversarial training alone with a normalized score of 0.773 (ranked No.56). The code is public available at https: //github.com/cihangxie/NIPS2017_adv_challenge_defense.

The authors propose a simple defense against adversarial attacks, which is to add randomization in the input of the CNNs. They experiment with different CNNs and published adversarial training techniques and show that randomized inputs mitigate adversarial attacks.
Pros:
(+) The idea introduced is simple and flexible to be used for any CNN architecture
(+) Experiments on ImageNet1k prove demonstrate its effectiveness
Cons:
(-) Experiments are not thorougly explained
(-) Novelty is extremely limited
(-) Some baselines missing
The experimental section of the paper was rather confusing. The authors should explain the experiments and the settings in the table, as those are not very clear. In particular, it was not clear whether the defense model was trained with the input randomization layers? Also, in Tables 1-6, how was the target model trained? How do the training procedures of target vs. defense model differ? In those tables, what is the testing procedure for the target model and how does it compare to the defense model?
The gap between the target and defense model in Table 4 (ensemble pattern attack scenario) shrinks for single step attack methods. This means that when the attacker is aware of the randomization parameters, the effect of randomization might diminish. A baseline that reports the performance when the attacker is fully aware of the randomization of the defender (parameters, patterns etc.) is missing but is very useful.
While the experiments show that the randomization layers mitigate the effect of randomization attacks, it's not clear whether the effectiveness of this very simple approach is heavily biased towards the published ways of generating adversarial attacks and the particular problem (i.e. classification). The form of attacks studied in the paper is that of additive noise. But there is many types of attacks that could be closely related to the randomization procedure of the input and that could lead to very different results.


We propose a novel unsupervised representation learning framework called neighbor-encoder in which domain knowledge can be trivially incorporated into the learning process without modifying the general encoder-decoder architecture. In contrast to autoencoder, which reconstructs the input data, neighbor-encoder reconstructs the input data's neighbors. The proposed neighbor-encoder can be considered as a generalization of autoencoder as the input data can be treated as the nearest neighbor of itself with zero distance. By reformulating the representation learning problem as a neighbor reconstruction problem, domain knowledge can be easily incorporated with appropriate definition of similarity or distance between objects. As such, any existing similarity search algorithms can be easily integrated into our framework. Applications of other algorithms (e.g., association rule mining) in our framework is also possible since the concept of "neighbor" is an abstraction which can be appropriately defined differently in different contexts. We have demonstrated the effectiveness of our framework in various domains, including images, time series, music, etc., with various neighbor definitions. Experimental results show that neighbor-encoder outperforms autoencoder in most scenarios we considered.

This paper presents a variant of auto-encoder that relaxes the decoder targets to be neighbors of a data point. Different from original auto-encoder, where data point x and the decoder output \hat{x} are forced to be close, the neighbor-encoder encourage the decoder output to be similar to the neighbors of the input data point. By considering the neighbor information, the decoder targets would have smaller intra-class distances, thus larger inter-class distances, which helps to learn better separated latent representation of data in terms of data clusters. The authors conduct experiments on several real but relative small-scale data sets, and demonstrate the improvements of learned latent representations by using neighbors as targets.
The method of neighbor prediction is a simple and small modification of the original auto-encoder, but seems to provide a way to augment the targets such that intra-class distance of decoder targets can be tightened. Improvements in the conducted experiments seem significant compared to the most basic auto-encoder.
Major issues:
There are some unaddressed theoretical questions. The optimal solution to predict the set of neighbor points in mean-squared metric is to predict the average of those points, which is not well justified as the averaged image can easily fall off the data manifold. This may lead to a more blurry reconstruction when k increases, despite the intra-class targets are tight. It can also in turn harm the latent representation when euclidean neighbors are not actually similar (e.g. images in cifar10/imagenet that are not as simple as 10 digits). This seems to be a defect of the neighbor-encoder method and is not discussed in the paper.
The data sets used in the experiments  are relatively small and simple, larger-scale experiments should be conducted. The fluctuations in Figure 9 and 10 suggest the significant variances in the results. Also, more complicated data/images can decrease the actual similarities of euclidean neighbors, thus affecting the results.
The baselines are weak. Only the most basic auto-encoder is compared, no additional variants or other data augmentation techniques are compared. It is possible other variants improve the basic auto-encoder in similar ways.
Some results are not very well explained. It seems the performance increases monotonically as the number of neighbors increases (Figure 5, 9, 10). Will this continue or when will the performance decrease? I would expect it to decrease as the far away neighbors will be dissimilar. The authors can either attach the nearest neighbors figures or their statistics, and provide explanations on when and why the performance decrease is expected.
Some notations are confusing and need to be improved. For example, X and Y are actually the same set of images, the separation is a bit confusing; y_i \in y in last paragraph of page 4 is incorrect, should use something like y_i in N(y).


Convolutional neural networks (CNN) have become the most successful and popular approach in many vision-related domains. While CNNs are particularly wellsuited for capturing a proper hierarchy of concepts from real-world images, they are limited to domains where data is abundant. Recent attempts have looked into mitigating this data scarcity problem by casting their original single-task problem into a new multi-task learning (MTL) problem. The main goal of this inductive transfer mechanism is to leverage domain-specific information from related tasks, in order to improve generalization on the main task. While recent results in the deep learning (DL) community have shown the promising potential of training task-specific CNNs in a soft parameter sharing framework, integrating the recent DL advances for improving knowledge sharing is still an open problem. In this paper, we propose the Deep Collaboration Network (DCNet), a novel approach for connecting task-specific CNNs in a MTL framework. We define connectivity in terms of two distinct non-linear transformation blocks. One aggregates taskspecific features into global features, while the other merges back the global features with each task-specific network. Based on the observation that task relevance depends on depth, our transformation blocks use skip connections as suggested by residual network approaches, to more easily deactivate unrelated task-dependent features. To validate our approach, we employed facial landmark detection (FLD) datasets as they are readily amenable to MTL, given the number of tasks they include. Experimental results show that we can achieve up to 24.31% relative improvement in landmark failure rate over other state-of-the-art MTL approaches. We finally perform an ablation study showing that our approach effectively allows knowledge sharing, by leveraging domain-specific features at particular depths from tasks that we know are related.

This paper proposes a multi-pathway neural network for facial landmark detection with multitask learning. In particular, each pathway corresponds to one task, and the intermediate features are fused at multiple layers. The fused features are added to the task-specific pathway using a residual connection (the input of the residual connection are the concatenation of the task-specific features and the fuse features). The residual connection allows each pathway to selectively use the information from other pathways and focus on its own task.
This paper is well written. The proposed neural network architectures are reasonable.
The residual connection can help each pathway to focus on its own task (suggested by Figure 8). This phenomenon is not guaranteed by the training objective but happens automatically due to the architecture, which is interesting.
The proposed model outperforms several baseline models. On MTFL, when using the AlexNet, the improvement is significant; when using the ResNet18, the improvement is encouraging but not so significant. On AFLW (trained on MTFL), the improvements are significant in both cases.
What is missing is the comparison with other methods (besides the baseline). For examples, it will be helpful to compare with existing non-multitask learning methods, like TCDCN (Zhang et al., 2014) (it seems to achieve 25% failure rate on AFLW, which is lower than the numbers in Figure 5), and  multi-task learning method, like MTCNN (Zhang et al., 2016). It is important to show that proposed multitask learning method is useful in practice.
In addition, many papers take the average error as the performance metric. Providing results in the average error can make the experiments more comprehensive.
The proposed architecture is a bit huge. It scales linearly with the number of tasks, which is not quite preferable. It is also not straightforward to add new tasks to finetune a trained model.
In Figure 5 (left), it is a bit weird that the pretrained model underperforms the nonpretrained one.
I am likely to change the rating based on the comparison with other methods.


Recently, researchers have discovered that the state-of-the-art object classifiers can be fooled easily by small perturbations in the input unnoticeable to human eyes. It is known that an attacker can generate strong adversarial examples if she knows the classifier parameters. Conversely, a defender can robustify the classifier by retraining if she has the adversarial examples. The cat-and-mouse game nature of attacks and defenses raises the question of the presence of equilibria in the dynamics. In this paper, we present a neural-network based attack class to approximate a larger but intractable class of attacks, and formulate the attacker-defender interaction as a zero-sum leader-follower game. We present sensitivity-penalized optimization algorithms to find minimax solutions, which are the best worst-case defenses against whitebox attacks. Advantages of the learning-based attacks and defenses compared to gradient-based attacks and defenses are demonstrated with MNIST and CIFAR-10.

The authors describe a mechanism for defending against adversarial learning attacks on classifiers. They first consider the dynamics generated by the following procedure. They begin by training a classifier, generating attack samples using FGSM, then hardening the classifier by retraining with adversarial samples, generating new attack samples for the retrained classifier, and repeating.
They next observe that since FGSM is given by a simple perturbation of the sample point by the gradient of the loss, that the fixed point of the above dynamics can be optimized for directly using gradient descent. They call this approach Sens FGSM, and evaluate it empirically against the various iterates of the above approach.
They then generalize this approach to an arbitrary attacker strategy given by some parameter vector (e.g. a neural net for generating adversarial samples). In this case, the attacker and defender are playing a minimax game, and the authors propose finding the minimax (or maximin) parameters using an algorithm which alternates between maximization and minimization gradient steps. They conclude with empirical observations about the performance of this algorithm.
The paper is well-written and easy to follow. However, I found the empirical results to be a little underwhelming. Sens-FGSM outperforms the adversarial training defenses tuned for the “wrong” iteration, but it does not appear to perform particularly well with error rates well above 20%. How does it stack up against other defense approaches (e.g. https://arxiv.org/pdf/1705.09064.pdf)? Furthermore, what is the significance of FGSM-curr (FGSM-81) for Sens-FGSM? It is my understanding that Sens-FGSM is not trained to a particular iteration of the “cat-and-mouse” game. Why, then, does Sens-FGSM provide a consistently better defense against FGSM-81? With regards to the second part of the paper, using gradient methods to solve a minimax problem is not especially novel (i.e. Goodfellow et al.), thus I would liked to see more thorough experiments here as well. For example, it’s unlikely that the defender would ever know the attack network utilized by an attacker. How robust is the defense against samples generated by a different attack network? The authors seem to address this in section 5 by stating that the minimax solution is not meaningful for other network classes. However, this is a bit unsatisfying. Any defense can be *evaluated* against samples generated by any attacker strategy. Is it the case that the defenses fall flat against samples generated by different architectures?
Minor Comments:
Section 3.1, First Line. ”f(ul(g(x),y))” appears to be a mistake.


Truncated Backpropagation Through Time (truncated BPTT, Jaeger (2005)) is a widespread method for learning recurrent computational graphs. Truncated BPTT keeps the computational benefits of Backpropagation Through Time (BPTT Werbos  (1990)) while relieving the need for a complete backtrack through the whole data sequence at every step. However, truncation favors short-term dependencies: the gradient estimate of truncated BPTT is biased, so that it does not benefit from the convergence guarantees from stochastic gradient theory. We introduce Anticipated Reweighted Truncated Backpropagation (ARTBP), an algorithm that keeps the computational benefits of truncated BPTT, while providing unbiasedness. ARTBP works by using variable truncation lengths together with carefully chosen compensation factors in the backpropagation equation. We check the viability of ARTBP on two tasks. First, a simple synthetic task where careful balancing of temporal dependencies at different scales is needed: truncated BPTT displays unreliable performance, and in worst case scenarios, divergence, while ARTBP converges reliably. Second, on Penn Treebank character-level language modelling Mikolov et al. (2012), ARTBP slightly outperforms truncated BPTT.
Backpropagation Through Time (BPTT) Werbos (1990) is the de facto standard for training recurrent neural networks. However, BPTT has shortcomings when it comes to learning from very long sequences: learning a recurrent network with BPTT requires unfolding the network through time for as many timesteps as there are in the sequence. For long sequences this represents a heavy computational and memory load. This shortcoming is often overcome heuristically, by arbitrarily splitting the initial sequence into subsequences, and only backpropagating on the subsequences. The resulting algorithm is often referred to as Truncated Backpropagation Through Time (truncated BPTT, see for instance Jaeger (2005)). This comes at the cost of losing long term dependencies.
We introduce Anticipated Reweighted Truncated BackPropagation (ARTBP), a variation of truncated BPTT designed to provide an unbiased gradient estimate, accounting for long term dependencies. Like truncated BPTT, ARTBP splits the initial training sequence into subsequences, and only backpropagates on those subsequences. However, unlike truncated BPTT, ARTBP splits the training sequence into variable size subsequences, and suitably modifies the backpropagation equation to obtain unbiased gradients.
Unbiasedness of gradient estimates is the key property that provides convergence to a local optimum in stochastic gradient descent procedures. Stochastic gradient descent with biased estimates, such as the one provided by truncated BPTT, can lead to divergence even in simple situations and even with large truncation lengths (Fig. 3).
ARTBP is experimentally compared to truncated BPTT. On truncated BPTT failure cases, typically when balancing of temporal dependencies is key, ARTBP achieves reliable convergence thanks to unbiasedness. On small-scale but real world data, ARTBP slightly outperforms truncated BPTT on the test case we examined.
ARTBP formalizes the idea that, on a day-to-day basis, we can perform short term optimization, but must reflect on long-term effects once in a while; ARTBP turns this into a provably unbiased overall gradient estimate. Notably, the many short subsequences allow for quick adaptation to the data, while preserving overall balance. 1 RELATED WORK BPTT Werbos (1990) and its truncated counterpart Jaeger (2005) are nearly uncontested in the recurrent learning field. Nevertheless, BPTT is hardly applicable to very long training sequences, as it requires storing and backpropagating through a network with as many layers as there are timesteps Sutskever (2013). Storage issues can be partially addressed as in Gruslys et al. (2016), but at an increased computational cost. Backpropagating through very long sequences also implies performing fewer gradient descent steps, which significantly slows down learning Sutskever (2013).
Truncated BPTT heuristically solves BPTT deficiencies by chopping the initial sequence into evenly sized subsequences. Truncated BPTT truncates gradient flows between contiguous subsequences, but maintains the recurrent hidden state of the network. Truncation biases gradients, removing any theoretical convergence guarantee. Intuitively, truncated BPTT has trouble learning dependencies above the range of truncation.

This is an interesting paper.
It is well known that TBPTT is biased because of a fixed truncation length. The authors propose to make it  unbiased by sampling different truncation lengths and hence changing  the optimization procedure which corresponds to adding noise in the gradient estimates which leads to  unbiased gradients.
Pros:
- Its a well written and easy to follow paper.
- If I understand correctly, they are changing the optimization procedure so that the proposed approach is able to find a local minima, which was not possible by using truncated backpropagation through time.
- Its interesting to see in there PTB results that they get better validation score as compared to truncated BPTT.
Cons:
- Though the approach is interesting, the results are quite preliminary. And given the fact there results are worse than the LSTM baseline (1.40 v/s 1.38). The authors note that it might be because of they are applying without sub-sequence shuffling.
- I'm not convinced of the approach yet. The authors could do some large scale experiments on datasets like Text8 or speech modelling.
Few points
- If I'm correct that the proposed approach indeed changes the optimization procedure, than I'd like to know what the authors think about exposure bias issue. Its a well known[1, 2] that we can't sample from RNN's for more number of steps, than what we used for trained (difference b/w teacher forcing and free running RNN). I'd like to know how does there method perform in such a regime (where you sample for more number of steps than you have trained for)
- Another thing, I'd like to see is the results of this model as compared to truncated backpropagation when you increase the sequence length. For example, Lets say you are doing language modelling on PTB, how the result varies when you change the length of the input sequence. I'd like to see a graph where on X axis is the length of the input sequence and on the Y axis is the bpc score (for PTB) and how does it compare to truncated backpropagation through time.
-  PTB dataset has still not very long term dependencies, so I'm curious what the authors think about using there method for something like speech modelling or some large scale experiments.
- I'd expect the proposed approach to be more computationally expensive as compared to Truncated Back-propagation through time. I dont think the authors mentioned this somewhere in the paper. How much time does a single update takes as compared to Truncated Back-propagation through time ?
- Does the proposed approach help in flow of gradients?
- In practice for training RNN's people use gradient clipping which also makes the gradient biased. Can the proposed method be used for training longer sequences?
[1] Scheduled Sampling For Sequence Prediction with RNN's https://arxiv.org/abs/1506.03099
[2] Professor Forcing  https://arxiv.org/abs/1610.09038
Overall, Its an interesting paper which requires some more analysis to be published in this conference. I'd be very happy to increase my score if the authors can provide me results what I have asked for.


INTERPRETABLE COUNTING FOR VISUAL QUESTION ANSWERING
Questions that require counting a variety of objects in images remain a major challenge in visual question answering (VQA). The most common approaches to VQA involve either classifying answers based on fixed length representations of both the image and question or summing fractional counts estimated from each section of the image. In contrast, we treat counting as a sequential decision process and force our model to make discrete choices of what to count. Specifically, the model sequentially selects from detected objects and learns interactions between objects that influence subsequent selections. A distinction of our approach is its intuitive and interpretable output, as discrete counts are automatically grounded in the image. Furthermore, our method outperforms the state of the art architecture for VQA on multiple metrics that evaluate counting.

------------------
Summary:
------------------
This work introduces a discrete and interpretable model for answering visually grounded counting questions. The proposed model executes a sequential decision process in which it 1) selects an image region to "add to the count" and then 2) updates the likelihood of selecting other regions based on their relationships (defined broadly) to the selected region. After substantial module pre-trianing, the model is trained end-to-end with the REINFORCE policy gradient method (with the recently proposed self-critical sequence training baseline). Compared to existing approaches for counting (or VQA in general), this approach not only produces lower error but also provides a more human-intuitive discrete, instance-pointing representation of counting.
-----------------------
Preliminary Evaluation:
-----------------------
The paper presents an interesting approach that seems to outperform existing methods. More importantly in my view, the model treats counting as a discrete human-intuitive process. The presentation and experiments are okay overall but I have a few questions and requests below that I feel would strengthen the submission.
------------------
Strengths:
------------------
- I generally agree with the authors that approaching counting as a region-set selection problem provides an interpretable and human-intuitive methodology that seems more appropriate than attentional or monolithic approaches.
- To the best of my knowledge, the writing does a good job of placing the work in the context of existing literature.
- The dataset construction is given appropriate attention to restrict its instances to counting questions and will be made available to the public.
- The model outperforms existing approaches given the same visual and linguistic inputs / encodings. While I find improvements in RMSE a bit underwhelming, I'm still generally positive about the results given the improved accuracy and human-intuitiveness of the grounded outputs.
- I appreciated the analysis of the effect of "commonness" and think it provides interesting insight into the generalization of the proposed model.
- Qualitative examples are interesting.
------------------
Weaknesses:
------------------
- There is a lot going on in this paper as far as model construction and training procedures go. In its current state, many of the details are pushed to the supplement such that the main paper would be insufficient for replication. The authors also do not promise code release.
- Maybe it is just my unfamiliarity with it, but the caption grounding auxiliary-task feels insufficiently introduced in the main paper.  I also find it a bit discouraging that the details of joint training is regulated to the supplementary material, especially given that the UpDown is not using it! I would like to see an ablation of the proposed model without joint training.
- Both the IRLC and SoftCount models are trained with objectives that are aware of the ordinal nature of the output space (such that predicting 2 when the answer is 20 is worse than predicting 19). Unfortunately, the UpDown model is trained with cross-entropy and lacks access to this notion. I believe that this difference results in the large gap in RMSE between IRLC/SoftCount and UpDown. Ideally an updated version of UpDown trained under an order-aware loss would be presented during the rebuttal period. Barring that due to time constraints, I would otherwise like to see some analysis to explore this difference, maybe checking to see if UpDown is putting mass in smooth blobs around the predicted answer (though there may be better ways to see if UpDown has captured similar notions of output order as the other models).
- I would like to see a couple of simple baselines evaluated on HowMany-QA. Specifically, I think the paper would be stronger if results were put in context with a question only model and a model which just outputs the mean training count. Inter-human agreement would also be interesting to discuss (especially for high counts).
- The IRLC model has a significantly large (4x) capacity scoring function than the baseline methods. If this is restricted, do we see significant changes to the results?
- This is a relatively mild complaint. This model is more human-intuitive than existing approaches, but when it does make an error by selecting incorrect objects or terminating early, it is no more transparent about the cause of these errors than any other approach. As such, claims about interpretability should be made cautiously.
------------------
Curiosities:
------------------
- In my experience, Visual Genome annotations are often noisy, with many different labels being applied to the same object in different images. For per-image counts, I don't imagine this will be too troubling but was curious if you ran into any challenges.
- It looks like both IRLC and UpDown consistently either get the correct count (for small counts) or underestimate. This is not the Gaussian sort of regression error that we might expect from a counting problem.
- Could you speak to the sensitivity of the proposed model with respect to different loss weightings? I saw the values used in Section B of the supplement and they seem somewhat specific.
------------------
Minor errors:
------------------
[5.1 end of paragraph 2] 'that accuracy and RSME and not' -> 'that accuracy and RSME are not'
[Fig 9 caption] 'The initial scores are lack' -> 'The initial scores lack'


Published as a conference paper at ICLR 2018 UNDERSTANDING IMAGE MOTION WITH GROUP REPRESENTATIONS
Motion is an important signal for agents in dynamic environments, but learning to represent motion from unlabeled video is a difficult and underconstrained problem. We propose a model of motion based on elementary group properties of transformations and use it to train a representation of image motion. While most methods of estimating motion are based on pixel-level constraints, we use these group properties to constrain the abstract representation of motion itself. We demonstrate that a deep neural network trained using this method captures motion in both synthetic 2D sequences and real-world sequences of vehicle motion, without requiring any labels. Networks trained to respect these constraints implicitly identify the image characteristic of motion in different sequence types. In the context of vehicle motion, this method extracts information useful for localization, tracking, and odometry. Our results demonstrate that this representation is useful for learning motion in the general setting where explicit labels are difficult to obtain.

The paper presents a method that given a sequence of frames, estimates a corresponding motion embedding to be the hidden state of an RNN (over convolutional features) at the last frame of the sequence. The parameters of the motion embedding are trained to preserve properties of associativity  and invertibility of motion, where the frame sequences have been recomposed (from video frames) in various way to create pairs of frame sequences with those -automatically obtained- properties. This means, the motion embedding is essentially trained without any human annotations.
Experimentally, the paper shows that in synthetic moving MNIST frame sequences motion embedding discovers different patterns of motion, while it ignores image appearance (i.e., the digit label). The paper also shows that linear regressor trained in KITTI on top of the unsupervised motion embedding to estimate camera motion performs better than chance.
Q to the authors: what labelled data were used to train the linear regressor in the KITTI experiment?
Empirically, it appears that supervision by preserving group transformations may not  be immensely valuable for learning motion representations.
Pros
1)The neural architecture for motion embedding computation appears reasonable
2)The paper tackles an interesting problem
Cons
1)For a big part of the introduction the paper refers to the problem of `` ````"learning motion” or `''understanding motion”  without being specific what it means by that.
2)The empirical results are not convincing of the strength of imposing group transformations for self-supervised learning of motion embeddings.
3)The KITTI experiment is not well explained as it is not clear how this regressor was trained to predict egomotion out of the motion embedding.


Published as a conference paper at ICLR 2018 DISTRIBUTED PRIORITIZED EXPERIENCE REPLAY
We propose a distributed architecture for deep reinforcement learning at scale, that enables agents to learn effectively from orders of magnitude more data than previously possible. The algorithm decouples acting from learning: the actors interact with their own instances of the environment by selecting actions according to a shared neural network, and accumulate the resulting experience in a shared experience replay memory; the learner replays samples of experience and updates the neural network. The architecture relies on prioritized experience replay to focus only on the most significant data generated by the actors. Our architecture substantially improves the state of the art on the Arcade Learning Environment, achieving better final performance in a fraction of the wall-clock training time.

A parallel aproach to DQN training is proposed, based on the idea of having multiple actors collecting data in parallel, while a single learner trains the model from experiences sampled from a central replay memory. Experiments on Atari game playing and two MuJoCo continuous control tasks show significant improvements in terms of training time and final performance compared to previous baselines.
The core idea is pretty straightforward but the paper does a very good job at demonstrating that it works very well, when implemented efficiently over a large cluster (which is not trivial). I also appreciate the various experiments to analyze the impact of several settings (instead of just reporting a new SOTA). Overall I believe this is definitely a solid contribution that will benefit both practitioners and researchers... as long as they got the computational resources to do so!
There are essentially two more things I would have really liked to see in this paper (maybe for future work?):
- Using all Rainbow components
- Using multiple learners (with actors cycling between them for instance)
Sharing your custom Tensorflow implementation of prioritized experience replay would also be a great bonus!
Minor points:
- Figure 1 does not seem to be referenced in the text
- « In principle, Q-learning variants are off-policy methods » => not with multi-step unless you do some kind of correction! I think it is important to mention it even if it works well in practice (just saying « furthermore we are using a multi-step return » is too vague)
- When comparing the Gt targets for DQN vs DPG it strikes me that DPG uses the delayed weights phi- to select the action, while DQN uses current weights theta. I am curious to know if there is a good motivation for this and what impact this can have on the training dynamics.
- In caption of Fig. 5 25K should be 250K
- In appendix A why duplicate memory data instead of just using a smaller memory size?
- In appendix D it looks like experiences removed from memory are chosen by sampling instead of just removing the older ones as in DQN. Why use a different scheme?
- Why store rewards and gamma’s at each time step in memory instead of just the total discounted reward?
- It would have been better to re-use the same colors as in Fig. 2 for plots in the appendix
- Would Fig. 10 be more interesting with the full plot and a log scale on the x axis?


Image generation has been successfully cast as an autoregressive sequence generation or transformation problem. Recent work has shown that self-attention is an effective way of modeling textual sequences. In this work, we generalize a recently proposed model architecture based on self-attention, the Transformer, to a sequence modeling formulation of image generation with a tractable likelihood. By restricting the self-attention mechanism to attend to local neighborhoods we significantly increase the size of images the model can process in practice, despite maintaining significantly larger receptive fields per layer than typical convolutional neural networks. We propose another extension of self-attention allowing it to efficiently take advantage of the two-dimensional nature of images. While conceptually simple, our generative models trained on two image data sets are competitive with or significantly outperform the current state of the art in autoregressive image generation on two different data sets, CIFAR-10 and ImageNet. We also present results on image super-resolution with a large magnification ratio, applying an encoder-decoder configuration of our architecture. In a human evaluation study, we show that our super-resolution models improve significantly over previously published autoregressive super-resolution models. Images they generate fool human observers three times more often than the previous state of the art. Table 1: Three outputs of a CelebA super-resolution model followed by three image completions by a conditional CIFAR-10 model, with input, model output and the original from left to right

Summary
This paper extends self-attention layers (Vaswani et al., 2017) from sequences to images and proposes to use the layers as part of PixelCNNs (van den Oord et al., 2016). The proposed model is evaluated in terms of visual appearance of samples and log-likelihoods. The authors find a small improvement in terms of log-likelihood over PixelCNNs and that super-resolved CelebA images are able to fool human observers significantly more often than PixelRNN based super-resolution (Dahl et al., 2017).
Review
Autoregressive models are of large interest to the ICLR community and exploring new architectures is a valuable contribution. Using self-attention in autoregressive models is an intriguing idea. It is a little bit disappointing that the added model complexity only yields a small improvement compared to the more straight-forward modifications of the PixelCNN++. I think the paper would benefit from a little bit more work, but I am open to adjusting my score based on feedback.
I find it somewhat surprising that the proposed model is only slightly better in terms of log-likelihood than a PixelRNN, but much better in terms of human evaluation – given that both models were optimized for log-likelihood. Was the setup used with Mechanical Turk exactly the same as the one used by Dahl et al.? These types of human evaluations can be extremely sensitive to changes in the setup, even the phrasing of the task can influence results. E.g., presenting images scaled differently can mask certain artifacts. In addition, the variance between subjects can be very high. Ideally, each method included in the comparison would be re-evaluated using the same set of observers. Please include error bars.
The CelebA super-resolution task furthermore seems fairly limited. Given the extreme downsampling of the input, the task becomes similar to simply generating any realistic image. A useful baseline would be the following method: Store the entire training set. For a given query image, look for the nearest neighbor in the downsampled space, then return the corresponding high-resolution image. This trivial method might not only perform well, it also highlights a flaw in the evaluation: Any method which returns stored high-resolution images – even if they don’t match the input – would perform at 50%. To fix this, the human observers should also receive the low-resolution image and be asked to identify the correct corresponding high-resolution image.
Using multiplicative operations to model images seems important. How does the self-attention mechanism relate to “gated” convolutions used in PixelCNNs? Could gated convolutions not also be considered a form of self-attention?
The presentation/text could use some work. Much of the text assumes that the reader is familiar with Vaswani et al. (2017) but could easily be made more self-contained by directly including the definitions used. E.g., the encoding of positions using sines and cosines or the multi-head attention model. I also felt too much of the architecture is described in prose and could be more efficiently and precisely conveyed in equations.
On page 7 the authors write “we believe our cherry-picked images for various classes to be of higher perceptual quality”. This is a meaningless result, not only because the images were cherry-picked. Generating realistic images is trivial - you just need to store the training images. Analyzing samples generated by a generative model (outside the context of an application) should therefore only be used for diagnostic purposes or to build intuitions but not to judge the quality of a model.
Please consider rephrasing the last sentence of the abstract. Generating images which “look pretty cool” should not be the goal of a serious machine learning paper or a respected machine learning conference.


Published as a conference paper at ICLR 2018 LIFELONG LEARNING WITH DYNAMICALLY EXPANDABLE NETWORKS
We propose a novel deep network architecture for lifelong learning which we refer to as Dynamically Expandable Network (DEN), that can dynamically decide its network capacity as it trains on a sequence of tasks, to learn a compact overlapping knowledge sharing structure among tasks. DEN is efficiently trained in an online manner by performing selective retraining, dynamically expands network capacity upon arrival of each task with only the necessary number of units, and effectively prevents semantic drift by splitting/duplicating units and timestamping them. We validate DEN on multiple public datasets under lifelong learning scenarios, on which it not only significantly outperforms existing lifelong learning methods for deep networks, but also achieves the same level of performance as the batch counterparts with substantially fewer number of parameters. Further, the obtained network fine-tuned on all tasks obtained siginficantly better performance over the batch models, which shows that it can be used to estimate the optimal network structure even when all tasks are available in the first place.

The paper was clearly written and pleasant to read. I liked the use of sparsity- and group-sparsity-promoting regularizers to select connections and decide how to expand the network.
A strength of the paper is that the proposed algorithm is interesting and intuitive, even if relatively complex, as it requires chaining a sequence of sub-algorithms. It was good to see the impact of each sub-algorithm studied separately (to some degree) in the experimental section. The results are overall strong.
It’s hard for me to judge the novelty of the approach though, as I’m not an expert on this topic.
Just a few points below:
- The experiments focus on a relevant continual learning problem, where each new task corresponds to learning a new class. In this setup, the method consistently outperforms EWC (e.g., Fig. 3), as well as the progressive network baseline.
Did the authors also check the performance on the permuted MNIST benchmark, as studied by Kirkpatrick et al. and Zenke et al.? It would be important to see how the method fares in this setting, where the tasks are the same, but the inputs have to be remapped, and network expansion is less of an issue.
- Fig. 4 would be clearer if the authors showed also the performance and how much the selected connection subsets would change if instead of using the last layer lasso + BFS, the full L1-penalized problem was solved, while keeping the rest of the pipeline intact.
- Still regarding the proposed selective retraining, the special role played by the last hidden layer seems slightly arbitrary. It may well be that it has the highest task-specificity, though this is not trivial to me. This special role might become problematic when dealing with deeper networks.


Published as a conference paper at ICLR 2018 TOWARDS IMAGE UNDERSTANDING FROM DEEP COMPRESSION WITHOUT DECODING
Motivated by recent work on deep neural network (DNN)-based image compression methods showing potential improvements in image quality, savings in storage, and bandwidth reduction, we propose to perform image understanding tasks such as classification and segmentation directly on the compressed representations produced by these compression methods. Since the encoders and decoders in DNN-based compression methods are neural networks with feature-maps as internal representations of the images, we directly integrate these with architectures for image understanding. This bypasses decoding of the compressed representation into RGB space and reduces computational cost. Our study shows that accuracies comparable to networks that operate on compressed RGB images can be achieved while reducing the computational complexity up to 2×. Furthermore, we show that synergies are obtained by jointly training compression networks with classification networks on the compressed representations, improving image quality, classification accuracy, and segmentation performance. We find that inference from compressed representations is particularly advantageous compared to inference from compressed RGB images for aggressive compression rates.

Thanks for addressing most of the issues. I changed my given score from 3 to 6.
Summary:
This work explores the use of learned compressed image representation for solving 2 computer vision tasks without employing a decoding step.
The paper claims to be more computationally and memory efficient compared to the use of original or the decompressed images. Results are presented on 2 datasets "Imagenet" and "PASCAL VOC 2012". They also jointly train the compression and classification together and empirically shows it can improve both classification and compression together.
Pros:
+ The idea of learning from a compressed representation is a very interesting and beneficial idea for large-scale image understanding tasks.
Cons:
- The paper is too long (13 pages + 2 pages of references). The suggested standard number of pages is 8 pages + 1 page of references. There are many parts that are unnecessary in the paper and can be summarized. Summarizing and rewording them makes the paper more consistent and easier to read:
  ( 1. A very long introduction about the benefits of inferring from the compressed images and examples.
    2. A large part of the intro and Related work can get merged.
    3. Experimental setup part is long but not well-explained and is not self-contained particularly for the evaluation metrics.
    “Please briefly explain what MS-SSIM, SSIM, and PSNR stand for”. There is a reference to the Agustsson et al 2017 paper
     “scalar quantization”, which is not well explained in the paper. It is better to remove this part if it is not an important part or just briefly but clearly explain it.
     4. Fig. 4 is not necessary. 4.3 contains extra information and could be summarized in a more consistent way.
     5. Hyperparameters that are applied can be summarized in a small table or just explain the difference between the
      architectures that are used.)
- There are parts of the papers which are confusing or not well-written. It is better to keep the sentences short and consistent:
E.g: subsection 3.2, page 5: “To adapt the ResNet … where k is the number of … layers of the network” can be changed to 3 shorter sentences, which is easier to follow.
There are some typos: e.g: part 3.1, fever ---> fewer,
- As it is mentioned in the paper, solving a Vision problem directly from a compressed image, is not a novel method (e.g: DCT coefficients were used for both vision and audio data to solve a task without any decompression). However, applying a deep representation for the compression and then directly solving a vision task (classification and segmentation) can be considered as a novel idea.
- In the last part of the paper, both compression and classification parts are jointly trained, and it is empirically presented that both results improved by jointly training them. However, to me, it is not clear if the trained compression model on this specific dataset and for the task of classification can work well for other datasets or other tasks.
The experimental setup and the figures are not well explained and well written.


Published as a conference paper at ICLR 2018 FASTGCN: FAST LEARNING WITH GRAPH CONVOLU- TIONAL NETWORKS VIA IMPORTANCE SAMPLING
The graph convolutional networks (GCN) recently proposed by Kipf and Welling are an effective graph model for semi-supervised learning. This model, however, was originally designed to be learned with the presence of both training and test data. Moreover, the recursive neighborhood expansion across layers poses time and memory challenges for training with large, dense graphs. To relax the requirement of simultaneous availability of test data, we interpret graph convolutions as integral transforms of embedding functions under probability measures. Such an interpretation allows for the use of Monte Carlo approaches to consistently estimate the integrals, which in turn leads to a batched training scheme as we propose in this work-FastGCN. Enhanced with importance sampling, FastGCN not only is efficient for training but also generalizes well for inference. We show a comprehensive set of experiments to demonstrate its effectiveness compared with GCN and related models. In particular, training is orders of magnitude more efficient while predictions remain comparably accurate.

Update:
I have read the rebuttal and the revised manuscript. Additionally I had a brief discussion with the authors regarding some aspects of their probabilistic framework. I think that batch training of GCN is an important problem and authors have proposed an interesting solution to this problem. I appreciated all the work authors put into the revision. In this regard, I have updated my rating. However, I am not satisfied with how the probabilistic problem formulation was presented in the paper. I would appreciate if authors were more upfront about the challenges of the problem they formulated and limitations of their results. I briefly summarize the key missing points below, although I acknowledge that solution to such questions is out of scope of this work.
1. Sampling of graph nodes from P is not iid. Every subsequent node can not be equal to any of the previous nodes. Hence, the distribution changes and subsequent nodes are dependent on previous ones. However, exchangeability could be a reasonable assumption to make as order (in the joint distribution) does not matter for simple choices of P. Example: let V be {1,2,3} and P a uniform distribution. First node can be any of the {1,2,3}, second node given first (suppose first node is '2') is restricted to {1,3}. There is clearly a dependency and change of distribution.
2. Theorem 1 is proven under the assumption that it is possible to sample from P and utilize Monte Carlo type argument. However, in practice, sampling is done from a uniform distribution over observed samples. Also, authors suggest that V may be infinite. Recall that for Monte Carlo type approaches to work, sampling distribution is ought to contain support of the true distribution. Observed samples (even as sample size goes to infinity) will never be able to cover an infinite V. Hence, Theorem 1 will never be applicable (for the purposes of evaluating population loss). Also note that this is different from a more classical case of continuous distributions, where sampling from a Gaussian, for instance, will cover any domain of true distribution. In the probabilistic framework defined by the authors it is impossible to cover domain of P, unless whole V is observed.
----------------------------------------------------------------------
This work addresses a major shortcoming of recently popularized GCN. That is, when the data is equipped with the graph structure, classic SGD based methods are not  straightforward to apply. Hence it is not clear how to deal with large datasets (e.g., Reddit). Proposed approach uses an adjacency based importance sampling distribution to select only a subset of nodes on each GCN layer. Resulting loss estimate is shown to be consistent and its gradient is used to perform the weight updates.
Proposed approach is interesting and the direction of the work is important given recent popularity of the GCN. Nonetheless I have two major question and would be happy to revisit my score if at least one is addressed.
Theory:
SGD requires an unbiased estimate of the gradient to converge to the global optima in the convex loss case. Here, the loss estimate is shown to be consistent, but not guaranteed to be unbiased and nothing is said about the gradient in Algorithm 1. Could you please provide some intuition about the gradient estimate? I might not be familiar with some relevant results, but it appears to me that Algorithm 1 will not converge to the same solution as full data GD would.
Practice:
Per batch timings in Fig. 3 are not enough to argue that the method is faster as it might have poor convergence properties overall. Could you please show the train/test accuracies against training time for all compared methods?
Some other concerns and questions:
- It is not quite cleat what P is. You defined it as distribution over vertices of some (potentially infinite) population graph. Later on, sampling from P becomes equivalent to uniform sampling over the observed nodes. I don't see how you can define P over anything outside of the training nodes (without defining loss on the unobserved data), as then you would be sampling from a distribution with 0 mass on the parts of the support of P, and this would break the Monte Carlo assumptions.
- Weights disappeared in the majority of the analysis. Could you please make the representation more consistent.
- a(v,u) in Eq. 2 and A(v,u) in Eq. 5 are not defined. Do they both correspond to entries of the (normalized) adjacency?


Published as a conference paper at ICLR 2018 MEMORY ARCHITECTURES IN RECURRENT NEURAL NETWORK LANGUAGE MODELS
We compare and analyze sequential, random access, and stack memory architectures for recurrent neural network language models. Our experiments on the Penn Treebank and Wikitext-2 datasets show that stack-based memory architectures consistently achieve the best performance in terms of held out perplexity. We also propose a generalization to existing continuous stack models (Joulin & Mikolov, 2015;Grefenstette et al., 2015) to allow a variable number of pop operations more naturally that further improves performance. We further evaluate these language models in terms of their ability to capture non-local syntactic dependencies on a subject-verb agreement dataset (Linzen et al., 2016) and establish new state of the art results using memory augmented language models. Our results demonstrate the value of stack-structured memory for explaining the distribution of words in natural language, in line with linguistic theories claiming a context-free backbone for natural language.

The authors propose to compare three different memory architecture for recurrent neural network language models:
vanilla LSTM, random access based on attention and continuous stack. The second main contribution of the paper is to propose an extension of continuous stacks, which allows to perform multiple pop operations at a single time step.
The way to do that is to use a similar mechanism as the adaptive computation time from Graves (2016): all the pop operations are performed, and the final state of the continuous stack is weighted average of all the intermediate states. The different memory models are evaluated on two standard language modeling tasks: PTB and WikiText-2, as well as on the verb number prediction dataset from Linzen et al (2016). On the language modeling tasks, the stack model performs slightly better than the attention models (0-2 ppl points) which performs slightly better than the plain LSTM (2-3 ppl). On the verb number prediction tasks, the stack model tends to outperforms the two other models (which get similar results) for hard examples (2 or more attractors).
Overall, I enjoy reading this paper: it is clearly written, and contains interesting analysis of different memory architecture for recurrent neural networks. As far as I know, it is the first thorough comparison of the different memory architecture for recurrent neural network applied to language modeling. The experiments on the Linzen et al. (2016) dataset is also interesting, as it shows that for hard examples, the different models do have different behavior (even when the difference are not noticeable on the whole test set).
One small negative aspect of the paper is that the substance might be a bit limited. The only technical contribution is to merge the ideas from the continuous stack with the adaptive computation time to obtain the "multi-pop" model. In the experimental section, which I believe is the main contribution of the paper, I would have liked to see more "in-depth" analysis of the different models. I found the experiments performed on the Linzen et al. (2016) dataset (Table 2) to be quite interesting, and would have liked more analysis like that. On the other hand, I found Figures 2 or 3 not very informative, as it is (would like to see more). For example, from Fig. 2, it would be interesting to get a better understanding of what errors are made by the different models (instead of just the distribution).
Finally, I have a few questions for the authors:
- In Figure 1. shouldn't there be an arrow from h_{t-1} to m_t instead of x_{t-1} to m_t?
- What are the equations to update the stack? I assume something similar to Joulin & Mikolov (2015)?
- Do you have any ideas why there is a sharp jump between 4 and 5 attractors (Table 2)?
- Why no "pop" operations in Figure 3 and 4?
pros/cons:
+ clear and easy to read
+ interesting analysis
- not very original
Overall, while not groundbreaking, this is a serious paper with interesting analysis. Hence, I am weakly recommending to accept this paper.