As deep CNN classifier performance using ground-truth labels has begun to asymptote at near-perfect levels, a key aim for the field is to extend training paradigms to capture further useful structure in natural image data and improve model robustness and generalization. In this paper, we present a novel natural image benchmark for making this extension, which we call CIFAR10H. This new dataset comprises a human-derived, full distribution over labels for each image of the CIFAR10 test set, offering the ability to assess the generalization of stateof-the-art CIFAR10 models, as well as investigate the effects of including this information in model training. We show that classification models trained on CIFAR10 do not generalize as well to our dataset as it does to traditional extensions, and that models fine-tuned using our label information are able to generalize better to related datasets, complement popular data augmentation schemes, and provide robustness to adversarial attacks. We explain these improvements in terms of better empirical approximations to the expected loss function over natural images and their categories in the visual world.

The authors create a new dataset with label distributions (rather than one-hot annotations) for the CIFAR-10 test set. They then study the effect of fine-tuning using this dataset on the generalization performance of SOTA deep networks. They also study the effects on adversarial robustness.
I think that datasets such as the one generated in this paper could indeed be a valuable testbed to study deep network generalization and robustness. There are many nice benefits of label distributions over one hot labels (that the authors summarize in Section 2.) The paper is also clear and well-written.
That being said, I do not find the investigation of this paper completely satisfactory. For instance in the generalization experiments, the numbers presented seem to show some interesting (and somewhat surprising) trends, however the authors do not really pursue these or provide any insight as to why this is the case. I also find the section on robustness very weak.
Detailed comments:
- The theoretical contribution mentioned in the appendix does not really seem to be a contribution - it is just a simple derivation of the loss under label distributions. Theoretical contributions are not necessary for a paper to have merit - the authors should remove this statement from the introduction as it detracts from the value of the paper.
- I find it somewhat surprising that the accuracy of the models does not change on training with Cifar10H. Do the authors have any intuition as to why this is the case? The model cross entropy seems to go down, indicating that probability assigned to the correct class increases. I would think that training with label distributions would actually reduce misclassification on confusing instances. It would be interesting to see how the logit distributions change for different examples. For instance, how does the model confidence change on correctly vs wrongly classified examples?
- The authors mention that they run each hyperparameter configuration for three random seeds. It would be nice then to see error bars for the results reported Tables 1 and 2, particularly because the differences in accuracy are small. Did the authors try different train-test splits of the test set? It would also be helpful if the authors could make plots for the results in these tables (at least in the appendix). It is hard to compare numbers across different tables.
-I find the results in Table 2 confusing. Comparing the numbers to Table 1, it seems that mixup does not really change accuracies/loss. The model names in Table 2 do not exactly match Table 1 so it is hard to identify the additional gain from using mixup that the authors mention. The authors should add plots for these results to illustrate the effect of adding mixup more clearly.
-I am not convinced by the section on robustness. Firstly, it is not clear to me why the authors chose FGSM which is known to be a somewhat simple attack to illustrate improved robustness of their model. To perform a useful study of robustness, the authors should study SOTA attacks such as PGD [Madry et al., 2017]. I also do not understand the claim that the top-1 choice becomes less confident after training with CIFAR10H -- this seems to be contradicted by the fact that the cross entropy loss goes down. The authors should provide supporting evidence for this claim by looking at changes in confidence (see point 3 above). Also, the comment about the trade-off between accuracy and robustness seems vague - could the authors clarify what they mean?
Overall, I like the premise of this paper and agree that with the potential benefits of the dataset generated. However, I think that the current experiments are not strong enough to corroborate this.


A known weakness of deep networks is a failure to perform well when evaluated on data which differ from the training distribution, even if these differences are very small, as is the case with adversarial examples. We propose Fortified Networks, a simple extension of existing networks, which "fortifies" the hidden layers in a deep network by identifying when the hidden states are off of the data manifold, and maps these hidden states back to parts of the data manifold where the network performs well. Our principal contribution is to show that fortifying these hidden states improves the robustness of deep networks and our experiments (i) demonstrate improved robustness to standard adversarial attacks in both black-box and white-box threat models; (ii) suggest that our improvements are not primarily due to the problem of deceptively good results due to degraded quality in the gradient signal (the gradient masking problem) and (iii) show the advantage of doing this fortification in the hidden layers instead of the input space. We demonstrate improvements in adversarial robustness on three datasets (MNIST, Fashion MNIST, CIFAR10), across several attack parameters, both white-box and black-box settings, and the most widely studied attacks (FGSM, PGD, Carlini-Wagner). We show that these improvements are achieved across a wide variety of hyperparameters.

This paper proposes a new defense to adversarial examples based on the 'fortification' of hidden layers using a denoising autoencoder. While building models that are robust to adversarial examples is an important and relevant research problem, I am not convinced by the evaluation of the defense.  Specific comments:
- The authors mostly evaluate their defense using FGSM (particularly on CIFAR). To truly establish the merit of a new defense, the authors must benchmark against state-of-the-art defenses such as PGD. It also seems like the epsilon values used for the PGD attacks are fairly small. The authors should report accuracies to a range of epsilon values for the PGD attack, as is standard.
- When the authors attack their models using PGD/FGSM, is this only on the classification loss or does this also include the denoising terms? Similar defenses which use denoisers have been broken once you run PGD on the full model [1].
- I do not really understand the motivation behind using an autoencoder here. Firstly, it is not clear that adversarial examples lie off the data manifold - they could form a very small set on the data manifold and thereby not affect standard generalization. Secondly, have the authors tried a simple regularization loss based on the error between hidden layer representations to a natural examples and the corresponding adversarial example? I think the authors must motivate the use of denoising autoencoders here by comparing to such a simple baseline.
General comment: The results hard to parse given the arrangement of figures and tables. Also, which approach are the authors denoting as ‘baseline adv. Train’ in the tables?
Overall I feel like building defenses to adversarial examples is a challenging problem and the empirical investigation in this paper is not sufficient to illustrate any real progress on this front.
[1] Athalye, A., & Carlini, N. (2018). On the Robustness of the CVPR 2018 White-Box Adversarial Example Defenses. arXiv preprint arXiv:1804.03286.


We propose Cooperative Training (CoT) for training generative models that measure a tractable density for discrete data. CoT coordinately trains a generator G and an auxiliary predictive mediator M . The training target of M is to estimate a mixture density of the learned distribution G and the target distribution P , and that of G is to minimize the Jensen-Shannon divergence estimated through M . CoT achieves independent success without the necessity of pre-training via Maximum Likelihood Estimation or involving high-variance algorithms like REINFORCE. This low-variance algorithm is theoretically proved to be superior for both sample generation and likelihood prediction. We also theoretically and empirically show the superiority of CoT over most previous algorithms in terms of generative quality and diversity, predictive generalization ability and computational cost.

The paper proposes an interesting method, where the discriminator is replaced by a component that estimates the density that is the mixture of the data and the generator's distributions. In a sense, that component is only a device that allows estimating a Jensen-Shannon divergence for the generator to then be optimized against. Other GAN papers have replaced their discriminator by a similar device (e.g., WGANs, ..), but the present formulation seems novel. The numerical experiments presented on a synthetic Turing test and text generation from EMNLP's 2017 news dataset appear promising.
Overall, the mediator seems to allow to achieve lower Jensen-Shannon (JS) divergence values in the experiments (and is kind of designed for that). Although this may be an improvement with respect to existing methods for discrete sequential data, it may also be limited in that it may not easily extend to other types of divergences that have proved superior to JS in some continuous settings.
The paper is rather clear, although there are lots of small grammatical errors as well as odd formulations which end up being distracting or confusing. The language should be proof-read carefully.
Pros:
- Generative modeling of sequence data still in its infancy
- Potentially lower variance than policy gradient approaches
- Experiments are promising
Cons:
- Lots of grammatical errors and odd formulations
Questions:
- Equation 14: what does it mean to find the "maximum entropy solution" for the given optimization problem?
- Figure 2: how do (b) and (c) relate to each other?
Remarks, small typos and odd formulations:
- "for measuring M_\/phi": what does measuring mean in this context?
- What does small m refer to? Algorithm 1 says the total number of steps  but it is also used in the main text as an index for J and \pi (for mediator?)
- Equation block 8: J_m has not been defined yet
- "the supports of distributions G and P"... -> G without subscript has now been defined in this context
- "if the training being perfect"
- "tend to get stuck in some sub-optimals"
- the learned distribution "collapseS"
- "since  the data distribution is, thus ..."
- "that measures a" -> "that estimates a ..."?
- "a predictive module": a bit unclear - generative v. discriminative is more usual terminology
- "is well ensured"
- "with the cost of diversity" -> "at the cost of diversity"?
- "has theoretical guarantee"
- in the references: "ALIAS PARTH GOYAL" (all caps)
- "let p denote the intermediate states": I don't understand what this is. Where is "p" used? (proof of Theorem 3)
- "CoT theoretically guarantees the training effectiveness": what does that mean?
- Figure 3: "epochs" -> "Epochs"
- Algorithm 1: what does "mixed balanced samples" mean? Make this more precise
- "wide-ranged"
- Equation 10 is too long and equation number is not properly formatted
- Figures hard to read in black & white
- Figure 2 doesn't use the same limits for the Y axis of the two NLL plots, making comparisons difficult. The two NLL plots are also not side-by-side


With the recently rapid development in deep learning, deep neural networks have been widely adopted in many real-life applications. However, deep neural networks are also known to have very little control over its uncertainty for test examples, which potentially causes very harmful and annoying consequences in practical scenarios. In this paper, we are particularly interested in designing a higher-order uncertainty metric for deep neural networks and investigate its performance on the out-of-distribution detection task proposed by Hendrycks & Gimpel (2016). Our method first assumes there exists an underlying higher-order distribution P(z), which generated label-wise distribution P(y) over classes on the K-dimension simplex, and then approximate such higher-order distribution via parameterized posterior function p θ (z|x) under variational inference framework, finally we use the entropy of learned posterior distribution p θ (z|x) as uncertainty measure to detect out-of-distribution examples. However, we identify the overwhelming over-concentration issue in such a framework, which greatly hinders the detection performance. Therefore, we further design a log-smoothing function to alleviate such issue to greatly increase the robustness of the proposed entropy-based uncertainty measure. Through comprehensive experiments on various datasets and architectures, our proposed variational Dirichlet framework with entropy-based uncertainty measure is consistently observed to yield significant improvements over many baseline systems.

This paper proposes a new framework for out-of-distribution detection, based on variational inference and a prior Dirichlet distribution. The Dirichlet distribution is presented, and the way it is used withing the method is discussed (i.e. clipping, scaling, etc). Experiments on several datasets and comparison with the state of the art is reported and extensively discussed.
The motivation of the proposed approach is clear, and I agree with the attempt to regularize the network output. The choice of the Dirichlet distribution is quite natural, as each of its samples are the prior weights of a multinomial distribution. However, some other choices are less clear (clipping, scaling, decision, and the use of a non-informative prior). The overall inference procedure appears to be advantageous in the many experiments that the authors report (several datasets, and several baselines).
The first thought that came to my mind, is that out-of-distribution detection is to classification what outlier detection is to regression. Therefore, relevant and recent work on the topic deserves to be cited, for instance:
S. Lathuiliere, P. Mesejo, X. Alameda-Pineda and R. Horaud, DeepGUM: Learning Deep Robust Regression with a Gaussian-Uniform Mixture Model, In ECCV, 2018.
One thing that I found quite strange at first sight is the choice of clipping the parameters of the Dirichlet distribution. It is said that this is done in order to choose an appropriate prior distribution. However, the choice is not very well motivated, because what "appropriate" means is not discussed. So why do you clip to 1? What would happen if some of the alpha's go over 1? Is it a numerical problem, a modeling problem, a convergence issue?
I would also like the authors to comment on the use of a non-informative Dirichlet distribution within the KL-divergence. The KL divergence measures the deviation between the approximate a posteriori distribution and the true one. If one selects the non-informative Dirichlet distribution, this is not only a brutal approximation of the true posterior, but most importantly a distribution that does not depend on x, and that therefore cannot be truly called posterior.
It is also strange to take a decision based on the maximum alpha. On the contrary, the smallest alpha should be taken, since it is the one that concentrates more probability mass to the associated corner in the simplex.
Regarding the scaling function, it is difficult to grasp its motivation and effects. It is annouced that the aim of the smoothing function is to smooth the concentration parameters alpha. But in what sense? Why do they need to be smoothed? Is this done to avoid numerical/convergence problems? Is this a key part of the model? The same stands, by the way, for the form of the input perturbation.
The experiments are plenty, and I appreciated the sanity check done after introducing the datasets. However, I did not manage to understand why some methods appear in some tables and not in other (for example "Semantic"). I also have the feeling that the authors could have chosen CIFAR-100 in Table 2, since most of the metrics reported are quite high (the problems are not much challenging).
Regarding the parameter eta, I would say that its discussion right after Table 3 is not long enough. Specially, given the high sensitivity of this parameter, as reported in the Table of Figure 4. What is the overall interpretation of this sensitivity?
From a quantitative perspective, the results are impressive, since the propose methods systematically outperforms the baselines (at least the ones reported). However, since these baselines are not the same in all experiments, CIFAR-100 is not used, and the discussion of the results could be richer, I conclude that the experimental section is not too strong.
In addition to all my comments, I would say that the authors chose to exceed the standard limit of 8 pages. Even if this is allowed, the extra pages should be justified. I am affraid that there are many choices not well motivated, and that the discussion of the results is not informative enough. I am therefore not inclined to accept the paper as it is.


Reinforcement learning agents are typically trained and evaluated according to their performance averaged over some distribution of environment settings. But does the distribution over environment settings contain important biases, and do these lead to agents that fail in certain cases despite high average-case performance? In this work, we consider worst-case analysis of agents over environment settings in order to detect whether there are directions in which agents may have failed to generalize. Specifically, we consider a 3D first-person task where agents must navigate procedurally generated mazes, and where reinforcement learning agents have recently achieved human-level average-case performance. By optimizing over the structure of mazes, we find that agents can suffer from catastrophic failures, failing to find the goal even on surprisingly simple mazes, despite their impressive average-case performance. Additionally, we find that these failures transfer between different agents and even significantly different architectures. We believe our findings highlight an important role for worst-case analysis in identifying whether there are directions in which agents have failed to generalize. Our hope is that the ability to automatically identify failures of generalization will facilitate development of more general and robust agents. To this end, we report initial results on enriching training with settings causing failure.

Update:
I appreciate the clarifications and the extension of the paper in response to the reviews. I think it made the work stronger. The results in the newly added section are interesting and actually suggest that by putting more effort into training set design/augmentation, one could further robustify the agents, possibly up to the point where they do not break at all in unnatural ways. It is a pity the authors have not pushed the work to this point (and therefore the paper is not as great as it could be), but still I think it is a good paper that can be published.
-----
The paper analyzes the performance of modern reinforcement-learning-based navigation agents by searching for “adversarial” maze layouts in which the agents do not perform well. It turns out that such mazes  exist, and moreover, one can find even relatively simple maze configurations that are easily solved by humans, but very challenging for the algorithms.
Pros:
1) Interesting and relevant topic: it is important not only to push for best results on benchmarks, but also understand the limitations of existing approaches.
2) The paper is well written
3) The experiments are quite thorough and convincing. I especially appreciate that it is demonstrated that there exist simple mazes that can be easily solved by humans, but not by algorithms. The analysis of transferability of “adversarial” mazes between different agents is also a plus.
Cons:
1) I am not convinced worst-case performance is the most informative way to evaluate models. Almost no machine learning model is perfect, and therefore almost for any model it would be possible to find training or validation samples on which it does not perform well. Why is it so surprising that this is also the case for navigation models? Why would one assume they should be perfect? Especially given that the generated “adversarial” mazed lie outside of the training data distribution, seemingly quite far outside. Are machine learning models ever expected to perfectly generalize outside of the training data distribution? Very roughly speaking, the key finding of the paper can be summarized as “several recent navigation agents have problems finding and entering small rooms of the type they never saw during training” - is this all that significant?
To me, the most interesting part of the paper is that the models generalize as well as they do. I would therefore like to see if it is possible to modify the training distribution - by adding “adversarial” mazes, potentially in an iterative fashion, or just by hand-designing a wider distribution of mazes - so that generalization becomes nearly perfect and the proposed search method is not anymore able to find “adversarial” mazes that are difficult for the algorithm, but easy for humans.
2) On a related note, to me the largest difference between the mazes generated in this paper and the classical adversarial images is that the modification of the maze is not constrained to be small or imperceptible. In fact, it is quite huge - the generated mazes are far from the training distribution. This is a completely different regime. This is like training a model to classify cartoon images and then asking it to generalize to real images (or perhaps other way round). Noone would expect existing image classification models to do this. This major difference with classical adversarial examples should be clearly acknowledged.
3) It would be interesting to know the computational requirements of the search method. I guess it can be estimated from the information in the paper, but would be great to mention it explicitly. (I am sorry if it is already mentioned and I missed it)
To conclude, I think the paper is interesting and well executed, but the presented results are very much to be expected. To me the most interesting aspect of the work is that the navigation agents generalize surprisingly well. Therefore, I believe the work would be much more useful if it focused more on how to make the agents generalize even better, especially since there is a very straightforward way to try this - by extending the training set. I am currently in the borderline mode, but would be very happy to change my evaluation if the focus of the paper is somewhat changed and additional experiments on improving generalization (or some other experiments, but making the results a bit more useful/surprising) are added.


LEARNING PROCEDURAL ABSTRACTIONS AND EVAL- UATING DISCRETE LATENT TEMPORAL STRUCTURE
Clustering methods and latent variable models are often used as tools for pattern mining and discovery of latent structure in time-series data. In this work, we consider the problem of learning procedural abstractions from possibly highdimensional observational sequences, such as video demonstrations. Given a dataset of time-series, the goal is to identify the latent sequence of steps common to them and label each time-series with the temporal extent of these procedural steps. We introduce a hierarchical Bayesian model called PRISM that models the realization of a common procedure across multiple time-series, and can recover procedural abstractions without supervision. We also bring to light two characteristics ignored by traditional evaluation criteria when evaluating latent temporal labelings (temporal clusterings) -segment structure, and repeated structure -and develop new metrics tailored to their evaluation. We demonstrate that our metrics improve interpretability and ease of analysis for evaluation on benchmark time-series datasets. Results on benchmark and video datasets indicate that PRISM outperforms standard sequence models as well as state-of-the-art techniques in identifying procedural abstractions.

This is a hybrid paper, making contributions on two related fronts:
1. the paper proposes a performance metric for sequence labeling, capturing salient qualities missed by other metrics, and
2. the paper also proposes a new sequence labeling method based on inference in a hierarchical Bayesian model, focused on simultaneously labeling multiple sequences that have the same underlying procedure but with varying segment lengths.
This paper is not a great topic fit for ICLR: it's primarily about a hand-designed performance metric for sequence labeling and a hierarchical Bayesian model with Gaussian observations and fit with Gibbs sampling in a full-batch setting. The ICLR 2019 reviewer guidelines suggest "Ask yourself: will a substantial fraction of ICLR attendees be interested in reading this paper?" and based on my understanding of the ICLR audience I suspect not. Based on looking at past ICLR proceedings, this paper's topic and collection of techniques is not in the ICLR mainstream (though it's not totally unrelated). The authors could convince me that I'm mistaken by pointing out closely related ICLR papers (e.g. with a similar mix of techniques in their methods, or similarly proposing a hand-designed performance metric); as far as I can tell, none of the papers cited in the references are from ICLR, but rather from e.g. NIPS, AISTATS, and IEEE TPAMI, which I believe would be better fits for this kind of work.
One way to make this work more relevant to the ICLR audience would be to add feature learning (especially based on neural network architectures). That might also entail additional technical contributions, like how to fit models like these in the minibatch setting (where the current Gibbs sampling method might not apply).
On the proposed performance metric, the discussion of existing metrics as they apply to the example in Fig 3 was really helpful. (I assume, but didn't check, that the authors' characterization of the published performance metrics is accurate, e.g. "no traditional clustering criteria can distinguish C_2 from C_3".) The proposed metric seems to help.
But it's a bit complicated, with several free design decisions involved (e.g. choosing the scoring function \mathcal{H} in Sec 3.1, the choice of conditional entropy H in Sec 3.2, the choice of \beta in Sec 3.3, the choice of the specific algebraic forms of RSS, LASS, SSS, and TSS). Certainly the proposed metrics incorporate the kind of information that the authors argue can be important, but the design details of how that information is summarized into a single number aren't really explored or weighed against alternative designs choices.
If a primary aim of this paper is to propose a new performance metric, and presumably to have it catch on with the rest of the field, then the contribution would be much greater if the design space was clearly articulated, alternatives were considered, and multiple proposals were validated. Validation could be done with human labelers ranking the intuitive 'goodness' of labeling results (and then compared to rankings derived from the proposed performance metrics), and with comparing how the metrics correlate with performance on various downstream tasks.
Another idea is to take advantage of a better segmentation performance metric and use it to automatically tune the hyperparameters of the sequence labeling methods considered in the experiments section. (IIUC hyperparameters were set by hand in the experiments.). That would make for more interesting experiments that give a more comprehensive summary of how these techniques can compare.
However, as it stands, while the performance metric itself may have merit, in this paper it is not sufficiently well validated or compared to alternatives.
On the hierarchical Bayesian model, the current model design andinference algorithm are okay but don't constitute major technical contributions. I was surprised by some model details: for example, in "Modeling the procedure" of Sec 4.1, it would be much more satisfying to generate the (p_1, ..., p_s) sequence from an HMM instead of sampling the elements of the sequence independently, dropping any chance to learn transition structure as part of the Bayesian inference procedure. More importantly, it wasn't made clear if 'self-transitions' where p_s = p_{s+1} were ruled out, though such transitions might confuse the model's semantics. As another example, in "Modeling the realizations in each time-series" of Sec 4.1, the procedure based on iid sampling and sorting seems unnatural, and might make inference more complex. Why not just sample the durations directly (rather than indirectly defining them via sorting independently-generated indices)? If there's a good reason, it should probably be discussed (e.g. maybe parameterizing the durations directly would make it easier to express prior distributions over *absolute* segment lengths, but harder to express distributions over *relative* segment lengths?). Finally, the restriction to conditionally iid Gaussian observations was disappointing.
The experimental results were solid on the task for which the model's extra assumptions paid off, but that's a niche comparison.
One suggestion on the baseline front: you can tie multiple HMMs to have the same procedure (i.e. the same state sequences not counting repeats) by fixing the number of states to be s (the length of the procedure sequence) and fixing the transition matrices to have an upper-bidiagonal support structure. A similar construction can be used for HSMMs. I think a natural Gibbs sampling procedure would emerge. This approach is probably written down in the HMM literature (it seems every conceivable HMM variant has been studied!) but I don't have a reference for it.
Overall, this paper needs more work.
Minor suggestions:
- maybe refer to "segment structure" (e.g. in Sec 3), as "changepoint structure" (and consider looking into changepoint performance metrics if you haven't already)
- if you used code from other authors in your baselines, it would be good to cite that code (e.g. GitHub links)


Exploration in environments with sparse rewards is a key challenge for reinforcement learning. How do we design agents with generic inductive biases so that they can explore in a consistent manner instead of just using local exploration schemes like epsilon-greedy? We propose an unsupervised reinforcement learning agent which learns a discrete pixel grouping model that preserves spatial geometry of the sensors and implicitly of the environment as well. We use this representation to derive geometric intrinsic reward functions, like centroid coordinates and area, and learn policies to control each one of them with off-policy learning. These policies form a basis set of behaviors (options) which allows us explore in a consistent way and use them in a hierarchical reinforcement learning setup to solve for extrinsically defined rewards. We show that our approach can scale to a variety of domains with competitive performance, including navigation in 3D environments and Atari games with sparse rewards.

This paper proposed an algorithm for structured exploration in deep reinforcement learning via learning the visual abstractions from pixels. The proposed method learns discrete visual abstractions and derives intrinsic reward functions from them so as to help the agent to optimize the policy.
The proposed method is interesting in that learning the visual abstractions together with the policy may assist in computing an optimal policy. The method is learning a meta Q function and (E * M+1) other Q functions. The authors mentioned that their work is most similar to hierarchical-DQN (Kulkarni et al., 2016) but this work required hand-crafted instance segmentation and the agent architecture do not learn about many intrinsic rewards learners. However, I am concerned if the proposed method solved the problem with the need of hand-crafted instance segmentation since, as shown in the Algorithm 1 and the caption of Figure 2, Q_{meta} acts every T steps. I do not understand why the meta Q function is used to propose actions for every fixed number of steps. Besides that, though the proposed method does have many intrinsic reward functions (in fact, there are E * M additional intrinsic reward functions). However, the authors did not show in the experiments if having too many intrinsic reward functions helps a lot. It will be better if the authors can show that, larger values for E or M can make the performances better.
Another concern I have is on some of the experiment results. For the experiment results in Figure 5 and 6, only in the left figures can the results of the proposed methods outperform the baselines. Besides that, the authors may need to describe the baseline methods in the experiments in more details.
Also, it will be better if the authors can improve the paper a little bit with the writing. For example, it will be better if the authors can explain the variables X, Y and the distribution q when mentioning Equation 1 so that it is easier to understand the paper. Also, there are some typos, such as the section reference on line 10 of Algorithm 1, the definition of the g function on the last line of page 3 (I guess the authors want to write "{0...E}" instead of "{0, E}") and the second sentence of the experiment section (at least I did not see the supplementary sections, but the authors mentioned that). It is better if these typos can be fixed.

References:
Tejas D Kulkarni, Karthik Narasimhan, Ardavan Saeedi, and Josh Tenenbaum. Hierarchical deep reinforcement learning: Integrating temporal abstraction and intrinsic motivation. In Advances in neural information processing systems, pp. 3675–3683, 2016.


DYREP: LEARNING REPRESENTATIONS OVER DYNAMIC GRAPHS
Representation Learning over graph structured data has received significant attention recently due to its ubiquitous applicability. However, most advancements have been made in static graph settings while efforts for jointly learning dynamic of the graph and dynamic on the graph are still in an infant stage. Two fundamental questions arise in learning over dynamic graphs: (i) How to elegantly model dynamical processes over graphs? (ii) How to leverage such a model to effectively encode evolving graph information into low-dimensional representations? We present DyRep -a novel modeling framework for dynamic graphs that posits representation learning as a latent mediation process bridging two observed processes namely -dynamics of the network (realized as topological evolution) and dynamics on the network (realized as activities between nodes). Concretely, we propose a two-time scale deep temporal point process model that captures the interleaved dynamics of the observed processes. This model is further parameterized by a temporal-attentive representation network that encodes temporally evolving structural information into node representations which in turn drives the nonlinear evolution of the observed graph dynamics. Our unified framework is trained using an efficient unsupervised procedure and has capability to generalize over unseen nodes. We demonstrate that DyRep outperforms state-of-the-art baselines for dynamic link prediction and time prediction tasks and present extensive qualitative insights into our framework.

Overall, the contribution of the paper is somewhat limited [but a little more than my initial assessment, thanks to the rebuttal]. It is essentially an extension of (Trivedi et al. 2017), adding attention to provide self-exciting rates, applied to two types of edges (communication edges and “friendship” edges). Conditioned on past edges, future edges are assumed independent, which makes the math trivial. The work would be better described as modeling a Marked Point Process with marks k \in {0,1}.
Other comments:
1.	[addressed] DyRep-No-SP is as good as the proposed approach, maybe because the graph is assumed undirected and the embedding of u can be described by its neighbors (author rebuttal describes as Localized Propagation), as the neighbors themselves use the embedding of u for their own embedding (which means that self-propagation is never "really off"). Highly active nodes have a disproportional effect in the embedding, resulting in the better separated embeddings of Figure 4. [after rebuttal: what is the effect of node activity on the embeddings?]
2.	[unresolved, comment still misundertood] The Exogenous Drive W_t(t_p – t_{p−1}) should be more personalized. Some nodes are intrinsically more active than others. [after rebuttal: answer "$W_t(t_p - t_{p-1})$ is personalized as $t_p$ is node specific", I meant personalized as in Exogenous Drive of people like Alice or Bob]
3.	[unresolved] Fig 4 embeddings should be compared against (Trivedi et al. 2017) [after rebuttal: author revision does not make qualitative comparison against Trivedi et al. (2017)]
Besides the limited innovation, the writing needs work.
4.	[resolved] Equation 1 defines $g_k(\bar{t})$ but does not define \bar{t}. Knowing (Trivedi et al. 2017), I immediately knew what it was, but this is not standard notation and should be defined.
5.	[resolved] $g_k$ must be a function of u and v
6.	[resolved] “$k$ represent the dynamic process” = >  “$k$ represent the type of edge” . The way it is written $k$ would need to be a stochastic process (it is just a mark, k \in {0,1})
7.	[resolved] Algorithm 1 is impossibly confusing. I read it 8 times and I still cannot tell what it is supposed to do. It contains recursive definitions like $z_i = b + \lambda_k^{ji}(t)$, where $\lambda_k^{ji}(t)$ itself is a function of $z_i(t)$. Maybe the z_i(t) and z_i are different variables with the same name?
8.	[resolved] The only hint that the graph under consideration is undirected comes from Algorithm 1, A_{uv}(t) = A_{vu}(t) = 1. It is *very* important information for the reader.
Related work (to be added to literature):
Dynamic graph embedding: (Yuan et al., 2017) (Ghassen et al., 2017)
Dynamic sub-graph embedding: (Meng et al., 2018)
Minor:
state-of-arts => state-of-the-art methods
list enumeration “1.)” , “2.)” is strange. Decide either 1) , 2) or 1. , 2. . I have never seen both.
MAE => mean absolute error (MAE)
Yuan, Y., Liang, X., Wang, X., Yeung, D. Y., & Gupta, A., Temporal Dynamic Graph LSTM for Action-Driven Video Object Detection. ICCV, 2017.
Jerfel,  , Mehmet E. Basbug, and Barbara E. Engelhardt. "Dynamic Collaborative Filtering with Compound Poisson Factorization." AISTATS 2017.
Meng, C., Mouli, S.C., Ribeiro, B. and Neville, J., Subgraph Pattern Neural Networks for High-Order Graph Evolution Prediction. AAAI 2018.
--- --- After rebuttal
Authors addressed most of my concerns. The paper has merit and would be of interest to the community. I am increasing my score.


Hierarchical clustering is a common approach to analysing the multi-scale structure of graphs observed in practice. We propose a novel metric for assessing the quality of a hierarchical clustering. This metric reflects the ability to reconstruct the graph from the dendrogram encoding the hierarchy. The best representation of the graph for this metric in turn yields a novel hierarchical clustering algorithm. Experiments on both real and synthetic data illustrate the efficiency of the approach.

This paper proposes a new formulation of hierarchical clustering for graphs.
Quality:
The proposed formulation has not been analyzed in detail and its advantage is not clear compared to existing approaches.
In addition, there are some existing measures for hierarchical clustering, for example, the dendrogram purity[1].
It would be interesting to analyze the relationship between the proposed method and such existing measures.
[1] Heller, K. A. and Ghahramani, Z.: Bayesian Hierarchical Clustering, ICML2005
Clarity:
This paper is clearly written and easy to read.
The proposed criteria is carefully derived and well explained.
I feel that the title of the paper is not appropriate as it says the paper is about graph representation learning while a graph (representation) is already given as an input in the setting discussed in this paper. "Learning hierarchical representation ..." would be better.
Originality:
The originality is not high as most of theoretical discussion is based on the existing work and the resulting hierarchical clustering algorithm is a straightforward extension of the average linkage method.
Of course, it is quite interesting if a minor change makes a big difference in clustering performance (theoretically and/or empirically), but such result is not given.
Significance:
Significance of the contribution is not high as the advantage of the proposed formulation is not clear.
One of interesting questions is: how about higher order relationships between nodes?
The proposed method takes up to second order relationships between nodes, that is, edges into account.
Since the proposed formulation can naturally include higher order relationships, it would be interesting to analyze such relationships in hierarchical clustering.
Pros:
- The paper is clearly written.
- The proposed formulation of hierarchical clustering is interesting.
Cons:
- The advantage of the proposed formulation is not presented.
- Experiments are not thorough.


ANTISYMMETRICRNN: A DYNAMICAL SYSTEM VIEW ON RECURRENT NEURAL NETWORKS
Recurrent neural networks have gained widespread use in modeling sequential data. Learning long-term dependencies using these models remains difficult though, due to exploding or vanishing gradients. In this paper, we draw connections between recurrent networks and ordinary differential equations. A special form of recurrent networks called the AntisymmetricRNN is proposed under this theoretical framework, which is able to capture long-term dependencies thanks to the stability property of its underlying differential equation. Existing approaches to improving RNN trainability often incur significant computation overhead. In comparison, AntisymmetricRNN achieves the same goal by design. We showcase the advantage of this new architecture through extensive simulations and experiments. AntisymmetricRNN exhibits much more predictable dynamics. It outperforms regular LSTM models on tasks requiring long-term memory and matches the performance on tasks where short-term dependencies dominate despite being much simpler.

In this paper, the authors provide a new approach to analyze the behavior of
RNNs by relating the RNNs with ODE numerical schemes. They provide analysis on
the stability of forward Euler scheme and proposed an RNN architecture called
AntisymmetricRNN to solve the gradient exploding/vanishing problem.
The paper is well presented although more recent works in this direction
should be cited and discussed. Also, some important issues are omitted and not
explained.
For example, the analysis begins with "RNNs with feedback" rather than vanilla
RNN, since vanilla RNN does not have the residual structure as eq(3). The
authors should note that clearly in the paper.
Although there are previous works relating ResNets with ODEs, such as [1],
this paper is original as it is the first work that relates the stability of
ODE numerical scheme with the gradient vanishing/exploding issues in RNNs.
In general, this paper provides a novel approach to analyze the gradient
vanishing/exploding issue in RNNs and provides applicable solutions, thus I
recommend to accept it.
Detailed comments:
The gradient exploding/vanishing issue has been extensively studied these
years and more recent results should be discussed in related works.
Author mentioned that existing methods "come with significant computational
overhead and reportedly hinder representation power of these models". However
this is not true for [2] which achieves full expressive power with
no-overhead.
It is true that "orthogonal weight matrices alone does not prevent exploding
and vanishing gradients", thus there are architectural approaches that can
bound the gradient norm by constants [3].
The authors argued that the critical criterion is important in preserving the
gradient norm. However, later on added a diffusion term to maintain the
stability of forward Euler method. Thus the gradient will vanish
exponentially w.r.t. time step t as: (1-\gamma)^t. Could the authors provide
more detailed analysis on this issue?
Since eq(3) cannot be regarded as vanilla RNN, it would be better begin the
analysis with advanced RNN architectures that fit in this form, such as
Residual RNN, Statistical Recurrent Units and Fourier Recurrent Units.
Why sharing the weight matrix of gated units and recurrent units? Is there any
other reason to do this other than reducing the number of parameters?
More experiment should be conducted on real applications of RNN, such as
language model or machine translation.
[1] Yiping Lu, Aoxiao Zhong, Quanzheng Li, and Bin Dong. Beyond finite layer
neural networks: Bridging deep architectures and numerical differential
equations. In ICML, pp. 3276–3285, 2018.
[2] Zhang, Jiong, Qi Lei, and Inderjit S. Dhillon. "Stabilizing Gradients for
Deep Neural Networks via Efficient SVD Parameterization." In ICML, pp.
5806-5814, 2018.
[3] Zhang, Jiong, Yibo Lin, Zhao Song, and Inderjit S. Dhillon. "Learning Long
Term Dependencies via Fourier Recurrent Units." In ICML, pp. 5815-5823, 2018.


GENERATING HIGH FIDELITY IMAGES WITH SUBSCALE PIXEL NETWORKS AND MULTIDIMENSIONAL UPSCALING
The unconditional generation of high fidelity images is a longstanding benchmark for testing the performance of image decoders. Autoregressive image models have been able to generate small images unconditionally, but the extension of these methods to large images where fidelity can be more readily assessed has remained an open problem. Among the major challenges are the capacity to encode the vast previous context and the sheer difficulty of learning a distribution that preserves both global semantic coherence and exactness of detail. To address the former challenge, we propose the Subscale Pixel Network (SPN), a conditional decoder architecture that generates an image as a sequence of sub-images of equal size. The SPN compactly captures image-wide spatial dependencies and requires a fraction of the memory and the computation required by other fully autoregressive models. To address the latter challenge, we propose to use Multidimensional Upscaling to grow an image in both size and depth via intermediate stages utilising distinct SPNs. We evaluate SPNs on the unconditional generation of CelebAHQ of size 256 and of ImageNet from size 32 to 256. We achieve state-of-the-art likelihood results in multiple settings, set up new benchmark results in previously unexplored settings and are able to generate very high fidelity large scale samples on the basis of both datasets.

Summary:
This paper addresses an important problem in density estimation which is to scale the generation to high fidelity images. Till now, there have been no good density modeling results on large images when taken into account large datasets like Imagenet (there have been encouraging results like with Glow, but on 5-bit color intensities and simpler datasets like CelebA). This paper is the first to successfully show convincing Imagenet samples with 128x128 resolution for a likelihood density model, which is hard even for a GAN (only one GAN paper (SAGAN) prior to this conference has managed to show unconditional 128x128 Imagenet samples). The ideas in this paper to pick an ordering scheme at subsampled slices uniformly interleaved in the image and condition slice generation in an autoregressive way is very likely to be adopted/adapted to more high fidelity density modeling like videos. Another important idea in this paper is to do depth upscaling, focusing on salient color intensity bits first (first 3 bits per color channel) before generating the remaining bits. The color intensity dependency structure is also neat: The non-salient bits per channel are conditioned on all previously generated color bits (for all spatial locations). Overall, I think this paper is a huge advance in density modeling, deserves an oral presentation and deserves as much credit as BigGAN, probably more, given that it is doing unconditional generation.
Details:
Major:
-1. Can you point out the total number of parameters in the models? Also would be good to know what hardware accelerators were used. The batch sizes mentioned in the Appendix (2048 for 256x256 Imagenet) are too big and needs TPUs? If TPU pods, which version (how many cores)? If not, I am curious to know how many GPUs were used.
0. I would really like to know the sampling times. The model still generates the image pixel by pixel. Would be good to have a number for future papers to reference this.
1. Any reason why 256x256 Imagenet samples are not included in the paper? Given that you did show 256x256 CelebA samples, sampling time can't be an issue for you to not show Imagenet 256x256. So, it would be nice to include them. I don't think any paper so far has shown good 256x256 unconditional samples. So showing this will make the paper even stronger.
2. Until now I have seen no good 64x64 Imagenet samples from a density model. PixelRNN samples are funky (colorful but no global structure). So I am curious if this model can get that. It may be the case that it doesn't, given that subscale ordering didn't really help on 32x32.  It would be nice to see both 5-bit and 8-bit, and for 8-bit, both the versions: with and without depth upscaling.
3. I didn't quite understand the architecture in slice encoding (Sec 3.2).  Especially the part about using a residual block convnet to encode the previous slices with padding, and to preserve relative meta-position of the slices. The part I get is that you concatenate the 32x32 slices along the channel dimension, with padded slices. I also get that padding is necessary to have the same channel dimension for any intermediate slice. Not sure if I see the whole point of preserving ordering. Isn't it just normal padding -> space to depth in a structured block-wise fashion?
4. Can you clarify how you condition the self-attention + Gated PixelCNN block on the previous slice embedding you get out of the above convnet? There are two embeddings passed in if I understand correctly: (1) All previous slices, (2) Tiled meta-position of current slice.  It is not clear to me how the conditioning is done for the transformer pixelcnn on this auxiliary embedding. The way you condition matters a lot for good performance, so it would be helpful for people to replicate your results if you provide all details.
5. I also don't understand the depth upscaling architecture completely. Could you provide a diagram clarifying how the conditioning is done there given that you have access to all pixels' salient bits now and not just meta-positions prior to this slice?
6. It is really cool that you don't lose out in bits/dim after depth upscaling that much. If you take Grayscale PixelCNN (pointed out in the anonymous comment), the bits/dim isn't as good as PixelCNN though samples are more structured. There is 0.04 b.p.d  difference in 256x256, but no difference in 128x128. Would be nice to explain this when you add the citation.
7. The architecture in the Appendix can be improved. It is hard to understand the notations. What are residual channels, attention channels, attention ffn layer, "parameter attention", conv channels?
Minor:
Typo: unpredented --> unprecedented


Our goal is to train a policy for autonomous driving via imitation learning that is robust enough to drive a real vehicle. We find that standard behavior cloning is insufficient for handling complex driving scenarios, even when we leverage a perception system for preprocessing the input and a controller for executing the output on the car: 30 million examples are still not enough. We propose exposing the learner to synthesized data in the form of perturbations to the expert's driving, which creates interesting situations such as collisions and/or going off the road. Rather than purely imitating all data, we augment the imitation loss with additional losses that penalize undesirable events and encourage progress -the perturbations then provide an important signal for these losses and lead to robustness of the learned model. We show that the model can handle complex situations in simulation, and present ablation experiments that emphasize the importance of each of our proposed changes and show that the model is responding to the appropriate causal factors. Finally, we demonstrate the model driving a car in the real world.

The paper describes a framework for training a self-driving policy by augmenting imitation loss with additional loss terms that penalize undesired behaviors and that encourage progress. The policy takes as input a parsed representation of the scene (rather than raw images) and outputs pose trajectories for a down-stream controller. The method is trained on simulated data that includes perturbations to improve generalizability. The framework is evaluated in simulation through a series of ablations to better understand the contribution of the different loss terms.
STRENGTHS
+ Paper acknowledges the difficulty of end-to-end (pixels-to-torque) learning for autonomous driving and instead reasons over pre-processed inputs in the form of lower-dimensional images (and image sequences) that capture obstacles as bounding boxes and simple lines for routes, grayscale intensities, etc. Similarly, the output is a trajectory that is then fed to a controller responsible for tracking this trajectory.
WEAKNESSES
- The insufficiency of behavioral cloning is not surprising, as noted, given the covariate shift. It would be interesting to consider a  no-regret formulation analogous to Ross et al., 2011, even though it would require interaction with a human.
- The limitation of producing paths in this way is that the network does not explicitly reason over the feasibility of the path, which is important for non-holonomic vehicles. Instead, the network must learn the kinematic and dynamic constraints.
- Perturbations of the simulated trajectories are used to expose the model to collisions and other rare events, but is not clear that simple trajectory perturbations such as those used here provide a sufficient exposure to these rare events.
- The fact that the 2D image that expresses the vehicle's position is absolute limits the environment in which the network is valid. The experiments are conducted on images corresponding to an 80m x 80m environment, which is trivially small.
- The proposed framework is highly specific to self-driving and the extent to which it provides insights for other domains is not clear.
- The ablation experiments are not very compelling. In the case of the nudging experiment, all models result in collisions with M4 being the best model with a 10% collision rate. The trajectory perturbation results are better. In the case of the slowing experiment, M3 is the only version to not result in collision, whereas M4 collides 5% of the time. It isn't clear than which model is preferable since, while M3 never collides in the case of the slowing down experiment, it collides 45% of the time in the nudging experiment, almost as frequently as the M0 baseline.
- The paper claims that the model was run on a real robot, but there is no experimental evaluation of the results, only a reference to videos. The results of these experiments should be quantified and discussed or the reference to running on a real vehicle should be toned down, if not removed.
- Equation 3 requires knowledge of the ground-truth distribution. How is this determined?


While neural networks can be trained to map from one specific dataset to another, they usually do not learn a generalized transformation that can extrapolate accurately outside the space of training. For instance, a generative adversarial network (GAN) exclusively trained to transform images of black-haired men to blondhaired men might not have the same effect on images of black-haired women. This is because neural networks are good at generation within the manifold of the data that they are trained on. However, generating new samples outside of the manifold or extrapolating "out-of-sample" is a much harder problem that has been less well studied. To address this, we introduce a technique called neuron editing that learns how neurons encode an edit for a particular transformation in a latent space. We use an autoencoder to decompose the variation within the dataset into activations of different neurons and generate transformed data by defining an editing transformation on those neurons. By performing the transformation in a latent trained space, we encode fairly complex and non-linear transformations to the data with much simpler distribution shifts to the neuron's activations. We motivate our technique on an image domain and then move to our two main biological applications: removal of batch artifacts representing unwanted noise and modeling the effect of drug treatments to predict synergy between drugs.

The authors present a way to transform data from a source distribution to have characteristics of a target distribution.
This is accomplished by applying a "NeuronEdit" function to the encoding of the input; this edited input is then decoded.
The NeuronEdit function is parametrized by the target distribution's statistics. The edit function does a sort of simple histogram matching, so that the ith percentile values of the source distribution's bottleneck representations instead become the ith percentile values of the target distribution's bottleneck representations.
Experiments are on CIFAR-10 and biology datasets (the latter of which are not my strong suit).
This paper is well-written and original. It is original because there are only a few works which directly manipulate the latent space (one example is latent space interpolation used to visualize GANs), and this is distinct from those.
The problem they aim to solve also has not received much attention, which enhances the novelty of this paper.
The presented method is simple and easy to implement, since the editing function is not learned but is instead deterministic. It is encapsulated in Equation 1.
The fact that the editing function is fixed may greatly hinder its flexibility and applicability.
In Section 3.1 and Figure 1, we are shown that NeuronEditting can turn images of horses with white backgrounds into images of horses with dark backgrounds (horses are an unseen class).
NeuronEditting turns the horse darker as well. It seems that one could change the brightness and contrast of the image to obtain a similar effect, or one could or take the geometric mean of the image in [0,1] with the average target image and obtain a similar effect. Such traditional methods are also robust to unseen classes. Moreover, NeuronEditting's ability to change the brightness of the image is not that surprising given that brightness is some of the most basic image information. (In point of fact it is captured by the DC Coefficient, the very first coefficient from the discrete cosine transform which is used in JPEG.) What else can NeuronEditting do in the image domain? Can this be used to rotate or reflect MNIST digits? The biological experiments also appear to involve simple input transformations.
Fine points:
- "an edit function between the the"
- I am not sure the speculation about this method's loose relation to word2vec belongs in a scientific work. Both involve modifications to a neural representations, but no further relation is justified in the paper.
- Was the dataset partitioning for the CIFAR-10 experiment done manually? If not, what process partitioned the dataset?
Edit: Some of my suggestions were incorporated in the rebuttal, but my sentiment is still that this is almost at the acceptance threshold. The large focus on biology makes much of this paper harder to evaluate or appreciate.


With the rapidly scaling up of deep neural networks (DNNs), extensive research studies on network model compression such as weight pruning have been performed for efficient deployment. This work aims to advance the compression beyond the weights to the activations of DNNs. We propose the Integral Pruning (IP) technique which integrates the activation pruning with the weight pruning. Through the learning on the different importance of neuron responses and connections, the generated network, namely IPnet, balances the sparsity between activations and weights and therefore further improves execution efficiency. The feasibility and effectiveness of IPnet are thoroughly evaluated through various network models with different activation functions and on different datasets. With < 0.5% disturbance on the testing accuracy, IPnet saves 71.1% ∼ 96.35% of computation cost, compared to the original dense models with up to 5.8× and 10× reductions in activation and weight numbers, respectively. The source codes are available at (omitted for blind review).

This article presents a novel approach called Integral Pruning (IP) to reduce the computation cost of Deep neural networks (DNN) by integrating activation pruning along with weight pruning. The authors show that common techniques of exclusive weight pruning does compress the model size, but increases the number of non-zero activations after ReLU. This would counteract the advantage of DNN accelerator designs (Albericio et al., 2016; Reagen et al., 2016) that leverage activation sparsity to speed up the computations. IP starts with pruning the weights using an existing technique to mask out weights under a threshold and then fine-tune the network in an iterative fashion to maintain the accuracy. After weight pruning, IP further masks out the activations with smaller magnitude to reduce the computation cost. Unlike weight pruning techniques that use static masks, the authors propose to use dynamic activation masks for activation sparsity in order to account for various patterns that are being activated in DNN for different input samples. In order to do this, the 'winner rate' measure for every layer (or for a group of layers in deep networks like ResNet32) is defined, to dynamically set the threshold for the generation of activation masks which eventually controls the amount of non-zero activation entries. The article empirically analyzes the sensitivity of activation pruning on validation data by setting different winner rates at every layer in DNN and decides upon a set of winner rates accordingly followed by an iteration of fine-tuning the network to maintain its performance. The authors show that their technique produced lower number of non-zero activations in comparison with the intrinsic sparse ReLU activations and weight pruning techniques.
The topic of reducing network complexity for embedded implementations of DNNs is highly relevant, in particular for the ICLR community.
The IP technique yields significantly reduced number of multiply-accumulate operations (MACs) across different models like MLP-3, ConvNet-5, ResNet32 and AlexNet and on different datasets like MNIST, CIFAR10 and ImageNet. They also depicted that pruning the activations with dynamic activation masks followed by fine-tuning the network yields more sparse activations and negligible loss in accuracy when compared against using static activation masks.

Strengths of the paper:
- The motivation to extend compression beyond the weights to activations in order to support the DNN accelerator designs and the technical details are clearly explained.
- The proposed technique indeed produces sparser activations than intrinsic ReLu sparse activations and can also applied to any network regardless of the choice of activation function.
- The proposed technique is evaluated across different network architectures and datasets.
- The advantage of adapting dynamic activation masks over static ones is clearly demonstrated.
 Weaknesses of the paper:
- The originality of the approach is limited because it is a relatively straightforward combination of existing techniques for weight and activation pruning.
- The "winner rate" measure is defined for every layer and should be explored over different values in order to find the equilibrium to reduce the number of non-zero activations and maintain the accuracy. This search of winner rates will become inefficient as the depth of the network increases. However, the authors used a single winner rate for a group of layers in case of ResNet-32 to reduce the exploration of search space but this choice might lead to suboptimal results.
- The authors compare the resultant number of MAC operations against numbers from the weight pruning technique. However, there also exist different works on group pruning techniques like Liu et al. (2017), Huang & Wang (2017), Ye et al. (2018) to prune entire channels / feature maps and thus yield more compact networks. Since these approaches prune the channels, they show a direct impact on the computation complexity and greatly reduce the computation time. A proper and fair comparison would be to compare the numbers of IP against such group pruning techniques. This comparison is highly important to highlight the significance of the approach on speeding up the DNNs and it is missing from the paper.
- At several locations in Section 4, e.g. Sec. 4.1, 4.3, and 4.4. there is no precise statement about the incurred accuracy loss (or no statement at all). The connection to Figures 4 and 5 is not immediately clear and should be made explicit.

References:
- Zhuang Liu, Jianguo Li, Zhiqiang Shen, Gao Huang, Shoumeng Yan, and Changshui Zhang. Learning efficient convolutional networks through network slimming.
- Jianbo Ye, Xin Lu, Zhe Lin, and James Z Wang. Rethinking the smaller-norm-less-informative assumption in channel pruning of convolution layers
- Zehao Huang and Naiyan Wang. Data-driven sparse structure selection for deep neural networks.

Overall Evaluation:
The authors integrate activation pruning along with the weight pruning and show that the number of MAC operations are greatly reduced by their technique when compared to the numbers of weight pruning alone. 	However, I am not convinced regarding the reported number of MAC operations since the number of MAC operations of sparse weight matrices and activations would remain the same as the original models unless some of the filters/activation maps are pruned from the network.  On the other hand, comparisons against group pruning techniques are highly necessary to evaluate the potential impact of the approach on speeding up of DNNs. My preliminary rating is a weak reject but I am open to revise my rating based on the authors response to the above stated major weaknesses.
Minor comments:
- Caption of Fig. 4 should mention the task on which the results were obtained.
- There are occasional grammar errors and typos that should be corrected.


REVEALING INTERPRETABLE OBJECT REPRESENTA- TIONS FROM HUMAN BEHAVIOR
To study how mental object representations are related to behavior, we estimated sparse, non-negative representations of objects using human behavioral judgments on images representative of 1,854 object categories. These representations predicted a latent similarity structure between objects, which captured most of the explainable variance in human behavioral judgments. Individual dimensions in the low-dimensional embedding were found to be highly reproducible and interpretable as conveying degrees of taxonomic membership, functionality, and perceptual attributes. We further demonstrated the predictive power of the embeddings for explaining other forms of human behavior, including categorization, typicality judgments, and feature ratings, suggesting that the dimensions reflect human conceptual representations of objects beyond the specific task.

This is a paper that communicates a large scale experiment on human object/semantic representations and a model of such representations.   The experiment could have been more carefully controlled (and described in the paper) and the modeling work is inconclusive.
Quality,
The experiment design is conventional, based on rating pair-wise similarity among triplets. Compared to earlier experiments, this data has more objects and more triplets.  Additional control experiments on smaller subsets have been carried out to further address hypotheses.  The description of the experiment could have been more careful: What are the precise instructions, how are the object/images presented (it is well known that relative positions, asymmetry, etc can play an important role), are there any temporal/learning effects (how clear is the task to the workers?).
The modeling work is basic and contains a number of steps that have unknown influence on the final outcome. For example model dimension: Is you claim that "D=49" is a law of human nature?  Model predictive performance seems excellent, that is interesting! But we do not know how robust this is to the many heuristics
Clarity,
The presentation of the inference process is clear. Not so clear what the uncertainties are
Originality
Limited. Mainly related to scale. But the data quality is unclear. The modeling approach involves a number of untested heuristics (non-negative, exponentiation etc).
Significance
Mostly related to the data.  I did not understand if it is planned to release the data.
Pros and cons
+Large scale experiment
+simple model, seem to have good accuracy
-experiment needs more careful description
-too many heuristics in model and inference, unclear how general the conclusions are
Other comments:
References have many issues
The authors have done a good job in the revision and have clarified points that were unclear in the first version.
I have remaining reservations on significance, but move rating up a notch to reflect the extensive improvements and  the authors' confirmation that they will release the data.


Neural network-based language models deal with data sparsity problems by mapping the large discrete space of words into a smaller continuous space of realvalued vectors. By learning distributed vector representations for words, each training sample informs the neural network model about a combinatorial number of other patterns. In this paper, we exploit the sparsity in natural language even further by encoding each unique input word using a fixed sparse random representation. These sparse codes are then projected onto a smaller embedding space which allows for the encoding of word occurrences from a possibly unknown vocabulary, along with the creation of more compact language models using a reduced number of parameters. We investigate the properties of our encoding mechanism empirically, by evaluating its performance on the widely used Penn Treebank corpus. We show that guaranteeing approximately equidistant vector representations for unique discrete inputs is enough to provide the neural network model with enough information to learn -and make use-of distributed representations for these inputs.

This paper studied a random projection of word embeddings in neural language modeling. Instead of having |V| x m embeddings, the author(s) represented a word with a random, sparse, linear combination {1, 0, -1} of k vector of size m. The experiment on PTB dataset showed that k had to be somewhat close to |V| in order to achieve the comparable perplexity to a feed-forward NLM.
Overall, I am not sure what we could gain from this research direction. The advantage of this random encoding was to reduce the number of parameters for an embedding layer, but the results showed we gained much PPL from a 25% reduction in embedding size (Table 1). In addition, the fact that the random projections preserved the inner product (centered at zero) was probably not desirable. It might be more fruitful if these linear combinations were learned or sub-senses of words (e.g. [1]).
The experiments were quite extensive on the hyper-parameters and showed how the models performed under different settings. However, these were done using 1 dataset and also a simple feed-forward network (rather than LSTM). I can understand the point that training NNLM accelerates the experiments, but the author(s) should consider trying a simply LSTM model after the best settings had been discovered (e.g. Table 1). PTB also has a very unnatural vocabulary distribution as pointed out in [2]. Thus, it might be helpful to test the result on another dataset (e.g. WikiText).
Other comments
1. I do not get the point of bringing up NCE. Did you actually use NCE loss? Did you only refer to NCE as a weight tying which can be used in a standard XENT loss [3]? The first paragraph of 3.3 did not help clarify this point either.
2. In Figure 3, the baseline got different perplexity between 3(a) and 3(b).
3. Shouldn't random indexing produce non-uniform numbers of non-zero entries depending on alpha? Why did you have an exact number of non-zero entries, s, in the experiments?
3. Some typos
- "... is that instead of trying to probability ..." => "... tying ..."
- "... All models sare trained ..." => "... are ..."
- "... Tho get the feature ..." => ?
References
[1] S. Arora et al., 2016. Linear Algebraic Structure of Word Senses, with Applications to Polysemy
[2] S. Merity et al., 2016. Pointer Sentinel Mixture Models
[3] Y. Gal et al., 2015. A Theoretically Grounded Application of Dropout in Recurrent Neural Networks


ADASHIFT: DECORRELATION AND CONVERGENCE OF ADAPTIVE LEARNING RATE METHODS
Adam is shown not being able to converge to the optimal solution in certain cases. Researchers recently propose several algorithms to avoid the issue of nonconvergence of Adam, but their efficiency turns out to be unsatisfactory in practice. In this paper, we provide a new insight into the non-convergence issue of Adam as well as other adaptive learning rate methods. We argue that there exists an inappropriate correlation between gradient g t and the second moment term v t in Adam (t is the timestep), which results in that a large gradient is likely to have small step size while a small gradient may have a large step size. We demonstrate that such unbalanced step sizes are the fundamental cause of non-convergence of Adam, and we further prove that decorrelating v t and g t will lead to unbiased step size for each gradient, thus solving the non-convergence problem of Adam. Finally, we propose AdaShift, a novel adaptive learning rate method that decorrelates v t and g t by temporal shifting, i.e., using temporally shifted gradient g t−n to calculate v t . The experiment results demonstrate that AdaShift is able to address the non-convergence issue of Adam, while still maintaining a competitive performance with Adam in terms of both training speed and generalization.

This manuscript contributes a new online gradient descent algorithm with adaptation to local curvature, in the style of the Adam optimizer, ie with a diagonal reweighting of the gradient that serves as an adaptive step size. First the authors identify a limitation of Adam: the adaptive step size decreases with the gradient magnitude. The paper is well written.
The strengths of the paper are a interesting theoretical analysis of convergence difficulties in ADAM, a proposal for an improvement, and nice empirical results that shows good benefits. In my eyes, the limitations of the paper are that the example studied is a bit contrived and as a results, I am not sure how general the improvements.
# Specific comments and suggestions
Under the ambitious term "theorem", the results of theorem 2 and 3 limited to the example of failure given in eq 6. I would have been more humble, and called such analyses "lemma". Similarly, theorem 4 is an extension of this example to stochastic online settings. More generally, I am worried that the theoretical results and the intuitions backing the improvements are built only on one pathological example. Are there arguments to claim that this example is a prototype for a more general behavior?
Ali Rahimi presented a very simple example of poor perform of the Adam optimizer in his test-of-time award speech at NIPS this year (https://www.youtube.com/watch?v=Qi1Yry33TQE): a very ill-conditioned factorized linear model (product of two matrices that correspond to two different layers) with a square loss. It seems like an excellent test for any optimizer that tries to be robust to ill-conditioning (as with Adam), though I suspect that the problem solved here is a different one than the problem raised by Rahimi's example.
With regards to the solution proposed, temporal decorrelation, I wonder how it interacts with mini-batch side. With only a light understanding of the problem, it seems to me that large mini-batches will decrease the variance of the gradient estimates and hence increase the correlation of successive samples, breaking the assumptions of the method.
Using a shared scalar across the multiple dimensions implies that the direction of the step is now the same as that of the gradient. This is a strong departure compared to ADAM. It would be interesting to illustrate the two behaviors to optimize an ill-conditioned quadratic function, for which the gradient direction is not a very good choice.
The performance gain compared to ADAM seems consistent. It would have been interesting to see Nadam in the comparisons.
I would like to congratulate the authors for sharing code.
There is a typo on the y label of figure 4 right.


Building upon the recent success of deep reinforcement learning methods, we investigate the possibility of on-policy reinforcement learning improvement by reusing the data from several consecutive policies. On-policy methods bring many benefits, such as ability to evaluate each resulting policy. However, they usually discard all the information about the policies which existed before. In this work, we propose adaptation of the replay buffer concept, borrowed from the off-policy learning setting, to create the method, combining advantages of on-and off-policy learning. To achieve this, the proposed algorithm generalises the Q-, value and advantage functions for data from multiple policies. The method uses trust region optimisation, while avoiding some of the common problems of the algorithms such as TRPO or ACKTR: it uses hyperparameters to replace the trust region selection heuristics, as well as the trainable covariance matrix instead of the fixed one. In many cases, the method not only improves the results comparing to the state-of-the-art trust region on-policy learning algorithms such as PPO, ACKTR and TRPO, but also with respect to their off-policy counterpart DDPG.

The authors introduce a off-policy method for TRPO by suggesting to use replay buffers to store trajectories and sample from them during training. To do this they extend the definition of the Q function to multiple policies where the Q_pi bar is then the expectation over the several policies. They propose the same for the value function and consequently the advantage function.
In my opinion this is some interesting work, but there are some details that are not clear to me, so i have several questions.
1. Why is it necessary to define these generalized notions of the Q, Value and Advantage functions? You motivate this by the fact the samples stored in the replay buffer will be generated by different policies, i.e. by differently parametrized policies at a certain time step. But this also holds almost all algorithms using replay buffers. Could you plese explain this part further?
2. In eq. (26) you introduce the parameter alpha as a sort of Lagrange multiplier to turn the unconstrained optimization problem defined by TRPO into a constrained one. This is was also proposed early by Schulman et al. in Proximal Policy Optimization. Yet, it is not cited or referenced. In the discussion of the experimental results go further into this. Please explain this part in more detail.
3. Another point of your work is the learnable diagonal covariance matrix. How can you be sure that the improvements you show are due to the replay buffers and not due to learning these? Or learning covariance in combination with the penalty term alpha?
4. Can you provide comparative results for PPO? PPO outperforms DDP and TRPO on most tasks so it would be interessting to see
5. How many trajectory samples do you store in the replay buffers? Can you provide results where you use your method but without any replay buffers, i.e. by using the last batch of data points?
Minor Suggestions:
- The references for the figures in the Experiments part are off. In fig. 1 you cite Todorov et al. for Mujoco but not TRPO and ACKTR, the same in fig. 2. Then in fig. 3 you cite DDPG also with Todorov et al.
- Some parts of the text is a bit unorganized. In section 2.1 you introduce AC algorithms and on the next page you give the definitions for all components but you don't say anything about how the interact. Also, the definition of the expected return was not "invented" by Schulman et al, and neither were Advantages, Q-, and Value functions. Maybe add a second or third reference.


Building on the success of deep learning, two modern approaches to learn a probability model of the observed data are Generative Adversarial Networks (GANs) and Variational AutoEncoders (VAEs). VAEs consider an explicit probability model for the data and compute a generative distribution by maximizing a variational lower-bound on the log-likelihood function. GANs, however, compute a generative model by minimizing a distance between observed and generated probability distributions without considering an explicit model for the observed data. The lack of having explicit probability models in GANs prohibits computation of sample likelihoods in their frameworks and limits their use in statistical inference problems. In this work, we show that an optimal transport GAN with the entropy regularization can be viewed as a generative model that maximizes a lower-bound on average sample likelihoods, an approach that VAEs are based on. In particular, our proof constructs an explicit probability model for GANs that can be used to compute likelihood statistics within GAN's framework. Our numerical results on several datasets demonstrate consistent trends with the proposed theory.

The contribution of the paper is to show that WGAN with entropic regularization maximize a lower bound on the likelihood of the observed data distribution. While the WGAN formulation minimizes the Wasserstein distance of the transformed latent distribution and the empirical distribution which is already a nice measure of "progress", having a bound on the likelihood can be interesting.
Pros:
+ I like the entropic GAN formulation and believe it is very interesting as it gives access to the joint distribution of latent and observed variables.
+ While there are some doubtful statements, overall the paper is well written and easy to read.
Cons:
- The assumption of injectivity of the generator could be problematic, as it might not be fulfilled due to mode collapse.
- I feel the theory is not very deep. Since one has a closed form of the transportation map (Eq. 3.7), the likelihood of the data is obtained by marginalizing out the latent space. However, this assumes that the inner dual maximization problem is solved to stationarity so that Eq 3.7 holds, which is not the case in practice (5 discriminator updates).
- Thus in Sec. 4.1 for the likelihood at various points in training it is not clear what is actually happening.
- Sec 4.3 for unregularized GANs might be problematic. In general, the transportation plan is not a density function, so I'm not certain whether Theorem 1 / Corollary 2 still hold. Furthermore, the heuristic for "inverting" G^* is very crude.
- There are also some minor problematic statements in the paper. While they can be easily fixed, they give me doubts:
  * The original VAE paper is not cited in the introduction for VAEs
  * The 2013 paper by Cuturi cited on page 2 has nothing to do with "computational aspects of GANs". It is about fast computation of approximate OT between two discrete prob. measures.
  * First-order / second-order Wasserstein distance is I think a bit unusual name for W_1, W_2
  * On pg. 4, the point of the entropy term is to make the objective strongly convex. Strict convexity has no computational benefits.


Recent work on bilingual lexicon induction (BLI) has frequently depended either on aligned bilingual lexicons or on distribution matching, often with an assumption about the isometry of the two spaces. We propose a technique to quantitatively estimate this assumption of the isometry between two embedding spaces and empirically show that this assumption weakens as the languages in question become increasingly etymologically distant. We then propose Bilingual Lexicon Induction with Semi-Supervision (BLISS) -a novel semi-supervised approach that relaxes the isometric assumption while leveraging both limited aligned bilingual lexicons and a larger set of unaligned word embeddings, as well as a novel hubness filtering technique. Our proposed method improves over strong baselines for 11 of 14 language pairs on the MUSE dataset, particularly for languages whose embedding spaces do not appear to be isometric. In addition, we also show that adding supervision stabilizes the learning procedure, and is effective even with minimal supervision.

This paper presents a new semi-supervised method to learn cross-lingual word embeddings mappings combining unsupervised distribution matching, alignment over a training dictionary, and a weak orthogonality constraint. The paper also shows that the underlying isometry assumption in orthogonal mappings weakens as the languages involved are more distant, proposes a new method to quantify the strength of the said assumption, and argues that the proposed semi-supervised mapping method is particularly suited for these more challenging cases.
I think that this is a solid work that explores an interesting research direction within cross-lingual embedding mappings. While the basic ingredients of the proposed method are not new, their combination is certainly original. In that regard, I think that the paper is rather incremental, but still has enough substance to make an interesting contribution. However, I think that some parts of the paper are too obscure, and I am not fully convinced by the experiments. I would appreciate if the authors could address my concerns below, and I would be happy to modify my score accordingly:
1) My understanding is that the proposed method (the one named BLISS in the experiments) only makes use of the proposed semi-supervised framework (Section 3.2) and is not followed by the iterative procrustes refinement (Section 3.3), but this is not clear at all from the paper. Could you clarify this?
2) It is well known that the retrieval method can have a big impact in bilingual dictionary induction due to the hubness problem. However, the paper does not detail which retrieval method is used in the experiments. I assume that MUSE uses CSLS and Vecmap uses nearest neighbor over cosine. Is this correct? What retrieval method does BLISS use?
3) I assume that when you talk about the "CSLS metric" in page 7 and 13 you refer to the unsupervised validation criterion of Lample et al. (Section 3.5 in their paper), and not to CSLS itself (Section 2.3 in their paper). In either case, this needs some clarification.
4) Unlike the "train" dictionaries, the "full" dictionaries from MUSE as provided at github also include the test set. Do you preprocess them to exclude the test set? If so, this should be clearly stated in the paper. If not, this would invalidate all these experiments.
5) The authors use different language pairs in their different result tables, which I find very confusing. For instance, none of the language pairs in table 5 (except for en-ru), are included in the main results (table 3), so we do not know how the different baselines and variants perform in them. Is there any reason for that?
6) Could you include all MUSE variants in Table 4?
7) While you compare your method to different versions of Vecmap (Artetxe et al., ACL 2017 & AAAI 2018), the last one (Artetxe et al., ACL 2018) (http://aclweb.org/anthology/P18-1073) is missing. That paper reports 48.1% and 48.2% accuracy for en-it and en-de in the unsupervised case, which is substantially better than your results for en-it (45.9%) and at par for en-de (48.3%). This goes against the main motivation of the paper (i.e. unsupervised distribution information and supervision from dictionaries can be combined for best results), as a completely unsupervised method seems to perform better than (or at least at par with) the proposed semi-supervised method. I think that the paper should include some discussion on this. In particular, I would like to know whether you have any argument to believe that both works are complementary.


While there has been an explosion in the number of experimentally determined, atomically detailed structures of proteins, how to represent these structures in a machine learning context remains an open research question. In this work we demonstrate that representations learned from raw atomic coordinates can outperform hand-engineered structural features while displaying a much higher degree of transferrability. To do so, we focus on a central problem in biology: predicting how proteins interact with one another-that is, which surfaces of one protein bind to which surfaces of another protein. We present Siamese Atomic Surfacelet Network (SASNet), the first end-to-end learning method for protein interface prediction. Despite using only spatial coordinates and identities of atoms as inputs, SASNet outperforms state-of-the-art methods that rely on hand-engineered, high-level features. These results are particularly striking because we train the method entirely on a significantly biased data set that does not account for the fact that proteins deform when binding to one another. Demonstrating the first successful application of transfer learning to atomic-level data, our network maintains high performance, without retraining, when tested on real cases in which proteins do deform.

For the task of predicting interaction contact among atoms of protein complex consisting of two interacting proteins, the authors propose to train a Siamese convolutional neural network, noted as SASNet, and to use the contact map of two binding proteins’ native structure.
The authors claim that the proposed method outperforms methods that use hand crafted features; also the authors claim that the proposed method has better transferability.
My overall concern is that the experiment result doesn’t really fully support the claim in the two aspects: 1) the SASNet takes the enriched dataset as input to the neural net but it also uses the complex (validation set) to train the optimal parameters, so strictly it doesn’t really fit in the “transfer” learning scenario. Also, the compared methods don’t really use the validation set from the complex data for training at all. Thus the experiment comparison is not really fair. 2) The experiment results include standard errors for different replicates where such replicates correspond to different training random seeds (or different samples from the enriched set?), however, it doesn’t include any significance of the sampling. Specifically, the testing dataset is fixed. A more rigorous setting is to, for N runs, each run splitting the validation and testing set differently.
Since this paper is an application paper, rather than a theoretical paper that bears theoretical findings, I would expect much more thorough experimental setup and analysis. Currently it is still missing discussion such as, when SASNet would perform better and when it would perform worse, what it is that the state of the art features can’t capture while SASNet can. Moreover, it is the prediction performance that matters to such task, but the authors remove the non-structure features from the compared methods. Results and discussion about how the previous methods with full features perform compared to SASNet, and also how we can include those features into SASNet should complete the paper.
Overall the paper is well written, and I do think the paper could be much stronger the issues above are addressed.
Some minor issues:
1)	on page 4, Section 3, the first paragraph, shouldn’t “C_p^{val} of 55” be “C_p^{test} of 55”?
2)	It is not clear what the “replicates” refer to in the experiments.
3)	Some discussion on why the “SASNet ensemble” would yield better performance would be good; could it be overfitting?


STRUCTURED NEURAL SUMMARIZATION
Summarization of long sequences into a concise statement is a core problem in natural language processing, requiring non-trivial understanding of the input. Based on the promising results of graph neural networks on highly structured data, we develop a framework to extend existing sequence encoders with a graph component that can reason about long-distance relationships in weakly structured data such as text. In an extensive evaluation, we show that the resulting hybrid sequence-graph models outperform both pure sequence models as well as pure graph models on a range of summarization tasks.

Note: I changed my original score from 4 to 7 based on the new experiments that answer many of the questions I had about the relative performance of each part of the model. The review below is the original one I wrote before the paper changes.
# Positive aspects of this submission
- The intuition and motivation behind the proposed model are well explained.
- The empirical results on the MethodNaming and MethodDoc tasks are very promising.
# Criticism
- The novelty of the proposed model is limited since it is essentially adding an existing GGNN layer, introduced by Li et al. (2015), on top of an existing LSTM encoder. The most important novelty seems to be the custom graph representation for these sequence inputs to make them compatible with the GGNN, which should then deserve a more in-depth study (i.e. ablation study with different graph representations, etc).
- Since you compare your model performance against Alon et al. on Java-small, it should be fair to report the numbers on Java-med and Java-large as well.
- The "GNN -> LSTM+POINTER" experiment results are reported on the MethodDoc task, but not for MethodNaming. Reporting this number for MethodNaming is essential to show the claimed empirical superiority of the hybrid encoder compared to GNN only.
- I have doubts about the usefulness of the proposed model for natural language summarization, for the following reasons:
    - The comparison of the proposed model for NLSummarization against See et al. is a bit unfair, since it uses additional information through the CoreNLP named entity recognizer and coreference models. With the experiments listed in Table 1, there is no way to know whether the increased performance is due to the hybrid encoder design or due the additional named entity and coreference information. Adding the entity and coreference data in a simpler way (i.e. at the token embedding level with a basic sequence encoder) in the ablation study would very useful to answer that question.
    - In NLSummarization, connecting sentence nodes using a NEXT edge can be analogous to using a hierarchical encoder, as used by Nallapati et al. ("Abstractive Text Summarization using Sequence-to-sequence RNNs and Beyond", 2016). Ignoring the other edges of the GNN graph, what are the theoretical and empirical advantages of your method compared to this sentence-level hierarchical encoder?
    - Adding the coverage decoder introduced by See et al. to your model would have been very useful to prove that the current performance gap is indeed due to the simplistic decoder and not something else.
- How essential is the weighted averaging for graph-level document representation (Gilmer et al. 2017) compared to uniform averaging?
- A few minor comments about writing:
    - In Table 1, please put the highest numbers in bold to improve readability
    - On page 7, the word "summaries" is missing in "the model produces natural-looking with no noticeable negative impact"
    - On page 9, "cove content" should be "core content"