DIVERSITY AND DEPTH IN PER-EXAMPLE ROUTING MODELS
Routing models, a form of conditional computation where examples are routed through a subset of components in a larger network, have shown promising results in recent works. Surprisingly, routing models to date have lacked important properties, such as architectural diversity and large numbers of routing decisions. Both architectural diversity and routing depth can increase the representational power of a routing network. In this work, we address both of these deficiencies. We discuss the significance of architectural diversity in routing models, and explain the tradeoffs between capacity and optimization when increasing routing depth. In our experiments, we find that adding architectural diversity to routing models significantly improves performance, cutting the error rates of a strong baseline by 35% on an Omniglot setup. However, when scaling up routing depth, we find that modern routing techniques struggle with optimization. We conclude by discussing both the positive and negative results, and suggest directions for future research.

The paper "Diversity and Depth in Per-Example Routing Models" extends previous work on routing networks by adding diversity to the type of architectural unit available for the router at each decision and by scaling to deeper networks. They evaluate their approach on Omniglot, where they achieve state of the art performance.
Overall, the paper is very well written and every aspect can be easily understood. The overview over related work given in the paper is thorough, and the authors explain very well how their approach relates to previous approaches.
The architecture presented is a natural and important extension of previous work. Adding diversity in routing units has indeed not been investigated well and is an important contribution to the community. Additionally, the authors do a good job of identifying problems with existing approaches (overfitting, routing depth) and offer a empirically convincing solutions.
The result section given in the paper is its weakness and requires a more in-depth analysis:
+ the results given for Omniglot are impressive
+ the experiments analyzing the impact of diversity and routing depth are interesting and offer interesting insight into the architecture
- the results do not show learning behavior over epochs; this is not necessary, but would give an additional insight into the learning behavior of the architecture
- the experimental settings are confusing: why are the different experiments performed with different datasets? This makes it seem as if the authors cherry-picked the best results for the different experiments (this might not be the case, but the results on Omniglot alone are good enough that negative results and a detailed discussion of them would not have hurt the paper, but enriched the discourse)
- additional experiments that offer a transition from larger datasets to smaller ones would be interesting; seeing how the performance of the architecture behaves e.g. on CIFAR10 for 1k, 5k, 10k, 25k and 50k would have illustrated how well the architecture is able to generalize from different numbers of samples
In summary, I think the paper analyzes a very important problem and has a lot of potential. However, it needs more extensive experiments that illustrate how the proposed architecture behaves over a wider variety of datasets.
UPDATE AFTER REBUTTAL:
I am still torn about this paper. On one hand, I still think that the topic and discourse provided by this paper is extremely important. On the other, the results - even after the revision - do not completely convince me. I might update my score after some discussion with the other reviewers.
2ND UPDATE:
After giving it some more thought, I find myself convinced that this paper has a contribution important enough to be accepted. I increase my score to 7.


In this paper, we introduce a novel method, called step-wise sensitivity analysis, which makes three contributions towards increasing the interpretability of Deep Neural Networks (DNNs). First, we are the first to suggest a methodology that aggregates results across input stimuli to gain model-centric results. Second, we linearly approximate the neuron activation and propose to use the outlier weights to identify distributed code. Third, our method constructs a dependency graph of the relevant neurons across the network to gain fine-grained understanding of the nature and interactions of DNN's internal features. The dependency graph illustrates shared subgraphs that generalise across 10 classes and can be clustered into semantically related groups. This is the first step towards building decision trees as an interpretation of learned representations.

Summary:
The paper introduces a new approach for interpreting deep neural networks called step-wise sensitivity analysis. The approach is conceptually quite simple and involves some interesting ideas, but I have some serious concerns whether the output produced by this method carries any meaning at all. If the authors were able to refute my concerns detailed below, I would raise my score substantially.
Strengths:
+ Potentially interesting heuristic to identify groups of feature channels in DNNs that encode image features in a distributed way
Weaknesses:
- Using the magnitude of the gradient in intermediate layers of ReLU networks is not indicative of importance
- No verification of the method on a simple toy example
Details:
Main issue: Magnitude of the gradient as a measure of importance.
I have trouble with the use of the gradient to identify "outliers," which are deemed important. Comparing the magnitude of activations across features does not make sense in a convnet with ReLUs, because the scale of activations in each feature map is arbitrary and meaningless. Consider a feature map h^l[i,x,y,f] (l=layer, i=images, x/y=pixels, f=feature channels), convolution kernels w^l[x,y,k,f] (k=input channels, f=output channels) and biases b^l[f]:
h^l[i,:,:,f] = ReLU(b^l[f] + \sum_k h^(l-1)[i,:,:,k] * w^l[:,:,k,f])
Assume, without loss of generality, the feature map h^l[:,:,:,f] has mean zero and unit variance, computed over all images (i) in the training set and all pixels (x,y). Let's multiply all "incoming" convolution kernels w^l[:,:,:,f] and biases b^l[f] by 10. As a result, this feature map will now have a variance of 100 (over images and pixels). Additionally, let's divide all "outgoing" kernels w^(l+1)[:,:,f,:] by 10.
Simple linear algebra suffices to verify that the next layer's features h^(l+1) -- and therefore the entire network output -- are unaffected by this manipulation. However, the gradient of all units in this feature map is 10x as high as that of the original network. Of course the gradient in layer l-1 will be unaltered once we backpropagate through w^l, but because of the authors' selection of "outlier" units, their graph will look vastly different.
In other words, it is unclear to me how any method based on gradients should be able to meaningfully assign "importance" to entire feature maps. One could potentially start with the assumption of equal importance when averaged over all images in the dataset and normalize the activations. For instance, ReLU networks with batch norm and without post-normalization scaling would satisfy this assumption. However, for VGG-16 studied here, this is not the case.
On a related note, the authors' observation in Fig. 4b that the same features are both strongly positive and strongly negative outliers for the same class suggests that this feature simply has a higher variance than the others in the same layer and is therefore picked most of the time. Similarly, the fact that vastly different classes such as shark and German Sheppard share the same subgraphs speaks to the same potential issue.
Secondary issue: No verification of the method on simple, understandable toy example.
As shown by Kindermans et al. [1], gradient-based attribution methods fail to produce the correct result even for the simplest possible linear examples. The authors do not seem to be aware of this work (at least it's not cited), so I suggest they have a look and discuss the implications w.r.t. their own work. In addition, I think the authors should demonstrate on a simple, controlled (e.g. linear) toy example that their method works as expected before jumping to a deep neural network. I suppose the issue discussed above will also surface in purely linear multi-layer networks, where the intermediate layers (and their gradients) can be rescaled arbitrarily without changing the network's function.
References:
[1] Kindermans P-J, Schütt KT, Alber M, Müller K-R, Erhan D, Kim B, Dähne S (2017) Learning how to explain neural networks: PatternNet and PatternAttribution. arXiv:170505598. Available at: http://arxiv.org/abs/1705.05598


Deep learning, a rebranding of deep neural network research works, has achieved a remarkable success in recent years. With multiple hidden layers, deep learning models aim at computing the hierarchical feature representations of the observational data. Meanwhile, due to its severe disadvantages in data consumption, computational resources, parameter tuning costs and the lack of result explainability, deep learning has also suffered from lots of criticism. In this paper, we will introduce a new representation learning model, namely "Sample-Ensemble Genetic Evolutionary Network" (SEGEN), which can serve as an alternative approach to deep learning models. Instead of building one single deep model, based on a set of sampled sub-instances, SEGEN adopts a genetic-evolutionary learning strategy to build a group of unit models generations by generations. The unit models incorporated in SEGEN can be either traditional machine learning models or the recent deep learning models with a much "narrower" and "shallower" architecture. The learning results of each instance at the final generation will be effectively combined from each unit model via diffusive propagation and ensemble learning strategies. From the computational perspective, SEGEN requires far less data, fewer computational resources and parameter tuning efforts, but has sound theoretic interpretability of the learning process and results. Extensive experiments have been done on several different real-world benchmark datasets, and the experimental results obtained by SEGEN have demonstrated its advantages over the state-of-the-art representation learning models.

This manuscript introduces SEGEN, a model based on Evolutionary Computation for building deep models. Interestingly, the authors define deep models in a different way. Instead of stacking several hidden layers one after the other (as in traditional deep learning models), SEGEN uses the idea of generations in evolutionary models (Genetic Algorithms or GA) and puts the unit models in the successive generations into layers, i.e., “evolutionary layer”. Each layer then performs the validation, selection, crossover, and mutation operations, as in GA. Another interesting point of the proposed method is that the choice of unit models in SEGEN can be traditional machine learning or recent deep learning models.
The paper touches an interesting topic and proposes a sound method. However, there are several issues with the paper. There are several ungrounded and untested claims, as well as many unclear points in the method.
-	In page 5, Section 4.2.4, the authors introduce the loss function used to define the fitness for the evolutionary model. It is not clear why they use the difference between the latent representations of the autoencoders (z) from pairwise nodes to define the loss. There are no motivations or discussion for this. Two different representations of two nodes may both be good (e.g., in terms of classification of data), but they do not have to be necessarily identical.
-	Given the loss defined in Section 4.2.4, it is not clear how the authors ran their model for MNIST and other datasets, for which they used CNN and MLP unit models. In CNN and MLP there is not latent representation z.
-	Based on the model descriptions in Section 4.2 (and its Subsections), the proposed method transfers the learned models in previous generations to the next ones. But there is no explanation if the new models are again fine-tuned on the data? For instance, take the autoencoders, for two different unit models, the cross-over operator defuses the variables (weights and bias) from the two selected models to create an offspring. There is no guarantee that the new autoencoder model works properly on the same dataset. As a naïve example, if there are correlated and redundant features in the data, different autoencoders may separately focus on one/some of these features. Defusing weights of the two autoencoders (built upon different aspects of the data) may most probably ruin the whole model.
-	There are four claims in the paper on the advantages of the proposed model, compared to other deep learning algorithms. None of these claims are discussed in depth or at least illustrated experimentally.
*** Less Data for Unit Model Learning. The authors could have reported the number of variables used in each model in the experiments. It is important to see with how many of a larger number of variables a traditional deep model can result in comparable results to SEGEN.
*** Less Computational Resources. The model operates in several generations and in each generation, many unit models are built. It is not fair to say and not clear how it can occupy less space or time complexity than a regular GCNN or MLP.
*** Less Parameter Tuning. Again experiments could clarify this issue.
*** Sound Theoretic Explanation. The authors only refer to (Rudolph 1994) for the performance bounds of their model and claim that since they are using GA they are better than other deep learning models. However, performance bounds for GA models are very shallow and proximal.
-	To calculate the computational complexity of the model, the authors analyzed the time for learning one unit model. However, in GA models, the complexity is calculated using the bounds on the number of times the fitness function is called since the fitness function is the most computationally intensive task (please see: Pelikan and Lobo 1999 ‘Parameterless Genetic Algorithm A Worst-case Time and Space Complexity Analysis’).
-	One of the main fallacies of GAs and evolutionary algorithms is that they may lead to premature convergence. This is very common, especially at the presence of trap functions, such as non-convex functions that real-world problems deal with (please see: Goldberg et al. 1991 ‘Massive Multimodality, Deception, and Genetic Algorithms’). There are no discussions/experiments on how SEGEN may overcome the premature convergence, or even if it converges at all.


The tremendous success of deep generative models on generating continuous data like image and audio has been achieved; however, few deep graph generative models have been proposed to generate discrete data such as graphs. The recently proposed approaches are typically unconditioned generative models which have no control over modes of the graphs being generated. Differently, in this paper, we are interested in a new problem named Deep Graph Translation: given an input graph, the goal is to infer a target graph by learning their underlying translation mapping. Graph translation could be highly desirable in many applications such as disaster management and rare event forecasting, where the rare and abnormal graph patterns (e.g., traffic congestions and terrorism events) will be inferred prior to their occurrence even without historical data on the abnormal patterns for this specific graph (e.g., a road network or human contact network). To this end, we propose a novel Graph-Translation-Generative Adversarial Networks (GT-GAN) which translates one mode of the input graphs to its target mode. GT-GAN consists of a graph translator where we propose new graph convolution and deconvolution layers to learn the global and local translation mapping. A new conditional graph discriminator has also been proposed to classify target graphs by conditioning on input graphs. Extensive experiments on multiple synthetic and real-world datasets demonstrate the effectiveness and scalability of the proposed GT-GAN.

The paper presents an approach for translating graphs in one domain to graphs in the same domain using a GAN approach. A graph Translator approach is defined and a number of synthetic data sets and one real-world data set are used to evaluate the approach. Most of the paper is written well, though there are some odd sentence structure issues in places. The paper could do with a thorough check for grammatical and spelling mistakes. For example you miss-spell NVIDIA.
The main concerns with the work:
1) Equation 2 is used to minimise the distance between graphs from X and graphs in Y. Yet, the main metric which is used to evaluate the paper is this distance. This would seem to give an unfair advantage to your approach. I would also be concerned about the fact that later you use this for stating if a graph represents good or hacker activity. If you have drawn translated graphs towards real graphs, how do you know that you haven’t pulled a good graph closer to a hacker graph? This is more concerning considering work which came out of NIPS which suggested that GAN’s tend to favour producing similar output rather than spreading it evenly over the domain.
2) It isn’t entirely clear what your results are trying to show. Presumably P, R, AUC and F1 are generated from the results produced from your Discriminator? Were each of the other approaches optimised against your discriminator or not? Also, it is unclear as to what the Gold Standard method is - we’re only told that its a classifier, but what type and how constructed?
3) Your approach seems to be ‘fixed’ in the set of nodes which are in both in the input and output graphs - needing to be the same. This would seem significantly limiting as graphs are rarely of the same node set.
4) Although you comment on other graphs approaches being limited to very small graphs, you do not test your approach on graphs with over 150 nodes. These would also seem to be very small graphs in comparison to real-world graphs. Further evaluation on larger graphs would seem to be essential - how long would it take on graphs with 10^6 nodes?
5) The real-world dataset seems rather odd and not fully explored. Given that you have this data it is surprising that you didn’t complete the loop by showing that you could take data from before a hack attempt and show that you could predict that in the future you had a hack attempt. Perhaps this is due to the fact that you didn’t have the ground-truth data in here to show a graph going from good to bad? But if not it would have been good to have shown, either through this data or some other, how your approach does match in with real-world results.
Given the points above, I would be very concerned on an approach which used the above to identify a future hacking attempt.
Some more specific comments on the paper:
- "The tremendous success of deep generative models on generating continuous data like image and audio” - it is not clear what this continuous data is.
- Hard to parse : “which barely can be available for the accounts worth being monitored.”
- “This requires us to learn the generic distribution of theft behaviors from historical attacks and synthesize the possible malicious authentication graphs for the other accounts conditioning on their current computer networks” - given that these historical attacks are (hopefully) rare, is there enough data here to construct a model?
- Please define GCNN
- “Our GT-GAN is highly extensible where underlying building blocks, GCNN and distance measure in discriminator, can be replaced by other techniques such as (Kipf & Welling, 2017; Arjovsky et al., 2017) or their extensions.” - this sounds more like a feature of what you have contributed rather than a contribution in its own right.
- In the context of synthetic data, what is ground-truth?
- Hard to parse “Modern deep learning techniques operating on graphs is a new trending topic in recent years.”
- Hard to parse “However, these methods are highly tailored to only address the graph generation in a specific type of applications such as molecules generation”
- Hard to parse “Existing works are basically all proposed in the most recent year,”
- “Typically we focus on learning the translation from one topological patterns to the other one” -> “Typically we focus on learning the translation from one topological pattern to the other”
- It’s not clear in equation 1 how you represent G_X. Only much later is it mentioned about adjacency matrix.
- Hard to parse “Different and more difficult than graph generation designed only for learning the distribution of graph representations, for graph translation one needs to learn not only the latent graph presentation but also the generic translation mapping from input graph to the target graph simultaneously.“
- Hard to parse “graph translation requires to learn”
- Hard to parse “in most of them the input signal is given over node with a static set of edge and their weights fixed for all samples”
- “we propose an graph” -> “we propose a graph”
- Hard to parse “The two components of the formula refers to direction filters as talked above”
- Hard to parse “Next, graph translator requires to”
- “as shown in Equations equation 7 and Equations equation 6,” -> “as shown in Equation 6 and Equation 7”
- Hard to parse “The challenge is that we need not only to learn the”
- Figure 2 would seem to need more explanation.
- The end of section 3.3 is a bit vague and lacks enough detail to reproduce.
- “our GT-GAN is able to provide a scalable (i.e., O(N2)) algorithm that can generate general graphs.” - what sizes have you tested this up to?
- Hard to parse “we randomly add another kjEj edges on it to form the target graph”
- “The goal is to forecast and synthesize the future potential malicious authentication graphs of the users without any historical malicious behaviors, by the graph translator from normal to malicious graph trained based on the users with historical malicious-behavior records.” - This isn’t entirely clear. Are you trying to create new malicious graphs or show that a current graph will eventually go malicious?
- “All the comparison methods are directly trained by the malicious graphs without the conditions of input graphs as they can only do graph generation instead of translation.” - not clear. For the synthetic data sets how did you choose which ones were malicious?
- “GraphRNN is tested with graph size within 150. GraphGMG, GraphVAE is tested within size 10 and RandomVAE is tested on graphs within size 150.” -> “GraphRNN and RandomVAE are tested with graph up to size 150. GraphGMG, GraphVAE is tested with graphs up to  size 10.”
- “Here, beyond label imbalance, we are interested in “label missing” which is more challenging.” - “missing labels”?
- “In addition, we have also trained a “gold standard” classifier based on input graphs and real target
graphs.” - need to say more about this.


UNIVERSAL SUCCESSOR FEATURES APPROXIMATORS
The ability of a reinforcement learning (RL) agent to learn about many reward functions at the same time has many potential benefits, such as the decomposition of complex tasks into simpler ones, the exchange of information between tasks, and the reuse of skills. We focus on one aspect in particular, namely the ability to generalise to unseen tasks. Parametric generalisation relies on the interpolation power of a function approximator that is given the task description as input; one of its most common form are universal value function approximators (UVFAs). Another way to generalise to new tasks is to exploit structure in the RL problem itself. Generalised policy improvement (GPI) combines solutions of previous tasks into a policy for the unseen task; this relies on instantaneous policy evaluation of old policies under the new reward function, which is made possible through successor features (SFs). Our proposed universal successor features approximators (USFAs) combine the advantages of all of these, namely the scalability of UVFAs, the instant inference of SFs, and the strong generalisation of GPI. We discuss the challenges involved in training a USFA, its generalisation properties and demonstrate its practical benefits and transfer abilities on a large-scale domain in which the agent has to navigate in a first-person perspective three-dimensional environment.

Paper’s contributions:
This paper considers the challenging problem of generalizing well to new RL tasks, based on having learned on a set of previous related RL tasks.  It considers tasks that differ only in their reward function (assume the dynamics are identical), and where the reward functions are constrained to be linear combinations over a set of given features.  The main approach, Universal Successor Features Approximators (USFAs) is a combination of two recent approaches:  Universal Value Function Approximators (UVFAs) and Generalized Policy Improvement (GPI).  The main claim is that while each of these methods leverages different types of regularity when generalizing to new tasks, USFAs are able to jointly leverage both types (and elegantly have both other methods as special cases).
Summary of evaluation:
Overall the paper tackles an important problem, and provides careful explanation and reasonably extensive results showing the ability of USFA to leverage structure.  I’m on the fence because I really wish the combination of generalization properties could be understood in a more intuitive way.  There are some more minor issues, such as lack of complexity analysis and a few notation details, that can be easily fixed.
Pros:
-	The problem of generalizing to new tasks in RL is an important open problem.
-	The paper is carefully written and provides clear explanation of most of the methods & results.
Cons:
-	The authors are diligent about trying to explain what type of regularities are exploited by each of UVFAs and GPI, and how this can be combined in USFAs.  However despite reading these parts carefully, I could not get a really good intuition, either in the methods or in the results, for the nature of the regularities exploited, and how it really differs.  Top of p.4 says that GPI generalizes well when the policy \pi(s) does well on task w’.  Can you give a specific MDP where Q is not smooth, but the policy does well?
-	There is no complexity analysis.  I would like to know the computational complexity of each of the key steps in Algorithm 1 (with comparison to simple UVFA and GPI).
-	It would be useful to see the empirical comparison with the approach of Ma et al. (2018), which also combines SFs and UFVAs. I understand there are differences in the details, but I would like to see confirmation of whether the claims about USFA’s superior ability to exploit structure is supported by results.
Minor comments:
-	The limitation to linear rewards is a reasonably strong assumption.  It would be good to support this, e.g. by references to domain that meet this assumption.
-	It seems the mathematical properties in Sec.3.1 could be further developed.
-	P.4: “Given a deterministic policy \pi, one can easily define a reward function r_\pi”.  I did not think this mapping was unique (see the literature on IRL, e.g. Ross et al.).  Can you provide a proof or reference to support this statement?
-	The definition of Q(s,a,w,z) is interesting. Can this be seen as a kernel between w and z?
-	\theta suddenly shows up in Algorithm 1. I presume these are the parameters of Q?  Should be defined.
-	The distribution used to sample policies seems to be a key step of this approach, yet not much guidance is given on how to do this in general.


Metric learning aims at learning a distance which is consistent with the semantic meaning of the samples. The problem is generally solved by learning an embedding, such that the samples of the same category are close (compact) while samples from different categories are far away (spread-out) in the embedding space. One popular way of generating such embeddings is to use the second-to-last layer of a deep neural network trained as a classifier with the softmax cross-entropy loss. In this paper, we show that training classifiers with different temperatures of the softmax function lead to different distributions of the embedding space. And finding a balance between the compactness, "spread-out" and the generalization ability of the feature is critical in metric learning. Leveraging these insights, we propose a "heating-up" strategy to train a classifier with increasing temperatures. Extensive experiments show that the proposed method achieves state-of-the-art performances on a variety of metric learning benchmarks.

Summary:
This paper proposes a novel optimization strategy regarding softmax cross-entropy loss, to extract the effective features of well generalization in the framework of metric learning.
The authors focus on the "temperature" parameter in the softmax and through analyzing the role of the temperature in terms of gradient, propose the approach of heating-up softmax in which the temperature is varied from low to high in training.
And, the effects of normalization such as by l2 and BatchNorm are discussed in the framework of heated-up softmax.
The experimental results on metric learning tasks demonstrate the effectiveness of the proposed method in comparison with the other methods.
Comments:
Pros:
+ The idea of heating up the temperature in softmax is interesting, and seems novel in the literature of metric learning.
+ The performance improvement, especially produced by batchNorm-based normalization, is shown.
Cons:
- The formulation of tempered softmax with normalization is already presented in [Wang et al., 2017].
- The reason why the heating-up approach contributes to better metric learning is not clearly provided in a well convincing way.
- It lacks an important ablation study to fairly validate the method.
- The discussion/comparison is limited to the simple softmax function.
Although the reviewer likes the idea of heating up softmax, this paper can be judged as a borderline slightly leaning toward reject, due to the above weak points, the details of which are explained as follows.
- Formulation
The softmax equipped with temperature for the normalized features and weights are shown in [Wang et al., 2017]. The only difference from that work is the way to deal with temperature; in [Wang et al., 2017], the temperature is "optimized" as a trainable parameter, while it is dealt with in a hand-crafted way of heating up in this work. Honestly speaking, it is unclear which approach is better, though the optimization in [Wang et al., 2017] seems elegant as stated in that paper. The only way to validate this work compared to [Wang et al., 2017] is to empirically evaluate those two methods in the experiments. Such a comparison experiment is not found and it is a main flaw of this paper.
- Justification of the method
The gradients of the softmax cross-entropy loss parameterized with a temperature T are well analyzed in Sections 3.1&3.2. But, in Section 3.3, the reviewer cannot find the clear and convincing explanation for why the temperature T should be increased in the training. My question is: why don't you use alpha=4 consistently throughout the training?
 It might be related to the process of simulated annealing (though "temperature" is usually cooled down in SA), and more interestingly, it would also be possible to find connection with the work of [Guo et al., 2017]. In [Guo et al., 2017], the temperature in the softmax is optimized as a post processing for calibrating the classifier outputs. Though the calibration task itself is a little bit apart from the metric learning of the authors' interest, we can find in that paper an interesting result that the temperature is heated up to increase the confidence of the classifier outputs, which is quite similar to the process of fine-tuning by heating up softmax as done in this work. Therefore, the reviewer guesses that the effectiveness of heating up softmax can also be interpreted from the viewpoint of [Guo et al., 2017].
There is also less description about Figure 1; in particular, the reviewer cannot understand what Figure 1(d) means.
- Ablation study
To empirically resolve the above concerns, it is necessary to present the empirical comparison with the "static" softmax.
Namely, the methods of HLN/HBN should be carefully compared to LN/BN of "alpha=4", not only those of alpha=16 shown in Table 1&2; the comparison in Table 3 seems unfair since the authors apply the static softmax without normalization.
And, it would be better to show the performance of heated-up softmax "without" normalization to show the important role of the normalization, as done in [Wang et al., 2017].
In summary, since the proposed method is composed of a heating-up approach and feature normalization, the authors are required to validate the method from those two aspects, respectively, for increasing the significance of this paper.
- Other loss function
For achieving a compactness in feature representation, the simple softmax requires both temperature and normalization. It, however, is also conceivable to employ the other types of loss function for that purpose, such as [a] which is based on the (Mahalanobis) distance with taking into account the margin between categories. The distance based loss also embeds features into localized clusters, which satisfies the authors' objective in this work. To validate the proposed method, it is required to compare the method with such a different types of loss function.
[a] Wan, W., Zhong, Y., Li, T., & Chen, J. (2018). Rethinking Feature Distribution for Loss Functions in Image Classification, In CVPR2018, pp. 9117–9126.


We aim to model 3D properties of the scenes from single 2D images. Learning 3D scenes from 2D images is a long-standing problem in computer vision with applications in related fields such as simulation and robotics. We propose pix2scene, a deep generative-based approach that represents the 3D scene in a learnt latent variable decoded into a viewpoint-dependent representation that can be rendered. Our method learns the depth of the scene and leverages a local smoothness assumption to extract the orientation of visible scene points. We achieve this using an encoderdecoder adversarial learning mechanism and a novel differentiable renderer to train the 3D model in an end-to-end fashion, using only images. We showcase the generative ability of our model qualitatively on the ShapeNet dataset (Chang et al., 2015). We also demonstrate that our model can predict the structure of scenes from various, previously unseen view points. Finally, we evaluate the effectiveness of the learned 3D scene representation in supporting 3D spatial reasoning.

This paper introduces a method to create a 3D scene model given a 2D image and a camera pose. The method is: (1) an "encoder" network maps the image to some latent code vector, (2) a "decoder" network uses the code and the camera pose to create a depthmap, (3) surface normals are computed from the depthmap, and (4) these outputs are fed to a differentiable renderer which reconstructs the input image. At training time, a discriminator provides feedback to (and simultaneously trains on) the latent code and the reconstructions. The model is self-supervised by the reconstruction error and the GAN setup. Experiments show compelling results in 3D scene generation for simple monochromatic synthetic scenes composed of an empty room corner and floating ShapeNet shapes.
This is a nice problem, and if the approach ever works in the real world, it will be useful. On synthetic environments, the results are impressive.
The paper seems to claim more ground than it actually covers. The abstract says "Our method learns the depth and orientation of scene points visible in images", but really only the depth is learned, and the "orientation" is an automatically-computed surface normal, which is a free byproduct of any depth estimate. The "surfel" description includes a reflectance vector, but this is never estimated or further described in the paper, so my guess is that it is simply treated as a scalar (which equals 1). Taking this reflectance issue together with the orientation issue, the model is not really estimating surfels at all, but rather just a depthmap, which makes the method seem considerably less novel. Furthermore, the differentiable rendering (eq. 1) appears to assume that all light sources are known exactly -- this is not a trivial assumption, and yet it is never mentioned in the paper. The text suggests that only an image is required to run the model, but Figure 3 shows that the networks are conditioned on the camera pose -- exact knowledge of the camera pose is difficult to obtain precisely in real settings, so this again is not an assumption to ignore.
To rewrite the paper more plainly, one might say that it receives a monochrome image as input, estimates a depthmap, and then shades this depthmap using perfect knowledge of lighting and camera pose, which reconstructs the input. This may sound less appealing, but it also seems more accurate.
The paper is also missing some details of the method and evaluation, which I hope can be cleared up easily.
- What is happening with the light source? This is critical in the shading equation (eq. 1), and yet no information is given on it -- we need the color and the position of every light in the scene.
- How is the camera pose represented? Section 3.3.3 says conditional normalization is used, but what exactly is fed to the network that estimates these conditional normalization parameters?
- What is the exact form of the reconstruction error? An equation would be great.
- How is the class-conditioning done in 4.2?
- In Eq. 4, the first usage of D_\theta should use only the object part of the vectors, and the second usage should use only the geometric part, right? Maybe this can be cleared up with a second D_subscript.
- I do not understand the "interleaved" training setup in 4.4.1. Please explain that more.
- It is not clear to me why the task in 4.4.2 needs any supervised training at all, if the classification is just done by computing L2 distances in the latent space. What happens with "0 sampled labels"?
Overall, I like the paper, and I can imagine others in my group liking it. I hope it gets in, assuming the technical details get cleaned up and the language gets softer.


A COMPREHENSIVE, APPLICATION-ORIENTED STUDY OF CATASTROPHIC FORGETTING IN DNNS
We present a large-scale empirical study of catastrophic forgetting (CF) in modern Deep Neural Network (DNN) models that perform sequential (or: incremental) learning. A new experimental protocol is proposed that enforces typical constraints encountered in application scenarios. As the investigation is empirical, we evaluate CF behavior on the hitherto largest number of visual classification datasets, from each of which we construct a representative number of Sequential Learning Tasks (SLTs) in close alignment to previous works on CF. Our results clearly indicate that there is no model that avoids CF for all investigated datasets and SLTs under application conditions. We conclude with a discussion of potential solutions and workarounds to CF, notably for the EWC and IMM models.

# [Updated after author response]
Thank you for your response. I am happy to see the updated paper. In particular, the added item in section 1.3 highlights where the novelty of the paper lies, and as a consequence, I think the significance of the paper is increased. Furthermore, the clarity of the paper has increased.
In its current form, I think the paper would be a valuable input to the deep learning community, highlighting an important issue (CF) for neural networks. I have therefore increased my score.
------------------------------------------
# Summary
The authors present an empirical study of catastrophic forgetting (CF) in deep neural networks. Eight models are tested against nine datasets with 10 classes each but a varying number of samples. The authors construct a number of sequential learning tasks to test the model performances in different scenarios. The main conclusion is that CF is still a problem in all models, despite claims in other papers.
# Quality
The paper shows healthy criticism of the methods used to evaluate CF in previous works. I very much like this.
While I like the different experimental set-ups and the attention to realistic scenarios outlined in section 1.2, I find the analysis of the experiments somewhat superficial. The accuracies of each model for each task and dataset are reported, but there is little insight into what causes CF. For instance, do some choices of hyperparameters consistently cause a higher/lower degree of CF across models? I also think the metrics proposed by Kemker et al. (2018) are more informative than just reporting the last and best accuracy, and that including these metrics would improve the quality of the paper.
# Clarity
The paper is generally clearly written and distinct paragraphs are often highlighted, which makes reading and getting an overview much easier. In particular, I like the summary given in sections 1.3 and 1.4.
Section 2.4 describing the experimental setup could be clearer. It takes a bit of time to decipher Table 2, and it would have been good with a few short comments on what the different types of tasks (D5-5, D9-1, DP10-10) will tell us about the model performances. E.g. what do you expect to see from the experiments of D5-5 that is not covered by D9-1 and vice versa? And why are the number of tasks in each category so different (8 vs 3 vs 1)?
I am not a huge fan of 3D plots, and I don't think they do anything good in section 4. The perspective can make it tricky to compare models, and the different graphs overshadow each other. I would prefer 2D plots in the supplementary, with a few representative ones shown in the main paper. I would also experiment with turning Table 3 into a heat map.
# Originality
To my knowledge, the paper presents the largest evaluation of CF in terms of evaluated datasets. Kemker et al. (2018) conduct a somewhat similar experiment using fewer datasets, but a larger number of classes, which makes the CF even clearer. I think it would be good to cite this paper and briefly discuss it in connection with the current work.
# Significance
The paper is mostly a report of the outcome of a substantial experiment on CF, showing that all tested models suffer from CF to some extent. While this is interesting and useful to know, there is not much to learn in terms of what can cause or prevent CF in DNNs. The paper's significance lies in showing that CF is still a problem, but there is room for improvement in the analysis of the outcome of the experiments.
# Other notes
The first sentence of the second paragraph in section 5 seems to be missing something.
# References
Kemker, R., McClure, M., Abitino, A., Hayes, T., & Kanan, C. (2018). In AAAI Conference on Artificial Intelligence. https://aaai.org/ocs/index.php/AAAI/AAAI18/paper/view/16410


NEURAL SPEED READING WITH STRUCTURAL-JUMP- LSTM
Recurrent neural networks (RNNs) can model natural language by sequentially "reading" input tokens and outputting a distributed representation of each token. Due to the sequential nature of RNNs, inference time is linearly dependent on the input length, and all inputs are read regardless of their importance. Efforts to speed up this inference, known as "neural speed reading", either ignore or skim over part of the input. We present Structural-Jump-LSTM: the first neural speed reading model to both skip and jump text during inference. The model consists of a standard LSTM and two agents: one capable of skipping single words when reading, and one capable of exploiting punctuation structure (sub-sentence separators (,:), sentence end symbols (.!?), or end of text markers) to jump ahead after reading a word. A comprehensive experimental evaluation of our model against all five state-of-the-art neural reading models shows that Structural-Jump-LSTM achieves the best overall floating point operations (FLOP) reduction (hence is faster), while keeping the same accuracy or even improving it compared to a vanilla LSTM that reads the whole text.

The paper presents a novel model for neural speed reading. In this new model, the authors combined several existing ideas in a nice way, namely, the new reader has the ability to skip a word or to jump a sequence of words at once. The reward of the reader is mixed of the final prediction correctness and the amount of text been skipped. The problem is formulated as a reinforcement learning problem. The results compared with the existing techniques on several benchmark datasets show consistently good improvements.
In my view, one important (also a little surprising) finding of the paper is that the reader can make jump choices successfully with the help of punctuations. And, blindly jumping a sequence of words without even lightly read them can still make very good predictions.
The basic idea of the paper, the concepts of skip and jump, and the reinforcement learning formulation are not completely new, but the paper combined them in an effective way. The results show good improvements majorly in FLOPS.
The way of defining state, rewards and value function are not very clear to me. Two value estimates are defined separately for the skip agent and the jump agent. Why not define a common value function for a shared state? Two values will double count the rewards from reading. Also, the state of the jump agent may not capture all available information. For example, how many words until the end of the sentence if you make a jump. Will this make the problem not a MDP?
Overall, this is a good paper.
I read the authors' response. The paper should in its final version add the precise explanation of how the two states interact and how a joint state definition differs from the current one.


IMPROVING THE GENERALIZATION OF ADVERSARIAL TRAINING WITH DOMAIN ADAPTATION
By injecting adversarial examples into training data, adversarial training is promising for improving the robustness of deep learning models. However, most existing adversarial training approaches are based on a specific type of adversarial attack. It may not provide sufficiently representative samples from the adversarial domain, leading to a weak generalization ability on adversarial examples from other attacks. Moreover, during the adversarial training, adversarial perturbations on inputs are usually crafted by fast single-step adversaries so as to scale to large datasets. This work is mainly focused on the adversarial training yet efficient FGSM adversary. In this scenario, it is difficult to train a model with great generalization due to the lack of representative adversarial samples, aka the samples are unable to accurately reflect the adversarial domain. To alleviate this problem, we propose a novel Adversarial Training with Domain Adaptation (ATDA) method. Our intuition is to regard the adversarial training on FGSM adversary as a domain adaption task with limited number of target domain samples. The main idea is to learn a representation that is semantically meaningful and domain invariant on the clean domain as well as the adversarial domain. Empirical evaluations on Fashion-MNIST, SVHN, CIFAR-10 and CIFAR-100 demonstrate that ATDA can greatly improve the generalization of adversarial training and the smoothness of the learned models, and outperforms state-of-the-art methods on standard benchmark datasets. To show the transfer ability of our method, we also extend ATDA to the adversarial training on iterative attacks such as PGD-Adversial Training (PAT) and the defense performance is improved considerably.

The paper casts the problem of learning from adversarial examples to make models resistant to adversarial perturbations to a domain adaptation problem. The proposed method Adversarial training with Domain adapatation( ATDA) learns a representation that is invariant to clean and adversarial data achieving state of the art results on CIFAR.
quality - Paper is well written, explanation of the mathematical parts are good, experimental quality can be much better.
clarity - the problem motivation as well as the methodology is clearly explained. the learning from the experiments are unclear and need more work.
originality - The casting of the problem as domain adaptation is original but from the experiments it was not conclusive as to how much benefit we get.
significance of this work -  Current models being sensitive to adversarial perturbations is quite a big problem so the particular problem authors are trying to address is very significant.
pros
A good idea, enough experiments that indicate the benefit of casting this as a domain adaptation problem.
cons
I feel, the authors should have extended the experiments to ImageNet which is a much larger dataset and validate the findings still hold, I feel the discussion section and comparison to other methods needs to be worked to be more thorough and to tease out the benefit of each of the various terms added to the loss functions as currently all we have is final numbers without much explanation and details. TSNE embeddings part is also very qualitative and while the plots indicate a better separation for ATDA, I feel authors should do more quantitative analysis on the embeddings instead of just qualitative plots.


BACKPLAY: 'MAN MUSS IMMER UMKEHREN'
Model-free reinforcement learning (RL) requires a large number of trials to learn a good policy, especially in environments with sparse rewards. We explore a method to improve the sample efficiency when we have access to demonstrations. Our approach, Backplay, uses a single demonstration to construct a curriculum for a given task. Rather than starting each training episode in the environment's fixed initial state, we start the agent near the end of the demonstration and move the starting point backwards during the course of training until we reach the initial state. Our contributions are that we analytically characterize the types of environments where Backplay can improve training speed, demonstrate the effectiveness of Backplay both in large grid worlds and a complex four player zero-sum game (Pommerman), and show that Backplay compares favorably to other competitive methods known to improve sample efficiency. This includes reward shaping, behavioral cloning, and reverse curriculum generation.

This paper presents a method for increasing the efficiency of sparse reward RL methods through a backward curriculum on expert demonstrations. The method in the paper is as follows: assuming access to expert demonstration and a resettable simulator, the start state of the agent in the beginning of training is sampled from end of demonstration (close to the rewarding state) where the task of achieving the goal is easy. Then gradually through a curriculum this is shifted backwards in the demonstration, making the task gradually harder.
The proposed method is closely related to 1) “Learning Montezuma’s Revenge from a Single Demonstration” a blog post and open-source code release by Salimans and Chen (OpenAI Blog, 2018) where they show that constructing a curriculum that gradually moves the starting state back from the end of a single demonstration to the beginning helps solve Montezuma’s revenge game 2) “Reverse Curriculum Generation for Reinforcement Learning” by Florensa et al. (CoRL 2017) , where they start the training to reach a goal from start states nearby a given goal state and gradually the agent is trained to solve the task from increasingly distant start states.
The approach is evaluated on a pair of tasks, a maze environment and a stochastic four-player game, Pommerman. In the maze environment, they compare to vanilla PPO and Uniformly sampled starting points across the expert trajectory. The Backplay method outperforms the vanilla baseline, however, from the training curves (~3500 epochs) in the appendix A4, it looks like the Uniform sampling baseline is doing as well or better than the proposed method. As pointed out by the authors themselves the reverse curriculum does not seem necessary in this environment. Also, it is unclear to me whether the curves shown is comparable as the starting point of the agent, at least in the beginning of training, is close to the goal with higher success rate for the Backplay method compared to baselines. A good convincing assessment would be to report success rate against the same starting point for all methods preferably not from the starting point of the demonstrations to assess generalisation of these methods for which authors briefly report unsuccessful results.
The Pommerman environment is more complex and the results reported are more interesting. Figure 3 shows the results on four different maps for which expert demonstrations are generated from a Finite-Machine Tree-search method (a competitive method in this environment). I’m slightly confused by the plots and the significant drops in performance once the curriculum is finished and agent encounters the start position of the demonstration trajectory (epoch 250). Is this affected by the schedule of the curriculum? Also, the choice of terminating training at epoch 550 is not clear as the method does not seem to have converged yet (the variance is quite high) and would be interesting to observe the dynamics of learning as the training proceeds and whether it converges to a stable policy at all. I am also slightly unclear regarding the performance difference between Standard method in Figure 3(c) and 3(d). If the Standard method is still the same baseline, vanilla PPO, why such huge performance difference? In my understanding, only the Uniform and Backplay methods should be affected by the quality of demonstrations? I believe this figure needs more explanation and clarity. I am also not clear on why Standard method is terminated at epoch 450 while other methods are trained until epoch 550. Figure 4 reports results of generalisation to 10 unseen maps but again the choice of terminating training after 550 epochs is not clear to me as the method again does not seem to have converged.
Overall, the choice of parameters is not well motivated, these include the window size for sampling the start point, the schedule for shifting the start point, batch size (102400 seems large to me and this choice is never explained), horizon (in appendix A3 reported to be 1707 for Maze while in the main text it is reported as 200 steps), termination of training (3500 for Maze, Figure 7, and 550 in Pommerman, Figure 3).
I commend the authors for honestly reporting their method’s shortcomings such as failure in generalisation, however, I find that the work lacks significance and quality. There is not much novelty in the proposed method and there is a clear lack of comparisons to existing sample efficient LfD techniques such as Generative Adversarial Imitation Learning (GAIL). I believe this paper requires substantial improvements for publication and is not up to the ICLR standards in its current form.


Deep artificial neural networks (DNNs) are typically trained via gradient-based learning algorithms, namely backpropagation. Evolution strategies (ES) can rival backprop-based algorithms such as Q-learning and policy gradients on challenging deep reinforcement learning (RL) problems. However, ES can be considered a gradient-based algorithm because it performs stochastic gradient descent via an operation similar to a finite-difference approximation of the gradient. That raises the question of whether non-gradient-based evolutionary algorithms can work at DNN scales. Here we demonstrate they can: we evolve the weights of a DNN with a simple, gradient-free, population-based genetic algorithm (GA) and it performs well on hard deep RL problems, including Atari and humanoid locomotion. The Deep GA successfully evolves networks with over four million free parameters, the largest neural networks ever evolved with a traditional evolutionary algorithm. These results (1) expand our sense of the scale at which GAs can operate, (2) suggest intriguingly that in some cases following the gradient is not the best choice for optimizing performance, and (3) make immediately available the multitude of neuroevolution techniques that improve performance. We demonstrate the latter by showing that combining DNNs with novelty search, which encourages exploration on tasks with deceptive or sparse reward functions, can solve a high-dimensional problem on which reward-maximizing algorithms (e.g. DQN, A3C, ES, and the GA) fail. Additionally, the Deep GA is faster than ES, A3C, and DQN (it can train Atari in ∼4 hours on one workstation or ∼1 hour distributed on 720 cores), and enables a state-of-the-art, up to 10,000-fold compact encoding technique.

Post-rebuttal update: The review process has identified several issues such as missing citations and lack of clarity with respect to aims of the paper. Although the authors have failed to update the paper within the rebuttal period, their responses show an understanding of the issues that need to be addressed as well as a broad appreciation of work in EC that would be included in a final version, making it a useful resource for the wider ML community. On top of this they will include an even larger amount of empiricial data from the experiments they have already run, which is a valuable resource considering the amount of compute needed to obtain this data.
---
The current landscape of reinforcement learning - particularly in domains with high-dimensional structured input spaces such as images or text - relies heavily on backpropagation-based reinforcement learning algorithms.  An alternative that has re-emerged is ES, due to its simplicity and scalability. However, ES can also be considered a gradient-based method. In this paper, the authors apply a similar treatment to GAs, another simple method at its most basic. The authors claim to have 3 main contributions: extending the scale to which GAs can operate, suggesting that gradient-based methods may not achieve the best performance, and making available a vast array of techniques available from the EC literature; they demonstrate the latter by utilising novelty search (NS).
In my opinion the authors do indeed have a valuable contribution in a) demonstrating that a simple GA can successfully be applied to larger networks than was previously thought to be possible and b) introducing a novel software implementation that allows GAs to be efficiently scaled/distributed (similar in nature to the work of Salimans et al.). This is by itself valuable, as, along with recent work on ES, it potentially extends the range of problems that are perhaps best tackled using black-box optimisation techniques. Going against the prevailing trends in order to investigate alternative methods is an underappreciated service to the community, and I believe the evaluation of the methods and the choice of comparative methods to be just about satisfactory. As exemplified by NS, there is a wealth of techniques from the EC literature that could be applied to many topical problems, and the authors' main contributions opens up the road for this.
A lot of care has been put into evaluation on Atari games. The details in the main paper and supplementary material, with, e.g., clear definitions of "frames", make me believe that fair comparisons have taken place. All methods in the table, including GAs, perform best at some games (except for RS, which is a necessary baseline for GAs). It would be better to provide more data points that relate to prior works - such as scores at 200M frames to evaluate sample complexity (indeed, the authors note that good solutions can be found by GAs within a few generations, so it would be best to tabulate this) and at ~ 4d to evaluate wall-clock time (is it possible to push performance even further?). Since the GA presented is very rudimentary, I consider the baselines in the main paper to be reasonable, but it would be misleading to not present newer work. The authors do so in the supplementary material, and it is promising to note that the GA still achieves state-of-the-art performance in a few games even when compared to the most sophisticated/distributed state-of-the-art DRL algorithms developed in a concentrated effort over the last few years. Despite having an NS variant, it is a shame that the authors did not show that this could potentially improve performance on Atari, when BCs such as the game RAM or preferably random CNN features are easily available.
The authors also evaluate on a maze that is a staple task in the EC literature to demonstrate the power of NS. While the results are unsurprising, it is a reasonable sanity check. The final evaluation is on a difficult continuous control task, in which GAs solve the task, but have much poorer sample complexity than ES. Given the range of continuous control tasks used to benchmark RL algorithms nowadays, the authors would do well to present results across more of these tasks. Again, NS was not evaluated on this task.
A major weakness of this paper is the presentation. The authors discuss some interesting findings, but would be better served by being more concise and focused. In particular, the emphasis should be more on showcasing quantitative results. Doing so, with more continuous control tasks, would make the claims of this paper more substantiated.


IMPROVING MMD-GAN TRAINING WITH REPULSIVE LOSS FUNCTION
Generative adversarial nets (GANs) are widely used to learn the data sampling process and their performance may heavily depend on the loss functions, given a limited computational budget. This study revisits MMD-GAN that uses the maximum mean discrepancy (MMD) as the loss function for GAN and makes two contributions. First, we argue that the existing MMD loss function may discourage the learning of fine details in data as it attempts to contract the discriminator outputs of real data. To address this issue, we propose a repulsive loss function to actively learn the difference among the real data by simply rearranging the terms in MMD. Second, inspired by the hinge loss, we propose a bounded Gaussian kernel to stabilize the training of MMD-GAN with the repulsive loss function. The proposed methods are applied to the unsupervised image generation tasks on CIFAR-10, STL-10, CelebA, and LSUN bedroom datasets. Results show that the repulsive loss function significantly improves over the MMD loss at no additional computational cost and outperforms other representative loss functions. The proposed methods achieve an FID score of 16.21 on the CIFAR-10 dataset using a single DCGAN network and spectral normalization.

OVERALL COMMENTS:
I haven't had much time to write this, so I'm giving a low confidence score and you should feel free to correct me.
I didn't think this paper was very clear.
I had trouble grasping what the contributions were supposed to be
and I had trouble judging the significance of the experiments.
That said, now that (I think) I understand what's going on,
the idea seems well motivated, the connection between the repulsion and the use of label information in other
GAN variants makes sense to me, and the statements you are making seem (as much as I had time to check them) correct.
This leaves the issue of scientific significance.
I feel like I need to understand what specifically contributed to the improvements in table 1 to evaluate significance.
First of all, it seems like there are a lot of other 'good-scoring' models left out of this table.
I understand that you make the claim that your improvement is orthogonal, but that seems like something that needs to
be tested empirically. You have orthogonal motivation but it might be that in practice your technique works for a reason
similar to the reason other techniques work. I would like to see more exploration of this.
Second, are the models below the line the only models using spectral norm? I can't tell.
Overall, it's hard for me to envision this work really seriously changing the course of research on GANs,
but that's perhaps too high a bar for poster acceptance.
For these reasons, I am giving a score of 6.
DETAILED COMMENTS ON TEXT:
> their performance heavily depends on the loss functions used in training.
This is not true, IMO. See [1]
> may discourage the learning of data structures
What does 'data structures' mean in this case?
It has another more common usage that makes this confusing.
> Several loss functions have been proposed
IMO this list doesn't belong in the main body of the text.
I would move it to an appendix.
> We assume linear activation is used at the last layer of D
I'm not sure what this means?
My best guess is just that you're saying there is no activation function applied to the logits.
> Arjovsky et al. (2017) showed that, if the supports of PX and PG do not overlap, there exists a perfect discriminator...
This doesn't affect your paper that much, but was this really something that needed to be shown?
If the discriminator has finite capacity it's not true in general and if it has infinite capacity its vacuous.
> We propose a generalized power iteration method...
Why do this when we can explicitly compute the singular values as in [2]?
Genuine question.
> MS-SSIM is not compatible with CIFAR-10 and STL-10 which have data from many classes;
Just compute the intra-class MS-SSIM as in [3].
> Higher IS and lower FID scores indicate better image quality
I'm a bit worried about using the FID to evaluate a model that's been trained w/ an MMD loss where
the discriminator is itself a neural network w/ roughly the same architecture as the pre-trained image classifier
used to compute the FID. What can you say about this?
Am I wrong to be worried?
> Table 1:
Which models use spectral norm?
My understanding is that this has a big influence on the scores.
This seems like a very important point.
REFERENCES:
[1] Are GANs Created Equal? A Large-Scale Study
[2] The Singular Values of Convolutional Layers
[3] Conditional Image Synthesis With Auxiliary Classifier GANs


GRAPH HYPERNETWORKS FOR NEURAL ARCHITECTURE SEARCH
Neural architecture search (NAS) automatically finds the best task-specific neural network topology, outperforming many manual architecture designs. However, it can be prohibitively expensive as the search requires training thousands of different networks, while each can last for hours. In this work, we propose the Graph HyperNetwork (GHN) to amortize the search cost: given an architecture, it directly generates the weights by running inference on a graph neural network. GHNs model the topology of an architecture and therefore can predict network performance more accurately than regular hypernetworks and premature early stopping. To perform NAS, we randomly sample architectures and use the validation accuracy of networks with GHN generated weights as the surrogate search signal. GHNs are fast -they can search nearly 10× faster than other random search methods on CIFAR-10 and ImageNet. GHNs can be further extended to the anytime prediction setting, where they have found networks with better speed-accuracy tradeoff than the state-of-the-art manual designs.

This paper proposes to accelerate architecture search by replacing the expensive inner loop (wherein candidate architectures are trained to completion) with a HyperNetwork which predicts the weights of candidate architectures, as in SMASH. Contrary to SMASH, this work employs a Graph neural network to allow for the use of any feedforward architecture, enabling fast architecture search through parameter prediction using highly performant search spaces. The authors test their system and show that performance using Graph HyperNet-generated weights correlates with performance when trained normally. The authors benchmark their method against competing approaches ("traditional" NAS techniques which incur the full expense of the inner loop, and one-shot techniques which learn a large model then select architectures by searching for paths in said model) and show competitive performance.
This is a solid technical contribution with a well-designed set of experiments. While the novelty is not especially high, the paper does a good job of synthesizing existing tools and achieves reasonably strong results with much less compute, making for a strong entry into the growing table of fast architecture search methods. I argue in favor of acceptance.
Notes:
-Whereas SMASH is limited to architectures which can be described with its proposed encoding scheme, GHNs only requires that the architecture be represented as a graph (which, to my knowledge, means it can handle any feedforward architecture).
-Section 4.2: It's not entirely clear how this setup allows for variable sized kernels or variable #channels. Is the output of H simply as large as the largest allowable parameter tensor, and sliced as necessary? A snippet of code might be more illuminating here than a set of equations. Additionally (I may have missed this in the text) is the #channels in each node held fixed with a predfined pattern, or also searched for? Are the channels for each node within a block allowed to vary relative to one another?
-Do you sample a new, random architecture at every SGD step during training of the GHN?
-I have no expertise in graph neural networks, and I cannot judge the efficacy of this scheme wrt other GNN techniques, nor can I judge the forward-backward message passing scheme of section 4.4. If another reviewer has expertise in this area and can provide an evaluation that would be great.

-GPU-days is an okay metric, but it's also problematic, since it will of course depend on the choice of GPU (e.g. you can achieve a 10x speedup just from switching from a 600-series to a V100! How does using 4 GPUS for 1 hour compare to 1 GPU for 4 hours? How does this change if you have more CPU power and can load data faster? What if you're using a DL framework which is faster than your competitor's?) Given that the difference here is an order of magnitude, I don't think it matters, but if authors begin to optimize for GPU-milliseconds then it will need to be better standardized.

-Further empirical evidence showing the correlation between approximate performance and true performance is also strong. I very much like that this study has been run for a method based on finding paths in a larger model (ENAS) and shows that ENAS' performance does indeed correlate with true performance, *but* not perfectly, something which (if I recall correctly) is not addressed in the original paper.

-It is worth noting that for ImageNet-Mobile and CIFAR-10 they perform on par with the top methods but tend to use more parameters.
-I like figures 3 and 4, the comparisons against MSDNet and random networks as a function of op budget is good to see.
-Table 4 shows that the correlation is weaker (regardless of method) for the top architectures, which I don't find surprising as I would expect the variation in performance amongst top architectures to be lower. It would be interesting to also see what the range of error rates are; I would expect that the correlation is higher when the range of error rates across the population of architectures is large, as it is easier to distinguish very bad architectures from very good architectures. Distinguishing among a set of good-to-very-good architectures is likely to be more difficult.
-For Section 5.3, I found the choice to use unseen architectures a little bit confusing. I think that even for this study, there's no reason to use a held-out set, as we seek to scrutinize the ability of the system to approximate performance even with architectures it *does* see during training.
-How much does the accuracy drop when using GHN weights? I would like to see a plot showing true accuracy vs. accuracy with GHN weights for the random-100 networks, as using approximations like this typically results in the approximated weights being substantially worse. I am curious to see just how much of a drop there is.
-Section 5.4: it's interesting that performance is stronger when the GHN only sees a few (7) nodes during training, even though it sees 17 nodes during testing. I would expect that the best performance is attained with training-testing parity. Again, as I do not have any expertise in graph neural networks, I'm not sure if this is common (to train on smaller graphs and generalize to larger ones), so if the authors or another reviewer would like to comment and further illuminate this behavior, that would be helpful.
Some typos:
Abstract: "prematured"  should be "premature"
Introducton, last paragraph: "CNN networks." CNN already stands for Convolutional Neural Network.


SPECTRAL INFERENCE NETWORKS: UNIFYING DEEP AND SPECTRAL LEARNING
We present Spectral Inference Networks, a framework for learning eigenfunctions of linear operators by stochastic optimization. Spectral Inference Networks generalize Slow Feature Analysis to generic symmetric operators, and are closely related to Variational Monte Carlo methods from computational physics. As such, they can be a powerful tool for unsupervised representation learning from video or graph-structured data. We cast training Spectral Inference Networks as a bilevel optimization problem, which allows for online learning of multiple eigenfunctions. We show results of training Spectral Inference Networks on problems in quantum mechanics and feature learning for videos on synthetic datasets. Our results demonstrate that Spectral Inference Networks accurately recover eigenfunctions of linear operators and can discover interpretable representations from video in a fully unsupervised manner.

Spectral Inference Networks, Unifying Deep and Spectral Learning
This paper presents a framework to learn eigenfunctions via a stochastic process. They are exploited in an unsupervised setting to learn representation of video data. Computing eigenfunctions can be computationally challenging in large-scale context. This paper proposes to tackle this challenge b y approximating then using a two-phase stochastic optimization process. The fundamental motivation is to merge approaches from spectral decomposition via stochastic approximation and learning an implicit representation. This is achievement with a clever use of masked gradients, Cholesky decomposition and explicit orthogonalization of resulting eigenvectors. A bilevel optimization process finds local minima as approximate eigenfunction, mimicking Borkar’97. Results are shown to correctly recover known 2d- schrodinger eigenfunctions and interpretable latent representation a video dataset, with a practical promising results using the arcade learning environment.
Positive
+ Computation of eigenfunctions on very large settings, without relying on Nystrom approximation
+ Unifying spectral decomposition within a neural net framework
Specific comments
- Accuracy issue - Shape of eigenfunctions are said to be correctly recovered, but no words indicates their accuracy. If eigenfunction values are wrong, this may be critical to the generalization of the method.
- Clarity could be improved in the neural network implementation, what is exactly done and why, when building the network
- Algorithm requires computing the jacobian of the covariance, which can be large and computationally expensive - how to scale it to large settings?
- Fundamentally, a local minimum is reached - any future work on tackling a global solution?  Perhaps by exploring varying learning rates?
- Practically, eigenfunction have an ambiguity to rotation - how is this enforced and checked during validation? (e.g., rotating eigenfunctions in Fig 1c)
- Eigenfunction of transition matrix should, if not mistaken, be smooth, whereas Fig 2a shows granularity in the eigenfunctions values (noisy red-blue maps) - Is this regularization issue, and can this be explicitly correctly?
- Perhaps a word on computational time/complexity?


In distributed training, the communication cost due to the transmission of gradients or the parameters of the deep model is a major bottleneck in scaling up the number of processing nodes. To address this issue, we propose dithered quantization for the transmission of the stochastic gradients and show that training with Dithered Quantized Stochastic Gradients (DQSG) is similar to the training with unquantized SGs perturbed by an independent bounded uniform noise, in contrast to the other quantization methods where the perturbation depends on the gradients and hence, complicating the convergence analysis. We study the convergence of training algorithms using DQSG and the trade off between the number of quantization levels and the training time. Next, we observe that there is a correlation among the SGs computed by workers that can be utilized to further reduce the communication overhead without any performance loss. Hence, we develop a simple yet effective quantization scheme, nested dithered quantized SG (NDQSG), that can reduce the communication significantly without requiring the workers communicating extra information to each other. We prove that although NDQSG requires significantly less bits, it can achieve the same quantization variance bound as DQSG. Our simulation results confirm the effectiveness of training using DQSG and NDQSG in reducing the communication bits or the convergence time compared to the existing methods without sacrificing the accuracy of the trained model.

Authors establish a connection between communication reduction in distributed optimization and dithered quantization. This allows us to understand prior approaches in a new perspective, and also motivates authors to develop two new distributed training algorithms which communication overhead is significantly reduced. The first algorithm, DQSG, uses dithered quantization to reduce the communication bits. The second algorithm, NDQSG, uses nested dithered quantization to further reduce the amount of needed communication. The usefulness of these algorithms are empirically validated by computing the raw communication bits and average entropy of them. Therefore, dithered communication seems to provide both theory and algorithm which are useful.
The paper is clearly written. It provides a succinct review of dithered quantization and previous works, and figures provide a good insight into why the algorithm works, especially Figure 3.
Theorems in this paper are mostly about plugging in properties of dithered quantization into standard results in stochastic optimization, but they are still useful. The analysis of NDQSG does not seem to be as complete as that of DQSG, however. With NQSG, now workers are divided into two groups, and there would be an interesting tradeoff between assignments to these two: how should we balance two groups? This might be tricky to analyze, but it is still useful to clarify limitations and provide conjectures. At least, this could be analyzed empirically.
pros:
* establishing a connection to other topic of research often facilitates productive collaboration between two fields
* provides a new perspective to understand prior work
* provides new useful algorithms
cons:
* experiments were conducted on small models and small datasets
* unclear models are large enough to demonstrate the need for communication reduction; in other words, it is unclear wall-time would actually be reduced with these algorithms.


Deep generative models are capable of learning probability distributions over large, high-dimensional datasets such as images, video and natural language. Generative models trained on samples from p(x) ought to assign low likelihoods to out-of-distribution (OoD) samples from q(x), making them suitable for anomaly detection applications. We show that in practice, likelihood models are themselves susceptible to OoD errors, and even assign large likelihoods to images from other natural datasets. To mitigate these issues, we propose Generative Ensembles, a model-independent technique for OoD detection that combines density-based anomaly detection with uncertainty estimation. Our method outperforms the Outof-DIstribution detector for Neural networks (ODIN) and Variational Information Bottleneck (VIB) baselines on image datasets, and achieves comparable performance to a classification model on the Kaggle Credit Fraud dataset.

Note to Area Chair: Another paper submitted to ICLR under the title “Do Deep Generative Models Know What They Don’t Know?” shares several similarities with the current submission.
This paper highlights a deficiency of current generative models in detecting out-of-distribution based samples based on likelihoods assigned by the model (in cases where the likelihoods are well-defined) or the discriminator distribution for GANs (where likelihoods are typically not defined). To remedy this deficiency, the paper proposes to use ensembles of generative models to obtain a robust WAIC criteria for anomaly detection.
My main concern is with the level of technical rigor of this work. Much of this has to do with the presentation, which reads to me more like a summary blog post rather than a technical paper.
- I couldn’t find a formal specification of the anomaly detection setup and how generative models are used for this task anywhere in the paper.
- Section 2 seems to be the major contribution of this work. But it was very hard to understand what exactly is going on. What is the notation for the generative distribution? Introduction uses p_theta. Page 2, Paragraph 1 uses q_theta (x). Eq. (1) uses p_theta and then the following paragraphs use q_theta.
- In Eq. (1), is theta a random variable?
- How are generative ensembles trained?  All the paper says is “independently trained”. Is the parameter initialization different? Is the dataset shuffling different? Is the dataset sampled with replacement (as in bootstrapping)?
- “By training an ensemble of GANs we can estimate the posterior distribution over model deciscion boundaries D_theta(x), or equivalently, the posterior distribution over alternate distributions q_theta. In other words, we can use uncertainty estimation on randomly sampled discriminators to de-correlate the OoD classification errors made by a single discriminator” Why is the discriminator parameterized by theta? What is an ensemble of GANs? Multiple generators or multiple discriminators or both? What are “randomly sampled discriminators”? What do the authors mean by "posterior distribution over alternate distributions"?
With regards to the technical assessment, I have the following questions for the authors:
- In Figure 1, how do the histograms look for the training distribution of CIFAR? If the histograms for train and test have an overlap much higher than the overlap between the train of CIFAR and test set of any other distribution, then ensembling seems unnecessary and anomaly detecting can simply be done via setting a maximum and a minimum threshold on the likelihood for a test point. In addition to the histograms, I'd be curious to see results with this baseline mechanism.
- Why should the WAIC criteria weigh the mean and variance equally?
- Did the authors actually try to fix the posterior collapse issue in Figure 3b using beta-VAEs as recommended? Given the simplicity of implementing beta-VAEs, this should be a rather easy experiment to include.
Minor typos:
- ODIN and VIB are not defined in the abstract
- Page 3: “deciscion”
- Page 2, para 2: “log_\theta p(x)”


Events that we observe in the world may be caused by other, unobserved events. We consider sequences of discrete events in continuous time. When only some of the events are observed, we propose particle smoothing to infer the missing events. Particle smoothing is an extension of particle filtering in which proposed events are conditioned on the future as well as the past. For our setting, we develop a novel proposal distribution that is a type of continuous-time bidirectional LSTM. We use the sampled particles in an approximate minimum Bayes risk decoder that outputs a single low-risk prediction of the missing events. We experiment in multiple synthetic and real domains, modeling the complete sequences in each domain with a neural Hawkes process (Mei & Eisner, 2017). On held-out incomplete sequences, our method is effective at inferring the ground-truth unobserved events. In particular, particle smoothing consistently improves upon particle filtering, showing the benefit of training a bidirectional proposal distribution.

The authors propose a particle smoothing approach with an approximate minimum Bayes risk decoder to impute missing events in the Neural Hawkes Process (NHP). The main goal is to address the missing events problem in continuous-time event analysis, which is an important problem in practice. The core idea is within the framework of particle smoothing.
To formulate the posterior distribution of the missing event, the authors consider both the left-to-right past events and the right-to-left future events. The paper first applies the NHP to capture both the observed and inferred missing events to learn a representation of the past events, and then uses a similar NHP to learn the representation of the observed events from the future. Based on the two representations, it then formulates the intensity function of the missing events and uses the thinning algorithm to sample different particles. Based on the proposed distribution, the paper also considers to decode a single prediction achieving the Minimum Bayes Risk. Experiments on synthetic datasets with 10 different initializations and two real datasets show that the proposed smoothing approach is better than the filtering baseline.
In general, this paper considers an important problem which is under active research in literature recently. However, there are a few weaknesses of the paper that should be addressed.
1. The proposed technique is tightly connected to NHP, which could limit the applicability of the approach to other temporal point processes. The essential idea is similar to Bi-LSTM to learn the representation from both ends of a sequence of asynchronous temporal events. There are several different ways to represent the inter-event time to feed into the network other than NHP. Can the proposed method also be applied to other processes?
2. Within the particle filtering framework, each particle (hypothesis) is weighted by the likelihood of the sequence of observed events under that hypothesis. It turns out that the integral part of Equation 5 does not have an obvious analytical solution under NHP. Then, we first need a set of samples to approximate the likelihood evaluation. Later, we also need to sample particles. I am not quite convinced the computational efficiency of this approach in real applications of practice. Also, there is no analysis either empirically or analytically about the impact of the accumulative sampling errors on the inference performance. Furthermore, to learn the proposed distribution, the paper applies the REINFORCE algorithm under the proposed distribution q. But REINFORCE is known for large variance issue. Given that we already need lots of samples for the likelihood, it is unclear to me how stable the algorithm could be in practice.
3. The experimental evaluation is weak. For particle filtering and smoothing, it is known that the filtering techniques are candidates for solving the smoothing problem but perform poorly when T is large. That's why it is necessary to develop more sophisticated strategies for good smoothing
algorithms. As a result, it is unfair to only compare the smoothing approach with the filtering baseline.
Actually, what people really care about is how different techniques can behave in real data to impute realistic missing events. From this perspective, I suggest to use the QQ-plot to evaluate the goodness of fitting on the synthetic dataset. For example, given a sequence of events generated from an independent temporal point process, we can randomly delete events, and then apply different techniques, including Linderman et al. (2017), Shelton et al.(2018), to impute missing events. Finally, we can compare the imputed sequence of events with the groundtruth.
In addition, sequential monte carlo approach often suffers from skewed particle issue where one particle gradually dominates all the other particles with no diversity. It is unclear how the proposed approach is able to handle this.
One missing related paper is "Learning Hawkes Processes from Short Doubly-Censored Event Sequences"
Section 5.2 can be significantly strengthened if comparing with at least one of these approaches.
4. The paper is fairly written. I had some trouble reading back and forth for understanding Figure 1 since it has long caption that is not self-contained. The annotation of Section 2 is also too heavy to quickly skim through to memorize.


Current generation of memory-augmented neural networks has limited scalability as they cannot efficiently process data that are too large to fit in the external memory storage. One example of this is lifelong learning scenario where the model receives unlimited length of data stream as an input which contains vast majority of uninformative entries. We tackle this problem by proposing a memory network fit for long-term lifelong learning scenario, which we refer to as Long-term Episodic Memory Networks (LEMN), that features a RNN-based retention agent that learns to replace less important memory entries based on the retention probability generated on each entry that is learned to identify data instances of generic importance relative to other memory entries, as well as its historical importance. Such learning of retention agent allows our long-term episodic memory network to retain memory entries of generic importance for a given task. We validate our model on a path-finding task as well as synthetic and real question answering tasks, on which our model achieves significant improvements over the memoryaugmented networks with rule-based memory scheduling as well as an RL-based baseline that does not consider relative or historical importance of the memory.

This work tackles the problems encountered by bounded memory storage mechanisms when faced with abundant data, of which much may be irrelevant or redundant. Such a problem is faced in lifelong learning settings, where a limitless data stream must somehow be encoded and stored so as to be useful at later points in time.
The researchers propose a solution based on “learning what to remember”. That is, rather than encode every observation (which can quickly become problematic), the model learns to replace less important memories. The importance of a memory is determined by its correlation with future reward; a “memory retention policy” is learned via reinforcement learning, wherein the model learns to retain or discard memories based on these actions’ (i.e., retentions) impact on future reward. Experiments to show the effectiveness of this mechanism include gridworld IMaze and Random Mazes, bAbI question answering (task 2), and Trivia QA.
Altogether the work does well to clearly describe an interesting approach to an important problem. The model is motivated and explained well, and there were no issues with understanding its inner workings.
Regarding the work’s novelty, there is a precedent for using RL-based write schemes (DNTM from Gulcehre et al, 2016), which the authors point out. I am not entirely convinced that the proposed writing scheme is a substantial addition over this past work, but I am not overly concerned about this since proper due credit is assigned in the paper. Perhaps a bit more discussion about the advantages of the proposed writing scheme could go a long way, since as it stands now, the paper simply claims that this past work “only considers the pairwise relationships between the current data instance and each individual memory”, and I’m not sure how much substance actually underlies this difference.
Unfortunately I think there is a fundamental problem with the work. The model is a proposed solution for problems with vast amounts of streaming data; problems that, presumably, current memory models would struggle with. However, the tasks in the paper do not fall in this domain. Instead, the authors chose to artificially cripple the size of their memory (using, for example, just a handful of memory “slots”) and demonstrate its performance on tasks that are otherwise completely within the realm of being solved by conventional memory models. This is fine as a jumping off point for the research, but for the model to be taken seriously as a valid solution to problems involving such a scale of data that current models cannot even cope, then it needs to show its worth on problems involving such a scale of data that current models cannot cope.
Demonstrating success here is important for a few reasons. First, such high-data scenarios may involve situations where many, many memories need to be encoded and considered for the future, since they are all useful or necessary for future performance. The experiments do not show whether the model can scale to, say, 100 or 1000 memories, which is within the realm of being “reasonable” for current memory architectures. Second, high-data scenarios may involve an abundant amount of distracting, irrelevant data. This places particularly tough demands on the RL-based writing mechanism, which will undoubtedly face problems with temporal credit assignment if: (a) the time between encoding and retrieval is long, and (b) there is high reward noise in the intermediate time. Thus, the authors should stress-test the components of their model, since these stresses will undoubtedly exist in the problems that the model is proposed to solve.
Some other minor considerations include the following. (1) The use of a single bAbI task is questionable. Why not run the model on the full suite? (2) How do conventional memory models perform on the tasks? Why are the baselines only variants of the proposed model?
To conclude and summarize, as a proposed solution to scenarios with streams of abundant data -- which the authors claim is a domain that current memory models may struggle -- the proposed model should tackle problems that: 1) have characteristics more reminiscent of these scenarios, and 2) are problems on which current memory models struggle, for the reasons claimed in the paper. In particular, it would be valuable to see model performance on tasks wherein very long stretches of time need to be considered. This is important because it can address questions with memory scaling (how does the model cope with more than a handful of memories?), and issues that would crop up in a reinforcement learning-based approach to memory retention over long time intervals (namely, long-term temporal credit assignment).