Training a neural network for a classification task typically assumes that the data to train are given from the beginning. However, in the real world, additional data accumulate gradually and the model requires additional training without accessing the old training data. This usually leads to the catastrophic forgetting problem which is inevitable for the traditional training methodology of neural networks. In this paper, we propose a memory-based continual learning method that is able to learn additional tasks while retaining the performance of previously learned tasks. Composed of two complementary networks, the Hippocampus-Net (H-Net) and the Cortex-Net (C-Net), our model estimates the index of the corresponding task for an input sample and utilizes a particular portion of itself with the estimated index. The C-Net guarantees no degradation in the performance of the previously learned tasks and the H-Net shows high confidence in finding the origin of an input sample.

The key idea of this paper is to expand the network for training on new tasks which is termed as C-Net, and train an additional generative model which is used for predicting task id (which is called H-Net), and use the task id for selecting weights from the C-Net.
Pros:
1. It is relatively easy to understand the paper.
2. The originality of this paper lies in the usage of generative model to predict task id (H-Net). To my knowledge has not been proposed before.
3. In contrast to previous works in multi-task learning, which assumes task id is available both during training and inference, this work tries to remove the need of task id during inference, which makes it closer to the general definition of continual learning.
Cons:
1. Expanding the network for new tasks is not a novel contribution of this paper, it has already been proposed in previous works on multi-task learning. Doing expansion on all of the layers does not qualify for a major contribution in my opinion.
2. The experimental comparison is not very fair in my opinion,
     a. Comparing accuracy of C-Net to other methods is not very useful. Because this methods expands the network for every new task, while other methods (EWC, LwF) has limited to no expansion in the network. Given that the single network result is far from state of the art (table 3), I suppose model size could contribute to the accuracy boost.
     b. It is not explicitly stated in the paper whether the output neurons are shared between tasks or an individual set of output neurons are used for different tasks, but from the rest of the paper I suppose disjoint neurons are used. Then the comparison between EWC and this work is not fair because EWC shares the output neurons among tasks.
This is not to blame this paper for not making fair comparison, since given different assumptions between methods (availability of task id, shared output neurons etc.), it is usually difficult to fairly compare between continual learning methods.  This problem is raised in another submission https://openreview.net/forum?id=ByGVui0ctm. The point here is that the accuracy of C-Net is not a good measure of how good this method is.
3. I disagree with the point that MNIST and SVHN are similar, they have very different distributions and are very easy to tell apart with a model. One concern is that the generative H-Net may fail to work once the distributions of the tasks overlap to some extent. e.g. cifar10 vs cifar100.
As a conclusion, the key contribution of this work is using generative model to determine task id which removes the need for task id during inference. It is relatively insufficient for publication on ICLR.


Recently, progress has been made towards improving relational reasoning in machine learning field. Among existing models, graph neural networks (GNNs) is one of the most effective approaches for multi-hop relational reasoning. In fact, multi-hop relational reasoning is indispensable in many natural language processing tasks such as relation extraction. In this paper, we propose to generate the parameters of graph neural networks (GP-GNNs) according to natural language sentences, which enables GNNs to process relational reasoning on unstructured text inputs. We verify GP-GNNs in relation extraction from text. Experimental results on a human-annotated dataset and two distantly supervised datasets show that our model achieves significant improvements compared to baselines. We also perform a qualitative analysis to demonstrate that our model could discover more accurate relations by multi-hop relational reasoning.

This work proposes a method for parametrising relation matrices in graph neural networks (GNNs) using text. The model is applied in a relation extraction task, and specific dataset subsections are identified to test and analyse “hopping” behaviour: a model’s ability to combine multiple relations for inferring a new one.
Strengths
- strong results
- testing on both bag-level / single-level relation extraction
- Insights via multiple ablations — variation of the number of layers, exploring densely connected data subsections with cycles to identify examples for multi-hop inference
- evaluation on new, human-annotated test set
Issues
- evaluation only on one task, and one dataset (although, with more detail).
- unclear how/why the specific result metrics and their particular range of precision are chosen. Why is F1/Acc only reported for the human-labelled part, but not for the distantly labelled part? Why is precision cut off at ~25% recall? A comprehensive aggregate measure across all levels of recall would be more informative, e.g. PR-AUC., and consistently applied in all experiments.
- task varies from previous versions of the task, posing a potential problem with comparability. What is the motivation behind augmenting the dataset with “NA” labels? Why is the task from previous work altered to predicting relationships between _every_ entity pair? Also unclear: is predicting “NA” actually part of the training loss, and of the evaluation? Is training of the previous “baseline” models adapted accordingly?
- some claims appear too bold and vague — see below.
- comparatively small modelling innovation
- There is similar prior work to this, most prominently Schlichtkrull et al. 2017 [https://arxiv.org/pdf/1703.06103.pdf] who evaluate on Fb15k-237, but also De Cao et al. 2018 [https://arxiv.org/pdf/1808.09920.pdf, published within 1 month before ICLR submission deadline] who evaluate on Wikihop. These previous methods in fact do the following: “Most existing GNNs only process multi-hop relational reasoning on pre-defined graphs and cannot be directly applied in natural language relational reasoning.” It would be good to work out differences to these previous models in more detail, especially to Schlichtkrull et al. 2017.
- unclear how big the specific contribution of the language-generated matrices is. I would normally not obsess about such a baseline, but it seems that the “generate using NL” aspect is a core to the paper (even in the title), and this isn’t worked out clearly.
More comments / questions:
- “multi-hop relational reasoning is indispensable in many natural language processing tasks such as relation extraction”. This isn’t clear to me, “indispensable” is a very strong wording.
- “state-of-the-art baselines”. SOTA defines the best previous work. How can there be several baselines (plural) which are all best? What does SOTA mean when a slight redefinition of the task is proposed, as in this work?
- “Relation extraction from text is a classic natural language relational reasoning task” — reference would be useful.
- not a big issue, though this does sound somewhat contradictory: 1) “the number of layers K is chosen to be of the order of the graph diameter”. 2) “We treat K as a hyperparameter”
- not clear: is LSTM indeed exactly the same as GP-GNN with K=1? I assume there is a difference, as the LSTM encodes the entire sentence at once, conditioning entity representations on their local context, whereas in GP-GNN this would not be the case.
- The distinction to Context-Aware RE (CARE) is not clear. The authors argue that CARE models co-occurrence of multiple relations, but is this not what a multi-hop relation extraction _should_ learn? It is also not clear how GP-GNN differs in this respect.
- It would be interesting to compare with a model which does not use language to define relation matrices (A), but learns them directly as parameters (independently from the text).
- It would be interesting to see an analysis of the matrices A_{i,j}. What does the text generate, and how can one find this out?


Feedforward convolutional neural network has achieved a great success in many computer vision tasks. While it validly imitates the hierarchical structure of biological visual system, it still lacks one essential architectural feature: contextual recurrent connections with feedback, which widely exists in biological visual system. In this work, we designed a Contextual Recurrent Convolutional Network with this feature embedded in a standard CNN structure. We found that such feedback connections could enable lower layers to "rethink" about their representations given the top-down contextual information. We carefully studied the components of this network, and showed its robustness and superiority over feedforward baselines in such tasks as noise image classification, partially occluded object recognition and fine-grained image classification. We believed this work could be an important step to help bridge the gap between computer vision models and real biological visual system.

This paper presents a novel deep learning module for recurrent processes. The general idea and motivation are generally appealing but the experimental validation is a mess. Architectures and hyper-parameters are casually changed from experiment to experiment (please refer to Do CIFAR-10 Classifiers Generalize to CIFAR-10? By Recht et al 2018 to understand why this is a serious problem.) Some key evaluations are missing (see below). Key controls are also lacking. This study is not the first to look into recurrent / feedback processes. Indeed some (but not all) prior work is cited in the introduction. Some of these should be used as baselines as opposed to just feedforward networks. TBut with all that said, even addressing these concerns would not be sufficient for this paper to pass threshold since overall the improvements are relatively modest (e.g., see Fig. 4 right panel where the improvements are a fraction of a % or left panel with a couple % fooling rates improvements) for a module that adds significant computational cost to an architecture runtime. As a side note, I would advise to tone down some of the claims such as "our network could outperform baseline feedforward networks by a large margin”...
****
Additional comments:
The experiments are all over the place. What is the SOA on CIFAR-10 and CIFAR-100? If different from VGG please provide a strong rationale for testing the circuit on VGG and not SOA. In general, the experimental validation would be much stronger if consistent improvements were shown across architectures.
Accuracy is reported for CIFAR-10 and CIFAR-100 for 1 and 2 feedback iterations and presumably with the architecture shown in Fig. 1. Then robustness to noise and adversarial attacks tested on ImageNet and with a modification of the architecture. According to the caption of Fig. 4, this is done with 5 timesteps this time! Accuracy on ImageNet needs to be reported ** especially ** if classification accuracy is not improved (as I expect).
Then experiments on fine-grained with ResNet-34! What architecture is this? Is this yet another number of loops and feedback iterations? When reporting that "Our model can get a top-1 error of 25.1, while that of the ResNet-34 model is 26.5.” Please provide published accuracy for the baseline algorithm.
For the experiment on occlusions, the authors report using “a multi-recurrent model which is similar to the model mentioned in the Imagenet task”. Sorry but this is not good enough.
Table 4 has literally no explanation. What is FF? What are unroll times?
As a side note, VGG-GAP does not seem to be defined anywhere.
When stating "We investigated VGG16 (Simonyan & Zisserman, 2014), a standard CNN that closely approximate the ventral visual hierarchical stream, and its recurrent variants for comparison.”, the authors probably meant “coarsely” not “closely".


In many robotic applications, it is crucial to maintain a belief about the state of a system, like the location of a robot or the pose of an object. These state estimates serve as input for planning and decision making and provide feedback during task execution. Recursive Bayesian Filtering algorithms address the state estimation problem, but they require a model of the process dynamics and the sensory observations as well as noise estimates that quantify the accuracy of these models. Recently, multiple works have demonstrated that the process and sensor models can be learned by end-to-end training through differentiable versions of Recursive Filtering methods. However, even if the predictive models are known, finding suitable noise models remains challenging. Therefore, many practical applications rely on very simplistic noise models. Our hypothesis is that end-to-end training through differentiable Bayesian Filters enables us to learn more complex heteroscedastic noise models for the system dynamics. We evaluate learning such models with different types of filtering algorithms and on two different robotic tasks. Our experiments show that especially for sampling-based filters like the Particle Filter, learning heteroscedastic noise models can drastically improve the tracking performance in comparison to using constant noise models.

This is a well written paper which proposes to learn heteroscedastic noise models from data by optimizing the prediction likelihood end-to-end through differentiable Bayesian Filters. In addition to existing Bayesian filters, the paper also proposes two different versions of the [differentiable] Unscented Kalman Filter. Performance of the different filters and noise models is evaluated on two real-world robotic problems: Visual Odometry and visual tracking of an object pushed by the robot.
While the general idea of learning the noise variances through backpropagation are straightforward extensions of existing work on differential Bayesian filters, the questions that the paper explores are important to make end-to-end learning of Bayesian filter more common. The results will help future research select the correct differential filter for their use case, and insight in potential benefits (or lack thereof) by learning heteroscedastic or homoscedastic process noise, and/or observation noise.
A downside is that the paper does not further explore how to weigh different loss terms which are apparently important to successfully train such models. Also unfortunate is the footnote which states that the current results are incomplete and will be updated, hence as a reviewer I am not sure which results and conclusions are valid right now.
Pros:
+ clearly written
+ useful experiments for those seeking to select a differential Bayesian filter, and learning (heteroscedastic) noise from data.
+ experiments on real-world use cases rather than toy problems
Cons:
- Incomplete experiments according to footnote, thus results and conclusions might change after this review.
- Unclear what the effect of the selected process / observation model is on the learned noise
Below are more detailed comments and questions:
* p6. Footnote: "due to time constraints, ..., results will be updated" Is this acceptable? I have never seen such a notice when reviewing. So, are the current results on a single fold? Will the numbers in the tables, or the conclusions change after this review?
* If I understand correctly, the paper 'only' focuses on learning the heteroscedastic noise variance, but assumes that the deterministic non-linear parts of the process and observation models are fixed. I did not find this very clearly stated in the paper, though at least the Appendix explicitly states the used functions for the process models.
* I would have liked to see in the paper more explanation on how the process and observations models were selected and validated  in the experiments, since I expect that the validity of these functions affects the learned noise variances. Since the noise needs to account for the inaccuracies in the deterministic models, would the choice for these functions not impact your conclusions? And, would it or would it not be possible to learn both these deterministic models and the noise jointly from the training data?
* Is it possible to add priors on Q and R parameters for Bayesian treatment of learning model parameters? I can imagine that priors can guide the optimization to either adjust more of the Q or more of the R variance to improve the likelihood.
* Section 1:
	* "Our experiments show that ... " This may be a matter of taste, but I did not expect to see the main conclusions already in the introduction. They should appear in the abstract to help out the quick reader. In the introduction, it appears as if you are talking about some separate preliminary experiments, and which you base some conclusions that will be used in the remainder of this paper.
* Section 3:
	* So, mostly empirical study, since heteroscedastic noise models were already used?
	* "Previous work evaluated ... " please add citations
* Section 4.1:
	* "train a discriminative neural network o with parameters wo to preprocess the raw sensory data D and thus create a more compact representation of the observations z = o(D;wo)." At this point in the paper, I don't understand this. How is z learned, via supervised learning (what is the target value for z)? Or is z some latent representation that is jointly optimized with the filters? This only became somewhat clearer in Sec. 5.2 on p.8 where it states that "We ... train a neural network to extract the position of the object, the contact point and normal as well as ...". So if I understand correctly, the function o for z = o(D) is thus learned offline w.r.t. some designed observation variables for which GT is available (from manual annotations?).
* Section 4.2:
	* "we predict a separate Qi for every sigma point and then compute Q as the weighted mean" → So, separate parameters w_g for each sigma point i, or is a single learned non-linear function applied to all points?
* Section 4.3:
	* Equation 14: inconsistent use of boldface script: should use bold sigma_t, and bold l_t ?
	* "In practice, we found that during learning ... by only increasing the predicted variance" →  This is an interesting observation, which I would have liked to see explored more. I understand that term (ii) is needed to guide the learning processes, but in the end wouldn't we want to optimize the actual likelihood? So, could you (after the loss with (ii) converged) reduce \lambda_2 to zero to properly optimize only the log likelihood without guidance from a good initial state? Or is it not possible to reliably optimize the likelihood via back-propagation at all from some reason?
* Section 5.1.1
	* "... of varying length (from 270 to over 4500 steps) ..." it would be good to mention the fps, to get understand to what real-world time horizons 50 / 100 frames correspond.
* Section 5.1.2:
	* Table 1: How are the parameters of the filters in the "no learning" column obtained? Are these tuned in some other way, or taken form existing implementations? Also, can you clarify if the 'no learning' parameters served as the initial condition for the learning approaches?
	* Table 1, first row column Q+R: "0.2" → Is there a missing zero here, i.e. "0.20"? Otherwise, the precision of reported results in this table is not consistent. Hard to say: is the mean of R+Q 0.2, and slightly lower than R+Qh, or could it be as high as 0.24 ?
	* "learning a heteroscedastic process noise model leads to big improvements and makes the filters competitive with the EKF". Results for EKF still appear significantly better than the novel UKF, and even the PF (especially rotational error).
* Section 6:
	* "Large outliers in the prediction of the preprocessing networks were not associated with higher observation noise." I don't see on what presented results these conclusions were drawn, as this is the first time the word "outlier" is mentioned in the paper. Outliers seem indeed important, as they contradict the typical assumptions e.g. of Gaussian noise, so it would be useful to clarify how the proposed techniques handle such outliers.


THE LOTTERY TICKET HYPOTHESIS: FINDING SPARSE, TRAINABLE NEURAL NETWORKS
Neural network pruning techniques can reduce the parameter counts of trained networks by over 90%, decreasing storage requirements and improving computational performance of inference without compromising accuracy. However, contemporary experience is that the sparse architectures produced by pruning are difficult to train from the start, which would similarly improve training performance.
We find that a standard pruning technique naturally uncovers subnetworks whose initializations made them capable of training effectively. Based on these results, we articulate the lottery ticket hypothesis: dense, randomly-initialized, feed-forward networks contain subnetworks (winning tickets) that-when trained in isolationreach test accuracy comparable to the original network in a similar number of iterations. The winning tickets we find have won the initialization lottery: their connections have initial weights that make training particularly effective.
We present an algorithm to identify winning tickets and a series of experiments that support the lottery ticket hypothesis and the importance of these fortuitous initializations. We consistently find winning tickets that are less than 10-20% of the size of several fully-connected and convolutional feed-forward architectures for MNIST and CIFAR10. Above this size, the winning tickets that we find learn faster than the original network and reach higher test accuracy.

(Score raised from 8 to 9 after rebuttal)
The paper examines the hypothesis that randomly initialized (feed-forward) neural networks contain sub-networks that train well in the sense that they converge equally fast or faster and reach the same or better classification accuracy. Interestingly, such sub-networks can be identified by simple, magnitude-based pruning. It is crucial that these sub-networks are initialized with their original initialization values, otherwise they typically fail to be trained, implying that it is not purely the structure of the sub-networks that matters. The paper thoroughly investigates the existence of such “winning-tickets” on MNIST and CIFAR-10 on both, fully connected but also convolutional neural networks. Winning-tickets are found across networks, various optimizers, at different pruning-levels and across various other hyper-parameters. The experiments also show that iterative pruning (with re-starts) is more effective at finding winning-tickets.
The paper adds a novel and interesting angle to the question of why neural networks apparently need to be heavily over-parameterized for training. This question is intriguing and of high importance to further the understanding of how neural networks train. Additionally, the findings might have practical relevance as they might help avoid unnecessary over-parameterization which, in turn, might save use of computational resources and energy. The main idea is simple (which is good) and can be tested with relatively simple experiments (also good). The experiments conducted in the paper are clean (averaging over multiple runs, controlling for a lot of factors) and should allow for easy reproduction but also for clean comparison against future experiments. The experimental section is well executed, the writing is clear and good and related work is taken into account to a sufficient degree. The paper touches upon a very intriguing “feature” of neural networks and, in my opinion, should be relevant to theorists and practitioners across many sub-fields of deep learning research. I therefore vote and argue for accepting the paper for presentation at the conference. The following comments are suggestions to the authors on how to further improve the paper. I do not expect all issues to be addressed in the camera-ready version.
1) The main “weakness” of the paper might be that, while the amount of experiments and controls is impressive, the generality of the lottery ticket hypothesis remains somewhat open. Even when restricting the statement to feed-forward networks only, the networks investigated in the paper are relatively “small” and MNIST and CIFAR-10 bear the risk of finding patterns that do not hold when scaling to larger-scale networks and tasks. I acknowledge and support the author’s decision to have thorough and clean experiments on these small models and tasks, rather than having half-baked results on ImageNet, etc. The downside of this is that the experiments are thus not sufficient to claim (with reasonable certainty) that the lottery ticket hypothesis holds “in general”. The paper would be stronger, if the existence of winning tickets on larger-scale experiments or tasks other than classification were shown - even if these experiments did not have a large number of control experiments/ablation studies.
2)  While the paper shows the existence of winning tickets robustly and convincingly on the networks/tasks investigated, the next important question would be how to systematically and reliably “break” the existence of lottery tickets. Can they be attributed to a few fundamental factors? Are they a consequence of batch-wise, gradient-based optimization, or an inherent feature of neural networks, or is it the loss functions commonly used, …? On page 2, second paragraph, the paper states: ”When randomly reinitialized, our winning tickets no longer match the performance of the original network, explaining the difficulty of training pruned networks from scratch”. I don’t fully agree - the paper certainly sheds some light on the issue, but an actual explanation would result in a testable hypothesis. My comment here is intended to be constructive criticism, I think that the paper has enough “juice” and novelty for being accepted - I am merely pointing out that the overall story is not yet conclusive (and I am aware that it might need several more publications to find these answers).
3) Do the winning tickets generalize across hyper-parameters or even tasks. I.e. if a winning ticket is found with one set of hyper-parameters, but then Optimizer/learning-rate/etc. are changed, does the winning-ticket still lead to improved convergence and accuracy? Same question for data-sets: do winning-tickets found on CIFAR-100 also work for CIFAR-10 and vice versa? If winning-tickets turn out to generalize well, in the extreme this could allow “shipping” each network architecture with a few good winning-tickets, thus making it unnecessary to apply expensive iterative pruning every time. I would not expect generalization across data-sets, but it would be highly interesting to see if winning tickets generalize in any way (after all I am still surprised by how well adversarial examples generalize and transfer).
4) Some things that would be interesting to try:
4a) Is there anything special about the pruned/non-pruned weights at the time of initialization? Did they start out with very small values already or are they all “behind” some (dead) downstream neuron? Is there anything that might essentially block gradient signal from updating the pruned neurons? This could perhaps be checked by recording weights’ “trajectories” during training to see if there is a correlation between the “distance weights traveled” and whether or not they end up in the winning ticket.
4b) Do ARD-style/Bayesian approaches or second-order methods to pruning identify (roughly) the same neurons for pruning?
5) Typo (should be through): “we find winning tickets though a principled search process”
6) For the standard ConvNets I assume you did not use batchnorm. Does batchnorm interfere in any way with the existence of winning tickets? (at least on ResNet they seem to exist with batchnorm as well)


The ability to look multiple times through a series of pose-adjusted glimpses is fundamental to human vision. This critical faculty allows us to understand highly complex visual scenes. Short term memory plays an integral role in aggregating the information obtained from these glimpses and informing our interpretation of the scene. Computational models have attempted to address glimpsing and visual attention but have failed to incorporate the notion of memory. We introduce a novel, biologically inspired visual working memory architecture that we term the Hebb-Rosenblatt memory. We subsequently introduce a fully differentiable Short Term Attentive Working Memory model (STAWM) which uses transformational attention to learn a memory over each image it sees. The state of our HebbRosenblatt memory is embedded in STAWM as the weights space of a layer. By projecting different queries through this layer we can obtain goal-oriented latent representations for tasks including classification and visual reconstruction. Our model obtains highly competitive classification performance on MNIST and CIFAR-10. As demonstrated through the CelebA dataset, to perform reconstruction the model learns to make a sequence of updates to a canvas which constitute a parts-based representation. Classification with the self supervised representation obtained from MNIST is shown to be in line with the state of the art models (none of which use a visual attention mechanism). Finally, we show that STAWM can be trained under the dual constraints of classification and reconstruction to provide an interpretable visual sketchpad which helps open the 'black-box' of deep learning.

Revision:
The authors have took in the feedback from myself and the other reviewers wholeheartedly, and have clearly worked hard to improve the results, and the paper during the revision process. In addition, their code release encourages easy reproducibility of their model, which imo is needed for this work given the non-conventional nature of the model (that being said, the paper itself is well written and the authors have done sufficiently well in explaining their approach, and also the motivation behind it, as per my original review). The code is relatively clear and self-contained demonstrating their experiments on MNIST, CelebA demonstrating the use of the visual sketch model.
I believe the improvements, especially given the compute resources available to the authors, warrant a strong accept of this work, so I revised my score to 9. I also believe this work will be of value to the ICLR community as it offers alternate, less explored approaches compared to methods that are typically used in this domain. I'm excited to see more in the community explore biologically inspired approaches to generative models, and I think this work along with the code base will be an important base for other researchers to use as a reference point for future work.
Original Review below:
Summary: They propose a biologically motivated short term attentive working memory (STAWM) generative model for images. The architecture is based on Hebbian Learning (i.e. associative memories are represented in the weight matrices that are dynamically updated during inference by a modified version of Hebbian learning rule). These memories are sampled from glimpses on an input image (using attention on contextual states, similar to [1]), in addition to a latent, query state. This model learns a representation of images that can be used for sequential reconstruction (via a sequence of updates, like a sketchpad, like DRAW [1], trained in an unsupervised manner). These memories produced by drawing can also be used for semi-supervised classification (achieves very respectable and competitive results for MNIST and CIFAR-10).
This paper is beautifully written, and the biological inspiration, motivation behind this work, and links to neuroscience literature as well as relation to existing ML work (even recent papers) is well stated. The main strength of this paper is that the author went from a biologically inspired idea to a complete realization of the idea in algorithmic form. The semi-supervised classification results are competitive to SOTA, and although the CIFAR-10 reconstruction results are not great (especially compared to generative adversarial models or recent variation models [2]), I think the approach is coming from a very different angle that is different enough compared to the literature to warrant some attention, or at least a glimpse, so to speak, from the broader community. The method may offer new ways to interpret ML models that is current models lack, which in itself is an important contribution. That being said, the fact that most adversarial generative models achieved a far better performance raises concern on the generalization ability of these memory-inspired learned representations, and I look forward to seeing future work investigate this area in more detail.
The authors also took great care in writing details for important parts of the experiments in the Appendix section, and open sourced the implementation to reproduce all their experiments. Given the complex nature of this model, they did a great job in writing a clear explanation, and provided enough details for the community to build biologically inspired models for deep networks. Even without the code, I felt I might have been able to implement most of the model given the detail and clarity of the writing, so having both available is a great contribution.
I highly recommend this paper for acceptance, with a score of 8 (edit: revised to 9 after rebuttal period). The paper might warrant a score of 9 if they had also achieved higher quality results for image generation, on Celeb-A or demonstrated results on ImageNet, and provided more detailed analysis about drawbacks of their approach vs conventional generative models.
[1] https://arxiv.org/abs/1502.04623
[2] https://arxiv.org/abs/1807.03039


ADAPTIVE POSTERIOR LEARNING: FEW-SHOT LEARNING WITH A SURPRISE-BASED MEMORY MODULE
The ability to generalize quickly from few observations is crucial for intelligent systems. In this paper we introduce APL, an algorithm that approximates probability distributions by remembering the most surprising observations it has encountered. These past observations are recalled from an external memory module and processed by a decoder network that can combine information from different memory slots to generalize beyond direct recall. We show this algorithm can perform as well as state of the art baselines on few-shot classification benchmarks with a smaller memory footprint. In addition, its memory compression allows it to scale to thousands of unknown labels. Finally, we introduce a meta-learning reasoning task which is more challenging than direct classification. In this setting, APL is able to generalize with fewer than one example per class via deductive reasoning.

Summary: the authors propose a new algorithm, APL, for a few-shot and a life-long learning based on an external memory module. APL uses a surprise-based signal to determine which data points to store in memory and an attention mechanism to the most relevant points for prediction. The authors evaluate APL on a few-shot classification task on Omniglot dataset and on a number analogy task.
Quality: the authors consider interesting approach to life-long learning and I really liked the idea of a surprise-based signal to choose the data to store. However, I am not convinced by the learning setting that authors study. While a digit-symbol task from the introduction is interesting to study the properties of APL, I fail to see any real world analogy where it is useful. The same happens in a few-shot omniglot classification. The authors decided to shuffle the labels within episodes that, I guess, is supposed to represent different tasks in a typical life-long learning scenario. Again, it maybe interesting to study the behaviour of the algorithm, but I don't see any practical relevance here. It would make more sense to study the algorithm in a life-long learning setting, for example, considered in [1] and [2].
Clarity: the paper is well-written in general. I failed to decode the meaning behind the paragraph under Figure 3 on page 4 and would advise the authors to re-write it. The same goes to the first paragraph on page 3.
Originality: the paper builds on the prior work of Kaiser et al., 2017 and Santoro et al., 2016, but the proposed modifications are novel to my best knowledge.
Significance: below average: the paper combines interesting ideas that potentially can be used in different learning contexts and with other algorithms, however, the evaluation does not show the benefit in an obvious way.
Other comments:
* throughout the whole paper it is not clear if the embeddings are learned or not. I suppose they are, but what then happens to the ones in memory? If they are not, like in ImageNet example, where do they come from?
* the hyperparameter \sigma: the authors claim "the value of \sigma seems to not matter too much". Matter for what? It's great if the performance is stable for a wide range of \sigma, but it seems like it should have a great influence over the memory footprint of APL. I feel this is an important point that needs more attention.
* it would be interesting to see how APL performs with a simple majority vote instead the decoder layer. This would count for an ablation study and could emphasize the role of the decoder.
* Figure 4, b) plots are completely unreadable on black-and-white print, the authors might like to address that
* In conclusion, the first claim about state-of-the-art accuracy with smaller memory footprint: I don't think that the results of the paper justify this claim.
[1] Yoon et al, Lifelong Learning with Dynamically Expandable Networks, ICLR 2017
[2] Rebuffi et al,  iCaRL: Incremental Classifier and Representation Learning, CVPR 2017
********************
After authors response:
Thanks to the authors for a detailed response. The introduction led me to believe that the paper solves a different task from what it actually does. I still like the algorithm and, given that the scope of the paper is limited to a few-shot learning, I tend to change my evaluation and recommend to accept the paper. It was a good idea to change the title to avoid possible confusion by other readers. The introduction is still misleading though. It creates the impression that APL solves a more general problem where it would be good enough to limit the discussion to a few-shot learning setting and explain it in greater detail for an unfamiliar reader. Some details also seem to be missing, e.g. I didn't get that the memory is flushed after each episode and could not find where this is mentioned in the paper.


Deep Reinforcement Learning has managed to achieve state-of-the-art results in learning control policies directly from raw pixels. However, despite its remarkable success, it fails to generalize, a fundamental component required in a stable Artificial Intelligence system. Using the Atari game Breakout, we demonstrate the difficulty of a trained agent in adjusting to simple modifications in the raw image, ones that a human could adapt to trivially. In transfer learning, the goal is to use the knowledge gained from the source task to make the training of the target task faster and better. We show that using various forms of fine-tuning, a common method for transfer learning, is not effective for adapting to such small visual changes. In fact, it is often easier to re-train the agent from scratch than to fine-tune a trained agent. We suggest that in some cases transfer learning can be improved by adding a dedicated component whose goal is to learn to visually map between the known domain and the new one. Concretely, we use Unaligned Generative Adversarial Networks (GANs) to create a mapping function to translate images in the target task to corresponding images in the source task. These mapping functions allow us to transform between various variations of the Breakout game, as well as between different levels of a Nintendo game, Road Fighter. We show that learning this mapping is substantially more efficient than re-training. A visualization of a trained agent playing Breakout and Road Fighter, with and without the GAN transfer, can be seen in

# Summary
This paper proposes to improve the sample efficiency of transfer learning for Deep RL by mapping a new visual domain (target) onto the training one (source) using GANs. First, a deep RL policy is trained on a source domain (e.g., level 1 of the Atari Road Fighter game). Second, a GAN (e.g. UNIT or CycleGAN) is trained for unsupervised domain adaptation from target images (e.g., level 2 of Road Fighter) to source ones. Third, the policy learned in the source domain is applied directly on the GAN-translated target domain. The experimental evaluation uses two Atari games: i) transfer from Breakout to Breakout with static visual distractors inpainted on the screen, ii) from one Road Fighter level to others. Results suggest that this transfer learning approach requires less images than retraining from scratch in the new domain, including when fine-tuning does not work.
# Strengths
Controlled toy experiments of Deep RL generalization issues:
The experiments on Breakout quantify how badly A3C overfits in this case, as it shows catastrophic performance degradation even with trivial static visual input perturbations (which are not even adversarial attacks). The fine-tuning experiments also quantify well how brittle the initial policy is, motivating further the importance of the problem studied by the paper.
Investigating the impact of different GANs on the end task:
The experiments evaluate two different image translation algorithms: one based on UNIT, the other based on CycleGAN. The results suggest that this choice is key and depends on the target domain. This suggests that the adaptation is in fact task dependent, confirming the direction pursued by others in task-specific unsupervised domain adaptation (cf. below).
# Weaknesses
Discrepancy between quantitative and qualitative results:
The good quantitative results (accumulated rewards) reported in the experiments are not reflected in the qualitative results. As can be seen from the videos, these results seem more to be representative of a bias in the data. For instance, in the Road Fighter videos, one can clearly see that the geometry of the road (width, curves) and dynamic obstacles are almost completely erased in the image translation process. The main reasons the quantitative results are good seem to be i) in the non-translated case the agent crashes immediately, ii) the "translated" image is a wide straight road identical to level 1 where the policy just keeps the car in the middle (thus crashing as soon as there is a turn or a collision with an obstacle). Even in the breakout case, there are catastrophic translation failures for some of the studied variations although the domain gap is static and small. The image translation results look underwhelming compared to state of the art GANs used for much more complex tasks and environments (e.g., the original CycleGAN paper and follow-up works, or the ICLR'18 progressive growing of GANs paper). This might be due to a hyper-parameter tuning issue, but it is unclear why the adaptation results seem not on par with previous results although the paper is in a visually simpler domain (Atari games).
Does not address the RL generalization issues:
Although it is the main goal of the paper, the method is fundamentally side-stepping the problem as it does not improve in any way the policy or the Deep RL algorithm (they are left untouched). It is mapping the target environment to the source one, without consideration for the end task besides tuning GAN hyper-parameters. If the initial policy is very brittle (as convincingly shown in section 2), then just mapping to the source domain does not improve the generalization capabilities of the Deep RL algorithm, or even improves transfer learning: it just enables the policy to be used in other contexts that can be reduced to the training one (which is independent of the learning algorithm, RL or otherwise). So it is unclear whether the main contribution is the one claimed. The contribution seems instead an experimental observation that it might be easier to reduce related domains to the training one instead of retraining a new (specialised and brittle) policy. Existing works have actually gone further, learning jointly the image translation and task network, including for very challenging problems, e.g. in unsupervised sim-to-real visual domain adaptation (e.g., Unsupervised Pixel-Level Domain Adaptation with Generative Adversarial Networks from Bousmalis et al at CVPR'17, which is not cited here).
Experimental protocol:
The experimental conclusions are not clear and lack generality, because the optimal methods (e.g., choice of GAN, number of iterations) vary significantly depending on the task (cf. Table 3 for instance). Furthermore, the best configurations seem selected on the test set for every experiment.
Data efficiency vs actual training efficiency:
The main claim is that it is better to do image translation instead of fine-tuning or full re-training. The basis of that argument is the experimentally observed need for less frames to do the image translation (Table 2). However, it is not clear that training GANs for unsupervised image translation is actually any easier / faster. What about training instability, mode collapse, hyper-parameter tuning, and actual training time comparisons on the same hardware?
# First Recommendation
Using image translation via GANs for unsupervised domain adaptation is a popular idea, used in the context of RL for Atari games here. Although the experiments show that mapping a target visual domain to a source one can enable reusing a deep RL policy as is, the qualitative results suggest this is in fact due to a bias in the data used here and the experimental protocol does not yield general insights. Furthermore, this approach is not specific to RL and its observed generalization issues. It does not improve the learning of the policy or improve its transferability, thus having only limited new insights compared to existing approaches that jointly learn image translation and target task-specific networks in much more challenging conditions.
I believe this submission is at the start of an interesting direction, and requires further work on more challenging tasks, bigger domain gaps, and towards more joint training or actual policy transfer to go beyond this first set of encouraging but preliminary results.
# Post-rebuttal Recommendation
Thanks to the authors for their detailed reply. The clarifications around overfitting, UNIT-GAN in Section 4, and the paper claims are helpful. I  also agree that the quantitative experiments are serious. I have bumped my score by +1 as a result.
Nonetheless, the results still seem preliminary and limited in scope for the aforementioned reasons. The discussion in the comments about the learned policies and transfer are ad-hoc. A lot of the shortcomings mentioned in the review are outright dismissed (e.g., "de facto standard in RL"), downplayed (esp. generalization, which is puzzling for a transfer learning paper), or left for future work.
As there is no strong technical contribution beyond the experimental observations in the current submission, I suggest the authors try to address the GAN shortcomings both mentioned in reviews and their reply, instead of  just observing  / reporting them. As this paper's main focus is to use image translation in the proposed RL setting (with standard GAN and RL methods), I do not think it is just someone else's problem to improve the image translation part. Proposing a technical contribution there would make the paper much stronger and appealing to a broader ICLR audience.  This might also require adding a third game to ensure more generalizable experimental insights.


We show that finite-width deep ReLU neural networks yield rate-distortion optimal approximation (Bölcskei et al., 2018) of a wide class of functions, including polynomials, windowed sinusoidal functions, one-dimensional oscillatory textures, and the Weierstrass function, a fractal function which is continuous but nowhere differentiable. Together with the recently established universal approximation result for affine function systems (Bölcskei et al., 2018), this demonstrates that deep neural networks approximate vastly different signal structures generated by the affine group, the Weyl-Heisenberg group, or through warping, and even certain fractals, all with approximation error decaying exponentially in the number of neurons. We also prove that in the approximation of sufficiently smooth functions finite-width deep networks require strictly fewer neurons than finite-depth wide networks.

The focus of this paper is to show that finite-width deep neural networks with fully connected layers and ReLU activations are rate-distortion optimal approximators of certain classes of functions, meaning the approximation error decays exponentially in the number of neurons in the network. The function classes explored in this paper are: 1-d polynomials (on bounded intervals), 1-d sinusoidal functions (on bounded intervals), and other 1-d functions built from compositions or linear combinations of these, such as the so-called class of “oscillatory textures” and a class of continuous but nowhere differentiable functions known as Weierstrass functions. Finally, the paper also shows that as the desired approximation accuracy goes to zero finite-width deep ReLU networks require asymptotically fewer neurons than finite-depth wide ReLU networks in approximating a broad class of smooth functions.
The paper is well-written and the technical results are presented in a way that is easy to understand. The results are somewhat novel, although they do build off other recent works, namely Yarotsky (2016) and Telgarsky (2015). However, the authors were careful to cite when they reuse proof techniques from these and other works. The results in the main text appear to be technically sound. I did not check carefully all the proofs in the supplemental materials.
My major criticism is that the focus on certain specific function classes (oscillatory textures, Weierstrauss functions) seems arbitrary, and leaves open many questions. For example, there is existing work on the approximation ability of deep ReLU networks for functions in more general Holder and Sobolev spaces:
Hadrien Montanelli and Qiang Du. Deep ReLU networks lessen the curse of dimensionality. arXiv preprint arXiv:1712.08688, 2017.
J. Schmidt-Hieber. Nonparametric regression using deep neural networks with ReLU activation function. ArXiv e-prints, August 2017.
I was left wondering how the present results relate to these works, and what insight we get from understanding these particular function classes that we don't get from understanding Holder or Sobolev spaces.
Major comments
In Section 3, I found the progression of the results from approximation of x^2, to multiplication xy, and to general smooth functions to be very natural and well-motivated. However, sections 4 and 5 seem lack somewhat in motivation, since here the authors focus on very specific function classes (sinusoidal functions, oscillatory textures, and Weierstrass functions). While these results are still interesting, focusing on such specific functions is less satisfactory, since it raises questions about the true scope of the results (e.g., will similar approximation rates extend to other fractal functions, or just Weierstrauss functions?). Could the authors give further justification for why these function classes are interesting to focus on, or why they limit themselves in this way? Can the authors also put these results more into context with existing results on the approximation with ReLU networks?
The authors state multiple times that “all our results apply to the multivariate case” but that they restrict themselves to the univariate case for simplicity of presentation. While this is fine, some indication of how the results are altered in the multivariate case would be useful. For example, does the fixed-width M in multivariate generalizations of Prop 3.1--3.3 need to be bigger, smaller, or the same? What other constants are dimensionally dependent? Do the multivariate generalization of their results bear the "curse of dimensionality", i.e., does the number of neurons needed to reach epsilon accuracy depend geometrically on the dimension?
Minor comments
A conclusion or discussion section summarizing the overall technical contribution would be useful for the reader. Also, it would be useful to include some discussion on remaining open problems or future work.
On pg. 2, the authors state “the approximation results throughout the paper guarantee that the magnitude of the weights in the network does not grow faster than polynomially in the cardinality of of the domain over which the approximation takes place”. What does “cardinality of the domain” here mean? I think the authors mean the size D of the interval [-D,D] over which the approximation is valid.
On pg. 7, the authors say “We note that this result allows to show that local cosine bases (cite) can be approximated by deep ReLU networks with exponential error decay…”. I think the authors mean to say “...this result allows us to show…” or “this result allows one to show…”. Although it’s not clear to me whether this means it has been shown (it’s a direct corollary), or could possible be shown (it’s a corollary, but needs some non-trivial work). Also, one line to specify what a “local cosine basis” is would be helpful.


UNSUPERVISED DISCOVERY OF PARTS, STRUCTURE, AND DYNAMICS
Humans easily recognize object parts and their hierarchical structure by watching how they move; they can then predict how each part moves in the future. In this paper, we propose a novel formulation that simultaneously learns a hierarchical, disentangled object representation and a dynamics model for object parts from unlabeled videos. Our Parts, Structure, and Dynamics (PSD) model learns to, first, recognize the object parts via a layered image representation; second, predict hierarchy via a structural descriptor that composes low-level concepts into a hierarchical structure; and third, model the system dynamics by predicting the future. Experiments on multiple real and synthetic datasets demonstrate that our PSD model works well on all three tasks: segmenting object parts, building their hierarchical structure, and capturing their motion distributions.

==== Review Summary ====
The paper demonstrates an interesting and potentially useful idea.  But much of it is poorely explained, and experimental results are not strongly convincing.  The only numerical evaluations are on a simple dataset that the authors made themselves.  The most interesting claim - that this network can learn unsupervised hierarchical object segmentation based on unlabelled video data -  is not well supported by the paper.
==== Paper Summary ====
This paper presents a deep neural network which learns object Segmentation, Structure, and Dynamics from unlabelled video.  The idea is quite useful, as it is a step towards learning models that can "discover" the concept of objects in a visual scene without any supervision.
The central contributions of this paper are:
(1) To show how one can use coherent motion (the fact that different parts of an object move together) to learn unsupervised object segmentation.
(2) To show how once can learn the relation between objects (e.g. "arm" is part of "body") by observing relative motion between segments.
==== General Feedback ====
The paper would benefit a lot from better explanations and being better tied together (see "Details" below for examples).   Figure captions would benefit from much better explanations and integration with the text - each figure caption should at least describe what the figure is intended to demonstrate.  Variables such as ($\mathcal M$, $\mathcal S$, $\mathcal I$, $\mathcal L$) should be indicated in figures .
Many things would benefit from being defined precisely with Equations.  For example I have no idea how the "soft" structural descriptor S is computed.  Is it (A) a parameter that is shared across data points and learned?  or (B) is it computed per-frame from the network?  And after it is calculated, how are the S_{ik} values (which fall between 0 and 1) used to combine the motion maps?
==== Scientific Questions ===
I'm confused as to what the latent variables z "mean".  It seems strange that there is a 1-d latent variable representing the motion of each part.  Translation of a segment within an image is 2D.  3D if you include planar rotation, then there's scaling motion and out-of-plane rotations, so it seems an odd design choice that motion should be squeezed into a 1D representation.
I find it difficult to assess the quality of the "next frame predictions".  There's lots other literature on next-frame prediction to compare against (e.g. https://arxiv.org/abs/1605.08104).  At least you could compare to a naive baseline of simply shifting pixels based on the optical flow.
I'm confused about how you are learning the "estimated flow".  My impression is that the input flow is calculated between the last 2 frames $\hat M = flow(I_{t-1}, I_t)$.  And that the "estimated" flow is an estimate of $flow(I_{t}, I_{t+1})$.  But in Section 4.3 you seem to indicate that the "estimated" flow is just trained to "reconstruct" the input flow.... In that case why not just feed the input flow directly into the Image Decoder?  What I guess you're doing is trying to Predict the next flow ($flow(I_{t}, I_{t+1})$) but if you're doing this neither Figure 3 nor Section 4.2 indicates this, hence my confusion.
==== Details ====
Figure 3:
----
The "shapes" example is odd, because it's not obvious that there's a structural hierarchy linking the shapes.  Maybe a "torso/left arm/right arm" would be better for illustrative purposes?
It would be helpful to put the variable names ($\mathcal M_k$, etc) on the figure.
Should add a second arrow coming into the (f) the Structural descriptor from a leaf-variable $p_k$
Also it would be helpful to indicate where the losses are evaluated.
"Next Frame" should probably be "Reconstruction" (though "Prediction" might be a more accurate word).
---
Section 4.2:
Notational point, it seems k can be from 1 to d.  But in Section 3 you say it can be from 1 to "n". Maybe it would be clearer to change both ("n" and "d") to "K" to emphasize that "k" is an index which goes up to "K".  (edit... now after reading 5.1: Latent Representation, I understand.  If there are n parts in the dataset you use d>n dimensions and the network learns to "drop" the extra ones... it would help to clarify that here).
Structural Descriptor: You say "in practice, we relax the binary constraints"... Yet this is important.. should write down the equation and say how the "soft" version of [i \in S] calculated.
Section 4.3
"elbo" seems like the wrong name for this loss, as it's not an Evidence Lower BOund.  The elbo would be the sum of this loss and the first component of L_{recon}.  It is a regularizing term, so you could call it L_reg.
It's not obvious that sparse local motion maps mean a heirarchical tree structure, but I see how it could help.  I suggest that without evidence for this loss you soften the claim to "this is intended to help encourage the model to learn a heirarchical tree structure"
Figure 4:
It appears that each row indicates a different video in the dataset, but then in 4f you still have two rows but they appear to correspond to different algorithms... a vertical separator here might help show that the rows in 4f do not correspond to the rows in 4a-e.
"Next Frame" here appears to refer to ground truth, but in Figure 3 "Next Frame" appears to be the prediction (which you call reconstruction).
Section 5.1
Future Prediction: No explanation either here or in Figure 4 of what it actually shows.  (What would a failure look like?)
Hierarchical structure... You binarize... how?
Figure 9:
What does this show?  The figure does not obviously demonstrate anything.  Maybe compare to ground-truth future frames?
Section 5.3:
Future Prediction: These images are from the test set, right?  If so, that is worth mentioning.
Object Segmentation ("in several different dimensions"  -> "corresponding to the active latent dimensions?")
Object Segmentation: Visually, it looks nice, but I have now idea how good this segmentation is.  You compare verbally to R-NEM and PSD, but there're no numbers.
Human Studies... The line "For those whose predicted tree structures are not consistent with ours, they all agree with our results and believe ours are more reasonable than others" .. brings to mind an image of a room full of tortured experimental subjects not being allowed to leave until they sign a confession that their own tree structures were foolish mistakes and your tree structures are far superior.... So probably it should be removed because it sounds shady.


Motivated by applications to unsupervised learning, we consider the problem of measuring mutual information. Recent analysis has shown that naive kNN estimators of mutual information have serious statistical limitations motivating more refined methods. In this paper we prove that serious statistical limitations are inherent to any measurement method. More specifically, we show that any distributionfree high-confidence lower bound on mutual information cannot be larger than O(ln N ) where N is the size of the data sample. We also analyze the DonskerVaradhan lower bound on KL divergence in particular and show that, when simple statistical considerations are taken into account, this bound can never produce a high-confidence value larger than ln N . While large high-confidence lower bounds are impossible, in practice one can use estimators without formal guarantees. We suggest expressing mutual information as a difference of entropies and using cross entropy as an entropy estimator. We observe that, although cross entropy is only an upper bound on entropy, cross-entropy estimates converge to the true cross entropy at the rate of 1/ √ N .

This paper studied the Donsker-Varadhan lower bound of KL-divergence. The authors show that with high probability, the DV lower bound is upper bounded by log of the sample size, so if the true KL-divergence is very large, then exponential sample size is needed to make the DV lower bound tight. The same argument holds true for any distribution-free high-confidence lower bound (such as DV lower bound) for KL divergence. Then the authors proposed to use an upper bound for entropy instead of lower bound for mutual information.
The idea of the paper is interesting and the proof of Theorems 1 and 2 are valid. Especially I like the idea of Theorem 1, which proves that any distribution-free high-confidence lower bound for KL divergence is upper bounded by log of sample size. This idea is similar to the paper in (Gao et al 15') which shows that mutual information estimator is upper bounded by log(N).
However, this paper contains many fatal flaws, which significantly weaken the quality of this paper. Precisely,
1. The DV lower bound is just an alternative of mutual information, helping MMI predictive coding algorithm to find good coding functions C_x and C_y. The goal of MMI predictive coding is not to estimate the mutual information I(C_x(x), C_y(y)) precisely, instead, the goal is to find good coding functions. The fact that DV lower bound is small means that we can not estimate mutual information through DV lower bound, but it does not directly imply that we can not find the coding functions. I expect some experiments to show that when mutual information is large, MMI predictive coding using DV lower bound can not find good coding functions.
2. In Section 3, the formula after the proof of Outlier Risk Lemma and before Theorem 1 (btw, it is better to have numbers for these formula) seems to be problematic. The first formula shows that E_{z~q} e^{F(z)} >= (1/N)e^{F_max}, then we plug it in (4). But in (4) there is negative ln of E_{z~q} e^{F(z)}, so we should have KL(P,Q) <= something, correct? This may be a typo but this typo is so important such that it affect the readability a lot. Theorem 1 is correct but the paragraphs before Theorem 1 confuse the reader a lot.
3. In Section 4, are you considering classical entropy for discrete random variables, or differential entropy for continuous random variables? I assume you are considering the latter, since most people of machine learning community are interested in continuous random variables. Then your statement of I(X;Y) <= H(X) is incorrect, since for continuous random variables, H(X|Y) can be negative. See (Thomas & Cover, Chapter 8) for a reference.
4. Related to problem 3, if you are considering continuous random variables, then the statement I(X;Y)=H(X)+H(Y)-H(X,Y) is not always correct. There are cases that H(X) is infinite, H(Y) is infinite, H(X,Y) is infinite but I(X;Y) is finite. These cases does not only exist in mathematical books, but also exists in practice, especially when the data is located on a low-dimensional manifold embedded in a high-dimensional space. Therefore, your approach of decompose the mutual information is not always possible.
5. Regarding to your proposed optimization problem in Section 6 (also it is better to have a number), I have some concerns. Since it involves max over \Psi outside, and inf over \Theta and inf over \Phi inside, so I wonder how do you solve this problem? Can you guarantee that the solution can provide good coding functions C_x and C_y? Also, it seems that this optimization problem is proposed as an improvement over the DV lower bound method, so I wish to see some experiment showing that this method is better than the DV lower bound method, at least for some synthetic datasets.
Because of the above mentioned flaws (especially 3 and 4, and lack of experiments), I think the paper is below the standard of ICLR conference.
References:
[1] Efficient Estimation of Mutual Information for Strongly Dependent Variables, by Gao, Ver Steeg and Galstyan, AISTATS15'
[2] Elements of Information Theory, 2nd edition, by Thomas and Cover.


We propose a novel unsupervised classification method based on graph Laplacian. Unlike the widely used classification method, this architecture does not require the labels of data and the number of classes. Our key idea is to introduce a approximate linear map and a spectral clustering theory on the dimension reduced spaces into generative adversarial networks. Inspired by the human visual recognition system, the proposed framework can classify and also generate images as the human brains do. We build an approximate linear connector network C analogous to the cerebral cortex, between the discriminator D and the generator G. The connector network allows us to estimate the unknown number of classes. Estimating the number of classes is one of the challenging researches in the unsupervised learning, especially in spectral clustering. The proposed method can also classify the images by using the estimated number of classes. Therefore, we define our method as an unsupervised classification method.

This paper develops an unsupervised classification algorithm using the idea of CycleGAN. Specifically, it constructs a piece-wise linear mapping (the Connector Network) between the discriminator network and the generator network. The learning objective is based on the cycle-consistency loss. Experiments show that it can achieve reasonable loss. This paper addresses an important problem, namely, unsupervised image classification, and may present interesting ideas. However, the paper is not in a good shape for publication in its current form.
First, the paper is not well written and many of the key ideas are not clear. It devotes more than half of the pages to the review of the preliminary materials in Sections 2-3 while only briefly explained the main algorithm in Section 4. Many of the details are missing. For example, why L1-loss is used in (5)-(7) in Algorithm 1? What is the “random-walk Laplacian matrix L_{sym}” (never defined)? More importantly, it seems that Section 3.4 is a key section to explain how to perform unsupervised classification. However, the ideas (regarding the footprints and footprint mask etc.) are totally unclear. It is assumed that all different classes have equal probabilities. In this setting, it is unclear (in its intuition) why it is possible to assign a cluster index to its true class labels. What is the key signal in the data that enables the algorithm to relate different clusters to their corresponding classes, especially when all classes have equal probability? Furthermore, it is not clear why the mapping from H to Z can be written as a sum of C_1,…,C_k in Proposition 3.2. If the final mapping is piece-wise linear, how can it be written as a sum of linear mappings? Similar question arises in the first paragraph of Section 4.1: if the connector network is constructed as a piecewise linear function (as stated earlier in the paper in abstract and introduction), then how can it be written as a matrix? (Only linear mapping can be expressed as a matrix.)
Second, the experiment is insufficient. None of the experiment details are presented in the paper. Only the final accuracy of 0.874 is given without any further details. What is the model architecture and size? More experimental analysis should be presented. For example, there are many different hyperparameters in the algorithms. How are the \lambda_D, \lambda_G chosen when there is no labeled validation set? How sensitive is the algorithm to different model architecture and model size? Furthermore, none of the baselines results are presented and compared against.
A lot of related works are missing. There have been a lot of emerging works related to unsupervised classification recently, which should be discussed and compared:
[1] G. Lample, L. Denoyer, and M. Ranzato.  Unsupervised machine translation using monolingual corpora only. ICLR, 2018.
[2] M. Artetxe, G. Labaka, E. Agirre, and K. Cho.  Unsupervised neural machine translation. ICLR, 2018.
[3] Y. Liu, J. Chen, and L. Deng.  Unsupervised sequence classification using sequential output statistics. NIPS, 2017
[4] A. Gupta, A. Vedaldi, A. Zisserman. Learning to Read by Spelling: Towards Unsupervised Text Recognition. arXiv:1809.08675, 2018.
[5] G. Lample, M. Ott, A. Conneau, L. Denoyer, M. Ranzato. Phrase-based & neural unsupervised machine translation. EMNLP 2018.
The presentation of the paper should be significantly improved as it is currently hard to read due to many grammar and English usage issues as well as other unclear statements. Just to name a few examples below:
-	(1st paragraph of Introduction): “…imagine the learned objectconstruct…”
-	The last paragraph in Section 1 is not in the right position and should be placed somewhere else in the introduction.
-	In the first paragraph of Section 2.2, “one of the clustering algorithm” should be “one of the clustering algorithms”.
-	In the first paragraph of Section 3, it is not clear what it means by “we can make the tuples (Z,X,H) for the whole dataset”.
-	At the end of the first paragraph of Section 3, there is a missing reference in “network in section()”.
-	In the third paragraph on page 4, there is a grammar issue in “H and Z have greater than convexity than X…” and in “it allows the linear combination on these two manifolds in the feature spaces H and Z are and”.
-	In the first paragraph of Section 3.2, it is not clear what it means by “two feature spaces will be trained by the cycle consistency loss to obtain the tuples (Z,X,H) with the correct permutation, where all of elements is in the same-class manifold and shares same learned features.”
-	In the first paragraph of page 5, “cycle-consistency loss z C(D(G(z))) and backward cycle consistency loss x G(C(D(x)))” does not read well. It sounds like z is the loss C(D(G(z))) and x is the loss G(C(D(x)))?
-	Typo in Figure 4: “a shows” should be “(a) shows”.


Many works have been proposed in the literature to capture the dynamics of diffusion in networks. While some of them define graphical markovian models to extract temporal relationships between node infections in networks, others consider diffusion episodes as sequences of infections via recurrent neural models. In this paper we propose a model at the crossroads of these two extremes, which embeds the history of diffusion in infected nodes as hidden continuous states. Depending on the trajectory followed by the content before reaching a given node, the distribution of influence probabilities may vary. However, content trajectories are usually hidden in the data, which induces challenging learning problems. We propose a topological recurrent neural model which exhibits good experimental performances for diffusion modelling and prediction.

The authors of this paper are proposing a neural network approach for learning diffusion dynamics in networks. The authors argue that the main advantage of their framework is that it incorporates the structure of independent cascades into the model which predicts the diffusion process.
The primary difficulty in reviewing this paper is the poor presentation of the paper. There are many typos and mistakes (e.g., the last paragraph of the paper starts with a sentence that does not make any sense), missing references (e.g., there is an empty parenthesis at the end of the second paragraph on the second page) and in at least two cases, there are references to a formula that is not in the manuscript (e.g., reference to formula 15 on line 3 of page 5). This issues makes reviewing this paper very difficult.
In the modeling section, authors use p(I|D) as q^D(.) in the Eq. 12, where p(I|D) is the conditional probability that a particular node infected an observed infected node first. Plugging p(I|D) in Eq. 12 and using decomposition of p(D ,I) used in Eq. 10, we arrive at a formulation which drops all p(I|D) terms. This results in an objective function which only involves infected nodes (and no term associated with the parent node), weighted by likelihood of each node j infecting the node at step i. This should make the training more simplified than what is discussed in Algorithm 2. Beyond this simplification, I am not clear if that is actually intended by the authors.
The experiments demonstrate a superior performance of the proposed model compared to alternative benchmarks. The authors explain how they trained their own model but there is no mention on how they trained benchmark models. However, given that the datasets used in the experiments were not used in the associated benchmark papers, it is necessary for authors to explain how they trained competing models.
Due to several shortcomings of the paper, most important of which is on presentation of the paper, this manuscript requires a significant revision by the authors to reach the necessary standards for publication, moreover it would be helpful to clarify the modeling choices and consequences of these choices more clearly.


The large-scale surveillance video analysis becomes important as the development of intelligent city. The heavy computation resources neccessary for state-of-theart deep learning model makes the real-time processing hard to be implemented. This paper exploits the characteristic of high scene similarity generally existing in surveillance videos and proposes dynamic convolution reusing the previous feature map to reduce the computation amount. We tested the proposed method on 45 surveillance videos with various scenes. The experimental results show that dynamic convolution can reduce up to 75.7% of FLOPs while preserving the precision within 0.7% mAP. Furthermore, the dynamic convolution can enhance the processing time up to 2.2 times.

The paper addresses the problem of computational inefficiency in video surveillance understanding approaches. It suggests an approach called Dynamic Convolution consists of Frame differencing, Prediction, and Dyn-Convolution steps. The idea is to reuse some of the convolutional feature maps, and frame features particularly when there is a significant similarity among the frames. The paper evaluates the results on 4 public datasets. However, it just compares the approach to a baseline, which is indeed applying convnet on all frames.
- State of the art is not well-studied in the paper. Video understanding approaches usually are not just applying convnet on all frames. Many of the approaches on video analysis, select a random set of frames (or just a single frame) [5], and extract the features for them. There is another set of work on attention, that try to extracts the most important spatio-temporal [1-4] information to solve a certain task. These approaches are usually computationally less expensive than applying convnet on all video frames. I suggest the authors compare their model with these approaches.

[1] Spatially Adaptive Computation Time for Residual Networks., Figurnov et al.

[2] Recurrent Models of Visual Attention, Mnih et al.

[3] Action recognition using visual attention, Sharma et al.

[4] End-to-end learning of action detection from frame glimpses in videos, Yeung et al.

[5] Two-Stream Convolutional Networks for Action Recognition in Videos, Simonyan et al.

- In addition, car and pedestrian detection performance is part of the evaluation process. In this case, the approach should be also compared to the state-of-the-art tracking approaches (that are cheaper to acquire) in terms of computational efficiency and performance.
- The writing of the paper should also improve to make the paper more understandable and easier to follow.
Some examples:
1. Unnecessary information can be summarized. For example, many details on the computational costs in abstract and the introduction can just simply be replaced by stating that “these approaches are computationally costly”.
2. Using present tense for the SoTA approaches is more common.“ShuffleNet (Zhang et al. (2017)) proposed two new strategies”.
3. Long sentences are difficult to follow: “In real surveillance video application, although the calculation reduction on convolution is the main concern of speeding up the overall processing time, the data transfer is another important factor which contributes to the time”

+ The problem of large-scale video understanding is an important and interesting problem to tackle.


Deep neural networks (DNNs) typically have enough capacity to fit random data by brute force even when conventional data-dependent regularizations focusing on the geometry of the features are imposed. We find out that the reason for this is the inconsistency between the enforced geometry and the standard softmax cross entropy loss. To resolve this, we propose a new framework for data-dependent DNN regularization, the Geometrically-Regularized-Self-Validating neural Networks (GRSVNet). During training, the geometry enforced on one batch of features is simultaneously validated on a separate batch using a validation loss consistent with the geometry. We study a particular case of GRSVNet, the Orthogonal-Low-rank Embedding (OLE)-GRSVNet, which is capable of producing highly discriminative features residing in orthogonal low-rank subspaces. Numerical experiments show that OLE-GRSVNet outperforms DNNs with conventional regularization when trained on real data. More importantly, unlike conventional DNNs, OLE-GRSVNet refuses to memorize random data or random labels, suggesting it only learns intrinsic patterns by reducing the memorizing capacity of the baseline DNN.

The paper proposes a data-dependent regularization method which is coupled with softmax loss to train deep neural networks for classification. The paper turns to Orthogonal Low-rank Embedding (OLE) loss for the geometric constraint that one class of data/feature are assumed to reside on a low-rank subspace that subspaces of different classes are orthogonal ideally. The probability in the softmax is then modeled as cosine similarity between data feature and the class-specific subspaces. In this way, geometric loss and softmax loss have the common goal for optimization. Moreover, during training, the geometry enforced on one batch of features is simultaneously validated on a separate batch using a validation loss. The experiments seem to suggest such a model helps avoid overfitting/memorizing noisy training data. The paper reads well and is easy to follow.
However, the paper is limited in technical novelty and practical significance. Here are some concerns --
1) The paper only studies one method based on OLE, though it cites the center loss [19]. How does the center loss behave in face of noisy training label? Would it also be able to refuse to fit the noisy training data?
2) As each class has its own (low-rank) subspace, and the rank is reduced by imposing the nuclear norm. It seems that the proposed method is hard to extend to many classes (class number is larger than the dimension)?
3) The datasets in the experiments are quite small in scale and class number. It is not persuasive unless tested on larger scale data or with large class number.
4) The proposed method seems to be limited in deal with discrete labels (e.g., classification), is it easy to extend to continuous target, say regression problems like depth estimation and surface normal estimation?
5) While the authors claim as a main contribution that the proposed GRSVNet is a general framework, it is hard to see how this framework can be used in other tasks other than classification.
6) The experiments are less persuasive. It's better to add the error bar to see the improvement by the proposed method is not due to random initialization. Running time should also be compared, as nuclear norm seems to be time consuming.


G-SGD: OPTIMIZING RELU NEURAL NETWORKS IN ITS POSITIVELY SCALE-INVARIANT SPACE
It is well known that neural networks with rectified linear units (ReLU) activation functions are positively scale-invariant. Conventional algorithms like stochastic gradient descent optimize the neural networks in the vector space of weights, which is, however, not positively scale-invariant. This mismatch may lead to problems during the optimization process. Then, a natural question is: can we construct a new vector space that is positively scale-invariant and sufficient to represent ReLU neural networks so as to better facilitate the optimization process ? In this paper, we provide our positive answer to this question. First, we conduct a formal study on the positive scaling operators which forms a transformation group, denoted as G. We show that the value of a path (i.e. the product of the weights along the path) in the neural network is invariant to positive scaling and prove that the value vector of all the paths is sufficient to represent the neural networks under mild conditions. Second, we show that one can identify some basis paths out of all the paths and prove that the linear span of their value vectors (denoted as G-space) is an invariant space with lower dimension under the positive scaling group. Finally, we design stochastic gradient descent algorithm in G-space (abbreviated as G-SGD) to optimize the value vector of the basis paths of neural networks with little extra cost by leveraging back-propagation. Our experiments show that G-SGD significantly outperforms the conventional SGD algorithm in optimizing ReLU networks on benchmark datasets.

Summary:
In prior deep learning papers it has been observed that ReLU networks are positively scale invariant due to the non-negative homogeneity property of the max(0, x) function. The present paper proposes to change the optimization procedure so that it is done in a space where this invariance is preserved (in contrast to the weight space, where it is not). To do so, the authors define the group G of positive scaling operators, and note that the "value of path" (product of weights along the path) is G-invariant and together with "activation status of paths" allows for the definition of an equivalence class. They then build a G-space which has fewer dimensions than the weight space, and proposed g-SGD to optimize the network in this space. In g-SGD gradients are computed normally, then projected to G-space via a sparse matrix in order to update the values of paths. The weights are then updated based on a "weight allocation method" that involves the inverse projection.
The authors conduct experiments on CIFAR-10 and -100 with a ResNet-34 and a similarly deep variant of VGG, showing significant benefits from G-SGD training in all cases. They also evaluate the performance of a simple MLP on Fashion-MNIST as a function of the invariant ratio H/m.
Comments:
The paper is organized well (with technical details of the proofs delegated to the appendix), and discusses the differences in comparison prior work. While evaluations on large-scale datasets would be helpful here, the present experiments suggest that optimization in G-space indeed consistently improves results, so the proposed method seems promising.
For completeness, it would be great to include Path-SGD results for the CIFAR experiments in Table 1, together with runtime information to highlight the benefits of the g-SGD algorithm and provide experimental proof that the computational overhead is indeed low.
If the authors are hoping for a wider adoption of the method, it would be helpful for the community to have the g-SGD code released within one of the standard deep learning frameworks.
Questions:
- Does it matter which weights are chosen as "free skeleton weights"? If these weights never get updated in the optimization procedure, could you please comment on the intuitive interpretation of the necessity of their presence?
- The text states that the computational overhead of the gradient of path norm is "very high". The Path-SGD paper proposes a method costing (B + 1)T, where the overhead an be small for large batches. It would be good to clarify this a bit in the present text.
- Is the advantage of g-SGD over SGD expected to be proportional to the invariant ratio for CNNs as well?


Revealing latent structure in data is an active field of research, having introduced exciting technologies such as variational autoencoders and adversarial networks, and is essential to push machine learning towards unsupervised knowledge discovery. However, a major challenge is the lack of suitable benchmarks for an objective and quantitative evaluation of learned representations. To address this issue we introduce Morpho-MNIST, a framework that aims to answer: "to what extent has my model learned to represent specific factors of variation in the data?" We extend the popular MNIST dataset by adding a morphometric analysis enabling quantitative comparison of trained models, identification of the roles of latent variables, and characterisation of sample diversity. We further propose a set of quantifiable perturbations to assess the performance of unsupervised and supervised methods on challenging tasks such as outlier detection and domain adaptation.

This paper discusses the problem of evaluating and diagnosing the representations learnt using a generative model. This is a very important and necessary problem.
However, this paper lacks in terms of experimental evaluation and has some technical flaws.
1. Morphological properties deals with only the "shape" properties of the image object. However, when the entire image is subject to the generative model, it learns multiple properties from the image apart from shape too - such as texture and color. Additionally, there are lot of low level pixel relations that the model learns to fit the distribution of the given images. However, here the authors have assumed that the latent space of the generative models are influenced only by the morphological properties of the image - which is wrong. Latent space features could be affected by the color or texture of the image as well.
2. Extracting morphological properties of the image is straight-foward for MNIST kind of objects. However, it becomes really difficult for other datasets such as CIFAR or some real world images. Studying the properties of a generative model on such datasets is very challenging and the authors have not added a discussion around that.
3. Now assuming that my GAN model has learnt good representation in Morpho-MNIST dataset, is it guaranteed to learn good representations in other datasets as well? There is no guarantee on generalizability or extensibility of the work.


SNIP: SINGLE-SHOT NETWORK PRUNING BASED ON CONNECTION SENSITIVITY
Pruning large neural networks while maintaining their performance is often desirable due to the reduced space and time complexity. In existing methods, pruning is done within an iterative optimization procedure with either heuristically designed pruning schedules or additional hyperparameters, undermining their utility. In this work, we present a new approach that prunes a given network once at initialization prior to training. To achieve this, we introduce a saliency criterion based on connection sensitivity that identifies structurally important connections in the network for the given task. This eliminates the need for both pretraining and the complex pruning schedule while making it robust to architecture variations. After pruning, the sparse network is trained in the standard way. Our method obtains extremely sparse networks with virtually the same accuracy as the reference network on the MNIST, CIFAR-10, and Tiny-ImageNet classification tasks and is broadly applicable to various architectures including convolutional, residual and recurrent networks. Unlike existing methods, our approach enables us to demonstrate that the retained connections are indeed relevant to the given task.

Post rebuttal update/comment:
I thank the authors for the revision and have updated the score (twice!)
One genuinely perplexing result to me is that the method behaves better than random pruning, yet after selecting the salient neurons the weights can be reinitialized, as per the rebuttal:
> # Initialization procedure
- It is correct that the weights used to train the pruned model are possibly different from the ones used to compute the connection sensitivity. Given (variance scaled) initial weights, SNIP finds the architecturally important parameters in the network, then the pruned network is established and trained in the standard way.
First, there is work which states quite the opposite (e.g. https://arxiv.org/abs/1803.03635). Please relate to it.
Fundamentally, if you decouple weight pruning from initialization it also means that:
- the first layer will be pruned out of connections to constant pixels (which is seen in the visualizations), this remains meaningful even after a reinitialization
- the second and higher layers will be pruned somewhat randomly - even if the connections pruned were meaningful with the original weights, after the reinitialization the functions computed by the neurons in lower layers will be different, and have no relation to pruned weights. Thus the pruning will be essentially random (though possibly from a very specific random distribution). In other words - then neurons in a fully connected layer can be freely swapped, each neuron in the next layer behaves on al of them anyway we are thinking here about the uninitialized neurons, with each of them having a distribution over weights and not a particular set of sampled weights, this is valid because we will reinitialize the neurons). Because of that, I wouldn't call any particular weight/connection architecturally important and find it strange that such weights are found.
I find this behavior really perplexing, but I trust that your experiments are correct. however, please, if you have the time, verify it.
Original review:
The paper presents an intriguing result in which a salient, small subset of weights can be selected even in untrained networks given sensible initialization defaults are used. This result is surprising - the usual network pruning procedure assumed that a network is pretrained, and only then important connections are removed.
The contributions of the paper are two-fold:
1) it reintroduces a multiplicative sensitivity measure similar to the Breiman garotte
2) and shows which other design choices are needed to make it work on untrained networks, which is surprising.
While the main idea of the paper is clear and easy to intuitively understand, the details are not. My main concern is that paper differentiates between weights and connections (both terms are introduced on page iv to differentiate from earlier work). However, it is not clear what are the authors referring to:
- a conv layer has many repeated applications of the same weight. Am I correct to assume that a conv layer has many more connections, than weights? Furthermore, are the dramatic sparsities demonstrated over connections counted in this manner? This is important - on MNIST each digit has a constant zero border, all connections to the border are not needed and can be trivially removed (one can crop the images to remove them for similar results). Thus we can trivially remove connections, without removing weights.
- in paragraph 5.5 different weight initialization schemes are used for the purpose of saliency estimation, but the paragraph then says "Note that for training VS-X initialization is used in all the cases." Does it mean that first a set of random weights is sampled, then the sensitivities are computed, then a salient set of connections is established and the weights are REINITIALIZED from a distribution possibly different than the one used to compute the sensitivity? The fact that it works is very surprising and again suggests that the method identifies constant background pixels rather than important weights.
- on the other hand, if there is a one-to-one correspondence between connections and weights, then the differentiation from Karnin (1990) at the bottom of p. iv is misleading.
I would also be cautious about extrapolating results from MNIST to other vision datasets. MNIST has dark backgrounds. Let f(w,c) = 0*w*c. Trivially, df/dw = df/dc = 0. Thus the proposed sensitivity measure picks non-background pixels, which is also demonstrated in figure 2. However, this is a property of the dataset (which encodes background with 0) and not of the method! This should be further investigated - a quick check is to invert MNIST (make the images black-on-white, not white-on-black) and see if the method still works. Fashion MNIST behaves in a similar way. Thus the only non-trvial experiments are the ones on CIFAR10 (Table 2), but the majority of the analysis is conducted on white-on-black MNIST and Fashion-MNIST.
Finally, no experiment shows the benefit of introducing the variables "c", rather than using the gradient with respect to the weights. let f be the function computed by the network. Then:
- df/d(cw) is the gradient passed to the weights if the "c" variables were not introduced
- df/dw = df/d(cw) d(cw)/dw = df/d(cw) * c = df/d(cw)
- df/dc = df/d(cw) d(cw)/dc = df/d(cw) * w
Thus the proposed change seems to favor a combination of weight magnitude and the regular df/dw magnitude. I'd like to see how using the regular df/dw criterion would fare in single-shot pruning. In particular, I expect using the plain gradient to lead to similar selections to those in Figure 2, because for constant  pixels 0 = df/d(cw) = df/dc = df/dw.
Suggested corrections:
In related work (sec. 2) it is pointed that Hessian-based methods are unpractical due to the size od the Hessian. In fact OBD uses a diagonal approximation to the hessian, which is computed with complexity similar to the gradient, although it is typically not supported by deep learning toolkits. Please correct.
The description of weight initialization schemes should also be corrected (sec. 4.2). The sentence "Note that initializing neural networks is a random process, typically done using normal distribution with zero mean and a fixed variance." is wrong and artificially inflates the paper's contribution.  Variance normalizing schemes had been known since the nineties (see efficient backprop) and are the default in many toolkits, e.g. Pytorch uses the Kaiming rule which sets the standard deviation according to the fan-in: https://github.com/pytorch/pytorch/blob/master/torch/nn/modules/linear.py#L56.
Please enumerate the datasets (MNIST, Fashion-MNIST, CIFAR10) in the abstract, rather than saying "vision datasets", because MNIST in particular is not representative of vision datasets due to the constant zero padding, as explained before.
Missing references:
- Efficient Backprop http://yann.lecun.com/exdb/publis/pdf/lecun-98b.pdf discusses variance scaling initialization, and approximations to the hessian. Since both are mentioned in the text this should be cited as well.
- the Breiman non-negative garotte (https://www.jstor.org/stable/1269730) is a similar well-known technique in statistics
Finally, I liked the paper and wanted to give it a higher score, but reduced it because of the occurrence of many broad claims made in the paper, such as: 1) method works on MNIST => abstract claims it generally works on vision datasets 2) paper states "typically used is fixed variance init", but the popular toolkits (pytorch, keras) actually use the variance scaling one by default 3) the badly explained distinction between connection and weight and the relation that it implies to prior work. I will revise the score if these claims are corrected.