Task-Driven Convolutional Recurrent Models of the Visual System
Feed-forward convolutional neural networks (CNNs) are currently state-of-the-art for object classification tasks such as ImageNet. Further, they are quantitatively accurate models of temporally-averaged responses of neurons in the primate brain's visual system. However, biological visual systems have two ubiquitous architectural features not shared with typical CNNs: local recurrence within cortical areas, and long-range feedback from downstream areas to upstream areas. Here we explored the role of recurrence in improving classification performance. We found that standard forms of recurrence (vanilla RNNs and LSTMs) do not perform well within deep CNNs on the ImageNet task. In contrast, novel cells that incorporated two structural features, bypassing and gating, were able to boost task accuracy substantially. We extended these design principles in an automated search over thousands of model architectures, which identified novel local recurrent cells and long-range feedback connections useful for object recognition. Moreover, these task-optimized ConvRNNs matched the dynamics of neural activity in the primate visual system better than feedforward networks, suggesting a role for the brain's recurrent connections in performing difficult visual behaviors.

This paper introduces and evaluates neuroscience-inspired refinements of classic deep feedforward convolutional neural networks for object classification. Specifically, the authors experiment with new types of processing units which implement mechanisms of recurrence and top-down feedback within the network -- processes thought to be important functional mechanisms in biological vision. The paper is well-conceived and well-argued, and the model development and evaluation is extremely comprehensive and technically accomplished. Importantly, the authors demonstrate how their recurrent models achieve comparable performance to much deeper feedforward networks, while at the same time providing a better fit to neurophysiological data. Moreover, this paper is also an excellent demonstration of how processing principles inspired by neurocognitive theory can be investigated using machine learning and neural network modelling, leading to sophisticated, quantitative evaluation of hypotheses about the functional organisation of the brain (following, e.g., Cox, Seidenberg, & Rogers, 2015; Kriegeskorte, 2015).  Detailed comments  A clear strength of this paper is its comprehensive modelling rigor. For example, I particularly liked the use of carefully matched control networks (lines 119-128). Another example is the attempt to control for the trainability of different architectures through comprehensive hyperparameter tuning (lines 159-171). It is also great that the code for the models is being publically released.  Reciprocal Gated Cells (and their parameterised generalization), introduced in this paper, would seem to have the potential to make an important fundamental contribution to computer vision engineering (in addition to yielding more neurocognitively plausible models). I think the authors could expand a little on the design choices leading to the Reciprocal Gated Cell. For example, couldn’t the standard LSTM cell be adapted to include bypassing? Is there an intuitive reason why the mutual, symmetric, gating by the h and c layers of each others’ inputs can be regarded, a priori, as a desirable property?  My main critical comment is with the ethological relevance of an object labelling task (this is an issue for other similar work as well, not just this paper). As the authors note, primate vision has the goal of transforming the visual input into representations that “readily guide behaviour”. Is object labelling really the most suitable machine learning task for approximating successful behaviour? Why should accurate labelling of objects be the objective function of a model of biological visual systems? This seems particularly problematic for non-human primates, who do not use labels for objects at all, but nevertheless activate adaptation-optimised, behaviour-relevant high-level representations of objects. When a monkey sees either a lion or a leopard, presumably they activate a high-level representation we might gloss as “it’s dangerous; it’s fast; I need to get away right now”. This would be very different to the representation activated for, say, a banana. However, a labelling task will lead the network to learn representations that are best for visually discriminating between all objects (e.g. LION, LEOPARD, BANANA), not representations that reflect meaningful, behaviour-relevant similarities and differences between objects (representations in which LION and LEOPARD are presumably more similar to each other than to BANANA). It seems to me that training neural networks to decode categorical similarity or property knowledge rather than object labels (Devereux, Clarke, & Tyler, 2018; Jozwik, Kriegeskorte, Storrs, & Mur, 2017; Rabovsky & McRae, 2014; Rogers et al., 2004) may reflect more suitable objectives for networks intended to model meaningful object processing. Classic connectionist literature on concept learning (e.g. McClelland and others) may also be relevant here. However, I consider this criticism to be relatively minor, given the other significant contributions of the paper.  Given the stated motivation to model biological visual systems, I would have liked to have seen the models evaluated against a wider range of neuroscience data, including fMRI (perhaps using Representational Similarity Analysis (Kriegeskorte, Goebel, & Bandettini, 2006), following, for example, the analyses of (Khaligh-Razavi & Kriegeskorte, 2014)). Comparison to EEG, ECoG, or MEG data would also have been interesting (e.g. Cichy, Khosla, Pantazis, Torralba, & Oliva, 2016), given the modelling focus on plausible dynamics (lines 101-105).   Is it particularly surprising that vanilla RNNs and LSTMs do not increase performance on object labelling, since these architectures are designed for sequence modelling (which is not relevant to the object labelling task)? I would like to see a slightly more detailed discussion and rationale for why the authors considered these worth testing in the first place (in terms of how they might relate to known functional properties of the primate visual system). Perhaps some of the ideas in lines 143-158 belong earlier, in the introduction and methods sections.  Final comments  From a computational cognitive neuroscience perspective, it is very exciting that the recurrent models presented in this paper do as well as neuro-cognitively implausible ultra-deep feedforward models. This paper is a rarity in that it simultaneously makes fundamental contributions to both computer vision engineering and high-level computational neuroscience, and I therefore heartily recommend its acceptance.   References   Cichy, R. M., Khosla, A., Pantazis, D., Torralba, A., & Oliva, A. (2016). Comparison of deep neural networks to spatio-temporal cortical dynamics of human visual object recognition reveals hierarchical correspondence. Scientific Reports, 6, 27755. https://doi.org/10.1038/srep27755  Cox, C. R., Seidenberg, M. S., & Rogers, T. T. (2015). Connecting functional brain imaging and Parallel Distributed Processing. Language, Cognition and Neuroscience, 30(4), 380–394. https://doi.org/10.1080/23273798.2014.994010  Devereux, B. J., Clarke, A., & Tyler, L. K. (2018). Integrated deep visual and semantic attractor neural networks predict fMRI pattern-information along the ventral object processing pathway. Scientific Reports, 8(1), 10636. https://doi.org/10.1038/s41598-018-28865-1  Jozwik, K. M., Kriegeskorte, N., Storrs, K. R., & Mur, M. (2017). Deep Convolutional Neural Networks Outperform Feature-Based But Not Categorical Models in Explaining Object Similarity Judgments. Frontiers in Psychology, 8. https://doi.org/10.3389/fpsyg.2017.01726  Khaligh-Razavi, S.-M., & Kriegeskorte, N. (2014). Deep Supervised, but Not Unsupervised, Models May Explain IT Cortical Representation. PLoS Comput Biol, 10(11), e1003915. https://doi.org/10.1371/journal.pcbi.1003915  Kriegeskorte, N. (2015). Deep Neural Networks: A New Framework for Modeling Biological Vision and Brain Information Processing. Annual Review of Vision Science, 1(1), 417–446. https://doi.org/10.1146/annurev-vision-082114-035447  Kriegeskorte, N., Goebel, R., & Bandettini, P. (2006). Information-based functional brain mapping. Proceedings of the National Academy of Sciences of the United States of America, 103(10), 3863–3868. https://doi.org/10.1073/pnas.0600244103  Rabovsky, M., & McRae, K. (2014). Simulating the N400 ERP component as semantic network error: Insights from a feature-based connectionist attractor model of word meaning. Cognition, 132(1), 68–89. https://doi.org/10.1016/j.cognition.2014.03.010  Rogers, T. T., Lambon Ralph, M. A., Garrard, P., Bozeat, S., McClelland, J. L., Hodges, J. R., & Patterson, K. (2004). Structure and Deterioration of Semantic Memory: A Neuropsychological and Computational Investigation. Psychological Review, 111(1), 205–235. https://doi.org/10.1037/0033-295X.111.1.205


Scalable Methods for 8-bit Training of Neural Networks
Quantized Neural Networks (QNNs) are often used to improve network efficiency during the inference phase, i.e. after the network has been trained. Extensive research in the field suggests many different quantization schemes. Still, the number of bits required, as well as the best quantization scheme, are yet unknown. Our theoretical analysis suggests that most of the training process is robust to substantial precision reduction, and points to only a few specific operations that require higher precision. Armed with this knowledge, we quantize the model parameters, activations and layer gradients to 8-bit, leaving at a higher precision only the final step in the computation of the weight gradients. Additionally, as QNNs require batch-normalization to be trained at high precision, we introduce Range BatchNormalization (BN) which has significantly higher tolerance to quantization noise and improved computational complexity. Our simulations show that Range BN is equivalent to the traditional batch norm if a precise scale adjustment, which can be approximated analytically, is applied. To the best of the authors' knowledge, this work is the first to quantize the weights, activations, as well as a substantial volume of the gradients stream, in all layers (including batch normalization) to 8-bit while showing state-of-the-art results over the ImageNet-1K dataset.

At the hightest level, the manuscript contains three contributions: 1. Range BN. This is a clever way to make BN more low precision friendly. The authors motivate it thoroughly and back it up with convincing experiments. A good contribution 2. A largely heuristically derived training scheme that uses 8-bit weights and activations, a low (8-bit) and high (16-bit) copy of the deltas and some other tricks such as stochastic rounding. They show ImageNet ResNet18 to 65% accuracy matching full precision, which is a solid experimental validation of the method. 3. A lengthy section that tries to predict if a certain quantization scheme  will be successful based on the cosine similarity between the original and the quantized weights.   Overall, it appears to be a well-written, though out paper. Using INT8 for training is very novel and useful.   For me the weak part of the paper is clearly Section 5. The hypothesis that angles are predictive of performance seems quite disconnected from the rest of the paper. I don't see evidence in the rest of the paper that this analysis provides anything predictive that could guide an experimenter. At the same time, it's not a theoretical result that is impactful on it's own.   It seems like there might be possible ways the authors could have tried to make more of this analysis, for example injecting angle noise into a network and measuring how it affects performance. But there is neither intuition nor data what a certain angle means. What does it tell us if the angle for binary is 37°, which is almost right in the middle between no noise and orthogonal 90°? It's obvious that some error (in angle or by any other measure) will degrade performance, but to make this useful, there needs to be a functional relationship between the two. The relationship in Fig 2a) is tenuous at best, with a gradual fall-off in cosine and a sharp knee in accuracy.   The derivation of the 2^M >> sqrt(ln(N)) condition is nicely done and promising, but the authors do not show that  it's useful, which they could do e.g. by predicting the required number of bits for various operations and then showing convergence at this number of bits. Instead, they just observer that using 8 bits leaves a huge amount  of headroom, rather than probing how tight this error bound is.   My second, minor concern is that clarity should be improved. It's not exactly clear to me what data formats are used for which computations. It's laudable that the authors provide code to check, but this should be clear from the paper. Is the 8-bit format an integer format or fixed point, and how are exponents / scaling factors determined? What's the 16-bit format used for bifurcated gradients, is it fp16 or int16? Am I understanding correctly that weights and activations are in 8-bit and the gradients have 8- and 16-bit copies, or is there also a higher precision copy of the weights (common in many other low precision schemes)? What exactly do you mean by fp32 updates, does this imply there is a 32 bit copy of the weights, and if yes, where does stochastic rounding come in? This is partially addressed in the last paragraph, but should come earlier, and more clearly. Consider explaining it in form of a table, or better yet a diagram showing the various tensors that go into computing and updating a layer, and what type they are in.   Misc comments: - The section around lines 79 is incorrectly claiming that BatchNorm has not successfully been applied in 16 bit. Köster et al. 2017 (Flexpoint: An Adaptive Numerical Format for Efficient Training of Deep Neural Networks) have shown that a ResNet-110 with BN can be trained completely in 16 bit, with no high precision variance parameters.  - The statement in line 80 is not clear, is 42% supposed to be the overall SotA or for alexnet? It doesn't seem to be the right number for either - Can you elaborate on the serial dependence in line 140? Why does this matter? Does this assume specialized hardware where serial dependencies are more costly than raw FLOPS? - Figure 1: Caption should mention that this figure is for the ternary case - Figure 2a: Needs axis labels. bits on the horizontal and angle / accuracy on the vertical? - Fig2 caption: type, "with respect" should be "with respect to". Middle histogram in log scale -> all 3 are in log scale.  - line 238, 10.5% for Cifar10 ResNet50. This model should train to about 93% accuracy according to He 2016. Please provide details on exact model used and why it's so much worse.   - 245 typo "precising"  In conclusion, the paper is solidly borderline, by which I mean that a rebuttal that addresses these concerns would make me reconsider the rating.      -----------   Response to author rebuttal:  In response to the rebuttal and discussion with other reviewers I'd like to update my assessment, the paper should be accepted. The new experimental results make the paper much stronger, and the authors were able to clear up a lot of misunderstandings, which I hope will make it through to the final version of the paper (e.g. a figure showing the elements of a layer, indicating the data type for each operation)


Convex Elicitation of Continuous Real Properties
A property or statistic of a distribution is said to be elicitable if it can be expressed as the minimizer of some loss function in expectation. Recent work shows that continuous real-valued properties are elicitable if and only if they are identifiable, meaning the set of distributions with the same property value can be described by linear constraints. From a practical standpoint, one may ask for which such properties do there exist convex loss functions. In this paper, in a finite-outcome setting, we show that in fact essentially every elicitable real-valued property can be elicited by a convex loss function. Our proof is constructive, and leads to convex loss functions for new properties.

Summary:  This paper studies elicitation which is defined as the property or statistic of a distribution which can be described as the minimizer of some loss function in expectation. More specifically, the focus of this paper is in studying which properties can be explained as the minimizer of convex loss functions. The authors prove that in the finite outcome setting, under some assumptions, every elicitable real-valued property can also be elicited by a convex loss function. The authors start the paper by providing an overview of the prior literature and go over their assumptions. Then, they state the theorem and give some intuition on how the proof goes through. Finally, they conclude the paper by providing two examples and applying their constructive technique to yield corresponding convex losses.  Evaluation:  The paper is nicely written and well-motivated. As authors also mention, in many practical applications, dealing with convex losses is easier than general loss functions as various techniques can be applied for finding their minimum.   Authors have addressed the limitation of their method in their discussion, however I really think that the generalization to infinite outcome space is very important. In other words, I believe that this limitation is very restrictive and needs to be better addressed. For this more general case, authors could have either provided some proofs (positive results) with maybe some additional assumptions, or they could maybe provide some counter-examples (negative results).   Furthermore, even in the case of finite outcome setting, the second example in section 5 shows that even the constructive way of their theorem (or more precisely, the constructed weight function \lambda) produces a very complex \lambda. So other than a more simpler setting of Example 1, the proof is not practically constructive. I understand that the authors were able to address this concern for Example 2 (which I think it is nice), but still this construction might be far from practical purposes in general.  Based on these points, I think the paper is in a very good shape in terms of clarity, but can be further improved in technical and also practical aspects.  AFTER REBUTTAL:  The authors have addressed all my concerns and I am willing to increase my score from 6 to 8. I believe the idea of convex elicitation is very important and this paper makes a huge contribution to the field. Hopefully, the result can be extended to the infinite outcome space in the near future.


Snap ML: A Hierarchical Framework for Machine Learning
We describe a new software framework for fast training of generalized linear models. The framework, named Snap Machine Learning (Snap ML), combines recent advances in machine learning systems and algorithms in a nested manner to reflect the hierarchical architecture of modern computing systems. We prove theoretically that such a hierarchical system can accelerate training in distributed environments where intra-node communication is cheaper than inter-node communication. Additionally, we provide a review of the implementation of Snap ML in terms of GPU acceleration, pipelining, communication patterns and software architecture, highlighting aspects that were critical for achieving high performance. We evaluate the performance of Snap ML in both single-node and multi-node environments, quantifying the benefit of the hierarchical scheme and the data streaming functionality, and comparing with other widely-used machine learning software frameworks. Finally, we present a logistic regression benchmark on the Criteo Terabyte Click Logs dataset and show that Snap ML achieves the same test loss an order of magnitude faster than any of the previously reported results, including those obtained using TensorFlow and scikit-learn.

The work at hand describes a new distributed implementation (Snap ML) for training generalized linear models for very large datasets. More precisely, the authors extend the popular CoCoA method [15] by providing a hierarchical version that is optimized for distributed computing environments. One of the key ingredients of this new version is the reduction of the overhead caused by the induced inter-node communication (see Section 2.1). In addition to a theoretical analysis of their new approach (Equation (2)), the authors also provide various implementation details including details related to an efficient local GPU solver per compute node (Section 3.1), to buffering techniques to reduce the overhead caused by memory transfers between host and device per compute node (Section 3.2), to an efficient exchange of information between compute nodes (Section 3.3), and to the overall software architecture. The experimental evaluation indicates some benefits of the new hierarchical scheme (Figure 5). It also demonstrates the benefits of the new framework for the Terabyte Click Logs.  The paper is well written and technically sound. I also like the theoretical analysis (Appendix B) of the inter-node communication pattern that is induced by parallelyzing the work per compute node (i.e., by having two loops of parallelism). It think the paper definitely meets the very high NIPS standards. My two main concerns are (1) lack of novelty and (2) the experimental evaluation:  (1) The framework extends the CoCoA framework. While being technically and theoretically sound, I find the practical merits of the approach quite limited. In particular, I am not blown away by the performance gains obtained via the new hierarchical framework. For instance, even given a slow network, I do not find the runtime reduction significant (e.g., 87 seconds instead for t_2=1 compared to 67 seconds for t_2 \approx 10, see Figure 5). I have the feeling that a similar performance gain could be achieved by compressing a shared vector v prior to sending it over the network.  (2) The approach is clearly of practical importance. However, I was not fully convinced by the experimental evaluation. For instance, in Figure 4, I find the performance comparison between Snap ML, Scikit-Learn, and TensorFlow not entirely fair: Scikit-Learn only makes use of a CPU instead of a V100 GPU. Further, TensorFlow reads the data in batches from disk (as stated by the authors). Thus, Snap ML is faster than Scikit-Learn simply due to the GPU and faster than TensorFlow due to the slow transfer between disk and memory device. Can one improve the performance of TensorFlow by increasing the batch size or by buffering/caching? Also, while I find the results in Figure 7 impressive, I am missing one important baseline: Snap ML with t_2=1, that is, a Spark+GPU version of the previous CoCoA framework (without the additional hierarchical additions). What is the performance of this variant? Do the modifications proposed in the work yield significant better results compared to a rather direct distributed application of the existing framework (i.e., a direct Spark/GPU implementation of the CoCoA implementation)?  Based on the authors' comments, I have decided to increase my score a bit. Overall, it is a decent contribution. I nevertheless think that the experimental evaluation could be improved.  Minor comments: - Figure 5: hierarchical


Greedy Hash: Towards Fast Optimization for Accurate Hash Coding in CNN
To convert the input into binary code, hashing algorithm has been widely used for approximate nearest neighbor search on large-scale image sets due to its computation and storage efficiency. Deep hashing further improves the retrieval quality by combining the hash coding with deep neural network. However, a major difficulty in deep hashing lies in the discrete constraints imposed on the network output, which generally makes the optimization NP hard. In this work, we adopt the greedy principle to tackle this NP hard problem by iteratively updating the network toward the probable optimal discrete solution in each iteration. A hash coding layer is designed to implement our approach which strictly uses the sign function in forward propagation to maintain the discrete constraints, while in back propagation the gradients are transmitted intactly to the front layer to avoid the vanishing gradients. In addition to the theoretical derivation, we provide a new perspective to visualize and understand the effectiveness and efficiency of our algorithm. Experiments on benchmark datasets show that our scheme outperforms state-of-the-art hashing methods in both supervised and unsupervised tasks.

The paper presents a greedy approach to train a deep neural network to directly produce binary codes that build on the straight through estimator.  During forward propagation the model uses the sgn output whereas at the back-propagation stage it passes derivatives as it the output were a simple linear function.  There are relevant papers that already proposed such an approach and that are not referred to as earlier work: [1] Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation, Bengio Y. et al. [2] Techniques for learning binary stochastic feedforward neural networks, Tapani R. et al.  The experimental setting is not very clear and I would suggest the authors to better explain  the supervised setting. Do they produce a binary code of length k and then classify it  with a single final output layer? If so then I would like to see an ablation study on this. What is the relative contribution of using this loss vs the contrastive or triplet loss that is  used in many of the other approaches?   What is the influence of the pre-trained network? What would be the performance if the method were to be trained from scratch end-to-end? This would allow for a better assessment of the greedy relaxation proposed here proposed for binarizing the output layer.  Regarding other works that have tackled the problem of hashing with neural networks I see missing: [3] Multimodal similarity-preserving hashing, Masci J. et al. [4] Semantic Hashing, Salakhutdinov R. et al  The comparison in 3.5 claims a performance margin that is larger at short code lengths wrt other methods,  however I see only one of the algorithms in the figure. Please report more.  Overall an interesting application paper that is clearly written, but that should include further references and  readjust the contribution section considering that the straight through is not novel.   ============================================================== After having read the rebuttal I decided to increase my score as the authors addressed all my concerns.


Toddler-Inspired Visual Object Learning
Real-world learning systems have practical limitations on the quality and quantity of the training datasets that they can collect and consider. How should a system go about choosing a subset of the possible training examples that still allows for learning accurate, generalizable models? To help address this question, we draw inspiration from a highly efficient practical learning system: the human child. Using head-mounted cameras, eye gaze trackers, and a model of foveated vision, we collected first-person (egocentric) images that represent a highly accurate approximation of the "training data" that toddlers' visual systems collect in everyday, naturalistic learning contexts. We used state-of-the-art computer vision learning models (convolutional neural networks) to help characterize the structure of these data, and found that child data produce significantly better object models than egocentric data experienced by adults in exactly the same environment. By using the CNNs as a modeling tool to investigate the properties of the child data that may enable this rapid learning, we found that child data exhibit a unique combination of quality and diversity, with not only many similar large, high-quality object views but also a greater number and diversity of rare views. This novel methodology of analyzing the visual "training data" used by children may not only reveal insights to improve machine learning, but also may suggest new experimental tools to better understand infant learning in developmental psychology.

This paper explores the effect of data distributions on the performance of object classification systems, by specifically looking at data produced by toddlers. Specifically, the paper hypothesizes that the views of objects gathered by head-mounted cameras on toddlers have a combination of (1) high-quality, close-up views and (2) a large number and diversity of rare views (in contrast to views produced by adults in the same setting). The paper suggests that such high-quality datasets (such as those produced by infant wearing a head-mounted camera) may help with the problem of data efficiency.  Overall, I think this is an excellent paper and I think it would be great to have it presented at NIPS. It’s generally a good idea to look at how much different data distributions affect the performance of our algorithms, and it seems like an important piece of the puzzle to understand and quantify the data distributions actually perceived by human learners in comparison to our machine learniners. The choice of examining toddler data in particular seems quite unique and I think is very interesting. In general the experiments are quite nice, though the paper could be strengthened by the addition of a few more (described below).  Quality ---------  Generally the quality of the paper is quite high, but I think a few more experiments could be performed and a few more details given regarding the datasets and results.  I would appreciate it if the training accuracy could be reported as well as the test accuracy. In particular, it would be interesting to know whether the adult data was also more difficult to train on than the child data, or whether the difference only appeared at test time. If the latter is true, that would indeed be a very interesting finding.  I would like to see the analyses in Figure 3 performed for at least one standard computer vision dataset, like ImageNet, in order to better understand whether it is closer to the parent distribution or the child distribution. The paper suggests it would be closer to the parent distribution since the photos are taken by adults, but this is an empirical question.  An additional experiment would be to control for object size and test for the difference in diversity, by cropping the images so that the object takes approximately the same amount of space in the image in both the child and adult datasets. Then, any difference in performance on the test set would be due to diversity in view, rather than size. If there isn’t a difference in performance, then it suggests the main factor between adults and children is size, rather than diversity in viewpoint. This seems like it would be an important variable to isolate.  I think there is one issue in the experiments regarding object size, which has to do with the size of the objects in the test set. Specifically, if the objects in the test set are large, then it could be that by increasing the object size (decreasing the FOV around the gaze center) the training set is being brought closer to the test distribution. In that case, the argument becomes a bit weaker, in that it is more about the child data being closer to the test distribution than the training data. I think the paper should report the histogram of object sizes in the test set and analyze this more closely to see if this is indeed what is happening.  The similar/diverse analysis in Section 4.3 felt a bit weaker to me, because it seems to me that the important distinction isn’t just “similar images” vs. “diverse images” but “similar *canonical* images” vs. “diverse images”. However, Figure 6b suggests that the “similar” images aren’t necessarily canonical views. It’s not at all surprising to me that low diversity of non-canonical views leads to worse performance. A far more interesting comparison (with non-obvious results, at least to me) would be to see if low diversity of canonical views also leads to worse performance. A suggestion for how to achieve this would be to choose a small set of images which the authors feel are canonical views as a seed for their procedure for choosing the similar/diverse images, so that the similar images are all clustered around the canonical views instead.  Some more minor points:  - I would like to know what the overall accuracy of YOLO was, for all frames within a look. “At least one” doesn’t convey much, when the look durations were on average around 50 frames. - Thank you for reporting averages over 10 runs of the network, and error bars too! - Figure 6a might look a bit nicer if a different embedding was used, like t-SNE. Also in Figure 6a, it took me a bit to realize the black points were all the images---I would just clarify this in the caption. - The choice of adding in the blurring according to foveation is interesting, but also makes it a bit harder to compare this dataset to standard image datasets which do not have this information. Since the paper argues it’s not about improving performance so much as exploring the choice of data, this seems like an odd decision to me. Can the authors justify why they included the acuity simulation further, and perhaps also report results without that (maybe in an appendix)?  Clarity --------  The paper is very well written and easy to read and understand. I only had a few points of confusion:  In the second paragraph of the introduction, the paper states that “more quantity naturally leads to better overall quality”. This confused me until I read the rest of the paragraph, and I think the reason is that the paper uses a different definition of “high-quality” than is typically used in the machine learning literature. Usually, “high-quality” in ML means that there is very little noise/error in the labels, that objects are in a canonical view, etc. After reading the rest of the paragraph I understand what the paper means, but I would suggest this paragraph be rephrased just to avoid confusion over the definition of this term in the first place.  Line 78: “a few recent studies have applied deep learning models to data collected by humans” Most real-world machine learning datasets were collected by humans, so I think it would be more accurate to say “nearly all deep learning methods are applied to data collected by humans”. But given the citations, I think this sentence was perhaps meant to convey something different, and more about the intersection of psychological research/data with machine learning?  Originality -------------  While egocentric datasets have been previously explored in computer vision (e.g. [1, 2]), I do not believe toddler data has been examined in the same way, nor have there been comparisons between adult and toddler data in the context of machine learning performance (of course this has been extensively explored in the psychological literature). While the paper does not introduce a new model, it does perform a nice set of experiments that better characterize the dimensions that we care about in high-quality data for computer vision.  Significance ----------------  Overall, I think this paper could be quite useful to the machine learning and computer vision communities for thinking about the distributions of our datasets from a psychological perspective.  One question that I have regarding significance is whether the authors plan to open source the dataset? I think it would be a valuable contribution to the computer vision community and would increase the significance of the paper to release the dataset. However, I realize this might not be possible due to privacy concerns (but perhaps the dataset could be anonymized by blurring out faces, in that case?)  [1] http://vision.cs.utexas.edu/projects/egocentric_data/UT_Egocentric_Dataset.html [2] http://imagelab.ing.unimore.it/WESAX/related_datasets.asp  Edit after author response: I have read the author response and am quite satisfied with it; I am particularly happy that they have (1) included the results without the acuity blur, (2) included more information about the distribution of ImageNet images, and (3) included information about the size of objects in the test distribution (and I find it very interesting that the results are as good as they are when the size of the objects in the test distribution is 0.75, compared with the size of 0.129 in the children's distribution). I think the community at NIPS would find this paper and its results very interesting and have thus decided to increase my score by one.


Long short-term memory and learning-to-learn in networks of spiking neurons
Recurrent networks of spiking neurons (RSNNs) underlie the astounding computing and learning capabilities of the brain. But computing and learning capabilities of RSNN models have remained poor, at least in comparison with artificial neural networks (ANNs). We address two possible reasons for that. One is that RSNNs in the brain are not randomly connected or designed according to simple rules, and they do not start learning as a tabula rasa network. Rather, RSNNs in the brain were optimized for their tasks through evolution, development, and prior experience. Details of these optimization processes are largely unknown. But their functional contribution can be approximated through powerful optimization methods, such as backpropagation through time (BPTT). A second major mismatch between RSNNs in the brain and models is that the latter only show a small fraction of the dynamics of neurons and synapses in the brain. We include neurons in our RSNN model that reproduce one prominent dynamical process of biological neurons that takes place at the behaviourally relevant time scale of seconds: neuronal adaptation. We denote these networks as LSNNs because of their Long short-term memory. The inclusion of adapting neurons drastically increases the computing and learning capability of RSNNs if they are trained and configured by deep learning (BPTT combined with a rewiring algorithm that optimizes the network architecture). In fact, the computational performance of these RSNNs approaches for the first time that of LSTM networks. In addition RSNNs with adapting neurons can acquire abstract knowledge from prior learning in a Learning-to-Learn (L2L) scheme, and transfer that knowledge in order to learn new but related tasks from very few examples. We demonstrate this for supervised learning and reinforcement learning.

This paper aims to combine deep learning techniques (backpropagation through time and learning to learning) with recurrent networks of spiking neurons, which are more biologically realistic but typically have worse performance and learning capabilities than deep neural networks. They propose a modified version of recurrent spiking neural networks (RSNNs) that includes neural adaptation and BPTT (modified to work with spiking networks), which they term long short-term memory spiking neural networks (LSNNs). They test on two standard benchmark SL tasks: sequential MNIST and TIMIT, and find performance comparable to that of an LSTM. They also test the learning-to-learn capabilities of LSNNs by looking at two task domains: 1) function learning, where the function to be learned is the class of all continuous functions of two variables that can be computed by a 2-layer ANN, as well as simple sine functions, and 2) meta-reinforcement learning in a navigation task, where the goal is placed randomly every episode.  In all, I really enjoyed this paper. It addresses an important disconnect between our understanding of how the brain works, and state of the art deep neural network architectures, which presumably are implementing at least some similar cognitive functions such as visual recognition or sequence processing. The tasks they selected are appropriate, ranging from standard benchmarking tasks in SL to toy tasks which allow for more detailed analysis, and finally a larger scale RL task.   I mainly have a few concerns with regard to clarity and novelty, although I suspect these could be readily addressed in revision.   1. The paper currently feels as if it’s almost 2 separate papers put together, one about LSNNs and one about learning to learn. Are these two contributions completely orthogonal, or is there some interaction? How would LSNNs perform on for example omniglot classification or another benchmark task in which meta-learning is required (perhaps the one used in Santoro et al, 2016 on MANNs)?  2. With regard to clarity, I got confused while reading section 3, because I had expected these to be trained in the “meta-learning” sense, as the previous paragraph (end of section 2) discussed learning-to-learn at length. Going back to my point 1, it’d make much more sense of these tasks were also “meta-learning” tasks. 3. What is the relative importance of the adaptive threshold vs the modified BPTT procedure (with DEEP R) to the performance of the model?  4. It’s important to spell out what are the new contributions made in this paper vs Bellec et al, 2018, which is cited for DEEP R. Without reading this other paper, I can’t tell what is the proposed contribution (was it applied to only FF networks in the other paper?) 5. I’m curious about the disadvantages/advantages of LSNNs vs traditional DNNs. It seems like LSNNs might deal well with sequential data, but what about data without sequential structure, like images? Convnets can capitalize on spatial properties like translational invariance, etc, but would LSNNs always struggle with these? LSTMs are definitely not the first model you’d reach for when classifying MNIST.  Update after reading the authors' rebuttal: While I still think that the paper is pretty strong, I'm pretty disappointed in the response, which never stated what the authors would specifically revise to address the points raised, even with regard to easily addressable issues like clarity. Because of this, I have less confidence that the paper could be revised to be in the top 50% of papers accepted to NIPS. However my opinion is that it should still be accepted.


On gradient regularizers for MMD GANs
We propose a principled method for gradient-based regularization of the critic of GAN-like models trained by adversarially optimizing the kernel of a Maximum Mean Discrepancy (MMD). We show that controlling the gradient of the critic is vital to having a sensible loss function, and devise a method to enforce exact, analytical gradient constraints at no additional cost compared to existing approximate techniques based on additive regularizers. The new loss function is provably continuous, and experiments show that it stabilizes and accelerates training, giving image generation models that outperform state-of-the art methods on 160 ⇥ 160 CelebA and 64 ⇥ 64 unconditional ImageNet.
however, when the MMD kernel is not based directly on image pixels, but on learned features of images. Wasserstein-inspired gradient regularization approaches can be used on the MMD critic when learning these features: [27] uses weight clipping [3], and [5,7] use a gradient penalty [18].
The recent Sobolev GAN [33] uses a similar constraint on the expected gradient norm, but phrases it as estimating a Sobolev IPM rather than loosely approximating Wasserstein. This expectation can be taken over the same distribution as [18], but other measures are also proposed, such as (P + Q ✓ ) /2. A second recent approach, the spectrally normalized GAN [32], controls the Lipschitz constant of the critic by enforcing the spectral norms of the weight matrices to be 1. Gradient penalties also benefit GANs based on f -divergences [37]: for instance, the spectral normalization technique of [32] can be applied to the critic network of an f -GAN. Alternatively, a gradient penalty can be defined to approximate the effect of blurring P and Q ✓ with noise [40], which addresses the problem of non-overlapping support [2]. This approach has recently been shown to yield locally convergent optimization in some cases with non-continuous distributions, where the original GAN does not [30].
In this paper, we introduce a novel regularization for the MMD GAN critic of [5,7,27], which directly targets generator performance, rather than adopting regularization methods intended to approximate Wasserstein distances [3,18]. The new MMD regularizer derives from an approach widely used in semi-supervised learning [10,Section 2], where the aim is to define a classification function f which is positive on P (the positive class) and negative on Q ✓ (negative class), in the absence of labels on many of the samples. The decision boundary between the classes is assumed to be in a region of low density for both P and Q ✓ : f should therefore be flat where P and Q ✓ have support (areas with constant label), and have a larger slope in regions of low density. Bousquet et al. [10] propose as their regularizer on f a sum of the variance and a density-weighted gradient norm.
We adopt a related penalty on the MMD critic, with the difference that we only apply the penalty on P: thus, the critic is flatter where P has high mass, but does not vanish on the generator samples from Q ✓ (which we optimize). In excluding Q ✓ from the critic function constraint, we also avoid the concern raised by [32] that a critic depending on Q ✓ will change with the current minibatch -potentially leading to less stable learning. The resulting discrepancy is no longer an integral probability metric: it is asymmetric, and the critic function class depends on the target P being approximated.
We first discuss in Section 2 how MMD-based losses can be used to learn implicit generative models, and how a naive approach could fail. This motivates our new discrepancies, introduced in Section 3. Section 4 demonstrates that these losses outperform state-of-the-art models for image generation.

Summary:  The paper presents a gradient constrained MMD that generalizes  the Sobolev discrepancy  introduced in Sobolev GAN by adding an L_2 regularizer to the gradient regularizer (the kernel case was also discussed in the aforementioned paper). Computing this regularized  MMD is computationally expensive authors propose a bound on the this discrepancy , and optimize with the new scaled MMD , that changes the radius of the ball in the RKHS according to the kernel and its derivative statistics on the target distribution in this paper.  The paper then combines the Scaled MMD with the recently introduced spectral normalization in order to train GAN.   Comments:  - Line 88: phi_psi: X to R^s.  It seems throughout the paper that s is considered to be equal 1. It would be good to mention this early on. Have you tried s >1?  - Line 117 ||f||_{L} if this is the Lipchitz norm this is not correct one need to consider the definition with sup_{x,y} f(x)- f(y)/d(x,y), the one given is a bound.   - Line 208 the gaussian kernel is considered always with sigma =1?  - Table (2 b) SMMDGAN without SN is not given, it  would been nice to have it as well to tease apart the contribution of SN and the scaled MMD. Line 247 and appendix, I think some more ablation study is needed to understand how this scaling and the spectral normalization interact.( see point below on  tracking the RKHS norm and singular values with and without SN might help)  - Have you tracked the cost function in the training process? How does the numerator and denominator behave? It would be great to see how the radius in the RKHS behave in the training and how this relates to the plots of singular values you give in Figure 2 with SN and w/ SN.    - Line 242, on Sobolev GAN it seems that on CIFAR 10 there is an issue in the hyperparameter selection or the implementation  or the choice of mu as it is set in this paper to mu =P. In the reviewer experience Sobolev GAN behaves at least close to WGAN-GP on CIFAR 10.      Minor:   - The figure captions are too sparse, and one needs to jump to the text to know what is going on. It would be good to have better descriptive captions it helps in the readability of the paper.   -Figure 6 (a) and (b) are the same  - would be good to have on a toy data like the mixture of gaussians  to train the GAN with the original cost function given in proposition 4.


A Linear Speedup Analysis of Distributed Deep Learning with Sparse and Quantized Communication
The large communication overhead has imposed a bottleneck on the performance of distributed Stochastic Gradient Descent (SGD) for training deep neural networks. Previous works have demonstrated the potential of using gradient sparsification and quantization to reduce the communication cost. However, there is still a lack of understanding about how sparse and quantized communication affects the convergence rate of the training algorithm. In this paper, we study the convergence rate of distributed SGD for non-convex optimization with two communication reducing strategies: sparse parameter averaging and gradient quantization. We show that O(1/ p MK) convergence rate can be achieved if the sparsification and quantization hyperparameters are configured properly. We also propose a strategy called periodic quantized averaging (PQASGD) that further reduces the communication cost while preserving the O(1/ p MK) convergence rate. Our evaluation validates our theoretical results and shows that our PQASGD can converge as fast as full-communication SGD with only 3% 5% communication data size.

======== after rebuttal Some of the concerns I had about the paper have been addressed by the authors' response (eg in relation to convergence proofs), yet some still stand. Especially the ones related to experiments, and parameter tuning. Since the authors appropriately responded to some of my key criticisms I will upgrade my score from 5 to 6.   ========original review  This paper analyzes the convergence rate of distributed mini-batch SGD using sparse and quantized communication with application to deep learning. Based on the analysis, it proposes combining sparse and quantized communication to further reduce the communication cost that burdens the wall clock runtime in distributed setups.  The paper is generally well written. The convergence analysis does appear to make sense, and the proposed combination of sparsification and quantization seems to save runtime a little bit with proper parameter tuning.  However, I have a few concerns about this work, listed below:  - A concern about this work is the novelty of the analysis. The key idea of proving convergence with noisy gradient update is bounding the variance of the gradient while assuming the noisy gradient update is unbiased, which, has been known for a long time, eg see [1-3] below. In fact, the following two papers, although not discussed in the submitted paper, have given detailed ways of minimizing variance for sparsified+quantized gradients.  [1] Jakub Konečný and Peter Richtárik, Randomized Distributed Mean Estimation: Accuracy vs Communication, 2016. [2] Ananda Theertha Suresh et al, Distributed Mean Estimation with Limited Communication, 2017. [3] Jianqiao Wangni et al. Gradient sparsification for communication-efficient distributed optimization. arXiv preprint arXiv:1710.09854, 2017.  Given the above work, it is unclear what are the new insights provided by this paper's theoretical analysis. One can simply prove bounds on the Variance of the quant./sparse gradients (assuming they are unbiased), and a convergence rate can be shown in a plug an play way. As such, it is important to compare the variance of the proposed techniques, for a given communication budget, in comparison to prior art, as listed above.  - Furthermore, it is not clear why the proposed technique, i.e., using a combination of sparsification and quantization, will be significantly better than standalone sparsification or quantization. As the theoretical convergence bound in the paper (line 255) suggests, the proposed approach converge at rate O(1+q/ sqrt(MK) ), which is worse than that using sparse communication (O(1/sqrt(MK) ), line 173) and that of using quantized communication (O(1/sqrt(MK) ), line 211). The authors do argue that when the parameters are well chosen, the rate can be roughly the same, but this does not explain why a combination can provide an improvement. I might have missed something on that part of the text.   - A key challenge in understanding the performance of distributed SGD using sparse and quantized Communication is how the speedup changes as the compression ratio (either quant. or sparsification) changes. That is for epsilon accuracy, how faster does training happen on P machines versus 1, and what is the best selection of sparsity/quantization levels? Unfortunately, there is no detailed analysis or any experiment shown in the main paper about this. It is critical to have such results (at least an empirical study) to understand the convergence rate in terms of gradient compression efficiency.  - In their experimental section, the authors compare the proposed technique with QSGD, terngrad, and vanilla sgd on a specific set of parameters. Where the parameters all tuned for all methods? Eg were the learning rates optimized individually for both QSGD and terngrad, and all setups of compute nodes?  It is not clear if the speedup will be significant with well-chosen parameters for QSGD and terngrad, and this is necessary for a fair comparison with prior art.  I overall think that the ideas presented in this paper are useful and interesting, but some work still needs to be done in order to clarify some points of confusion. My most major concern is how the proposed algorithm performs experimentally when all other algorithms (eg QSGD) operate with tuned hyperparameters. Finally, since the paper claims to establish linear speedups, these should be also proved experimentally in comparison to prior art.


SplineNets: Continuous Neural Decision Graphs
We present SplineNets, a practical and novel approach for using conditioning in convolutional neural networks (CNNs). SplineNets are continuous generalizations of neural decision graphs, and they can dramatically reduce runtime complexity and computation costs of CNNs, while maintaining or even increasing accuracy. Functions of SplineNets are both dynamic (i.e., conditioned on the input) and hierarchical (i.e., conditioned on the computational path). SplineNets employ a unified loss function with a desired level of smoothness over both the network and decision parameters, while allowing for sparse activation of a subset of nodes for individual samples. In particular, we embed infinitely many function weights (e.g. filters) on smooth, low dimensional manifolds parameterized by compact B-splines, which are indexed by a position parameter. Instead of sampling from a categorical distribution to pick a branch, samples choose a continuous position to pick a function weight. We further show that by maximizing the mutual information between spline positions and class labels, the network can be optimally utilized and specialized for classification tasks. Experiments show that our approach can significantly increase the accuracy of ResNets with negligible cost in speed, matching the precision of a 110 level ResNet with a 32 level SplineNet.

The paper presents SplineNets that reformulate CNNs as neural decision graph using B-splines. It's comprised of four technical contributions, i.e., embedding trick, general formulation for a neural decision graph, a loss function of utilizing and specializing splines and a differentiable quantization method.  The idea is novel and sensible. It integrates the classic splines into convolutional neural networks which might be valuable for both of theoretical and practical aspects. However, the paper is difficult to read and follow. It is unclear how they implement in practice. Some equations and symbols need to be clarified, e.g. the symbols in equation 7-10, and d indicates degree of the polynomials in line 115 but hyperparameter delta in line 235.  I'm confused with several points.  a) what's the role of theta? Is the phi (i.e. D(x; theta) in line 137) analogous to attention (or gating) mechanism?  b)  Does the k/K in line 233 denote the knot number? Does the degree of the polynomial default to be k-1? Or is a fixed degree used? If larger K indicates higher-order smoothness in F-SN, does it mean linear spline works best as J-SN always works better than F-SN. b) how to demonstrate the dynamic (conditioned on the input) introduced by the method?  d) More experimental details need to be provided and clarified, e.g. optimization method, important hyperparameters for training, filter sizes.  I like the idea, but the current draft need to be improved.  After reading the comments of other reviewers and rebuttal, I think it's a good work. I hope the final manuscript could include the updates present in the rebuttal.


Uncertainty Sampling is Preconditioned Stochastic Gradient Descent on Zero-One Loss
Uncertainty sampling, a popular active learning algorithm, is used to reduce the amount of data required to learn a classifier, but it has been observed in practice to converge to different parameters depending on the initialization and sometimes to even better parameters than standard training on all the data. In this work, we give a theoretical explanation of this phenomenon, showing that uncertainty sampling on a convex (e.g., logistic) loss can be interpreted as performing a preconditioned stochastic gradient step on the population zero-one loss. Experiments on synthetic and real datasets support this connection.

The paper shows that after some reasonable assumptions are made, a modified version of uncertainty sampling could be viewed as optimizing zero-one loss of a binary classification problem. Before reading this paper, I also wondered why some papers report that only looking at a subset of samples could improve the classifier performance. I am glad to see a theoretical paper which tries to answer this mystery. I believe that the results will inspire more people to develop better active learning or optimization algorithms.  Strength: Except the typos, the authors seem to pay attention to subtle details in the proof. Some proving techniques they use are pretty clever, although I did not read enough theoretical papers to know how difficult you can come up with those techniques. It is impressive that the main result (theorem 9) does not require the convex assumption of the model. Except the proposition 11 and 12, which sound true but its proofs involve too advanced calculus concepts, I tried to follow every step in the proof. I think all steps are intuitively correct (after ignoring those typos) and are not redundant, although I did not derive every step by myself again to make sure it is 100% right.  Weakness: The main weakness is its writing. First, the author misses some directly related work such as [1] and [2]. [1] and [2] also show the connections between active learning and gradient descent, although they prove the properties of different algorithms and different models. Second, the paper has too many typos and inconsistent notations. In addition, the authors often use terminologies without citations or explanations. These make the readers hard to follow the math, especially the proofs. Third, the theorem actually requires many assumptions, which might not hold in practice and are not stated very clearly in the abstract and introduction. For example, the abstract does not mention that the theorem only holds when large regularization is applied to the classifier and uncertainty sampling selects the points on the decision boundary (as the pool of samples approach to infinity and the probability mass near decision boundary is not zero).  Typos and revision suggestions In the main paper: 22: At the first time of mentioning pre-condition SGD, providing a citation because I think not all people know what pre-condition means.  76: four parameters? Equation (5): saying that you are using Mean-Value theorem 152: Could you define smooth more clearly? 157 and 168: Theta_{valid} should be Theta_{regular} Equation (14): It took me some time to find what psi(x) is at line 50. Recommend to add the definition again here.  In supplementary materials: Equation (17): There is no n_{seed} in (4) Equation (20): B -> M_l Equation (23): I think n should be removed, but equation (24) is correct Equation (31): Missing ) after theta_{t-1}. Missing gradient sign before l(z^(t),theta_{t-1}) Equation (39): Remove extra ] for E Equation (43): Missing E_{y|x} Equation (47) and (48): phi() -> psi()   [1] Guillory, Andrew, Erick Chastain, and Jeff Bilmes. "Active learning as non-convex optimization." Artificial Intelligence and Statistics. 2009. [2] Ramdas, Aaditya, and Aarti Singh. "Algorithmic connections between active learning and stochastic convex optimization." International Conference on Algorithmic Learning Theory. Springer, Berlin, Heidelberg, 2013.  After rebuttal: I think the authors address the concerns of reviewer 1 effectively. I encourage the authors to add that convergence proof to supplementary material. I keep my original judgment because I think all reviewers did not find significant flaws other than typos in the proof and I think the proposed theorems are significant enough to be published in NIPS.


Testing for Families of Distributions via the Fourier Transform *
We study the general problem of testing whether an unknown discrete distribution belongs to a specified family of distributions. More specifically, given a distribution family P and sample access to an unknown discrete distribution P, we want to distinguish (with high probability) between the case that P ∈ P and the case that P is -far, in total variation distance, from every distribution in P. This is the prototypical hypothesis testing problem that has received significant attention in statistics and, more recently, in computer science. The main contribution of this work is a simple and general testing technique that is applicable to all distribution families whose Fourier spectrum satisfies a certain approximate sparsity property. We apply our Fourier-based framework to obtain near sample-optimal and computationally efficient testers for the following fundamental distribution families: Sums of Independent Integer Random Variables (SIIRVs), Poisson Multinomial Distributions (PMDs), and Discrete Log-Concave Distributions. For the first two, ours are the first non-trivial testers in the literature, vastly generalizing previous work on testing Poisson Binomial Distributions. For the third, our tester improves on prior work in both sample and time complexity.

Summary The paper studies the problem of testing membership to a family of distributions. In this problem the goal is to decide if an unknown distribution belongs in some pre-specified class of distributions or if it is at least eps-far in total variation distance from any distribution in the class. While the problem of testing identity to a single distribution is well understood, testing membership to a class is significantly more challenging.   This paper proposes a novel framework for testing membership to classes whose Fourier spectrum is sufficiently concentrated over some small domain. The main idea of the work is that in such cases it is possible to learn the distribution with few samples not only in total variation distance eps but also in L2 distance eps/sqrt(N) where N is the size of the effective support. This allows applying a testing via learning approach that does not work for TV distance as it requires a tolerant identity tester. In contrast, tolerant testing for L2 distance is known to be significantly more sample efficient.  The paper applies the framework to families of distributions that can be expressed as sums of independent random variables and log-concave distributions. The sampling complexities that are achieved are: - Poisson Binomial Distributions (PBD): ~n^(1/4)/eps^2 - Sums of k Integer Random Variables (kSIIRV): ~(kn^(1/4) + k^2)/eps^2 - Poisson Multinomial Distributions (PMD): ~n^((k-1)/4)/eps^2 - Log-Concave Distributions: ~sqrt(n)/eps^2 + 1/eps^2.5  The first result and fourth results are improvements over prior work removing additive terms of the sample complexity in the order of 1/eps^6 and 1/eps^5 respectively. The achieved results are nearly tight. Prior work has established that ~n^(1/4)/eps^2, sqrt(k) n^(1/4)/eps^2 are necessary for the cases of PBD and kSIIRV respectively, while a sqrt(n)/eps^2 lower bound follows from lower bounds for uniformity testing. In addition, this work provides a ~n^((k-1)/4)/eps^2 lower bound for the case of PMD.  Comments The proposed method based on Fourier Transform is quite interesting and the paper nicely illustrate its power and applicability through a range of different settings obtaining near optimal sample complexity bounds that qualitatively match the information theoretic lower bounds.   An important weakness of this work is its presentation. There is very little content in the main body of the paper. Apart from the framework, no detail is given about how it is applied in the results. The explanation in the introduction is quite vague and only mentions difficulties in applying the framework but not how it is applied. Right now everything is pushed to the supplementary materials which is highly technical and hard to read. It should be possible to give a more concrete outline with simplified versions of the algorithms in the main body of how the framework is applied and what Fourier properties of the families of distributions are used. I understand that the NIPS format gives limited space but I find that its use was not well optimized.  Another point missing in the paper is a unified summary of related work. There has been a lot of directly related work for testing families of distributions which is only partially invoked when comparing the results of the paper.    Recommendation Overall, I believe that the proposed framework is quite interesting and novel, and is able to improve on prior work of the literature and get near optimal bounds to many interesting problems. The main drawback I find is the presentation of the results. The current submission contains very little content in the main body. I think an outline of the algorithms and how the framework is applied to the families of distributions considered are necessary to be included.


Unsupervised Text Style Transfer using Language Models as Discriminators
Binary classifiers are often employed as discriminators in GAN-based unsupervised style transfer systems to ensure that transferred sentences are similar to sentences in the target domain. One difficulty with this approach is that the error signal provided by the discriminator can be unstable and is sometimes insufficient to train the generator to produce fluent language. In this paper, we propose a new technique that uses a target domain language model as the discriminator, providing richer and more stable token-level feedback during the learning process. We train the generator to minimize the negative log likelihood (NLL) of generated sentences, evaluated by the language model. By using a continuous approximation of discrete sampling under the generator, our model can be trained using back-propagation in an end-to-end fashion. Moreover, our empirical results show that when using a language model as a structured discriminator, it is possible to forgo adversarial steps during training, making the process more stable. We compare our model with previous work that uses convolutional networks (CNNs) as discriminators, as well as a broad set of other approaches. Results show that the proposed method achieves improved performance on three tasks: word substitution decipherment, sentiment modification, and related language translation.

This paper proposes using the language models' log-likelihood score as a discriminator in order to train style transfer generation models. In particular, given a text x, the goal is to train an encoder and a decoder. The encoder extracts vectors z_x and v_x from x, representing the content and the style of x respectively. The decoder should be able to restore x from (z_x, v_x), but generate a stylistically different sentence from z_x and a different vector v_y. The paper proposes to judge the style difference by a language model through its log-likelihood score.  Despite the simple idea, the authors show that it works well on three style transfer tasks, and achieves comparable or better performances than the state-of-the-art adversarially trained models. The results look promising. I agree with the authors that using language model as the discriminator is a compelling approach because it avoids the adversarial training's instability problem.  It was mentioned in the paper that the language model discriminator has its own problems. One of them is that the language model always prefers short sentences. Thus additional length normalization and training hacks are necessary to make it work. The adversarially trained models don't seem to have this problem.   A bigger concern of mine is that the proposed style transfer method could lose the semantics of the sentence more easily than the method of Shen et al. If x is a sentence from corpus X and y' is a sentence from corpus Y whose style is transferred to X. Shen et al.'s method requires the distributions of x and y' to be indistinguishable. In this paper, there is a penalty if y' has the same distribution as y, but there is no penalty if y' has a completely different distribution from that of x. In the extreme case, the total loss can be very low if the decoder learns to restore all the original input by memorizing and copying them, but always generates random strings if the content is paired with a transferred style vector. This doesn't happen in the paper's reported experiments, but I am worried that for harder tasks, given the strong overfitting capability of neural networks, the network would find it easier to learn such a copy-or-random generator than learning the actually useful style transfer model.  Overall the paper is clearly written and has throughout experiments. Therefore I tend to accept the paper.


Mean Field for the Stochastic Blockmodel: Optimization Landscape and Convergence Issues
Variational approximation has been widely used in large-scale Bayesian inference recently, the simplest kind of which involves imposing a mean field assumption to approximate complicated latent structures. Despite the computational scalability of mean field, theoretical studies of its loss function surface and the convergence behavior of iterative updates for optimizing the loss are far from complete. In this paper, we focus on the problem of community detection for a simple twoclass Stochastic Blockmodel (SBM). Using batch co-ordinate ascent (BCAVI) for updates, we show different convergence behavior with respect to different initializations. When the parameters are known, we show that a random initialization can converge to the ground truth, whereas in the case when the parameters themselves need to be estimated, a random initialization will converge to an uninformative local optimum.

Considers a simple variant of the stochastic block model (SBM) for networks with 2 communities/clusters.  Previous work has studied the asymptotic behavior of the global optimizer of the mean field variational inference objective; this paper instead analyzes the dynamics of mean field coordinate ascent updates.  It shows recovery of the correct community structure for most random initializations given correct model parameters, but for a particular way of initializing inference with unknown community interaction probabilities, inference converges to an uninformative local optimum with high probability.  ORIGINALITY:  Recent papers have given related analyses of inference algorithms for mixture models, but study of inference dynamics for the SBM appears novel.  CLARITY:  The fairly technical results here would clearly be easier to present in an extended journal paper.  This submission puts some proofs in the supplement, but in my view this only partially succeeds in making the main manuscript readable.  It would benefit from moving a few more details from Sec. 3 to the supplement, and adding additional explanation.  There is a lot of mathematical notation, some introduced with no explanation even on first use, and I found myself repeatedly hunting through the paper to find the definition of terms.  Some more detailed comments: 1) Sec. 2 does not mention that the reason probabilities B are constrained symmetric is that A is the symmetric adjacency matrix of an undirected graph.  (Several previous papers have developed SBM variants for directed graphs, with non-symmetric B.) 2) Sec. 2 needs to be slower in explaining how the algorithm is changing in the last two equations.  (Batch updates are not coordinate ascent on the mean field objective, and in general are not guaranteed convergent.  The "population" variant justification is ignored.)  Also \xi^(s) is not defined, and its connection to \psi^(s) is not specified. 3) Proposition 3.1 statement, second "eigenvalues" should be "eigenvectors".  Alpha variables are an example of notation introduced without comment in the middle of a technical argument, that you then assume the reader will remember when trying to understand your experiments in Sec. 4.  EXPERIMENTS:  The (limited) experiments in Sec. 4 are a useful supplement to the earlier theory, but the text here is terse and includes insufficient explanation for the conceptual interpretation of the variables involved.  In particular, a much clearer explanation for Figure 1(a) is needed.  TECHNICAL CONTENT & SIGNIFICANCE:   The technical derivations needed to establish the main theorems are non-trivial, and the key arguments in the main paper all seem correct.  (I did not verify all proofs from the appendices.)  Given that the model is restricted to two communities, it found it unsurprising to find (Theorem 3.3) that mean field converges to the correct community structure with known parameters.  (This is similar to the positive results of [25] for 2-cluster Gaussian mixtures, but I would argue that the negative results in [13] with 3 or more clusters are more illuminating and important.)  The negative results of Sec. 3.2 are more interesting.  They essentially say that if inference is initialized with random community assignments, and model parameters are updated from there, the probability of identification of correct communities is exponentially small.  However, I disagree with the claim in Sec. 5 that this is "typically done by most practitioner".  The researchers who I know that do applied work with cluster-based models know that initializing with random partitions tends to lead to very poor local optima.  For this SBM, here is one example of a more standard initialization: 1) Sample a "reasonable" set of initial block interaction probabilities.  (For this model, this proposal would pick p >> q.) 2) Do at least one, and perhaps several, updates of the community probabilities psi given the fixed initial (p,q). 3) From this point, update (p,q) and then iterate as analyzed in the paper.  The positive results in Sec. 3.1 with correct (p,q) suggest that if the (p,q) sampled in step #1 is within a neighborhood of the correct values, this algorithm may lead to a good optimum.  Analysis of such a more realistic inference scheme could be an interesting area for future research.


A Probabilistic U-Net for Segmentation of Ambiguous Images
Many real-world vision problems suffer from inherent ambiguities. In clinical applications for example, it might not be clear from a CT scan alone which particular region is cancer tissue. Therefore a group of graders typically produces a set of diverse but plausible segmentations. We consider the task of learning a distribution over segmentations given an input. To this end we propose a generative segmentation model based on a combination of a U-Net with a conditional variational autoencoder that is capable of efficiently producing an unlimited number of plausible hypotheses. We show on a lung abnormalities segmentation task and on a Cityscapes segmentation task that our model reproduces the possible segmentation variants as well as the frequencies with which they occur, doing so significantly better than published approaches. These models could have a high impact in real-world applications, such as being used as clinical decision-making algorithms accounting for multiple plausible semantic segmentation hypotheses to provide possible diagnoses and recommend further actions to resolve the present ambiguities.

Post rebuttal: Authors have responded well to the issues raised, and I champion publication of this work.    Main idea:   Use a conditional variational auto-encoder to produce well-calibrated segmentation hypotheses for a given input.   Strengths:   The application is well motivated and experiments are convincing and state of the art. Convincing baselines, good discussion and conclusion. Nice illustrations in the appendix.   Weaknesses:   No significant theoretical contribution.   Possibly in response, the manuscript is a little vague in its positioning relative to prior work. While relevant prior work is cited, the reader is left with some ambiguity and, if not familiar with this prior work, might be misled to think that there is methodological innovation beyond the specifics of architecture and application.   Based on its solid and nontrivial experimental contribution I advocate acceptance; but the manuscript would profit from a clearer enunciation of the fact that / what prior work is being built on.   Comments:   50: "treat pixels independently"; this is unclear (or faulty?), the quoted papers also use an encoder/decoder structure  As a consequence, contribution 1 (line 74) is dubious or needs clarification.   Contribution (2) is fine.   Contribution (3): The statement is true, but the contribution is unclear. If the claim refers to the fact that latent z is concatenated only at the fully connected layers, then this has been done before (e.g. in DISCO Nets by Bouchacourt et al., (NIPS 2016)).  Contribution (4): The claim is vague. If we take generative models in their generality, then it encompasses e.g. most of Bayesian Statistics and the claim of only qualitative evaluation is obviously wrong. If we only consider generative models in the deep learning world, the statement is correct insofar as many papers only contain qualitative evaluation of "My pictures are prettier than yours"; but there are nevertheless quantitative metrics, such as the inception score, or the metrics used in the Wasserstein AE.   Section 2:  It is not made very obvious to the reader which part of this VAE structure is novel and which parts are not. The paper does follow [4] and especially [5] closely (the latter should also be cited in line 82). So the only really novel part here is the U-net structure of P(y|z,x). Concatenating z after the U-net in (2) is new in this formulation, but not in general (e.g. as already mentioned by DISCO Nets (Bouchacourt at al., NIPS 2016)).   Finally, there is no justification for the appearance of $\beta$ in (4), but it is up to the parameter name identical to what Higgins et al., (ICLR 2017) do with their \beta-VAE. Especially since authors choose $\beta >= 1$, which follows the Higgins et al. disentangling argument, and not the usual $\beta_t <= 1$, in which case it would be a time dependent downscaling of the KL term to avoid too much regularization in the beginning of the training (but then again the references to earlier work would be missing).


Efficient Projection onto the Perfect Phylogeny Model
Several algorithms build on the perfect phylogeny model to infer evolutionary trees. This problem is particularly hard when evolutionary trees are inferred from the fraction of genomes that have mutations in different positions, across different samples. Existing algorithms might do extensive searches over the space of possible trees. At the center of these algorithms is a projection problem that assigns a fitness cost to phylogenetic trees. In order to perform a wide search over the space of the trees, it is critical to solve this projection problem fast. In this paper, we use Moreau's decomposition for proximal operators, and a tree reduction scheme, to develop a new algorithm to compute this projection. Our algorithm terminates with an exact solution in a finite number of steps, and is extremely fast. In particular, it can search over all evolutionary trees with fewer than 11 nodes, a size relevant for several biological problems (more than 2 billion trees) in about 2 hours.

This paper attempts to solve an optimization subproblem which arises in a matrix factorization which arises in studying the phylogenetic evolutionary tree.  In particular, the overall problem is given a matrix F, find the matrices U and M such that F=UM where U is a binary matrix and M is a positive matrix whose columns sum to 1.  The subproblem is, given U, to find the least-squares solution F,M such that || \hat{F} - F || is minimized subject to the constraints that F = UM where \hat{F} is the measured matrix.  The paper proposes an algorithm which can solve this subproblem directly in polynomial time (compared with iterative solvers) and the major contribution is the algorithm for computing this along with the theoretical proofs of its optimality.  Beyond this analysis, practical techniques are proposed for evaluating the algorithm efficiently.  The algorithm is then applied and compared against current state-of-the-art iterative solvers, specifically ADMM and PGD.  Experimental results show that the proposed algorithm is dramatically faster in practice when compared against existing iterative solvers.  Overall the paper seems well written and the results significant.  I suspect the paper will be of interest beyond those studying phyolgenetic trees as the main contribution is related to the optimization sub problem.   UPDATE: Having read the author feedback I remain largely positive on the paper.  In addition, the authors included additional experimental comparison against other methods seems to be a very positive and compelling addition to the paper.


PG-TS: Improved Thompson Sampling for Logistic Contextual Bandits
We address the problem of regret minimization in logistic contextual bandits, where a learner decides among sequential actions or arms given their respective contexts to maximize binary rewards. Using a fast inference procedure with Pólya-Gamma distributed augmentation variables, we propose an improved version of Thompson Sampling, a Bayesian formulation of contextual bandits with near-optimal performance. Our approach, Pólya-Gamma augmented Thompson Sampling (PG-TS), achieves state-of-the-art performance on simulated and real data. PG-TS explores the action space efficiently and exploits high-reward arms, quickly converging to solutions of low regret. Its explicit estimation of the posterior distribution of the context feature covariance leads to substantial empirical gains over approximate approaches. PG-TS is the first approach to demonstrate the benefits of Pólya-Gamma augmentation in bandits and to propose an efficient Gibbs sampler for approximating the analytically unsolvable integral of logistic contextual bandits.

This submission tries to propose Polya-gamma augmented Thompson sampling from a frequentist-derived algorithm whose likelihood is a mixture of Gaussian and PG mixing distribution when the reward is the logistic model with binary observation in contextual bandits to improve the Laplace approximation based Bayesian sampling. Experiments are demonstrated on the real world data including the public benchmark Yahoo data.  Detailed comments:  Like you admitted the logistic reward is pragmatic, that's why so far very few have been researched on this topic, which makes it difficult to get adopted by the community.   As you stated the example in the news recommendation setting case in the click or not over the item problem, which is the same or similar to recent advances that took over the contextual bandit, see below.  You adopted the traditional Gibbs sampler, where you sample from the PG distribution with M = 100 burn-in steps, for the experiment is fine, but lack of regret bounds to support, that you may want to study more, e.g., see [1]'s section or appendix about TS.  Laplace-TS's batch update is a good pro but did not fully explored at the moment.  Gibbs step M is too tiny and does not well theoretically motivated.  Laplace approximations are sensitive to multimodality is a con.  The sample complexity or size etc would be clearly stated and analyzed.  You played with a toy dataset on Forest Cover Type from UCI where you only reported the result over 100 runs with 1000 trials, I suggest adjusting the performance towards at least, say, 50k or 100k etc. There is no way to adopt some wrong results as illustrated in Figure 4, it may take a while for you to conquer though.  Your major issue is lack of theoretical guarantee or regret (either upper or lower) bound for the proposed algorithm, as it's standard in this field.  The regret gain is not significant from Figure 5 through the experimental report presented, and the benefit of arm's frequency is not conniving as far as I can tell.  You partitioned the UCI dataset to k = 32 clusters by the traditional k-means clustering where you set the reward of each arm to be the average reward in the corresponding cluster, where you might want to change k's values to see the difference.  Missing references: [1] ICML 2017 On Context-Dependent Clustering of Bandits [2] SIGIR 2016 Collaborative Filtering Bandits [3] ICML 2016 Distributed Clustering of Linear Bandits in Peer to Peer Networks  Where the exactly same Yahoo today module benchmark data is employed, in this sense that your proposed solutions should be able to compare with the state-of-the-art overtook traditional contextual bandit[1-3], which I am curious to see the empirical comparison at least with either one of [1-3].  Since there is no regret bound provided in this manuscript, the empirical result as described and in Figure 4 which showed that your result is wrong or inconsistent on your implementation for conducting the experiment on Yahoo, see more details on how to do legitimate experiments therein in [2,3] to observe the correct performance behavior you should achieve.  The discussion is not really a discussion part, and the real conclusion is missing. Please differentiate these two sections and wrote them separately.  Last but not least, this manuscript should be written in a more professional English writing including to fix typos and grammar errors etc, e.g., "Motivated by our results" and so forth. You should carefully address these concerns to increase the quality of this submission.  %Update after rebuttal% I read their rebuttal, and they did a good job on the clarifications. I have increased my score, and I have no problem to accept this submission.


A no-regret generalization of hierarchical softmax to extreme multi-label classification
Extreme multi-label classification (XMLC) is a problem of tagging an instance with a small subset of relevant labels chosen from an extremely large pool of possible labels. Large label spaces can be efficiently handled by organizing labels as a tree, like in the hierarchical softmax (HSM) approach commonly used for multi-class problems. In this paper, we investigate probabilistic label trees (PLTs) that have been recently devised for tackling XMLC problems. We show that PLTs are a no-regret multi-label generalization of HSM when precision@k is used as a model evaluation metric. Critically, we prove that pick-one-label heuristic-a reduction technique from multi-label to multi-class that is routinely used along with HSM-is not consistent in general. We also show that our implementation of PLTs, referred to as EXTREMETEXT (XT), obtains significantly better results than HSM with the pick-one-label heuristic and XML-CNN, a deep network specifically designed for XMLC problems. Moreover, XT is competitive to many state-of-the-art approaches in terms of statistical performance, model size and prediction time which makes it amenable to deploy in an online system.

Summary: This work investigates Probabilistic Label Trees (PLTs) in solving extreme multi-label classification (XMLC). The theoretical analysis shows PLT is a no-regret algorithm for precision@k, and the algorithmic improvement combines PLT and fastText to efficiently handle  extreme multi-label text classification problems, with a clustering-based tree structure building strategy.   This paper is comphrensive and well-written, including extensive experiments.  The theory part formally shows PLT outputing k labels with highest marginal probabilities is consistent with precision@k, given zero-regret node classifiers. The authors also provide some negative result on heuristic strategies, one is that pick-one-label heuristic is suboptimal in terms of precision@k, and another is that building Huffman trees for PLT does not minimize computational cost.  I think the consistency result is a bit straightforward since the zero-error assumption on marginal probabilities is strong enough to provide many nice properties. However, as the authors said in Line 231-233, if this assumption holds, there is no need to care about the tree structure in PLT, which is surely an important issue in practice. Also, it is easy to see that PLT is preferred than vanilla HSM in XMLC.  Personally speaking, I would like to see the authors put more theory parts into the supplementary and show more experiments on the tree structure decision in PLT. For example:  - The performance difference between original PLT paper vs. Huffman tree vs. top-down hierarchical clustering used in this paper. - The improvement by applying fastText into PLT. Does fastText help both in time cost and performance?  ============================== Post-author feedback comments:  I have read the author feedback and found out that their theoretical results are based on the maximum estimation error of marginal probability on nodes. Although this measure may not be available in practice, the theoretical contribution is more than what I expected before.   Therefore I increased my score.


How to Start Training: The Effect of Initialization and Architecture
We identify and study two common failure modes for early training in deep ReLU nets. For each, we give a rigorous proof of when it occurs and how to avoid it, for fully connected, convolutional, and residual architectures. We show that the first failure mode, exploding or vanishing mean activation length, can be avoided by initializing weights from a symmetric distribution with variance 2/fan-in and, for ResNets, by correctly scaling the residual modules. We prove that the second failure mode, exponentially large variance of activation length, never occurs in residual nets once the first failure mode is avoided. In contrast, for fully connected nets, we prove that this failure mode can happen and is avoided by keeping constant the sum of the reciprocals of layer widths. We demonstrate empirically the effectiveness of our theoretical results in predicting when networks are able to start training. In particular, we note that many popular initializations fail our criteria, whereas correct initialization and architecture allows much deeper networks to be trained.

The paper studies two failure modes during the initial phase of deep network training: 1. The mean length of activation vector in the last layer increases/decreases exponentially with depth. The mean is taken over the randomness in weights/biases while the input is considered fixed. 2. The variance of length of activation vector grows exponentially with depth.  It has been shown that FM1 in fully connected architectures can be avoided by sampling the weights of the network from a symmetric distribution with variance 2/fan-in. For ResNets, it has been shown that in addition to weight initialization, each residual block needs to be scaled appropriately such that the sum of reciprocal of these scaling factors is close to 1 (the latter is not mentioned explicitly/clearly).  It is shown that FM2 is avoided for fully connected networks when using large width in networks. More specifically, the variance in the length of activation vector is exponential in the sum of reciprocal of widths of layers. For ResNets it is shown that addressing FM1 automatically rules out FM2.   All theoretical results are complemented side-by-side with appropriate experiments. For FM1, the mean length of activations is shown with depth for different initializations along with the number of epochs needed to achieve 20% performance. For FM2, only the latter is shown (It would be informative to also show how the empirical variance behaves for different cases).  Comments:  FM1 is well studied in literature for fully connection architecture as acknowledged by authors as well. The novelty seems to be twofold: 1. FM1 has not been well studied for ResNet architecture. It is studied in this paper. 2. When studying FM1, previous literature assumes the input is random and the variance of activations is calculated over both the randomness in input and parameter initialization distribution. In this paper, input is considered fixed.  Regarding point 2 above, I am not sure I understand why preserving the mean length of activation vector in the last layer for a *fixed input* is important. In previous work, the motivation is to preserve the information content in the input. As a proxy, this information is measured by the variance in input and so calculating the variance of hidden units over the input/parameter distribution makes sense. The reasoning for considering fixed input is not discussed in the paper. Please clarify.  Also, since the theory predicts that FM2 in fully connected networks can be avoided by using network widths such that the sum of reciprocal is small, I am curious if it equivalent to use something like widths 10,20,30,40 vs widths 40,30,20,10 in a network. In these two cases, the sum of reciprocal of widths is identical, but one has width increasing with depth while the other has width decreasing with depth. My intuition is that the latter may have a poor "mean epochs to 20%". I recommend the authors to add this experiment. If the results are unexpected, it would be interesting to leave it as an open thread, and in the other case it would make the theory more compelling.  The authors should acknowledge in line 173 that the paper "A Closer Look at Memorization in Deep Networks" discussed (their Figure 6) that time to convergence is better for wider networks.  (I did not go through the proofs.)  Clarity:  The novelty of the paper is not stated clearly. For instance, the first two contributions mentioned in the paper are already known. The novelty in these contributions is not clear.  There are two problems in the way the authors have introduced FM1 and FM2:  1. Clearly identifying FM1 and FM2 in previous work and better motivating why they (can) cause failure is very important and does not seem to be done properly. Instead, after stating FM1 and FM2 in the introduction, the rest of the paper discusses how to solve it. This is especially the case with FM2 which is not well known. 2. While it is stated in the definition of FM1 and FM2 that quantities increase/decrease exponentially with depth, and that it is problematic, it is never stated explicitly how these quantities should ideally behave: mean length should be preserved and variance should be as small as possible.    Originality:  As mentioned above, the novelty needs to be stated clearly and the motivation for FM2 and why FM1 is considered for fixed input needs to be clarified.  Overall, I think the paper is somewhat original. The drawback is that initializing networks parameters from distributions with variance 2/fan-in is already known. But the part about ResNet is novel. The part about why large width is important is also novel and very interesting.  Significance: I think the intialization scheme for ResNet (scaling residual blocks seems to be the novel part) can be of importance for practitioners.   I have given an overall score of 6 mainly because of lack of clarity in the paper.  ## Post rebuttal update:  The authors clarified some of the questions I raised. Although the experiment on training a network with decreasing width is not conducted for the rebuttal as I proposed, the authors agree to include it in the final submission.   However I have two other concerns:  1. The paper talks about preserving activation length and variance which was originally proposed as a way to preserve information during the forward pass of a deep network. However there is no discussion on gradient norm preservation which is an important for avoiding vanishing/exploding gradients problem during initialization and is also important for avoiding the initial failure during training.  2. The constant C that appears in theorem 5 is only discussed to the point that it exists and is positive. But no further discussion is provided. This is a concern because a very large C satisfying theorem 5 would be trivial since the activation length will effectively not be preserved. This is further important because the theorem talks about any fixed input vector. When considering a large number of input vectors (a dataset), it is more likely for the variance in activation norm is large for some of the samples.  For these reasons, I am sticking with my original score.