We present a generic dynamic architecture that employs a problem specific differentiable forking mechanism to leverage discrete logical information about the problem data structure. We adapt and apply our model to CLEVR Visual Question Answering, giving rise to the DDRprog architecture; compared to previous approaches, our model achieves higher accuracy in half as many epochs with five times fewer learnable parameters. Our model directly models underlying question logic using a recurrent controller that jointly predicts and executes functional neural modules; it explicitly forks subprocesses to handle logical branching. While FiLM and other competitive models are static architectures with less supervision, we argue that inclusion of program labels enables learning of higher level logical operations -our architecture achieves particularly high performance on questions requiring counting and integer comparison. We further demonstrate the generality of our approach though DDRstack -an application of our method to reverse Polish notation expression evaluation in which the inclusion of a stack assumption allows our approach to generalize to long expressions, significantly outperforming an LSTM with ten times as many learnable parameters.

Summary:
The paper presents a generic dynamic architecture for CLEVR VQA and Reverse Polish notation problems. Experiments on CLEVR show that the proposed model DDRprog outperforms existing models, but it requires explicit program supervision. The proposed architecture for RPN, called DDRstack outperforms an LSTM baseline.
Strengths:
— For CLEVR VQA task, the proposed model outperforms the state-of-the-art with significantly less number of parameters.
— For RPN task, the proposed model outperforms baseline LSTM model by a large margin.
Weaknesses:
— The paper doesn’t describe the model clearly. After reading the paper, it’s not clear to me what the components of the model are, what each of them take as input and produce as output, what these modules do and how they are combined. I would recommend to restructure the paper to clearly mention each of the components, describe them individually and then explain how they are combined for both cases - DDRprog and DDRstack.
— Is the “fork” module the main contribution of the paper? If so, at least this should be described in detail. So, if no fork module is required for a question, the model architecture is effectively same as IEP?
— Machine accuracy is already par with human accuracy on CLEVR and very close to 100%. Why is this problem still important?
— Given that the performance of state-on-art on CLEVR dataset is already very high ( <5% error) and the performance numbers of the proposed model are not very far from the previous models, it is very important to report the variance in accuracies along with the mean accuracies to determine if the performance of the proposed model is statistically significantly better than the previous models or not.
— In Figure 4, why are the LSTM32/128 curves different for Length 10 and Length 30 till subproblem index 10? They are both trained on the same training data, only test data is of different length and ideally both models should achieve similar accuracy for the first 10 subproblems (same trend as DDRstack).
— Why is DDRstack not compared to StackRNN?
— Can the authors provide training time comparison of their model and other/baseline models? Because that is more important than the number of epochs required in training.
— There are only 3 curves (instead of 4) in Figure 3.
— In a number of places, the authors are referring to left and right program branches. What are they? These names have not being defined formally in the paper.
Overall:
I think the research work in the paper is interesting and significant, but given the current presentation and level of detail in the paper, I don’t think it will be helpful for the research community. By proper restructuring of paper and adding more details, the paper can be converted to a solid submission.


Distributional Semantics Models(DSM) derive word space from linguistic items in context. Meaning is obtained by defining a distance measure between vectors corresponding to lexical entities. Such vectors present several problems. This work concentrates on quality of word embeddings, improvement of word embedding vectors, applicability of a novel similarity metric used 'on top' of the word embeddings. In this paper we provide comparison between two methods for post process improvements to the baseline DSM vectors. The counter-fitting method which enforces antonymy and synonymy constraints into the Paragram vector space representations recently showed improvement in the vectors' capability for judging semantic similarity. The second method is our novel RESM method applied to GloVe baseline vectors. By applying the hubness reduction method, implementing relational knowledge into the model by retrofitting synonyms and providing a new ranking similarity definition RESM that gives maximum weight to the top vector component values we equal the results for the ESL and TOEFL sets in comparison with our calculations using the Paragram and Paragram + Counter-fitting methods. For SIMLEX-999 gold standard since we cannot use the RESM the results using GloVe and PPDB are significantly worse compared to Paragram. Apparently, counter-fitting corrects hubness. The Paragram or our cosine retrofitting method are state-of-the-art results for the SIMLEX-999 gold standard. They are 0.2 better for SIMLEX-999 than word2vec with sense de-conflation (that was announced to be state-of the-art method for less reliable gold standards). Apparently relational knowledge and counter-fitting is more important for judging semantic similarity than sense determination for words. It is to be mentioned, though that Paragram hyperparameters are fitted to SIMLEX-999 results. The lesson is that many corrections to word embeddings are necessary and methods with more parameters and hyperparameters perform better.

This paper proposes a ranking-based similarity metric for distributional semantic models. The main idea is to learn "baseline" word embeddings, retrofitting those and applying localized centering, to then calculate similarity using a measure called "Ranking-based Exponential Similarity Measure" (RESM), which is based on the recently proposed APSyn measure.
I think the work has several important issues:
1. The work is very light on references. There is a lot of previous work on evaluating similarity in word embeddings (e.g. Hill et al, a lot of the papers in RepEval workshops, etc.); specialization for similarity of word embeddings (e.g. Kiela et al., Mrksic et al., and many others); multi-sense embeddings (e.g. from Navigli's group); and the hubness problem (e.g. Dinu et al.). For the localized centering approach, Hara et al.'s introduced that method. None of this work is cited, which I find inexcusable.

2. The evaluation is limited, in that the standard evaluations (e.g. SimLex would be a good one to add, as well as many others, please refer to the literature) are not used and there is no comparison to previous work. The results are also presented in a confusing way, with the current state of the art results separate from the main results of the paper. It is unclear what exactly helps, in which case, and why.

3. There are technical issues with what is presented, with some seemingly factual errors. For example, "In this case we could apply the inversion, however it is much more convinient [sic] to take the negative of distance. Number 1 in the equation stands for the normalizing, hence the similarity is defined as follows" - the 1 does not stand for normalizing, that is the way to invert the cosine distance (put differently, cosine distance is 1-cosine similarity, which is a metric in Euclidean space due to the properties of the dot product). Another example, "are obtained using the GloVe vector, not using PPMI" - there are close relationships between what GloVe learns and PPMI, which the authors seem unaware of (see e.g. the GloVe paper and Omer Levy's work).

4. Then there is the additional question, why should we care? The paper does not really motivate why it is important to score well on these tests: these kinds of tests are often used as ways to measure the quality of word embeddings, but in this case the main contribution is the similarity metric used *on top* of the word embeddings. In other words, what is supposed to be the take-away, and why should we care?
As such, I do not recommend it for acceptance - it needs significant work before it can be accepted at a conference.
Minor points:
- Typo in Eq 10
- Typo on page 6 (/cite instead of \cite)


Searching for Activation Functions
The choice of activation functions in deep networks has a significant effect on the training dynamics and task performance. Currently, the most successful and widely-used activation function is the Rectified Linear Unit (ReLU). Although various hand-designed alternatives to ReLU have been proposed, none have managed to replace it due to inconsistent gains. In this work, we propose to leverage automatic search techniques to discover new activation functions. Using a combination of exhaustive and reinforcement learning-based search, we discover multiple novel activation functions. We verify the effectiveness of the searches by conducting an empirical evaluation with the best discovered activation function. Our experiments show that the best discovered activation function, f (x) = x · sigmoid(βx), which we name Swish, tends to work better than ReLU on deeper models across a number of challenging datasets. For example, simply replacing ReLUs with Swish units improves top-1 classification accuracy on ImageNet by 0.9% for Mobile NASNet-A and 0.6% for Inception-ResNet-v2. The simplicity of Swish and its similarity to ReLU make it easy for practitioners to replace ReLUs with Swish units in any neural network.

Authors propose a reinforcement learning based approach for finding a non-linearity by searching through combinations from a set of unary and binary operators. The best one found is termed Swish unit; x * sigmoid(b*x).
The properties of Swish like allowing information flow on the negative side and linear nature on the positive have been proven to be important for better optimization in the past by other functions like LReLU, PLReLU etc. As pointed out by the authors themselves for b=1 Swish is equivalent to SiL proposed in Elfwing et. al. (2017).
In terms of experimental validation, in most cases the increase is performance when using Swish as compared to other models are very small fractions. Again, the authors do state that "our results may not be directly comparable to the results in the corresponding works due to differences in our training steps."
Based on the Figure 6 authors claim that the non-monotonic bump of Swish on the negative side is very important aspect. More explanation is required on why is it important and how does it help optimization. Distribution of learned b in Swish for different layers of a network can interesting to observe.


Saliency methods aim to explain the predictions of deep neural networks. These methods lack reliability when the explanation is sensitive to factors that do not contribute to the model prediction. We use a simple and common pre-processing step -adding a mean shift to the input data-to show that a transformation with no effect on the model can cause numerous methods to incorrectly attribute. We define input invariance as the requirement that a saliency method mirror the sensitivity of the model with respect to transformations of the input. We show, through several examples, that saliency methods that do not satisfy a input invariance property are unreliable and can lead to misleading and inaccurate attribution.

Saliency methods are effective tools for interpreting the computation performed by DNNs, but evaluating the quality of interpretations given by saliency methods are often largely heuristic. Previous work has tried to address this shortcoming by proposing that saliency methods should satisfy "implementation invariance", which says that models that compute the same function should be assigned the same interpretation. This paper builds on this work by proposing and studying "input invariance", a specific kind of implementation invariance between two DNNs that compute identical functions but where the input is preprocessed in different ways. Then, they examine whether a number of existing saliency methods satisfy this property.
The property of "implementation invariance" proposed in prior work seems poorly motivated, since the entire point of interpretations is that they should explain the computation performed by a specific network. Even if two DNNs compute the same function, they may do so using very different computations, in which case it seems natural that their interpretations should be different. Nevertheless, I can believe that the narrower property of input invariance should hold for saliency methods.
A much more important concern I have is that the proposed input invariance property is not well motivated. A standard preprocessing step for DNNs is to normalize the training data, for example, by subtracting the mean and dividing by the standard deviation. Similarly, for image data, pixel values are typically normalized to [0,1]. Assuming inputs are transformed in this way, the input invariance property (for mean shift) is always trivially satisfied. The paper does not justify why we should consider networks where the training data is not normalized is such a way.
Even if the input is not normalized, the failures they find in existing saliency methods are typically rather trivial. For example, for the gradient times input method, they are simply noting that the interpretation is translated by the gradient times the mean shift. The paper does not discuss why this shift matters. It is not at all clear to me that the quality of the interpretation is adversely affected by these shifts.
I believe the notion that saliency methods should be invariant to input transformations may be promising, but more interesting transformations must be considered -- as far as I can tell, the property of invariance to linear transformations to the input does not provide any interesting insight into the correctness of saliency methods.


Generative models are important tools to capture and investigate the properties of complex empirical data. Recent developments such as Generative Adversarial Networks (GANs) and Variational Auto-Encoders (VAEs) use two very similar, but reverse, deep convolutional architectures, one to generate and one to extract information from data. Does learning the parameters of both architectures obey the same rules? We exploit the causality principle of independence of mechanisms to quantify how the weights of successive layers adapt to each other. Using the recently introduced Spectral Independence Criterion, we quantify the dependencies between the kernels of successive convolutional layers and show that those are more independent for the generative process than for information extraction, in line with results from the field of causal inference. In addition, our experiments on generation of human faces suggest that more independence between successive layers of generators results in improved performance of these architectures.

This paper examines the nature of convolutional filters in the encoder and a decoder of a VAE, and a generator and a discriminator of a GAN. The authors treat the inputs (X) and outputs (Y) of each filter throughout each step of the convolving process as a time series, which allows them to do a Discrete Time Fourier Transform analysis of the resulting sequences. By comparing the power spectral density of the input and the output, they get a Spectral Dependency Ratio (SDR) ratio that characterises a filter as spectrally independent (neutral), correlating (amplifies certain frequencies), or anti-correlating (dampens frequencies). This analysis is performed in the context of the Independence of Cause and Mechanism (ICM) framework. The authors claim that their analysis demonstrates a different characterisation of the inference/discriminator and generative networks in VAE and GAN, whereby the former are anti-causal and the latter are causal in line with the ICM framework. They also claim that this analysis can be used to improve the performance of the models.
Pros:
-- SDR characterisation of the convolutional filters is interesting
-- The authors show that filters with different characteristics are responsible for different aspects of image modelling
Cons:
-- The authors do not actually demonstrate how their analysis can be used to improve VAEs or GANs
-- Their proposed SDR analysis does not actually find much difference between the generator and the discriminator of the GAN
-- The clarity of the writing could be improved (e.g. the discussion in section 3.1 seems inaccurate in the current form). Grammatical and spelling mistake are frequent. More background information could be helpful in section 2.2. All figures (but in particular Figure 3) need more informative captions
-- The authors talk a lot about disentangling in the introduction, but this does not seem to be followed up in the rest of the text. Furthermore, they are missing a reference to beta-VAE (Higgins et al, 2017) when discussing VAE-based approaches to disentangled factor learning
In summary, the paper is not ready for publication in its current form. The authors are advised to use the insights from their proposed SDR analysis to demonstrate quantifiable improvements the VAEs/GANs.


HYPERPARAMETER OPTIMIZATION: A SPECTRAL APPROACH
We give a simple, fast algorithm for hyperparameter optimization inspired by techniques from the analysis of Boolean functions. We focus on the high-dimensional regime where the canonical example is training a neural network with a large number of hyperparameters. The algorithm -an iterative application of compressed sensing techniques for orthogonal polynomials -requires only uniform sampling of the hyperparameters and is thus easily parallelizable. Experiments for training deep neural networks on Cifar-10 show that compared to state-of-the-art tools (e.g., Hyperband and Spearmint), our algorithm finds significantly improved solutions, in some cases better than what is attainable by handtuning. In terms of overall running time (i.e., time required to sample various settings of hyperparameters plus additional computation time), we are at least an order of magnitude faster than Hyperband and Bayesian Optimization. We also outperform Random Search 8×. Our method is inspired by provably-efficient algorithms for learning decision trees using the discrete Fourier transform. We obtain improved sample-complexty bounds for learning decision trees while matching state-of-the-art bounds on running time (polynomial and quasipolynomial, respectively).

The paper is about hyperparameter optimization, which is an important problem in deep learning due to the large number of hyperparameters in contemporary model architectures and optimization algorithms.
At a high-level, hyperparameter optimization (for the challenging case of discrete variables) can be seen as a black-box optimization problem where we have only access to a function evaluation oracle (but no gradients etc.). In the entirely unstructured case, there are strong lower bounds with an exponential dependence on the number of hyperparameters. In order to sidestep these impossibility results, the current paper assumes structure in the unknown function mapping hyperparameters to classification accuracy. In particular, the authors assume that the function admits a representation as a sparse and low-degree polynomial. While the authors do not empirically validate whether this is a good model of the unknown function, it appears to be a reasonable assumption (the authors *do* empirically validate their overall approach).
Based on the sparse and low-degree assumption, the paper introduces a new algorithm (called Harmonica) for hyperparameter optimization. The main idea is to leverage results from compressed sensing in order to recover the sparse and low-degree function from a small number of measurements (i.e., function evaluations). The authors derive relevant sample complexity results for their approach. Moreover, the method also yields new algorithms for learning decision trees.
In addition to the theoretical results , the authors conduct a detailed study of their algorithm on CIFAR10. They compare to relevant recent work in hyperparameter optimization (Bayesian optimization, random search, bandit algorithms) and find that their method significantly improves over prior work. The best parameters found by Harmonica improve over the hand-tuned results for their "base architecture" (ResNets).
Overall, I find the main idea of the paper very interesting and well executed, both on the theoretical and empirical side. Hence I strongly recommend accepting this paper.
Small comments and questions:
1. It would be interesting to see how close the hyperparameter function is to a low-degree and sparse polynomial (e.g., MSE of the best fit).
2. A comparison without dummy parameters would be interesting to investigate the performance differences between the algorithms in a lower-dimensional problem.
3. The current paper does not mention the related work on hyperparameter optimization using reinforcement learning techniques (e.g., Zoph & Le, ICLR 2017). While it might be hard to compare to this approach directly in experiments, it would still be good to mention this work and discuss how it relates to the current paper.
4. Did the authors tune the hyperparameters directly using the CIFAR10 test accuracy? Would it make sense to use a slightly smaller training set and to hold out say 5k images for hyperparameter evaluation before making the final accuracy evaluation on the test set? The current approach could be prone to overfitting.
5. While random search does not explicitly exploit any structure in the unknown function, it can still implicitly utilize smoothness or other benign properties of the hyperparameter space. It might be worth adding this in the discussion of the related work.
6. Algorithm 1: Why is the argmin for g_i  (what does the index i refer to)?
7. Why does PSR truncate the indices in alpha? At least in "standard" compressed sensing, the Lasso also has recovery guarantees without truncation (and empirically works sometimes better without).
9. Definition 3: Should C be a class of functions mapping {-1, 1}^n to R?  (Note the superscript.)
10. On Page 3 we assume that K = 1, but Theorem 6 still maintains a dependence on K. It might be cleaner to either treat the general K case throughout, or state the theorem for K = 1.
11. On CIFAR10, the best hyperparameters do not improve over the state of the art with other models (e.g., a wide ResNet). It could be interesting to run Harmonica in the regime where it might improve over the best known models for CIFAR10.
12. Similarly, it would be interesting to see whether the hyperparameters identified by Harmonica carry over to give better performance on ImageNet. The authors claim in C.3 that the hyperparameters identified by Harmonica generalize from small networks to large networks. Testing whether the hyperparameters also generalize from a smaller to a larger dataset would be relevant as well.


In this paper we introduce a new speech recognition system, leveraging a simple letter-based ConvNet acoustic model. The acoustic model requires only audio transcription for training -no alignment annotations, nor any forced alignment step is needed. At inference, our decoder takes only a word list and a language model, and is fed with letter scores from the acoustic model -no phonetic word lexicon is needed. Key ingredients for the acoustic model are Gated Linear Units and high dropout. We show near state-of-the-art results in word error rate on the LibriSpeech corpus (Panayotov et al., 2015) with MFSC features, both on the CLEAN and OTHER configurations.

This paper applies gated convolutional neural networks [1] to speech recognition, using the training criterion ASG [2]. It is fair to say that this paper contains almost no novelty.
This paper starts by bashing the complexity of conventional HMM systems, and states the benefits of their approach. However, all of the other grapheme-based end-to-end systems enjoy the same benefit as CTC and ASG. Prior work along this line includes [3, 4, 5, 6, 7].
Using MFSC, or more commonly known as log mel filter bank outputs, has been pretty common since [8]. Having a separate subsection (2.1) discussing this seems unnecessary.
Arguments in section 2.3 are weak because, again, all other grapheme-based end-to-end systems have the same benefit as CTC and ASG. It is unclear why discriminative training, such as MMI, sMBR, and lattice-free MMI, is mentioned in section 2.3. Discriminative training is not invented to overcome the lack of manual segmentations, and is equally applicable to the case where we have manual segmentations.
The authors argue that ASG is better than CTC in section 2.3.1 because it does not use the blank symbol and can be faster during decoding. However, once the transition scores are introduced in ASG, the search space becomes quadratic in the number of characters, while CTC is still linear in the number characters. In addition, ASG requires additional forward-backward computation for computing the partition function (second term in eq 3). There is no reason to believe that ASG can be faster than CTC in both training and decoding.
The connection between ASG, CTC, and marginal log loss has been addressed in [9], and it does make sense to train ASG with the partition function. Otherwise, the objective won't be a proper probability distribution.
The citation style in section 2.4 seems off. Also see [4] for a great description of how beam search is done in CTC.
Details about training, such as the optimizer, step size, and batch size, are missing. Does no batching (in section 3.2) means a batch size of one utterance?
In the last paragraph of section 3.2, why is there a huge difference in real-time factors between the clean and other set? Something is wrong unless the authors are using different beam widths in the two settings.
The paper can be significantly improved if the authors compare the performance and decoding speed against CTC with the same gated convnet. It would be even better to compare CTC and ASG to seq2seq-based models with the same gated convnet. Similar experiments should be conducted on switchboard and wsj because librespeech is several times larger than switchboard and wsj. None of the comparison in table 4 is really meaningful, because none of the other systems have parameters as many as 19 layers of convolution. Why does CTC fail when trained without the blanks? Is there a way to fix it besides using ASG? It is also unclear why speaker-adaptive training is not needed. At which layer do the features become speaker invariant? Can the system improve further if speaker-adaptive features are used instead of log mels? This paper would be much stronger if the authors can include these experiments and analyses.
[1] R Collobert, C Puhrsch, G Synnaeve, Wav2letter: an end-to-end convnet-based speech recognition system, 2016
[2] Y Dauphin, A Fan, M Auli, D Grangier, Language modeling with gated convolutional nets, 2017
[3] A Graves and N Jaitly, Towards End-to-End Speech Recognition with Recurrent Neural Networks, 2014
[4] A Maas, Z Xie, D Jurafsky, A Ng, Lexicon-Free Conversational Speech Recognition with Neural Networks, 2015
[5] Y Miao, M Gowayyed, F Metze, EESEN: End-to-end speech recognition using deep RNN models and WFST-based decoding, 2015
[6] D Bahdanau, J Chorowski, D Serdyuk, P Brakel, Y Bengio, End-to-end attention-based large vocabulary speech recognition, 2016
[7] W Chan, N Jaitly, Q Le, O Vinyals, Listen, attend and spell, 2015
[8] A Graves, A Mohamed, G Hinton, Speech recognition with deep recurrent neural networks, 2013
[9] H Tang, L Lu, L Kong, K Gimpel, K Livescu, C Dyer, N Smith, S Renals, End-to-End Neural Segmental Models for Speech Recognition, 2017


AN ONLINE LEARNING APPROACH TO GENERATIVE ADVERSARIAL NETWORKS
We consider the problem of training generative models with a Generative Adversarial Network (GAN). Although GANs can accurately model complex distributions, they are known to be difficult to train due to instabilities caused by a difficult minimax optimization problem. In this paper, we view the problem of training GANs as finding a mixed strategy in a zero-sum game. Building on ideas from online learning we propose a novel training method named CHEKHOV GAN 1 . On the theory side, we show that our method provably converges to an equilibrium for semi-shallow GAN architectures, i.e. architectures where the discriminator is a one-layer network and the generator is arbitrary. On the practical side, we develop an efficient heuristic guided by our theoretical results, which we apply to commonly used deep GAN architectures. On several real-world tasks our approach exhibits improved stability and performance compared to standard GAN training.

The paper applies tools from online learning to GANs. In the case of a shallow discriminator, the authors proved some results on the convergence of their proposed algorithm (an adaptation of FTRL) in GAN games, by leveraging the fact that when D update is small, the problem setup meets the ideal conditions for no-regret algorithms. The paper then takes the intuition from the semi-shallow case and propose a heuristic training procedure for deep GAN game.
Overall the paper is very well written. The theory is significant to the GAN literature, probably less so to the online learning community. In practice, with deep D, trained by single gradient update steps for G and D, instead of the "argmin" in Algo 1., the assumptions of the theory break. This is OK as long as sufficient experiment results verify that the intuitions suggested by the theory still qualitatively hold true. However, this is where I have issues with the work:
1) In all quantitative results, Chekhov GAN do not significantly beat unrolled GAN. Unrolled GAN looks at historical D's through unrolled optimization, but not the history of G. So this lack of significant difference in results raise the question of whether any improvement of Chekhov GAN is coming from the online learning perspective for D and G, or simply due to the fact that it considers historical D models (which could be motivated by sth other than the online learning theory).
2) The mixture GAN approach suggested in Arora et al. (2017) is very related to this work, as acknowledged in Sec. 2.1, but no in-depth analysis is carried out. I suggest the authors to either discuss why Chekhov GAN is obviously superior and hence no experiments are needed, or compare them experimentally.
3) In the current state, it is hard to place the quantitative results in context with other common methods in the recent literature such as WGAN with gradient penalty. I suggest the authors to either report some results in terms of inception scores on cifar10 with similar architectures used in other methods for comparison. Alternatively please show WGAN-GP and/or other method results in at least one or two experiments using the evaluation methods in the paper.
In summary, almost all the experiments in the paper are trying to establish improvement over basic GAN, which would be OK if the gap between theory and practice is small. But in this case, it is not. So it is not entirely convincing that the practical Algo 2 works better for the reason suggested by the theory, nor it drastically improves practical results that it could become the standard technique in the literature.


We present Merged-Averaged Classifiers via Hashing (MACH) for Kclassification with large K. Compared to traditional one-vs-all classifiers that require O(Kd) memory and inference cost, MACH only need O(d log K) memory while only requiring O(K log K +d log K) operation for inference. MACH is the first generic K-classification algorithm, with provably theoretical guarantees, which requires O(log K) memory without any assumption on the relationship between classes. MACH uses universal hashing to reduce classification with a large number of classes to few independent classification task with small (constant) number of classes. We provide theoretical quantification of accuracy-memory tradeoff by showing the first connection between extreme classification and heavy hitters. With MACH we can train ODP dataset with 100,000 classes and 400,000 features on a single Titan X GPU (12GB), with the classification accuracy of 19.28%, which is the best-reported accuracy on this dataset. Before this work, the best performing baseline is a one-vs-all classifier that requires 40 billion parameters (160 GB model size) and achieves 9% accuracy. In contrast, MACH can achieve 9% accuracy with 480x reduction in the model size (of mere 0.3GB). With MACH, we also demonstrate complete training of fine-grained imagenet dataset (compressed size 104GB), with 21,000 classes, on a single GPU.

Thanks to the authors for their feedback.
==============================
The paper presents a method for classification scheme for problems involving large number of classes in multi-class setting. This is related to the theme of extreme classification but the setting is restricted to that of multi-class classification instead of multi-label classification. The training process involves data transformation using R hash functions, and then learning R classifiers. During prediction the probability of a test instance belonging to a class is given by the sum of the probabilities assigned by the R meta-classifiers to the meta-class in the which the given class label falls. The paper demonstrates better results on ODP and Imagenet-21K datasets compared to LOMTree, RecallTree and OAA.
There are following concerns regarding the paper which don't seem to be adequately addressed :

 - The paper seems to propose a method in which two-step trees are being constructed based on random binning of labels, such that the first level has B nodes. It is not intuitively clear why such a method could be better in terms of prediction accuracy than OAA. The authors mention algorithms for training and prediction, and go on to mention that the method performs better than OAA. Also, please refer to point 2 below.
 - The paper repeatedly mentions that OAA has O(Kd) storage and prediction complexity. This is however not entirely true due to sparsity of training data, and the model. These statements seem quite misleading especially in the context of text datasets such as ODP. The authors are requested to check the papers [1] and [2], in which it is shown that OAA can perform surprisingly well. Also, exploiting the sparsity in the data/models, actual model sizes for WikiLSHTC-325K from [3] can be reduced from around 900GB to less than 10GB with weight pruning, and sparsity inducing regularizers. It is not clear if the 160GB model size reported for ODP took the above suggestions into considerations, and which kind of regularization was used. Was the solver used from vowpal wabbit or packages such as Liblinear were used for reporting OAA results.
 - Lack of empirical comparison - The paper lacks empirical comparisons especially on large-scale multi-class LSHTC-1/2/3 datasets [4] on which many approaches have been proposed. For a fair comparison, the proposed method must be compared against these datasets. It would be important to clarify if the method can be used on multi-label datasets or not, if so, it needs to be evaluated on the XML datasets [3].
[1] PPDSparse - http://www.kdd.org/kdd2017/papers/view/a-parallel-and-primal-dual-sparse-method-for-extreme-classification
[2] DiSMEC - https://arxiv.org/abs/1609.02521
[3] http://manikvarma.org/downloads/XC/XMLRepository.html
[4] http://lshtc.iit.demokritos.gr/LSHTC2_CFP


Amortized inference has led to efficient approximate inference for large datasets. The quality of posterior inference is largely determined by two factors: a) the ability of the variational distribution to model the true posterior and b) the capacity of the recognition network to generalize inference over all datapoints. We analyze approximate inference in variational autoencoders in terms of these factors. We find that suboptimal inference is often due to amortizing inference rather than the limited complexity of the approximating distribution. We show that this is due partly to the generator learning to accommodate the choice of approximation. Furthermore, we show that the parameters used to increase the expressiveness of the approximation play a role in generalizing inference rather than simply improving the complexity of the approximation.

=======
Update:
The new version addresses some of my concerns. I think this paper is still pretty borderline, but I increased my rating to a 6.
=======
This article examines the two sources of loose bounds in variational autoencoders, which the authors term “approximation error” (slack due to using a limited variational family) and “amortization error” (slack due to the inference network not finding the optimal member of that family).
The existence of amortization error is often ignored in the literature, but (as the authors point out) it is not negligible. It has been pointed out before in various ways, however:
* Hjelm et al. (2015; https://arxiv.org/pdf/1511.06382.pdf) observe it for directed belief networks (admittedly a different model class).
* The ladder VAE paper by Sonderby et al. (2016, https://arxiv.org/pdf/1602.02282.pdf) uses an architecture that reduces the work that the encoder network needs to do, without increasing the expressiveness of the variational approximation. That this approach works well implies that amortization error cannot be ignored.
* The structured VAE paper by Johnson et al. (2016, https://arxiv.org/abs/1603.06277) also proposes an architecture that reduces the load on the inference network.
* The very recent paper by Krishnan et al. (posted to arXiv days before the ICLR deadline, although a workshop version was presented at the NIPS AABI workshop last year; http://approximateinference.org/2016/accepted/KrishnanHoffman2016.pdf) examines amortization error as a core cause of training failures in VAEs. They also observe that the gap persists at test time, although it does not examine how it relates to approximation error.
Since these earlier results existed, and approximation-amortization decomposition is fairly simple (although important!), the main contributions of this paper are the empirical studies. I will try to summarize the main novel (i.e., not present elsewhere in the literature) results of these:
Section 5.1:
Inference networks with FFG approximations can produce qualitatively embarrassing approximations.
Section 5.2:
When trained on a small dataset, training amortization error becomes negligible. I found this surprising, since it’s not at all clear why dataset size should lead to “strong inference”. It seems like a more likely explanation is that the decoder doesn’t have to work as hard to memorize the training set, so it has some extra freedom to make the true posterior look more like a FFG.
Also, I think it’s a bit of an exaggeration to call a gap of 2.71 nats “much tighter” than a gap of 3.01 nats.
Section 5.3:
Amortization error is an important contributor to the slack in the ELBO on MNIST, and the dominant contributor on the more complicated Fashion MNIST dataset. (This is totally consistent with Krishnan et al.’s finding that eliminating amortization error gave a bigger improvement for more complex datasets than for MNIST.)
Section 5.4:
Using a restricted variational family causes the decoder to learn to induce posteriors that are easier to approximate with that variational family. This idea has been around for a long time (although I’m having a hard time coming up with a reference).
These results are interesting, but given the empirical nature of this paper I would have liked to see results on more interesting datasets (Celeb-A, CIFAR-10, really anything but MNIST). Also, it seems as though none of the full-dataset MNIST models have been trained to convergence, which makes it a bit difficult to interpret some results.
A few more specific comments:
2.2.1: The \cdot seems extraneous to me.
5.1: What dataset/model was this experiment done on?
Figure 3: This can be inferred from the text (I think), but I had to remind myself that “IW train” and “IW test” refer only to the evaluation procedure, not the training procedure. It might be good to emphasize that you don’t train on the IWAE bound in any experiments.
Table 2: It would be good to see standard errors on these numbers; they may be quite high given that they’re only evaluated on 100 examples.
“We can quantitatively determine how close the posterior is to a FFG distribution by comparing the Optimal FFG bound and the Optimal Flow bound.”: Why not just compare the optimal with the AIS evaluation? If you trust the AIS estimate, then the result will be the actual KL divergence between the FFG and the true posterior.


Machine learning and in particular deep learning approaches have outperformed many traditional techniques in accomplishing complex tasks such as image classfication (Krizhevsky et al., 2012), natural language processing or speech recognition . Most of the state-of-the art deep networks have complex architecture and use a vast number of parameters to reach this superior performance. Though these networks use a large number of learnable parameters, those parameters present significant redundancy (de Freitas, 2013). Therefore, it is possible to compress the network without much affecting its accuracy by eliminating those redundant and unimportant parameters. In this work, we propose a three stage compression pipeline, which consists of pruning, weight sharing and quantization to compress deep neural networks. Our novel pruning technique combines magnitude based ones with dense sparse dense Han et al. (2016) ideas and iteratively finds for each layer its achievable sparsity instead of selecting a single threshold for the whole network. Unlike previous works, where compression is only applied on networks performing classification, we evaluate and perform compression on networks for classification as well as semantic segmentation, which is greatly useful for understanding scenes in autonomous driving. We tested our method on LeNet-5 and FCNs, performing classification and semantic segmentation, respectively. With LeNet-5 on MNIST, pruning reduces the number of parameters by 15.3 times and storage requirement from 1.7 MB to 0.006 MB with accuracy loss of 0.03%. With FCN8 on Cityscapes, we decrease the number of parameters by 8 times and reduce the storage requirement from 537.47 MB to 18.23 MB with class-wise intersection-over-union (IoU) loss of 4.93% on the validation data.

quality: this paper is of good quality
clarity: this paper is very clear
originality: this paper combines original ideas with existing approaches for pruning to obtain dramatic space reduction in NN parameters.
significance: this paper seems significant.
PROS
- a new approach to sparsifying that considers different thresholds for each layer
- a systematic, empirical method to obtain optimal sparsity levels for a given neural network on a task.
- Very interesting and extensive experiments that validate the reasoning behind the described approach, with a detailed analysis of each step of the algorithm.
CONS
- Pruning time. Although the authors argue that the pruning algorithm is not prohibitive, I would argue that >1 month to prune LeNet-5 for MNIST is certainly daunting in many settings. It would benefit the experimental section to use another dataset than MNIST (e.g. CIFAR-10) for the image recognition experiment.
- It is unclear whether this approach will always work well; for some neural nets, the currently used sparsification method (thresholding) may not perform well, leading to very little final sparsification to maintain good performance.
- The search for the optimal sparsity in each level seems akin to a brute-force search. Although possibly inevitable, it would be valuable to discuss whether or not this approach can be refined.
Main questions
- You mention removing "unimportant and redundant weights" in the pruning step; in this case, do unimportant and redundant have the same meaning (smaller than a given threshold), or does redundancy have another meaning (e.g. (Mariet, Sra, 2016))?
- Algorithm 1 finds the best sparsity for a given layer that maintains a certain accuracy. Have you tried using a binary search for the best sparsity instead of simply decreasing the sparsity by 1% at each step? If there is a simple correlation between sparsity and accuracy, that might be faster; if there isn't (which would be believable given the complexity of neural nets), it would be valuable to confirm this with an experiment.
- Have you tried other pruning methods than thresholding to decide on the optimal sparsity in each layer?
- Could you please report the final accuracy of both models in Table 2?
Nitpicks:
- paragraph break in page 4 would be helpful.


Published as a conference paper at ICLR 2018 DO GANS LEARN THE DISTRIBUTION? SOME THEORY AND EMPIRICS

This paper proposes a clever new test based on the birthday paradox for measuring diversity in generated samples. The main goal is to quantify mode collapse in state-of-the-art generative models. The authors also provide a specific theoretical construction that shows bidirectional GANs cannot escape specific cases of mode collapse.
Using the birthday paradox test, the experiments show that GANs can learn and consistently reproduce the same examples, which are not necessarily exactly the same as training data (eg. the triplets in Figure 1).
The results are interpreted to mean that mode collapse is strong in a number of state-of-the-art generative models.
Bidirectional models (ALI, BiGANs) however demonstrate significantly higher diversity that DCGANs and MIX+DCGANs.
Finally, the authors verify empirically the hypothesis that diversity grows linearly with the size of the discriminator.
This is a very interesting area and exciting work. The main idea behind the proposed test is very insightful. The main theoretical contribution stimulates and motivates much needed further research in the area. In my opinion both contributions suffer from some significant limitations. However, given how little we know about the behavior of modern generative models, it is a good step in the right direction.
1. The biggest issue with the proposed test is that it conflates mode collapse with non-uniformity. The authors do mention this issue, but do not put much effort into evaluating its implications in practice, or parsing Theorems 1 and 2. My current understanding is that, in practice, when the birthday paradox test gives a collision I have no way of knowing whether it happened because my data distribution is modal, or because my generative model has bad diversity. Anecdotally, real-life distributions are far from uniform, so this should be a common issue. I would still use the test as a part of a suite of measurements, but I would not solely rely on it. I feel that the authors should give a more prominent disclaimer to potential users of the test.
2. Also, given how mode collapse is the main concern, it seems to me that a discussion on coverage is missing. The proposed test is a measure of diversity, not coverage, so it does not discriminate between a generator that produces all of its samples near some mode and another that draws samples from all modes of the true data distribution. As long as they yield collisions at the same rate, these two generative models are ‘equally diverse’. Isn’t coverage of equal importance?
3. The other main contribution of the paper is Theorem 3, which shows—via a very particular construction on the generator and encoder—that bidirectional GANs can also suffer from serious mode collapse. I welcome and are grateful for any theory in the area. This theorem might very well capture the underlying behavior of bidirectional GANs, however, being constructive, it guarantees nothing in practice. In light of this, the statement in the introduction that “encoder-decoder training objectives cannot avoid mode collapse” might need to be qualified. In particular, the current statement seems to obfuscate the understanding that training such an objective would typically not result into the construction of Theorem 3.


Hierarchical Bayesian methods have the potential to unify many related tasks (e.g. k-shot classification, conditional, and unconditional generation) by framing each as inference within a single generative model. We show that existing approaches for learning such models can fail on expressive generative networks such as PixelCNNs, by describing the global distribution with little reliance on latent variables. To address this, we develop a modification of the Variational Autoencoder in which encoded observations are decoded to new elements from the same class; the result, which we call a Variational Homoencoder (VHE), may be understood as training a hierarchical latent variable model which better utilises latent variables in these cases. Using this framework enables us to train a hierarchical PixelCNN for the Omniglot dataset, outperforming all existing models on test set likelihood. With a single model we achieve both strong one-shot generation and near human-level classification, competitive with state-of-the-art discriminative classifiers. The VHE objective extends naturally to richer dataset structures such as factorial or hierarchical categories, as we illustrate by training models to separate character content from simple variations in drawing style, and to generalise the style of an alphabet to new characters.

- Good work on developing VAEs for few-shot learning.
- Most of the results are qualitative and I reckon the paper was written in haste.
- The rest of the comments are below:
- 3.1: I got a bit confused over what X actually is:
 -- "We would like to learn a generative model for **sets X** of the form".
 --"... to refer to the **class X_i** ...".
 -- "we can lower bound the log-likelihood of each **dataset X** ..."
- 3.2: "In general, if we wish to learn a model for X in which each latent variable ci affects some arbitrary subset Xi of the data (**where the Xi may overlap**), ...": Which is just like learning a Z for a labeled X but learning it in an unsupervised manner, i.e. the normal VAE, isn't it? If not, could you please elaborate on what is different (in the case of 3.2 only, I mean)? i.e. Could you please elaborate on what's different (in terms of learning) between 3.2 and a normal latent Z that is definitely allowed to affect different classes of the data without knowing the classes?
- Figure 1 is helpful to clarify the main idea of a VHE.
- "In a VHE, this recognition network takes only small subsets of a class as input, which additionally ...": And that also clearly leads to loss of information that could have been used in learning. So there is a possibility for potential regularization but there is definitely a big loss in estimation power. This is obviously possible with any regularization technique, but I think it is more of an issue here since parts of the data are not even used in learning.
- "Table 4.1 compares these log likelihoods, with VHE achieving state-of-the-art. To": Where is Table 4.1??
- This is a minor point and did not have any impact on the evaluation but VAE --> VHE, reparameterization trick --> resampling trick. Maybe providing rather original headings is better? It's a style issue that is up to tastes anyway so, again, it is minor.
- "However, sharing latent variables across an entire class reduces the encoding cost per element is significantly": typo.
- "Figure ?? illustrates".


Published as a conference paper at ICLR 2018 NOT-SO-RANDOM FEATURES
We propose a principled method for kernel learning, which relies on a Fourieranalytic characterization of translation-invariant or rotation-invariant kernels. Our method produces a sequence of feature maps, iteratively refining the SVM margin. We provide rigorous guarantees for optimality and generalization, interpreting our algorithm as online equilibrium-finding dynamics in a certain two-player min-max game. Evaluations on synthetic and real-world datasets demonstrate scalability and consistent improvements over related random features-based methods.

In this paper the authors consider learning directly Fourier representations of shift/translation invariant kernels for machine learning applications. They choose the alignment of the kernel to data as the objective function to optimize. They empirically verify that the features they learned lead to good quality SVM classifiers. My problem with that paper is that even though at first glance learning adaptive feature maps seems to be an attractive approach, authors' contribution is actually very little. Below I list some of the key problems. First of all the authors claim in the introduction that their algorithm is very fast and with provable theoretical guarantees. But in fact later they admit that the problem of optimizing the alignment is a non-convex problem and the authors end up with a couple of heuristics to deal with it. They do not really provide any substantial theoretical justification why these heuristics work in practice even though they observe it empirically. The assumptions that large Fourier peaks happen close to origin is probably well-justified from the empirical point of view, but it is a hack, not a well established well-grounded theoretical method (the authors claim that in their experiments they found it easy to find informative peaks, even in hundreds of dimensions, but these experiments are limited to the SVM setting, I have no idea how these empirical findings would translate to other kernelized algorithms using these adaptive features).  The Langevin dynamics algorithm used by the authors to find the peaks (where the gradient is available) gives only weak theoretical guarantees (as the authors actually admit) and this is a well known method, certainly not a novelty of that paper. Finally, the authors notice that "In the rotation-invariant case, where Ω is a discrete set, heuristics are available". That is really not very informative (the authors refer to the Appendix so I carefully read that part of the Appendix, but it is extremely vague, it is not clear at all how the Langevin dynamics can be "emulated" by a discrete Markov chain that mixes fast; the authors do not provide any justification of that approach, what is the mixing time ?; how the "good emulation property" is exactly measured ?).  In the conclusions the authors admit that: "Many theoretical questions remain, such as accelerating the search for Fourier peaks". I think that the problem of accelerating this approach is a critical point that this publication is missing. Without this, it is actually really hard to talk about general mechanism of learning adaptive Fourier features for kernel algorithms (which is how the authors present their contribution); instead we have a method heavily customized and well-tailored to the (not particularly exciting) SVM scenario (with optimization performed by the standard annealing method; it is not clear at all whether for other downstream kernel applications this approach for optimizing the alignment would provide good quality models) that uses lots of task specific hacks and heuristics to efficiently optimize the alignment. Another problem is that it is not clear at all to me how authors' approach can be extended to non shift-invariant kernels that do not benefit from Bochner's Theorem. Such kernels are very related to neural networks (for instance PNG kernels with linear rectifier nonlinearities correspond to random layers in NNs with ReLU) and in the NN context are much more interesting that radial basis function or in general shift-invariant kernels. A general kernel method should address this issue (the authors just claim in the conclusions that it would be interesting to explore the NN context in more detail).
To sum it up, it is a solid submission, but in my opinion without a substantial contribution and working only in a  very limited setting when it is heavily relying on many unproven hacks and heuristics.


Published as a conference paper at ICLR 2018 UNIVERSAL AGENT FOR DISENTANGLING ENVIRONMENTS AND TASKS
Recent state-of-the-art reinforcement learning algorithms are trained under the goal of excelling in one specific task. Hence, both environment and task specific knowledge are entangled into one framework. However, there are often scenarios where the environment (e.g. the physical world) is fixed while only the target task changes. Hence, borrowing the idea from hierarchical reinforcement learning, we propose a framework that disentangles task and environment specific knowledge by separating them into two units. The environment-specific unit handles how to move from one state to the target state; and the task-specific unit plans for the next target state given a specific task. The extensive results in simulators indicate that our method can efficiently separate and learn two independent units, and also adapt to a new task more efficiently than the state-of-the-art methods.

The authors propose to decompose reinforcement learning into a PATH function that can learn how to solve reusable sub-goals an agent might have in a specific environment and a GOAL function that chooses subgoals in order to solve a specific task in the environment using path segments. So I guess it can be thought of as a kind of hierarchical RL.
The exposition of the model architecture could use some additional detail to clarify some steps and possibly fix some minor errors (see below). I would prefer less material but better explained. I had to read a lot of sections more than once and use details across sections to fill in gaps. The paper could be more focused around a single scientific question: does the PATH function as formulated help?
The authors do provide a novel formulation and demonstrate the gains on a variety of concrete problems taken form the literature. I also like that they try to design experiments to understand the role of specific parts of the proposed architecture.
The graphs are WAY TOO SMALL to read. Figure #s are missing off several figures.
MODEL & ARCHITECTURE
The PATH function given a current state s and a goal state s', returns a distribution over the best first action to take to get to the goal P(A).  ( If the goal state s’ was just the next state, then this would just be a dynamics model and this would be model-based learning? So I assume there are multiple steps between s and s’?).
At the beginning of section 2.1, I think the authors suggest the PATH function could be pre-trained independently by sampling a random state in the state space to be the initial state and a second random state to be the goal state and then using an RL algorithm to find a path.
Presumably, once one had found a path ( (s, a0), (s1, a1), (s2, a2), …, (sn-1,an-1),  s’ ) one could then train the PATH policy on the triple (s, s’, a0) ? This seems like a pretty intense process: solving some representative subset of all possible RL problems for a particular environment … Maybe one choses s and s’ so they are not too far away from each other (the experimental section later confirms this distance is >= 7. Maybe bring this detail forward)?
The expression Trans’( (s,s’), a) = (Trans(s,a), s’) was confusing.  I think the idea here is that the expression
Trans’(  (s,s’) , a ) represents the n-step transition function and ‘a' represents the first action?
The second step is to train the goal function for a specific task. So I gather our policy takes the form of a composed function and the chain rule gives close to their expression in 2.2
    PATH(  s,  Tau( s, th^g ),  a ; th^p )
    d / { d th^g }  PATH(  s,  Tau( s, th^g ),  a ; th^p )
         =  {d / d {s’ }  PATH } ( s,  Tau( s, th^g ),  a )    d / {d th^g}  Tau( s, th^g)
What is confusing is that they define
    A( s, a, th^p, th^g, th^v ) = sum_i   gamma^i  r_{t+1}  +  gamma^k  V(  s_{t+k}  ;  th^v  )   -   V( s_t ;  th^v )
The left side contains th^p and th^g, but the right side does not. Should these parameters be take out of the n-step advantage function A?
The second alternative for training the goal function tau seems confusing. I get that tau is going to be constrained by whatever representation PATH function was trained on and that this representation might affect the overall performance - performance.  I didn’t get the contrast with method one.  How do we treat the output of Tau as an action? Are you thinking of the gradient coming back through PATH as a reward signal? More detail here would be helpful.
EXPERIMENTS:
Lavaworld: authors show that pretraining the PATH function on longer 7-11 step policies leads to better performance
when given a specific Lava world problem to solve.  So the PATH function helps and longer paths are better. This seems reasonable. What is the upper bound on the size of PATH lengths you can train?
Reachability: authors show that different ways of abstracting the state s into a vector encoding affect the performance of the system.  From a scientific point of view, this seems orthogonal to the point of the paper, though is relevant if you were trying to build a system.
Taxi: the authors train the PATH problem on reachability and then show that it works for TAXI. This isn’t too surprising. Both picking up the passenger (reachability) and dropping them off somewhere are essentially the same task: moving to a point. It is interesting that the Task function is able to encode the higher level structure of the TAXI problem’s two phases.
Another task you could try is to learn to perform the same task in two different environments. Perhaps the TAXI problem, but you have two different taxis that require different actions in order to execute the same path in state space. This would require a phi(s) function that is trained in a way that doesn’t depend on the action a.
ATARI 2600 games: I am not sure what state restoration is.  Is this where you artificially return an agent to a state that would normally be hard to reach? The authors show that UA results in gains on several of the games.
The authors also demonstrate that using multiple agents with different policies can be used to collect training examples for the PATH function that improve its utility over training examples collected by a single agent policy.
RELATED WORK:
Good contrast to hierarchical learning: we don’t have switching regimes here between high-level options
I don’t understand why the authors say the PATH function can be viewed as an inverse? Oh - now I get it.
Because it takes an extended n-step transition and generates an action.


Deep learning algorithms are increasingly used in modeling chemical processes. However, black box predictions without rationales have limited used in practical applications, such as drug design. To this end, we learn to identify molecular substructures -rationales -that are associated with the target chemical property (e.g., toxicity). The rationales are learned in an unsupervised fashion, requiring no additional information beyond the end-to-end task. We formulate this problem as a reinforcement learning problem over the molecular graph, parametrized by two convolution networks corresponding to the rationale selection and prediction based on it, where the latter induces the reward function. We evaluate the approach on two benchmark toxicity datasets. We demonstrate that our model sustains high performance under the additional constraint that predictions strictly follow the rationales. Additionally, we validate the extracted rationales through comparison against those described in chemical literature and through synthetic experiments.

This paper presents an interesting approach to identify substructural features of molecular graphs contributing to the target task (e.g. predicting toxicity). The algorithm first builds two conv nets for molecular graphs, one is for searching relevant substructures (policy improvement), and another for evaluating the contribution of selected substructures to the target task (policy evaluation). These two phases are iterated in a reinforcement learning manner as policy iterations. Both parts are based on conv nets for molecular graphs, and this framework is a kind of 'self-supervised' scheme compared to the standard situations that the environment provides rewards. The experimental validations demonstrate that this model can learn a competitive-performed conv nets only dependent on the highlighted substructures, as well as reporting some case study on the inhibition assay for hERG proteins.
Technically speaking, the proposed self-supervised scheme with two conv nets is very interesting. This demonstrates how we can perform progressive substructure selections over molecular graphs to highlight relevant substructures as well as maximizing the prediction performance. Given that conv nets for molecular graphs are not trivially interpretable, this would provides a useful approach to use conv nets for more explicit interpretations of how the task can be performed by neural nets.
However, at the same time, I had one big question about the purpose and usage of this approach. As the paper states in Introduction, the target problem is 'hard selection' of substructures, rather than 'soft selection' that neural nets (with attention, for example) or neural-net fingerprints usually provide. Then, the problem would become a combinatorial search problem, which has been long studied in the data mining and machine learning community. There would exist many exact methods such as LEAP, CORK, and graphSig under the name of 'contrast/emerging/discriminative' pattern mining exactly developed for this task. Also, it is widely known that we can even perform a wrapper approach for supervised learning from graphs simultaneously with searching all relevant subgraphs as seen in Kudo+ NIPS 2004, Tsuda ICML 2007, Saigo+ Machine Learning 2009, etc. It would be unconvincing that the proposed neural nets approach fits to this hard combinatorial task rather than these existing (mostly exact) methods.
In addition to the above point, several technical points below would also be unclear.
- A simple heuristic by adding 'selected or not' variables to the atom features works as intended? Because this is fed to the conv net, it seems we can ignore this elements of features by tweaking the weight parameters accordingly. If the conv net performs the best when we use the entire structure, then learning might be forced to ignore the selection. Can we guarantee in some sense this would not happen?
- Zeroing out the atom features also sounds quite simple and a bit groundless. Confusingly, the P network also has an attention mechanism, and it is a bit unclear to me what was actually worked.
- In the experiments, the baseline is based on LR, but this would not be fair because usually we cannot expect any linear relationship for molecular fingerprints. It's highly correlated due to the inclusion relationships between subgraphs. At least, any nonlinear baseline (e.g. Random forest or something?) should be presented for discussing the results.
Pros:
- interesting self-supervised framework provided for highlighting relevant substructures for a given prediction task
- the hard selection setting is encoded in input graph featurization
Cons:
- it would be a bit unconvincing that identifying 'hard selection' is better suited for neural nets, rather than many existing exact methods (without using neural networks). At least one of the typical ones should be compared or discussed.
- I'm still not quite sure whether or not some heuristic parts work as intended.


We introduce a new deep convolutional neural network, CrescendoNet, by stacking simple building blocks without residual connections. Each Crescendo block contains independent convolution paths with increased depths. The numbers of convolution layers and parameters are only increased linearly in Crescendo blocks. In experiments, CrescendoNet with only 15 layers outperforms almost all networks without residual connections on benchmark datasets, CIFAR10, CIFAR100, and SVHN. Given sufficient amount of data as in SVHN dataset, CrescendoNet with 15 layers and 4.1M parameters can match the performance of DenseNet-BC with 250 layers and 15.3M parameters. CrescendoNet provides a new way to construct high performance deep convolutional neural networks with simple network architecture. Moreover, through investigating the behavior and performance of subnetworks in CrescendoNet, we note that the high performance of CrescendoNet may come from its implicit ensemble behavior. Furthermore, the independence between paths in CrescendoNet allows us to introduce a new path-wise training procedure, which can reduce the memory needed for training.

The paper presents a new CNN architecture: CrescendoNet. It does not have skip connections yet performs quite well.
Overall, I think the contributions of this paper are too marginal for acceptance in a top tier conference.
The architecture is competitive on SVHN and CIFAR 10 but not on CIFAR 100. The performance is not strong enough to warrant acceptance by itself.
FractalNets amd DiracNets (https://arxiv.org/pdf/1706.00388.pdf) have demonstrated that it is possible to train deep networks without skip connections and achieve high performance. While CrescendoNet seems to slightly outperform FractalNet in the experiments conducted, it is itself outperformed by DiracNet. Hence, CrescendoNet does not have the best performance among skip connection free networks.
You claim that FractalNet shows no ensemble behavior. This is clearly not true because FractalNet has ensembling directly built in, i.e. different paths in the network are explicitly averaged. If averaging paths leads to ensembling in CrescendoNet, it leads to ensembling in FractalNet. While the longest path in FractalNet is stronger than the other members of the ensemble, it is nevertheless an ensemble. Besides, as Veit showed, ResNet also shows ensemble behavior. Hence, using ensembling in deep networks is not a significant contribution.
The authors claim that "Through our analysis and experiments, we note that the implicit ensemble behavior of CrescendoNet leads to high performance". I don't think the experiments show that ensemble behavior leads to high performance. Just because a network performs averaging of different paths and individual paths perform worse than sets of paths doesn't imply that ensembling as a mechanism is in fact the cause of the performance of the entire architecture. Similary, you say "On the other hand, the ensemble model can explain the performance improvement easily." Veit et al only claimed that ensembling is a feature of ResNet, but they did not claim that this was the cause of the performance of ResNet.
Path-wise training is not original enough or indeed different enough from drop-path to count as a major contribution.
You claim that the number of layers "increase exponentially" in FractalNet. This is misleading. The number of layers increases exponentially in the number of paths, but not in the depth of the network. In fact, the number of layers is linear in the depth of the network. Since depth is the meaningful quantity here, CrescendoNet does not have an advantage over FractalNet in terms of layer number. Also, it is always possible to simply add more paths to FractalNet if desired without increasing depth. Instead of using 1 long paths, one can simply use 2, 3, 4 etc. While this is not explicitly mentioned in the FractalNet paper, it clearly would not break the design principle of FractalNet which is to train a path of multiple layers by ensembling it with a path of fewer layers. CrescendoNets do not extend beyond this design principle.
You say that "First, path-wise training procedure significantly reduces the memory requirements for convolutional layers, which constitutes the major memory cost for training CNNs. For example, the higher bound of the memory required can be reduced to about 40% for a Crescendo block with 4 paths where interval = 1." This is misleading, as you need to store the weights of all convolutional layers to compute the forward pass and the majority of the weights of all convolutional layers to compute the backward pass, no matter how many weights you intend to update. In a response to a question I posed, you mentioned that we you meant was "we use about 40% memory for the gradient computation and storage". Fair enough, but "gradient computation and storage" is not mentioned in the paper. Also, the reduction to 40% does not apply e.g. to vanilla SGD because the computed gradient can be immediately added to the weights and does not need to be stored or combined with e.g. a stored momentum term.
Finally, nowhere in the paper do you mention which nonlinearities you used or if you used any at all. In future revisions, this should be rectified.
While I can definitely imagine that your network architecture is well-designed and a good choice for image classification tasks, there is a very saturated market of papers proposing various architectures for CIFAR-10 and related datasets. To be accepted to ICLR, either outstanding performance or truly novel design principles are required.


This work provides an automatic machine learning (AutoML) modelling architecture called Autostacker. Autostacker improves the prediction accuracy of machine learning baselines by utilizing an innovative hierarchical stacking architecture and an efficient parameter search algorithm. Neither prior domain knowledge about the data nor feature preprocessing is needed. We significantly reduce the time of AutoML with a naturally inspired algorithm -Parallel Hill Climbing (PHC). By parallelizing PHC, Autostacker can provide candidate pipelines with sufficient prediction accuracy within a short amount of time. These pipelines can be used as is or as a starting point for human experts to build on. By focusing on the modelling process, Autostacker breaks the tradition of following fixed order pipelines by exploring not only single model pipeline but also innovative combinations and structures. As we will show in the experiment section, Autostacker achieves significantly better performance both in terms of test accuracy and time cost comparing with human initial trials and recent popular AutoML system.

The authors introduce a simple hill climbing approach to (very roughly) search in the space of cascades of classifiers.
They first reinvent the concept of cascades of classifiers as an extension of stacking (https://en.wikipedia.org/wiki/Cascading_classifiers). Cascading is like stacking but carries over all original model inputs to the next classifier.
The authors cast this nicely into a network view with nodes that are classifiers and layers that use the outputs from previous layers. However, other than relating this line of work to the ICLR community, this interpretation of cascading is not put to any use.
The paper incorrectly claims that existing AutoML frameworks only allow using a specific single model. In fact, Auto-sklearn (Feurer et al, 2015) automatically constructs ensembles of up to 50 models, helping it to achieve more robust performance.
I have some questions about the hillclimbing approach:
- How is the "one change" implemented in the hill climber? Does this evaluate results for each of several single changes and pick the best one? Or does it simply change one classifier and continue? Or does it evaluate all possible individual changes and pick the best one? I note that the term "HillClimber" would suggest that some sort of improvement has to be made in each step, but the algorithm description does not show any evaluation step at this point. The hill climbing described in the text seems to make sense, but the pseudocode appears broken.
Section 4.2: I am surprised that there is only a comparison to TPOT, not one to Auto-sklearn. Especially since Auto-sklearn constructs ensembles posthoc this would be an interesting comparison.
As the maximum range of number of layers is 5, I assume that scaling is actually an issue in practice after all, and the use of hundreds of primitive models alluded to in the introduction are not a reality at this point.
The paper mentions guarantees twice:
- "This kind of guarantee of not being worse on average comes from the the characteristic of AutoStacked"
- "can be guaranteed to do better on average"
I am confident that this is a mistake / an error in choosing the right expression in English. I cannot see why there should be a guarantee of any sort.
Empirically, Autostacker appears better than RandomForest, but that is not a big feat. The improvements vs. TPOT are more relevant. One question: the data sets used in Olson et al are very small. Does TPOT overfit on these? Since AutoStacker does not search as exhaustively, could this explain part of the performance difference? How many models are evaluated in total by each of the methods?
I am unsure about the domain for the HillClimber. Does it also a search over which classifiers are used where in the pipeline, or only about their hyperparameters?
Minor issues:
- The authors systematically use citations wrongly, apparently never using citep but only having inline citations.
- Some parts of the paper feel unscientific, such as using phrases like "giant possible search space".
- There are also several English grammar mistakes (e.g., see the paragraph containing "for the discover") and typos.
- Why exactly would a small amount of data be more likely to be unbalanced?
- The data "cleaning" method of throwing out data with missing values is very unclean. I hope this has only been applied to the training set and that no test set data points have been dropped?
- Line 27 of Algorithm 1: sel_pip has not been defined here
Overall, this is an interesting line of work, but it does not seem quite ready for publication.
Pros:
- AutoML is a topic of high importance to both academia and industry
- Good empirical results
Cons:
- Cascading is not new
- Unclear algorithm: what exactly does the Hillclimber function do?
- Missing baseline comparison to Auto-sklearn
- Incorrect statements about guarantees


ENHANCING THE RELIABILITY OF OUT-OF-DISTRIBUTION IMAGE DETECTION IN NEURAL NETWORKS
We consider the problem of detecting out-of-distribution images in neural networks. We propose ODIN, a simple and effective method that does not require any change to a pre-trained neural network. Our method is based on the observation that using temperature scaling and adding small perturbations to the input can separate the softmax score distributions between in-and out-of-distribution images, allowing for more effective detection. We show in a series of experiments that ODIN is compatible with diverse network architectures and datasets. It consistently outperforms the baseline approach (Hendrycks & Gimpel, 2017) by a large margin, establishing a new state-of-the-art performance on this task. For example, ODIN reduces the false positive rate from the baseline 34.7% to 4.3% on the DenseNet (applied to CIFAR-10 and Tiny-ImageNet) when the true positive rate is 95%.

-----UPDATE------
The authors addressed my concerns satisfactorily. Given this and the other reviews I have bumped up my score from a 5 to a 6.
----------------------
This paper introduces two modifications that allow neural networks to be better at distinguishing between in- and out- of distribution examples: (i) adding a high temperature to the softmax, and (ii) adding adversarial perturbations to the inputs. This is a novel use of existing methods.
Some roughly chronological comments follow:
In the abstract you don't mention that the result given is when CIFAR-10 is mixed with TinyImageNet.
The paper is quite well written aside from some grammatical issues. In particular, articles are frequently missing from nouns. Some sentences need rewriting (e.g. in 4.1 "which is as well used by Hendrycks...", in 5.2 "performance becomes unchanged").
 It is perhaps slightly unnecessary to give a name to your approach (ODIN) but in a world where there are hundreds of different kinds of GANs you could be forgiven.
I'm not convinced that the performance of the network for in-distribution images is unchanged, as this would require you to be able to isolate 100% of the in-distribution images. I'm curious as to what would happen to the overall accuracy if you ignored the results for in-distribution images that appear to be out-of-distribution (e.g. by simply counting them as incorrect classifications). Would there be a correlation between difficult-to-classify images, and those that don't appear to be in distribution?
When you describe the method it relies on a threshold delta which does not appear to be explicitly mentioned again.
In terms of experimentation it would be interesting to see the reciprocal of the results between two datasets. For instance, how would a network trained on TinyImageNet cope with out-of-distribution images from CIFAR 10?
Section 4.5 felt out of place, as to me, the discussion section flowed more naturally from the experimental results. This may just be a matter of taste.
I did like the observations in 5.1 about class deviation, although then, what would happen if the out-of-distribution dataset had a similar class distribution to the in-distribution one? (This is in part, addressed in the CIFAR80 20 experiments in the appendices).
This appears to be a borderline paper, as I am concerned that the method isn't sufficiently novel (although it is a novel use of existing methods).
Pros:
- Baseline performance is exceeded by a large margin
- Novel use of adversarial perturbation and temperature
- Interesting analysis
Cons:
- Doesn't introduce and novel methods of its own
- Could do with additional experiments (as mentioned above)
- Minor grammatical errors


Published as a conference paper at ICLR 2018 WHEN IS A CONVOLUTIONAL FILTER EASY TO LEARN?
We analyze the convergence of (stochastic) gradient descent algorithm for learning a convolutional filter with Rectified Linear Unit (ReLU) activation function. Our analysis does not rely on any specific form of the input distribution and our proofs only use the definition of ReLU, in contrast with previous works that are restricted to standard Gaussian input. We show that (stochastic) gradient descent with random initialization can learn the convolutional filter in polynomial time and the convergence rate depends on the smoothness of the input distribution and the closeness of patches. To the best of our knowledge, this is the first recovery guarantee of gradient-based algorithms for convolutional filter on non-Gaussian input distributions. Our theory also justifies the two-stage learning rate strategy in deep neural networks. While our focus is theoretical, we also present experiments that justify our theoretical findings.

This paper studies the problem of learning a single convolutional filter using SGD. The main result is: if the "patches" of the convolution are sufficiently aligned with each other, then SGD with a random initialization can recover the ground-truth parameter of a convolutional filter (single filter, ReLU, average pooling). The convergence rate, and how "sufficiently aligned" depend on some quantities related to the underlying data distribution. A major strength of the result is that it can work for general continuous distributions and does not really rely on the input distribution being Gaussian; the main weakness is that some of the distribution dependent quantities are not very intuitive, and the alignment requirement might be very high.
Detailed comments:
1. It would be good to clarify what the angle requirement means on page 2. It says the angle between Z_i, Z_j is at most \rho, is this for any i,j? From the later part it seems that each Z_i should be \rho close to the average, which would imply pairwise closeness (with some constant factor).
2. The paper first proves result for a single neuron, which is a clean result. It would be interesting to see what are values of \gamma(\phi) and L(\phi) for some distributions (e.g. Gaussian, uniform in hypercube, etc.) to give more intuitions.
3. The convergence rate depends on \gamma(\phi_0), from the initialization, \phi_0 is probably very close to \pi/2 (the closeness depend on dimension), which is  also likely to make \gamma(\phi_0) depend on dimension (this is especially true of Gaussian).
4. More precisely, \gamma(\phi_0) needs to be at least 6L_{cross} for the result to work, and L_{cross} seems to be a problem dependent constant that is not related to the dimension of the data. Also \gamma(\phi_0) depends on \gamma_{avg}(\phi_0) and \rho, when \rho is reasonable (say a constant), \gamma(\phi_0) really needs to be a constant that is independent of dimension. On the other hand, in Theorem 3.4 we can see that the upperbound on \alpha (the quality of initialization) depends on the dimension.
5. Even assuming \rho is a constant strictly smaller than \pi/2 seems a bit strong. It is certainly plausible that nearby patches are highly correlated, but what is required here is that all patches are close to the average. Given an image it is probably not too hard to find an almost all white patch and an almost all dark patch so that they cannot both be within a good angle to the average.
Overall I feel the result is interesting but hard to interpret correctly. The details of the theorem do not really support the high level claims very strongly. The paper would be much better if it goes over several example distributions and show explicitly what are the guarantees. The reviewer tried to do that for Gaussian and as I mentioned above (esp. 4) the result does not seem very impressive, maybe there are other distributions where this result works better?
After reading the response, I feel the contribution for the single neuron case does not require too much assumptions and is itself a reasonable result. I am still not convinced by the convolution case (which is the main point of this paper), as even though it does not require Gaussian input (a major plus), it still seems very far from "general distribution". Overall this is a first step in an interesting direction, so even though it is currently a bit weak I think it is OK to be accepted. I hope the revised version will clearly discuss the limitations of the approach and potential future directions as the response did.


Massive data exist among user local platforms that usually cannot support deep neural network (DNN) training due to computation and storage resource constraints. Cloud-based training schemes provide beneficial services, but suffer from potential privacy risks due to excessive user data collection. To enable cloud-based DNN training while protecting the data privacy simultaneously, we propose to leverage the intermediate representations of the data, which is achieved by splitting the DNNs and deploying them separately onto local platforms and the cloud. The local neural network (NN) is used to generate the feature representations. To avoid local training and protect data privacy, the local NN is derived from pre-trained NNs. The cloud NN is then trained based on the extracted intermediate representations for the target learning task. We validate the idea of DNN splitting by characterizing the dependency of privacy loss and classification accuracy on the local NN topology for a convolutional NN (CNN) based image classification task. Based on the characterization, we further propose PrivyNet to determine the local NN topology, which optimizes the accuracy of the target learning task under the constraints on privacy loss, local computation, and storage. The efficiency and effectiveness of PrivyNet are demonstrated with CIFAR-10 dataset.

1. Paper summary
This paper describes a technique using 3 neural networks to privatize data and make predictions: a feature extraction network, an image classification network, and an image reconstruction network. The idea is to learn a feature extraction network so that the image classification network performs well and the image reconstruction network performs poorly.
2. High level paper - subjective
I think the presentation of the paper is somewhat scattered: In section 2 the authors introduce their network and their metric for utility and privacy and then immediately do a sensitivity analysis. Section 3 continues with a sensitivity analysis now considering performance and storage of the method. Then 2.5 pages are spent on channel pruning.
I would have liked if the authors spent more time justifying why we should trust their method as a privacy preserving technique (described in detail below).
The authors clearly performed an impressive amount of sensitivity experiments. Assuming the privacy claims are reasonable (which I have some doubts about below) then this paper is clearly useful to any company wanting to do privacy preserving classification. At the same time I think the paper does not have a significant amount of machine learning novelty in it.
3. High level technical
I have a few doubts about this method as a privacy-preserving technique:
- Nearly every privacy-preserving technique gives a guarantee, e.g., differential privacy guarantees a statistical notion of privacy and cryptographic methods guarantee a computational notion of privacy. In this work the authors provide a way to measure privacy but there is no guarantee that if someone uses this method their data will be private, by some definition, even under certain assumptions.
- Another nice thing about differential privacy and cryptography is that they are impervious to different algorithms because it is statistically hard or computationally hard to reveal sensitive information. Here there could be a better image reconstruction network that does a better job of reconstructing images than the ones used in the paper.
- It's not clear to my why PSNR is a useful way to measure privacy loss. I understand that it is a metric to compare two images that is based on the mean-squared error so a very private image should have a low PSNR while a not private image should have a high PSNR, but I have no intuition about how small the PSNR should be to afford a useful amount of privacy. For instances, in nearly all of the images of Figures 21 and 22 I think it would be quite easy to guess the original images.
4. 1/2 sentence summary
While the authors did an extensive job evaluating different settings of their technique I have serious doubts about it as a privacy-preserving method.


Lung cancer is the leading cause of cancer deaths in the world and early detection is a crucial part of increasing patient survival. Deep learning techniques provide us with a method of automated analysis of patient scans. In this work, we compare AlexNet, a multi-layered and highly flexible architecture, with a custom CNN to determine if lung nodules with patient scans are benign or cancerous. We have found our CNN architecture to be highly accurate (99.79%) and fast while maintaining low False Positive and False Negative rates (< 0.01% and 0.15% respectively). This is important as high false positive rates are a serious issue with lung cancer diagnosis. We have found that AlexNet is not well suited to the problem of nodule identification, though it is a good baseline comparison because of its flexibility.

The authors compare a standard DL machine (AlexNet) with a custom CNN-based solution in the well known tasks of classifying lung tumours into benign or cancerous in the Luna CT scan dataset, concluding that the proposed novel solution performs better.
The paper is interesting, but it has a number of issues that prevents it from being accepted for the ICLR conference.
First, the scope of the paper, in its present form, is very limited: the idea of comparing the novel solution just with AlexNet is not adding much to the present landscape of methods to tackle this problem.
Moreover, although the task is very well known and in the last few year gave rise to a steady flow of solutions and was also the topic of a famous Kaggle competition, no discussion about that can be found in the manuscript.
The novel solution is very briefly sketched, and some of the tricks in its architecture are not properly justified: moreover, the performance improvement w.r.t . to AlexNet is hardly supporting the claim.
Experimental setup consists of just a single training/test split, thus no confidence intervals on the results can be defined to show the stability of the solution.
The whole sections 2.3 and 2.4 include only standard material unnecessary to mention given the target venue, and the references are limited and incomplete.
This given, I rate this manuscript as not suitable for ICLR 2018.


Despite their popularity, even efficient implementations of Support Vector Machines (SVMs) have proven to be computationally expensive to train at a largescale, especially in streaming settings. In this paper, we propose a coreset construction algorithm for efficiently generating compact representations of massive data sets to speed up SVM training. A coreset is a weighted subset of the original data points such that SVMs trained on the coreset are provably competitive with those trained on the original (massive) data set. We provide lower and upper bounds on the number of samples required to obtain accurate approximations to the SVM problem as a function of the complexity of the input data. Our analysis also establishes sufficient conditions on the existence of sufficiently compact and representative coresets for the SVM problem. We empirically evaluate the practical effectiveness of our algorithm against synthetic and real-world data sets.

The paper suggests an importance sampling based Coreset construction for Support Vector Machines (SVM). To understand the results, we need to understand Coreset and importance sampling:
Coreset: In the context of SVMs, a Coreset is a (weighted) subset of given dataset such that for any linear separator, the cost of the separator with respect to the given dataset X is approximately (there is an error parameter \eps) the same as the cost with respect to the weighted subset. The main idea is that if one can find a small coreset, then finding the optimal separator (maximum margin etc.) over the coreset might be sufficient. Since the computation is done over a small subset of points, one hopes to gain in terms of the running time.
Importance sampling: This is based on the theory developed in Feldman and Langberg, 2011 (and some of the previous works such as Langberg and Schulman 2010, the reference of which is missing). The idea is to define a quantity called sensitivity of a data-point that captures how important this datapoint is with respect to contributing to the cost function. Then a subset of datapoint are sampled based on the sensitivity and the sampled data point is given weight proportional to inverse of the sampling probability. As per the theory developed in these past works, sampling a subset of size proportional to the sum of sensitivities gives a coreset for the given problem.
So, the main contribution of the paper is to do all the sensitivity calculations with respect to SVM problem and then use the importance sampling theory to obtain bounds on the coreset size. One interesting point of this construction is that Coreset construction involves solving the SVM problem on the given dataset which may seem like beating the purpose. However, the authors note that one only needs to compute the Coreset of small batches of the given dataset and then use standard procedures (available in streaming literature) to combine the Coresets into a single Coreset. This should give significant running time benefits. The paper also compares the results against the simple procedure where a small uniform sample from the dataset is used for computation.
Evaluation:
Significance: Coresets give significant running time benefits when working with very big datasets. Coreset construction in the context of SVMs is a relevant problem and should be considered significant.
Clarity: The paper is reasonably well-written. The problem has been well motivated and all the relevant issues point out for the reader. The theoretical results are clearly stated as lemmas a theorems that one can follow without looking at proofs.
Originality: The paper uses previously developed theory of importance sampling. However, the sensitivity calculations in the SVM context is new as per my knowledge. It is nice to know the bounds given in the paper and to understand the theoretical conditions under which we can obtain running time benefits using corsets.
Quality: The paper gives nice theoretical bounds in the context of SVMs. One aspect in which the paper is lacking is the empirical analysis. The paper compares the Coreset construction with simple uniform sampling. Since Coreset construction is being sold as a fast alternative to previous methods for training SVMs, it would have been nice to see the running time and cost comparison with other training methods that have been discussed in section 2.