MAX-MIG: AN INFORMATION THEORETIC APPROACH FOR JOINT LEARNING FROM CROWDS
Eliciting labels from crowds is a potential way to obtain large labeled data. Despite a variety of methods developed for learning from crowds, a key challenge remains unsolved: learning from crowds without knowing the information structure among the crowds a priori, when some people of the crowds make highly correlated mistakes and some of them label effortlessly (e.g. randomly). We propose an information theoretic approach, Max-MIG, for joint learning from crowds, with a common assumption: the crowdsourced labels and the data are independent conditioning on the ground truth. Max-MIG simultaneously aggregates the crowdsourced labels and learns an accurate data classifier. Furthermore, we devise an accurate data-crowds forecaster that employs both the data and the crowdsourced labels to forecast the ground truth. To the best of our knowledge, this is the first algorithm that solves the aforementioned challenge of learning from crowds. In addition to the theoretical validation, we also empirically show that our algorithm achieves the new state-of-the-art results in most settings, including the real-world data, and is the first algorithm that is robust to various information structures.

Update after feedback: I would like to thank the authors for their detailed answers, it would be great to see some revisions in the paper also though (except new experimental results).
Especially thank you for providing details of a training procedure which I was missing in the initial draft. I hope to see them in the paper (at least some of them).
I have increased the rating to 6. Given new experimental results both on real data and forecaster comparison I would like to increase the rating to 7. However, I am not sure that this is fair to other authors who would might not be physically able to provide new experimental results due to computational constraints, please note that the experiments in this paper are rather 'light' in the standards of modern deep learning experiments and can be done within the rebuttal period.
====================================================
The paper finds a practical implementation of ideas from Kong & Schoenebeck (2018) for the learning with crowd problem. It proofs the claims from Kong & Schoenebeck (2018) for the specific family of data classifiers and crowd aggregators. From the general perspective, the papers proposes a method for joint training a classifier and a crowd label aggregator with particular consideration of correlated crowd labels.
The paper is fairly well-written and well-balanced between theoretical and empirical justification of the method. I see 1 major and 1 big issues with the paper.
Major issue: I am missing details of the actual procedure of training the model. Is MIG set as a loss function for the data classifier NN? Is crowd aggregator trained also as an NN with MIG as a loss function? How do the authors find the optimal p? Also, in order all the provided theory to work all the found data classifier NN, the aggregator and p should be exact maximisers of MIG as far as I understand. How do the author ensure that they find the exact maximisers? Also related to understanding how training works: on p.15 the authors claim “Note that our method can handle this simple correlated mistakes case and will give all useless experts weight zero based on Theorem 3.4.” I have trouble understanding why the proposed method should find these zero weights rather than it is just able to find them?
I am willing to change my judgement if the authors provide convincing details on the training procedure.
Big issue: Experimental settings.
a) Though it is interesting to see the analysis of the method under controlled environments of synthetic crowd labels with different properties that show benefits of the proposed method (such as dealing with correlated crowd labels), it would be also appealing to see the results with real labels, for example, Rodrigues & Pereira (2017) provide Amazon MTurk crowd labels for the LabelMe data
b) Is the proposed data-crowd forecaster the only method that uses crowd labels on the test data? While it can be argued that it is not straightforward in the test regime to include crowd labels into Crowd Layer, for example, without retraining the neural net, AggNet can use crowd labels without retraining the neural net part. In the presented format, it is unfair to compare the forecaster with the other methods because it uses more information, and essentially, the forecaster is not compared with anything (that uses the same information). It can be compared, at least, with pure Majority Voting, or more advanced pure crowdsourcing aggregation methods. Yes, they won’t use image data, but at least they can use the same amount of crowd label information, which would make a nice comparison with the presented related work and proposed NN: this is what you can get using just image data during test (Crowd Layer, Max-MIG, and others from the current paper), this is what you can get using just crowd labels during test (Majority Voting or, preferably, more advanced pure crowdsourcing aggregators), and this is what you can get using both image and crowd labels during test (the proposed forecaster and AggNet, for example)
Questions out of curiosity:
i). Does Max-MIG handle missing crowd labels for some data points? Did the author use missing labels in the experiments?
ii). Both the Dogs vs. Cats and CIFAR-10 datasets have more or less balanced data, i.e., the number of data points belonging to each ground truth class is similar between classes. Is this true for the LUNA16 dataset? If yes, have the authors tried their method with heavily imbalanced data? In my experience, some crowdsourcing methods may suffer with imbalanced data, for example, Crowd Layer does so on some data. This tendency of Crowd Layer is kind of confirmed on the provided Dogs vs. Cats in the naïve majority case, where based on crowd labels the first class dominates the second.
Other questions/issues/suggestions:
1. Until the formal introduction of the forecaster on page 4, it is not entirely clear what is the difference between the data classifier and data-crowd forecaster. It should be explained more clearly at the beginning that the 3 concepts (data classifier, crowd label aggregator and "data-crowd forecaster") are separated. Also some motivation why we should care about forecaster would be beneficial because one can argue that if we could train a NN that would make good enough predictions why we should waste resources on crowd labels. For example, the provided empirical results can be used as an argument for this.
2. From the introduction it is unclear that there are methods in crowdsourcing that do not rely on the assumption that data and crowd labels are independent given the ground truth labels. As mentioned in related works there are methods dealing with difficulty of data points, where models assume that crowd labels maybe biased on some data points due to their difficulty, e.g., if images are blurred, which violates this assumption.
Also the note that considering image difficulty violates the independence assumption could be added on page 3 around "[we] do not consider the image difficulty"
3. The beginning of page 4. I think it would be more clear to replace "5 experts' labels:" by $y^{[5]}=$
4. I suggest to move the caption of Figure 3 into the main text.
5. p.3 "However, these works are still not robust to correlated mistakes" - Why?
6. Data-crowds forecaster equation. It would be good to add some intuition about this choice. The product between the classifier and aggregator predictions seems reasonable, division on p_c is not that obvious. This expression presumably maximises the information gain introduced below. Some link between this equation and the gain introduction would be nice. Also, minor point – it is better to enlarge inner brackets ()_c
7. The formulation “To the best of our knowledge, our approach is a very early algorithm”, and namely “a very early algorithm” is unclear for me
8. Dual usage of “information intersection” as an assumption and as something that Max-MIG finds is confusing
9. Any comments how the learning rates were chosen are always beneficial
10. Proof of Proposition C.3: “Based on the result of Lemma C.2, by assuming that h ∗ ∈ H_{NN} , we can see (h ∗ , g∗ ,p ∗ ) is a maximizer of max_{h∈H_{NN} ,g∈G_{W A},p∈∆_C} MIGf (h, g,p)” – is expectation missing in the max equation? Is this shown below on page 13? If yes, then the authors should paraphrase this sentence as it does not imply that this is actually shown below
11. p.12 (and below) – what is $\mathbf{C}^m$? Is it $\mathbf{W}^m$?
12. p.15 (at the end of proof) $p \log q$ and $p \log p$ are not formally defined
Minor:
1. p.1 "of THE data-driven-based machine learning paradigm"
2. "crowds aggregator" -> "crowd aggregator"?
3. p.2 (and below) "between the data and crowdsourced labels i.e. the ground truth labelS"
4. Rodrigues & Pereira (2017) has a published version (AAAI) of their paper
5. p.2 "that model multiple experts individually and explicitly in A neural network"
6. p.3 "model the crowds by A Gaussian process"
7. p.3 "We model the crowds via confusion matriCES"
8. p.3 "only provide A theoretic framework and assume AN extremely high model complexity"
9. p.4 "forecast" for h and g -> "prediction"?
10. p.6 “between the data and the crowdsourced labelS”?
11. p.6 “However, in practice, with A finite number of datapoints”
12. p.6 “the experiment section will show that our picked H_{NN} and G_{W A} are sufficientLY simple to avoid over-fitting”
13. p.6 “We call them A Bayesian posterior data classifier / crowds aggregator / data-crowds forecaster, RESPECTEVILY”
14. p.6 “Theorem 3.4. With assumptionS 3.1, 3.3”
15. p.7 “DoctOr Net, the method proposed by Guan et al. (2017)”
16. p.7 “including the naive majority case since naive expert is independent with everything” – rephrasing is required, unclear what “independent with everything” means and who is “naïve expert”
17. Please capitalised names of conferences and journals in References
18. p.10 “she labels the image as “dog”/“cat” with THE probability 0.6/0.8 respectively”, “(e.g. B labels the image as “cat” with THE probability 0.5 and “dog” with THE probability 0.5 when the image has cats or dogs)”
19. p.12 “Lemma C.2. (Kong & Schoenebeck, 2018) With assumptionS 3.1, 3.3”, “Proposition C.3. [Independent mistakes] With assumptionS 3.1, 3.3”


In this paper, we consider the problem of learning a policy by observing numerous non-expert agents. Our goal is to extract a policy that, with high-confidence, acts better than the agents' average performance. Such a setting is important for real-world problems where expert data is scarce but non-expert data can easily be obtained, e.g. by crowdsourcing. Our approach is to pose this problem as safe policy improvement in reinforcement learning. First, we evaluate an average behavior policy and approximate its value function. Then, we develop a stochastic policy improvement algorithm that safely improves the average behavior. The primary advantages of our approach, termed Rerouted Behavior Improvement (RBI), over other safe learning methods are its stability in the presence of value estimation errors and the elimination of a policy search process. We demonstrate these advantages in the Taxi grid-world domain and in four games from the Atari learning environment.

The paper looks at learning a policy from multiple demonstrators which should also be safely improved by an reinforcement learning signal. They define the policy as a mixture of policies from the single demonstrators. The paper gives a new way to estimate the value function of each policy where the overall policy is defined as mixture of the single policies. The paper subsequently looks at the standard error of the value function estimation and then define the policy improvement step in the presence of value estimation error. The resulting reroute constraint for the policy improvement step is evaluated on the taxi toy task as well as on 4 different atari domains.
This paper presents an interesting ideas which is also based on an exhaustive theoretical derivation. However, the paper is lacking clarity and motivation which makes it almost impossible to understand at the first pass. Moreover, the presented results are promising but not exhaustive and the resulting algorithm is also restricted to discrete action domains. More comments see below:
- The paper consists of 2 parts, the average behavior policy and its value function and the safe policy improvement step. The relation between these two parts are not clear. Is the policy improvement step only working if the policy is defined as in section 4 and the value function computed as in section 4?
- Proposition 4.2 needs to be much better motivated and explained. It is totally unclear at this part of the paper why proposition 4.2 is used.
- Please explain why proposition 4.2 indicates that Q^D \approx Q^\beta
- The selection type of S in the taxi example is also unclear.
- How would we solve Equation 8 with continuous actions / parametrized policies \pi? Without this extension, the algorithm is quite restricted.
- the figure captions need to be much more exhaustive. I am not sure I understand the x axis of Figure 4 (right). What iterations are shown here? We only do one improvement step of the behavior policy, without any resembling, is that right?
- Could we also use a similar policy update for policy improvement in reinforcement learning?
- Could you add an algorithm box for estimating the Q-function? Do we estimate every Q-function in isolation using MC estimates and then just use the weighted average?
- It would be interesting to also compare the value function learning method proposed in the paper in isolation to other value function learning methods such as DQN. while the presented method is simple (learn from MC estimates), this is also known to be very data inefficient.


META-LEARNING WITH DIFFERENTIABLE CLOSED-FORM SOLVERS
Adapting deep networks to new concepts from a few examples is challenging, due to the high computational requirements of standard fine-tuning procedures. Most work on few-shot learning has thus focused on simple learning techniques for adaptation, such as nearest neighbours or gradient descent. Nonetheless, the machine learning literature contains a wealth of methods that learn non-deep models very efficiently. In this paper, we propose to use these fast convergent methods as the main adaptation mechanism for few-shot learning. The main idea is to teach a deep network to use standard machine learning tools, such as ridge regression, as part of its own internal model, enabling it to quickly adapt to novel data. This requires back-propagating errors through the solver steps. While normally the cost of the matrix operations involved in such a process would be significant, by using the Woodbury identity we can make the small number of examples work to our advantage. We propose both closed-form and iterative solvers, based on ridge regression and logistic regression components. Our methods constitute a simple and novel approach to the problem of few-shot learning and achieve performance competitive with or superior to the state of the art on three benchmarks.

This paper proposes a meta-learning approach for the problem of few-shot classification. Their method, based on parametrizing the learner for each task by a closed-form solver, strikes an interesting compromise between not performing any adaptation for each new task (as is the case in pure metric learning methods) and performing an expensive iterative procedure, such as MAML or Meta-Learner LSTM where there is no guarantee that after taking the few steps prescribed by the respective algorithms the learner has converged. For this reason, I find that leveraging existing solvers that admit closed-form solutions is an attractive and natural choice.
Specifically, they propose ridge regression as their closed-form solver (R2-D2 variant). This is easily incorporated into the meta-learning loop with any hyperparameters of this solver being meta-learned, along with the embedding weights as is usually done. The use of the Woodbury equation allows to rewrite the closed form solution in a way that scales with the number of examples instead of the dimensionality of the features; therefore taking advantage of the fact that we are operating in a few-shot setting. While regression may seem to be a strange choice for eventually solving a classification task, it is used as far as I understand due to the availability of this widely-known closed-form solution. They treat the one-hot encoded labels of the support set as the regression targets, and additionally calibrate the output of the network (via a transformation by a scale and bias) in order to make it appropriate for classification. Based on the loss of ridge regression on the support set of a task, a parameter matrix is learned for that task that maps from the embedding dimensionality to the number of classes. This matrix can then be used directly to multiply the embedded (via the fixed for the purposes of the episode embedding function) query points, and for each query point, the entry with the maximum value in the corresponding row of the resulting matrix will constitute the predicted class label.
They also experimented with a logistic regression variant (LR-D2) that does not admit a closed-form solution but can be solved efficiently via Newton’s Method under the form of Iteratively Reweighted Least Squares. When using this variant they restrict to tackling the case of binary-classification.
A question that comes to mind about the LR-D2 variant: while I understand that a single logistic regression classifier is only capable of binary classification, there seems to be a straightforward extension to the case of multiple classes, where one classifier per class is learned, leading to a total of N one-vs-all classifiers (where N is the way of the episode). I’m curious how this would compare in terms of performance against the ridge regression variant which is naturally multi-class. This would allow to directly apply this variant in the common setting and would enable for example still oversampling classes at meta-training time as is done usually.
I would also be curious to see an ablation where for the LR-D2 variant SGD was used as the optimizer instead of Newton’s method. That variant may require more steps (similar to MAML), but I’m curious in practice how this performs.
A few other minor comments:
- In the related work section, the authors write: “On the other side of the spectrum, methods that optimize standard iterative learning algorithms, [...] are accurate but slow.” Note however that neither MAML nor MetaLearner LSTM have been showed to be as effective as Prototypical Networks for example. So I wouldn’t really present this as a trade-off between accuracy and speed.
- I find the term multinomial classification strange. Why not use multi-class classification?
- In page 8, there is a sentence that is not entirely grammatically correct: ‘Interestingly, increasing the capacity of the other method it is not particularly helpful’.
Overall, I think this is good work. The idea is natural and attractive. The writing is clear and comprehensive. I enjoyed how the explanation of meta learning and the usual episodic framework was presented. I found the related work section thorough and accurate too. The experiments are thorough as well, with appropriate ablations to account for different numbers of parameters used between different methods being compared. This approach is evidently effective for few-shot learning, as demonstrated on the common two benchmarks as well as on a newly-introduced variant of cifar that is tailored to few-shot classification. Notably, the ridge regression variant can reach results competitive with SNAIL that uses significantly more weights and is shown to suffer when its capacity is reduced. Interestingly, other models such as MAML actually suffer when given additional capacity, potentially due to overfitting.


Asynchronous distributed gradient descent algorithms for training of deep neural networks are usually considered as inefficient, mainly because of the Gradient delay problem. In this paper, we propose a novel asynchronous distributed algorithm that tackles this limitation by well-thought-out averaging of model updates, computed by workers. The algorithm allows computing gradients along the process of gradient merge, thus, reducing or even completely eliminating worker idle time due to communication overhead, which is a pitfall of existing asynchronous methods. We provide theoretical analysis of the proposed asynchronous algorithm, and show its regret bounds. According to our analysis, the crucial parameter for keeping high convergence rate is the maximal discrepancy between local parameter vectors of any pair of workers. As long as it is kept relatively small, the convergence rate of the algorithm is shown to be the same as the one of a sequential online learning. Furthermore, in our algorithm, this discrepancy is bounded by an expression that involves the staleness parameter of the algorithm, and is independent on the number of workers. This is the main differentiator between our approach and other solutions, such as Elastic Asynchronous SGD or Downpour SGD, in which that maximal discrepancy is bounded by an expression that depends on the number of workers, due to gradient delay problem. To demonstrate effectiveness of our approach, we conduct a series of experiments on image classification task on a cluster with 4 machines, equipped with a commodity communication switch and with a single GPU card per machine. Our experiments show a linear scaling on 4-machine cluster without sacrificing the test accuracy, while eliminating almost completely worker idle time. Since our method allows using commodity communication switch, it paves a way for large scale distributed training performed on commodity clusters.

The paper proposes an algorithm to restrict the staleness in ASGD (asynchronous SGD), and also provides theoretical analysis. This is an interesting and important topic. However, I do not feel that this paper solves the fundamental issue - the staleness will be still very larger or some workers need to stay idle for a long time in the proposed algorithm if there exists some extremely slow worker. To me, the proposed algorithm is more or less just one implementation of ASGD, rather than a new algorithm. The key trick in the algorithm is collecting all workers' gradients in the master machine and update them at once, while hard limiting the number of updates in each worker. The theoretical analysis is not brand new. The
line 6 in Algorithm 1 makes the delay a random variable related to the speed of a worker. The faster a worker is, the larger the tau is, which invalidates the assumption implicitly used in the theoretical analysis.
The experiment is done with up to 4 workers, which is not sufficient to validate the advantages of the proposed algorithm compared to state of the art ASGD algorithms. The comparison to other ASGD implementations is also missing, such as Hogwild! and Allreduce.
In addition, I am so surprised that this paper only have 10 references (the last one is duplicated). The literature review is quite shallow and many important work about ASGD are missing, e.g.,
- Parallel and distributed computation: numerical methods, 1989.
- Distributed delayed stochastic optimization, NIPS 2011.
- Hogwild!, NIPS 2011
- Asynchronous Parallel Stochastic Gradient for Nonconvex Optimization, NIPS 2015
- An asynchronous mini-batch algorithm for regularized stochastic optimization, 2016.


RANDOM MESH PROJECTORS FOR INVERSE PROBLEMS
We propose a new learning-based approach to solve ill-posed inverse problems in imaging. We address the case where ground truth training samples are rare and the problem is severely ill-posed-both because of the underlying physics and because we can only get few measurements. This setting is common in geophysical imaging and remote sensing. We show that in this case the common approach to directly learn the mapping from the measured data to the reconstruction becomes unstable. Instead, we propose to first learn an ensemble of simpler mappings from the data to projections of the unknown image into random piecewise-constant subspaces. We then combine the projections to form a final reconstruction by solving a deconvolution-like problem. We show experimentally that the proposed method is more robust to measurement noise and corruptions not seen during training than a directly learned inverse.

This paper describes a novel method for solving inverse problems in imaging.
The basic idea of this approach is use the following steps:
1. initialize with nonnegative least squares solution to inverse problem (x0)
2. compute m different projections of x0
3. estimate x from the m different projections by solving "reformuated" inverse problem using TV regularization.
The learning part of this algorithm is in step 2, where m different convolutional neural networks are used to learn m good projections. The projections correspond to computing a random Delaunay triangulation over the image domain and then computing pixel averages within each triangle. It's not clear exactly what the learning part is doing, i.e. what makes a "good" triangulation, why a CNN might accurately represent one, and what the shortcomings of truly random triangulations might be.
More specifically, for each projection the authors start with a random set of points in the image domain and compute a Delaunay triangulation. They average x0 in each of the Delaunay triangles. Then since the projection is constant on each triangle, the projection into the lower-dimensional space is given by the magnitude of the function over each of the triangular regions. Next they train a convolutional neural network to approximate the above projection. The do this m times. It's not clear why the neural network approximation is necessary or helpful.
Empirically, this method outperforms a straightforward use of a convolutional U-Net to invert the problem.
The core novelty of this paper is the portion that uses a neural network to calculate a projection onto a random Delaunay triangulation. The idea of reconstructing images using random projections is not especially new, and much of the "inverse-ness" of the problem here is removed by first taking the pseudoinverse of the forward operator and applying it to the observations. Then the core idea at the heart of the paper is to speed up this reconstruction using a neural network by viewing the projection onto the mesh space as a set of special filter banks which can be learned.
At the heart of this paper is the idea that for an L-Lipschitz function f : R^k → R the sample complexity
is O(L^k), so the authors want to use the random projections to essentially reduce L. However, the Cooper sample complexity bound scales with k like k^{1+k/2}, so the focus on the Lipschitz constant seems misguided.
This isn't damning, but it seems like the piecewise-constant estimators are a sort of regularizer, and that's where we
really get the benefits.
The authors only compare to another U-Net, and it's not entirely clear how they even trained that U-Net. It'd be nice to see if you get any benefit here from their method relative to other approaches in the literature, or if this is just better than inversion using a U-Net. Even how well a pseudoinverse does would be nice to see or TV-regularized least squares.
Practically I'm quite concerned about their method requiring training 130 separate convolutional neural
nets. The fact that all the different datasets give equal quality triangulations seems a bit odd, too. Is
it possible that any network at all would be okay? Can we just reconstruct the image from regression
on 130 randomly-initialized convolutional networks?
The proposed method isn't bad, and the idea is interesting. But I can't help but wonder whether it works just because what we're doing is denoising the least squares reconstruction, and regression on many random projections might be pretty good for that. Unfortunately, the experiments don't help with developing a deeper understanding.


We propose a guided dropout regularizer for deep networks based on the evidence of a network prediction: the firing of neurons in specific paths. In this work, we utilize the evidence at each neuron to determine the probability of dropout, rather than dropping out neurons uniformly at random as in standard dropout. In essence, we dropout with higher probability those neurons which contribute more to decision making at training time. This approach penalizes high saliency neurons that are most relevant for model prediction, i.e. those having stronger evidence. By dropping such high-saliency neurons, the network is forced to learn alternative paths in order to maintain loss minimization, resulting in a plasticity-like behavior, a characteristic of human brains too. We demonstrate better generalization ability, an increased utilization of network neurons, and a higher resilience to network compression using several metrics over four image/video recognition benchmarks.

The authors propose a data-dependent dropout variant that produces dropout candidates based on their predictive saliency / relevance. Results are reported for 4 datasets (Cifar10, Cifar100, Caltech256 and UCF101) and 4 different models (CNN-2, AlexNet, VGG16 and VGG19), and suggest an increase in generalization performance over other dropout approaches (curriculum dropout, standard dropout, no dropout), as well as increase in the network's plasticity as measured by some existing metrics from the literature. The authors conclude that Excitation Dropout results in better network utilization and offers advantages for network compression (in the sense of neuron pruning).
Overall I find the idea to be interesting and fairly novel, and commend the authors for the fluid writing style. However, I find key issues with the testing and experiments. Specifically, the lack of confidence bounds for individual results makes it impossible to determine whether the reported incremental improvements are actually significant over those of existing approaches. Likewise, I criticize the choice of methods the authors have chosen to compare against, as several other data-dependent dropout approaches (e.g. Information Dropout) exist that may be conceptually closer (and therefore more comparable) to the proposed approach. I also question the choice of tested network architectures and the placement of the dropout layer.
The paper could be of high significance if all claims in the paper could be backed up by experiments that show the advantage of Excitation Dropout to be not a random effect. I will therefore give a lower score for the paper in its current form, but am willing to revise my rating the major points below are properly addressed.
Pros:
+ novel mechanism to improve dropout, results seemingly superior over other methods
+ achieves better utilization of network resources and achieves robustness to dropping out neurons at test time
Cons:
- results without error bars, unclear if advantage is significant
- did not compare against most relevant competing methods
MAJOR POINTS
Section 2 - The comparison to Moreiro et al. is not entirely clear. A fairer comparison would be with some of the other methods listed which also focus on answering the question of which neurons to dropout, or approaches which determine the dropout policy based on information gained from the data, such as Information Dropout (Achille & Soatto). The authors state that Morerio et al are the state-of-the-art in dropout techniques, however based on the results presented here (Figure 3) it seems to perform just as well as standard dropout. Perhaps there are architecture-specific or data-specific issues? In any case this example undermines the confidence of the claims.
Section 3.2, equation 3 - is there some theoretical underpinning as to how this equation was modelled, or was it chosen simply because it covers the expected corner cases described in paragraph 4 of this section? Also, given the intuition in this paragraph (e.g. p_EB = 1 / N), it is correct to assume this equation models the dropout probability but only for fully connected layers? What about dropout in convolutional layers? Though some previous statements do point to the usage of dropout predominantly for fully connected layers, I feel that this context is missing here and should be explicitly addressed. The caption to e.g. Table 1 seems to imply the authors add a single dropout layer in one of the fully connected layers, however this begs the question as to why this positioning was chosen - why only one dropout layer, and why precisely at that location? The scope of the claims should be adapted accordingly.
Section 4.2 - "After convergence, ED demonstrates a significant improvement in performance compared to other methods". If five trained models were used, then some sense of measure of uncertainty should be given throughout. For example, in the Cifar10 results for Figure 3, it is difficult to say whether the marginal improvement from about 80% (standard dropout and curriculum dropout) to about 82% (excitation dropout) is significant or not. Perhaps this would be less of an issue if the authors had worked with e.g. ImageNet, but for these smaller datasets it would definitely be worth to be on the safe side. I highly suspect that statistically speaking (perhaps with the exception of the results on Caltech256), the effects of all of these dropout variants are indistinguishable from each other. I urge the authors to include a measure of the standard deviation / 95% confidence interval across the models that were tested.
The results presented sub-section 4.3 do not justify the claim that the models trained with Excitation Dropout tend to be more informative. Perhaps the definition of "informative" should be expanded upon in length. Can the authors show that the alternative paths learned by the models augmented with Excitation Dropout indeed carry complimentary information and not just redundant information?
Figure 5 shows interesting results, but once again begs the question of whether there is any significant difference between standard dropout and curriculum dropout. I encourage the authors to include confidence bounds for each trace. Likewise, there is an inherent bias in the results, in that the leftmost figure compares EB and CD in the context in which EB was trained, i.e. dropping of "most salient" neurons. The comparison is one-sided, however, as no results are reported from the context in which CD was trained, i.e. dropping neurons more frequently as training progresses. Comparing these results would bring to light whether the performance boost see in Figure 5 is a function of "overfitting" to the training manner or not.
Also, I believe the results for the second column (dropping out least relevant neurons) are misleading. To the best of my understanding, as p_c increases, at some point neurons start to be dropped that actually have high relevance. This could explain why all curves start out similarly and EB slowly begins to stick out - at this point the EB models once again start to be used in the context within which they were trained, in contrast to the other approaches. The authors should perhaps also explicity clarify why this second column gives any more information than the first.
MINOR POINTS
The authors propose Excitation Dropout as a guided regularization technique. Batch normalization is another standard regularization technique which is often compared with dropout. In particular, for deep CNNs, batch normalization is known to work very well, often better than the standard dropout. In the experiments here, to what extent was batch normalization and / or any other widely utilized network regularizers used? Is it possible that the regularizing effect found here actually comes from one of these? I.e. were the models that were not trained from scratch trained with batch normalization? It would be good if more data could be provided for EB vs. other regularizing techniques, if the claim is that EB is a novel regularizer.
Section 3.1 - "We choose to use EB since it produces a valid probability distribution for each network layer". Though this is a nice property, were there any other considerations for choosing the saliency method? Recent work (Adebayo et al, "Sanity Checks for Saliency Maps") has shown that even some well established saliency techniques are actually independent from both the model and data. As this approach relies heavily on the correctness of EB, I feel that a further justification should be given to validate its use for this scenario other than just based on the type of output it produces.
Section 3.1, equation 2 - more detail and reasoning should be given as to why connections with negative weights are excluded from the computation of the conditional probability, if possible without referring the reader to the EB paper. Why is this justified? Is this probability modelled for a specific activation function?
The authors do not provide the details of the CNN-2 architecture (even in the appendix) and simply refer to another article. If the majority of the results presented in the paper are based on this network (including a reference made to a specific layer of the network in subsection 4.2) – which is not commonly known – why not to detail the network architecture and save additional effort for the reader?
How are the class-wise training and test images chosen for Caltech256 dataset?
The authors test the CNN-2 architecture on Cifar10 and Cifar100, and AlexNet, VGG16, and VGG19 on UCF101. I feel that at least a couple architectures should be validated with more than a single dataset, or the authors should justify the current matching between architectures and datasets. Table 2 is unclear regarding what models were used for what datasets (caption could be interpreted to mean that VGG16 was also used for Cifar and Caltech, however other statements seem to say otherwise).
"To prove that the actual boost in accuracy with ED is not provided by the choice of specific masks,..." I suggest that the authors rephrase or explain this sentence in more detail. To the best of my understanding, it is precisely the fact that different masks are used, each reflective of the particular input used to generate the forward activations, that gives boost in performance over "standard" dropout methods by identifying salient paths in the network.
Although it is a very important experimental detail, only in the end of sub-section 4.2, it becomes clear in which layers Excitation Dropout was applied.
Y-axis labels are missing for the left panels in Figure 3.
The authors randomly choose to abbreviate Excitation Dropout as ED in some paragraphs, while write the full form in others.
Table 2 - It is not clear that the "Neurons ON" metric refers to the "average percentage of zero activations" explained below.
Table 2 - How is peak p_EB measured? Is this an average over a set of test images after convergence? If so, I similarly suggest for confidence bounds to be introduced. It would be interesting to compare this to intermediate values (e.g. after every epoch) during training. Same question for entropy of activations and entropy of pEB. This information would be useful for reproducibility.
Table 2 - Where do the delta values in Table 2 come from? If empirically determined, it should be stated explicitly.
Table 2 - In general, because the metrics provided in Table 2 are averages (second paragraph of this Section 4.3), both (to the best of my understanding) across input subsets (e.g. averging results over many test inputs) and models (caption to Table 1), I feel Table 2 in its current form raises confusion given the lack of confidence bounds. I recommend the authors to clarify what type of averaging was done and to introduce e.g. standard deviations across all reported scores. The authors should refrain from using the term "significantly" while describing results if no statistical testing was done, or explicitly clarify their usage of this term.
Table 2 - In general, Table 2 reports results on selected metrics which, if the authors' hypothesis is correct, should have a clear trend as training progresses. An interesting idea to explore would be to include an analysis (in the appendix) of how these factors change over the course of the training procedure. Intuitively, it seems plasticity is something that should be learned slowly over time, so if these plots were to reveal something different, it would be indicative that something else is going on.
Figure 4 - Judging heatmaps is difficult as it depends on the visual perception of the reader. Thus, it is difficult to judge whether, as the authors claim, ED is indeed less "peaky" than the other alternatives. I suggest that the authors use a perceptually uniform heatmap, and to acompany these figures with e.g. the histogram of the heatmap values. Likewise, it is unclear how the multi-model aspect of the testing plays a role in generating these results. From the 5 originally trained models, how was the model selected that generated these results? Was there averaging of any kind?
Figure 5: the text is too small to be readble
Is "re-wiring" the most appropriate term to use to describe what is happening at inference time? Although different paths may be used, the network connections themselves are fixed and thus this is a potential source for confusion.
What do numbers in Table-4 in the appendix represent? Test accuracy?


SNAS: STOCHASTIC NEURAL ARCHITECTURE SEARCH
We propose Stochastic Neural Architecture Search (SNAS), an economical endto-end solution to Neural Architecture Search (NAS) that trains neural operation parameters and architecture distribution parameters in same round of backpropagation, while maintaining the completeness and differentiability of the NAS pipeline. In this work, NAS is reformulated as an optimization problem on parameters of a joint distribution for the search space in a cell. To leverage the gradient information in generic differentiable loss for architecture search, a novel search gradient is proposed. We prove that this search gradient optimizes the same objective as reinforcement-learning-based NAS, but assigns credits to structural decisions more efficiently. This credit assignment is further augmented with locally decomposable reward to enforce a resource-efficient constraint. In experiments on CIFAR-10, SNAS takes fewer epochs to find a cell architecture with state-of-theart accuracy than non-differentiable evolution-based and reinforcement-learningbased NAS, which is also transferable to ImageNet. It is also shown that child networks of SNAS can maintain the validation accuracy in searching, with which attention-based NAS requires parameter retraining to compete, exhibiting potentials to stride towards efficient NAS on big datasets.

Summary:
This paper proposes Stochastic Neural Architecture Search (SNAS), a method to automatically and efficiently search for neural architectures. It is built upon 2 existing works on these topics, namely ENAS (Pham et al 2018) and DARTS (Liu et al 2018).
SNAS provides nice theory and explanation of gradient computations, unites the strengths and avoid the weaknesses of ENAS and DARTS. There are many details in the paper, including the Appendix. The idea is as follows:
+------------+---------------------+-------------------------+
| Method | Differentiable | Directly Optimize |
|                |                           |    NAS reward       |
+------------+---------------------+-------------------------+
| ENAS     |      No                |        Yes                   |
| DARTS   |      Yes               |        No                    |
| SNAS     |      Yes               |        Yes                   |
+------------+---------------------+-------------------------+
SNAS inherits the idea of ENAS and DARTS by superpositioning all possible architectures into a Directed Acyclic Graph (DAG), effectively sharing the weights among all architectures. However, SNAS improves over ENAS and DARTS as follows (Section 2.2):
1. SNAS improves over ENAS in that it allows independent sampling at edges in the shared DAG, leading to a more tractable gradient at the edges of the DAG, which in turn allows more tractable Monte Carlo estimation of the gradients with respect to the architectural parameters.
2. While DARTS also has the property (1), DARTS implements this by computing the expected value at each node in the DAG, with respect to the joint distribution of the input edges and the operations. This makes DARTS not optimize the direct NAS objective. SNAS, due to their smart manipulation of architectural gradients using Gumbel variables, still optimizes the same objective with NAS and ENAS, but has a smoother gradients.
Experimental results in the paper show that SNAS finds architectures on CIFAR-10 that are comparable to those found by ENAS and DARTS, using a reasonable amount of computing resource. These architectures can also be transferred to learn competent models on ImageNet, like those of DARTS. Furthermore, experimental observations (Figure 3) are consistent with the theory above, that is:
1. The search process of SNAS is more stable than that of ENAS (as SNAS samples with a smaller variance).
2. Architectures found by SNAS perform better than those of DARTS, as SNAS searches directly for the NAS reward of the sampled models.
Strengths:
1. SNAS unites the strengths and avoids the weaknesses of ENAS and DARTS
2. SNAS provides a nice theory, which is verified through their experimental results.
Weaknesses:
I don’t really have any complaints about this paper. Some presentations of the paper might have been improved, e.g. the discussion on the ZERO operation in other comments should have been included.


We present an end-to-end reinforcement learning approach to hierarchical text classification where documents are labeled by placing them at the right positions in a given hierarchy. While existing "global" methods construct hierarchical losses for model training, they either make "local" decisions at each hierarchy node or ignore the hierarchy structure during inference. To close the gap between training/inference and capture label dependencies in an end-to-end manner, we propose to learn a label assignment policy to determine where to place the documents and when to stop. The proposed method, HiLAP, directly optimizes metrics over the hierarchy, makes inter-dependent decisions during inference, and can be combined with different text encoding models for end-to-end training. Experiments on three public datasets show that HiLAP yields an average improvement of 33.4% in Macro-F1 and 5.0% in Samples-F1, outperforming state-of-the-art methods by a large margin.

This papers uses the label hierarchy to drive the search process over a set of labels using reinforcement learning. The approach offers clever and promising techniques to force the inference process in structured classification to converge, but experiments seem to lack apple-to-apple comparisons.
However, I think the authors should rather present this work as structured classification, as labels dependencies not modeled by the hierarchy are exploited, and as other graph structure could be exploited to drive the RL search.
I tend to see hierarchical classification as an approach to multi-label classification justified by a greedy decomposition that reduced both training and test time. This view has been outmoded for more than an decade, first as flat approaches became feasible, and now as end-to-end  structured classification is implementable with DNNs (see for instance David Belanger work with McCallum)
Compared to other structured classification approaches whose scope is limited by the complexity of the inference process, this approaches is very attractive. The authors open the optimization black box of the inference process by adding a few very clever tricks that facilitate convergence:
- Intermediate rewards based on the gain on F1 score
- Self critical training approach
- "Clamped" pre-training enabled by the use of state embeddings that are multiplied my a transition to any state in the free mode, and just the next states in the hierarchy in the clamped mode
- Addition of a flat loss to improve the quality of the document representation
While those tricks may have been used for other applications, they seem new in the context of hierarchical/multi-label/structured classification.
While the experiments appear thorough, they could be the major weakness of this paper. The results the authors quote as representative of other approaches seem in fact entirely reproduced on datasets that were not used on the original papers, and the authors do not try an apple-to-apple comparison to determine if this 'reproduction' is fair. None of the quoted work used the 2018 version of Yelp, and I could only find RCV1 Micro-F1 experiments in Johnson and Yang, who report a 84% micro-F1, far better than the 76.6% reported on their behalf here, and better than the 82.7% reported  by the authors. I read note 4 about the difference in the way the threshold is computed, but I doubt it can explain such a large difference. I did not check everything, but could not find and apple-to-apple comparison?
Have the network architecture been properly optimized in terms of hyper-parameters?
In particular, having tried Kim CNN on large label sets, I suspect the author settings using a single layer after the convolution is sub-optimal. I concur with the following paper than an additional hidden layer is essential: Liu et al "Deep Learning for Extreme Multi-label Text Classification". I also note the 32 batch size could be way too small for sparse label sets (I tend to use a batch size of 512 on this type of data).


We study the problem of learning classifiers robust to universal adversarial perturbations. While prior work approaches this problem via robust optimization, adversarial training, or input transformation, we instead phrase it as a two-player zero-sum game. In this new formulation, both players simultaneously play the same game, where one player chooses a classifier that minimizes a classification loss whilst the other player creates an adversarial perturbation that increases the same loss when applied to every sample in the training set. By observing that performing a classification (respectively creating adversarial samples) is the best response to the other player, we propose a novel extension of a game-theoretic algorithm, namely fictitious play, to the domain of training robust classifiers. Finally, we empirically show the robustness and versatility of our approach in two defence scenarios where universal attacks are performed on several image classification datasets -CIFAR10, CIFAR100 and ImageNet.

Being familiar but not an expert in either game theory or adversarial training, my review will focus on the overall soundness of the proposed method
Summary:
The authors propose to tackle the problem of adversarial training.
Deep networks are know to be susceptible to adversarial attacks.
Adversarial training is concerned with the training of networks that both achieve good performance for the original task while being robust to adversarial attacks.
They propose to focus on universal adversarial perturbations, as opposed to per-sample perturbations. The latter is a subclass of the former.
It doesn’t strike as the most natural scenario: I can’t really think of a practical image classification scenario where one would want to perturb a whole dataset of image with a single perturbation. That said, this focus leads to simpler algorithms (complexity and storage wise) which are worth exploring.
The authors first present the min-max problem of adversarial training at hand where a classifier f mimizes a loss L for a dataset D, while the conman maximizes the loss over perturbation of the dataset \epsilon.
They then introduce an algorithm to solve it inspired by fictitious play:
A sequence of classifiers and perturbed datasets are created iteratively by the two players (classifier, conman) and each player uses the complete history of its opponent to make its next move.
The objective solved by each player  is :
conman: fool all past classifiers with a single new perturbation
classifier: be robust to all past perturbations so far.
Although it makes intuitive sense, it is unclear from the manuscript whether this formulation provides any convergence guarantees. It would be great to know whether the connection to fictitious play is purely inspirational or if any of the theoretical guarantees from game theory apply here.
The conman’s objective to fool all past classifiers is the bottleneck (in terms of storage) and an approximation is proposed: the mean loss over past classifiers is replaced by the loss under a single ‘average’ classifier trained on all past dataset, with the intuition that this average classifier summarizes all past classifiers
A particular algorithm for perturbation learning is described and the proposed algorithm is compared against two baselines: a pre-existing adversarial training algorithm, an non-adversarial algorithm
The metrics chosen are accuracy and adversarial accuracy.
On standard classification tasks, adversarial algorithms perform slightly less well on the original task (accuracy) but are robust to perturbation as expected,
It would be interesting to know if these good performances extend to per-sample perturbations: Do a network trained on universal perturbations perform well against per sample perturbation?
Remarks:
sgn missing in the adversarial patch update (and who is alpha?)
introduce terminology: white box black box


In this paper, we adopt distributionally robust optimization (DRO) (Ben-Tal et al., 2013) in hope to achieve a better generalization in deep learning tasks. We establish the generalization guarantees and analyze the localized Rademacher complexity for DRO, and conduct experiments to show that DRO obtains a better performance. We reveal the profound connection between SGD and DRO, i.e., selecting a batch can be viewed as choosing a distribution over the training set. From this perspective, we prove that SGD is prone to escape from bad stationary points and small batch SGD outperforms large batch SGD. We give an upper bound for the robust loss when SGD converges and keeps stable. We propose a novel Weighted SGD (WSGD) algorithm framework, which assigns high-variance weights to the data of the current batch. We devise a practical implement of WSGD that can directly optimize the robust loss. We test our algorithm on CIFAR-10 and CIFAR-100, and WSGD achieves significant improvements over the conventional SGD.

This paper motivates performing a “robustified” version of SGD. It attempts to formalize this notion, proves some variants of generalization bounds, and proposes an algorithm that claims to implement such a modified SGD.
The clarity of the paper can be improved. There are several notational and language ambiguities throughout. Most importantly, a couple of paragraphs that are meant to convey key intuitions about the results are very badly written (the one following Theorem 1, the one preceding Theorem 2, the last one in Section 5.1, and the one preceding Theorem 5, more on these later).
Apart from these clarity issues, the significance of the results is weak. This is because although the technical statements seem correct, the leap from them to measurable outcomes (such as actual generalization bounds) are missing. Part of this is due to a lack of a good notion of “true” robust risk. Moreover the algorithmic contribution does not connect well with the suggested theory and the experimental results are modest at best. Here is a more detailed breakdown of my objections.
-	The notion of distributional robust loss is sound, i.e. R(\theta, K). Its empirical variant is also good, \hat{R}(\theta, K). But the notion of robust loss defined in the paper, \hat R_{S,P}(\theta) with the weights on the samples, breaks from this notion. The reason is that in the former case the weights depend on the value of the sample (z) whereas in the latter they depend on the index (i). It is not evident how to map one to the other.
-	This makes the question of what is the “true” robust risk unclear. It is tempting to simply say it is its expectation with respect to a generic sample. This is the view taken in Theorem 3, which offers a kind of generalization bound. But if one looks carefully at this, the location of the expectation and supremum should be swapped. Here is an alternative view: if want to think of the infinite sample limit, then we need to have a sequence of robustness classes P_m, that vary with m (say those that put weight only on a q-fraction of the samples, just like in the suggested WSGD). The “true” robust risk would be the limit of the sup of the empirical risk, this keeps the sup and expectation in the right order. Under the right conditions, this limit would indeed exist. And it is difficult to know, for a given m, how *far* the generic-sample expectation of Theorem 3 is from it. Without this knowledge, it is difficult to interpret Theorem 3 as a generalization bound.
-	Theorem 1 itself is a standard result. The discussion after Theorem 1 is the kind of argument that also explains Fisher information, and can be presented more clearly. I’m not sure whether Theorem 2 exactly proves what the paragraph before it is trying to explain. The fact that SGD converges into the ball seems contrived, since the quantities that we are trying to bound have nothing to do with the optimization method. If the optimum is within the ball (+/- something) then the same result should hold with the step size replaced with the (+/- something). So how does this explain escaping stationary points?
-	If we accept Theorem 3 as a generalization bound, alongside with the Rademacher bounds of Theorem 4, I don’t think the paper treats the balance between the various terms adequately enough. In particular we see that the |P|_\infty term in Theorem 3 has to balance out the (robust) Rademacher bound, and need it to be of the order of (1+RAD_2(P)\sqrt{\log N}/m). For P that puts weight k over 1/k points, |P|_\infty = 1/k. RAD_2(P) is bounded by 1/\sqrt{k}, so it’s negligible next to 1. But the covering number N can grow exponentially with k (when it’s not too large, and for small \epsilon, just by counting arguments). So this seems to say that for a good tradeoff in the bounds will lead to k having to be a growing fraction of m. This intuition, if true, is not presented. Not only that, but it also goes against the suggested approach of choosing some constant fraction of m.
-	Theorem 5 gives a local Rademacher complexity. But again there is a conceptual step missing from this to strong generalization bounds, partly because we are not exactly minimizing the empirical risk within the considered class. Also, the discussion that follows bounding the rad_\infty with |P|_\infty is deficient, because it misses again the fact there are two salient terms that need to balance out.
-	Algorithm 1 (WSGD) needs to specify (q,r) and which G (G_1 or G_2) as inputs too.
-	Most importantly, WSGD does not seem to be minimizing the robust risk at all. First, I’m not really sure what the G_1 variant does. If we were to follow the intuition of Theorem 1, we should be looking at the gradients, not the loss values. As for G_2, by sampling we are in fact replacing the sup over p with an average over P. This can have a significantly different behavior, and we could possibly interpret it as a slightly reduced effective batch size, especially in the case of G_2. In fact, in the experiments, when r is set to 0, this is exactly what is happening! At any rate, it is not clear at all how any of the earlier sections connect with sections 6 or 7.
-	In the experimental section it is not clarified which of the latter two is used (I assume G_2, the randomized one, given the discussion in the end of Section 6.) When the authors write “accuracy improvement”, they should more clearly say “relative decrease in misclassification error”. That’s the only thing that makes sense with the numbers, and if it does in fact  the authors mistakenly say that the 5-15% improvement is for CIFAR 100 and the 5% is for CIFAR 10, it’s the other way around! And the exception (least) improvement seems to be ResNet-34 on CIFAR-100 (not VGG-16, as they claim, unless the table is wrong.) All in all, these are all pretty weak results, albeit consistent. A better benchmark would have been to compare against various batch sizes, and somehow show that the results do *not* follow from batch size effects.


NEURAL PERSISTENCE: A COMPLEXITY MEASURE FOR DEEP NEURAL NETWORKS USING ALGEBRAIC TOPOLOGY
While many approaches to make neural networks more fathomable have been proposed, they are restricted to interrogating the network with input data. Measures for characterizing and monitoring structural properties, however, have not been developed. In this work, we propose neural persistence, a complexity measure for neural network architectures based on topological data analysis on weighted stratified graphs. To demonstrate the usefulness of our approach, we show that neural persistence reflects best practices developed in the deep learning community such as dropout and batch normalization. Moreover, we derive a neural persistencebased stopping criterion that shortens the training process while achieving comparable accuracies as early stopping based on validation loss.

The authors, motivated by work in topological graph analysis, introduce a new broadly applicable complexity measure they call  neural persistence--essentially a sum over norms of persistence diagrams (objects from the study of persistent homology).  The also provide experiments testing their parameter, primarily on MNIST with some work on CIFAR-10.
I'd like to preface my criticism with the following: this work is extremely compelling, and the results and experiments are sound.  I'm very interested to see where this goes.  Figure 2 is particularly compelling!
That said, I am extremely suspicious of proposals for measures of generalization which (1) do not make contact with the data distribution being studied, and (2) which are only tested on MNIST and CIFAR-10.  Additionally, (3) it is not clear what a "good" neural persistence is, a priori, and (4) I'm not entirely sure I agree with the author's assessment of their numerical data.
In more detail below:
1. At this point, there's a tremendous number of different suggested ways to measure "generalization" by applying different norms and bounds and measures from all of the far reaches of mathematics.  A new proposed measure **really needs** to demonstrate a clear competitive measure against other candidates.  The authors make a strong case that this measure is better than competitors from TGA, but I'm not yet convinced this measure is doing enough legwork.  For example, is it possible that a network has high neural persistence, but still has terrible test or validation error?  Why or why not?  Are there obvious counterexamples?  Are there reasons to think those obvious counterexamples aren't like trained neural networks?  These are all crucial questions to ask and answer if you want this sort of measure to be taken seriously.
2.  Most of your numerical experiments were on MNIST, and MNIST is weird.  It's getting to be a joke now in the community that your idea works on MNIST, but breaks once you try to push it to something harder.  Even Cifar-10 has its quirks, and observations that are true of some networks absolutely do not generalize to others.
3. While I'm convinced that neural persistence allows you to distinguish between networks trained in different ways, it isn't clear why I should expect a particular neural persistence to mean anything at all w.r.t. validation loss.  Are there situations in which the neural persistence has stopped changing, but validation loss is still changing appreciably?  Why or why not?
4. I'm concerned that the early stopping procedure used as a benchmark wasn't tuned as carefully as neural persistence was.  I also honestly cannot determine anything from Figure 4 except that your "Fixed" baseline is bad, and that persistence seems to do about the same as validation loss.  It even seems that Training loss is a better early stopping criteria (better than both validation and persistence!) from this plot, because it seems to perform just as well, and systematically stop earlier.  Am I reading this plot right (particularly for 1.0 fraction MNIST)?
This work currently seems like a strong candidate for the workshop track.  I would have difficulty raising my score above much more than a 6 without much more numerical data, and analysis of when the measure fails.
Edit: The authors have made a significant effort to address my concerns, and I'm updating my score to 7 from 5 in response.


IMPROVING GENERALIZATION AND STABILITY OF GENERATIVE ADVERSARIAL NETWORKS
Generative Adversarial Networks (GANs) are one of the most popular tools for learning complex high dimensional distributions. However, generalization properties of GANs have not been well understood.
In this paper, we analyze the generalization of GANs in practical settings. We show that discriminators trained on discrete datasets with the original GAN loss have poor generalization capability and do not approximate the theoretically optimal discriminator. We propose a zero-centered gradient penalty for improving the generalization of the discriminator by pushing it toward the optimal discriminator. The penalty guarantees the generalization and convergence of GANs. Experiments on synthetic and large scale datasets verify our theoretical analysis.

The primary innovation of this paper seems focused towards increasing the generalization of GANs, while also maintaining convergence and preventing mode collapse.
The authors first discuss common pitfalls concerning the generalization capability of discriminators, providing analytical underpinnings for their later experimental results. Specifically, they address the problem of gradient explosion in discriminators.
The authors then suggest that a zero-centered gradient penalty (0-GP) can be helpful in addressing this issue. 0-GPs are regularly used in GANs, but the authors point out that the purpose is usually to  provide convergence, not to increase generalizability. Non-zero centered penalties can give a convergence guarantee but, the authors, assert, can allow overfitting. A 0-GP can give the same guarantees but without allowing overfitting to occur.
The authors then verify these assertions through experimentation on synthetic data, as well as MNIST and ImageNet. My only issue here is that very little information was given about the size of the training sets. Did they use all the samples? Some portion? It is not clear from reading. This would be a serious impediment to reproducibility.
All in all, however, the authors provide a convincing  combination of analysis and experimentation. I believe this paper should be accepted into ICLR.
Note: there is an error on page 9, in Figure 3. The paragraph explanation should list that the authors' 0-GP is figure 3(e). They list (d) twice.


While it has not yet been proven, empirical evidence suggests that model generalization is related to local properties of the optima which can be described via the Hessian. We connect model generalization with the local property of a solution under the PAC-Bayes paradigm. In particular, we prove that model generalization ability is related to the Hessian, the higher-order "smoothness" terms characterized by the Lipschitz constant of the Hessian, and the scales of the parameters. Guided by the proof, we propose a metric to score the generalization capability of the model, as well as an algorithm that optimizes the perturbed model accordingly.

This paper gives various PAC-Bayesian generalization guarantees and some
empirical results on parameter perturbation in training using an algorithm
motivated by the theory.
The fundamental issue addressed in this paper is whether parameter
perturbation during training improves generalization and, if so, what
theoretical basis exists for this phenomenon.  For continuously
parameterized models, PAC-Bayesian bounds are fundamentally based on
parameter perturbation (non-singular posteriors).  So PAC-Bayesian
theory is naturally tied to parameter perturbation issues.  A more
refined question is whether the size of the perturbation should be
done on a per-parameter bases and whether per-parameter noise levels
should be adaptive --- should the appropriate noise level for each
parameter be adjusted on the basis of statistics in the training data.
Adam and RMS-prop both adapt per-parameter learning rate eta_i to be
proportional to 1/((E g_i^2) + epsilon) where E g_i^2 is some running
estimate of the expectation over a draw of a training point of the
square of the gradient of the loss with respect to parameter i.  At
the end of the day, this paper, based on PAC-Bayesian analysis,
proposes that a very similar adaptation be made to per-parameter noise
during training but where E g_i^2 is replaced by the RMS value \sqrt{E
g_i^2}.  It seems that all theoretical analyses require the square
root --- the units need to work.  A fundamental theoretical question,
perhaps unrelated to this paper, is why in learning rate adaptation the
square root hurts the performance.
This paper can be evaluated on both theoretical and empirical grounds.
At a theoretical level I have several complaints.  First, the
theoretical analysis seem fairly mechanical and without theoretical
innovation. Second, the analysis obscures the prior being used (the
learning bias). The paper first states an assumption that each
parameter is a-priori taken to be uniform over |w_i| <= \tau_i and the
KL-divergence in the PAC-Bayes bound is then log tau_i/sigma_i where
sigma_i is the width of a uniform posterior over a smaller interval.
But later they say that they approximate tau_i by |w_i| + kappa_i with
kappa_i = \gamma |w_i| + epsilon.  I believe this works out to be
essentially a log-uniform prior on |w_i| (over some finite range of
log |w_i|).  This seems quite reasonable but should be made explicit.
The paper ignores the possibility that the prior should be centered at
the random initialization of the parameters.  This was found to be
essential in Dziugaite and Roy and completely changes the dependence
of k_i on w_i.
Another complaint is that the Hoefding bound is very loose in cases
where the emperical loss is small compared to its upper bound.  The
analysis can be more intuitively related to practice by avoiding the
rescaling of the loss into the interval [0,1] and writing expressions
in terms of a maximum bound on the loss L_max.  When hat{L} << L_max
(almost always the case in practice) the relative Chernoff bound is
much tighter and significantly alters the analysis.  See McAllester's
PAC-Bayesian tutorial.
The theoretical discussion on re-parameterization misses an important
point, in my opinoin, relative to the need to impose a learning bias
(the no-free-lunch theorem).  All L_2 generalization bounds can be
interpreted in terms of a Gaussian prior on the parameters.  In all
such cases the prior (the learning bias) is not invariant to
re-parameterization.  All L_2 generalization bounds are subject to the
same re-parameterization criticism.  A prior tied to a particular
parameterization is standard practice in machine learning for in all
L_2 generalization bounds, including SVMs.  I do think that a
log-uniform prior (rather than a Gaussian prior) is superior and
greatly reduces sensitivity to re-parameterization as noted by the
authors (extremely indirectly).
I did not find the empirical results to very useful.  The value of
parameter perturbation in training remains an open question. Although
it is rarely done in practice today, it is an important fundamental
question. A much more thorough investigation is needed before any
conclusions can be drawn with confidence. Experimentation with
perturbation methods would seem more informative than theory given the
current state of the art in relating theory to practice.


Data parallelism has become a dominant method to scale Deep Neural Network (DNN) training across multiple nodes. Since the synchronization of the local models or gradients can be a bottleneck for large-scale distributed training, compressing communication traffic has gained widespread attention recently. Among several recent proposed compression algorithms, Residual Gradient Compression (RGC) is one of the most successful approaches-it can significantly compress the transmitting message size (0.1% of the gradient size) of each node and still preserve accuracy. However, the literature on compressing deep networks focuses almost exclusively on achieving good compression rate, while the efficiency of RGC in real implementation has been less investigated. In this paper, we develop an RGC method that achieves significant training time improvement in real-world multi-GPU systems. Our proposed RGC system design called RedSync, introduces a set of optimizations to reduce communication bandwidth while introducing limited overhead. We examine the performance of RedSync on two different multiple GPU platforms, including a supercomputer and a multi-card server. Our test cases include image classification on Cifar10 and ImageNet, and language modeling tasks on Penn Treebank and Wiki2 datasets. For DNNs featured with high communication to computation ratio, which has long been considered with poor scalability, RedSync shows significant performance improvement.

Quality and clarity:
The paper proposes an approach to reduce the communication bandwidth and overhead in distributed deep learning. The approach leverages on previous work (mainly the residual gradient compression (RGC) algorithm), and proposes several implementation optimizations. From what I can read, it is the basic RGC algorithm that is used, but with some clever optimization to improve the performance of it.
The quality of the paper is good, it is well-written and easy to read. The evaluation of the proposed approach is well done, using several diverse datasets and models, and executed on two different parallel systems. However, the reasons why RGC and qRGC sometimes have better accuracy than SGD needs to be analyzed and explained.
Originality and significance:
The originality of the paper is relatively low (optimization of an existing algorithm) and the contributions are incremental. However, the paper addresses an important practical problem in distributed learning, and thus can have a significant practical impact on how distributed deep learning systems are implemented.
Pros:
* Addresses an important issue.
* Good performance.
* Good evaluation on two different systems.
Cons:
* Limited contribution. Although I like implementation papers (very important), I think the contribution is to low for ICLR.
Minor:
* In general, the figures are hard to read (the main problem is to small text)
* Compression in the title is slightly misleading, since it's mainly selection that is done (top-0.1% gradients). Although the values are packed in a data structure for transmission, it's not compression in a information theory perspective.


Distributed computing can significantly reduce the training time of neural networks. Despite its potential, however, distributed training has not been widely adopted: scaling the training process is difficult, and existing SGD methods require substantial tuning of hyperparameters and learning schedules to achieve sufficient accuracy when increasing the number of workers. In practice, such tuning can be prohibitively expensive given the huge number of potential hyperparameter configurations and the effort required to test each one. We propose DANA, a novel approach that scales out-of-the-box to large clusters using the same hyperparameters and learning schedule optimized for training on a single worker, while maintaining similar final accuracy without additional overhead. DANA estimates the future value of model parameters by adapting Nesterov Accelerated Gradient to a distributed setting, and so mitigates the effect of gradient staleness, one of the main difficulties in scaling SGD to more workers. Evaluation on three state-of-the-art network architectures and three datasets shows that DANA scales as well as or better than existing work without having to tune any hyperparameters or tweak the learning schedule. For example, DANA achieves 75.73% accuracy on ImageNet when training ResNet-50 with 16 workers, similar to the non-distributed baseline.

# overview
In this work, Nesterov Accelerated gradient based updates are applied in a distributed fashion to scale SGD based training to multiple nodes without the introduction of further hyperparameters or having to adapt the learning rate schedule from that of single node training on the same data.
Evaluation is carried out on image classification workloads using ResNet model variants across CIFAR10,100, and ImageNet datasets, utilizing from 8-32 nodes. In contrasting test error relative to single node performance, the authors find their method degrades less than other synchronous and asynchronous SGD based approaches as node count increases.
Overall, this work is presented in a fairly clear and logical manner, and the writing is easy to follow.  However the approach described appears to be contingent on very specific worker communication patterns and timing which seem unrealistic for real-world settings (namely that each worker sends exactly one update per N sized block received).  Extrapolating from the curvature of the results shown it doesn't appear that DANA would continue to outperform other methods like ASGD once the worker count scales beyond the 32 node limit evaluated.
# pros
* no additional hyperparameter tuning required
* should be easy to drop into existing asynchronous SGD implementations, just need to modify the worker side.
* does appear to scale slightly better from an accuracy perspective in 16-32 node counts
# cons
* Biggest criticism is the assumption of block random or round-robin worker update scheduling. Presuming each worker will update master exactly once to determine future parameter position is far from realistic on real hardware (varying capacity, performance, system loads, dealing with stragglers) and should probably be considered a synchronous not asynchronous update.
* only evaluated on image classification tasks on cifar10, cifar100, imagenet on resnet-20 and resnet-50. Would have been better to evaluate on a more varied set of tasks/models/datasets
# other comments
* Figure 2 baseline performance reported is a bit misleading/confusing since it was only evaluated on a single worker. Would suggest restricting to a single point rather than some extrapolated line that seems to indicate being run on multiple-workers.
* Figure 3 should should also show multi-node speedups for the other methods compared for completeness.
* Section 5.2 should report on percentage scaling efficiency rather than using speedup as it doesn't normalize for worker count.  For instance 16x could be interpreted as good or poor if it was achieved using 16 vs 160 nodes.
* Section 5.2 there's a small typo: GPUs -> GPU
* Consider https://arxiv.org/abs/1705.07176 in related work?


DEEP LEARNING 3D SHAPES USING ALT-AZ ANISOTROPIC 2-SPHERE CONVOLUTION
The ground-breaking performance obtained by deep convolutional neural networks (CNNs) for image processing tasks is inspiring research efforts attempting to extend it for 3D geometric tasks. One of the main challenge in applying CNNs to 3D shape analysis is how to define a natural convolution operator on noneuclidean surfaces. In this paper, we present a method for applying deep learning to 3D surfaces using their spherical descriptors and alt-az anisotropic convolution on 2-sphere. A cascade set of geodesic disk filters rotate on the 2-sphere and collect spherical patterns and so to extract geometric features for various 3D shape analysis tasks. We demonstrate theoretically and experimentally that our proposed method has the possibility to bridge the gap between 2D images and 3D shapes with the desired rotation equivariance/invariance, and its effectiveness is evaluated in applications of non-rigid/ rigid shape classification and shape retrieval.

# Summary
This paper proposes a new kind of spherical convolution for use in spherical CNNs, and evaluates it on rigid and non-rigid 3D shape recognition and retrieval problems. Previous work has either used general anisotropic convolution or azimuthally isotropic convolution. The former produces feature maps on SO(3), which is deemed undesirable because processing 3-dimensional feature maps is costly. The latter produces feature maps on the sphere, but requires that filters be circularly symmetric / azimuthally isotropic, which limits modeling capacity. This paper proposes an anisotropic spherical convolution that produces 2D spherical feature maps. The paper also introduces an efficient way of processing geodesic / icosahedral spherical grids, avoiding complicated spectral algorithms.
# Strengths
The paper has several strong points. It is well written, clearly structured, and the mathematics is clear and precise while avoiding unnecessary complexity. Much of the relevant related work is discussed, and this is done in a balanced way. Although it is not directly measured, it does seem highly likely that the alt-az convolution is more computationally efficient than SO(3) convolution, and more expressive than isotropic S2 convolution. The most important contribution in my opinion is the efficient data structure presented in section 4, which allows the spherical convolution to be computed efficiently on GPUs for a grid that is much more homogeneous than the lat/lon grids used in previous works (which have very high resolution near the poles, and low resolution at the equator). The idea of carving up the icosahedral grid in just the right way, so that the spherical convolution can be computed as a planar convolution with funny boundary conditions, is very clever, elegant, and practical.
# Weaknesses
There is however a misunderstanding about the properties of the alt-az convolution that must be cleared up before this paper can be published. To start with, the set of rotations R(phi, nu, 0) called the alt-az group in this paper is not a group in the mathematical sense. This easy to see, because a composition of rotations of the form Rz(phi) Ry(nu) is not generally of that form. For instance we can multiply Rz(phi) Ry(nu) by the element Rz(omega)Ry(0) = Rz(omega), which gives the element Rz(phi) Ry(nu) Rz(omega). As noted in the paper, this is a general element of SO(3) (and hence not in the set of alt-az rotations). So the closure axiom of a group is violated.
This matters, because the notion of equivariance really only makes sense for a group. If a layer l satisfies l R = R l  (for R a alt-az rotation), then it automatically satisfies l RR' = RR' l, which means l is equivariant to the whole group generated by the set of alt-az rotations. As we saw before, this is the whole rotation group. This would mean that the layer is actually SO(3)-equivariant, but it has been proven [1], that any rotation equivariant layer between scalar spherical feature maps can be expressed as an azimuthally isotropic convolution. Since the alt-az convolution is not isotropic and maps between scalars on S2, it cannot be equivariant. This also becomes apparent in the experiments section, where rotational data augmentation is found to be necessary. The paper does not contain an attempted proof of equivariance, and if one tries to give one, the impossibility of doing so will become apparent.
I note that the alt-az convolution *is* equivariant to rotations in the subgroup SO(2) of rotations around the Z-axis.
Another somewhat jarring fact about the alt-az convolution is that it is not well defined on the south pole. The south pole can be represented by any pair of coordinates of the form phi in [0, 2pi], nu = +/- pi. But it is easy to see that eq. 10 will give different results for each of these coordinates, because they correspond to different rotations of the filter about the Z-axis. This is ultimately due to the fact that the set of alt-az rotations is not the same as the set of points on the sphere, topologically speaking. The set of points on the sphere can only be viewed as the quotient SO(3)/S(2).
The paragraph motivating the alt-az convolution on page 4 is not very clear, and some claims are questionable. I agree that local SO(2) invariance is too limiting. But it is not true that rotating filters is not effective in planar/volumetric CNNs, as shown by many recent papers on equivariant networks. I would suggest rewriting this paragraph to make it clearer and less speculative, and acknowledge that although rotating filters might increase computational complexity, it has often been shown very effective.
# Other comments
The experiments show that the method is quite effective. For instance, the SHREC17 results are on par with Cohen et al. and Esteves et al., presumably at a significantly reduced computational cost. That they do not substantially outperform these and other methods is likely due to the input representation, which is lossy, leading to a maximal performance shared by all three methods. An application to omnidirectional vision might more clearly show the strength of the method, but this would be a lot of work so I do not expect the authors to do that for this paper.
It would be nice to see a more direct comparison between the three definitions of spherical convolution (general SO3, isotropic S2, and anisotropic S2). Right now, the numbers reported in Cohen et al. and Esteves et al. are copied over, but there are probably many differences between the precise setup and architectures used in these papers. It would be interesting to see what happens if one uses the same architecture on a number of problems, changing only the convolution in each case.
Initially, I was a bit puzzled about why SO(3) augmentation seems to reduce accuracy in table 1. I think this is because SO(3) augmentation actually makes the classification problem harder if the input is initially aligned. Some more explanation / discussion would be good.
It would be nice to explain the spherical parameterization in more detail. Is this operation itself rotation equivariant?
Typos & minor issues
- Abstract: "to extract non-trivial features". The word non-trivial really doesn't add anything here. Similarly "offers multi-level feature extraction capabilities" is almost meaningless since all DL methods can be said to do so.
- Below eq. 5, D_R^{-1} should equal D_R(-omega, -nu, -phi). The order is reversed when inverting.
- "Different notations of convolutions" -> notions
- "For spherical functions there is no consistent and well defined convolution operators." As discussed above, the issue is quite a bit more subtle. There are exactly two well-defined convolution operators, but they have some characteristics deemed undesirable by the authors.
- "rationally symmetric" -> rotationally
- "exact hierarchical spherical patterns" -> extract
- It seems quite likely that the unpacking of the icosahedral/hexagonal grid as done in this paper has been studied before in other fields. References would be in order. Similarly, hexagonal convolution has a history in DL and outside.
- Bottom of page 7, capitalize "for".
- "principle curvatures" -> principal.
- "deferent augmentation modes" -> different
- "inspite" -> in spite
- "reprort" -> report
- "utlize" -> utilize
- "computer the convolution" -> compute
# Conclusion
Although the alt-az convolution lacks the mathematical elegance of the general anisotropic and azimuthally isotropic spherical convolutions, it still seems like a practically useful operation for some kinds of data, particularly when implemented using the homogeneous icosahedral/hexagonal grid and fast algorithm presented in this paper. Hence, I would wholeheartedly recommend acceptance of this paper if the authors correct the factual errors (e.g. the claim of SO(3)-equivariance) and provide a clear discussion of the issues. For now I will give an intermediate rating to the paper.
[1] Kondor, Trivedi, "On the Generalization of Equivariance and Convolution in Neural Networks to the Action of Compact Groups"


Dropout is a simple yet effective technique to improve the generalization performance and prevent overfitting in deep neural networks (DNNs). In this paper, we discuss three novel observations about dropout to better understand the generalization of DNNs with rectified linear unit (ReLU) activations: 1) dropout is a smoothing technique that encourages each local linear model of a DNN to be trained on data points from nearby regions; 2) a constant dropout rate can result in effective neural-deactivation rates that are significantly different for layers with different fractions of activated neurons ; and 3) the rescaling factor of dropout causes an inconsistency to occur between the normalization during training and testing conditions when batch normalization is also used. The above leads to three simple but nontrivial improvements to dropout resulting in our proposed method "Jumpout." Jumpout samples the dropout rate using a monotone decreasing distribution (such as the right part of a truncated Gaussian), so the local linear model at each data point is trained, with high probability, to work better for data points from nearby than from more distant regions. Instead of tuning a dropout rate for each layer and applying it to all samples, jumpout moreover adaptively normalizes the dropout rate at each layer and every training sample/batch, so the effective dropout rate applied to the activated neurons are kept the same. Moreover, we rescale the outputs of jumpout for a better trade-off that keeps both the variance and mean of neurons more consistent between training and test phases, which mitigates the incompatibility between dropout and batch normalization. Compared to the original dropout, jumpout shows significantly improved performance on CIFAR10, CIFAR100, Fashion-MNIST, STL10, SVHN, ImageNet-1k, etc., while introducing negligible additional memory and computation costs.

The paper proposes yet another variant of the celebrated Dropout algorithm. Specifically, the proposed method attempts to address the obvious drawbacks of Dropout: (i) the need to heuristically select the Dropout rate; and (ii) the universality of this selection across a layer.
As the authors have admitted in the paper (Sec. 1.2), there is a variety of methods already addressing the same problem. They argue that contrary to some of these methods "jumpout does not rely on additional trained models: it adjusts the dropout rate solely based on the ReLU activation patterns. Moreover, jumpout introduces negligible computation and memory overhead relative to the original dropout methods, and can be easily incorporated into existing model architectures."
However, this is argument is certainly untrue and rather misleading. The works of Kingma et al. (2015) and Molchanov et al. (2017), that the authors cite, does not introduce additional trained models. In addition, there is additional related work that the authors do not cite, but ought to:
[1] Yarin Gal, Jiri Hron, Alex Kendall, "Concrete Dropout," Proc. NIPS 2017.
[2] Yingzhen Li, Yarin Gal, "Dropout Inference in Bayesian Neural Networks with Alpha-divergences," Proc ICML 2017.
[3] Harris Partaourides, Sotirios Chatzis, “Deep Network Regularization via Bayesian Inference of Synaptic Connectivity,” J. Kim et al. (Eds.): PAKDD 2017, Part I, LNAI 10234, pp. 30–41, 2017.
These methods also address a similar problem, without introducing extra networks or imposing extra costs art inference time. Thus, citing them, as well as COMPARING to them, is a necessity for this paper to be convincing.
These crucial shortcoming aside, there are various theoretical claims in this paper that are not sufficiently substantiated. To begin with, the arguments used in the last paragraph of page 4 seem at least speculative; then,  the authors proceed to propose a solution to the alleged problem in the beginning of page 5. They suggest sampling from a truncated Gaussian, but they do not elaborate on why this selection solves the problem; they limit themselves to noting that other selections, such as the Beta distribution, may also be considered in the future. We must also underline that [3] have suggested exactly that; sampling from a Beta.
Finally, the last two modifications the authors propose seem reasonable, yet they are extremely heuristic. No one knows (which can be guaranteed through theoretical proofs or solid experimental evidence) that without these the method would not work. In addition, previous papers, e.g. [1-3] achieve similar goals in a principled fashion (ie by inferring proper posterior densities); without experimental comparisons, nobody knows which paradigm is best to adopt.


Modeling sequential data has become more and more important in practice. Some applications are autonomous driving, virtual sensors and weather forecasting. To model such systems, so called recurrent models are frequently used. In this paper we introduce several new Deep Recurrent Gaussian Process (DRGP) models based on the Sparse Spectrum Gaussian Process (SSGP) and the improved version, called Variational Sparse Spectrum Gaussian Process (VSSGP). We follow the recurrent structure given by an existing DRGP based on a specific variational sparse Nyström approximation, the recurrent Gaussian Process (RGP). Similar to previous work, we also variationally integrate out the input-space and hence can propagate uncertainty through the Gaussian Process (GP) layers. Our approach can deal with a larger class of covariance functions than the RGP, because its spectral nature allows variational integration in all stationary cases. Furthermore, we combine the (Variational) Sparse Spectrum ((V)SS) approximations with a well known inducing-input regularization framework. For the DRGP extension of these combined approximations and the simple (V)SS approximations an optimal variational distribution exists. We improve over current state of the art methods in prediction accuracy for experimental data-sets used for their evaluation and introduce a new data-set for engine control, named Emission.

Overall Score: 7/10.
Confidence Score: 7/10.
Detailed Comments: This paper introduces various Deep Recurrent Gaussian Process (DRGP) models based on the Sparse Spectrum Gaussian Process (SSGP) models and the Variational Sparse Spectrum Gaussian Process (VSSGP) models. This is a good paper and proposed models are very sound so I recommend for acceptance although as main weakness I can say that is very technical so it can be difficult to follow. Adding more intuitive ideas, motivation and maybe a figure for each step would be a solution. Apart from that it is a really good paper, congratulations.
Related to: RNN models and Sparse Nystrom approximation.
Strengths: Models are very sound, solutions are solid, the proposed methodology is correct and the empirical results and experiments are valid and properly done.
Weaknesses: It is too difficult to follow and it is written in an extreme technical way. More intuitions and a proper motivation both in the abstract and introduction may be put in order to make the paper easier to read and, hence, more used by researchers and data scientists.
Does this submission add value to the ICLR community? : Yes it does, the experiments show the efficiency of the proposed methods in some scenarios and are valid methodologies.
Quality:
Is this submission technically sound?: Yes it is.
Are claims well supported by theoretical analysis or experimental results?: Experimental results prove empirically the methods and appendixes show the analysis performed in a clear and elegant way.
Is this a complete piece of work or work in progress?: Complete piece of work.
Are the authors careful and honest about evaluating both the strengths and weaknesses of their work?: Yes, and I would enfatize that I have liked that some experiments are won by other methods such as GP-LSTM, they are very honest.
Clarity:
Is the submission clearly written?: Yes, but it is difficult for newcomers due to the reasons that I have stated before.
Is it well organized?: Yes it is.
Does it adequately inform the reader?: Yes it is.
Originality:
Are the tasks or methods new?: Yes, they are sound.
Is the work a novel combination of well-known techniques?: Yes it is.
Is it clear how this work differs from previous contributions?: Yes.
Is related work adequately cited?: Yes, being a strength of the paper.
Significance:
Are the results important?: I would argue that they are and are a clear alternative to consider in order to solve these problems.
Are others likely to use the ideas or build on them?: If the paper is written in a more friendly way, yes.
Does the submission address a difficult task in a better way than previous work?: Yes I think.
Does it advance the state of the art in a demonstrable way?: Yes, empirically.
Arguments for acceptance: Models are very sound, solutions are solid, the proposed methodology is correct and the empirical results and experiments are valid and properly done
Arguments against acceptance: Clarity of the paper.
Minor issues and typos:
-> (V)SS not defined before being used.
-> Abstract should be rewritten adding a motivation and focusing more on the problems being solved and less in the details of the solutions.
-> Recurrent indexes that go backwards (i) of Eq. 1. should be explained why are going backwards before being used like that. Newcomers may be confused.
-> Section 2 writing style lacks a bit of cohesion, relating the paragraphs may be a solution.
-> Q is not defined in section 3.1 paragraph 1.
-> A valid covariance function must produce a PSD matrix, put that in section 3.1.
-> I do not see how U marginalizes in Eq. 7, kind of confused about that, I think that it should be p(y|X,U).
-> Section 3.4 statistics should be explained.
Reading thread and authors response rebuttal decision:
=================================================
I consider that the authors have perfomed a good rebuttal and reading the other messages and the authors response I also consider that my issue with clarity is solved. Hence, I upgrade my score to 7 and recommend the paper for publication.