THE COMPARATIVE POWER OF RELU NETWORKS AND POLYNOMIAL KERNELS IN THE PRESENCE OF SPARSE LATENT STRUCTURE
There has been a large amount of interest, both in the past and particularly recently, into the relative advantage of different families of universal function approximators, for instance neural networks, polynomials, rational functions, etc. However, current research has focused almost exclusively on understanding this problem in a worstcase setting: e.g. characterizing the best L 1 or L ∞ approximation in a box (or sometimes, even under an adversarially constructed data distribution.) In this setting many classical tools from approximation theory can be effectively used. However, in typical applications we expect data to be high dimensional, but structured -so, it would only be important to approximate the desired function well on the relevant part of its domain, e.g. a small manifold on which real input data actually lies. Moreover, even within this domain the desired quality of approximation may not be uniform; for instance in classification problems, the approximation needs to be more accurate near the decision boundary. These issues, to the best of our knowledge, have remain unexplored until now. With this in mind, we analyze the performance of neural networks and polynomial kernels in a natural regression setting where the data enjoys sparse latent structure, and the labels depend in a simple way on the latent variables. We give an almost-tight theoretical analysis of the performance of both neural networks and polynomials for this problem, as well as verify our theory with simulations. Our results both involve new (complex-analytic) techniques, which may be of independent interest, and show substantial qualitative differences with what is known in the worst-case setting.

The paper studies the representational power of two-layer ReLU networks and polynomials for approximating a linear generative model for data with sparsity in the latent vector. They show that ReLU networks achieve optimal rate whereas low degree polynomials get a much worse rate.
Overall, the results are strong, the authors provide a lower bound on the degree of polynomial needed to approximate the model indicating the power of non-linearity. The observation of moving away from uniform approximators is well-motivated. The approximation theorem for ReLU is intriguing and uses new ideas which I have not seen before and are potentially useful in other applications. So far, only rational functions have been able to give such approximation guarantees. However, the motivation for studying sparse linear regression from a representation view-point is not very clear. Ideally, you would like to study representation for more complex models.
Questions/Comments:
- Related work is missing prior work at the intersection of kernel methods and neural networks, please update.
- Define notation before using, for example, \rho_\tau^{⨂m}
- Expand proof sketches, they are not very clear, also full proofs are written with not much detail.
- Is the dependence on \mu tight? The current dependence sort of suggests that you need the observation matrix to be very close to identity.
- Proof of Lemma B.1 is unclear, could you explain how you deduce the lemma from the inequality?


Model compression has been introduced to reduce the required hardware resources while maintaining the model accuracy. Lots of techniques for model compression, such as pruning, quantization, and low-rank approximation, have been suggested along with different inference implementation characteristics. Adopting model compression is, however, still challenging because the design complexity of model compression is rapidly increasing due to additional hyper-parameters and computation overhead in order to achieve a high compression ratio. In this paper, we propose a simple and efficient model compression framework called DeepTwist which distorts weights in an occasional manner without modifying the underlying training algorithms. The ideas of designing weight distortion functions are intuitive and straightforward given formats of compressed weights. We show that our proposed framework improves compression rate significantly for pruning, quantization, and low-rank approximation techniques while the efforts of additional retraining and/or hyper-parameter search are highly reduced. Regularization effects of DeepTwist are also reported.

This paper proposed a general framework, DeepTwist, for model compression. The so-called weight distortion procedure is added into the training every several epochs. Three applications are shown to demonstrate the usage of the proposed approach.
Overall, I think the novelty of the paper is very limited, as all the weight distortion algorithms in the paper can be formulated as the proximal function in proximal gradient descent. See http://www.stat.cmu.edu/~ryantibs/convexopt-S15/scribes/08-prox-grad-scribed.pdf for a reference.
Specifically, the proposed framework can be easily reformulated as a loss function plus a regularizer for proximal gradient. Using gradient descent (GD), there will be two steps: (1) finding a new solution using GD, and (2) project the new solution using proximal function. Now in deep learning, since SGD is used for optimization, several steps are need to locate reasonable solutions, i.e. the Distortion Step in the framework. Then proximal function can be applied directly after Distortion Step to project the solutions. In this way, we can easily see that the proposed framework is a stochastic version of proximal gradient descent. Since SGD is used for training, several minibatches are needed to achieve a relatively stable solution for projection using the proximal function, which is exactly the proposed framework in Fig. 1.
PS: After discussion, I think the motivation of the method is not clear to understand why the proposed method works.


Meta-learning is a powerful tool that learns how to quickly adapt a model to new tasks. In the context of reinforcement learning, meta-learning algorithms can acquire reinforcement learning procedures to solve new problems more efficiently by meta-learning prior tasks. The performance of meta-learning algorithms critically depends on the tasks available for meta-training: in the same way that supervised learning algorithms generalize best to test points drawn from the same distribution as the training points, meta-learning methods generalize best to tasks from the same distribution as the meta-training tasks. In effect, meta-reinforcement learning offloads the design burden from algorithm design to task design. If we can automate the process of task design as well, we can devise a meta-learning algorithm that is truly automated. In this work, we take a step in this direction, proposing a family of unsupervised meta-learning algorithms for reinforcement learning. We describe a general recipe for unsupervised meta-reinforcement learning, and describe an effective instantiation of this approach based on a recently proposed unsupervised exploration technique and model-agnostic meta-learning. We also discuss practical and conceptual considerations for developing unsupervised meta-learning methods. Our experimental results indicate that unsupervised meta-reinforcement learning effectively acquires accelerated reinforcement learning procedures without the need for manual task design, significantly exceeds the performance of learning from scratch, and even matches performance of meta-learning methods that use hand-specified task distributions in many environments.

The paper considers a particular setting of so-called meta-reinforcement learning (meta-RL) where there is a distribution over reward functions (the transition function is fixed) and with some access to this distribution, the goal is to produce a learning algorithm that  "learns well" on the distribution. At training time, a sample of reward functions are drawn and an algorithm returns a learning program that at test time, can reinforcement learn the test environment as specified by a test reward function. The paper proposes to generate a set of training reward functions instead of relying on some "manual" specification, thus the term "unsupervised" in the title. It also proposes an algorithm, basing on a recent work in skill discovery (DIAYN, not yet peer-reviewed), to find such reward functions.
Firstly, the exposition is hard to follow. For example, the "Task acquisition via random discriminators" subsection, without first mentioning DIAYN, seems out-of-context: what is D_phi_rand(z|s)? a joint distribution of (reward function, state) makes no sense. It only makes sense when there is a stochastic process, e.g. MDP coupled with a policy.
Secondly, the reasoning is very informal based on a vague vocabulary (not trying to pick on the authors, these are not uncommon in deep learning literature) are used without rigorous arguments. Section 3.1 brought up a natural objection -- I applaud the authors' self-critique -- based on "no free lunch theorem," but it dismisses it via "the specific choice for the unsupervised learning procedure and meta-learning algorithm can easily impose an inductive bias" without specifying what (and how) choice leads to what "inductive biases." This is crucial as the authors seem to suggest that although the "inductive bias" is important -- task design expresses such -- an unsupervised method, which requires no supervision, can do as well.
Thirdly, the baseline comparison seems inappropriate to me. The "fair" baseline the authors proposed was to RL a test task from scratch. But this is false as the meta-RL agent enjoys access to the transition dynamics (controlled Markov process, CMP in the paper) during the so-called meta-training (before meta-testing on the test task). In fact, a more appropriate baseline would be initialize an RL agent with with a correct model (if sample complexity in training is not a concern, which seems to be the case as it was never addressed in the paper) or a model estimated from random sample transitions (if we are mindful of sample complexity which seems more reasonable to me). One may object that a (vanilla) policy-gradient method cannot incorporate an environment model but why should we restrict ourselves to these model-free methods in this setting where the dynamics can be accessed during (meta-)training?
Pros:
1. It connects skill discovery and meta-RL. Even though such connection was not made explicitly clear in the writing, its heavy reliance on a recent (not yet peer-reviewed) paper suggests such. It seems to want to suggest it through a kind of "duality" between skills/policies and rewards/tasks (z in the paper denotes the parameter of a reward function and also the parameter of policy). But are there any difference between the two settings?
Cons:
1. The writing is imprecise and often hard to follow.
2. The setting considered is not well motivated. How does an unsupervised method provide the task distribution before seeing any tasks?
3. The restriction of tasks to different reward functions made the proposed baseline seem unfairly weak.
In summary, I could not recommend accepting this work as it stands. I sincerely hope that the authors will be more precise in their future writing and focus on articulating and testing their key hypotheses.


GRAPH WAVELET NEURAL NETWORK
We present graph wavelet neural network (GWNN), a novel graph convolutional neural network (CNN), leveraging graph wavelet transform to address the shortcomings of previous spectral graph CNN methods that depend on graph Fourier transform. Different from graph Fourier transform, graph wavelet transform can be obtained via a fast algorithm without requiring matrix eigendecomposition with high computational cost. Moreover, graph wavelets are sparse and localized in vertex domain, offering high efficiency and good interpretability for graph convolution. The proposed GWNN significantly outperforms previous spectral graph CNNs in the task of graph-based semi-supervised classification on three benchmark datasets: Cora, Citeseer and Pubmed.

This is an empirical paper that proposes to design wavelets on graphs, that can be integrated to neural networks on graphs. It permits to reducing the number of parameters of the « convolution » and exploits the sparsity of sparse weighted graphs for computations. I think it’s an interesting work.
The perspective I enjoy in using wavelets is that they typically provide a good trade-off in localization in the spectral and graph domains. For instance, large eigenvalues of the laplacian could be potentially captured in a more stable way. This type of work might be a first step.
Pros :
- good numerical results
- nice incorporation of structure via wavelets
Cons :
- Sometimes the paper is not really clear
I have severals comments:
1/ (1) « Some subsequent works devote to making spectral methods spectrum-free… avoiding high computational cost ». I think also those types of representations can be potentially unstable. That’s a second reason.
2/ I’m not sure to understand the point (1) of the fourth paragraph of the introduction. (1.)« graph wavelet does not depend on the eigen decomposition of Laplacian matrix ». Does it mean numerically ? This sentence is not clear, because even numerically, it can be done in a dependent way to this matrix. However, if it is implied that the fastest algorithm can be obtained without eigen-decomposition of the Laplacian Matrix, in a cheap way, then I agree and a small rephrasing could be nice.
3/ Please remove all the sentences that are supposed to be an introduction for a section, i.e. the sentences between 2 and 2.1 (« We use graph… ») and 3 and 3.1 (« In many research… »). They are poorly written and do not help the reader.
5/ .(2.2) “However, such a polynomial approximation also limits the flexibility to define appropriate convolution on graph” I’m not sure to understand. In the paper you refer to, the set of filters span the polynomial of degree less than n, of a diagonalizable matrix of size nxn. Thus, lagrange polynomial basis could be used to interpolate any desired values? Does it signify learning non-diagonal(in the appropriate basis) matrix?
6/ (2.3) $s$ how is this parameter chosen, crossvalidation? Is it adjusted such that the singular values of $\psi_s$ have a certain decay? Why is $s$ constant across layers? How is the spectrum of psi_s? Is it well conditionned? A huge difference with (Hammond et al,2011) is that they use a collection of wavelets. Did you consider this kind of approach ? Is there a loss of information in some cases?(like if $\psi_s$ has a fast decay)
7/ (2.3) “The matrix $\psi_s$ and $\psi_s^{-1}$ are both sparse”. This is a critical affirmation which is not in general true. It is possibly sparse if the weighted laplacian graph is sparse as well, as explained by the remark after theorem 5.5 of page 16 of (Hammond et al, 2011). However, I do agree this typically happens in the application you present.
8/ The (c) of Figure 1 is missing.(in my version at least)
9/ In section 5.3, why not trying to compare the number of parameters with other papers? I think also more results, with maybe a bit higher performances, could be reported, such as: GraphSGAN, GAT. But that’s fine to me.
10/ Appendix A, isn’t it a simple rephrasing of (Hammond et al, 2011)?
11/ How do you compare in term of interpretability with [1]?
12/ Just as a suggestion and/or a comment: it seems similar to approaches such as lifting schemes(which basically builds wavelets on graphs/manifolds), except that there is no learning involved. (e.g. [2]) I think there could be great connexions.
[1] Relational inductive biases, deep learning, and graph networks, Battaglia et al 2011?
[2] THE LIFTING SCHEME: A CONSTRUCTION OF SECOND GENERATION WAVELETS, Wim Sweldgens


In this paper, we describe a general framework to systematically analyze current neural models for multi-task learning, in which we find that existing models expect to disentangle features into different spaces while features learned in practice are still entangled in shared space, leaving potential hazards for other training or unseen tasks. We propose to alleviate this problem by incorporating a new inductive bias into the process of multi-task learning, that different tasks can communicate with each other not only by passing hidden variables but gradients explicitly. Experimentally, we evaluate proposed methods on three groups of tasks and two types of settings (IN-TASK and OUT-OF-TASK). Quantitative and qualitative results show their effectiveness.
Multi-task learning has been established as an effective learning method in deep learning, driving state-of-theart results to new levels in a number of tasks, ranging from computer vision (Misra et al., 2016;Zamir et al., 2018) to natural language processing (Collobert & Weston, 2008;Luong et al., 2015).
While various kinds of neural multi-task learning approaches have been proposed, a commonality is the two types of feature spaces they defined to store taskspecific features and task-invariant features. And then interaction mechanisms among different feature spaces will be explored in diverse ways, such as hard sharing (Collobert & Weston, 2008) or soft sharing (Misra et al., 2016;Ruder12 et al., 2017) schemes , flat Liu et al., 2016) or structural sharing schemes (Hashimoto et al., 2016;Søgaard & Goldberg, 2016).
Despite their great success, a potential limitation of most existing works is that shared space is entangled with private features since it is not enforced through the use of explicit loss functions (Bousmalis et al., 2016). Specifically, we would have extracted shared features and disentangled them into separated spaces. However, in reality, methods in use just collect all the features together into a common space, instead of learning shared rules across different tasks. This problem, which we define as "pretend-to-share", clearly suffers from the following issues: 1) IN-TASK SETTING: shared space is redundant and contaminated by task-specific features, leaving potential hazards for other training tasks. 2) OUT-OF-TASK SETTING: the learned model can not effectively adapt to new tasks since a real common structure is not discovered.
A few works explore this problem with adversarial training and shows that explicitly disentangling representations into private and shared spaces will improve model's performance (Bousmalis et al., 2016;Liu et al., 2017). In this paper, we propose an orthogonal technique to alleviate above problem. Specifically, we first describe a general framework for multi-task learning called Parameters ReadWrite Networks (PRaWNs) that can simply abstract the commonalities between existing typical neural models for multi-task learning. Then, we explain the reason for pretend-to-share problem within PRaWN that each task has the permission to modify shared parameters without any communication protocol, which makes it easier to update parameters along diverse directions.
Therefore, the problem is translated into this question: how to address the conflict among different tasks happening in the process of parameter updating. We resolve this conflict by introducing an inductive bias for multi-task learning. Specifically, different tasks can communicate with each other by sending (or receiving) both hidden variables and gradients. As shown in Fig.1(b), during the forward phase, tasks A, B, C are capable of sending (and receiving) the gradients to (and from) each other. Then during the backward phase, each task is aware of how other tasks are modifying the parameters, making it possible to modify in a more consistent way. Concretely, by communicating with gradients, we can apply consistency constraint to each task during the process of parameter updating, which can prevent private features from slipping into shared space thereby alleviating the pretend-to-share problem. Technically, we explore two kinds of communication mechanisms for gradient passing: pairwise and list-wise communication, where the latter one can take task relatedness into consideration.
We evaluate our models on three groups of multi-task learning datasets, range from natural language processing tasks to computer vision. The results show that our models are more expressive since we can learn a shared space which are more pure, driving task-dependent features away.
To summarize, we make the following contributions:
• We describe a general framework, which can not only help us analyze current models in an unified way [Section 3], but find potential limitations, and move forward with suitable approaches [Section 4].
• We study a task-agnostic problem existing in many multi-task learning models, and propose to alleviate it by allowing different tasks to communicate with gradients.
• We present an elaborate qualitative analysis, giving some insight into the internal mechanisms of proposed gradient passing method.

This paper tries to address the"pretend-to-share" problem by designing the gradient passing schemes in which the gradient updates to specific parameters of tasks are passed to the shared parameters. Besides, the authors summarize existing multitask learning algorithms in a framework called Parameters Read-Write Networks (PRAWN).
Pros:
- The view of putting existing multi-task learning algorithms in a read-write framework is quite intriguing and inspiring.
Cons:
- Motivation: The whole paper is assumed to address the "pretend-to-share" problem, while the authors never provide any evidence that such problem really exists for any other algorithm. It seems to be an assumption without any support.
- Method:
   - Though the read-write framework is very interesting, the authors do not clearly present it, so that the readers can be totally get lost. For example, what do you mean by writing {\Theta^{*r}_k - \theta^{swr}_k}? In the line of structural read-op, where are \theta_3 and \theta_4  in the column of the constituent para. ? What do you mean by writing the equation (4)? How do you define g() in equation (8)? This is a research paper which should be as clear as possible for the readers to reproduce the results, rather than a proposal only with such abstract and general functions defined.
   - In the list-wise communication scheme, you define the task relationship in equation (11). The problem is how do you evaluate the effectiveness of such definition, since massive works in multitask learning pursue to learn the task relationship automatically to guarantee the effectiveness instead of such heuristic definition.
- Related works: The authors do not clearly and correctly illustrate the connections between this work and meta-learning/domain adaptation. To my best knowledge, meta-learning, including MAML (Finn et al. 2017), can obviously solve both in-task setting and out-task setting. In some sense, I think this work is almost equivalent to MAML.
- Experiments:
   - First, several state-of-the-art baselines including MAML and cross-stitch networks should be compared. Specifically, for the text classification dataset, there have been a lot of domain adaptation works discovering the transferable pivots (shared features) and non-pivots (specific features), which the authors should be aware of and compare in Table 3.
   - The Figure 5 is not clear to me, and so is the discussion. The authors try to explain that the updating direction of shared parameters for PGP-SR is an integration of two private updating directions. I tried hard to understand, but still think that Figure 5(a) is even better than Figure 5(b). The updating direction of the shared parameters is almost the same as the cyan line.
- Presentation: there are so many grammatical errors and typos. For example,
   - In the introduction, "...datasets, range from natural" -> "...datasets, ranging from natural"
  - In the related work, "and they propose address it with adversarial" -> "and they propose to address it with adversarial"
 - In the beginning of Section 4, " an general" -> "a general"


Adversarial examples remain an issue for contemporary neural networks. This paper draws on Background Check (Perello-Nieto et al., 2016), a technique in model calibration, to assist two-class neural networks in detecting adversarial examples, using the one dimensional difference between logit values as the underlying measure. This method interestingly tends to achieve the highest average recall on image sets that are generated with large perturbation vectors, which is unlike the existing literature on adversarial attacks (Cubuk et al., 2017). The proposed method does not need knowledge of the attack parameters or methods at training time, unlike a great deal of the literature that uses deep learning based methods to detect adversarial examples, such as Metzen et al. (2017), imbuing the proposed method with additional flexibility.

This paper addresses adversarial detection through the absolute-value difference between the two logit vector values of a DNN binary classifier, with one class associated with normal data and the other with adversarial data. Assignment of examples to an "adversarial" class is problematic in that adversarial examples are typically generated in regions for which training data is very sparse. To cope with this, the authors propose use of the Background Check calibration techniques recently proposed by Perello-Nieto et al. (ICDM 2016).
Here, BC is used to estimate probabilities in a sparse "background" class (here, the adversarial class)  through a form of interpolation based on foreground and background densities. The underlying distributional assumption used for estimating foreground densities was that of a gamma function.  Rather than using BC's affine bias for estimating background density from the foreground density, the authors adapt it by raise the weighting for the "adversarial" decision to the fifth power. Unfortunately, no justification for this choice is given, other than to say that this was done with "domain knowledge informing the use of a power value".
In their experimentation, the authors generate from CIFAR-10 data four kinds of adversarial attacks: noise alone, images with moderate noise, clear images with noticeable noise, and clear images with imperceptible noise. For a variety of attacks, they showed (in Table 2) differences between the average recall for normal examples vs the average recall for normal plus adversarial images. However, without knowing the proportion of adversarial examples used in testing, the significance of the reported differences cannot be judged. They also list the true positive rates of adversarial examples, which showed much variation from experiment to experiment (trending to rather poor performance for attacks with imperceptible noise). Again, the significance of the results cannot be judged without knowing the false negative rate, true negative rate, etc. Moreover, the results are reported without clearly identifying two of the attacks used ("Mom." is presumably Dong et al.'s attack using momentum in gradient descent, and Miyato et al.'s "VAT" is not properly introduced in the related work). Crucially, no evaluation of their method is made with respect to other adversarial detection strategies.
Pros:
* Overall, the calibration approach is well motivated, and likely to be of some benefit.
* The paper is generally readable and understandable. The issues behind calibration and the use of BC are well explained.
Cons:
* The result is a simple and straightforward application of an existing technique - not greatly original.
* Many design choices in the model (particularly the raising of one of the weights to a seemingly-arbitrary power) are mysterious. No indication is given as to other alternatives or how they might perform.
* The experimental results are inadequate to judge the impact of the proposed calibration approach.
* There is no comparison against other detection methods.


PREDICT THEN PROPAGATE: GRAPH NEURAL NETWORKS MEET PERSONALIZED PAGERANK
Neural message passing algorithms for semi-supervised classification on graphs have recently achieved great success. However, for classifying a node these methods only consider nodes that are a few propagation steps away and the size of this utilized neighborhood is hard to extend. In this paper, we use the relationship between graph convolutional networks (GCN) and PageRank to derive an improved propagation scheme based on personalized PageRank. We utilize this propagation procedure to construct a simple model, personalized propagation of neural predictions (PPNP), and its fast approximation, APPNP. Our model's training time is on par or faster and its number of parameters on par or lower than previous models. It leverages a large, adjustable neighborhood for classification and can be easily combined with any neural network. We show that this model outperforms several recently proposed methods for semi-supervised classification in the most thorough study done so far for GCN-like models. Our implementation is available online.

The thurst behind this paper is that graph convolutional networks (GCNs) are constrained by construction
to focus on small neighborhoods around any given node. Large neighborhoods introduce in principle
a large number of parameters (while as the authors point out, weight sharing is an option to avoid this issue),
plus even worse oversmoothing may occur. Specifically, Xu et al. (2018) showed that for a k-layer GCN one can
think of the influence score of a node x on node y as the probability  that a walker that starts at x,
lands on y after k steps of random walk (modulo some details).
Therefore, as k increases the random walks reaches its stationary distribution, forgetting any local information that is useful,
e.g., for node classification. To avoid this problem, the authors propose the following: use personalized Pagerank
instead of the standard Markov chain of Pagerank. In PPR there is a restart probability, which allows
their algorithm to avoid “forgetting” the local information around a walk, thus allowing for an arbitrary
number of steps in their random walk. The authors define two methods PEP, and PEPa based on PPR. The latter
method is faster in practice since it approximates the PPR.
A key advantage of the proposed method is the separation of the node embedding part from the propagation scheme. In this sense,
following the categorization of existing methods into three categories, PEP is a hybrid of message passing algorithms,
and random walk based node embeddings. The experimental evaluation tests certain basic properties of the proposed method. One interesting performance feature of
PEP and PEPa is that they can perform well using few training examples. This is valuable especially when obtaining labeled
examples is expensive.  Finally, the authors compare their proposed methods against state-of-the-art GCN-based methods.
Some remarks follow.
- The idea of using PPR for node embeddings has been suggested in recent prior work “LASAGNE: Locality and structure aware graph node embeddings”
By Faerman et al.  While according to the authors’ categorization of the existing methods in the intro, LASAGNE
falls under the “random walk” family  of methods, the authors should compare against it.

- Continuing the previous point,  even simpler baselines would be desirable. How inferior is for instance
an approach on one-vs-all classification using the approximate personalized Pagerank node embedding and
support vector machines?

- Also, the authors mention “since our datasets are somewhat similar…”. Please clarify with respect to
which aspects? Also, please use datasets that are different. For instance, see the LASAGNE paper for
more datasets that have different number of classes.
- In the experiments the authors use two layers for fair comparison. Given that one of the advantages of the
proposed method is the  ability to have more layers without suffering from the GCN shortcomings
with large neighborhood exploration, it would be interesting to see an experiment where the number of layers is a variable.


LABEL SUPER-RESOLUTION NETWORKS
We present a deep learning-based method for super-resolving coarse (lowresolution) labels assigned to groups of image pixels into pixel-level (highresolution) labels, given the joint distribution between those low-and highresolution labels. This method involves a novel loss function that minimizes the distance between a distribution determined by a set of model outputs and the corresponding distribution given by low-resolution labels over the same set of outputs. This setup does not require that the high-resolution classes match the lowresolution classes and can be used in high-resolution semantic segmentation tasks where high-resolution labeled data is not available. Furthermore, our proposed method is able to utilize both data with low-resolution labels and any available high-resolution labels, which we show improves performance compared to a network trained only with the same amount of high-resolution data. We test our proposed algorithm in a challenging land cover mapping task to super-resolve labels at a 30m resolution to a separate set of labels at a 1m resolution. We compare our algorithm with models that are trained on high-resolution data and show that 1) we can achieve similar performance using only low-resolution data; and 2) we can achieve better performance when we incorporate a small amount of highresolution data in our training. We also test our approach on a medical imaging problem, resolving low-resolution probability maps into high-resolution segmentation of lymphocytes with accuracy equal to that of fully supervised models.

Paper summary:
This paper presents a deep-learning based method for super-resolving low-resolution labels into high-resolution labels given the joint distribution between those low- and high- resolution labels. This is useful for many semantic segmentation tasks where high-resolution ground truth data is hard and expensive to collect. Its main contribution is a novel loss function that allows to minimize the distance between the distribution determined by a set of model outputs and the corresponding distribution given by low-resolution label over the same set of outputs. The paper also thoroughly evaluates the proposed method for two main tasks, the first being a land cover mapping task and the second being a medical imaging problem.
For the land cover application, adding low-resolution data to high-resolution data worsens the results when evaluating on the geographic area from which the high-resolution data was taken. However, when testing the model on new geographic areas and only adding the low-resolution data from this new area in training makes significant improvements.
Generally the paper is very well written, well structured, all explanations are clear, examples and figures are presented when needed and convey helpful information for the reader. The overall idea is fun, original, useful (especially in remote sensing) and is presented in a a convincing way. All major claims are supported by experimental evaluation. There are nevertheless a few concerns:
Major Concerns:
On a conceptual level, the main concern is that the paper assumes we are given a joint distribution of low and high resolution labels, “where we are given the joint distribution P(Y,Z)”, which seems the main limitation of this method. In fact, to correctly estimat this joint distribution either requires additional knowledge about low-resolution data such as the example presented on the NCLD data : “For instance, the “Developed, Medium Intensity” class [...] of the coarse classes”, or it requires actual high-resolution labelled data to correctly estimate this joint distribution. I think the paper would greatly benefit from including a section that discusses the impact of this limitation.
Another point is footnote 3 on page 5. This argument is valid but it would be more convincing to give a thorough explanation on why the choice of the presented loss function is better compared to the KL divergence based loss function or at least some evidence that the two perform similarly when evaluating the method.
Minor Concerns:
-	“such as CRFs or iterative evaluation” I would include a citation on this type of work.
-	Format of some references in the text need to be corrected, e.g. “into different land cover classes Demir et al. (2018); Kuo et al. (2018); Davydow et al. (2018); Tian et al. (2018).”


In recent years, deep neural networks have demonstrated outstanding performance in many machine learning tasks. However, researchers have discovered that these state-of-the-art models are vulnerable to adversarial examples: legitimate examples added by small perturbations which are unnoticeable to human eyes. Adversarial training, which augments the training data with adversarial examples during the training process, is a well known defense to improve the robustness of the model against adversarial attacks. However, this robustness is only effective to the same attack method used for adversarial training. Madry et al. (2017) suggest that effectiveness of iterative multi-step adversarial attacks and particularly that projected gradient descent (PGD) may be considered the universal first order adversary and applying the adversarial training with PGD implies resistance against many other first order attacks. However, the computational cost of the adversarial training with PGD and other multi-step adversarial examples is much higher than that of the adversarial training with other simpler attack techniques. In this paper, we show how strong adversarial examples can be generated only at a cost similar to that of two runs of the fast gradient sign method (FGSM), allowing defense against adversarial attacks with a robustness level comparable to that of the adversarial training with multi-step adversarial examples. We empirically demonstrate the effectiveness of the proposed two-step defense approach against different attack methods and its improvements over existing defense strategies.

Summary. The authors propose a novel adversarial training method, e2SAD, that relies on a two-step process for generating sets of two training adversarial samples for each clean training sample. The first step is a classical FGSM that yields the first adversarial sample. The second adversarial sample is calculated with a FGSM that is based on the cross-entropy between the probabilities generated by the first adversarial sample and the probabilities generated by the second adversarial sample. The method is computationally efficient (two forward/backward passes per clean sample) w.r.t. powerful iterative attacks such as IFGSM or PGD requiring 40+ steps and the authors claim it gives comparable results to adversarial training with multi-step attacks methods in white and black-box settings.
Clarity. Part 1 and 2 of the paper are well written and summarize the existing attacks/defense mechanisms, their pros and cons as well as the contributions clearly. The next sections could be made shorter (see comments below) to match ICLR’s recommended soft limit of 8 pages instead of the 10 pages hard limit. This would also help the reader grasp the key ideas faster and have a standard formatting (no negative spaces for instance).
Novelty. The idea of simulating the effect of iterative attacks using two distinct steps is novel and appealing to me. The first step increases the loss while the second step shifts the probability distributions apart.
Pros and cons.
(+) The paper is clear and easy to follow, although a bit long.
(+) The idea is interesting and clearly motivated in terms of computational efficiency and in terms of desired properties (Figure 2 illustrates this point well).
(-) Only one aspect of the idea is exploited in the article. It would be interesting to compare this method as an attacker (both in terms of performance and in terms of generated samples, see comment below). Powerful adversarial training should indeed rely on powerful generated adversarial samples.
(-) The results seem somewhat mitigated in terms of significance and conclusions drawn by the authors. Also, the experimental setup is quite light, notably the used CNN architectures are quite small and other datasets could have been used (also linked to the significance of the results).
Comments.
- Shorter paper. Here are suggested modifications for the paper that could help strengthen the impact of your paper. Section 3.1 could be almost entirely discarded as it brings no new ideas w.r.t sections 1 and 2. Figure 1 summarizes the method well, thus the description in Section 3.2 could be made shorter, especially when displaying Equation (8) right after Figure 1. This would then help reduce the size of Sections 3.2.1 and 3.2.2 (because Equation (8) and Figure 1 would prevent you from repeating claims made earlier in the paper). Algorithm 1 is straightforward and could be placed in Appendix. Conclusion and Result sections could be shortened a little as well (not as much as Section 3 though).
- Significance of the results. The significance of some results is unclear to me. Could the authors provide the standard deviation over 3 or 5 runs? For example, in rows 1, 3, 4, 5, 6 of Table 2, it is not clear it e2SAD performs better than FGSM adversarial training, thus raising the question of the necessity of Step 2 of the attack (which is the core contribution of the paper).
- Experimental setup. The last two rows of Table 1 are encouraging for e2SAD. However, the authors could introduce another dataset, e.g. CIFAR10 or 100 or even ImageNet restricted to 20 or 100 random classes/with fewer samples per class and use deeper modern CNN architectures like ResNets (even a ResNet18). Those models are widely adopted both in the research community and by the industry, thus defense mechanisms that provably work for such models can have a huge impact.
- Defense setup. Is the order of Steps 1 and 2 relevant? What if the authors use only iterations of Step 2?
- Attack setup. Here are a few suggestions for assessing your method in an attack setting: what is the precision of the network, without any defense, given an average dissimilarity L2 budget in the training/test samples, in a white/black box setting? How does it compare to standard techniques (e.g. FGSM, IFGSM, DeepFool, Carlini)? What happens if the authors use their method both for both defense and attack? Could the authors display adversarial samples generated by their method?
Conclusion. The idea presented in the paper is interesting, but (1) the experimental results are not entirely satisfactory for the moment and (2) only one aspect of the idea is exploited in the paper, which can be made more interesting and impactful while studying both attack and defense setups. I strongly encourage the authors to continue their research in this area due to the high potential impact and benefits for the whole community.


Dropout has been one of standard approaches to train deep neural networks, and it is known to regularize large models to avoid overfitting. The effect of dropout has been explained by avoiding co-adaptation. In this paper, however, we propose a new explanation of why dropout works and propose a new technique to design better activation functions. First, we show that dropout can be explained as an optimization technique to push the input towards the saturation area of nonlinear activation function by accelerating gradient information flowing even in the saturation area in backpropagation. Based on this explanation, we propose a new technique for activation functions, gradient acceleration in activation function (GAAF), that accelerates gradients to flow even in the saturation area. Then, input to the activation function can climb onto the saturation area which makes the network more robust because the model converges on a flat region. Experiment results support our explanation of dropout and confirm that the proposed GAAF technique improves performances with expected properties.

This paper offers the argument that dropout works not due to preventing coadaptation, but because it gives more gradient, especially in the saturated region. However, previous works have already characterized how dropout modifies the activation function, and also the gradient in a more precise way than what is proposed in this paper.
## Co-adaptation
co-adaptation does not seem to mean correlation among the unit activations. It is not too surprising units need more redundancy with dropout, since a highly useful feature might not always be present, but thus need to be replicated elsewhere.
Section 8 of this paper gives a definition of co-adaptation,
based on if the loss is reduced or increased based on a simultaneous change in units.
https://arxiv.org/abs/1412.4736
And this work, https://arxiv.org/abs/1602.04484, reached a conclusion similar to yours
that for some notion of coadaptation, dropout might increase it.
## Gradient acceleration
It does not seem reasonable to measure "gradient information flow" simply as the norm of the gradient, which is sensitive to scales, and it is not clear if the authors accounted for scaling factor of dropout in Table 2.
The proposed resolution, to add this discontinuous step function in (7) with floor is a very interesting idea backed by good experimental results. However, I think the main effect is in adding noise, since the gradient with respect to this function is not meaningful. The main effect is optimizing with respect to the base function, but adding noise when computing the outputs. Previous work have also looked at how dropout noise modifies the effective activation function (and thus its gradient). This work, http://proceedings.mlr.press/v28/wang13a.html, give a more precise characterization instead of treating the effect as adding a function with constant gradient multiplied by an envelop. In fact, the actual gradient with dropout does involve the envelope by chain rule, but the rest is not actually constant as in GAAF.


We propose a robust Bayesian deep learning algorithm to infer complex posteriors with latent variables. Inspired by dropout, a popular tool for regularization and model ensemble, we assign sparse priors to the weights in deep neural networks (DNN) in order to achieve automatic "dropout" and avoid over-fitting. By alternatively sampling from posterior distribution through stochastic gradient Markov Chain Monte Carlo (SG-MCMC) and optimizing latent variables via stochastic approximation (SA), the trajectory of the target weights is proved to converge to the true posterior distribution conditioned on optimal latent variables. This ensures a stronger regularization on the over-fitted parameter space and more accurate uncertainty quantification on the decisive variables. Simulations from large-p-small-n regressions showcase the robustness of this method when applied to models with latent variables. Additionally, its application on convolutional neural networks (CNN) leads to state-of-the-art performance on MNIST and Fashion MNIST datasets and improved resistance to adversarial attacks.

TITLE
Bayesian deep learning via stochastic gradient mcmc with a stochastic approximation adaptation
REVIEW SUMMARY
Fairly well written paper on SG-MCMC type inference in neural networks with slab and spike priors. In my view, the originality and significance is limited.
PAPER SUMMARY
The paper develops a method for sampling/optimization of a Bayesian neural network with slab and spike priors on the weights.
QUALITY
I belive the contribution is technically sound (but I have not checked all equations or the proof of Theorem 1). The empirical evaluation is not unreasonable, but also not strongly convincing.
CLARITY
The paper is fairly well written, but grammar and use of English could be slightly improved (not so important).
ORIGINALITY
The paper builds on existing work on EM-type algorithms for slab and spike models and SG-MCMC for Bayesian inference in neural networks. The novelty of the contribution is limited: The main contribution is the combination of the two methods and some theoretical results. I am not able to judge if there is significant originality in the theoretical results (Theorem 1 + Corr 1+2) but if I am not mistaken it is more or less an application of a known result to this particular setting?
SIGNIFICANCE
While I think the proposed algorithm is reasonable and most likely useful in practice, I am not sure the contribution is substantial enough to gain large interest in the community.
FURTHER COMMENTS
Figure 2 (d+e) are in my view not so useful for assessing the training/test performance, but I am not even completely sure what the figures shows, as there are no axis labels. I would prefer some results on the loss, perhaps averaged over multiple data sets.


ENERGY-CONSTRAINED COMPRESSION FOR DEEP NEURAL NETWORKS VIA WEIGHTED SPARSE PROJEC- TION AND LAYER INPUT MASKING
Deep Neural Networks (DNNs) are increasingly deployed in highly energyconstrained environments such as autonomous drones and wearable devices while at the same time must operate in real-time. Therefore, reducing the energy consumption has become a major design consideration in DNN training. This paper proposes the first end-to-end DNN training framework that provides quantitative energy consumption guarantees via weighted sparse projection and input masking. The key idea is to formulate the DNN training as an optimization problem in which the energy budget imposes a previously unconsidered optimization constraint. We integrate the quantitative DNN energy estimation into the DNN training process to assist the constrained optimization. We prove that an approximate algorithm can be used to efficiently solve the optimization problem. Compared to the best prior energy-saving methods, our framework trains DNNs that provide higher accuracies under same or lower energy budgets.

This paper describes a procedure for training neural networks via an explicit constraint on the energy budget, as opposed to pruning the model size as commonly done with standard compression methods.  Comparative results are shown on a few data sets where the proposed method outperforms multiple different approaches.  Overall, the concept is interesting and certainly could prove valuable in resource-constrained environments.  Still I retain some reservations as detailed below.
My first concern is that this paper exceeds the recommended 8 page limit for reasons that are seemingly quite unnecessary.  There are no large, essential figures/tables, and nearly the first 6 pages is just introduction and background material.  Likewise the paper consumes a considerable amount of space presenting technical results related to knapsack problems and various epsilon-accurate solutions, but this theoretical content seems somewhat irrelevant and distracting since it is not directly related to the greedy approximation strategy actually used for practical deployment.  Much of this material could have been moved to the supplementary so as to adhere to the 8 page soft limit.  Per the ICLR reviewer instructions, papers deemed unnecessarily long relative to this length should be judged more critically.
Another issue relates to the use of a mask for controlling the sparsity of network inputs.  Although not acknowledged, similar techniques are already used to prune the activations of deep networks for compression.  In particular, various forms of variational dropout essentially use multiplicative weights to remove the influence of activations and/or other network components similar to the mask M used is this work.  Representative examples include Neklyudov et al., "Structured Bayesian Pruning via Log-Normal Multiplicative Noise," NIPS 2017 and Louizos et al., "Bayesian Compression for Deep Learning," NIPS 2017, but there are many other related alternatives using some form of trainable gate or mask, possibly stochastic, to affect pruning (the major ML and CV conferences over the past year have numerous related compression papers).  So I don't consider this aspect of the paper to be new in any significant way.
Moreover, for the empirical comparisons it would be better to compare against state-of-the-art compression methods as opposed to just the stated MP and SSL methods from 2015 and 2016 respectively.  Despite claims to the contrary on page 9, I would not consider these to be state-of-the-art methods at this point.
Another comment I have regarding the experiments is that hyperparameters and the use of knowledge distillation were potentially tuned for the proposed method and then simultaneously applied to the competing algorithms for the sake of head-to-head comparison.  But to me, if these enhancements are to be included at all, tuning must be done carefully and independently for each algorithm.  Was this actually done?  Moreover it would have been nice to see results without the confounding influence of distillation to isolate sources of improvement, but no ablation studies were presented.
Finally, regarding the content in Section 5, the paper carefully presents an explicit bound on energy that ultimately leads to a constraint that is NP-hard just to project on to, although approximate solutions exist that depend on some error tolerance.  However, even this requires an algorithm that is dismissed as "complicated."  Instead a greedy alternative is derived in the Appendix which presumably serves as the final endorsed approach.  But at this point it is no longer clear to me exactly what performance guarantees remain with respect to the energy bound.  Theorem 3 presents a fairly inscrutable bound, and it is not at all transparent how to interpret this in any practical sense.  Note that after Theorem 3, conditions are described whereby an optimal projection can be obtained, but these seem highly nuanced, and unlikely to apply in most cases.
Additionally, it would appear that crude bounds on the energy could also be introduced by simply penalizing/constraining the sparsity on each layer, which leads to a much simpler projection step.  For example, a simple affine function of the L0 norm would be much easier to optimize and could serve as a loose bound on the energy, given that the latter should be a non-decreasing function of the L0 norm.  Any idea how such a bound compares to those presented given all the approximations and greedy steps that must be included?
Other comments:
- As an implementation heuristic, the proposed Algorithm 1 gradually decays the parameter q, which controls the sparsity of the mask M.  But this will certainly alter the energy budget, and I wonder how important it is to employ a complex energy constraint if minimization requires this type of heuristic.
- I did not see where the quantity L(M,W) embedded in eq. (17) was formally defined, although I can guess what it is.
- In general it is somewhat troublesome that, on top of a complex, non-convex deep network energy function, just the small subproblem required for projecting onto the energy constraint is NP-hard.  Even if approximations are possible, I wonder if this extra complexity is always worth it relative so simple sparsity-based compression methods which can be efficiently implemented with exactly closed-form projections.
- In Table 1, the proposed method is highlighted as having the smallest accuracy drop on SqueezeNet.  But this is not true, EAP is lower.  Likewise on AlexNet, NetAdapt has an equally optimal energy.


We introduce a new method for training GANs by applying the Wasserstein-2 metric proximal on the generators. This approach is based on the gradient operator induced by optimal transport theory, which connects the geometry of the sample space and the parameter space in implicit deep generative models. From this theory, we obtain an easy-to-implement regularizer for the parameter updates. Our experiments demonstrate that this method improves the speed and stability in training GANs in terms of wallclock time and Fréchet Inception Distance (FID) learning curves.

[summary]
This paper considers natural gradient learning in GAN learning, where the Riemannian structure induced by the Wasserstein-2 distance is employed. More concretely, the constrained Wasserstein-2 metric $d_W$, the geodesic distance on the parameter space induced by the Wasserstein-2 distance in the ambient space, is introduced (Theorem 1). The natural gradient on the parameter space with respect to the constrained Wasserstein-2 metric is then derived (Theorem 2). Since direct evaluation of $G^{-1}$ poses difficulty, the authors go on to considering a backward scheme using the proximal operator (3), yielding:
(i) The Semi-Backward Euler method is proposed via a second-order Taylor approximation of the proximal operator $d_W^2$ (Proposition 3).
(ii) From an alternative formulation for $d_W$ (Proposition 4), the authors propose dropping the gradient constraint to define a relaxed Wasserstein metric $d$, yielding a simple proximal operator given by the expected squared Euclidean distance in the sample space used as a regularizer (equation (4)). The resulting algorithm is termed the Relaxed Wasserstein Proximal (RWP) algorithm.
[pros]
The proposal provides an easy-to-implement drop-in regularizer framework, so that it can straightforwardly be combined with various generator update schemes.
[cons]
Despite all the theoretical arguments given to justify the proposal, the resulting proposal may simply be viewed as a naive application of the proximal operator.
[Quality]
See [Detailed comments] section below.
[Clarity]
This paper is basically clearly written.
[Originality]
Providing justification to the proximal operator approach in GAN learning via natural gradient with respect to the Riemannian structure seems original.
[Significance]
See [Detailed comments] section below.
[Detailed comments]
To the parameter space $\Theta\subset\mathbb{R}^d$, one can consider introducing several different Riemannian structures, including the conventional Euclidean structure and that induced by the Wasserstein-2 metric. Which Riemannian structure among all these possibilities would be natural and efficient in GAN training would not be evident, and this paper discusses this issue only in the very special single instance in Section 2.3. A more thorough argument supporting superiority of the Riemannian structure induced by the Wasserstein-2 metric would thus be needed in order to justify the proposed approach.
In relation to this, the result of comparison between WGAN-GP with and without SBE shown in Figure 5 is embarrassing to me, since it might suggest that the proposed framework aiming at performing Wasserstein natural gradient is not so efficient if combined with WGAN-GP. The natural gradient is expected to be efficient when the underlying coordinate system is non-orthonormal (Amari, 1998). Starting with the gradient descent iteration derived from the backward Euler method in (3), which is computationally hard, the argument in this paper goes on to propose two methods: the Semi-Backward Euler method via a second-order Taylor approximation to the backward Euler scheme (Proposition 3), and RWP in (4) via approximation (dropping of the gradient constraint and finite-difference approximation in the integral with respect to $t$) of an alternative simpler formulation for the Wasserstein metric (Proposition 4). These two methods involve different approximations to the Semi-Backward Euler, and one would like to know why the approximations in the latter method is better in performance than those in the former. Discussion on this point is however missing in this paper.
In Section 3, it would have been better if the performance be compared not only in terms of FID but also the loss considered (i.e., Wasserstein-1), since the latter is exactly what the algorithms are trying to optimize.
Minor points:
Page 4: The line just after equation (4) should be moved to the position following the equation giving $d(\theta_0,\theta_1)^2$.
In the reference list, the NIPS paper by Gulrajani et al. appears twice.


Long training times have increasingly become a burden for researchers by slowing down the pace of innovation, with some models taking days or weeks to train. In this paper, a new, general technique is presented that aims to speed up the training process by using a thinned-down training dataset. By leveraging autoencoders and the unique properties of embedding spaces, we are able to filter training datasets to include only those samples that matter the most. Through evaluation on a standard CIFAR-10 image classification task, this technique is shown to be effective. With this technique, training times can be reduced with a minimal loss in accuracy. Conversely, given a fixed training time budget, the technique was shown to improve accuracy by over 50%. This technique is a practical tool for achieving better results with large datasets and limited computational budgets.

This paper presents the idea of splitting the training process into two phases: fast training on a subset of the original dataset and finetining on the full dataset. To find a good subset of the training dataset it is proposed to train an autoencoder and use its embeddings to choose examples that have large values of the embedding features. The experiments show that on CIFAR-10 dataset this may speed up the convergence.
In general, I like the idea of being smart about which data and in which order to feed to the learner.
Nonetheless, I disagree with several premises of this paper. The paper claims that by making the dataset smaller one can speed up the training by the means of fitting the dataset into the accelerator memory and thus avoiding slow memory copies from CPU to accelerator memory. However, modern deep learning data pipelines are built in a way that has virtually zero overhead, since the data is loaded from disk and preprocessed on CPU and then copied on the accelerator asynchronously (i.e. the GPU doesn’t have to wait for the data, it can process the current batch and at the same time load the next one). Moreover, moving the data to GPU will introduce additional overheads in the case of random data augmentation, since this additional work would have to be done by GPU (while current deep learning frameworks asynchronously do this work on CPU). And finally, the authors claim that most modern datasets can fit to the accelerator memory if reduced 10x, but in my experience the network and it’s activations (which are stored during training) occupies most of the GPU memory even on high end accelerators, not leaving enough space to store a large dataset even after data reduction.
The paper cites Dunner  et al. (2017) as related work that focus on the similar problem: how to find a subset of the dataset to fit it into the GPU memory. However, I would argue that their setup is very different because they are using linear models (such as SVM): their learning steps are very fast compared to CNNs (which makes the memory bandwidth much slower in comparison), they don’t have to store activations of the layers (which allows them to fit much more training samples into GPU memory), and they don’t use data augmentation.
Also, I don’t think that the experimental comparison provides a strong enough evidence supporting the benefits of the proposed scheme. First, the experiments are only done on a small scale dataset (CIFAR-10), which is OK in general, but questionable when the proposed method explicitly targets big data regime and making the training faster. Second, the only baseline considered is choosing subset of data randomly. Third, the optimization method is plain SGD with momentum, while when presenting techniques for faster convergence it would make sense to compare on at least several standard optimization algorithms (e.g. Adam). Finally, the presented results are weak: in Fig 4 any improvements over random baseline are noticeable only after degrading the performance of the network by a large margin (to less than 67% accuracy on CIFAR-10);
Fig 5 looks like it has an error: training on the full dataset performs on the level of random guess after some training, which contradicts the fact that the same network converged to something reasonable in Fig. 6. Also, I believe that the training on the full dataset is strictly better than training on a random subset of data for a few epochs and then finetuning on the full dataset (with the same time budget). The latter sees the same number of updates but with less data, which should only decrease the test performance. If it’s correct, the results in Fig 5 for the full training should look similar (or better) than the results in Fig 4 for the random subset baseline, but it’s very far from being the case.
In Fig 6 I’m not sure what is being compared. Is a train or test loss? If it’s train, then it’s not a fair comparisons, since the two network are optimizing different train losses. I’m also very surprised not to see the moment of training mode transition on the plot (i.e. the moment when the model switched from restricted dataset to the full one), the lack of it can indicate an implementation error.
And finally, I would like to see the text being improved. Right now the language is confusing, for example: “this technique is shown to be effective” (what does it mean “effective” and compared to what?), “Unfortunately, while these techniques may be viable for smaller networks or datasets, large datasets have shown that they do not scale well.” (who have shown that the techniques doesn’t work on large dataset?), “the testing network is initialized using a weighted average of the final weights learned during training” (what does it mean?), “Qualitatively, the trained autoencoder succeeded in learning an adequate embedding.” (what does it mean?), etc.
Also, there is a typo in formulas 1, 2, and 3: it probably should sum up to n-1.
And I didn’t get what formula 4 means, what is “union of (for all i in n)”?
This bit I also didn’t get: “This simple loss function, in essence, forces the network to learn to extract the key features from the input, so that it can reproduce it using said features only. If desired, one could elect to use a more sophisticated loss, such as the Wasserstein distance metric (Gulrajani et al., 2017; Arjovsky et al., 2017), that takes more into account than raw pixel values.”. How can you substitute L2 loss in an autoencoder with Wasserstein metric (which is a metric between probability distributions, not images)?
There is also some missing related work, e.g. the idea mentioned in conclusion on augmenting the dataset in the latent space is presented in DeVries et al. “Dataset augmentation in feature space”.
It would be interesting to connect this work with importance sampling off-policy RL (see e.g. “Prioritized Experience Replay”) and look into sampling dataset points proportional to some importance probability with importance sampling correction.
On the positive side, I really enjoyed the look of the figures and diagrams.


PROXQUANT: QUANTIZED NEURAL NETWORKS VIA PROXIMAL OPERATORS
To make deep neural networks feasible in resource-constrained environments (such as mobile devices), it is beneficial to quantize models by using low-precision weights. One common technique for quantizing neural networks is the straightthrough gradient method, which enables back-propagation through the quantization mapping. Despite its empirical success, little is understood about why the straight-through gradient method works. Building upon a novel observation that the straight-through gradient method is in fact identical to Nesterov's dual-averaging algorithm on a quantization constrained optimization problem, we propose a more principled alternative approach, called PROXQUANT, that formulates quantized network training as a regularized learning problem instead and optimizes it via the prox-gradient method. PROX-QUANT does back-propagation on the underlying full-precision vector and applies an efficient prox-operator in between stochastic gradient steps to encourage quantizedness. For quantizing ResNets and LSTMs, PROXQUANT outperforms state-of-the-art results on binary quantization and is on par with state-of-the-art on multi-bit quantization. We further perform theoretical analyses showing that PROXQUANT converges to stationary points under mild smoothness assumptions, whereas variants such as lazy prox-gradient method can fail to converge in the same setting.

This paper proposed ProxQuant method to train neural networks with quantized weights. ProxQuant relax the quantization constraint to a continuous regularizer and then solve the optimization problem with proximal gradient method. The authors argues that previous solvers straight through estimator (STE) in BinaryConnect (Courbariaux et al. 2015) may not converge, and the proposed ProxQuant is better.
 I have concerns about both theoretical and experimental contributions
1. The proposed regularizer for relaxing quantized constraint looks similar to BinaryRelax (Yin et al. 2018 BinaryRelax: A Relaxation Approach For Training Deep Neural Networks With Quantized Weights.), which is not cited. I hope the authors can discuss this work and clarify the novelty of the proposed method. One difference I noticed is that BinaryRelax use lazy prox-graident, while the proposed ProxQuant use non-lazy update. It is unclear which one is better.
2. On page 5, the authors claim ‘’Our proposed method can be viewed as … generalization ...’’ in page 5. It seems inaccurate because unlike proposed method, BinaryConnect use lazy prox-gradient.
3. What’s the purpose of equation (4)? I am confused and did not find it explained in the content.
4. The proposed method introduced more hyper-parameters, like the regularizer parameter \lambda, and the epoch to perform hard quantization. In section 4.2, it is indicated that parameter \lambda is tuned on validation set. I have doubts about the fairness comparing with baseline BinaryConnect. Though BC does not have this parameter, we can still tune learning rate.
5. ProxQuant is fine-tuned based on the pre-trained real-value weights. Is BinaryConnect also fine-tuned? For a CIFAR-10 experiments, 600 epochs are a lot for fine-tuning. As a comparison, training real-value weights usually use less than 300 epochs. BinaryConnect can be trained from scratch using same number of epochs. What does it mean to hard-quantize BinaryConnect? The weights are already quantized after projection step in BinaryConnect.
6. The authors claim there are no reported results with ResNets on CIFAR-10 for BinaryConnect, which is not true. (Li et al. 2017 Training Quantized Nets: A Deeper Understanding) report results on ResNet-56, which I encourage authors to compare with.
7. What is the benefit of ProxQuant? Is it faster than BinaryConnect? If yes, please show convergence curves. Does it generate better results? Table 1 and 2 does not look convincing, especially considering the fairness of comparison.
8. How to interpret Theorem 5.1? For example,  Li et al. 2017 show the real-value weights in BinaryConnect can converge for quadratic function, does it contradict with Theorem 5.1?
9. I would suggest authors to rephrase the last two paragraphs of section 5.2. It first states ‘’one needs to travel further to find a better net’’, and then state ProxQuant find good result nearby, which is confusing.
10.  The theoretical benefit of ProxQuant is only intuitively explained, it looks to me there lacks a rigorous proof to show ProxQuant will converge to a solution of the original quantization constrained problem.
11. The draft is about 9 pages, which is longer than expected. Though the paper is well written and I generally enjoyed reading, I would appreciate it if the authors could shorten the content.
My main concerns are novelty of the proposed method, and fairness of experiments.

======================= after rebuttal =======================
I appreciate the authors' efforts and am generally satisfied with the revision. I raised my score.
The authors show advantage of the proposed ProxQuant over previous BinaryConnect and BinaryRelax in both theory and practice. The analysis bring insights into training quantized neural networks and should be welcomed by the community.
However, I still have concerns about novelty and experiments.
- The proposed ProxQuant is similar to BinaryRelax except for non-lazy vs. lazy updates. I personally like the theoretical analysis showing ProxQuant is better, although it is based on smooth assumptions. However, I am quite surprised BinaryRelax is so much worse than ProxQuant and BinaryConnect in practice (table 1). I would encourage the authors to give more unintuitive explanation.
-  The training time is still long, and the experimental setting seems uncommon. I appreciate the authors' efforts on shortening the finetuning time, and provide more parameter tuning.  However, 200 epochs training full precision network and 300 epochs for finetuning is still a long time, consider previous works like BinaryConnect can train from scratch without a full precision warm start. In this long-training setting, the empirical advantage of ProxQuant over baselines is not much (less than 0.3% for cifar-10 in table 1, and comparable with Xu 2018 in table 2).


We consider the problem of balancing exploration and exploitation in sequential decision making problems. This trade-off naturally lends itself to probabilistic modelling. For a probabilistic approach to be effective, considering uncertainty about all immediate and long-term consequences of agent's actions is vital. An estimate of such uncertainty can be leveraged to guide exploration even in situations where the agent needs to perform a potentially long sequence of actions before reaching an under-explored area of the environment. This observation was made by the authors of the Uncertainty Bellman Equation model (O'Donoghue et al., 2018), which explicitly considers full marginal uncertainty for each decision the agent faces. However, their model still considers a fully factorised posterior over the consequences of each action, meaning that dependencies vital for correlated long-term exploration are ignored. We go a step beyond and develop Successor Uncertainties, a probabilistic model for the state-action value function of a Markov Decision Process with a non-factorised covariance. We demonstrate how this leads to greatly improved performance on classic tabular exploration benchmarks and show strong performance of our method on a subset of Atari baselines. Overall, Successor Uncertainties provides a better probabilistic model for temporal difference learning at a similar computational cost to its predecessors.

This paper tackles the classical exploration / exploitation problem in reinforcement learning.
The paper argues that it is necessary to propagate uncertainty correctly and argue that they can do so using the successor representation to compute the Bayesian posterior over Q-values conditioned on the data already observed.
Novelty:
This work is similar to “The uncertainty bellman equation” (UBE) (O’Donoghue et al. 2018) which adds a head to a regular DQN agent to predict a function u which is an upper bound of the variance of the posterior distribution over Q-values. The difference here is that the successor features are used here to predict Q-values and the function u.
The successor features can be seen as a discounted state occupancy of the current policy and carry information about the future. While the relation with the UBE is highlighted in section 4.6 an empirical evaluation between the two methods would also be needed.
Clarity: The method is detailed comparing to contextual bandit methods, the authors then argue that applying directly these methods to reinforcement learning case does not propagate uncertainty over several timesteps. While this indeed true it is misleading and imply that propagating uncertainty is not considered by current exploration methods in RL.
Soundness:
The method presented here is relatively reductive. The estimated posterior over the value function is correct only if the transition model P is already known, otherwise Equation 4 would also need to incorporate uncertainty over P. Similarly, as the authors point out, this doesn’t include the max operation. At best, we are learning a posterior over the value function for a fixed policy, for when only the reward is unknown.
As a whole, the authors argue that their method allows a better propagation of Q-values uncertainty but provide little theoretical or experimental evidence that would back this claim.
From a deep RL perspective, the features \phi^l only carry local information. The authors argue that this leads to more stable features as these feature do not depend on the current policy. However it also means that in a sparse reward setting the reward observed would be zero most of the time and no useful features would be learned. In practice methods using the successor representation usually share parts of the network with other tasks to improve representation learning (see e.g. Figure 1 of Machado et al., Eigenoption discovery through the deep successor representation, 2017, & also their 2018 paper).
Experiments:
The experiment are disappointing as they are only limited to tabular and deterministic problems. An obvious missing comparison is to UBE, at the minimum; and other “deep” algorithms such as BDQN, Bootstrap DQN, etc. Some of these algorithms have been shown to perform well on the Atari benchmark, and that seems like a reasonable point of comparison also.
The method also relies on knowing the successor features. While they can be learned easily in a tabular, deterministic MDP it is not clear how the posterior would behave in larger and/or non stochastic domains when it takes more time to learn these successor features.
Overall, I am not sure what I learned from reading this paper. While the idea of using the successor representation in exploration is interesting and has been considered recently, the method presented in this paper needs to be better justified and evaluated on more challenging tasks.
Minor comments
I would like to see a proof of Equation 4, which may be simple but is not immediate.
Some papers of relevance here:
An analysis of model-based Interval Estimation for Markov Decision
Processes, Strehl & Littman (2008)
(More) efficient reinforcement learning via posterior sampling. Osband et al (2013)
Count-Based Exploration with the Successor Representation, Machado et al. (2018)


Embeddings of tree-like graphs in hyperbolic space were recently shown to surpass their Euclidean counterparts in performance by a large margin. Inspired by these results, we present an algorithm for learning word embeddings in hyperbolic space from free text. An objective function based on the hyperbolic distance is derived and included in the skip-gram negative-sampling architecture from word2vec. The hyperbolic word embeddings are then evaluated on word similarity and analogy benchmarks. The results demonstrate the potential of hyperbolic word embeddings, particularly in low dimensions, though without clear superiority over their Euclidean counterparts. We further discuss subtleties in the formulation of the analogy task in curved spaces.

This paper presents a technique for embedding words in hyperbolic space, which extends previous non-euclidean methods to non-structured data like free text. The authors provide a new gradient based method for creating the embeddings and then evaluate them on standard word embedding benchmarks. Overall the paper is very well written and well executed. They find that in the low dimensions the approach outperforms standard Euclidean space methods while in higher dimensions this advantage disappears.
The results do not try to claim state of the art on all benchmarks, which I find refreshing and I appreciate the authors candor in giving an honest presentation of their results. Overall, I enjoyed this paper and am eager to see how the authors develop the approach further.
However, along these same lines it would be great to have the authors provide more discussion about the next steps and potential applications for this approach. Is the interest here purely methodological? Are there potential use cases where they believe this approach might be superior to Euclidean approaches? More detail in the discussion and intro about the trajectory of this work would help the reader understand the methodological and application-specific implications.
Pros:
- Clearly written and results are presented in a straightforward manner.
- Extension of analogy reasoning to non-euclidean spaces.
Cons:
- Lack of clear motivation and compelling use case.
- It would be nice to have a visualization of the approach in 2-dimensions. While Figure 3 is instructive for how analogies work in this space, it would be great to visualize an entire dataset. I'm sure that the proposed embeddings would result in  a very different space than euclidean embeddings (as the Poincare embedding paper showed), so it would be great to have at least one visualization of an embedded dataset. Presumably this would play to the strengths of the approach as it excels in lower dimensions.
-  The largest of embedding dimension tested was 100, and it is common to use much larger embeddings of 500-d. Do the trends they observe continue to larger dimensions, e.g. is the performance gap even larger in higher dimensions?


To realize the promise of ubiquitous embedded deep network inference, it is essential to seek limits of energy and area efficiency. To this end, low-precision networks offer tremendous promise because both energy and area scale down quadratically with the reduction in precision. Here, for the first time, we demonstrate , and VGG-16bn networks on the ImageNet classification benchmark that, at 8-bit precision exceed the accuracy of the full-precision baseline networks after one epoch of finetuning, thereby leveraging the availability of pretrained models. We also demonstrate ResNet-18, ResNet-34, and ResNet-50 4-bit models that match the accuracy of the full-precision baseline networks -the highest scores to date. Surprisingly, the weights of the low-precision networks are very close (in cosine similarity) to the weights of the corresponding baseline networks, making training from scratch unnecessary. We find that gradient noise due to quantization during training increases with reduced precision, and seek ways to overcome this noise. The number of iterations required by stochastic gradient descent to achieve a given training error is related to the square of (a) the distance of the initial solution from the final plus (b) the maximum variance of the gradient estimates. By drawing inspiration from this observation, we (a) reduce solution distance by starting with pretrained fp32 precision baseline networks and fine-tuning, and (b) combat noise introduced by quantizing weights and activations during training, by using larger batches along with matched learning rate annealing. Sensitivity analysis indicates that these techniques, coupled with proper activation function range calibration, offer a promising heuristic to discover low-precision networks, if they exist, close to fp32 precision baseline networks.

This manuscript joins a crowded space of methods for low bit quantization to enable inference on more efficient hardware. In the past, these methods often were limited to 8-bit quantization, or smaller networks, or result in accuracy degradation. This paper is part of a recent crop of methods that achieve full accuracy on ResNet50 with 4-bit weights and activations.
The method in this paper is based around a simple, yet powerful observation: Fine-tuning at low precision introduces noise in the gradient. Using the relationship between noise, batch-size and learning rate, that has recently been receiving a lot of attention in the context of large batch training, they compensate for this added noise by increasing the batch size.
I like the simplicity and effectiveness, and believe that this method will be a useful addition to the toolbox for low-precision inference.
Overall, the paper is well written, and the claims are well supported experimentally. Results are demonstrated on a wide range of networks, including various configurations of ResNet, DenseNet, Inception. It's not clear whether these experiments are from a single run. If they are, with sub 1% differences between methods we are getting close to the run-to-run variability, and it would be preferable to see results averaged across multiple runs.
Ultimately, I am on the fence if this is a sufficient contribution for acceptance. In particular, this paper claims "first evidence ... matching the accuracy of full precision". While this may in a narrow technical sense be the case, PACT https://arxiv.org/abs/1805.06085 also works on ResNet50 without an accuracy drop. While this is not published work, it was rejected at ICLR last year, making it hard to recommend acceptance here. There is also work concurrently submitted to this forum (which I obviously don't expect the authors to cite or take into account, but want to mention for the sake of completeness) such as https://openreview.net/forum?id=HyfyN30qt7 which achieves the same or better results, and does not require 8-bit BN scale factors and 32-bit bias.
This manuscript could be made stronger in multiple ways, e.g. by combining with the recently proposed clipping techniques like Choi et al. (2018), and pushing towards 2 or 3 bit training, or eliminating all larger bit-width parameters to make for easier hardware design.


APPROXIMABILITY OF DISCRIMINATORS IMPLIES DIVERSITY IN GANS
While Generative Adversarial Networks (GANs) have empirically produced impressive results on learning complex real-world distributions, recent works have shown that they suffer from lack of diversity or mode collapse. The theoretical work of Arora et al. (2017a)  suggests a dilemma about GANs' statistical properties: powerful discriminators cause overfitting, whereas weak discriminators cannot detect mode collapse. By contrast, we show in this paper that GANs can in principle learn distributions in Wasserstein distance (or KL-divergence in many cases) with polynomial sample complexity, if the discriminator class has strong distinguishing power against the particular generator class (instead of against all possible generators). For various generator classes such as mixture of Gaussians, exponential families, and invertible and injective neural networks generators, we design corresponding discriminators (which are often neural nets of specific architectures) such that the Integral Probability Metric (IPM) induced by the discriminators can provably approximate the Wasserstein distance and/or KL-divergence. This implies that if the training is successful, then the learned distribution is close to the true distribution in Wasserstein distance or KL divergence, and thus cannot drop modes. Our preliminary experiments show that on synthetic datasets the test IPM is well correlated with KL divergence or the Wasserstein distance, indicating that the lack of diversity in GANs may be caused by the sub-optimality in optimization instead of statistical inefficiency.

[pros]
This paper proposes the notion of restricted approximability, and provides a sample complexity bound, polynomial in the dimension, for GANs.
The proposal is especially useful in investigating possible cause of the lack of diversity in GANs.
[cons]
Whether it proposes use of properly-designed discriminator architecture in GAN learning is not clear enough.
The claimed ability of the proposed method to avoid mode collapse is not directly addressed in the experiments presented in the appendices.
[quality]
The contents of Section 3 may be useful as case studies but are not used in the following sections on neural network generators. It would thus be better to include experimental results into the main part of the paper rather than the current contents in Section 3.
[clarity]
In most parts of this paper, the authors seem to propose designing a proper discriminator architecture according to the generator class, and the discriminator architecture is to be used in GAN learning. It seems, however, that a "properly-designed discriminator architecture" is not used at all in the experiments in Appendix F. A comparison between a "properly-designed discriminator architecture" and a "vanilla fully-connected distriminator" is found in Appendix G.4, where the advantage of the former seems marginal. The authors also seem to use the proposal not to improve GAN learning but rather as a tool for evaluation, in order to see whether the lack of diversity in GANs comes either from failure of properly evaluating the Wasserstein distance or from insufficient optimization in learning. These two distinct subjects are discussed in a mixed way, which reduces clarity of the presentation.
In the experiments in Appendix G, it is claimed that a discriminator with the architecture specified in Lemma 4.1 is used in GAN learning, but either weight clamping or gradient penalty is used as well. It is unclear how the specifications in Lemma 4.1 for the parameter $\phi$ are combined with weight clamping or gradient penalty.
Some statements include forward reference, which obscure readability. For example, in the last paragraph of Section 1.1 "the statistical properties of GANs" are mentioned without an explicit statement as to what they mean, which are given later in page 3, lines 6-12. As another example, in the third paragraph of Section 1.3 the authors start discussing the KL-divergence, but at this point it is not evident at all why they do it. It is not until Section 4.1 that the reader can understand the reason by observing that the main theorem (Theorem 4.2) is proved by making use of KL-divergence.
[originality]
The idea of introducing the notion of restricted approximability and discussing a sample complexity bound, polynomial in the dimension, for GANs are considered original.
[significance]
The whole arguments in this paper are based on the assumption that both $p$ and $q$ are in the class $\mathcal{G}$. In the context of GAN learning, it poses no problem for the generator since we explicitly parameterize it, for example using a neural network, but in practice there is no guarantee that the target distribution also belongs to the same class, and this point would affect significance of the proposal. One may argue that when one employs a certain neural network architecture for the generator one expects that the target distribution is well expressed by a network with the prescribed architecture. But the question as to what will happen when the target distribution does not belong to the class $\mathcal{G}$ remains. In any case, no discussion is presented in this paper as for this question.
[minor points]
Page 3, line 45: for low-dimensional (dimensions -> distributions)
Page 4, line 8: Remove the parentheses enclosing Lopez-Paz & Oquab, 2016.
Page 4, lines 20-21: Duplicate parentheses.
Page 4, line 7: the true and estimated distribution(s) exist.
Page 5, line 33: the lower and upper bound(s) differ
Page 7, line 9: What do "some assumptions" refer to?
Page 8, line 44: The(re) exists a discriminator class
Page 19, line 1: there exi(s)ts a coupling


To educate hyper deep learners, Curriculum Learnings (CLs) require either human heuristic participation or self-deciding difficulties of training instances. These coaching manners are blind to the coherent structures among examples, categories and tasks, which are pregnant with more knowledgeable curriculum-routed teachers. In this paper, we propose a general methodology Teaching to Teach (T2T). T2T is facilitated by Structured Dark Knowledge (SDK) that constitutes a communication between structured knowledge prior and teaching strategies. On one hand, SDK adaptively extracts structured knowledge by selecting a training subset consistent with the previous teaching decisions. On the other hand, SDK teaches curriculumagnostic teachers by transferring these knowledge to update their teaching policy. This virtuous cycle can be flexibly-deployed in most existing CL platforms and more importantly, very generic across various structured knowledge characteristics, e.g., diversity, complementarity and causality. We evaluate T2T across different learners, teachers and tasks, which significantly demonstrates that structured knowledge can be inherited by the teachers to further benefit learners' training.

Summarization: This paper studies how to inject structured prior knowledge into the teaching model for machine learning. The authors propose a very general framework called `teach to teach’, in which 1) the knowledge is distilled via subset selection that matches the teacher decision distribution 2) the distilled knowledge is then transferred into the teacher model by reweighting the contributions of teacher objective and coherence constraints. Extensive experiments are conducted on image classification, unsupervised domain adaptation and sequence learning.

Questions: 1) Can the teacher models, like those in L2T, be successfully transferred? For example, the teacher model trained with task 1 (with prior knowledge 1) successfully applied to task 2?
2) I’m not that clear about the relationship with knowledge distillation (Hinton et. al 2015). Per my understanding, the authors seem to make the distribution of the knowledge (specified by the set prior function F) coherent with teacher model and let the both influence each other (in section 3.2). In that sense I do not know what is the `dark’ knowledge here.

Pros: In general I think this paper is a decent work that the structural prior knowledge is elegantly combined with teaching strategy (a.k.a. the teacher models in curriculum learning). The proposed method is intuitive and natural. The empirical verifications are deep and comprehensive to demonstrate the effectiveness of the `teaching to teach’ framework.

Cons: 1) I think the authors should compare with self-paced learning with diversity (SPLD) since you also take diversity as a form of structural knowledge.
2) The writing needs to be significantly polished. First, please simply the writing both in terms of general logic and language. I spent quite a few efforts in figuring out the meaning of some notations and complicated terms such as `curriculum-routed’ and `g_i’. Furthermore, I see no reason of putting so much fancy decorations on an essentially iterative algorithm (the bottom part of page 5 and all page 6). Second,  I suggest the authors give more intuitive and concrete examples towards what is the structural prior knowledge at the earlier phase of the paper, rather than putting most of them into appendix. Last but not least, please use more clear citation formats: currently quite a few citations are missing of publishing venues such as Fan et.al 2018 and Furlanello et.al 2018.