Backpropagation is driving today's artificial neural networks (ANNs). However, despite extensive research, it remains unclear if the brain implements this algorithm. Among neuroscientists, reinforcement learning (RL) algorithms are often seen as a realistic alternative: neurons can randomly introduce change, and use unspecific feedback signals to observe their effect on the cost and thus approximate their gradient. However, the convergence rate of such learning scales poorly with the number of involved neurons. Here we propose a hybrid learning approach. Each neuron uses an RL-type strategy to learn how to approximate the gradients that backpropagation would provide. We provide proof that our approach converges to the true gradient for certain classes of networks. In both feedforward and convolutional networks, we empirically show that our approach learns to approximate the gradient, and can match or the performance of exact gradient-based learning. Learning feedback weights provides a biologically plausible mechanism of achieving good performance, without the need for precise, pre-specified learning rules.

This paper proposes a method that addresses the "weight transport" problem [1] (not cited by the authors) emphasizing the biological infeasibility of artificial neural networks (ANNs) which are trained by gradients computed by the backpropagation algorithm [2a, 2b]. It is arguably the most eminent criticism (among many many others) when learning in the brain is modelled by such ANNs.
The authors cite only Rumelhart et al. (1986) for backpropagation (which is the same as the reverse mode of automatic differentiation), although Rumelhart cited neither Linnainmaa, the inventor of the method [2a], nor Werbos, who first applied it to ANNs [2b].
The authors propose to estimate the weights for the backward pass using a noise-based estimator and provide theoretical and empirical arguments. The proposed method is compared with backpropagation (the ground truth) and direct feedback alignment (DFA; which resorts to random matrices for the backward pass) on small fully-connected, convolutional, and fully-connected auto-encoder networks for MNIST, CIFAR10, and CIFAR100 and offers analysis based on insights from recent work.
Generally, this work is well-placed and the method straightforward and novel. That said, we found the paper to be difficult to parse with some questionable claims which we don't believe are sufficiently backed up by experiments. Due to the following issues, we think that the paper is not yet quite ready for acceptance.
1.) As the authors point out, the recent work by Akrout et al. [3] is very closely related but the paper is lacking a significant discussion let alone an empirical comparison. Extrapolating from the experiments in this paper we think that the presented method is likely going to underperform since Akrout et al. show on-par performance with backprop on imagenet.
2.) The paper appears to blend synthetic gradients [4a, 4b] and feedback alignment to a somewhat questionable degree. A connection that is usually not done in other recent papers about biologically feasible ANNs. And that for good reasons: synthetic gradients do not address the weight transport problem. They backprop the synthetic gradients in order to train the gradient estimators of the previous layer.
Also, the paper cites only the recent reference [4b] (2016) on synthetic gradients but not the original work [4a] (1990).
3.) The authors claim "Thus our method illustrates a biologically realistic way by which the brain could perform gradient descent learning". Even though their method is a more plausible way by which the brain could perform gradient descent learning, it is still far from realistic. The proposed method still implicitly "transports" a lot of information from the forward pass to the backward pass (such as a. topology or b. the derivative of the activation function or c. the activations of the forward pass (h^i) that are necessary to compute the gradient estimate in either case).
4.) The experimental results of the proposed method but also the baseline are simply too far from the state of the art. This is partially dismissed in section 5 due to regularization and data augmentation "tricks". We'd like to point out that feedback alignment has been reported to achieve <2% error on the MNIST dataset [6] which is significantly better than the results reported in this work without any tricks. Similar seems to be the case for CIFAR10 and CIFAR100. The reported results on CIFAR 10 and 100 are over 20% below the state of the art. Furthermore, many previous papers provide code which arguably makes such experiments particularly easy (see [7]). We do not expect the authors to provide state of the art results but they should be at least in the same ball-park.
5.) The authors claim "(...) this hybrid approach can solve large-scale problems." We don't think this is backed up by experiments as most models are rather small (all < 50k parameters except the CNN architecture) and evaluated on toy datasets (for vision standards; while underperforming as already mentioned).
6.) It is not obvious how the noise-based estimation will scale with width and depth. The authors make the observation that "The number of neurons has an effect". This can be seen in the appendix in Fig 4E where the relative error increases exponentially with the number of neurons in the first layer (note that the second layer doesn't increase since it remains unchanged). This seems to indicate that the method would not scale well.
7.) Why do the authors decide to train the MNIST networks with such a small learning rate (4e-4) for 2 million steps? Better results can be achieved with just a few thousand steps. Does the method work in the "general" setting?
8.) With regards to Figure 2C for layer 2: Given a random initialization of the backward pass for feedback alignment but also the presented method, we would expect that the sign congruence of the first few iterations to be the roughly the same on average. That doesn't seem to be the case according to the figure. Is there are an explanation for this observation?
9.) Why the sudden switch from feedback alignment to direct feedback alignment for the CNN experiment in section 4.3? Does the approach of section 4.2 not work with CNNs?
10) We hope the authors could clear up some confusion with regards to section 3 and the consistency proofs in the appendix.
10.a) Apparently, the notation slightly changes? In section 2, layers are indicated by 1 <= i <= N as the superscript. In section 3, i prevails but a new superscript 1 <= n <= N is used for the layers. Theorem 1 then states that the least-squares estimator for B^(i+1)\tilde{e}^{i+1} is (\hat{B}^{N+1})'. There seems to be a notation issue since it is our understanding that every layer should have its own estimated weights.
10.b) We believe A03 should state that E[\tilde{e}^i (\tilde{e}^i)'] is of full rank and not simply \tilde{e}^i (\tilde{e}^i)' assuming that \tilde{e} stands for the equivalent X of regular least-squares. From the main text (specifically the eq in section 2.1) one might assume that \tilde{e}^i are vectors but the proof seems to treat them like the design matrices X in the regular least-squares estimator.
10.c) It is awkward that in the proof of Theorem 1 and 2 the population size is suddenly T, the same symbol for the transpose.
10.d) The regular OLS estimator of the coefficients is b = (X'X)(X'y) where X is a full-rank design matrix and y are the targets. It is our understanding that \lambda, as defined by eq 12, is equivalent to the y of the regular OLS estimator. But in the following equation (the bottom of page 14), the W^{N+1} of \lambda is factored out, apparently assuming e^{N+1}e^{N+1}' = I. Maybe we made a mistake, we'd appreciate if the authors could clarify this step in the proof.
11.) Finally, we add some additional minor comments or sources of confusion:
- we'd appreciate if all equations were numerated
- In section 2, the definition of the loss is verbose and partially unnecessary. L(y,\hat{y}(x)) never appears in the text. Furthermore, \hat{y} is not introduced as a function so writing \hat{y}(x) is awkward.
- Similar for the equation in section 2.2.
- 4.1 first line: the tilde is misplaced
- Figure 3A, the black line has no label (assumed to be DAE which is missing)
- 4.2 first inline equation: the tilde is misplaced
- Table 1: no decimal values, not std. Not clear if the difference between DFA and note perturbation is significant.
- Figure 4A-E is missing the crucial comparison with feedback alignment
References:
[1] Grossberg, Stephen. "Competitive learning: From interactive activation to adaptive resonance." Cognitive science 11.1 (1987): 23-63.
[2a] Seppo Linnainmaa. The representation of the cumulative rounding error of an algorithm as a Taylor expansion of the local rounding errors. Master's Thesis, Univ. Helsinki, 1970. FORTRAN code on pages 58-60. See also BIT 16, 146-160, 1976.
[2b] Werbos, Paul J. "Applications of advances in nonlinear sensitivity analysis." System modeling and optimization. Springer, Berlin, Heidelberg, 1982. 762-770.
[3] Akrout, Mohamed MA, et al. "Deep learning without weight transport." Advances in Neural Information Processing Systems. 2019.
[4a] J.  Schmidhuber. Networks adjusting networks. In J. Kindermann and A. Linden, editors, Proceedings of `Distributed Adaptive Neural Information Processing', St. Augustin, 24.-25.5. 1989, pages 197-208. Oldenbourg, 1990. Extended version: TR FKI-125-90 (revised), Institut für Informatik, TUM. http://people.idsia.ch/~juergen/FKI-125-90ocr.pdf See section "An Approach to Local Supervised Learning in Recurrent Networks."
[4b] Jaderberg, Max, et al. "Decoupled neural interfaces using synthetic gradients." Proceedings of the 34th International Conference on Machine Learning-Volume 70. JMLR. org, 2017.
[5] Czarnecki, Wojciech Marian, et al. "Understanding synthetic gradients and decoupled neural interfaces." Proceedings of the 34th International Conference on Machine Learning-Volume 70. JMLR. org, 2017.
[6] Nøkland, Arild. "Direct feedback alignment provides learning in deep neural networks." Advances in neural information processing systems. 2016.
[7] https://paperswithcode.com/sota/image-classification-on-cifar-10?p=maxout-networks
For now, we'd lean towards rejecting this submission, but we might change our minds, provided the comments above were addressed in a satisfactory way. Let us wait for the rebuttal.
Edit: After the rebuttal, we increased our score.


This paper analyzes how generalization works in meta-learning. Our core contribution is an information-theoretic generalization bound for meta-learning, which identifies the expressivity of the task-specific learner as the key factor that makes generalization to new datasets difficult. Taking inspiration from our bound, we present Discrete InfoMax Codes (DIMCO), a novel meta-learning model that trains a stochastic encoder to output discrete codes. Experiments show that DIMCO requires less memory and less time for similar performance to previous metric learning methods and that our method generalizes particularly well in a challenging small-data setting.

This paper presents DIMCO, a meta-learner that is trained by maximizing mutual information between a discrete data representation and class labels across tasks. DIMCO is inspired by an information theoretic lower bound on the generalisation gap for meta-learning, which the authors argue identifies overfitting in the task learner as the bottleneck.
This work proposes to constrain a learner to output discrete codes that are learned to capture the mutual information with class labels. While the idea of using discrete codes is interesting, its presentation in the manuscript is not well motivated and at times hard to follow. This makes it challenging to evaluate the novelty, validity, and generality of the proposed approach. Meanwhile, the empirical evaluation is somewhat lacking. Thus, I do not believe this work is ready for publication in its current form.
Detailed comments
My main concern is with respect to the primary contribution of this paper, a generalisation bound on meta-learning. The bound appears to be on a multi-task loss without task adaptation, and thus the claims made with respect to the theorem seem somewhat over-reaching. I also believe the VC-dimensionality of the encoder is missing in Eq. 4? If so, this changes the interpretation since the length of the code and the expressivity of the encoder are interrelated. Further, I would welcome a deeper analysis of the theorem and its implications. The current interpretation states that the number of tasks is independent of the size of each task, hence given many tasks, using minimal representations is an effective approach to meta-generalisation. Yet minimal representations is a well-known idea and has features in several works that use mutual information as a regularizer, most notably works on the Information Bottleneck.
Another reservation I have is the use of mutual information between encoder representations and class labels as a loss function (Eq. 1). It lacks context and a proper motivation, especially since the analysis of [1] shows that the loss function in Eq. 1 is the cross-entropy objective. The authors make a similar analysis in Appendix A and argue that Eq. 1 differs in that cross-entropy is an approximation because it adds a parametrized linear layer on top of \tilde{X}. Thus, in the absence of that layer they collapse to the same objective. As DIMCO itself directly extract class label predictions from \tilde{X}, I fail to see a difference between the loss in Eq. 1 and a cross-entropy objective.
The main motivation behind their loss objective is that it does not require a support / query set. This does not seem to be a feature of the mutual information objective itself, but rather a choice made by the authors. I would have liked a deeper discussion of this seeing as the authors make it a central tenet of DIMCO. Prior works use a support set as a principled means of doing meta-learning: meta-training explicitly takes into account that at test time, the learner will be given a small support set from which to learn how to query points. As far as I understand, DIMCO does not take this into account during meta-training.  At meta-test time however, DIMCO does use a support set to map query points (Eqs. 10 and 11). Why should we break protocols between meta training and testing? Are there any downsides to doing so?
Empirically, I find the CUB experiment compelling but would welcome some ablations. What are the trade-offs between p and d? Can DIMCO outperform N-pair when number of bits are unconstrained?
miniImagenet is a standard benchmark in few-shot learning, but I am unable to find a results table - could the authors please provide results on the standard setup so that the method can be compared against known baselines? Further, would the results currently presented hold in a N-way-5-shot setup?
As for the constrained version of miniImagenet that the authors propose, I am not convinced this is an interesting protocol. In general, the miniImagenet task distribution is created by N-way permutations of the classes in the meta-set (e.g. meta-training tasks are combinations of the 64 classes in the meta-training set). By keeping the number of classes constant but reducing the number of images per class, this protocol is not reducing samples per task: a task is always defined as 5/20-way-1-shot (Fig. 4). Instead, the effect should be that tasks are in (greater) violation of the task i.i.d. assumption. Thus, I question whether this setup demonstrates the trade-offs the authors present in Theorem 1 and whether the results can be interpreted in light of it.
Finally, that both experiments are image-based raises questions as to the generality of the method. The paper could be considerably strengthened by evaluating DIMCO on a non-image task, or if not discuss the method’s limitations.
The idea of discrete codes for few-shot classification is interesting and sufficiently novel, I am likely to increase my score if my concerns are addressed and the experimental section is strengthened.
Further questions and comments:
- I am unable to parse Eq. 11 - what does the notation \prod_i p_{\tilde{x}_i, i} mean?
- It is unnecessarily hard to follow the proof of theorem 1. It would help the reader if you restated relevant definitions, such as Eq. 1, since the difference with Eq. 23 is very subtle. It would also be helpful to explain how the summand in Eq. 25 differs from either, and Eq. 26 could be expanded or briefly explained after the derivation.
-  Because DIMCO is trained with backpropagation, the interpretation of Eq. 5 as d independent events seems invalid. How does it affect the method if they are not independent?
- Overloading X and Y as both random variables and mini-batch samples creates unnecessary confusion. I believe the objective in Eq. 1 is approximated, not calculated exactly? For instance, the mutual information in Eq. 8 is with respect to a mini-batch, so should it not be \hat{I}?
- p^j_{ik} in Eq. 9 is undefined.
- Eq. 9 is interpreted as an exact entropy, however it appears to be a mini-batch approximation to the true entropy?
References
[1] Achille and Soatto. Emergence of Invariance and Disentanglement in Deep Representations. JMLR. 2018.


The problem of accelerating drug discovery relies heavily on automatic tools to optimize precursor molecules to afford them with better biochemical properties. Our work in this paper substantially extends prior state-of-the-art on graph-to-graph translation methods for molecular optimization. In particular, we realize coherent multi-resolution representations by interweaving the encoding of substructure components with the atom-level encoding of the original molecular graph. Moreover, our graph decoder is fully autoregressive, and interleaves each step of adding a new substructure with the process of resolving its attachment to the emerging molecule. We evaluate our model on multiple molecular optimization tasks and show that our model significantly outperforms previous state-of-the-art baselines.

The authors present a heirarchical graph-to-graph translation method for generating novel organic molecules.
Working from the model of Jin et al. (2019), the authors introduce a three step heirarchy - the model first determines where a substructure should be generated, what is the substructure, then the attachments to the existing molecule.
All steps of this uses embeddings generated from a message passing network - these embeddings are input into a few bilinear attention layers to obtain the heirarchical generation scheme.
The model is trained with molecular pairs (X, Y), and a VAE loss - a hidden z vector controls the way to modify X to improve its properties.
The encoder is just a MLP over the difference between sum of embeddings at a atom level and at the substructure level.
The model is evaluated on accuracy and diversity, in both conditional and unconditional settings.
The experiments show a small improvement over previous SOTA algorithms.
This is a borderline paper, and I'm leaning towards a weak reject, because I don't believe the model is well motivated enough:
- Sec 3.1 it's unclear how the substructures are generated - they provide a lot of inductive bias for the algorithm.
  Are they automatically generated or built from a database of substructures?
- Variational decoding does not seem well motivated enough - would a stochastic decoding procedure not work as well as having a latent vector that essentially adds noise to the training?
- The experiments seem interesting and comprehensive - it seems that the model learns to exploit the biases and increase logP, as well as showing the ability to conditionally turn off DRD2-active properties of the molecules.
Some questions:
- Why not use a Transformer instead of an LSTM or GRU? The cell naturally acts over sets of neighbors and transformers are a natural model to tackle this problem.
- Sec 3.1 Topological Prediction, the attention is over c_{X}^{S} but the text claims it should be over c_{X}^{G}? Is ^G the attention substructure?
- Sec 3.1 Attachment Layer MPN: the A_i seem to be a tuple (S_i, {v_j}). The set of attaching atoms is limited to 2 right? It might be more clear to simply enumerate them here if so.
- Sec 3.1 Substructure Tree: Since tree decompositions are not unique, does this work use the different tree decompositions and DFS traversals as data augmentations?
- Table 2b: What is a "two-layer" and "one layer" encoder? Is it the size of the MLP or the removal of the attachment MPNs?
- Ablation study: Since the Attachment Layer has all the substructure information, this ablation should ideally make sure the models all have a similar number of parameters, and the decrease in performance isn't due to the decrease in parameters.
Nits:
- Sec 3.1 "bi-linear" should not have a dash, bilinear is one word.


In this paper, we develop an unsupervised generative clustering framework that combines variational information bottleneck and the Gaussian Mixture Model. Specifically, in our approach we use the variational information bottleneck method and model the latent space as a mixture of Gaussians. We derive a bound on the cost function of our model that generalizes the evidence lower bound (ELBO); and provide a variational inference type algorithm that allows to compute it. In the algorithm, the coders' mappings are parametrized using neural networks and the bound is approximated by Markov sampling and optimized with stochastic gradient descent. Numerical results on real datasets are provided to support the efficiency of our method.

The author(s) posit a Mixture of Gaussian's prior for a compressed latent space representation of high-dimensional data (e.g. images and documents). They propose fitting this model using the Variational Information Bottleneck paradigm and explicate its derivation and tie it to the variational objective used by similar models. They empirically showcase their model and optimization methodology on the MNIST, STL-10, and Reuters10k benchmarks.
The idea of using a latent mixture of Gaussian's to variationally encode high-dimensional data is not new. The author(s) appropriately cite VaDE (Jiang, 2017) and DEC (Xie, 2016), "Deep Unsupervised Clustering with Gaussian Mixture Variational Autoencoders" (Dilokthanakul, 2017). However, the author(s) unfortunately did not mention "Approximate Inference for Deep Latent Gaussian Mixtures" (Nalisnick, 2016). Nalisnick et al. consider the exact generative process as the proposed by the author(s), but with the addition of a Dirichlet prior on the Categorical distribution. Nalisnick et al. fit their model with a VAE and circumvent Dirichlet intractability by positing a Kumarswamy stick-breaking variational posterior (the resulting distribution is on the simplex). They achieve 91.58% accuracy, but do so after fitting a KNN to the latent space. New techniques such as "Implicit Reparameterization Gradients" (Figurnov, 2018) and "Pathwise Derivatives Beyond the Reparameterization Trick" (Jankowiak, 2018) allow direct use of a Dirichlet posterior in VAEs. These methods are respectively implemented in the TensorFlow and PyTorch APIs. My point here is that there are many ways to fit this pre-existing model. From my review of these other works we have, for "best run" on MNIST, that author(s) > VaDE > Nalisnick > Dilokthanakul > DEC. Thus, the author(s) are SoTA for this particular generative process for their best run. It would be nice to see their standard deviation to gauge how statistically significant their results are. However, I am dubious of the SoTA claim as I detail later.
The author(s) derivation was sound, but a bit confusing for me. In particular, I found keeping track of P's and Q's very burdensome after reading sections 2.1 and 2.2. In my experience, Q is typically reserved for a variational distribution that approximates some intractable P distribution, while P is used to describe the generative process (i.e. likelihood) and other exact distributions. Furthermore, one typically introduces the generative process first using P distributions. Once, I got past this confusion everything else made sense. I might suggest introducing the generative process first and with P distributions instead of Q's. The variational model can follow with the Q distributions as the author(s) have it. Equations 4, 5, and 6 all exactly match the unsupervised information bottleneck objective (Alemi, 2017)--see appendix B. I am therefore confident in those equations. I carefully checked their derivation of the VADe comparison (equation 10 and appendix B). Their derivation is straightforward. The principle trick is using the MC assumption C->X->U for P distributions to claim p(u|x,c) = p(u|x). Equations 12-16 all follow naturally from equation 11. If equation 17 is indeed an approximation or bound approximation, I would suggest not using the equals sign. Instead, consider another appropriate operator or rename Dkl to indicate it is the approximated version (just as in equation 24).
If the author(s) like my suggestion regarding generative vs variational nomenclature, I would also change the second sentence of page 6 to something like "We use our variational Qc|u distribution to compute assignments." Thereafter, I would drop the star indicator for optimal parameters. These parameters are not necessarily optimal given the non-convexity of the DNN. Replace with, "after convergence, we ..." Similarly, drop optimal from line 2 of algorithm 1.
Circling back to the experiments, the author(s) use reported values from DEC and VaDE. Those works compute cluster assignments using a KNN classifier on the latent space. This paper however uses the arg max of equation 19. I much prefer this article's method, but for comparison purposes, the author(s) should similarly use a KNN classifier on their latent space to compute accuracy in the same manner. The use of KNN in these other works allows them to consider a number of latent clusters larger than the number of true classes (e.g. 20 clusters for MNIST). I like that the author(s) stick to 10 clusters for MNIST, but for comparison I would have liked to see a KNN generated accuracy alongside their equation 19 based accuracy. It looks like the author(s) implemented VADe for STL-10 based on table 1. If so, it seems they could easily implement equation 19 for their STL-10 value for VADe. If not, please correct this. Not to belabor further, but I would really like to see a table 2 that reports both equation 19 and KNN accuracies when available. Namely, the author(s) should report both values for their model and can leave equation 19 accuracies blank for reported values.
Lastly, I always raise an eyebrow when I see tSNE latent space representations. STL-10 was used to generate figure 6, where one needs dim(u) >> 2. However, MNIST can be well reconstructed using just 2 latent dimensions. In this case, tSNE is unnecessary. The author(s) state "Colors are used to distinguish between clusters." This statement is unclear as to whether the author(s) are using the class label or learned latent cluster. If it is the former, figure 6 makes sense in that it shows misclassifications (i.e. red dots in the green cluster). However, if it is the latter then I am concerned tSNE is doing something weird.
To summarize, I enjoyed the paper. My only concern is novelty. Being the first to pair an existing model with an existing method, in my eyes, does not necessarily meet the ICLR bar. The author(s) seemingly achieve SoTA, but without KNN-based accuracies for their model it is hard to compare to cited works. Having these KNN results and error bars would strengthen their case substantially to: achieving SoTA for an existing model by being the first to pair it with an existing method.


Existing graph embedding models often have weaknesses in exploiting graph structure similarities, potential dependencies among nodes and global network properties. To this end, we present U2GNN, a novel embedding model leveraging on the strength of the recently introduced universal self-attention network (Dehghani et al., 2019), to learn low-dimensional embeddings of graphs which can be used for graph classification. In particular, given an input graph, U2GNN first applies a self-attention computation, which is then followed by a recurrent transition to iteratively memorize its attention on vector representations of each node and its neighbors across each iteration. Thus, U2GNN can address the weaknesses in the existing models in order to produce plausible node embeddings whose sum is the final embedding of the whole graph. Experimental results in both supervised and unsupervised fashions show that our U2GNN produces new state-of-the-art performances on a range of well-known benchmark datasets for the graph classification task.

In this paper, the authors developed a graph embedding method called U2GAN based on self-attention mechanism. Similar to many existing graph neural network, U2GAN samples and aggregate neighboring features for each node in a graph. The aggregation function is similar to GAT, i.e., a query based attention layer. The difference is an incorporation of a transition function followed the attention layer. By minimizing Eq. 7, node embeddings can be inferred, which are summed up to obtain a graph embedding, for the downstream graph classification task.
There are several points that are unclear in the paper.
1.  The major argument of the advantage of using self-attention for neighborhood aggregation is to facilitate memorizing the dependencies between nodes and explore the graph structure similarities locally and globally. This argument, however, was not clearly discussed in the paper. First, it is not clear on why existing GNN methods, such as GCN, GraphSage, and GAT, cannot do so. Second, it is not clear on how does the proposed U2GAN achieve it. The current paper only provides some high-level descriptions. A more specific or theoretical discussion is desired.
2. Since the attention based aggregation is similar to GAT, a discussion on the difference is important.
3. Several model designs are not well justified. In Eq. 2, 3, the reason to employ Layernorm is missing. In Eq. 7, the intuition on how does the loss function help learn effective embeddings remains to be clarified. Also, it may be better to evaluate different pooling method to obtain graph embedding to justify the choice of sum in Eq. 1.
4. Since the proposed method aims to learn node embeddings in an unsupervised manner, it is better to see some descriptions on why graph classification was selected as the task in evaluation, instead of node classification.
5. In the experiments, some methods such as deepwalk, node2vec, graphsage and GAT are missing in comparison. In particular, due to the similarity between the proposed method GAT, it is interesting to evaluate GAT by replacing its supervised loss by Eq. 7 as a compared method. Moreover, in fig. 4, other visualizations of other methods can be compared to demonstrate the difference between the proposed method and others.


Goal recognition based on the observations of the behaviors collected online has been used to model some potential applications. Newly formulated problem of goal recognition design aims at facilitating the online goal recognition process by performing offline redesign of the underlying environment with hard action removal. In this paper, we propose the stochastic goal recognition control (S-GRC) problem with two main stages: (1) deceptive opponent modeling based on maximum entropy regularized Markov decision processes (MDPs) and (2) goal recognition control under proactively static interdiction. For the purpose of evaluation, we propose to use the worst case distinctiveness (wcd) as a measure of the nondistinctive path without revealing the true goals, the task of S-GRC is to interdict a set of actions that improve or reduce the wcd. We empirically demonstrate that our proposed approach control the goal recognition process based on opponent's deceptive behavior.

This paper studies goal recognition control given a deceptive opponent, who selects actions to intentionally mislead or confusing the learner to learn the true goal. The problem has been studied in the security game, resource allocation game and Stackelberg game, where the defender is to play a resource allocation game given the best response of the attacker. in this paper, the authors use stochastic-shortest-path MDP to model the attacker's planning problem. The defender's objective is stated in eq (2) and (3) but not explained clearly.
The work considers a deceptive attacker whose objective is not to maximize the total reward/minimize the total cost, but rather to balance two objectives: minimizing the total cost while maximizing the KL divergence between its current policy and the optimal goal-achieving policy. This opponent model seems to be simplistic, given security game has studied the opponent using more rigorous equilibrium analysis and with dynamic Bayesian game formulation.
see Manshaei, Mohammad Hossein, et al. "Game theory meets network security and privacy." ACM Computing Surveys (CSUR) 45.3 (2013): 25.
The practical application is well motivated.
The paper itself is incomplete and contains many typos. The paper is unfinished.


We tackle the problem of discovering novel classes in an image collection given labelled examples of other classes. This setting is similar to semi-supervised learning, but significantly harder because there are no labelled examples for the new classes. The challenge, then, is to leverage the information contained in the labelled images in order to learn a general-purpose clustering model and use the latter to identify the new classes in the unlabelled data. In this work we address this problem by combining three ideas: (1) We suggest that the common approach of bootstrapping an image representation using the available labels introduces an unwanted bias, which we avoid by using self-supervised learning to train the representation from scratch on the union of labelled and unlabelled data; (2) We use rank statistics to transfer the model's knowledge of the labelled classes to the problem of clustering the unlabelled images; Finally, (3) we train the data representation by optimizing a joint objective function on the labelled and unlabelled subsets of the data, improving both the supervised classification of the labelled data, and the clustering of the unlabelled, simultaneously. We evaluate our approach on standard classification benchmarks and outperform current methods for novel category discovery by a significant margin.

This paper tackles the problem of unsupervised object discovery, whereby a labeled dataset must be leveraged in order to then cluster an unlabeled dataset with a set of unknown categories. The paper contributes three main ideas to succeed at this task, namely 1) use of self-supervised learning to initialize the representations in a way that doesn't bias them to the labeled data, 2) a robust rank-based metric to generate estimates of similarity/dissimilarity along with consistency-based regularization to improve optimization, and 3) Joint optimization/refinement using a combination of labeled/unlabeled losses, as well as ability to learn incrementally without forgeting the original labeled classes. Results are shown on a range of datasets including OmniGlot, ImageNet, CIFAR-10, CIFAR-100, and SVHN. The results demonstrate improvement over the current state of art for this task.
  Overall, my current vote for this paper is a weak reject. The main reason is that the paper really combines a set of known methods (self-supervised learning, consistency-based regularization, and an ad-hoc training regimen. On the other hand, the paper is well-written and provides nice rigorous experiments showing clear improvements over state of art by porting these known techniques from different domains (self/semi-supervised learning). However, if satisfactory answers to questions below are given, I am willing to change my rating.
Main Argument
Strengths
  - Overall the paper tackles an interesting problem, and does so in a way that achieves good results beyond state of art.
  - An ablation study is provided which shows the contribution of different parts of the method.
  - The paper is very well-motivated, written, and methods are described nicely and succinctly.
Weaknesses
  - Clearly the methods employed are, by themselves, not novel and have been used for a range of other ML problem formulations. What is the clear contribution/novelty of the work?
  - While the paper raises an interesting motivation about not biasing feature learning by using self-supervised learning, it's not clear to me that this claim is justified. What is the evidence for this, besides better performance? While the ablation w/o self-supervised learning performs more poorly than everything combined, the other parts of the ablation (with self-supervised learning) also perform poorly. Clearly, there is some interaction between the different aspects of the method, but I am not sure what that is. Why is the full combination so much better than if any one thing is removed?
  - The method is very similar to KCL (e.g. loss (3) is the same and justified via a graphical model formulation in that paper). The paper does not really read like it is building off of that though, which seems a bit misleading. Do the authors believe there is additional novelty, or is it a matter of adding the three contributions to KCL? What is the major reason that self-supervision with KCL still doesn't do as well? As far as I can tell, the only difference is self-supervised learning, the fact that you have a manual 3-stage curriculum, and the robust ranking method/consistency loss (i.e. the three contributions).
  - You assume that the number of clusters is known; this is one of the advantages of all of the prior work in that they can estimate this. How well does the method work if the number of clusters is not known?
Additional comments not related to final vote:
  - The paper seems to start out in a way that implies the problem is new; citations should be provided to the three main compared works in the intro to make it clear that this is a follow-on to an existing problem. Further, as mentioned above MCL/KCL (especially the latter) are very similar in nature and introduced this problem (including [1] which also used unsupervised feature learning in one condition). These should be included as "work most related to ours" given that they came before Han et al.
  [1] Deep Image Category Discovery using a Transferred Similarity Function, https://arxiv.org/abs/1612.01253.


We propose a novel method for compressed sensing recovery using untrained deep generative models. Our method is based on the recently proposed Deep Image Prior (DIP), wherein the convolutional weights of the network are optimized to match the observed measurements. We show that this approach can be applied to solve any differentiable linear inverse problem, outperforming previous unlearned methods. Unlike various learned approaches based on generative models, our method does not require pre-training over large datasets. We further introduce a novel learned regularization technique, which incorporates prior information on the network weights. This reduces reconstruction error, especially for noisy measurements. Finally we prove that, using the DIP optimization approach, moderately overparameterized single-layer networks trained can perfectly fit any signal despite the nonconvex nature of the fitting problem. This theoretical result provides justification for early stopping.

== After reading authors' response ==
The authors didn't really offer a response. Their new PDF just mentions that their proof for the case $A=I$ can be extended to any orthogonal matrix $A$ (since if $G$ is a iid Gaussian matrix with unit variance, so is $AG$ for any orthogonal matrix $A$).  Hence, I don't change my response, and still support "weak accept"
== Original review ==
The paper proposes an extension to the 2017 "Deep Image Prior" (DIP) idea. DIP is a method to use untrained neural nets for compressed sensing, but uses the structure of the neural net to implicitly regularize and constrain the solution (though, as a major result of this paper shows, it doesn't regularize -- it can learn noise). It's an interesting combination of compressed sensing and deep learning.
This particular paper adds the following to the DIP idea:
(1) it adds a total-variation (TV) semi-norm regularization term,
(2) it adds a weighted Tikhonov/ridge-regression style regularization term, with learned weight-matrix, and some overall lambda_L hyper-parameter,
(3) they provide a theorem, based on a 2019 preprint, that a very simplified version of the unregularized DIP model (1 layer, with one set of weights fixed and assumed drawn Gaussian) can be fully trained by gradient descent, and hence it can overfit to noise, and thus necessarily must be stopped early.
They provide some numerical experiments to demonstrate the performance of their approach.
== I have some high-level criticism of the approach (things that cannot be fixed):
-- The idea of an algorithm that must be stopped early is distasteful to me and perhaps to others as well. It's vague, and it means the model is wrong. It doesn't feel like math, it feels like sorcery. You haven't actually provided a result that says when to stop, or that stopping early at all will help, only that running until convergence is a bad idea.
-- Compressed sensing is a bit old these days, and part of the greatness of CS was what it inspired, even if single-pixel cameras have not overturned the camera industry. But pure CS, with a Gaussian measurement matrix, and noiseless measurements, is quite academic and not applicable to the real-world.  In particular, your method seems to do best when there are a lot of measurements (for low measurement numbers, the Bora et al approach does well, as does TVAL on the MNIST data -- see Fig 1b). The experiments were also noiseless, and they had Gaussian or the Fourier sampling (which is not random, but rather heavily biased toward DC). I'm not that convinced this is useful for medical imaging.
On the plus side, any approach for medical imaging is welcome, even if the connection is a bit academic. The writing was overall good.  Furthermore, the proof of the theorem seems like it combines quite a bit of good math. It wasn't clear if empirical process theory was used in their proof, or if that's from the 2019 result they cite.
== Some major comments (things that *can* be addressed)
-- Experimentally, I wasn't convinced that the learned regularization (LR) was useful. Table 1 appears to show it is helpful, but this is just saying that some kind of l2 regularization is useful. I would be convinced if you showed that vanilla Tikhonov regularization is not as helpful as your learned l2 regularization.
-- Furthermore, for the benefit of your LR approach, you did a grid search on the lambda_L parameter to choose the best value. Since you never mentioned trainining/validation/testing data, I assume you did this on the final set of data? In that case, you've tuned your hyperparameter to your testing data, and so I don't find the results that convincing. I think this is a big deal.  Or maybe I don't understand what you've done, because the language ".. and selected the best one" was unclear what "best" meant, e.g., if you look at a residual, or if you use ground-truth data.
-- For both your LR and vanilla Tikhonov, you can try selecting lambda_L based on the residual or other non-oracle method. Standard methods include generalized cross-validation, Morozov’s discrepancy principle, and the unbiased predictive risk estimator method.
-- For the results comparing with Bora et al., how much of your performance advantage was due to the TV term? Please re-do Fig 4b with variants of your method with lambda_T=0, lambda_L=0, and both lambdas=0 added to the plot.
-- The handling of the TV term was not discussed. It seems you fed this into PyTorch and hoped for the best. Actually minimizing the usual isotropic 2D TV is tricky, and there are many optimization papers written about it. It's not differentiable, and if you actually converge, then you usually land at points of non-differentiability, so it cannot just be ignored. Sometimes it is smoothed, or a separable but non-isotropic version is used. I think your response to this point is, "but we don't actually run the algorithm to convergence" (since it will overfit), which I still find bizarre (imagine saying that in a paper submitted to an optimization journal!).
== Summary of my opinion ==
I liked the idea, but I think I liked the core idea of DIP more than the extensions that this paper provides. The experiments show enough promising results to make this interesting, but they have not explored the details of the extensions.  The theorem is very complex but quite limited in scope, and maybe would be better as its own (math) result rather than as an add-on to this paper.  The application of CS is a bit limited, and the biggest benefits seem to be in the high-measurement and low or zero-noise case.
Overall, I lean toward weak reject, as I think the regularization parts of the paper would benefit from a major revision of the paper.
== Minor comments ==
- In section 3.1, the description of G is confusing. Writing G(z;w): R^k --> R^n is unclear. You mean G(.;w): R^k --> R^n. And calling z a "seed" confused me, until I realized you meant this is the "input" to the layer.
- statements like "we use early stopping, a phenomena that will be analyzed theoretically in Section 4" are misleading, since you don't analyze early stopping at all. You show that your model doesn't regularize, so that it will fit noise (in a special simplified case), so that running to convergence is a bad idea. This doesn't mean you know when to stop.
- Section 3.2, for p(y|w) and p(w), why not just say these are Gaussian and give the parameters?  In the sentence above eq (4), missing a "log" before "posterior".  The justification for the Gaussian weights (last paragraph on page 4) felt rather weak to me.  Like everyone for the past 60 years, you are adding a l2 regularization term because it's simple and it works fairly well and there are not that many alternatives.
- It took me a while to figure out that Sigma was L x L. The description here was confusing (in section 3.2.1), and it was unclear of Sigma_U was a sub-matrix (and if it was diagonal or a scaled identity) or a scalar.
- Thm 4.1:
-- "V is fixed at random" sounds like it would make a proper probabilist squirm!  The statement of the theorem is that "... holds ... with probability at least ...". You mean that this probability is due to the randomness in V?
-- You have a 1-layer network with 2 sets of weights, but in fact fix one set of weights; and it's not compressed sensing but denoising since A=I. I understand that results in this area are hard, but make sure not to oversell your result earlier in the paper, since this limited setting doesn't actually apply to the main problem you're interested in.
-- "While our proof is for the case of A=I, a similar result can be shown for other measurement matrices."  If that's true, then please show it!
- Table 1: for with and without LR, did you have a TV term for both?
- Section 5.2.1, "thus we can infer that assuming a learned Gaussian distribution over weights is useful." No, I don't think this is an appropriate assumption.  Maybe you can infer that shrinking the estimator toward a fixed vector (e.g., 0), and training to find how much to shrink, makes your solution more robust (a bias-variance tradeoff).  In particular, in Table 1, I'd like to see the improvements for one of these entries as a box plot, to see if the improvement is due to controlling bad outliers, or if actually the bulk is really improved.
- In bibliography, capitalize acronyms like "admm", "amp", "mri", etc.


In modern supervised learning, many deep neural networks are able to interpolate the data: the empirical loss can be driven to near zero on all samples simultaneously. In this work, we explicitly exploit this interpolation property for the design of a new optimization algorithm for deep learning. Specifically, we use it to compute an adaptive learning-rate in closed form at each iteration. This results in the Adaptive Learning-rates for Interpolation with Gradients (ALI-G) algorithm. ALI-G retains the main advantage of SGD which is a low computational cost per iteration. But unlike SGD, the learning-rate of ALI-G uses a single constant hyper-parameter and does not require a decay schedule, which makes it considerably easier to tune. We provide convergence guarantees of ALI-G in the stochastic convex setting. Notably, all our convergence results tackle the realistic case where the interpolation property is satisfied up to some tolerance. We provide experiments on a variety of architectures and tasks: (i) learning a differentiable neural computer; (ii) training a wide residual network on the SVHN data set; (iii) training a Bi-LSTM on the SNLI data set; and (iv) training wide residual networks and densely connected networks on the CIFAR data sets. ALI-G produces state-of-the-art results among adaptive methods, and even yields comparable performance with SGD, which requires manually tuned learning-rate schedules. Furthermore, ALI-G is simple to implement in any standard deep learning framework and can be used as a drop-in replacement in existing code.

This paper proposes a new gradient descent methods for training deep neural network which can take the adaptive step size with only one hyper-parameter to tune -- the maximum learning rate -- and achieve comparable results to stochastic gradient descent (SGD) on various tasks and models. In order to achieve that, they develop a stochastic extension of the Polyak step-size for the non-convex setting, namely the adaptive learning-rates for interpolation with gradients (ALI-G), in which the minimal value of the objective loss is set to 0 due to interpolation in neural networks and the learning rates are clipped by a chosen maximal value. The problem is formulated clearly, and the review on the Polyak step-size and related works are well done. Another main contribution of the paper is to provide the convergence guarantees for ALI-G in the convex setting where the objective loss is Lipschitz-continuous (Theorem 1 in the paper). Their theorem also takes into account the error in the estimate of the minimal value of the objective loss. In addition, they derive the connections between  ALI-G and SGD and show that compared to SGD, ALI-G take into consideration that the objective loss is non-negative and set the loss to 0 when it is negative. They perform empirical study to compare their algorithm with other methods including Adagrad, Adam, DFW, L4Adam and SGD on learning a differentiable neural computer, object recognition, and a natural language processing task. Their experimental results show that ALI-G performance is comparable with that of SGD with schedule learning rate.
Overall, this paper could be an interesting contribution. However, some points in the theory and experiments need to be verified. I weakly reject this paper, but given these clarifications in an author response, I would be willing to increase the score.
For the algorithm and theory, there are some points that need to be verified and further clarification on novelty:
1. When there are regularization such as the weight decay regularization, the minimal objective loss will not be 0. In such cases, Theorem 1 in the paper only guarantees that ALI-G reaches an objective loss less than or equal to a multiple of the estimate error \epsilon of the true minimal objective loss. It cannot guarantee that the objective loss reached by ALI-G can converge to the true minimall objective loss. Furthermore, when training neural networks with, for example, the weight decay regularization, often times the value of the regularization loss, i.e.  the estimate error \epsilon, is not small. Therefore, the upper bound given by Theorem 1 is rather loose.
2.  The paper mentions that when no regularization is used, ALI-G and Deep Frank-Wolfe (DFW) are identical algorithms. The difference between the two algorithms are when regularization is used. However, given my concern for Theorem 1 above, the convergence of ALI-G and advantage of ALI-G over DFW in this setting is questionable, and the claim that “ALI-G can handle arbitrary (lower-bounded) loss functions” also needs to be verified.
For the experiments, the following should be addressed:
1. In the experiment with the differentiable neural computers, even though ALI-G obtains better performance for a large range of \eta, its best objective loss is still worse than RMSProp, L4Adam, and L4Mom.
2. Given the merit of Theorem 1 is that the convergence guarantee takes into account the estimate error of the minimal objective loss, an ablation study that compares ALI-G with other methods in the same setting with and without regularization are needed. For example, it would be more convincing if similar results to those in Table 2 or 3 but without regularization are provided and discussed.
3. In Section 5.5, given that ALI-G and DFW are related, why is there no result for DFW in Figure
4.?
4.  As the paper mentions, AProx algorithm and ALI-G are related, why is there no comparison with AProx in the experiments?
Things to improve the paper that did not impact the score:
1. In all experiments, the performance differences between ALI-G and competitive methods are small. Thus, error bars are needed for these results.
2. In Table 3, the gap between SGD and ALI-G can be significantly different on different architectures. For example, in CIFAR-100 experiments, while ALI-G achieves the same result as SGD with the DenseNet, ALI-G’s performance on Wide ResNet is much worse than SGD. Do you have any explanation for this?
3. How is ALI-G compared with the methods proposed in the paper “Stochastic Gradient Descent with Polyak's Learning Rate” (https://arxiv.org/abs/1903.08688)
The following paper also proved the convergence of adaptive gradient methods for nonconvex optimization:
Dongruo Zhou*, Yiqi Tang*, Ziyan Yang*, Yuan Cao, Quanquan Gu. On the Convergence of Adaptive Gradient Methods for Nonconvex Optimization.
-------------------------------------------------------------
After rebuttal:
I noticed that the authors:
1. did not compare with SGD or Adam with good hyperparameters (https://arxiv.org/abs/1907.08610)
2. did not test on the large scale datasets, e.g., imagenet
3. the proposed algorithm is not as good as SGD on most numerical experiments.
For the above reasons, I think this paper should be rejected.


The trade-off between robustness and standard accuracy has been consistently reported in the machine learning literature. Although the problem has been widely studied to understand and explain this trade-off, the problem seems to remain as an open problem. In this paper, motivated by the fact that the high dimensional distribution is poorly represented by limited data samples, we introduce sensible adversarial learning and demonstrate the synergistic effect between pursuits of natural accuracy and robustness. Specifically, we define a sensible adversary which is useful for learning a defense model and keeping a high natural accuracy simultaneously. We theoretically establish that the Bayes rule is the most robust multi-class classifier with the 0-1 loss under sensible adversarial learning. We propose a novel and efficient algorithm that trains a robust model with sensible adversarial examples, without a significant drop in natural accuracy. Our model on CIFAR10 yields state-of-the-art results against various attacks with perturbations restricted to ∞ with = 8/255, e.g., the robust accuracy 57.23% against PGD attacks as well as the natural accuracy 91.51%.

Summary:
The paper studies the phenomenon of trade-off between robust and standard accuracies that is usually observed in adversarial training. Many existing studies try to understand this trade-off and show that it is unavoidable. In contrast, this work shows that under a sensible definition of adversarial risk, there is no trade-off between standard accuracy and sensible adversarial accuracy. It is shown that Bayes optimal classifier has optimal standard and sensible adversarial accuracies. The authors then go on to propose a new adversarial training algorithm which tries to minimize the sensible adversarial risk. Experimental results show that models learned through the proposed technique have high adversarial and standard accuracies.
Comments:
1) Sensible Adversarial Risk: The first contribution of the work is to define a new notion of adversarial risk which the authors call ''sensible adversarial risk'' and study its properties.  There is a recent work[1] which also proposes a new definition of adversarial risk and which is similar in spirit to what the current work tries to achieve.  In [1], the authors define a perturbation as adversarial only if it doesn't change the label of the Bayes optimal classifier, which is similar to what the current paper does. Owing to this similarity, the properties of sensible adversarial risk obtained in Theorems 1,2 in the current work look similar to [1]. So the authors should discuss/compare their results with [1].
2) Sensible Adversarial Training: I believe the major contribution of the paper is to propose an algorithm for minimizing sensible adversarial risk. However, I have some concerns with the proposed algorithm. The authors say that since the Bayes optimal classifier is unknown, they use a reference model f_r (which can be naturally trained). Consider the following scenario. Suppose the true data is given by (x, f^B(x)), for some unknown f^B; that is, the Bayes optimal classifier has perfect standard accuracy. Suppose f_r has perfect standard accuracy, but very bad adversarial accuracy on train and test sets. Suppose f_r is substituted for f^B in the sensible adversarial risk (with 0-1 loss). Then it is easy to see that f_r is a minimizer of the resulting objective. So the proposed algorithm will just output f_r in this scenario. This is clearly not desirable. Given this, I believe a more thorough understanding of the proposed algorithm is needed. When will the algorithm converge to non-robust classifiers? How should one initialize the algorithm to avoid such undesirable behavior?
While the notion of sensible adversarial risk is sensible, it is not clear why it should result in such high adversarial accuracies as reported in Tables 1,5. The adversarial perturbation of epsilon=8/255 on cifar10 is considered so small that sensible adversarial risk (with reference model f^B) at any point will almost always be equal to the existing notion of adversarial risk at that point. So, minimizing sensible adversarial risk (assuming you are given f^B) is exactly equivalent to minimizing adversarial risk. But it is known that minimizing adversarial risk results in models with low standard accuracies. So I feel sensible adversarial risk is not the reason behind such high accuracies. Could the authors explain what is the reason for such good adversarial accuracies reported in the paper?
3) Experiments:  While the experimental results on cifar10 look impressive, I have some concerns about the way the PGD attacks are run. I downloaded the model provided by the authors and ran PGD attack on it.  I ran L_infty attacks with epsilon=8/255, step size=2/255. The results I obtained seem to differ from the results presented in the paper:
PGD Steps | Adversarial accuracy of SENSE
20            62.09
50            60.34
100           59.99
I believe PGD attack with multiple random restarts will reduce the adversarial accuracy even further. Given this, I'd appreciate if the authors perform more careful attacks (with appropriate hyper-parameters) on their model. It'd also be great if the authors report the performance of PGD trained model using the same attacks used to report the performance on their model.
4) Other comments:  I'm not sure if the toy example (cheese hole distribution) in Section 2 is helpful. What the authors seem to conclude from it is that adversarial training can improve standard accuracy. But I do not agree with these conclusions. What if Figure 1b is the true distribution? Will the same conclusions hold? In general, these toy examples need not be illustrative of the behavior on real datasets. So instead of having these toy examples, I'd suggest the authors have a thorough discussion on theoretical and experimental results.
[1] Suggala, A. S., Prasad, A., Nagarajan, V., & Ravikumar, P. (2018). Revisiting Adversarial Risk. arXiv preprint arXiv:1806.02924.


We investigate the use of ellipsoidal trust region constraints for second-order optimization of neural networks. This approach can be seen as a higher-order counterpart of adaptive gradient methods, which we here show to be interpretable as first-order trust region methods with ellipsoidal constraints. In particular, we show that the preconditioning matrix used in RMSProp and Adam satisfies the necessary conditions for provable convergence of second-order trust region methods with standard worst-case complexities. Furthermore, we run experiments across different neural architectures and datasets to find that the ellipsoidal constraints constantly outperform their spherical counterpart both in terms of number of backpropagations and asymptotic loss value. Finally, we find comparable performance to state-ofthe-art first-order methods in terms of backpropagations, but further advances in hardware are needed to render Newton methods competitive in terms of time.

In this paper, the authors investigate the use of ellipsoidal trust region constraints for second order optimization. The authors first show that adaptive gradient methods can be viewed as first-order trust region methods with ellipsoid constraints. The authors then show that the preconditioning matrix of RMSProp and Adam can be used as norm inducing matrices for second order trust region methods. This ellipsoidal trust region method is empirically compared with first order gradient methods and spherical second order trust region methods.
Overall the paper is nicely written and very easy to read. The ideas are interesting. However, I have a number of concerns/questions about the work, that I list below.
1. Why is the preconditioning matrix of RMSProp/Adam a reasonable norm inducing matrix? One can show that the empirical Fisher is not an accurate curvature matrix in general, and so there is no reason to believe this would in fact enforce the proper ellipsoidal trust region for the method? See for example: https://arxiv.org/pdf/1905.12558.pdf.
2. I am also not convinced that Figure 2 actually shows that the curvature matrix is diagonally dominant. How do I interpret a value of 40 or 50 for this metric, and why does it imply that it is diagonally dominant?
3. The experiments also do not look very convincing to me. How sensitive is the algorithm to the hyperparameters like lambda1 and lambda2? I am also a bit confused about why different batch sizes were used for the first order gradient methods and the second order TR methods? The method overall doesn't seem to be able to match first order gradient methods, and it is not clear whether this is because of using the RMSProp/Adam preconditioner as a curvature matrix.
Given these concerns, I consider this paper to be borderline. I am happy to have a discussion with the authors and the other reviewers and change my score however.
=========================================
Edit after rebuttal:
I thank the reviewers for their response. While the updated paper has certainly improved, I think the paper still requires a much more thorough experimental evaluation. I am sticking to my score.


Convolutional Neural Networks (CNNs) typically treat normalization methods such as batch normalization (BN) and rectified linear function-like activations (e.g. ReLU) as building blocks. Previous work pointed out that learning feature channels with equal magnitudes is important for a CNN to achieve good generalization ability. However, the above "Norm+ReLU-like" basic block often learns inhibited channels that has small magnitude (i.e. contributes little to the feature representation), impeding both learning and generalization ability of CNNs. This problem is seldom explored in the literature. To mitigate the inhibited channels and encourage channels to contribute equally to the feature representation, we propose a new building block, Channel Equilibrium (CE), which is able to prevent inhibited channels in both experiments and theory. CE has several appealing properties. First, CE can be stacked after many different normalization methods such as BN and Group Normalization (GN), as well as integrated into many advanced CNN architectures such as ResNet and MobileNet V2 to form a series of CE networks (CENets), outperforming existing network architectures. Second, CE has an interesting connection with the Nash Equilibrium, a well-known solution of a non-cooperative game. Third, extensive experiments show that CE achieves state-of-the-art results on various challenging benchmarks such as ImageNet and COCO. The models and codes will be released.

This paper proposes a new block that gives diminishing
Pros)
(+) This paper is well-written and the idea looks interesting.
(+) The provided experiments look quite extensive. I like the experiments with both heavy and light network architectures to show the effectiveness of the proposed method.
(+) The authors provided plausible and sufficient backups for the claim.
(+) Connection with Nash equilibrium seems to be slightly overclaimed, but the attempt is novel and interesting.
Cons)
(-) The overall paper flow could be organized much better. I think some of the materials in the appendix needs to be placed in the main paper.
(-) Some of the notations in Section 3 need to be rewritten for better clarity.
(-) The terminology such as channel-level sparsity, channel equalization, and equivalent lambda looks vague. I recommend the authors define them for clarity.
Comments)
- I think the relationship between diminishing the sparsity on channels and improving the evaluation performance should be more addressed. If we change ReLU after Conv with other nonlinear functions such as ELU or PReLU, then the output hardly becomes zero, but as you know the performance usually gets worse. Then, how do you explain the necessity of reducing sparsities?
- How did you measure the sparsity in a plain network in Figure 1 and Figure 3? Did you pick a single layer or gather all the layers' sparsity? Please specify the way of measuring it.
- How did you plot the graphs in Figure 3.(a) and 3.(b). Did you pick a single channel randomly?
- To support the authors' claim, it would be better to show the sparsity ratio for a model (ResNet18, MobileNetV2, and so on) which shows better performance than each original one.
- How did you determine the transformation of F in the AII branch? Please provide any intuitions why the authors choose the transformation like in eq.(9).
- If CE-block actually conducts well right after a BN layer, then why CE block is attached only after the final BN layer in a bottleneck module? It would be better to provide any results by doing some studies when dealing with the intermediate BNs in a bottleneck module.
- I am wondering whether adaptive normalization techniques such as Instance-Batchnormalziation or Switchable Normalization methods could also give less sparsified features. Please clarify this by comparing with the proposed method.
- It seems that CE would have some extra computational costs compared with SE's. Please clarify why CE-module does have small computational costs.
- Please specify what MobileNetV2 has used. Looks like MobileNetV2x1.0 would be used.
- Why do you think the Top-5 accuracy of MobileNetV2 has not been improved?
- Can your method combine with ResNet with SE such as ResNet50SE?
- Please clarify the way of measuring the sparsity ratio in Figure 3d.   Why did you consider the sparseness of a feature by measuring the gamma in Figure 4? There may exist a channel that contains large magnitude values then gamma cannot make the output close to zero.
About overall rating)
The theoretical backups for the authors' claim look sound, and all the experiments including the performance improvement with several models seem to support the effectiveness of the idea very well. Specifically, I like all the analyses and the connection with Nash equilibrium, which are very intuitive and may provide further insights to researchers in this field.  I think this paper is clearly above the standard of ICLR. If the authors could address all the concerns above and refine the paper with better readability, then I could increase the score.


This work addresses class-specific object boundary extraction, i.e., retrieving boundary pixels that belong to a class of objects in the given image. Although recent ConvNet-based approaches demonstrate impressive results, we notice that they produce several false-alarms and misdetections when used in real-world applications. We hypothesize that although boundary detection is simple at some pixels that are rooted in identifiable high-frequency locations, other pixels pose a higher level of difficulties, for instance, region pixels with an appearance similar to the boundaries; or boundary pixels with insignificant edge strengths. Therefore, the training process needs to account for different levels of learning complexity in different regions to overcome false alarms. In this work, we devise a curriculumlearning-based training process for object boundary detection. This multi-stage training process first trains the network at simpler pixels (with sufficient edge strengths) and then at harder pixels in the later stages of the curriculum. We also propose a novel system for object boundary detection that relies on a fully convolutional neural network (FCN) and wavelet decomposition of image frequencies. This system uses high-frequency bands from the wavelet pyramid and augments them to conv features from different layers of FCN. Our ablation studies with contourMNIST dataset, a simulated digit contours from MNIST, demonstrate that this explicit high-frequency augmentation helps the model to converge faster. Our model trained by the proposed curriculum scheme outperforms a state-of-the-art object boundary detection method by a significant margin on a challenging aerial image dataset.
two-stage training scheme for class-specific object boundary estimation. This is a sample result from the model when applied for the problem of extracting road contours from aerial image tiles.
Class-specific object boundary extraction from images is a fundamental problem in Computer Vision (CV). It has been used as a basic module for several applications including object localization [Yu et al. (2018a); Wang et al. (2015)], 3D reconstruction [Lee et al. (2009); Malik & Maydan (1989); ], image generation [Isola et al. (2017); ], multi-modal image alignment Kuse & Shen (2016), and organ feature extraction from medical images [Maninis et al. (2016)]. Inspired from the sweeping success of deep neural networks in several CV fields, recent works [Yu et al. (2017); Acuna et al. (2019); Yu et al. (2018b)] designed ConvNet-based architectures
for object boundary detection and demonstrated impressive results. However, we notice that the results from these methods, as shown in Figure 1b, still suffer from significant false-alarms and misdetections even in regions without any clutter. We hypothesize that although boundary detection is simple at some pixels that are rooted in identifiable high-frequency locations, other pixels pose a higher level of difficulties, for instance, region pixels with an appearance similar to boundarypixels; or boundary pixels with insignificant edge strengths (ex: camouflaged regions). Therefore, the training process needs to account for different levels of learning complexity around different pixels to achieve better performance levels.
In classical CV literature, the different levels of pixel complexities are naturally addressed by decomposing the task into a set of sequential sub-tasks of increasing complexities [Leu & Chen (1988); Lamdan & Wolfson (1988); Kriegman & Ponce (1990); Ramesh (1995)]. Most often, boundary detection problem has been decomposed into three sub-tasks: (a) low-level edge detection such as Canny [Canny (1986)]; (b) semantic tagging/labeling of edge pixels [Prasad et al. (2006)] and (c) edge linking/refining [Jevtić et al. (2009)] in ambiguous regions. These approaches first solve the problem for simpler pixels (with sufficient edge strength) and then reason about harder pixels in the regions with ambiguous or missing evidence. However, with the advent of ConvNets, this classical perspective towards boundary extraction problem has been overlooked. New end-to-end trainable ConvNets have pushed the boundaries of state-of-the-art significantly compared to classical methods. However, we believe that classical multi-stage problem-solving schemes can help to improve the performance of ConvNet models. A parallel machine learning field, Curriculum Learning [Bengio et al. (2009)] also advocates this kind of multi-stage training schemes which train the network with a smoother objective first and later with the target task objective. These schemes are proven to improve the generalization of the models and convergences of training processes in several applications. Motivated by these factors, this work devises a curriculum-learning inspired two-stage training scheme for object boundary extraction that trains the networks for simpler tasks first (subtasks a and b) and then, in the second stage, trains to solve the more complex sub-task (c). Our experimental results on a simulated dataset and a real-world aerial image dataset demonstrate that this systematic training indeed results in better performances.
As mentioned already, the task of predicting object boundaries is mostly rooted in identifiable higher-frequency image locations. We believe that explicit augmentation of high-frequency contents to ConvNet will improve the convergence of training processes. Hence, this work designs a simple fully convolutional network (FCN) that takes in also high-frequency bands of the image along with the RGB input. Here in this work, we choose to use high-frequency coefficients from wavelet decomposition [Stephane (1999)] of the input image and augment them to conv features at different levels. These coefficients encode local features which are vital in representing sharp boundaries. Our empirical results convey that this explicit high-frequency augmentation helps the model to converge faster, especially in the first stage of curriculum learning.
In summary, our contributions in this work are the following:
• A novel two-stage training scheme (inspired from curriculum-learning) to learn classspecific object boundaries.
• A novel ConvNet augmented by high-frequency wavelets.
• A thorough ablation study on a simulated MNIST digit-contour dataset.
• Experiments with a challenging aerial image dataset for road contour extraction • A real-world application of road contour extraction for aligning geo-parcels to aerial imagery.
Related Work: The problem of extracting object boundaries has been extensively studied in both classical and modern literature of CV. Most of the classical methods start with low-level edge detectors and use local/global features to attach semantics to the detected pixels. They, later, use object-level understanding or mid-level Gestalt cues to reason about missing edge links and occluded boundaries. The work by Prasad et al. (2006) used local texture patterns and a linear SVM classifier to classify edge pixels given by Canny edge detector. The work by Mairal et al. (2008) reasoned on low-level edges, but learned dictionaries on multiscale RGB patches with sparse coding and used the reconstruction error curves as features for a linear logistic classifier. The work of Hariharan et al. (2011) proposed a detector that combines low-level edge detections and semantic outputs from pre-trained object detectors to localize class-specific contours.
Several recent works Yang et al. (2016); Yu et al. (2017) adopted fully convolutional networks (FCN) for the task of semantic boundary extraction. The work by Yang et al. (2016) proposed a FCN-based encoder-decoder architecture for object contour detection. The work of Bertasius et al. (2015) first used VGG-based network to locate binary semantic edges and then used deep semantic segmentation networks to obtain category labels. More recently, the work of Yu et al. (2017) proposed an FCN architecture, CASENet, with a novel shared concatenation scheme that fuses low-level features with higher conv layer features. The shared concatenation replicates lower layer features to separately concatenate each channel of the class activation map in the final layer. Then, a K-grouped 1 × 1 conv is performed on fused features to generate a semantic boundary map with K channels, in which the k-th channel represents the edge map for the k-th category. A few recent works Acuna et al. (2019); Yu et al. (2018b) integrated an alignment module into the network to account for noise in the contour labels. Most of these networks are trained in an end-to-end manner, using crossentropy-based objective functions. These objectives treat all boundary pixels equally irrespective of the complexity around them. Unlike existing methods, we use explicit high-frequency augmentation to ConvNet and train it in a curriculum learning scheme that accounts for different levels of pixel complexities with two stages.
2 THE PROPOSED CURRICULUM LEARNING SCHEME  (2012)] start with a smoothed objective function and gradually move to less smoothed functions. In other terms, these methods consider a class of objective functions that can be expressed as,
where C o (θ) is smoother or simple objective and C t (θ) is the target objective we wish to optimize. There are several ways to choose C o ; it can either be the same loss function as C t but solving the task on simpler examples, or be a proxy task simpler than the target task. In general, λ is a (binary) variable that takes values zero or one. It is set to zero initially and later increases to one. The epoch where it changes from zero to one is referred to as the switch epoch.
Curriculum Learning in CV: CL-inspired training schemes are recently gaining attention in CV fields. Recently some popular CV architectures leveraged CL based schemes to improve model generalization and training stabilities. In FlowNet 2.0 [Ilg et al. (2017)] for optical flow prediction, simpler training data are fed into the network first and then the more difficult dataset. The object detection framework of  first trains simpler networks (proposal, refiner-nets) and then trains the final output-net in the end. Here we propose a two-stage CL scheme for object boundary detection methods.
The proposed CL based training scheme for learning Object Boundaries: ConvNets for classspecific object boundary detection are in general trained with multi-label cross entropy-based objectives [Yu et al. (2017)].
where θ denotes the weights of the network; and p and k represent indices of pixel and class labels respectively.Ŷ and Y represent prediction and groundtruth label maps. β is the percentage of non-edge pixels in the image to account for skewness of sample numbers [Yu et al. (2017)]. This objective treats all the contour pixels equally and does not account for the complexity of the task around them. Here, we consider this as the target objective function, C t , that we wish to optimize.
We start the training, however, with a simpler task C o . We believe the pixels with strong edge strength are easy to localize and semantically identify. Hence, we propose to solve the task around those pixels in the first stage. We take element-wise multiplication between canny edge map of the input and dilated groundtruth label to prepare supervisory signal Z for this stage.
where E I is the Canny edge map of image I and Y D the dilated groundtruth map. This is as shown in Figure 2e. The objective function for this stage becomes;
Since we use a dilated version of GT in preparing Z, it also contains some non-object contour pixels. However, these might be refined in the second stage of CL, when trained with Y (Eq 2). Hence, the CL objective function in Eq 1 uses Eq 4 and Eq 2 as initial and target objective functions respectively. In the CL training scheme, we set the switch epoch as T /2, where T is the total number of training epochs.

Summary: The suggest two improvements to boundary detection models: (1) a curriculum learning approach, and (2) augmenting CNNs with features derived from a wavelet transform. For (1), they train half of the epochs with a target boundary that is the intersection between a Canny edge filter and the dilated groundtruth. The second half of epochs is with the normal groundtruth. For (2), they compute multiscale wavelet transforms, and combine it with each scale of CNN features. They find on a toy MNIST example that the wavelet transform doesn’t impact results very much and curriculum learning seems to provide some gains. On the Aerial Road Contours dataset, they find an improvement of ~15% mAP over the prior baseline (CASENet).
I have several concerns with this work:
* The idea of using wavelet transforms to augment CNNs has been more thoroughly explored in prior work (e.g., see [1]).
* No comparison to existing SOTA segmentation models (e.g., [2]). These semantic / instance segmentation models can easily be adapted to the task of boundary detection. I suspect the baseline here is weak.
* Section 6 is severely unfinished. The explanation is sparse and there are no quantitative results -- just the output of the model overlaid on one example.
* The choice of curriculum learning task is arbitrary, and there are no ablations explaining why this is a reasonable task. For example, what about random subsets of pixels? At the moment, it offers no insight for practitioners.
* There are no ablations for the Aerial Road Contours experiments. This seems necessary because it is the only realistic dataset evaluated in this work. The MNIST experimental results appear qualitatively different from the Contours experiment. For example, they show that wavelet features do not make much of a difference, but does it make a difference for Contours?
Altogether, this work unfortunately offers few insights to vision practitioners, let alone general practitioners. Substantial work needs to be devoted to expanding experimental coverage.
[1] Wavelet Convolutional Neural Networks. Shin Fujieda, Kohei Takayama, Toshiya Hachisuka
[2] TensorMask: A Foundation for Dense Object Segmentation. Xinlei Chen, Ross Girshick, Kaiming He, Piotr Dollár


Graph representation learning serves as the core of many important prediction tasks, ranging from product recommendation in online marketing to fraud detection in financial domain. Real-life graphs are usually large with complex local neighborhood, where each node is described by a rich set of features and easily connects to dozens or even hundreds of neighbors. Most existing graph learning techniques rely on neighborhood aggregation, however, the complexity on reallife graphs is usually high, posing non-trivial overfitting risk during model training. In this paper, we present Neural Sparsification (NeuralSparse), a supervised graph sparsification technique that mitigates the overfitting risk by reducing the complexity of input graphs. Our method takes both structural and non-structural information as input, utilizes deep neural networks to parameterize the sparsification process, and optimizes the parameters by feedback signals from downstream tasks. Under the NeuralSparse framework, supervised graph sparsification could seamlessly connect with existing graph neural networks for more robust performance on testing data. Experimental results on both benchmark and private datasets show that, NeuralSparse can effectively improve testing accuracy and bring up to 7.4% improvement when working with existing graph neural networks on node classification tasks.

The authors propose a supervised graph sparsification technique that "mitigates the overfitting risk by reducing the
complexity of input graphs."
The idea is as follows: there is a sparsification network which samples subgraphs (adjacency matrices) by computing a probability distribution for each edge and drawing the existence of each edge from this distribution. The sparsified graph is then fed to a GNN that computes a classification loss. Since the authors use the Gumbel-softmax trick the method is end-to-end differentiable and the output consists of a node classifier and a graph generative model that can be used to sample sparsified graphs.
The paper covers an interesting topic and results in some good numbers on standard benchmark datasets. I also like the idea to use the Gumbel-softmax trick to make the entire model differentiable.
Unfortunately, the authors miss to cite and discuss highly related work [1] (ICML 2019). In this work, the authors also maintain a graph generative model (also by parameterizing edges but with iid Bernoulli RVs), also sampling graphs from this generative model, and also using these sampled graphs to train a GNN for semi-supervised node classification. The resulting model is also end-to-end differentiable. Instead of using a Gumbel-softmax to keep the method differentiable (given that we have discrete random variables) the authors propose a novel way to compute gradients for the parameters of the Bernoulli RVs by posing the problem as a bilevel optimization problem. In [1] the graph cannot only be sparsified but also enriched with edges that might be beneficial for the classification accuracy. Indeed, it was shown that adding edges is more beneficial than removing edges. The results on Cora and Citeseer are better than the results reported by the authors in this submission. At the very least, the authors should familiarize themselves, discuss, and compare empirically to [1].
The existence of this previous work also reduces the novelty of the proposed approach. However, if a  comparison to [1] would be added, the submission could be seen as an alternative instance of the framework presented in [1]. I would be willing to increase my score to accept, if the authors provide such a comparison in an updated version.
[1] https://arxiv.org/abs/1903.11960
-----
I've read the rebuttal and while I don't agree necessarily with all statements made by the authors, I am happy to increase my score based on the discussion of previous work.


Deep Graph Neural Networks (GNNs) are instrumental in graph classification and graph-based regression tasks. In these tasks, graph pooling is a critical ingredient by which GNNs adapt to input graphs of varying size and structure. We propose a new graph pooling operation based on compressive Haar transforms, called HaarPooling. HaarPooling is computed by following a chain of sequential clusterings of the input graph. The input of each pooling layer is transformed by the compressive Haar basis of the corresponding clustering. HaarPooling operates in the frequency domain by the synthesis of nodes in the same cluster and filters out fine detail information by compressive Haar transforms. Such transforms provide an effective characterization of the data and preserve the structure information of the input graph. By the sparsity of the Haar basis, the computation of HaarPooling is of linear complexity. The GNN with HaarPooling and existing graph convolution layers achieves state-of-the-art performance on diverse graph classification problems.

Summary
This paper proposed a new graph pooling method based on the Haar basis on graphs. The authors argued that existing graph pooling methods have drawbacks (ignored node features, ignored the hierarchical structure of a graph or, computationally expensive in terms of space or time) and that the proposed method overcame these drawbacks. The proposed method extracts the low-frequency components of signals in terms of Haar decomposition on a chain of graphs. By the property of the Haar basis, we can compute the pooling matrix with the complexity proportional to the graph size. The paper empirically compared the proposed method with state-of-the-art pooling methods and graph NNs using well-known datasets.
Decision
Although this paper gave a novel pooling method by incorporating the Wavelet theory, I still have questions for the effectiveness of the proposed method in the real datasets and whether the proposed method solved the problems of existing methods the authors mentioned in the introduction (see the Suggestion section). Besides, I think the authors can improve the organization of the paper to maximize the value of the paper. Therefore, I judge the paper as a border, tending to reject for now.
Suggestions
- The main part of the paper is 10 pages long. I think the authors can polish the organization of the paper to fit the recommended page size (i.e., 8 pages).
- I want to know how many times the authors ran experiments for each configuration.  I think the variance of test accuracies are critical since the difference in performance between methods is not significant. Adding error bars to the experiment results are preferrable if authors ran experiments multiple times.
- The authors claimed in the introduction that the drawback of existing methods is their time or space complexity and that the proposed method is computationally inexpensive. I think it is better to emphasize that HaarPooling does solve the questions. Although the authors demonstrated that the proposed algorithm is fast compared to the naive implementation, the comparison with other pooling methods are missing. For example, adding a comparison table in terms of time and space complexity of pooling methods is one idea. Another idea is to demonstrate it empirically by providing time and memory consumption to the experiment results.
Minor Comments
- Introduction
	- Haar Pooling is computed following a chain... → Haar Pooling is computed by following ...
- page 1, section 1, paragraph 1
	- Graph classification and regression are a very different kind of task
	- → At first sight, I have thought that this sentence discusses the difference between classification and regression tasks. Could you reconsider the sentence?
- page 3, section 2, paragraph 1
	- The authors used the term "chain," whose definition is available in the later section (Sec. 3). I think this terminology is not common in the literature of graph NNs (at least I did not come up with the definition from this term). Could you consider to add the definition (or brief explanation) of the term when you use it for the first time?
- page 5, section 3, Chain of graph by clustering
	- Write what $w$ is in the definition of a graph $\mathcal{G}=(V, E, w)$.
- page 5, section 3.1, Orthogonality
	- Define $l_2(\mathcal{G}_j)$ (I can imagine its definition, though).
- page 5, section 3.1, Haar basis
	- For two consective layers $j$, $j+1$ ...
	- → I had a little difficulty in understanding this sentence. Could you reconsider the wording?
-  page 13, Appendix B, (10)
	-  I think "1" in this equation is an all-one vector. It is better to write differently the all-one vector and the scalar one.
-  page 5, section 3.1 and page 7, (8)
	-  For the node $v$ in the $j$-th graph and $v'$ in the $(j+1)$-th graph, the authors use both of $v\in v'$ (section 3.1) and $v'\in Pa_{\mathcal{G}}(v)$ (equation (8)) to denote the parent-child relationship. It is better to align the notation.
- page 13, Appendix B
	- Could you write in the appendix how to construct the compressed Haar from the full counterpart (I suppose we should choose bases corresponding to $k=1$)?
Questions
- In the paragraph starting with Locality on page 5, the paper says that the number of different values of the Haar basis is bounded by constant. I want the authors to make this sentence more precise. To be more specific, what "the different values" means (I imagine that it means the number of scalar values appearing as a component of some basis) and what "constant" means (with respect to which variable the "constant" is constant?)
- I think that the complexity of the direct matrix product is $O(N^2C)$, where $N$ is the graph size and $C$ is the channel size (correct me if I am wrong). Therefore the result of Figure 5 was surprising to me because the direct matrix product takes approximately $O(N^3)$ time.  These look inconsistent to me unless the channel size $C$ is proportional to the node size $N$.
- Equation (5) claimed that the compressive Haar basis approximately keeps the length of an input signal. I want to know how we can justify it (theoretically or empirically).  I think that it is empirically correct for the Fourier basis that signal carries "information" in the low-frequency domains and that high-frequency areas are noisy (e.g., NT and Maehara (2019); Ma et al., 2019). If this is the case for Haar transform, too, I understand that equation (5) holds and gives the justification that the proposed method can extract the information from lower frequencies.