Attention mechanisms have been boosting the performance of deep learning models on a wide range of applications, ranging from speech understanding to program induction. However, despite experiments from psychology which suggest that attention plays an essential role in visual reasoning, the full potential of attention mechanisms has so far not been explored to solve abstract cognitive tasks on image data. In this work, we propose a hybrid network architecture, grounded on selfattention and relational reasoning. We call this new model Attention Relation Network (ARNe). ARNe combines features from the recently introduced Transformer and the Wild Relation Network (WReN). We test ARNe on the Procedurally Generated Matrices (PGMs) datasets for abstract visual reasoning. ARNe excels the WReN model on this task by 11.28 ppt. Relational concepts between objects are efficiently learned demanding only 35% of the training samples to surpass reported accuracy of the base line model. Our proposed hybrid model, represents an alternative on learning abstract relations using self-attention and demonstrates that the Transformer network is also well suited for abstract visual reasoning.

This work proposes a new architecture for abstract visual reasoning, based on Transformer-style soft attention and relation networks. The authors test their network on the PGM dataset, and demonstrate a non-trivial improvement over previously reported baselines.
In general, abstract reasoning is an important field of current study in neural network-based machine learning, as it is an area that has notoriously eluded these types of models historically. The paper is reasonably well put together, and I have no reason to question the various technical aspects of the work.
Unfortunately, I think there are significant shortcomings. Firstly, the PGM dataset was designed to stress out-of-distribution generalization, and performance on the Neutral split was not proposed as a particularly interesting challenge on its own. This is because, as the name implies, abstract reasoning requires the ability to identify abstract conceptual features of the data and compose them in novel ways at test time, which is *not* a feature of the neutral split.  The authors are encouraged to run their model on these other generalization splits.
Second, there seems to be little value to the field overall for research involving minor architectural improvements for single datasets. If the authors believe in this method, they are encouraged to demonstrate its effectiveness on a wide variety of data types. On this note, I should add that the authors are incorrect to state that this is the first work to use self-attention for abstract reasoning (please see Zambaldi, 2018 for one example of many papers that have incorporated self-attention into convolutional architectures).
So to sum up, while this work broaches an interesting subject and is technically fine, it does not surpass the threshold for acceptance because it fails to demonstrate the usefulness of the method on the task at hand, as well as broad utility of the proposed method.


While meta-learning approaches that utilize neural network representations have made progress in few-shot image classification, reinforcement learning, and, more recently, image semantic segmentation, the training algorithms and model architectures have become increasingly specialized to the few-shot domain. A natural question that arises is how to develop learning systems that scale from few-shot to many-shot settings while yielding competitive performance in both. One scalable potential approach that does not require ensembling many models nor the computational costs of relation networks, is to meta-learn an initialization. In this work, we study first-order meta-learning of initializations for deep neural networks that must produce dense, structured predictions given an arbitrary amount of training data for a new task. Our primary contributions include (1), an extension and experimental analysis of first-order model agnostic meta-learning algorithms (including FOMAML and Reptile) to image segmentation, (2) a novel neural network architecture built for parameter efficiency and fast learning which we call EfficientLab, (3) a formalization of the generalization error of meta-learning algorithms, which we leverage to decrease error on unseen tasks, and (4) a small benchmark dataset, FP-k, for the empirical study of how meta-learning systems perform in both fewand many-shot settings. We show that meta-learned initializations for image segmentation provide value for both canonical few-shot learning problems and larger datasets, outperforming random and ImageNet-trained initializations for up to 400 densely labeled examples. Finally, we show both theoretically and empirically that a key limitation of MAML-type algorithms is that when adapting to new tasks, a single update routine is used that is not conditioned on the available data for a new task. We find that our network, with an empirically estimated optimal update procedure yields state of the art results on the FSS-1000 dataset, while only requiring one forward pass through a single model at evaluation time.

This paper proposes to apply MAML-style meta-learning to few-shot semantic segmentation in images. It argues that this type of algorithm may be more computationally-efficient than existing methods and may offer better performance with a higher number of examples. They further propose to perform hyper-parameter search to choose a new learning rate for the inner learning process after optimizing for the network parameters.
As far as I know, this is the first paper to apply gradient-based (i.e. MAML-style) meta-learning to this specific problem. Existing approaches to few-shot semantic segmentation have mostly used multi-branch conv-networks to condition the output on the training examples. This paper shows that (FO)MAML achieves similar accuracy to the FSS-1000 baseline. This is an empirical contribution in itself. The paper also demonstrates that the EfficientNet architecture can be applied to segmentation.
Major concerns:
(1.1) The improvement obtained by the hyper-parameter optimization seems quite marginal (79.0 - 81.4 and 73.3 - 73.9) and there is no study of the variance of the results. The fact that better performance is obtained by tuning the learning rate on the *training* set suggests to me that the improvement might not be significant. You could repeat the experiment by sampling multiple different training and testing sets (with different classes) to estimate (some of) the variance.
(1.2) The formalization in Section 4 is mathematically appealing but seems unnecessary. In the end, the paper is essentially arguing that it's better to use different learning rates (for the inner loop) during meta-training and meta-testing. This seems obvious, since this includes equal learning rates as a special case. The paper proposes to optimize the latter learning rate using a gradient-free method. This argument can be made without considering generalization bounds.
(1.3) One of the central claims of the paper is that "meta-learned representations smoothly transition as more data becomes available." It's not entirely clear what this means. I suppose it means that, with few shots, it should perform as well as existing few-shot methods, but with many shots, it should perform as well as a standard learning algorithm. The paper failed to present any evidence that existing algorithms for few-shot segmentation do not satisfy this property. It would strengthen the argument to include an existing few-shot segmentation algorithm in Figure 2.
(1.4) The details of the experiment in Figure 2 are not clear. Was a different number of iterations used when there are hundreds of examples? Were different hyper-parameters used when optimizing from a pre-trained checkpoint? It would be unfair to use the same hyper-parameters which had been optimized specifically for the meta-learned initialization.
Other issues:
(2.1) It's not clear what it means to achieve human-level performance in the few-shot task and the many-shot task. What is human-level performance at few-shot segmentation? It seems to me that humans are capable of segmenting novel objects (i.e. zero-shot) with almost perfect accuracy. Does this mean that your method should achieve the same accuracy with few- and many-shots?
(2.2) The use of early stopping was unclear. Do you use a fixed number of SGD iterations during training and a variable number of iterations (determined by a stopping criterion) during testing? However, this seems to be contradicted by the statement that the UHO algorithm determined an optimal number of iterations (8) for testing? On the other hand, this seems like too few iterations with hundreds of shots. Maybe the automatic stopping criterion was only used with many shots? Or maybe training proceeds until either the stopping criterion is satisfied or the maximum number of iterations is reached? Furthermore, the early stopping criterion was not specified.
(2.3) Missing reference: Meta-SGD (arxiv 2017) considers a different learning rate for every parameter and updates the learning rates during meta-training .
(2.4) There was no discussion of the running time of different methods. This would be particularly interesting in the many-shot regime. How slow are the RelationNet approaches?
(2.5) It is claimed that Figure 1 demonstrates that  "the estimated optimal hyperparameters for the update routine ... are not the same as those specified a priori during meta-training". However, it seems that the optimal learning rate is awfully close to the dotted blue line (within the variance of the results).
(2.6) For the IOU loss (equation 10), what are the predicted y values? Are they arbitrary real numbers? Do you use a sigmoid to constrain them to real numbers in [0, 1]?
Minor:
(3.1) It is not worth stating the optimal learning rate to more than 3 or 4 significant figures.
(3.2) Use 1 \times 1 instead of 1 x 1.
(3.3) Is there a reference for the Dice score? Where does the name come from?


Learning in the presence of label noise is a challenging yet important task, and it is crucial to design models that are robust when part of the dataset are mislabeled. In this paper, we discover that a new class of loss functions called the gambler's loss provides strong robustness to label noise across various levels of corruption. Training with this modified loss function reduces memorization of data points and is a simple yet effective method to improve robustness and generalization. Moreover, using this loss function allows us to derive an analytical early stopping criterion that accurately estimates when over-memorization begins to occur. Our overall approach achieves strong results and outperforming existing baselines.

Update after rebuttal:
The good:
The rebuttal and updated paper address many of my concerns. Most importantly, the updated paper demonstrates the three-stage phenomenon on Open Images and adds experiments on IMDB showing that the Gambler's loss with AES helps a lot. The LAES iteration introduced in the updated paper alleviates my concern about performance drop compared to the CT baseline at certain corruptions on CIFAR-10.
The bad:
- From Figure 12, it looks like the three-stage phenomenon doesn't hold on IMDB. Does AES provide additional benefit beyond the Gambler's loss on IMDB? This needs to be clarified with the way Figure 12 turned out.
- There is a serious missing citation [1] that should be included as a baseline. The proposed method in [1] is at least superficially similar to the Gambler's loss and also makes use of the fact that it is easier to fit clean labels than noisy labels. My apologies for not noticing this earlier.
Overall:
I would still suggest acceptance, because the three-stage phenomenon is an interesting find that the authors make good use of. In light of the missing citation, though, I cannot raise my score.
[1]: Zhilu Zhang, Mert R. Sabuncu. "Generalized Cross Entropy Loss for Training Deep Neural Networks with Noisy Labels". NeurIPS 2018.
-----------------------------------------------------------------------
Summary:
This paper proposes a method to alleviate label noise. It opens with the observation of three distinct stages when training in the presence of label noise. Importantly, there is a ‘gap’ stage during which the network has not begun memorizing noisy labels and early stopping is ideal. The authors then observe that the Gambler’s loss (Ziyin et al., 2019) elongates the gap stage and propose an analytic early stopping (AES) criterion for identifying when to stop training.
The analysis of the AES criterion, e.g. in Figure 5, and the observation of a phase transition when tuning the o hyperparameter are quite interesting, and the latter observation is of practical value when using the AES criterion.
The AES criterion seems to be well-motivated, and the empirical evaluation of the Gambler’s loss with and without early stopping is good. The results are strong on MNIST but somewhat weak on CIFAR-10. Specifically, the improvements on CIFAR-10 only appear for large corruption rates (0.7+), and performance is lower than the baselines for other corruption rates. This is a worrying problem, because it calls into question the value of the method on larger problems. However, seeing as this is a distinct approach from the baselines and that it demonstrates some promise, I recommend borderline accept. The authors could raise my score by demonstrating more consistent gains on another larger-than-MNIST CV dataset or an NLP/speech dataset. Other points of concern that I have are listed below.
Major points:
At the top of page 3, the authors say that the idealized gap assumption “holds well for simple datasets such as MNIST and on datasets with very high corruption rate, where our method achieves best results, and less so on more complicated datasets such as CIFAR10”. The idealized gap assumption is behind the AES criterion, but Figure 5 suggests that the AES criterion works well on CIFAR-10, so what do the authors mean when they say the assumption doesn’t work as well on CIFAR-10? Is this just referring to the results?
Saying traditional label noise correction methods are “of no use when one is not aware of the existence of label noise” seems unfair. The FC method and others do not require foreknowledge of the corruption rate and do not harm performance in the absence of label noise, so they can also be said to automatically correct label noise.
“FC, however, requires knowing the whole transition matrix, and is outperformed significantly by our method.”
This is not quite true, because Patrini et al. propose an estimate of the transition matrix as part of the Forward correction. Did you use the estimated or true transition matrix for the FC method? It would be good to clarify this in the paper.
Minor points:
There are a few grammatical errors and typos in the paper:
“or explicit regularization, this is also what is suggested by Abiodun et al. (2018)” (run-on sentence)\
“CIFAR10” should be “CIFAR-10”


Network quantization is one of the most hardware friendly techniques to enable the deployment of convolutional neural networks (CNNs) on low-power mobile devices. Recent network quantization techniques quantize each weight kernel in a convolutional layer independently for higher inference accuracy, since the weight kernels in a layer exhibit different variances and hence have different amounts of redundancy. The quantization bitwidth or bit number (QBN) directly decides the inference accuracy, latency, energy and hardware overhead. To effectively reduce the redundancy and accelerate CNN inferences, various weight kernels should be quantized with different QBNs. However, prior works use only one QBN to quantize each convolutional layer or the entire CNN, because the design space of searching a QBN for each weight kernel is too large. The hand-crafted heuristic of the kernel-wise QBN search is so sophisticated that domain experts can obtain only sub-optimal results. It is difficult for even deep reinforcement learning (DRL) Deep Deterministic Policy Gradient (DDPG)-based agents to find a kernel-wise QBN configuration that can achieve reasonable inference accuracy. In this paper, we propose a hierarchical-DRL-based kernel-wise network quantization technique, AutoQ, to automatically search a QBN for each weight kernel, and choose another QBN for each activation layer. Compared to the models quantized by the state-of-the-art DRL-based schemes, on average, the same models quantized by AutoQ reduce the inference latency by 54.06%, and decrease the inference energy consumption by 50.69%, while achieving the same inference accuracy.

This paper proposes a new method for quantizing neural network weights and activations that uses deep reinforcement learning to select the appropriate bitwidth for individual kernels in each layer. The algorithm uses a reward function that weights accuracy of the quantized model with latency, energy, and FPGA area, and leverages a high level and low-level controller to create quantized models that can take into account these factors. Compared to prior approaches taht only perform layer-wise instead of kernel-wise quantization, the quantized models can achieve better performance, or latency.
While the problem of more effectively quantizing neural network weights and activations is interesting, I found this paper hard to follow and evaluate. The proposed hierarchical deep RL method mentions a number of tricks and design decisions that are not ablated, and the notation and text around the method were difficult to follow. There were also no baselines comparing to other approaches for performing kernel-wise quantization (e.g. random search, quantize based off variance, etc.). Without improved baselines and text I would recommend rejecting this paper.
Major comments:
* Based on the analysis of kernel-wise results, it seems that a very simple strategy that chooses QBNs based on weight variance could be sufficient to achieve good performance. However, there’s no comparisons to these kinds of heuristics or even to random search over QBN per-kernel. There are also no ablations that study whether the hierarchical-RL based approach beats a baseline that just directly predicts the kernel-wise QBNs independently.  Without these baselines, it’s hard to know how well AutoQ works.
* The text that details the hierarchical RL approach with multiple controllers that is core to the new AutoQ approach is extremely hard to follow, with many undefined symbols.
Minor comments:
* FIg 1: where in the neightwork do these weight kernels come from? What network/dataset?
* You repeatedly state that ML experts obtain only sub-optimal results at selecting QBNs, can you cite something for this? What if you give experts access to additiional information like visualizations of weight distributions?
* What’s the tradeoff between using different QBNs for each kernel and specifying this number? Is there additional memory overhead?
* Table 1: hard to read the exponent in kernel-wise math, add parentheses
* Please use \citep vs. \citet where appropriate
* Why not also search for kernel-specific activation quantization?
* The notation in Fig 3 and surrounding text is really hard to follow, e.g. w_w, h_w, and indices into states (roundup could be ceil, names are confusing iRd, eRd?).
* Eqn 2 comes out of nowhere… why optimize for log(accuracy) vs accuracy? How do you choose the user-defined constants? Decay factor?
* Hardware overhead estimator: are these models accurate independent of QBN?
* Why do you have to perform Gaussian augmentation? I didn’t follow the re-labeling of transitions after Eqn. 6, is this needed?
* Define \mu when you first use it (Eqn 4?)
* Is \delta_a in implementation details the same as \sigma around Eqn 4?
* Table 3: Is there variance in these results based off fine-tuning? Could you include error bars or stderr on these estimates?
* Figure 6: are the search spaces different for these different approaches? I.e. does AutoQ perform better due to the upper/lower bounds you set on QBN?
=============
Thank you to the authors for addressing many of my concerns. The updated paper still does not present any baselines for methods that can search or learn kernel-wise quantization parameters, e.g. run random search and select the best-performing models according to your reward function. Additionally, there were no revisions made to the text to improve the presentation, and thus I continue to recommend rejection of this work.


Graph Neural Networks (GNNs) broadly follow the scheme that the representation vector of each node is updated recursively using the message from neighbor nodes, where the message of a neighbor is usually pre-processed with a parameterized transform matrix. To make better use of edge features, we propose the Edge Information maximized Graph Neural Network (EIGNN) that maximizes the Mutual Information (MI) between edge features and message passing channels. The MI is reformulated as a differentiable objective via a variational approach. We theoretically show that the newly introduced objective enables the model to preserve edge information, and empirically corroborate the enhanced performance of MI-maximized models across a broad range of learning tasks including regression on molecular graphs and relation prediction in knowledge graphs.

This paper proposed an auxiliary loss based on mutual information for graph neural network. Such loss is to maximize the mutual information between edge representation and corresponding edge feature in GNN ‘message passing’ function. By assuming a gaussian distribution of edge feature given edge representation, the training can be done efficiently with tractable density. Experiments on molecule regression and knowledge graph completion show better performance than MPNN.
Overall the paper is written in a clear way which is easy to follow. The idea of using mutual information as some kind of regularization is also interesting. However, there are some concerns I have with the paper:
Regarding formulation
1. The derivation up to Eq(8) looks fine to me, where the assumptions are reasonable. However from Eq(8) one can see this is reduced to an ‘auto-encoder’ type of regularization, where one can have a trivial solution for reconstruction -- the identity network, when the hidden dimension is larger than input dimension. And in this paper, dimension of W should always be larger than the dimension of e (for example, in molecules e should be low dimension vector with the bond type, distance, etc., while W should have dimension that matches the node embeddings).
I think the original loss (i.e., the supervised MSE, cross entropy etc) would help a bit with such degenerated case, but it is possible that the learned f(e) contains both identity mapping (or equivalent) and the representation that contributes to original loss.
2. Actually I’m also not sure if I get the motivation here. If one needs to do this regularization for edges, why don’t we consider this auxiliary loss for node embeddings as well? As in molecules, atoms have more interesting features than bonds, which should account more if the mutual information loss is needed.
Regarding experiment
1. In Figure 1, the training loss of EIGNN is better than MPNN. This is a bit counterintuitive to me, as I think the auxiliary is a kind of regularization -- which might help with generalization but not necessarily the training loss.
2. The original paper of MPNN reports the relative MAE. Is it possible to report the results using the same metric as previous paper? It would make the comparison more consistent -- though showing the improvement in current way is not too bad.
3. I think one simple ablation study would to concat the edge feature directly inside the ‘message passing’ procedure, or have some ‘residual’ type of connection for edge features.


Graphs are known to have complicated structures and have myriad applications. How to utilize deep learning methods for graph classification tasks has attracted considerable research attention in the past few years. Two properties of graph data have imposed significant challenges on existing graph learning techniques. (1) Diversity: each graph has a variable size of unordered nodes and diverse node/edge types. (2) Complexity: graphs have not only node/edge features but also complex topological features. These two properties motivate us to use a multiplex structure to learn graph features in a diverse way. In this paper, we propose a simple but effective approach, MxPool, which concurrently uses multiple graph convolution networks and graph pooling networks to build hierarchical learning structure for graph representation learning tasks. Our experiments on numerous graph classification benchmarks show that our MxPool has marked superiority over other stateof-the-art graph representation learning methods. For example, MxPool achieves 92.1% accuracy on the D&D dataset while the second best method DiffPool only achieves 80.64% accuracy.

This paper introduces a new hierarchical graph representation learning method for graph classification. It builds upon the diffpool method. Specifically, the authors propose the multiplex convolution and the multiplex pooling operation. The multiplex convolution learns multiple graph convolution operations (potentially with different parameters) and merges them using a neural network. Similarly, the multiplex pooling learns multiple assignment matrices utilizing diffpool and merges them using a neural network. The effectiveness of the proposed model is demonstrated with experiments in several standard benchmark datasets for graph classification.
Strengths
The overall architecture of the proposed method can be regarded as an extension of diffpool method by using multiple convolution and pooling operations. An analysis on how the number of convolution and pooling operations can affect the performance of the model shows the effectiveness of the proposed multiplex convolution/pooling operations. The effectiveness of the proposed model is further verified by the experimental results on 5 standard benchmark datasets.
Weakness
The novelty of the paper is limited. The major contribution of this paper is to utilize multiple convolution/pooling operations and merge them with neural networks.
It is not clear how issues mentioned in the motivational examples shown in Figure 1 can be solved by the proposed model. As discussed in Figure 1(a) and Figure 1(b), small graphs may prefer small embedding while large graphs may prefer large embedding. However, it is unclear how the proposed model achieves this goal as the combination of the embeddings from different convolution operations is through a neural network that is shared by all graphs. Similarly, for Figure 1(c) and Figure 1(d), it is not clear how the proposed model can focus more on one type of information than the other. It would be helpful if there are some empirical results to demonstrate this.
Recommendations
It would be better if the authors can provide some analysis on time complexity of the proposed model. The major concern of the efficiency is the introducing of fully connected graph after each pooling as mentioned in Section 3.3.
Minor comments
In section 3.3, “is a GCN different” -> “is a GCN variant”?
In Section 3.3, should there be a softmax function after Eq.(5)?
In section 4.2, “in ter90o8ims” -> “in terms”?


The question of whether deep neural networks are good at generalising beyond their immediate training experience is of critical importance for learning-based approaches to AI. Here, we consider tests of out-of-sample generalisation that require an agent to respond to never-seen-before instructions by manipulating and positioning objects in a 3D Unity simulated room. We first describe a comparatively generic agent architecture that exhibits strong performance on these tests. We then identify three aspects of the training regime and environment that make a significant difference to its performance: (a) the number of object/word experiences in the training set; (b) the visual invariances afforded by the agent's perspective, or frame of reference; and (c) the variety of visual input inherent in the perceptual aspect of the agent's perception. Our findings indicate that the degree of generalisation that networks exhibit can depend critically on particulars of the environment in which a given task is instantiated. They further suggest that the propensity for neural networks to generalise in systematic ways may increase if, like human children, those networks have access to many frames of richly varying, multi-modal observations as they learn.

=============================== Update after revisions =====================================================
In my initial review, I had raised some issues with the interpretation of the results and suggested some control experiments to tighten the conclusions. The authors chose to weaken their initial claims by rephrasing their conclusions instead. I understand that there may not have been enough time to run many of the experiments I suggested, but I still think they are worth considering for the future. I'm mostly satisfied with the rephrasing of the conclusions in the revised paper, so as promised, I'm happy to increase my score and recommend acceptance.
I spotted several typos in the revised paper, however: section 4.1: "we choose to consider negation ...", p. 5: "for for ...", a citation on p. 5 is not compiled correctly. There may be more. For the final version please make sure to go through the paper thoroughly a couple of times and fix all the typos.
========================================================================================================
The authors present a systematic study of generalization in agents embedded in a simulated 3d environment. I think there are some interesting results in this paper that might be useful for people to know about. I appreciate the thoroughness of the experiments, in particular. I have, however, some issues with the interpretation of several of the main results. I would be happy to increase my score if we can resolve some of these issues. Here are my main concerns:
1) In the experiments in section 3, only a limited test set is used. How is the train/test split decided in these experiments? Table 6 suggests that you have a much larger repository of objects. Why not use all possible objects in the test set? It is a bit premature to declare your results as systematic generalization if you can’t show that it actually works for a much larger set of test objects (ideally for all possible objects).
2) Section 4.1: in these experiments, the training set size is increased, but the test set size is kept constant (and small), so the train/test size ratio also increases. So, an alternative explanation of the results in this section is that the model behaves largely according to visual similarity and as the training set size is increased, it becomes easier to find a training set object that is visually similar to any test set object. I think the authors should run an experiment where both training and test set sizes increase by the same amount so that the train/test set size ratio stays constant. If the model can’t achieve systematic generalization in that case, it would be wrong to conclude, as the authors do now, that increasing the training set size itself improves systematic generalization. The correct conclusion would rather be that increasing the train/test size ratio improves generalization, which is a weaker conclusion. Please note that the results in this section are quite similar to those in Lake & Baroni (2018) and in Bahdanau et al. (2018) (see their Figure 3). Bahdanau et al. (2018), for example, also show that increasing train/test set size ratio (their “rhs/lhs” ratio) improves generalization in generic neural networks. It is interesting to note, however, that neither Lake & Baroni (2018) nor Bahdanau et al. (2018) interpret these results positively (i.e., these results don’t show systematicity), whereas the current paper seems to put a more positive spin on essentially the same result. I think these earlier results should be explicitly discussed here and the authors should justify why they are interpreting the results differently (if they are). It should also be noted that in the real world the train/test size ratio for humans is presumably very small, perhaps zero (given the compositional abilities of humans).
3) Section 4.3: I don’t think the results in this section are sufficient to establish the egocentric frame per se as the key factor. One possibility is that perhaps the frame doesn’t have to be centered on the agent, but as long as it has some systematic relationship to the agent’s location (for example, the center of the visibility window could be some distance away from the agent, and the agent itself may or may not be inside this window), that’s good enough to get generalization improvements. An even weaker possibility is that simply a moving frame is enough for improved generalization. In this case, the reference frame doesn’t even need to have a systematic relationship to the agent’s location. For example, the frame could be relative to a fictitious agent that randomly explores the environment. I think the authors should run some experiments to rule out these possibilities if they want to claim that the egocentric frame itself is responsible for generalization improvements.
4) Section 4.4: In the experiments in this section, I think there are two relevant factors that need to be better disentangled: 1) the number and variability of image frames experienced by the two models; 2) the active perception aspect (the fact that the agent interacts with the environment and affects its own perceptual experience in one case). The authors claim the second factor as the key aspect enabling better generalization, but 1) is equally likely (this would be more in line with a standard data augmentation type result). A good control experiment here would be to not just use the first frame but a larger number of more variable frames for training the non-situated agent (for example, one can use image frames that would be seen by a camera that more or less randomly moves in front of the objects perhaps with the constraint that both objects are always at least partially visible). If the classification model generalizes as well as the situated agent in this control condition, you cannot claim active perception as the key factor.
5) As a more general point, it’s a bit frustrating to have to judge systematic generalization by only looking at the results of some limited set of experiments. How do I interpret the results if the agent achieves only 84% accuracy in some experiment (as opposed to 100%)? It would be much better if the authors could somehow more rigorously prove systematicity. Here, I don’t necessarily mean “prove” in a mathematical sense, but just analyzing the learned representations a bit more rigorously and being able to say something along the lines of: here’s exactly how the trained model represents “lift”; because of reason X, Y, Z, this representation is completely disentangled from all object representations in the dataset (and ideally from all possible object representations, because that’s really what true systematicity entails, although I highly doubt that any generic model of the type studied in this paper will be able to achieve this, regardless of the amount and type of input it receives).
More minor issues:
6) In Table 5, “table lamp” appears both in training and test sets. Is this a typo?
7) Some results are presented in the appendix without any mention in the main text (Appendix D. 2). I think this is not a good practice in general. In the main text, please make sure to mention, however briefly, every result that appears in the appendix (something along the lines of "This result could not be explained by confound X or Y (Appendix Z)" would suffice).
8) Font size in Figure 2 is tiny (axis labels are impossible to read), please make it bigger. You don’t need that many ticks on the axes.


The tremendous success of deep neural networks has motivated the need to better understand the fundamental properties of these networks, but many of the theoretical results proposed have only been for shallow networks. In this paper, we study an important primitive for understanding the meaningful input space of a deep network: span recovery. For k < n, let A ∈ R k×n be the innermost weight matrix of an arbitrary feed forward neural network M : R n → R, so M (x) can be written as M (x) = σ(Ax), for some network σ : R k → R. The goal is then to recover the row span of A given only oracle access to the value of M (x). We show that if M is a multi-layered network with ReLU activation functions, then partial recovery is possible: namely, we can provably recover k/2 linearly independent vectors in the row span of A using poly(n) non-adaptive queries to M (x). Furthermore, if M has differentiable activation functions, we demonstrate that full span recovery is possible even when the output is first passed through a sign or 0/1 thresholding function; in this case our algorithm is adaptive. Empirically, we confirm that full span recovery is not always possible, but only for unrealistically thin layers. For reasonably wide networks, we obtain full span recovery on both random networks and networks trained on MNIST data. Furthermore, we demonstrate the utility of span recovery as an attack by inducing neural networks to misclassify data obfuscated by controlled random noise as sensical inputs.

** Summary
The paper studies how to recover the span of a NN from a limited number of queries. The problem belongs to the general question of how to reconstruct functions from black-box interaction and it may find application in obfuscation attacks where very large perturbations of the input do not affect the output. The main contribution of the paper is on the theoretical analysis of a simple non-adaptive and a more sophisticated adaptive algorithm. The main finding is that under mild conditions on the structure of the NN partial recovery is possible. The empirical validation show that in practice, it is often the case the full span recover is actually possible, as the structure and weights of common NN are "friendly" enough.
** Evaluation
While the content of the paper lies a bit off my expertise, my impression is that this is a solid technical and theoretical contribution.
Detailed comments:
1- Theoretical results: The properties proved in Thm.3.4 and 4.3 are quite powerful, showing that (partial/approximate) span recovery is possible with a relatively small amount of samples (of order of n*k, where n is the original input size and k is the size of the span), in a computationally efficient way (in particular for the non-adaptive algorithm), and for Relu NN or NN with differentiable layers and final threshold function. The main question is of course the validity of the assumptions needed to prove the theorems. Asm.1 and 2 are overall reasonable and they are very well supported by Lemma 3.1. The first two assumptions in page 6 are straightforward, while I'll less convinced of 3 and 4. In fact, they need to hold for any subspace V of any dimension smaller than k. I wonder whether the assumptions may become less and less likely as the size of the subspace decrease.
2- The theorems and the paper are mostly well written but some parts may be clearer.
3- Alg.1: the computation of the gradient is never really explained apart from the high-level lemma 3.2. While an actual algorithm is reported in the appendix, it would be better to have it explained already in the main text.
4- Right after Lemma 3.2 it is said "which demonstrates the claim". I am not sure which claim it refers to.
5- In alg.1 there is a parameter r which defines the number of queries of the algorithm. Thm 3.4 provides an upper bound on the number of queries needed and it does depend on k. Since k is initially unknown, how do you actually parameterize the algorithm? is there a stopping condition that can be tested?
6- In Thm 3.4 it is said that the algorithm returns the subspace V in time that is polynomial in the main parameters of the problem. Yet, I'm not sure where such complexity comes from. In Alg.1 it seems like the subspace is the direct output of the algorithm, so the complexity is r times the cost of computing the gradient, which according to Lem3.2. is poly(n). Is this the way you finally obtain the complexity?
7- One thing I'm doubtful about is the fact that the result in Thm 3.4 seems to be independent from the depth d and width k_i of the different layers. Some conditions may be implicit in the Asm.1 and 2, though. Furthermore, in the experiments it is clearly showed that thin NNs may make the support not recoverable. Could you please make such limit more explicit in the theory?
8- In alg.4 I think lines 5-7 are just the way to execute line 4. Is that correct? If not, how do you execute line 4?
9- In alg.4 line 8 and 9 are not easy to follow and they are not really discussed in the main text. Could you please clarify?
10- The empirical validation is relatively simple but it illustrates quite well the theory. Still I wish the authors could report results that dig more in detail in the theoretical results showing how tight they are (e.g., in the dependency on n, k, and other factors). The current results provide just a hint on how accurate/informative the theory is.
11- In the empirical result, it would be great to have a much more thorough validation of the difference between the non-adaptive and the adaptive algorithms. In the current results it seems like there is very limited difference.


Variance reduction methods which use a mixture of large and small batch gradients, such as SVRG (Johnson & Zhang, 2013) and SpiderBoost (Wang et al.,  2018), require significantly more computational resources per update than SGD (Robbins & Monro, 1951). We reduce the computational cost per update of variance reduction methods by introducing a sparse gradient operator blending the top-K operator (Stich et al., 2018;Aji & Heafield, 2017) and the randomized coordinate descent operator. While the computational cost of computing the derivative of a model parameter is constant, we make the observation that the gains in variance reduction are proportional to the magnitude of the derivative. In this paper, we show that a sparse gradient based on the magnitude of past gradients reduces the computational cost of model updates without a significant loss in variance reduction. Theoretically, our algorithm is at least as good as the best available algorithm (e.g. SpiderBoost) under appropriate settings of parameters and can be much more efficient if our algorithm succeeds in capturing the sparsity of the gradients. Empirically, our algorithm consistently outperforms SpiderBoost using various models to solve various image classification tasks. We also provide empirical evidence to support the intuition behind our algorithm via a simple gradient entropy computation, which serves to quantify gradient sparsity at every iteration.

This paper aims at improving the computational cost of variance reduction methods while preserving their benefits regarding the fast provable convergence. The existing variance reduction based methods suffer from higher per-iteration gradient query complexity as compared to the vanilla mini-batch SGD, which limits their utility in many practical settings. This paper notices that, for many models, as the training progresses the gradient vectors start exhibiting structure in the sense that only a small number of coordinates have large magnitude. Based on this observation, the paper proposes a modified variance reduction method (by modifying the SpiderBoost method), where a 'memory vector' keeps track of the coordinates of the gradient vectors with large variance. Let $d$ be the size of the model parameter. During each iteration, one computes the gradient for $k_1$ coordinates with the highest variance (according to the memory vector) and an additional $k_2$ random coordinates.
The paper shows that in the worst case, the proposed method has the same gradient query complexity as the SpiderBoost variance reduction method. Assuming that the proposed method can track the sparsity of the gradient vector, the proposed method achieves a gradient query complexity which is $O(\sqrt{(k_1 + k_2)/d})$ times that of the SpiderBoost method. The paper demonstrates the gradient query complexity improvement over the SpiderBoost method on MNIST and CIFAR-10 data set.
The paper presents novel results by utilizing the ideas from the field of communication-efficient distributed optimization. As far as the reviewer can tell, the results in the paper are correct. That said, there is quite a bit of room for improvement in terms of the writing of the paper.
The paper appears to have way too many typos. For example,
In Section 2.1:
- Why is $k$ introduced?
- $S$ denotes a random subset with size $k$ ---> $k_2$?
- drawn from the set ${\ell : |y_{\ell}| < |y_{(k)}|}$ ----> ${\ell : |x_{\ell}| < |x_{(k_2)}|}$?
- $rtop(x, y) = (0, 12, 0, 0, 1)$ --> $rtop(x, y) = (0, 16, 0, 0, 1)$
In Lemma 1:
 - while defining $top_{-k_1}(x, y)$, ".... if |x_{\ell}| >= |x_{(k_1)}|" ----> ".... if |x_{\ell}| <= |x_{(k_1)}|"?
In Section 2.2:
 - What are $g_0, g_1,..., g_{L-1}$? Shouldn't these be $\phi_0, \phi_1,..., \phi_{L-1}$?
In A1:
 - right after (5), what is $\tilde{x}_0$ in the definition of $\Delta_f$?
The authors may also consider making the empirical evaluation more comprehensive by considering tasks from the NLP domain, e.g., language modeling. This would further help asses the utility of the proposed method.


This paper proposes a differentiable linear quadratic Model Predictive Control (MPC) framework for safe imitation learning. The infinite-horizon cost is enforced using a terminal cost function obtained from the discrete-time algebraic Riccati equation (DARE), so that the learned controller can be proven to be stabilizing in closed-loop. A central contribution is the derivation of the analytical derivative of the solution of the DARE, thereby allowing the use of differentiation-based learning methods. A further contribution is the structure of the MPC optimization problem: an augmented Lagrangian method ensures that the MPC optimization is feasible throughout training whilst enforcing hard constraints on state and input, and a prestabilizing controller ensures that the MPC solution and derivatives are accurate at each iteration. The learning capabilities of the framework are demonstrated in a set of numerical studies.

This paper shows how to make infinite-horizon MPC differentiable
by differentiating through the terminal cost function and controller.
Recent work in non-convex finite-horizon continuous control [1,2,3] face
a huge issue in selecting the controller's horizon length and
better-understanding differentiable infinite horizon
control has potentially strong applications in these domains.
As a step in this non-convex direction, this paper provides a nice
investigation in the convex LTI case.
The imitation learning experiments on a small spring dynamical
system are a necessary sanity check for further work, but
many other more complex systems could be empirically studied
and would have made this paper stronger.
One point that would be useful to clarify: the DARE solution in (7,8) is
derived to optimally control a LTI system *without* control/state bounds but
is then used to control the LTI system *with* control/state bounds in (4).
Does this lead to suboptimal solutions to the true infinite-horizon problem?
[1] Chua, K., Calandra, R., McAllister, R., & Levine, S. Deep reinforcement learning in a handful of trials using probabilistic dynamics models. NeurIPS 2018.
[2] Hafner, D., Lillicrap, T., Fischer, I., Villegas, R., Ha, D., Lee, H., & Davidson, J. Learning latent dynamics for planning from pixels. ICML 2019.
[3] Tingwu Wang, Xuchan Bao, Ignasi Clavera, Jerrick Hoang, Yeming Wen, Eric Langlois, Shunshi Zhang, Guodong Zhang, Pieter Abbeel, Jimmy Ba. Benchmarking Model-Based Reinforcement Learning. arXiv 2019.


This paper studies theoretically the capacity limits of graph neural networks falling within the message-passing framework (GNN mp ). Two main results are presented. First, GNN mp are shown to be Turing universal under sufficient conditions on their depth, width, node identification, and layer expressiveness. Second, it is discovered that GNN mp can lose a significant portion of their power when their depth and width is restricted. The proposed impossibility statements stem from a new technique that enables the repurposing of seminal results from theoretical computer science and leads to lower bounds for an array of decision, optimization, and estimation problems involving graphs. Strikingly, several of these problems are deemed impossible unless the product of a GNN mp 's depth and width exceeds (a function of) the graph size; this dependence remains significant even for tasks that appear simple or when considering approximation.

Summary
This paper studied the expressive power of graph NNs, specifically, their universality and limitations under the non-anonymous setting, via the theory of distributed computations. For the universality, it proved the Turing completeness of graph NNs if messaging and aggregation functions are sufficiently strong. For the limitation, it characterized the lower bound of width for solving graph-theoretic tasks (such as subgraph detection, subgraph verification, approximate, and exact optimization problems) using graph NNs. The key idea is to reduce the computation model of graph NNs to LOCAL (for Turing completeness) or CONGEST (for limitations), which are well-studied in the literature of distributed computations and use the known results for these models.
Decision
This paper gave us a new approach to analyzing the expressive power of graph NNs. Not only does this paper give new theoretical results, but also it opens the door to a new research direction by bridging the theories of graph NNs and distributed computations. However, I cannot confirm the correctness of the proof of Theorem 3.1 (see Suggestions section). For now, I am tending to accept the paper. But I want to determine the final decision after I am certain that the proof of the theorem is correct.
We can roughly divide existing approaches for studying the expressive power of graph NNs into two. One is to compare the power of discriminating non-isomorphic graph pairs with isomorphism tests such as the WL isomorphism test (Xu et al., 2019). The other one is to theoretically justify the oversmoothing phenomena (Li et al., 2018). The proof techniques the authors used are different from both of the two. It related a graph NN to the computational models LOCAL and CONGEST, and enabled to incorporate the theory of distributed computations. By doing so, the authors successfully derived many lower bounds in a systematic way, proving the effectiveness of their strategy. I think we can expect that a more refined analysis inspired by this approach will appear in the future.
Regarding the Experience Assessment: I have published several papers in graph NNs (4). But I do not know much about the area of the theory of distributed algorithms (1--3).
Suggestions
- Section 3.2
	- Theorem 3.1 proves the equivalence of GNN_n and LOCAL. However, the definition of equivalence is missing. Please write it in the main part, since this theorem is the key result of this paper.
	- I could not find any reference for the Turing completeness of the LOCAL model. Could you add the reference for it?
	- The description of the CONGEST model is only available in the appendix informally (Appendix B.3). Could you write it in the main part?
	- The authors emphasized the importance of the universality and limitation results in the introduction and paragraph after Corollary 3.1. In my opinion, the importance of such tasks is in machine learning community (Cybenko's paper on the universality of MLPs (Cybenko, 1989) is one of the most cited papers in the community). Rather, I think many graph NN researchers who are expected to read this paper are not familiar with the theory of distributed computations. Therefore, I would recommend to use page resources to explain the basic concepts of the distributed computation theory.
Questions
- Is there any existing work which tries to solve the graph theoretical tasks using graph NNs? If there is, can the theorems in this paper give explanations for the results?
[Cybenko, 1989] Cybenko, George. Approximation by superpositions of a sigmoidal function. Mathematics of control, signals and systems 2.4 (1989): 303-314.
[Li et al., 2018] Qimai Li, Zhichao Han, and Xiao-Ming Wu. Deeper Insights into Graph Convolutional Networks for Semi-Supervised Learning. In Proceedings of the 32nd AAAI Conference on Artificial Intelli- gence, pp. 3538–3545, 2018.
[Xu et al., 2019] Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural networks? In International Conference on Learning Representations, 2019.


Statistical inference methods are fundamentally important in machine learning. Most state-of-the-art inference algorithms are variants of Markov chain Monte Carlo (MCMC) or variational inference (VI). However, both methods struggle with limitations in practice: MCMC methods can be computationally demanding; VI methods may have large bias. In this work, we aim to improve upon MCMC and VI by a novel hybrid method based on the idea of reducing simulation bias of finite-length MCMC chains using gradient-based optimisation. The proposed method can generate low-biased samples by increasing the length of MCMC simulation and optimising the MCMC hyper-parameters, which offers attractive balance between approximation bias and computational efficiency. We show that our method produces promising results on popular benchmarks when compared to recent hybrid methods of MCMC and VI.

This paper proposes a new combination of Markov chain Monte Carlo (MCMC) and variational inference (VI) for improving approximate inference. The main contribution is the optimization objective that allows improving the quality of samples obtained from the combination of VI and MCMC. Specifically, the authors minimize the "approximate" version of the Kullback-Leibler (KL) divergence between the distribution of MCMC + VI and the true distribution. The authors validate the effectiveness of their formulation through experiments on 6 synthetic benchmarks and generative modeling of MNIST (experiments on Bayesian neural networks are also provided in the appendix).
Overall, I think the paper provides a solid contribution towards combining MCMC and VI by proposing a way to optimize the MCMC part. The experiments validate the method by showing consistent improvement over existing methods. However, I believe the justification behind the proposed formulation, i.e., Equation (4) and (5), needs to be improved before being published at the conference.
First, for Equation (4), the explanation behind "replacing" H(P_{T}) with ELBO w.r.t. P_{0} is confusing. Specifically, it is reasoned that ELBO w.r.t. P_{t} only increase after MCMC steps. This statement is misleading since the replacement was done for H(P_{T}), not the ELBO w.r.t. P_{T}.
I also think the Equation (5) is not properly justified. it is stated that the constraint is needed for preventing P_{T} to be closer to P_{0}. However, nothing is stated about the reason on why P_{T} gets closer to \pi when Equation (5) is satisfied.  Note that even if the expected log-likelihood of the distribution is high, it does not necessarily mean that the distribution is more similar.
Minor comments:
- I was unable to understand why the algorithm is named "ergodic" inference. Both HVI and the proposed EI rely on the ergodic property of Markov chain for improving the variational distribution. I hope the authors could better illustrate on this point. I also think the term "ergodic approximation" in page 3. is hard to understand.
- I (weakly) suggest changing y-axis of Figure 5. to log-scale for better readability. It almost seems that the brown plot does not converge in Fig 5-(a).
- The paper could have been strengthened by performing experiments on more challenging datasets, e.g., CIFAR-10 or CIFAR-100.


Maximum Common Subgraph (MCS) is defined as the largest subgraph that is commonly present in both graphs of a graph pair. Exact MCS detection is NPhard, and its state-of-the-art exact solver based on heuristic search is slow in practice without any time complexity guarantee. Given the huge importance of this task yet the lack of fast solver, we propose an efficient MCS detection algorithm, NEURALMCS, consisting of a novel neural network model that learns the nodenode correspondence from the ground-truth MCS result, and a subgraph extraction procedure that uses the neural network output as guidance for final MCS prediction. The whole model guarantees polynomial time complexity with respect to the number of the nodes of the larger of the two input graphs. Experiments on four real graph datasets show that the proposed model is 31.78× faster than the exact solver while achieving near-perfect accuracy in MCS detection.

This paper proposes a novel algorithm NeuralMCS for maximum common subgraph (MCS) identification. The proposed algorithm consists of two components. One is a neural-network model based on Graph Matching Networks (GMN, Li et al, 2019) to learn a node-to-node matching matrix from examples of the ground-truth MCS results. Another is the algorithm called GSE (Guided Subgraph Extraction) to obtain an MCS by making an explicit assignment from the estimated matching matrix by the NN model. Experimental comparisons are made to other NN-based approaches combined with threshold-based assignments by the Hungarian algorithm (w.r.t the accuracy) and to a state-of-the-art exact algorithm MCSplit, and show the effectiveness of NeuralMCS.
This paper proposes an interesting method for MCS detection which would have large application interests. Though the basic idea is nice,  the reported performance gains would be a bit less convincing due to the following evaluation problem and its weak novelty.
The algorithm has two parts, and the first NN part to learn a matching matrix is mostly based on the already existing algorithm of GMN (Li et al, 2019). The novel part would be primarily in post-processing normalization (described in 3.1) for the matching matrix and seems to also be applicable to other NNs (for example, GAT?). The second part GSE to get an explicit subgraph also seem to be applied independently to the first part. We can see that combining these two parts worked, but it is unclear how each component contributes to the performance gain compared to any possible alternatives of each part.
I understand that MCS detection from a matching matrix (and node state vectors) is not exact if we just use Hungarian-like linear assignment problem (LAP) solvers for a submatrix obtained by a simple thresholding, but both post-processing normalization and GSE parts (which brings the novelty) can be more carefully evaluated through some 'ablation studies' using some simple alternative substitutes.


The lottery ticket hypothesis proposes that over-parameterization of deep neural networks (DNNs) aids training by increasing the probability of a "lucky" sub-network initialization being present rather than by helping the optimization process . Intriguingly, this phenomenon suggests that initialization strategies for DNNs can be improved substantially, but the lottery ticket hypothesis has only previously been tested in the context of supervised learning for natural image tasks. Here, we evaluate whether "winning ticket" initializations exist in two different domains: natural language processing (NLP) and reinforcement learning (RL). For NLP, we examined both recurrent LSTM models and large-scale Transformer models (Vaswani et al., 2017). For RL, we analyzed a number of discrete-action space tasks, including both classic control and pixel control. Consistent with work in supervised image classification, we confirm that winning ticket initializations generally outperform parameter-matched random initializations, even at extreme pruning rates for both NLP and RL. Notably, we are able to find winning ticket initializations for Transformers which enable models one-third the size to achieve nearly equivalent performance. Together, these results suggest that the lottery ticket hypothesis is not restricted to supervised learning of natural images, but rather represents a broader phenomenon in DNNs.

This paper studies the existence of “winning ticket” initialization in natural language processing (NLP) and reinforcement learning (RL). The lottery ticket hypothesis has been found effective in over-parameterized deep neural networks, which provides better sub-network initialization if not outperforming the original full network. Experiments show that winning ticket initializations generally outperform parameter-matched random initializations on recurrent LSTM models, large-scale Transformer, and discrete-action space RL tasks, including both classic control and pixel control.  A substantial number of parameters can be saved.
This paper is clearly motivated and easy to follow.  The results are interesting.  However, my major concern is its intellectual merit.  The paper does not seem to propose any new algorithm, and the application of lottery ticket (late rewinding) to LSTM, Transformer, and RL looks quite straightforward. There are a few insights drawn from the experiment, such as when only the Transformer weights are pruned.  However they does not appear substantial. Overall, I do not find the paper innovative enough for publication at top conferences like ICLR.  I understand this is subjective, so I leave it to the AC for further evaluation.


The posteriors over neural network weights are high dimensional and multimodal. Each mode typically characterizes a meaningfully different representation of the data. We develop Cyclical Stochastic Gradient MCMC (SG-MCMC) to automatically explore such distributions. In particular, we propose a cyclical stepsize schedule, where larger steps discover new modes, and smaller steps characterize each mode. We prove non-asymptotic convergence theory of our proposed algorithm. Moreover, we provide extensive experimental results, including ImageNet, to demonstrate the effectiveness of cyclical SG-MCMC in learning complex multimodal distributions, especially for fully Bayesian inference with modern deep neural networks.

The paper propose a new MCMC scheme which is demonstrated to perform well for estimating Bayesian neural networks. The key idea is to not keep lowering the step sizes, but -- at pre-specified times -- go back to large step sizes.
The paper is timely, the proposed algorithm is novel, and the theoretical analysis also seem quite novel.
My key concern is that with MCMC sampling it is often quite difficult to tune parameters, and by introducing more parameters to tune when step sizes should increase, I fear that we end up in a "tuning nightmare". How sensitive is the algorithm to choice of parameters?
I would expect that the proposed algorithm is quite similar to just running several MCMCs in parallel. The authors does a comparison to this and show that their approach is significantly faster due to "warm restarts". Here I wonder how sensitive this conclusion is to choice of parameters (see nightmare above) ? I would guess that opposite conclusions could be reached by tuning the algorithms differently -- is that a reasonable suspicion ?
It is argued that the cyclic nature of the algorithms gives a form of "warm start" that is beneficial for MCMC. My intuition dictate that this is only true of the modes of the posterior are reasonable close to each other; otherwise I do not see how this warm starting is helpful. I would appreciate learning more about why this intuition is apparently incorrect.
Minor comments:
* on page 4 it is stated that the proposed algorithm "automatically" provide the warm restarts -- but is it really automatic? Isn't this a priori determined by choice of parameters for the algorithm?
* It would be good to use \citet instead of \cite at places, e.g. "discussed in (Smith & Topin, 2017)" should be "discussed by Smith & Topin (2017)". This would improve readability (which is generally very good).
* For the empirical studies I think it would be good to report state-of-the-art results as well. I expect that the Bayesian nets still are subpar to non-Bayesian methods, and I think the paper should report this.


How well can we estimate the probability that the classification predicted by a deep neural network is correct (or in the Top 5)? It is well-known that the softmax values of the network are not estimates of the probabilities of class labels. However, there is a misconception that these values are not informative. We define the notion of implied loss and prove that if an uncertainty measure is an implied loss, then low uncertainty means high probability of correct (or top k) classification on the test set. We demonstrate empirically that these values can be used to measure the confidence that the classification is correct. Our method is simple to use on existing networks: we proposed confidence measures for Top k which can be evaluated by binning values on the test set.

Summary: This paper proposes an uncertainty measure called an implied loss. The authors suggest that it is a simple way to quantify the uncertainty of the model. It is suggested that "Low implied loss (uncertainty) means a high probability of correct classification on the test set.". They suggest that the analysis of an implied loss justifies the maximum confidence value of softmax-cross entropy. They also extend to evaluate Top-k uncertainty (the uncertainty whether our prediction is in the Top-5 maximum values of our confidence score or not).
========================================================
Clarity:
I found that this paper content does not seem to be difficult mathematically, however, it is difficult to follow the paper and here I list several parts that can potentially be improved:
1. INTRO: the sentence "Our implied loss interpretation justifies both methods, since we demonstrate that "both these quantities" are uncertainty measures.". What is both quantities here, the maximum softmax probability and something?
2. INTRO: first contribution, accurate estimates of the probability that the classification of the model on a test set is correct. What do you mean by this sentence? I couldn't see the accuracy of the estimation in Table 2.
3. second contribution in INTRO: what is the meaning of "a consistent manner" here. If it means "successfully", is there a case that your method fail? It would be nice to be precise when describing the contribution. Figure 2 is more like a small example but can be considered difficult to convince to the readers about the capability of the proposed method.
4. Figure 1 can be much improved. I found the caption is hard to understand. Loss (y-axis) may be written as Kullback-Leibler loss to be more precise in this context (if I understood correctly).
minor comment: Figure 1(a) [colon missing], since Figure 1(b) has ":" right after. The authors made an effort to explain the color in the caption of Figure 1(b) but the explanation of (yellow) is missing. The explanation of how to train a network to get Figure 1(a) seems to be missing. It seems -log(f_(1)) should be -log(f^{sort}_(1)). Finally, I'm not sure what is P(Correct|x). Is this histogram suggested that when the U_1 is large and the P(Correct|x) is small, we get a correct prediction, or maybe this means the ratio of correct prediction under the value of loss (histogram)? I think it would help the reader to make it more concise. In the description in page 4 -\log(p_{max}) seems never define before, was it a typo?
5.1 First page, last sentence: what do you mean by the current setting. Before this point there is no explanation about the problem setting, only we are interested in quantifying an uncertainty measure of the networks.
5.2 First page, last sentence:  I'm not familiar with Bayes factors, is the last sentence your contribution or it's the finding of the existing work, if it's the latter one, it would be nice to cite them. However, I found this sentence a bit vague: Bayes factor more informative (not sure what is the definition of informative here) better than Brier score under the situation where prob of correct classification is high (is this means high accuracy on the test set?).

6. PRIOR-WORK: Although the authors suggested many existing works, it would be highly useful if the authors discuss the relationship between existing works and their proposed work, e.g., where to put your work in the literature. And since they proposed an uncertainty function, it would be nice to see a few definitions of uncertainty existing works described (doesn't need to be mathematical formulation, I think just an intuitive explanation is sufficient).
7. I am confused with the definition of an implied loss. It is first defined in page 3 with a fixed k (as 1) as a loss where the prediction is correct but in Eq. (4), it looks like a set with one element where y is the maximum prediction score, not a correct label. Then there is U_k(x), if k!=1, is this an implied loss? Although in (5) it is a real number, not a set anymore, I read this paper and took it as a "yes U_k(x) is also an implied loss". Also it would be better to define Kullback-Leibler here to be concise and kind to the readers. Then in def 4.1, it's mentioned that the uncertainty measure U_k(x) is an implied loss if the event has high expected loss, does that mean if the event has low expected loss, it is not an implied loss? My opinion is that the authors may use a definition environment and define precisely what is an implied loss. For example, given "\ell: X x Y \to R, a correct label y \in Y, and an integer k <= K (number of classes), an implied loss is defined as" to avoid confusion.
8. Def 4.1: U(x) without subscript is undefined (perhaps U_k(x)?). What is an element of the set S^\eps_k? If it is a set then what is the meaning of the event S has a high expected loss? Does the definition of implied loss after Eq. (6) and the definition of implied loss in Eq. (5) identical when it is Kullback-Leibler loss?
9. Theorem 4.2: S^\eps seems to be undefined without k. Moreover, how to interpret the bound in (7), it would be nicer to explain the bound after stating the theorem. It is only an inequality that says the left-hand side is smaller or equal to the right-hand side. And does (7) hold for any y? And how tight is the bound?
10. Remark 4.3: what is e_k? what is k here if k in U is set to one? And the name of the remark, how to interpret (8) as "Neural networks are always overconfident?" Is this about neural networks or this apply to any function?
11. Sec 5: Figure 6 is not in the main body but the appendix... If this a mistake (and the paper is supposed to be 9 pages without ref.) or it is supposed to be in the appendix? If It's in the appendix, it would be better to mention that this figure is in the appendix.
12. Sec5: since Bayes factor is highly used in this paper to motivate the use of the measure, I don't think it is a good idea to put the explanation of Bayes factor in the appendix, i.e., it is impossible to understand this paper without knowing Bayes factor.
13: Fig 6: I think it is better and kinder to use U_1, U_5 in the legend of the plot instead of f. What is the model entropy?
14. Table 1: why there is "-" in CIFAR-10, it is better to clarify it in the paper (or maybe I missed it). I am not sure how to interpret the result, if the higher the better, does that mean the Loss is great?, I'm confused with the experimental results.
15. Before the beginning of 6.2: Tables 7 and 6 are in the appendix and we should state clearly it is in the appendix.
========================================================
Comments:
This paper lacks of clarity and difficult to understand. Although it is claimed to be better than existing measure, I am not convinced about that despite many experiments were conducted unfortunately.
For the criticism of using the maximum confidence of the softmax score from the softmax-cross entropy loss may not quantify the uncertainty, it is known theoretically that the score of softmax-cross entropy corresponds to p(y|x) if our prediction function achieve the global minimizer this loss function and our function class to be considered is all measurable functions (Zhang, JMLR2004: Statistical Analysis of Some Multi-Category Large Margin Classification Method). For other losses, see Williamson+ JMLR2016: Composite Multiclass Losses. However, it may not be accurate empirically when we use a deep network as it is reported in Guo+, 2017. Thus, one direction is to do post-processing or finding a way to modify a network. For U_1(k), I feel that it should suffer from the same problem as using maximum confidence score of softmax. Extending to top-k may have a good point when discussing about uncertainty and I believe it is good to explore that direction. For experiments, I would like to know how many trials did the authors run the experiment? and it would be helpful to see the standard deviation of the reported value. I believe this paper can still be improved a lot. For these reasons, I vote to reject this paper this time.
========================================================
Minor comments:
there exists the writing convention of "Top 5", "top 5", "top5". It's better to pick one way to describe it if there is no reason to make it different.
========================================================
After the rebuttal:
I have read the rebuttal.
I appreciate the authors' effort to modify the paper.
Also, please let me state that I totally agree that the problem the authors try to solve is indeed important and relevant for using machine learning in the real-world.
I feel that the structure of the modified version is better than the first version. It is a nice to include the explanation of the Bayes factor in the main body. Appendix C is also very useful for everyone to understand the Bayes factor.
As the author requested, I have read through the whole revised paper (including the appendix) carefully . I am aware of the positive sides of this paper. For example, it is interesting that we can find wrongly labeled data. Utilizing the uncertainty information for several applications. However, I found that the paper still requires a lot of modifications. The authors have modified many of my concerns, but still several of them were not addressed. I also emphasized the comments for parts that are unrelated to clarity (please see below). For these reasons, I decide not to change my score.
Below are my comments after reading the rebuttal (which some of them may be overlapped with the issues that were not addressed in the revised version).
============================
General comments:
1. Although the work is about tackling the empirical confidence estimation problem, Theorem 3.4 and Eq. (5) are providing insights about the population, not finite data points for empirical estimation. If we focus on the population case, it is known that the minimizer of the softmax cross-entropy loss must be a conditional probability $p(y|x)$. Thus, it is natural that as we minimize such a loss, the probability of correct classification must be high, since we can pick the best choice for classification, i.e., argmax of $p(y|x)$. But the most challenging part of the research in this direction is that, although the theory suggests we can get nice confidence information (in population), when we use the deep neural networks, the quality of confidence estimation can be very bad (overconfidence in empirical estimation) compared with the high accuracy we can achieve. As a result, a finer theory that can quantify the quality of confidence estimate for the finite sample case is highly needed, but I think Theorem 3.4 fails to capture this. I am aware that this theorem only concerns the KL loss. I believe that even only for KL-loss, the result can be significant if we discuss a finer theory for empirical estimation.
2. Regarding the Remark 3.5 (Neural networks are always overconfident), in my understanding, the result has no relationship with neural networks at all since it is true regardless of the hypothesis class of interest (e.g., linear models, kernel models, deep networks). This is because it is simply the definition of a loss function (and the result in the paper focus on KL loss, but I believe we can derive for many other losses). We know that the quality of confidence estimation of simple models can be better although the accuracy is worse. Thus, if the objective is to visualize the problem of neural networks, Remark 3.5 does not seem to help and adding neural networks in the remark title can be misleading. Thus, the implication of Remark 3.5 is insufficient to state that Neural networks are always overconfident.
3. What is the advantage of an implied loss? It seems the paper has two separate stories, the first one is implied loss (Sec. 3) and then move on to Bayes factor (Sec.4). Then, there is an adversarial detection problem using the gradient norm in the last experiment. From Sec. 4, the discussion about implied loss is very limited and if I understand correctly, the $-\log p_\max$ and $-log\sum p_{1:5}$ (the latter seems to require a superscript $sort$) are the implied loss, which does not seem to have the clear advantages over other methods. My impression is that the contributions of this paper are unclear and I do not know what is the main point of this paper. While the abstract dedicates most space to highlight the implied loss (nothing about Bayes factor), the conclusion dedicates most space to highlight the Bayes factor. Improving the connection between two parts may help to signify the contributions. Despite all that, I do like the idea of introducing the Bayes factor in this paper.
============================
Clarity:
1. Most importantly, I think the clear and solid definition of implied loss is missing.
According to Definition 3.1, "The uncertainty measure $U_k(x)$ is an implied loss if the event $S^\epsilon_k$ has high expected loss". I believe the implied loss is one of the most important contributions of this paper. I am not sure what does it means by "if the event $S^\epsilon_k$ has high expected loss" How can we define "high expected loss"?. I tried to read the paper many times to understand what exactly is the implied loss, and what is the scope of implied loss (what is and what is not an implied loss).
2. Following my first issue on clarity, what is the definition of uncertainty measure in this paper? According to the paper, it is defined roughly as $U_k$, whose statistics allow us to better estimate $p_k$. And in the abstract, it is emphasized that if uncertainty measure is an implied loss, then low uncertainty means a high probability of correct classification. Is there any uncertainty measure that is not implied loss? Also in the introduction, it seems that both softmax and the model entropy are uncertainty measures, are they implied losses? Is MC-dropout an uncertainty measure or even an implied loss in this context?
3. Eq. (11) left side: I think it is useful to the expectation with respect to which variable, I believe it is $B$. As a result, is it a typo to have $Y_i$ instead of $B_i$ on the left-hand side of (11)?
4. Example 3.3: If we set $k=1$ according to the definition of the implied loss at the last sentence of this example. Will it contradict $U_1(x)$ defined just right before that sentence? Because $y_w$ will become the $2$-nd ranked label, which I feel it is different from $U_1(x)$ defined in Example 3.3
5. I think it is better to clearly state that the figures/tables are in the appendix when referring to them from the main body. I saw the authors refer to Table 5 in Sec. 4.1, Figure 5 in Sec. 4.2 and Tables 6 and 7 in Sec. 4.3. All of them are in the appendix. And it seems some of them are highly needed. For example, the authors said that "by fine graining the bins we can capture relatively small ... on the order of 20" (before Sec. 4.3), then suggested the reader to see Figure 5. I feel Figure 5 must be included in the main body because it is hard to understand that without seeing the figure.
6. I think all figures that have $f_{(1)}$ (Figures 1a, 1b, and 5) must have a superscript $sort$ for all of them. Otherwise, it is wrong.
7. Kullback-Leibler loss is used extensively here without definition. It is important to clarify the clear definition of it. $U_1(x)$ also used extensively for the KL loss case and non-KL loss cases. This can make the paper hard to read. I suggest using $U^{\mathrm{KL}}_1(x)$ when referring to the uncertainty measure with respect to the KL-loss.
8. I saw $-\log(p_\max), -\log(f_{(1)}), -\log(f_1^{sort}), U_1$. Are these all refer to the same thing? And I found that sometimes the argument $(x)$ is ignored in the paper sometimes it doesn't in a quite random way. If they are the same, it would be nice to unify them.
9. The caption of figure 2(b) is uninformative. The authors may consider improving it.
10. It would be very helpful to add the implication or interpretation of the theoretical results to help the readers understand the intuition of the proven results. For example, how does Eq. (3) implies that when small uncertainty implies high chance of correct classification.
============================
Minor typos:
INTRO: "uncertainly measures" -> "uncertainty measures"
5.2: "on-distribution" -> "in-distribution"
Conclusion: logpmax -> write clearly with $$ should be better
Minor comments in the appendix:
1. How does Appendix A related to implied loss or Bayes factor in this paper? Did I miss something?
2. Figure 5:
	2.1 y-axis: is it Bayes ratio or Bayes factor? It seems the Bayes ratio and Bayes factor to be a different thing. Even if it is the same in some literature, I think it's better to use the Bayes factor here for the consistency of this paper.
	2.2 Caption: "entropy" -> "model entropy".
	2.3 Caption: Is it mistakes or do you want to insist on using $U_1$ and $U_5$ in the caption but using different notions in the figure? (in figure, they are $-\log(f_{(1)})$ and $-\log(\sum f_{(1:5)})$).
3. Appendix C: Eq. 16 and Eq. 19: I believe there is a typo. I think it should be $BF(X|Y_1), BF(X|Y_2), BF(X|Y_3)$, respectively.
4. Appendix C: Eq. 18-right: does it need to sum to 1 not 0.3+0.5+0.3 = 1.1?


Variational autoencoders (VAEs) defined over SMILES string and graph-based representations of molecules promise to improve the optimization of molecular properties, thereby revolutionizing the pharmaceuticals and materials industries. However, these VAEs are hindered by the non-unique nature of SMILES strings and the computational cost of graph convolutions. To efficiently pass messages along all paths through the molecular graph, we encode multiple SMILES strings of a single molecule using a set of stacked recurrent neural networks, harmonizing hidden representations of each atom between SMILES representations, and use attentional pooling to build a final fixed-length latent representation. By then decoding to a disjoint set of SMILES strings of the molecule, our All SMILES VAE learns an almost bijective mapping between molecules and latent representations near the high-probability-mass subspace of the prior. Our SMILES-derived but moleculebased latent representations significantly surpass the state-of-the-art in a variety of fully-and semi-supervised property regression and molecular property optimization tasks.

This paper presents a VAE-based method to predict molecular properties as well as to design a novel molecular architecture.
The method employs as a basic input the SMILES representation of molecules which are not well defined in terms of the representation, though.
To tackle the issue, the authors formulate the method based on (1) multiple inputs of SMILES strings, (2) the character-wise feature fusion across those multiple strings and (3) network training by multiple output targets of SMILES strings different from the input ones.
As a result, the method provides the fixed-length latent representation which is robust against the variations of the SMILES representation and is useful for predicting the molecular properties and optimizing the molecular design.
The experimental results on the tasks of the property prediction and molecular optimization using the benchmark datasets demonstrate the effectiveness of the proposed method in comparison with the others.
Though this paper provides some contributions to the field of molecular graph representation, it contains flaws in presentation, lacking details of technical contents; thus, the paper is regarded as "borderline". I would like the authors to properly provide the details in the following points.
* This paper lacks some important technical contents, making it hard to understand.
- What is the actual form of the property predictor used in p(\rho|z). And, in the first place of the paper, it would be better to explain what kind of and how many properties of the molecular are considered.
- How is the decoder constructed and trained? Though the authors state that a single-layer LSTM is employed as the decoder, there is no clear description how to cope with the multiple outputs (decoder targets) in training the LSTM.
- What does \theta in p_\theta(z) mean? In the VAE, p(z) works as a simple prior on z.
- The description, especially in Sec. 3 for the proposed method, is rather poorly presented by using less amount of math. At least, the authors should first depict the overall architecture of the method through clarifying the above-mentioned technical points.
- It is unclear how to apply the proposed method to the semi-supervised learning framework? Is it re-formulated in the SS-VAE [a] framework?
- The comparison experiments seem to be inconsistent. The proposed method is compared with different methods in different datasets/tasks. Toward fair comparison, it should be evaluated in comparison with some baselines including such as JT-VAE consistently.
[a] Kingma, D. P.; Mohamed, S.; Rezende, D. J.; Welling, M. Semi-Supervised Learning with Deep Generative Models. Proceedings of the 28th Annual Conference on Neural Information Processing Systems. 2014; pp 3581–3589.
* The authors cope with the fluctuations regarding the SMILES representation by means of RNN. There, however, are some approaches to canonicalize the SMILES representation itself such as by canonical SMILES. The authors have to mention those other approaches and discuss the superiority of the method over them; hopefully the proposed method should be compared with the naive one simply combining VAE [Kang&Cho18] and canonical SMILES representation. And to further understand the difficulty in SMILES variations, it would be better to show the averaged cardinality of SMILES representation per molecular.
Minor comment:
* In aggregating the RNN features for each character in SMILES strings, it might be possible to incorporate some structural knowledge into the process; for example, "C"s in benzene share the identical feature representation through fusing all those features.


We propose a novel training approach for improving the learning of generalizing features in neural networks. We augment the network with a reverse pass which aims for reconstructing the full sequence of internal states of the network. Despite being a surprisingly simple change, we demonstrate that this forward-backward training approach, i.e. racecar training, leads to significantly more general features to be extracted from a given data set. We demonstrate in our paper that a network obtained in this way is continually trained for the original task, it outperforms baseline models trained in a regular fashion. This improved performance is visible for a wide range of learning tasks from classification, to regression and stylization. In addition, networks trained with our approach exhibit improved performance for task transfers. We additionally analyze the mutual information of our networks to explain the improved generalizing capabilities.

This paper attempts to learn general and reusable features for transfer learning tasks. The authors propose a training paradigm called Racecar Training. The core idea of it is to operate a reverse pass for the network. The authors use mutual information to analyze the network for its improved generalizing capabilities. They also conduct experiments on classification, regression and stylization to validate their method’s effectiveness.
Pros:
The reverse pass idea is similar to auto-encoder paradigm that discards redundant information and only save the essential low dimensional one by comparing the original data and the decoding data. The general feature the authors mentioned is like the essential low dimensional information, which is reasonable.
Cons:
1.	I think the main drawbacks of this paper is that the authors make a poor presentation. The authors talk about learning general features with which the model can use on new task in the title, introduction and even the whole paper. However, there are no explicit general features learned during the learning procedure. They only perform the reverse pass when learning in the original task. Even the structure of networks does not change at all. The general and reusable feature is only an explanation of the improved performance. I think the authors should change this explanation to a more convincing one. For example, the network may learn general and reusable weights or parameters since it is the model learned that will be applied to new tasks instead of the features.
2.	The analysis by mutual information makes the paper hard to follow. The figures such as figure 2 are so confusing. There are so many points and lines in each picture. What do they mean? What are the x-axis and y-axis? The authors also do not explain what the meaning of different mutual information are.
3.	The symbols are chaotic. The authors explain “RR^3” means n=1 in equation 1. Then what does n equal to in RR^1? The authors explain “AB” means the model was trained for task A during phase I, and is then trained for task B as transfer in phase II. Then what does AA/AB in “Std_{AA/AB}” mean?
4.	In the last paragraph of page 3, the authors mention orange color. However, there are only green, blue and yellow color in Figure 1.
5.	In experiments, I do not see obvious advantage of the proposed method. For example, in Figure 6, the test accuracy of Std_{AA} is 0.9842 while that of RR^3_{AA} is only 0.9859. In Figure 7, the test accuracy of Std_{AA} is 0.8377 while that of RR^13_{AA} is 0.8711. The transfer tasks (referred as AB) also show minor advantages. With minor advantages but increased requirements for memory and additional computations (e.g., 61.13% slower per epoch for the MNIST tests mentioned by the authors), the proposed method shows very limited values.


Recently, many stochastic gradient descent algorithms with variance reduction have been proposed. Moreover, their proximal variants such as Prox-SVRG can effectively solve non-smooth problems, which makes that they are widely applied in many machine learning problems. However, the introduction of proximal operator will result in the error of the optimal value. In order to address this issue, we introduce the idea of extragradient and propose a novel accelerated variance reduced stochastic extragradient descent (AVR-SExtraGD) algorithm, which inherits the advantages of Prox-SVRG and momentum acceleration techniques. Moreover, our theoretical analysis shows that AVR-SExtraGD enjoys the best-known convergence rates and oracle complexities of stochastic first-order algorithms such as Katyusha for both strongly convex and non-strongly convex problems. Finally, our experimental results show that for ERM problems and robust face recognition via sparse representation, our AVR-SExtraGD can yield better performance than state-of-the-art algorithms such as Prox-SVRG and Katyusha. The asynchronous variant of AVR-SExtraGD outperforms KroMagnon and ASAGA, which are the asynchronous variants of SVRG and SAGA, respectively.

This paper proposed a new stochastic algorithm: AVR-ExtraGD. AVR-ExtraGD combines the extended extragradient method proposed in [3] and the accelerated SVRG method in [1][2]. In their experiments, AVR-ExtraGD outperforms [2] in running time for sparse linear regression.
This paper presents their convergence analysis using results in [1][2]. They showed that the proposed algorithm can achieve O(sqrt{kappa n} log (1 / \epsilon)) complexity for strongly convex problem and O(1/sqrt{epsilon}) for convex problem, which are the best results for both cases.
The idea of an accelerated version of variance reduced stochastic extragradient method is novel. However, there are some issues that the authors should address in order for the paper to match the quality of ICLR.
Each step of extragradient approximates the proximal operator x_{k+1} = argmin_x P(x) + 1/(2 eta_k)\|x – x_k\|_2^2, therefore we would expect a faster and more stable convergence from this method. This paper claims that extragradient reduces the gap between the obtained optimal value and the real optimal value, which is confusing. The update of extragradient is actually biased towards x_k. The claim is then discussed in section 3.1 and 3.2 but is not clearly explained. Besides, how this claim is reflected in the convergence result is not discussed. I encourage the authors to clearly elaborate this claim and make relevant remarks after the main theorems.
To better understand the convergence result, it is important to know how extragradient affects the complexity and choice of hyperparameters such as K, eta_1, eta_2, and beta. Such discussion is not in this paper. I suggest the authors to make these aspects clear.
The experiments compare the proposed algorithm with other algorithms by their running time for lasso and elastic-net. The comparisons show the efficiency of the proposed algorithm. However, a more careful experimental design is required to better demonstrate the performance:
1.	For the choice of inner iterations, choosing m=2n for Katyusha actually requires calculating 5n stochastic gradient because each iteration of Katyusha does gradient updates twice.
2.	This paper only presents comparisons of running time. I encourage the authors to also plots comparisons on number of iterations, which will help revealing where the speed up of AVR-ExtraG comes from.
3.	It is also preferable that the author compare with MiG [1], since the proposed algorithm is an extragradient version of [1].
4.	Please at least solve two different optimization programs (e.g. logistic regression, neural network) so any conclusions are not specific to the oddities of a particular program.
The presentation and structure of this paper need to be improved. Here are some suggestions:
1.	 In Section 1, only provide a high-level literature review and then motivate the work. A comprehensive review can come after the introduction.
2.	In Section 4, put all the lemmas into the appendix while giving more intuitions and remarks.
3.	Issues including notions without pre-definition or reference, typos, and incorrect gramma need to be fixed.
Detailed comments:
1.	From the title, the main application of this work is sparse learning problem. However, how the proposed algorithm benefits sparsity is not discussed. Besides, I suggest the authors to move the asynchronous algorithm in the appendix to the main paper.
2.	The paragraph before section 1.1: lasso and elastic-net are used without citation.
3.	Section 1.1: PGD and SGD are used without citation.
4.	Beginning of page 2: “And” should be “Besides”
5.	“Besides, for accelerating the algorithm and …”: “for accelerating” should be “to accelerate”
6.	Section 1.2: “Nguyen et al. (2017) proposed the idea of extragradient which can be seen as a guide during the process, and introduced it into the optimization problems.” What does “the process” and “it” refers to is unclear.
7.	Section 1.2: the claim extragradient examines the geometry and curvature of the problem is confusing. The geometry of the problem is inspected through a line search step in [3]. However, line search is not discussed in this paper.
8.	Section 1.2: “reduce the gap between the optimal value we get and the real optimal value”, these two kinds optimal values are important notions of this paper but they are not defined.
9.	In Assumption 2, you can refer to Part 2, Section 7 of [5] for the definition of semi-continuity.
10.	“dw is the gradient of the function at w”, what does “the function” refers to?
11.	“APG and Acc-Prox-SVRG” needs citation.
12.	“was proposed to simply the structure of Katyusha”, “simply” should be “simplify”
13.	Section 3.1: “updated with the update rules of MiG”: “with” should be “by”
14.	In the equations of Section 3.2, the equivalent of gradient norm square and function f is incorrect, and the purpose of this equation is unclear.
15.	Section 4.1 Theorem 1: The inequality in theorem 1 is not intuitively related to the convergence rate. I suggest the author to simplify the inequality (For example, Theorem 2.1 in [2]).
16.	The references are not in a uniform format. Conference/Journal names are missing for some references.
17.	One useful reference for this paper is [4], it discussed extragradient for online convex learning.
Additional question:
[5] update the extragradient step by sampling a new stochastic gradient while in this paper the same sample is used twice. How you compare these two approaches in terms of their performance and convergence?
[1] A simple stochastic variance reduced algorithm with fast convergence rates, Zhou et al., 2018.
[2] Katyusha: the first direct acceleration of stochastic gradient methods, Z. Allen-Zhu, 2017
[3] Extragradient method in optimization: Convergence and complexity, T. Nguyen et al., 2017
[4] Online Optimization with Gradual Variations, Chiang et al., 2012
[5] Convex Analysis, R. Rockafella, 1970
[6] Reducing Noise in GAN Training with Variance Reduced Extragradient, Chavdarova et al.  2019