Extracting saliency maps, which indicate parts of the image important to classification, requires many tricks to achieve satisfactory performance when using classifier-dependent methods. Instead, we propose classifier-agnostic saliency map extraction, which finds all parts of the image that any classifier could use, not just one given in advance. We observe that the proposed approach extracts higher quality saliency maps and outperforms existing weakly-supervised localization techniques, setting the new state of the art result on the ImageNet dataset.

This paper focuses on the extraction of high-quality model-agnostic saliency maps. The authors argue that when an extracted saliency map is directly dependent on a model, then it might not be useful for a different classifier and thus not general enough. To overcome this problem, they consider all the possible classifiers weighted by their posterior probabilities. This problem cannot be solved explicitly, and the authors suggest a scheme to approximate the solution using two networks. That is, pretrain an initial classifier and then, following an adversarial training procedure, one network is trying to confuse the classifier and the other one to maximize its accuracy. Using this formulation, the authors report state-of-the-art results for salience map extraction.
SUMMARY/OVERALL COMMENTS
The authors present a simple and effective way to produce classifier-agnostic saliency maps. The argument for the approach is well justified and the results seem convincing on a first read. However, the novelty of the method is a concern given the previous work of Fan et al. (2017), and the manuscript is not upfront about the differences between the two works. The experiments are another cause for concern: Fan et al. should have been tested as a baseline with similar implementation (controlling for architecture and \lambda), and implementation differences in prior works of Table 1 make it difficult to draw conclusions.
RELATED WORKS
* In the introduction, the authors mention related works but fail to mention the work of Fan et al. (2017) which is clearly the most relevant. The first mention of Fan et al is on page 4 in a very specific discussion the regularization coefficient. The problem formulation in Section 2 and the approach is Section 3 is largely borrowed from Fan et al but not acknowledged until the last page. This introduces bias and confusion to the reader in regards to the novelty of the approach. Please, mention the work of Fan et al. (2017) in the introduction and clearly delineate the differences in the works earlier in the text. (--)
* Du et al. (2018), “Towards Explanation of DNN-based Prediction with Guided Feature Inversion”, use the VGG models for saliency map extraction and achieve a LE of 38.2. Note that Du et al. (2018), suggest that this modification could lead to SOTA results. I would like to see a comparison with this method. (-)
* The work of Kindermans, et al. (2017), “Learning how to explain neural networks: PatternNet and Pattern Attribution”, although they do not aim for weakly supervised localization and thus, they do not present the LE, they produce saliency maps. I would like to see a LE comparison with that method. (minor -)
* In the introduction, p1 (last paragraph) other methods are briefly mentioned (Extracted saliency maps show all the evidence….superpixels), etc.) without references. Please add references when needed. (-)
* The framework presented in this paper was first proposed by Fan et al. (2017). The authors claim four main differences in their approach. In my eyes, not all of them are major or novel - probably the most impactful is removing superpixels as it simplifies the problem and implementation. (+)
APPROACH
* The authors aim for simplicity (strong +)
* The authors justify their approach and present their arguments clearly (strong ++)
* In the algorithm section, the authors first mention the sampling procedure and then their motivation. Please alter the ordering of these to be conceptually easier to understand your approach
* In the first sentence after the equation 6 I guess that “(cf. Alg. 1)” is a typo and should be modified to (Alg. 1)”
* After Equation 6 it is argued that the method resembles the training procedure of GANs (Godfellow et al., 2014) but not the work of Fan et al., 2017. (--)
EXPERIMENTS
* The authors define as their baseline the F thinning strategy (i.e. use only the first classifier) which is a model dependent salience map. While this is a useful comparison against the classifier dependent methods, given the similarity to the work of Fan et al. (2017), experiments comparing the proposed model to Fan et al. are necessary. It is important to control for network architecture (ResNet-50) and choice of \lambda to properly determine if the four changes outlined in Section 6 result in any real improvement over Fan et al. (strong --)
* The authors use the Table 1 (borrowed from Fong and Vedaldi (2017)) to compare their results against other methods. This comparison is problematic as different approaches are using different models as classifiers which may lead to increase or decrease of the LE. (--)
* In Table 2 the authors do not report how many times they run the same experiments to get these values. They also run less experiments with non-shared weights and they report only the LE. In my eyes it looks that the authors are trying to force their argument that the sharing weights helps (probably because it is one of their novelties). Please report the statistics of your experiments and fill the empty entries in the table. (--)
* In Table 3, what does the last row represent?
* Table 1 errors: (1) You write “Localization evaluation using OM, LE and F1 scores”. Please remove the F1 score as you do not report it. Also, correct the first sentence of the “Localization” subsection which states that you use three different metrics to “two different metrics”. (2) The LE from Fong and Vedaldi (2017) should be 43.2 and not 43.1.
* Regarding the unseen classes (section 5): (1) Please report in the appendix the classes that you are using in each subset. Are there classes correlated? (-)  (2) I see that there is a strong correlation between the LE on subset A and E. It looks like you are training on E and you generalize on A.
NOVELTY/IMPACT
* Novelty is a strong concern, given the work of Fan et al. (2017) (strong --). Nevertheless, the authors propose some changes that can be seen as more general, but the effectiveness of the changes is clearly established.
* This paper’s strongest point is the simplicity (conceptually and implementation-wise) of the method, an advantage over previous works (+)
OTHER COMMENTS
* Fan et al. (2017), use an adaptive λ that pushes the mask to 10% of the image whereas you are using a fixed one that pushes the mask to approximately 50% of the image. How can you make sure that this is not the reason that you are getting better results?
If the authors can clearly and fairly demonstrate that the changes they propose over Fan et al (2017) result in improved performance, and the manuscript is adjusted to be more upfront about this prior work, I would consider increasing my rating.


In this paper, a new interactive parallel learning scheme is proposed to enhance the performance of off-policy continuous-action reinforcement learning. In the proposed interactive parallel learning scheme, multiple identical learners with their own value-functions and policies share a common experience replay buffer, and search a good policy in collaboration with the guidance of the best policy information. The information of the best policy is fused in a soft manner by constructing an augmented loss function for policy update to enlarge the overall search space by the multiple learners. The guidance by the previous best policy and the enlarged search space by the proposed interactive parallel learning scheme enable faster and better policy search in the policy parameter space. Working algorithms are constructed by applying the proposed interactive parallel learning scheme to several off-policy reinforcement learning algorithms such as the twin delayed deep deterministic (TD3) policy gradient algorithm and the soft actor-critic (SAC) algorithm, and numerical results show that the constructed IPE-enhanced algorithms outperform most of the current state-of-the-art reinforcement learning algorithms for continuous action control.

This paper describes a new architecture for parallelizing off-policy reinforcement learning systems with a pool of independent learners trained on identical, but independent instances of the environment with a scheme for periodically synchronizing the the policy knowledge across the pool. The paper provides demonstrations in several continuous control domains.
I think this paper should be rejected because: 1) the approach is not well justified or placed within the large literature on parallel training architectures and population-based training methods, (2) the results are competitive with the best in each domain, but there are many missing details. Since the contribution is entirely support by empirical evidence, these issues need to be clarified. I look forward to the author response, as I will pose several questions below and my final score will carefully take the answers into account.
Justification of decision. There are numerous papers on parallel architectures for training deep RL systems [1,2, 6] and you cited a few, and while not all of them focus on continuous control there are design decisions and insights in those works must be relevant to your efforts. You should make those connections clear in the paper. One line of the paper is not nearly enough. The stated focus of the paper is exploration-exploitation yet there is little to no discussion of other ideas including noisy networks, intrinsic motivation, or count-based exploration methods. The paper is missing a lot of key connections to the literature.
I am certainly not a fan of architectures that assume access to many instances of the environment. In this case that assumption seems worse because of the target application: continuous control domains. These domains are simulations of physical control systems; on a robot the agent receives only one stream of experience and thus these architectures would not work well. Though there is some work on applying these multi-environment architectures to farms of robot arms; the reality of the real-world is that the arms end up being very different due to wear and tear, and engineers must constantly fix the hardware because these multi-environment architectures do not work when the environments are different. We cannot loose sight of the goal here—maximizing these simulation environments is not of interest itself, its a stepping stone—architectures that only work on simulations that afford multiple identical environments but fail in the real world have very limited application. I think this paper needs to motivate why parallel training in this way in these robotics inspired domains is interesting and worthwhile.
The main area of concern with this paper is the experiment section. There are several issues/questions I would like the authors to address:
1) You built on top of TD3, however you just used the parameter settings of TD3 as published and didn’t tune them. This is a problem because it could just be that the existing parameter choices for TD3 were just better for the new approach. You have to take additional effort in this case to ensure your method is actually better than just using TD3. Additional parameter tuning of TD3 is required here.
2) I think its an odd choice for TD3 to have an infinite buffer, as recent work has show at least for DQN that large buffers can hurt performance [7].  Can you justify this choice beyond “the authors of TD3 did it that way”?
3) Why is R_eval different for each method?
4) Why did you not compare to TD3 on the same set of domains as used in the TD3 paper? Why a subset? Why these particular domains?
5) In 2 of the 4 domains the proposed method ties or is worse than the baselines. In half-cheetah it looks close to significant, and in the ant domain the result is unlikely to be significant because the error-bars overlap and the error-bars of TD3 are wider than the other methods so a simple visual inspection is not enough. There does not seem to be a strong case for the new method here. I may be misunderstanding the results. Help me see the significance.
6) The paper claims improvement in variance, but this requires additional analysis in the form of an F-test of better.
7) Why these baselines (e.g., SAC) and not others? Why did you not include D4PG [6]? Soft Q-learning? A population-based training method [3,4,5,8] to name a few?
[1] GPU-Accelerated Robotic Simulation for Distributed Reinforcement Learning
[2] IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures
[3] Human-level performance in first-person multiplayer games with population-based deep reinforcement learning
[4] Improving Exploration in Evolution Strategies for Deep Reinforcement Learning via a Population of Novelty-Seeking Agents
[5] Structured Evolution with Compact Architectures for Scalable Policy Optimization
[6] Distributed Distributional Deterministic Policy Gradients
[7] A Deeper Look at Experience Replay
[8] Evolution Strategies as a Scalable Alternative to Reinforcement Learning

Small things that did not impact the score:
1) references to “search interval” in the abstract are confusing because the reader has not read the paper yet
2) Description of the method in abstract is too specific
3) P1 intro, not a topic sentence for what follows
4) “performs an action to its environment” >> grammar
5) “One way to parallel learning…” >> grammar
6) “that the value parameter and” >> grammar
7) “pitfalls” >> minima
8) Did you try combining you method with other base off-policy methods? how did it work?
9) GAE undefined?
10) “among the baseline and”>>grammar…there are many grammar errors


LARGE-SCALE ANSWERER IN QUESTIONER'S MIND FOR VISUAL DIALOG QUESTION GENERATION
Answerer in Questioner's Mind (AQM) is an information-theoretic framework that has been recently proposed for task-oriented dialog systems. AQM benefits from asking a question that would maximize the information gain when it is asked. However, due to its intrinsic nature of explicitly calculating the information gain, AQM has a limitation when the solution space is very large. To address this, we propose AQM+ that can deal with a large-scale problem and ask a question that is more coherent to the current context of the dialog. We evaluate our method on GuessWhich, a challenging task-oriented visual dialog problem, where the number of candidate classes is near 10K. Our experimental results and ablation studies show that AQM+ outperforms the state-of-the-art models by a remarkable margin with a reasonable approximation. In particular, the proposed AQM+ reduces more than 60% of error as the dialog proceeds, while the comparative algorithms diminish the error by less than 6%. Based on our results, we argue that AQM+ is a general task-oriented dialog algorithm that can be applied for non-yes-or-no responses.

=================
Updated Thoughts
=================
I was primarily concerned about a lack of analysis regarding the technical contributions moving from AQM to AQM+. The revisions and author comments here have addressed the specific experiments I've asked for and more generally clarified the contributions made as part of AQM+. I've increased my rating to reflect my increased confidence in this paper. Overall, I think this is a good paper and will be interesting to the community.
I also thank the authors for their substantial efforts to revise the paper and address these concerns.
===========
Strengths:
===========
The approach is a sensible application of AQM to the GuessWhich setting and results in significant improvements over existing approaches both in terms of quantitative results and qualitative examples.
===========
Concerns:
===========
[A] Technical Novelty is Limited Compared to AQM
The major departures from the AQM approach claimed in the paper (Section 3.3) are:
	[1] the generation of candidate questions through beam search rather than predefined set
	[2.1] The approximate answerer being an RNN generating free-form language instead of a binary classifier.
	[2.2] Dropping the assumption that \tilde p(a_t | c, q_t) = \tilde p (a_t | c, q_t, h_{t-1}).
	[3] Estimate approximate information gain using subsets of the class and answer space corresponding to the beam-search generated question set and their corresponding answers.
I have some concerns about these:
For [1], the original AQM paper explores this exact setting for GuessWhat in Section 5.2 -- generating the top-100 questions from a pretrained RNN question generator via beam search and ranking them based on information gain. From my understand, this aspect of the approach is not novel.
For [2.1] I disagree that this is a departure from the AQM approach, instead simply an artifact of the experimental setting. The original AQM paper was based in the GuessWhat game in which the answerer could only reply with yes/no/na; however, the method itself is agnostic of this choice. In fact, the detailed algorithm explanation in Appendix A of the AQM paper explicitly discusses the possibility of the answer generator being an RNN model.
Generally, the modifications to AQM largely seem like necessary, straight-forward adjustments to the problem setting of GuessWhich and not algorithmic advances. That said, the changes make sense and do adapt the method to this more complex setting where it performs quite well!
[B] Design decisions are not well justified experimentally
Given that the proposed changes seem rather minor, it would be good to see strong analysis of their effect. Looking back at the claimed difference from AQM, there appear to be a few ablations missing:
- How useful is generating questions? I would have liked to see a comparison to a Q_fix set samples from training. (This corresponds to difference [1] above.)
- How important is dialog history to the aprxAns model? (This corresponds to difference [2.2] above).
- How important is the choice to restrict to |C| classes? Figure 4b begins to study this question but conflates the experiment by simultaneously increasing |Q| and |A|. (This correspond to difference [3] above.)
[C] No evaluation of Visual Dialog metrics
It would be useful to the community to see if this marked improvement in GuessWhich performance also results in improved ability to predict human response to novel dialogs. I (and I imagine many others) would like to see evaluation on the standard Visual Dialog test metrics. If this introspective inference process improves these metrics, it would significantly strengthen the paper!
[D] No discussion of inference time
It would be useful to include discussion of relative inference time. The AQM framework requires substantially more computation than an non-introspective model. Could authors report this relative increase in inference efficiency (say at K=20)?
[E] Lack of Comparison to Base AQM
I would expect explicit comparison to AQM for a model named AQM+ or a discussion on why this is not possible.
===========
Minor Things:
===========
- I don't understand the 2nd claimed contribution from the introduction "At every turn, AQM+ generates a question considering the context of the previous dialog, which is desirable in practice." Is this claim because the aprxAns module uses history?
- Review versions of papers often lack polished writing. I encourage the authors to review their manuscript for future versions with an eye for clarity of terminology, even if it means a departure from established notation in prior work.
- The RL-QA qualitative results, are these from non-delta or delta? Is there a difference between the two in terms of interpretability?
===========
Overview:
===========
The modifications made to adapt AQM to the GuessWhich setting presented here as AQM+ seem to be somewhat minor technical contributions. Further, where these difference could be explored in greater detail, there is a lack of analysis. That said, the proposed approach does make significant qualitative and quantitative improvements in the target problem. I'm fairly on the fence for this paper and look forward to seeing additional analysis and the opinions of other reviewers.


Recently there has been a surge of interest in designing graph embedding methods. Few, if any, can scale to a large-sized graph with millions of nodes due to both computational complexity and memory requirements. In this paper, we relax this limitation by introducing the MultI-Level Embedding (MILE) framework -a generic methodology allowing contemporary graph embedding methods to scale to large graphs. MILE repeatedly coarsens the graph into smaller ones using a hybrid matching technique to maintain the backbone structure of the graph. It then applies existing embedding methods on the coarsest graph and refines the embeddings to the original graph through a graph convolution neural network that it learns. The proposed MILE framework is agnostic to the underlying graph embedding techniques and can be applied to many existing graph embedding methods without modifying them. We employ our framework on several popular graph embedding techniques and conduct embedding for real-world graphs. Experimental results on five large-scale datasets demonstrate that MILE significantly boosts the speed (order of magnitude) of graph embedding while generating embeddings of better quality, for the task of node classification. MILE can comfortably scale to a graph with 9 million nodes and 40 million edges, on which existing methods run out of memory or take too long to compute on a modern workstation.

In this submission, the authors propose a three-stage framework for large-scale graph embedding. The proposed method first constructs a small graph by graph coarsening, then applies any existing graph embedding method, and last refines the learned embeddings. It is useful, however, the experimental results are not convincing and cannot support the authors' claims about the proposed method.
First, in many places, the authors claim that the embedding quality of the proposed method is improved. For example, the last sentence of Section 1, and "MILE improves quality" paragraph on Page 7. However, the experimental results fail to support this. As the proposed method is for the large-scale graph, let's focus on the results of YouTube dataset and Yelp dataset first. For Youtube dataset ((d) of Table 2), when m is set to be 8, for all the cases, the performance drops. For Yelp dataset (Figure 3), the authors do not provide Micro-f1 for the original graph (m = 0) or m = 1, 2, so it is hard or impossible to demonstrate that the quality of the proposed method is still good.
Second, the comparison with existing methods is not sufficient. For the most important Yelp dataset (as this dataset fits the motivation scenario (large-scale graph) of this submission), the authors fail to report any comparison. Thus it might not be weak to demonstrate the benefit of the proposed method.
Third, some experiment details are missing. For example, how the authors compute the running time of the proposed method? All the three stages are included? How the authors implement the existing methods? Are these implementations good enough to ensure a fair comparison?
*******
Some other questions:
a) On page 2, the authors mention that the proposed method "can be easily extended to directed graph". However, based on my understanding, directly graph will affect both the graph coarsening and embedding refining steps, and it seems not so easy to extend. Do the authors have the solution and experiments for directed graph? It would be interesting to see such results, which enlarges the application scope of the proposed method.
b) The toy example on page 3 is very clear. However, for real-world graphs, does the proposed graph coarsening work well? For example, one property the proposed method utilizes is "structurally equivalent". What is the percentage of the nodes that can have such property for real-world graphs?
********
Some other comments:
Generally speaking, this submission studies a very practical task. Although the authors claim that the proposed method has great efficiency while the embedding quality is comparable good or even better than the existing methods, I think that there is an efficiency-quality trade-off based on the experimental results in this submission. When m increases, the graph coarsening step causes more information loss, and thus the quality may decrease. Embedding refining step can be regarded as a procedure to reduce such information loss, but may not improve the embedding quality better than the original graph. So to me, it would be more meaningful to study such efficiency-quality trade-off for large-scale graph embedding.


While extremely successful in several applications, especially with low-level representations; sparse, noisy samples and structured domains (with multiple objects and interactions) are some of the open challenges in most deep models. Column Networks, a deep architecture, can succinctly capture such domain structure and interactions, but may still be prone to sub-optimal learning from sparse and noisy samples. Inspired by the success of human-advice guided learning in AI, especially in data-scarce domains, we propose Knowledge-augmented Column Networks that leverage human advice/knowledge for better learning with noisy/sparse samples. Our experiments demonstrate how our approach leads to either superior overall performance or faster convergence.

This paper formulates a new method called human-guided column networks to handle sparse and noisy samples. Their main idea is to introduce human knowledge to guide the previous column network for robust training.
Pros:
1. The authors find a fresh direction for learning with noisy samples. The human advice can be viewed as previledged information.
2. The authors perform numerical experiments to demonstrate the efficacy of their framework. And their experimental result support their previous claims.
Cons:
We have three questions in the following.
1. Motivation: The authors are encouraged to re-write their paper with more motivated storyline. The current version is okay but not very exciting for idea selling. For example, human guidance should be your selling point, and you may not restrict your general method into ColumnNet, which will limit the practical usage.
2. Related works: In deep learning with noisy labels, there are three main directions, including small-loss trick [1], estimating noise transition matrix [2,3], and explicit and implicit regularization [4]. I would appreciate if the authors can survey and compare more baselines in their paper instead of listing some basic ones.
3. Experiment:
3.1 Baselines: For noisy labels, the authors should add MentorNet [1] as a baseline https://github.com/google/mentornet From my own experience, this baseline is very strong.
3.2 Datasets: For datasets, I think the author should first compare their methods on symmetric and aysmmetric noisy data. Besides, the authors are encouraged to conduct 1 NLP dataset.
By the way, if your human guidance is totally wrong, how your model handle such extreme cases? Could you please discuss this important point in your paper?
References:
[1] L. Jiang, Z. Zhou, T. Leung, L. Li, and L. Fei-Fei. Mentornet: Learning data-driven curriculum for very deep neural networks on corrupted labels. In ICML, 2018.
[2] G. Patrini, A. Rozza, A. Menon, R. Nock, and L. Qu. Making deep neural networks robust to label noise: A loss correction approach. In CVPR, 2017.
[3] J. Goldberger and E. Ben-Reuven. Training deep neural-networks using a noise adaptation layer. In ICLR, 2017.
[4] T. Miyato, S. Maeda, M. Koyama, and S. Ishii. Virtual adversarial training: A regularization method for supervised and semi-supervised learning. ICLR, 2016.


In this work the authors show that Bayesian Neural Networks (BNNs) can be efficiently applied to calibrate state-of-the-art Deep Neural Networks (DNN). Our approach acts offline, i.e., it is decoupled from the training of the DNN to be calibrated. This offline approach allow us to apply our BNN calibration to any model regardless of the limitations that the model may present during training. Note that this offline setting is also appropriate in order to deal with privacy concerns of DNN training data or implementation, among others. We show that our approach clearly outperforms other simple maximum likelihood based solutions that have recently shown very good performance, as temperature scaling (Guo et al., 2017). As an example, we reduce the Expected Calibration Error (ECE%) from 0.52 to 0.24 on CIFAR-10 and from 4.28 to 2.46 on CIFAR-100 on two Wide ResNet with 96.13% and 80.39% accuracy respectively, which are among the best results published for these tasks. Moreover, we show that our approach improves the performance of online methods directly applied to the DNN, e.g. Gaussian processes or Bayesian Convolutional Neural Networks. Finally, this decoupled approach allows us to apply any further improvement to the BNN without considering the computational restrictions imposed by the deep model. In this sense, this offline setting is a practical application where BNNs can be considered, which is one of the main criticisms to these techniques. In terms of reproducibility, we provide all the implementation details in https://github.com/2019submission/bnn.2019.

-- Paper summary --
The primary goal of this paper is to investigate the suitability of BNNs for carrying out post-calibration on trained deep learning models. The results are compared to equivalent models calibrated using temperature scaling, and the proposed technique is shown to yield superior uncertainty calibration.
-- General Commentary --
The overall goal of this work is rather modest and the scope of the evaluation is limited. While not without challenges, carrying out offline calibration as a corrective measure is a simpler problem to tackle than developing well-calibrated models upfront, and limiting the comparison to just one other post-calibration method greatly narrows the overall vision such a paper should have. For instance, isn’t post-calibration more likely to result in overfitting than a model that is implicitly calibrated at training time?
I have plenty of concerns with the submission itself, listed below:
- First and foremost, the paper is full of typos and grammatical errors. I genuinely struggled to read the paper end-to-end without being continually distracted by these issues. While some mistakes may indeed be genuine, others are only there due to sheer negligence and because the authors didn’t properly check the paper before submission.
- While the overall objective of this work (i.e. improving calibration of deep models) is clearly established, the overall presentation of ideas is very muddled and I initially struggled to properly understand what’s being proposed.  A simple diagram or illustration would have clarified some of the notation at the very least.
- The sloppiness in the presentation is also manifested in other ways. For example, in Figure 1, the plots should be individually titled (‘uncalibrated’, ‘temp-scal' and ‘BNN') in order to immediately distinguish between them; instead, all this information is contained in the caption whereas it could just as easily have been added to the plot.
- As alluded to earlier, I am disappointed by the lack of scope in the paper. The experimental evaluation should have been widened to include direct comparisons against BNN models which one might expect to be slightly better-calibrated upfront. There has also been significant interest in improving the calibration of deep models by stacking different architectures in such a way that the model is implicitly calibrated at training time. Examples of such papers include ‘Adversarial Examples, Uncertainty, and Transfer Testing Robustness in Gaussian Process Hybrid Deep Networks’ (Bradshaw et al, 2017), and ‘Calibrating Deep Convolutional Gaussian Processes’ (Tran et al, 2018). The deep kernel learning schemes developed by Wilson et al. also discuss similar hybrid models.

- With reference to the papers cited above, one possible extension the authors could consider is to use a Gaussian process for post-calibration instead of a BNN, although I suspect this may have already been investigated in the past. In any case, this warrants further discussion.

- I can’t disentangle the two contributions listed at the bottom of Pg 2 and the top of Pg 3. There is no theoretical evaluation of the ‘alternative hypothesis’ being mentioned, and the investigation is entirely limited to the offline setting, so I’m not entirely sure what distinction the authors are trying to make here.

- In the same section, the authors then remark that ‘Our results open new perspectives to improve the variational approximation…’ and ‘we believe our results might foster further research in…’, before proceeding to list a dozen or so papers which might be inspired by this work. However, I can’t really see how the single contribution being presented in this paper can have significant impact on the related work. I encourage the authors to substantiate their claims with more concrete examples rather than simply include vague mentions of other papers.

- The structure and content of Section 3 is quite perplexing. Effectively, up until Equation 3, the authors are simply restating how to use VI for BNNs, with no mention whatsoever of how this fits in the storyline of model calibration. Whereas such a section should have contained novel methodology and/or intuition, the only reference to using BNNs for post-calibration is found in a small paragraph at the end of Pg 4, before immediately proceeding to the Experiments section. Once again, this makes any contributions of the paper unclear and inconclusive. Spurious comments such as the inconsequential connection to MDL further accentuate the paper’s lack of identity and focus.
- There are also some problematic technical details in this section, such as the definitive choice of using a two-layered BNN with no justification whatsoever. It is well known than plain BNNs also struggle to deliver well-calibrated outputs, and yet the authors immediately settle on a two-layered fully-connected network without stopping to consider whether some other network configuration or initialisation scheme might be more appropriate. Some introspection is later given in the experiment accompanied by Figure 2, but the analysis carried out there is just not sufficient.

- There are some instances where the authors use text while in math mode, which gives poor formatting as exemplified by ‘conf’ in Equation 4.

- Referring to ‘datasets’ as ‘databases’ in Section 4.1 is unusual. Some of the commentary in this subsection is also very difficult to interpret. For example, what is meant by ‘uses BNNs’? Does this mean that a BNN appears in the model being calibrated or is this referring to the BNN used to carry out calibration? The majority of these ambiguous statements could have been avoided had more care been given to checking the paper properly before submission.
- In their discussion of the results, the authors state that ‘We cannot conclude that BNNs are calibrating at the cost of losing accuracy’, which I consider to be an overly sunny view of the results. Even if minor, a dip in accuracy is observed in almost every example provided in the Experiments section, dropping as much as 3% for CIFAR-100. Given that calibration is the primary focus of this paper, it might also be worth including another metic for validating this criteria, such as the Brier score.
-- Recommendation --
Unfortunately, the material presented here is neither significant enough nor sufficiently explored to spark much interest. The overall scope of the paper is disappointingly limited, while novel ideas and design choices are poorly motivated and communicated throughout. This submission feels rushed and incomplete, and consequently well below the conference’s standards.
Pros/Cons summary:
+  The proposal yields good results in the provided experiments
-   Minor contributions that are not convincing enough
-   Muddled presentation of ideas
-   Dubious or weakly motivated design choices
-   Poorly written with plenty of typos
-   Difficult to follow


In this paper, we investigate the underlying factor that leads to the failure and success in training of GANs. Specifically, we study the property of the optimal discriminative function f * (x) and show that f * (x) in most GANs can only reflect the local densities at x, which means the value of f * (x) for points in the fake distribution (P g ) does not contain any information useful about the location of other points in the real distribution (P r ). Given that the supports of the real and fake distributions are usually disjoint, we argue that such a f * (x) and its gradient tell nothing about "how to pull P g to P r ", which turns out to be the fundamental cause of failure in training of GANs. We further demonstrate that a well-defined distance metric (including the dual form of Wasserstein distance with a compacted constraint) does not necessarily ensure the convergence of GANs. Finally, we propose Lipschitz-continuity condition as a general solution and show that in a large family of GAN objectives, Lipschitz condition is capable of connecting P g and P r through f *

[pros]
- It proposes a general formulation of GAN-type adversarial learning as in (1), which includes the original GAN, WGAN, and IPM-type metrics as special cases.
- It also proposes use of the penalty term in terms of the Lipschitz constant  of the discriminative function.
[cons]
- Some of the arguments on the Wasserstein distance and on WGAN are not sound.
- Theorem 3 does not make sense.
- The proposed scheme is eventually similar to the gradient-penalty-based formulation in Gulrajani et al. (2017).
[Quality]
I found some weaknesses in this paper, so that I judge the quality of this paper not to be high. For example, the criticisms on the Wasserstein distance in Section 2.3 and in Section 4.4, as well as the argument on WGAN at the end of Section 3.1, is not sound. The claim in Theorem 3 does not make sense, if we literally take its statement. All these points are detailed below.
[Clarity]
The main paper is clearly written, whereas in the appendices I noticed several grammatical and spelling errors as well as unclear descriptions.
[Originality]
Despite that the arguments in this paper are interesting, the proposed scheme is somehow eventually similar to the gradient-penalty-based formulation in Gulrajani et al. (2017), with differences being introduction of loss metrics $\phi,\varphi,\psi$ and the form of the gradient penalty, $\max \|\nabla f(x)\|_2^2$ in this paper versus $E[(\|\nabla f(x)\|_2-1)^2]$ in Gulrajani et al. (2017). This fact has made me to think that the originality of this paper is marginal.
[Significance]
This paper is significant in that it would stimulate empirical studies on what objective functions and what types of gradient penalty are efficient in GAN-type adversarial learning.
Detailed comments:
In Section 2.3, the authors criticize use of the Wasserstein distance as the distance function of GANs, but their criticism is off the point. It is indeed a problem not of the Wasserstein distance itself, but of its dual formulation.
It is true mathematically that $f$ in equation (8) does not have to be defined outside the supports of $P_g$ and $P_r$ because it does not affect the expectations in (8). In practice, however, one may regard that $f$ satisfies the condition $f(x)-f(y)\le d(x,y)$ not only on the supports of $P_g$ and $P_r$ but throughout the entire space $\mathbb{R}^n$. It is equivalent to requiring $f$ to satisfy the 1-Lipschitz condition on $\mathbb{R}^n$, and is what WGAN (Arjovsky et al., 2017) tries to do in its implementation of the "critic" $f$ via a multilayer neural network with weight clipping.
One can also argue that, if one defines $f$ only on the supports of $P_g$ and $P_r$, then it should trivially be impossible to obtain gradient information which can change the support of $P_g$. The common practice of requiring the Lipschitz condition throughout $\mathbb{R}^n$ is thus reasonable from this viewpoint. This is therefore not the problem of the Wasserstein distance itself, but the problem regarding how the dual problem is implemented in learning of GANs. In this regard, the discussion in this section, as well as that in Section 4.4, is misleading.
On optimizing $k$, I do not agree with the authors's claim at the end of Section 3.1 that WGAN may not have zero gradient with respect to $f$ even when $P_g=P_r$. Indeed, when $P_g=P_r$, for any measurable function $f$ one trivially has $J_D[f]=E_{x\sim P_g}[f(x)]-E_{x\sim P_r}[f(x)]=0$, so that the functional derivative of $J_D$ with respect to $f$ does vanish identically.
I do not understand the claim of Theorem 3. I think that the assumption is too strong. If one literally takes "$\forall x \not= y$", then one can exchange $x$ and $y$ in the condition $f(y)-f(x)=k\|x-y\|$ to obtain $f(x)-f(y)=k\|y-x\|$, which together would imply $k=0$, and consequently $f$ is constant. One would be able to prove that if there exists $(x,y)$ with $x \not= y$ such that $f(y)-f(x)=k\|x-y\|$ holds then the gradient of $f$ at $x_t$ is equal to $k(y-x)/\|x-y\|$ under the Lipschitz condition.
Appendix G: Some notations should be made more precise. For example, in the definition of J_D the variable of integration $x$ has been integrated out, so that $J_D$ no longer has $x$ as its variable. The expression $\partial J_D/\partial x$ does not make any sense. Also, $J_D^*(k)$ is defined as "arg min" of $J_D$, implying as if $J_D^*(k)$ were a $k$-Lipschitz function.
Page 5, line 36: $J_D(x)$ appears without explicit definition.
Page 23, lines 34 and 38: Cluttered expression $\frac{\partial [}{\partial 2}]$ makes the statements not understandable. It also appears on page 24 several times.


Understanding the behavior of stochastic gradient descent (SGD) in the context of deep neural networks has raised lots of concerns recently. Along this line, we theoretically study a general form of gradient based optimization dynamics with unbiased noise, which unifies SGD and standard Langevin dynamics. Through investigating this general optimization dynamics, we analyze the behavior of SGD on escaping from minima and its regularization effects. A novel indicator is derived to characterize the efficiency of escaping from minima through measuring the alignment of noise covariance and the curvature of loss function. Based on this indicator, two conditions are established to show which type of noise structure is superior to isotropic noise in term of escaping efficiency. We further show that the anisotropic noise in SGD satisfies the two conditions, and thus helps to escape from sharp and poor minima effectively, towards more stable and flat minima that typically generalize well. We verify our understanding through comparing this anisotropic diffusion with full gradient descent plus isotropic diffusion (i.e. Langevin dynamics) and other types of position-dependent noise.

The paper studies the benefit of an anisotropic gradient covariance matrix in SGD optimization for training deep network in terms of escaping sharp minima (which has been discussed to correlate with poor generalization in recent literature).
In order to do so, SGD is studied as a discrete approximation of stochastic differential equation (SDE). To analyze the benefits of anisotropic nature and remove the confounding effect from scale of noise, the scale of noise in the SDE is considered fixed during the analysis. The authors identify the expected loss around a minimum as the efficient of escaping the minimum and show its relation with the hessian and gradient covariance at the minimum. It is then shown that when all the positive eigenvalues of the covariance matrix concentrate along the top eigenvector and this eigenvector is aligned with the top eigenvector of the Hessian of the loss w.r.t. the parameters, SGD is most efficient at escaping sharp minima. These characteristics are analytically shown to hold true for a 1 hidden layer network and experiments are conducted on toy and real datasets to verify the theoretical predictions.
Comments:
I find the main claim of the paper intuitive-- at any particular minimum, if noise in SGD is more aligned with the direction along which loss surface has a large curvature (thus the minimum is sharp along this direction), SGD will escape this minimum more efficiently. On the other hand, isotropic noise will be wasteful because a sample from isotropic noise distribution may point along flat directions of the loss even though there may exist other directions along which the loss curvature is large. However, I have several concerns which I find difficult to point out because *many equations are not numbered*.
1. In proposition 2, it is assumed under the argument of no loss of generality that both the loss at the minimum L_0=0 and the corresponding theta_0 =0. Can the authors clarify how both can be simultaneously true without any loss of generality?
2. A number of steps in proposition 2 are missing which makes it difficult to verify. When applying Ito's lemma and taking the integral from 0 to t, it is not mentioned that both sides are also multiplied with the inverse of exp(Ht).
3. In proposition 2, when computing E[L(theta_t)] on page 12, the equalities after line 3 are not clear how they are derived. Please clarify or update the proof with sufficient details.
4. It is mentioned below proposition 2 that the maximum of Tr(H. Sigma) under constraint (6) is achieved when Sigma* = Tr(Sigma). lambda_1 u1.u1^T, where lambda_1 is the top eigenvalue of H. How is lambda_1 a factor in Sigma*? I think Sigma* should be Tr(Sigma). u1.u1^T because this way the sum of eigenvalues of Sigma remains unchanged which is what constraint (6) states.
5. The proof of proposition 5 is highly unclear.Where did the inequality ||g_0(theta)||^2 <= delta.u^TFu + o(|delta|) come from? Also, the inequality right below it involves the assumption that u^Tg_0 g_0u <= ||g_0||^2 and no justification has been provided behind this assumption.
Regarding experiments, the toy experiment in section 5.1 is interesting, but it is not mentioned what network architecture is used in this experiment. I found the experiments in section 5.3 and specifically Fig 4 and Fig 7 insightful. I do have a concern regarding this experiment though. In the experiment on FashionMNIST in Fig 4, it can be seen that both SGD and GLD 1st eigvec escapes sharp minimum, and this is coherrent with the theory. However, for the experiment on CIFAR-10 in Fig 7, experiment with GLD 1st eigvec is missing. Can the authors show the result for GLD 1st eigvec on CIFAR-10? I think it is an important verification of the theory and CIFAR-10 is a more realistic dataset compared with FashionMNIST.
A few minor points:
1. In the last paragraph of page 3, it is mentioned that the probability of escaping can be controlled by the expected loss around minimum due to Markov's inequality. This statement is inaccurate. A large expected loss upper bounds the escaping probability, it does not control it.
2. Section 4 is titled "The anisotropic noise of SGD in deep networks", but the sections analyses a 1 hidden layes network. This seems inappropriate.
3. In the conclusion section, it is mentioned that the theory in the paper unifies various existing optimization mentods. Please clarify.
Overall, I found the argument of the paper somewhat interesting but I am not fully convinced because of the concerns mentioned above.


Machine learned models exhibit bias, often because the datasets used to train them are biased. This presents a serious problem for the deployment of such technology, as the resulting models might perform poorly on populations that are minorities within the training set and ultimately present higher risks to them. We propose to use high-fidelity computer simulations to interrogate and diagnose biases within ML classifiers. We present a framework that leverages Bayesian parameter search to efficiently characterize the high dimensional feature space and more quickly identify weakness in performance. We apply our approach to an example domain, face detection, and show that it can be used to help identify demographic biases in commercial face application programming interfaces (APIs).

Summary:
=========
The paper uses a proof-of-concept Bayesian parameter search-based simulation in virtual environment to probe biases of an already trained model towards specific categories that may have been sparsely represented in the training set. Understanding bias in trained models, especially in models involving end-to-end deep neural networks learners, is of high importance in machine learning. More specifically, probing the source of unintentional bias introduced as a result of skewed distribution in the training set and dissecting the biased performance is important for many applications such as surveillance, criminal profiling, medical diagnosis and predicting creditworthiness of a person.
The authors used four commercial face recognition APIs (by Microsoft, Google, IBM, and Face++) as test bed for this investigation.
Strength:
========
- The paper reads well and is easy to follow.
- The application of face detection and recognition is a good choice as it is precursor to detailed analysis in surveillance and criminal profiling.
- The choice of the controlled Bayesian parameter search enables one to control the amount of variation with respect to the expected uncertainty in performance of the classifier on the generated input from the simulator.
- The use of standardized measures such as Fitzpatrick’s skin tone and FACS intensity help in replication and consistency in the evaluation.
Weakness:
=========
- Although evaluating commercial APIs is good enough in demonstrating the existence of the bias, access to the trained model with possibility to retrain the model in such a way to mitigate a bias in particular parameter could have helped to further tie the drop in performance to the parameter variation.
- This work is preliminary and only involves a single person face manipulated in the parameter space. It lacks diversity of in samples and as such limits the analysis to draw strong conclusions. As such a generative network (such as a GAN trained to generate diverse samples while controlling for the parameters under investigation could have helped to draw more generalized conclusion.
- The age parameter variation is not convincingly different across the values considered. It would have been interesting to use models trained for age progression such as [1] for a more diverse variation in the age parameter space.
Minor comments:
===============
- Figures 4 and 5 could have used better captions describing the ranges for instance for age 1 is older and for skin tone 1 is darker (although indicated in Fig. 3). Captions should be self contained. Fig. 5 caption should describe the chance performance in each case.
- The manuscript should be revised to make in text citation formats consistent (some places it uses authors (year) and other places it uses (authors, year)). Also, minor punctuation and syntactic errors should be fixed.
[1] Yang, H., Huang, D., Wang, Y. and Jain, A.K., 2017. Learning face age progression: A pyramid architecture of gans. arXiv preprint arXiv:1711.10352.


One of the most notable contributions of deep learning is the application of convolutional neural networks (ConvNets) to structured signal classification, and in particular image classification. Beyond their impressive performances in supervised learning, the structure of such networks inspired the development of deep filter banks referred to as scattering transforms. These transforms apply a cascade of wavelet transforms and complex modulus operators to extract features that are invariant to group operations and stable to deformations. Furthermore, ConvNets inspired recent advances in geometric deep learning, which aim to generalize these networks to graph data by applying notions from graph signal processing to learn deep graph filter cascades. We further advance these lines of research by proposing a geometric scattering transform using graph wavelets defined in terms of random walks on the graph. We demonstrate the utility of features extracted with this designed deep filter bank in graph classification of biochemistry and social network data (incl. state of the art results in the latter case), and in data exploration, where they enable inference of EC exchange preferences in enzyme evolution.

# Summary of the paper
Inspired by the success of deep filter banks, this paper presents a designed deep filter bank for graphs that is based on random walks.  More precisely, the technique uses lazy random walks, expressed in terms of the graph Laplacian, and re-frames this in terms of graph signal processing. Similarly to wavelets, graph node features are calculated at different scales and subsequently summed in order to remain invariant under permutations. Several experiments on graph data sets demonstrate the performance of the new technique.
# Review
This paper is written very well and explains its method with high clarity. The principal issues I see are as follows:
- The originality of the contributions is not clear
- Missing theoretical discussion
- The experimental setup is terse and slightly confusing
Concerning the originality of the paper, the differences to Gama et al., 'Diffusion Scattering Transforms on Graphs' are not made clear. Cursory reading of this publication shows a large degree of similarity. Both of the papers make use of diffusion geometry, but Gama et al. _also_ define a multi-scale filter bank, similar to Eq. 4 and 5. The paper needs to position itself more clearly vis-à-vis this other publication. Is the present approach to be seen more as an application of the theory that was developed in the paper by Gama et al.? What are the key similarities and differences? In terms of space, this could be added to Section 3.2, which could be rephrased as a generic 'Differences to other methods' section and has to be slightly condensed in any case (see my suggestions below). Another publication by Zou & Lerman, 'Graph Convolutional Neural Networks via Scattering', is also cited as an inspiration, but here the differences are larger in my understanding and do not necessitate further justification. Last, the publication 'Graph Capsule Convolutional Neural Networks' by Verma & Zhang is also cited for the definition of 'scattering capsules'. Again, cursory reading of the publication shows that this approach is similar to the presented one; the only difference being which features are used for the definition of capsules. I recommend referring to the invariants as 'capsules' and link it back to Verma & Zhang so that the provenance of the terminology is clear.
Concerning the theoretical part of the paper, I miss a discussion of the complexity of the approach. Such a discussion does not have to be long, but in particular since the paper mentions that the applicability of scattering transforms for transfer learning (and also remarks about the universality of them in Section 4), some space should be devoted to theoretical considerations (memory complexity, runtime complexity). This would strengthen the paper a lot, in particular in light of the complexity of other approaches! Furthermore, an additional experiment about the stability of scattering transforms appears warranted. While I applaud the experimental description in the paper (number of scales, how the maximum scale is chosen, ...), an additional proof or experiment in the appendix should deal with the stability. Let's assume that for extremely large graphs, I am content with 'almost-but-not-quite-as-good' classification performance. Is it possible to achieve this by limiting the number of scales? How much to the results depend on the 'right' choice here?
Concerning the experimental setup, I think that the way (average) accuracies are reported at present is slightly misleading. The paper even remarks about this in footnote 2. While I understand the need of demonstrating the universality of these features, I think that the current setup is not optimal for this. I would recommend (in addition to reporting accuracies) a transfer learning setup rather in which the beneficial properties of the new method can be better explored. More precisely, the claim from Section 4, 4th paragraph ('Since the scattering transform...') needs to be further explored. This appears to be a unique feature of the new method. The current experimental setup does not exploit it. As a side-note, I realize that this might sound like a standard request for 'show more experiments', but I think the paper would be more impactful if it contained one scenario in which its benefits over other approaches are clear.
# Suggestions for improvement
The paper flows extremely well and it is clear that care has been taken to ensure that everything can be understood. I liked the discussion of invariance properties in particular. There are only a few minor things that can be improved:
- 'covariant' and 'equivariant', while common in (graph) signal processing, could be briefly explained to increase accessibility and impact
- 'order' and 'layer' are not used consistently: in the caption of Figure 2a, the term 'order' is used, but for Eq. 4 and 5, for example, the term 'layer' is employed. Since 'layer' is more reminiscent of a DNN, I would suggest to use 'order' throughout the paper, because it meshes better with the way the scattering invariants are defined.
- the notation $Sx$ is slightly overloaded; in Figure 2a, for example, it is not clear at first that the individual cascades are supposed to form a *set*; this is only explained at the end of Section 3.1; to make matters more consistent, the figure should be updated and the combination of individual cascades should be made clear
- In Eq. 5, the bars of the absolute value are not set correctly; the absolute value should cover $\psi_j x(v_i)$ and not $(v_i)$ itself.
- minor 'gripe': $\psi^{(J)}$ is defined as a set in Eq. 2, but it is treated as a matrix or an operator (and also referred to as such); this should be more consistent
- The discussion of the aggregation of multiple statistics in Section 3.2 appears to be somewhat redundant in light of the discussion for Eq. 4 and Eq. 5 in the preceding section
- in the appendix, more details about the training of the FCN should be added; all other parts of the experiments are described in sufficient detail, but the training process requires additional information about learning rates etc.


One intriguing property of neural networks is their inherent vulnerability to adversarial inputs, which are maliciously crafted samples to trigger target networks to misbehave. The state-of-the-art attacks generate adversarial inputs using either pixel perturbation or spatial transformation. Thus far, several provable defenses have been proposed against pixel perturbation-based attacks; yet, little is known about whether such solutions exist for spatial transformation-based attacks. This paper bridges this striking gap by conducting the first systematic study on provable defenses against spatially transformed adversarial inputs. Our findings convey mixed messages. On the impossibility side, we show that such defenses may not exist in practice: for any given networks, it is possible to find legitimate inputs and imperceptible transformations to generate adversarial inputs that force arbitrarily large errors. On the possibility side, we show that it is still feasible to construct adversarial training methods to significantly improve the resilience of networks against adversarial inputs over empirical datasets. We believe our findings provide insights for designing more effective defenses against spatially transformed adversarial inputs.

Summary: The paper studies a new attack model based on spatial transformations. The authors first formalize an attack model based on spatial transformation and then study attacks and defenses for this model.
Clarity: While the paper studies an important problem -- it's important to move out of the norm ball based attack models and consider different attacks like spatial transformations, in the current version, the presentation lacks clarity in both the formulation of the attack model, attacks, defenses and explanation of the results. For example, the impossibility result isn't clear: the claim is that any classifier has adversarial spatial transformations that are successful in causing misclassifcation for some threshold on the size of transformation. There is no explanation of how large this threshold is in practice. Is it small enough to be called an "impossibility result"? What does this threshold intuitively depend on?
Originality: The key contribution seems to be the formalization of some notion of spatial transformation. However, the final expression (Proposition 1) basically looks just like an l_p norm but after transforming it by some "fixed" matrix M. The expressions for this new attack model where || M r|| < \eps for some perturbation \eps look pretty similar to the case previously considered (where M was essentially identity). For example, Raghunathan et al. 2018 and Hein & Andriushchenko 2017. The paper is also missing discussion on the structure of this matrix M, and how it changes the attacks and defenses in practice
Significance: I think the problem of spatial transformation based adversarial examples is important and the authors have the right goals. However, the current presentation makes it hard to understand the main results provided and hence I would rate that the contribution is not very significant.
Overall: I highly recommend the authors to revise the presentation and clarify a) the main conceptual differences of the new attack model (matrix M of proposition 1) b) Formalize the impossibility and possibility results carefully with concrete theoretical/empirical results to back the claims


DELTA: DEEP LEARNING TRANSFER USING FEATURE MAP WITH ATTENTION FOR CONVOLUTIONAL NET- WORKS
Transfer learning through fine-tuning a pre-trained neural network with an extremely large dataset, such as ImageNet, can significantly accelerate training while the accuracy is frequently bottlenecked by the limited dataset size of the new target task. To solve the problem, some regularization methods, constraining the outer layer weights of the target network using the starting point as references (SPAR), have been studied. In this paper, we propose a novel regularized transfer learning framework DELTA, namely DEep Learning Transfer using Feature Map with Attention. Instead of constraining the weights of neural network, DELTA aims to preserve the outer layer outputs of the target network. Specifically, in addition to minimizing the empirical loss, DELTA aligns the outer layer outputs of two networks, through constraining a subset of feature maps that are precisely selected by attention that has been learned in a supervised learning manner. We evaluate DELTA with the state-of-the-art algorithms, including L 2 and L 2 -SP . The experiment results show that our method outperforms these baselines with higher accuracy for new tasks.

This is a reasonable paper based on a simple intuition. The authors have noticed that some of the state of the art methods (they use Li et al - ICML18 as the main reference) are using only some simple normalization for improving the transfer learning and as such they propose preserving the outer layer output of the target network and aligning it with the one of the source network. On top of that they also propose modeling the difference of feature maps considering an attention mechanism obtain through supervised learning.
The idea in itself is interesting and valuable. However, I have had some difficulty in understanding precisely how the "behavior" is really regularized. While I understand what is depicted in Figure 1 I'm not completely sure this really means that the network behavior is regularized rather than simply correlating the two outputs. In the evaluation, the authors present in Figure 4 some qualitative examples but I would have expected to see some quantitative evaluation of this. I would have liked to see experiments on some larger datasets that are commonly used in computer vision (e.g., Caltech 256 is rather old even if it has been used in Li et al.). The quantitative results in Table 1 and 2 indicate some slight improvement but I'm not completely convinced that this is really significant in the end. The results in Figure 4 tend to show that with the attention mechanism there is a central bias and most of the results tend to be concentrated on the center of the image (in this case the result might also be correct but the examples presented are not too eloquent).


Hierarchical Reinforcement Learning is a popular method to exploit temporal abstractions in order to tackle the curse of dimensionality. The options framework is one such hierarchical framework that models the notion of skills or options. However, learning a collection of task-agnostic transferable skills is a challenging task. Option discovery typically entails using heuristics, the majority of which revolve around discovering bottleneck states. In this work, we adopt a method complementary to the idea of discovering bottlenecks. Instead, we attempt to discover "landmark" sub-goals which are prototypical states of well connected regions. These sub-goals are points from which a densely connected set of states are easily accessible. We propose a new model called Successor options that leverages Successor representations to achieve the same. We also design a novel pseudo-reward for learning the intra-option policies. Additionally, we describe an Incremental Successor options model that iteratively builds options and explores in environments where exploration through primitive actions is inadequate to form the Successor representations. Finally, we demonstrate the efficacy of our approach on a collection of grid worlds and on complex high dimensional environments like Deepmind-Lab.

Summary: This paper tries to tackle the option discovery problem, by building on recent work on successor representation and eigenoptions. Although this is an extreme important problem, I feel the paper fails to deliver on its promise. The authors propose a way of clustering states via their SR/SF representation and they argue that this would lead to the discovery of subgoals that are fundamentally different from the popular choices in literature, like bottleneck states. They argue that this discovery procedure would lead to states “better for exploration”, “provide greater accessibility to a larger number of states”. Both of which sound promising, but I felt the actual evaluation fails to show or even assess either of these rigorously. Overall, after going through the paper, it is not clear what are the properties of these discovered subgoal states and why they would be better for exploration and/or control.
Clarity: Can be improved significantly! It requires several reads to get some of the important details. See detailed comments.
Originality and Significance: Very limited, at least in this version. The quantitative, and in some cases qualitative, evaluation lacks considerably. The comparison with the, probably most related, method (eigenoption) yield some slight improvement. But unfortunately, I was not conceived that this grain of empirical evidence would transfer to other scenarios. I can’t see why that would that be the case, or in which scenarios this might happen. At least those insights seem to be missing from the discussion of the results.
Detailed comments and questions:
1) Section 3.1: Latent Learning. There are a couple of design choices here that should have been more well explained or motivated:
i) The SR were built under the uniformly random policy. This is a design choice that might work well for gridworld/navigation type of domains but there are MDP where the evaluation under this particular policy can yield uninformative evaluations. Nevertheless this is an interesting choice that I think deserved more discussion, especially the connection to previous work on proto-value functions and eigenoptions. For instance, if both of these representations -- eigenoptions and the proposed successor option model -- aim to represent the SR under the uniform policy, why does know do (slightly) better than the other? Moreover, how would these compare under a different policy (non-uniform).
ii) The choice of reward. The notation is a bit confusing here, as it’s somewhat inconsistent with the definitions (2-4). Also, more generally, It is not clear throughout if we are using the discounted or undiscounted version of SR/SFs -- (2-3) introduce the discounted version, (4) seems to be undiscounted. Not clear if (5) refers to the discounted or undiscounted version. Nevertheless, I am guessing this was meant as a shaping reward, thus \gamma=1 for (5). But if that’s the case, according to eq. (2), most of the time I would expect \psi_s(s_{t+1}) and \psi_s(s_{t}) to have the same value. Could you explain why that is not true (at least in your examples)?
iii) Termination set: Q(s,a)<=0. This again seems a bit of an arbitrary choice and it’s not clear which reward this value function takes into account.
2) Figure 2: The first 2 figures representing the SRs for the two room domain: the values for one of the rooms seems to be zero, although one would expect a smoother transition around the ‘doorway’, otherwise the shaping won’t point in the right direction for progression. Again, this might suggest that more informative(control) policy might give you more signal.
3) Section 3.2: ‘The policy used for learning the SR is augmented with the previously learnt options‘. Can you be more precise about how this is done? Which options used? How many of them? And in which way are they used? This seems like a very important detail. Also is this augmented policy used only for exploration?
4) SRmin < \sum_{s’} ψ(s, :) < SRmax. Is this meant to be an expectation over all reachable next states or all states in the environment? How is this determined or translated in a non-tabular setting. Not sure why this is a proxy to how ‘developed’ this learning problem or approximation is. Can you please expand on your intuition here?
5) Section 3.3. The reward definition seems to represent how much the progress between \phi(s_t+1) - \phi(s) aligns with the direction of the goal. This is very reminest of FuN [2] -- probably a connect worth mentioning and exploring.
6) Figure 4: Can you explain what rho is? It seems to be an intermediate representation for shared representation \phi. Where is this used?
7) Experiments:
“a uniformly random policy among the options and actions (typically used in exploration) will result in the agent spending a large fraction of it’s time near these sub-goals”. Surely this is closely linked to the termination condition of the option and the option policy. How is this assessed?
“in order to effectively explore the environment using the exploration policy, it is important to sample actions and options non-uniformly”. It would be good to include such a comparison, or give a reason why this is the case. It’s also not clear how many of the options we are considering in this policy and how extensive their horizons will be. This comes back to the termination condition in Section 3.1 which could use an interpretation.
“In all our experiments, we fix the ratio of sampling an option to an action as 1:19.” This seems to be somewhat contradictory to the assumption that primitive actions are not enough to explore effectively this environment.
Figure 8. I think this experiment could use some a lot more details. Also it would be good to guide the reader through the t-SNE plot in Figure 8a. What’s the observed pattern? How does this compare to the eigenoption counterpart.
8) General comment on the experiments: There seems to be several stages in learning, with non-trivial dependencies. I think the exposition would improve a lot if you were more explicit about these: for instance, if the representation continually refined throughout the process; when the new cluster centers are inferred are the option policies learnt from scratch? Or do they build on the previous ones? Does this procedure converge -- aka do the clusters stabilize?
9) Quantitative performance evaluation was done only for the gridworld scenarios and felt somewhat weak. The proposed tasks (navigation to a goal location) is exactly what SFs are trained to approximate. No composition of (sub)tasks, nor tradeoff-s of goals were studied [1,3] -- although they seem natural scenario of option planning and have been studied in previous SFs work. Moreover, if the SFs are built properly, in these gridworlds acting greedily with respect to the SFs (under the uniformly random policy) should be enough to get you to the goal. Also, probably this should be a baseline to begin with.
References:
[1] Andre Barreto, Will Dabney, Remi Munos, Jonathan J Hunt, Tom Schaul, Hado P van Hasselt, and ´ David Silver. Successor features for transfer in reinforcement learning. In Advances in Neural Information Processing Systems, pp. 4055–4065, 2017.
[2] Vezhnevets, A.S., Osindero, S., Schaul, T., Heess, N., Jaderberg, M., Silver, D. and Kavukcuoglu, K., 2017, July. FeUdal Networks for Hierarchical Reinforcement Learning. In International Conference on Machine Learning (pp. 3540-3549).
[3] Barreto, A., Borsa, D., Quan, J., Schaul, T., Silver, D., Hessel, M., Mankowitz, D., Zidek, A. and Munos, R., 2018, July. Transfer in deep reinforcement learning using successor features and generalised policy improvement. In International Conference on Machine Learning (pp. 510-519).


This paper presents the Variation Network (VarNet), a generative model providing means to manipulate the high-level attributes of a given input. The originality of our approach is that VarNet is not only capable of handling pre-defined attributes but can also learn the relevant attributes of the dataset by itself. These two settings can be easily combined which makes VarNet applicable for a wide variety of tasks. Further, VarNet has a sound probabilistic interpretation which grants us with a novel way to navigate in the latent spaces as well as means to control how the attributes are learned. We demonstrate experimentally that this model is capable of performing interesting input manipulation and that the learned attributes are relevant and interpretable.

This paper introduces a new framework for learning an interpretable representation of images and their attributes. The authors suggest decomposing the representation into a set of 'template' latent features, and a set of attribute-based features. The attribute-based features can be either 'free', i.e. discovered from the data, or 'fixed', i.e. based on the ground truth attributes. The authors encourage the decomposition of the latent space into the 'template' and the 'attributes' features by training a discriminator network to predict whether the attributes and the template features come from the same image or not.
While the idea is interesting, the paper is lacking an experimental section, so the methodology is impossible to evaluate. Furthermore, while the authors spend many pages describing their methodology, the writing is often hard to follow, so I am still confused about the exact implementation of the attribute features \phi(x, m) for example. The authors do point to the Appendix for their Experiments section, however this is not a good idea. The paper should be self-contained and the authors should not assume that their readers will read the information presented in the Appendix, which is always optional.
Unfortunately, even the experimental section presented in the Appendix is not comprehensive enough to evaluate the proposed method. The authors train the model on a single dataset (MNIST), no baseline or ablation results are presented, and all the results are purely qualitative. Given that the ground truth attribute decomposition for MNIST is not known, even the qualitative results are impossible to evaluate. I recommend that the authors present quantitive results in the updated version of their paper (i.e. disentanglement metric scores, the log-likelihood of the reconstructions), including new experiments on a dataset like dSprites or CelebA, where the ground truth attributes are known.


SIGNSGD WITH MAJORITY VOTE IS COMMUNICATION EFFICIENT AND FAULT TOLERANT
Training neural networks on large datasets can be accelerated by distributing the workload over a network of machines. As datasets grow ever larger, networks of hundreds or thousands of machines become economically viable. The time cost of communicating gradients limits the effectiveness of using such large machine counts, as may the increased chance of network faults. We explore a particularly simple algorithm for robust, communication-efficient learning-SIGNSGD. Workers transmit only the sign of their gradient vector to a server, and the overall update is decided by a majority vote. This algorithm uses 32× less communication per iteration than full-precision, distributed SGD. Under natural conditions verified by experiment, we prove that SIGNSGD converges in the large and mini-batch settings, establishing convergence for a parameter regime of ADAM as a byproduct. Aggregating sign gradients by majority vote means that no individual worker has too much power. We prove that unlike SGD, majority vote is robust when up to 50% of workers behave adversarially. The class of adversaries we consider includes as special cases those that invert or randomise their gradient estimate. On the practical side, we built our distributed training system in Pytorch. Benchmarking against the state of the art collective communications library (NCCL), our framework-with the parameter server housed entirely on one machine-led to a 25% reduction in time for training resnet50 on Imagenet when using 15 AWS p3.2xlarge machines.

The authors present a distributed implementation of signSGD with majority vote as aggregation. The result is a  communication efficient and byzantine robust distributed training method. This is an interesting and relevant problem. There are two parts in this paper: first the authors prove a convergence guarantee for signSGD, and then they prove that under a weak adversary attack signSGD will be robust to a constant fraction of adversarial nodes. The authors conclude with some limited experiments.
Overall, the idea of combining low-communication methods with byzantine resilience is quite interesting. That is, by limiting the domain of the gradients one expects that the power of an adversary would be limited too. The application of the majority vote on the gradients is an intuitive technique that can resolve weak adversarial attacks. Overall, I found the premise quite interesting.
There are several issues that if fixed this could be a great paper, however I am not sure if there is enough time between rebuttals to achieve this for this round of submissions. I will summarize these key issues below.
1) Although the authors claim that this is a communication efficient technique, signSGD (on its communication merit) is not compared with any state of the art communication efficient training algorithm, for example:
- 1Bit SGD [1]
- QSD [2]
- TernGrad [3]
- Deep Gradient compression [4]
I think it is important to include at least one of those algorithms in a comparison. Due to the lack of comparisons with state of the art it is hard to argue on the relative performance of signSGD.
2) Although the authors claim byzantine resilience, this is against a very weak type of adversary, eg one that only sends back the opposite sign of the local stochastic gradient. An omniscient adversary can craft attacks that are significantly more sophisticated, for which a simple majority vote would not work. Please see the results in [b1].
3) The authors although reference some limited literature on byzantine ML, they do not compare with other byzantine tolerant ML methods. For example check [eg, b1-b4] below. Again, due to the lack of comparisons with state of the art it is hard to argue on the relative performance of signSGD.
Overall, although the presented ideas are promising, a substantial revision is needed before this paper is accepted for publication. I think it is extremely important that an extensive comparison is carried out with respect to both communication efficient algorithms, and/or byzantine tolerant algorithms, since signSGD aims to be competitive with both of these lines of work. This is a paper that has potential, but is currently limited by its lack of appropriate comparisons.
[1] https://www.microsoft.com/en-us/research/wp-content/uploads/2016/02/IS140694.pdf
[2] https://papers.nips.cc/paper/6768-qsgd-communication-efficient-sgd-via-gradient-quantization-and-encoding.pdf
[3] https://papers.nips.cc/paper/6749-terngrad-ternary-gradients-to-reduce-communication-in-distributed-deep-learning.pdf
[4] https://arxiv.org/pdf/1712.01887.pdf
[b1] https://arxiv.org/pdf/1802.07927.pdf
[b2] https://arxiv.org/pdf/1803.01498.pdf
[b3] https://dl.acm.org/citation.cfm?id=2933105
[b4] https://arxiv.org/pdf/1804.10140.pdf
[b5] https://arxiv.org/pdf/1802.10116.pdf
########################
I would like to commend the authors for making a significant effort in revising their manuscript. Specifically, I think adding the experiments for QSGD and Krum are an important addition. However, I still have a few major that in my opinion are significant:
- The experiments for QSGD are only carried for the 1-bit version of the algorithm. It has been well observed that this is by far the least well performing variant of QSGD. That is, 4 or 8 bit QSGD seems to be significantly more accurate for a given time budget. I think the goal of the experiments should not be to compare against other 1-bit algorithms (though to be precise, 1-bit QSGD is a ternary algorithm) , but against the fastest low-communication algorithm. As such, although the authors made an effort in adding more experiments, I am still not convinced that signSGD will be faster than 4 or 8 bit QSGD. I want to also acknowledge in this comment the fact that these experiments do take time, and are not easy to run, so I commend them again for this effort.
- My second comment relates to comparisons with state of the art algorithms in byzantine ML. The authors indeed did compare against Krum, however, as noted in my original review there are many works following Blanchard et al.
For example as I noted https://arxiv.org/pdf/1802.07927.pdf (the Bulyan algorithm) shows that there exist significantly stronger defense mechanisms for byzantine attacks. I think it would have been a much stronger comparison to compare with Bulyan.
Overall, I think the paper has good content, and the authors significantly revised their paper according to the reviews. However, several more experiments are needed for convincing a potential reader of the main claims of the paper, i.e., that signSGD is a state of the art communication efficient and byzantine tolerant algorithm.
I will increase my score from 5 to 6, and I will not oppose the paper being rejected or accepted. My personal opinion is that a resubmission for a future venue would yield a much stronger and more convincing paper assuming more extensive and thorough comparisons are added.


We address a fundamental problem in chemistry known as chemical reaction product prediction. Our main insight is that the input reactant and reagent molecules can be jointly represented as a graph, and the process of generating product molecules from reactant molecules can be formulated as a sequence of graph transformations. To this end, we propose Graph Transformation Policy Network (GTPN) − a novel generic method that combines the strengths of graph neural networks and reinforcement learning to learn the reactions directly from data with minimal chemical knowledge. Compared to previous methods, GTPN has some appealing properties such as: end-to-end learning, and making no assumption about the length or the order of graph transformations. In order to guide model search through the complex discrete space of sets of bond changes effectively, we extend the standard policy gradient loss by adding useful constraints. Evaluation results show that GTPN improves the top-1 accuracy over the current state-of-the-art method by about 3% on the large USPTO dataset. Our model's performances and prediction errors are also analyzed carefully in the paper.

Update:
Score increased.
___________________________________
Original review:
The paper presents an approach to predict the products of chemical reactions, given the reactants and reagents. It works by stepwise predicting the atom pairs that change their bonds in course of reaction, and then adjusting the bonds between them. This can be interpreted as a stepwise graph transformation.
I think this is an interesting applied ML paper with fair results. The presentation is clear and understandable. The experimental setup is reasonable. However, the paper is not ready yet to be accepted in my opinion.
I think a higher score is justified if the authors address the following points:
- Relation to previous work, originality
In contrast to what the authors claim, what is predicted here is not exactly the reaction mechanism, but an implementation of the principle of minimal chemical distance, which was already described by Ugi and coworkers in 1980 [see Jochum, Gasteiger, Ugi, The Principle of Minimum Chemical Distance Angew. Chem. Int. Ed. Engl. 1980, 19, p 495-505].
The “insight” the authors have about treating reagents and reactants jointly is Organic Chemistry 101, and that reactions are stepwise graph transformations was also reported by Ugi et al, however, already in 1979! [Ugi, et al. "New applications of computers in chemistry." Angewandte Chemie International Edition in English 18.2 (1979): 111-123. ] I assume the authors were not aware of these papers, but now they are, so this needs to be modified accordingly, and these papers need to be referred to in the introduction.
- Questions:
The authors suggest that graph neural networks are more generic that so-called heuristic features (fingerprints) – which, as Duvenaud et al have elaborated, can be interpreted just as graph neural networks themselves – with fixed weights. Also, there are results by the Hochreiter group which show that graph neural networks perform worse that classical chemical features under rigorous testing { DOI: 10.1039/C8SC00148K } Do the authors think their models could also improve if they used the classical fingerprints?
Is the GRU really needed to encode the past bond changes? What happens if you remove it?
The statement that the method has the advantage of not relying on handcrafted reaction templates is somewhat overselling, because instead it uses a handcrafted complex neural network architecture. How complicated is it to train the network? If you remove some of the “tricks” of shaping the loss function, does it still train well?
To what degree is the ranking of the different models just a matter of hyperparameter tuning or different architectures? If you used a different graph neural net instead of an MPNN on top of your GTPN method, what would you expect? Are the differences between the models significant?
During prediction, you apply a flag to the starting molecules if they are a reagent or reactant. How do you know upfront what a reagent or reactant is during inference?
On page 5 and 7, you speak of the correct sequence of reaction triples (which implies an ordering), even though earlier you claim the algorithm is order-invariant? Where do you get the ground truth labels from? I assume these are already annotated in the data.
In the appendix, please replace the pie chart with a bar chart.
- Language:
I would suggest the authors to adapt the language of their paper towards a more academic tone. Science is not a sports competition of getting slightly higher numbers in benchmarks, but rather about providing insights and explanations. Words like “beating” or “record” are locker room talk, and to be avoided.


Parametric texture models have been applied successfully to synthesize artificial images. Psychophysical studies show that under defined conditions observers are unable to differentiate between model-generated and original natural textures. In industrial applications the reverse case is of interest: a texture analysis system should decide if human observers are able to discriminate between a reference and a novel texture. For example, in case of inspecting decorative surfaces the detection of visible texture anomalies without any prior knowledge is required. Here, we implemented a human-vision-inspired novelty detection approach. Assuming that the features used for texture synthesis are important for human texture perception, we compare psychophysical as well as learnt texture representations based on activations of a pretrained CNN in a novelty detection scenario. Additionally, we introduce a novel objective function to train one-class neural networks for novelty detection and compare the results to standard one-class SVM approaches. Our experiments clearly show the differences between human-vision-inspired texture representations and learnt features in detecting visual anomalies. Based on a digital print inspection scenario we show that psychophysical texture representations are able to outperform CNN-encoded features.

The submission investigates the problem of detecting perceptual anomalies in visual textures.
It proposes features from three different models, the Portilla & Simoncelli texture model (PS), the Spatial Vision model by Schuett and Wichmann (SW) and CNN features from the VGG network. From these features it trains two anomaly detectors: one out of the box one-class SVM and a 3 layer neural network. The network is optimised with a loss function that encourages output values for the original texture to be larger than for a white-noise image obtained by shuffling its pixels. At the same time the range of output values for the original texture is encouraged to be small.

The performance of the different approaches is evaluated using synthetic anomalies. However no distinction between perceptually striking and perceptually negligible anomalies is made and quantitative results are only reported for all synthetically generated anomalies.
Two attempts are made to control if an approach specifically picks up on perceptually striking anomalies.
a) detection rate on gaussian noise as a proxy for perceptually negligible anomalies
b) anecdotal evidence from visual inspection.
I do not think that either of the two controls is sufficient to make a clear statement about which method is best in detecting perceptually striking anomalies. Therefore my main concern is that the performance evaluation is not suitable to achieve meaningful results.
Furthermore the technical depth of the submission appears fairly limited. The main original contribution is the CNN loss that is introduced. However, the loss does not strike me as particularly compelling. It resembles a classifier between textures and white noise samples with the same pixel-wise statistics. I am not sure why this should be particularly suited to detect perceptual anomalies.
Finally, showing quantitative results from only two textures does not feel like a very comprehensive analysis.
In general  the submission tackles an interesting research topic. However, to show meaningful results I believe that one has to collect psychophysical data for the anomalies of interest to distinguish between anomalies that are perceptually detectable and those that are not.
With such a test set one could then start testing hypothesis on which feature representation is most appropriate to model the perceptual results or optimise features directly to match human psychophysical results (similar to the study by Berardino et al. 2017 [1]). In its current form I am not sure what I can learn from the submission both in terms of anomaly detection and feature spaces particularly suited to detect perceptual anomalies in visual textures.
[1] Eigen-distortions of hierarchical representations
A Berardino, V Laparra, J Ballé, E Simoncelli
Advances in neural information processing systems, 3530-3539