Many (but not all) approaches self-qualifying as "meta-learning" in deep learning and reinforcement learning fit a common pattern of approximating the solution to a nested optimization problem. In this paper, we give a formalization of this shared pattern, which we call GIMLI, prove its general requirements, and derive a general-purpose algorithm for implementing similar approaches. Based on this analysis and algorithm, we describe a library of our design, unnamedlib, which we share with the community to assist and enable future research into these kinds of meta-learning approaches. We end the paper by showcasing the practical applications of this framework and library through illustrative experiments and ablation studies which they facilitate.

Summary:
The authors present a PyTorch based framework for performing second-order
reverse mode autodiff for meta-learning.
First, the authors present a formalization of a general prototypical
meta-learning setting.
They then provide an algorithm that solves this problem via gradient based
optimization.
Finally, perhaps the main contribution is a specific PyTorch implementation
of said algorithm.
The type of meta-learning setting the authors consider is one where a gradient
based inner loop optimizer finds $\theta^\star$ by performing a finite number of steps.
The inner loop optimizer is parameterized through $\varphi$ that consists of
two parts $\varphi^\text{loss}$ and $\varphi^\text{opt}$.
The parameters $\varphi^\text{loss}$ are somehow part of the loss used for
training in the inner loop. Example: Regularization paramter.
The parameters $\varphi^\text{opt}$ do not occur in the loss but in the
optimizer step. Example: Learning rate.
Example of an inner loop step:
$\theta^{k+1} := \theta^k - \alpha (\nabla L(\theta^k) + \lambda \nabla R(\theta^k))$
where $\theta$ are the parameters of a neural network, $L$ is the training loss,
$R$ is the regularizer, $\alpha$ is the learning rate, $\lambda$ is the regularization parameter.
In this example we would have $\varphi = (\alpha \lambda)^T$.
The authos assume $\theta^K$, the output of the inner loop after $K$ steps to
be differentiable wrt $\varphi$.
Furthermore, the meta-learning loss is assumed to be differentiable wrt $\theta$
so that a gradient of the meta-learning loss wrt to $\varphi$ can be computed.
The authors also assume the meta-learning loss to be sufficient smooth in $\varphi$
such that a gradient based optimization can even be used for meta learning to
a local optimum.
The authors explicitly write down the reverse mode auto differentiation of
the inner loop and show how to, in that way, compute the gradient of the
meta-learning loss wrt to $\varphi$.
The reverse (adjoint) mode auto differentiation of the above example inner loop step
is the following step (iterated over in reverse down from $k = K$ to $k = 1$:
$\bar \theta^k := (I - \alpha (\nabla^2 L(\theta^k) + \lambda \nabla^2 R(\theta^k)))^T \bar \theta^{k+1}$
$\bar \alpha := \bar \alpha - (\nabla l(\theta^k) + \lambda \nabla R(\theta^k))^T \bar \theta^{k+1}$
$\bar \lambda := \bar \lambda - \alpha \nabla R(\theta^k)^T \bar \theta^{k+1}$
where $\bar \alpha$ accumulates the gradient of $\theta$ wrt $\alpha$ and
$\bar \lambda$ accumulates the gradient of $\theta$ wrt $\lambda$.
The authors give some implementation details specific to some frameworks necessary
for implementing such "gradient of an inner loop".
The authors present experiments where they show how to meta-learn learning rates
with their framework.
They also how their framework can be used to quickly implement a MAML type
meta-learning optimizer ablation study comparing various combinations of
architecture, optimizer and inner loop steps etc..
Recommendation:
I propose to reject the paper.
In my eyes the only contribution is the implementation of a meta-learner in
PyTorch based on well known methods.
The provided unifying formalization is theoretically inaccurate (see below)
and to me come across as merely a motivation for their framework
(but no value added compared to existing literature).
There is no new insight provided on the software engineering level either as
far as I can see.
Detailed comments:
- Page 1: ...provides tooling for analysing the provable requirements...
	seems like a complicated way of saying something that could be said simple
- Page 2: Without loss of generality, ...
	You are assuming a parametric model. Not sure what generality this phrase
	refers to.
- Page 2: A formalization as $\theta^\star = argmin(\theta, L(\theta, \varphi))$
	is inaccurate in the sense that it does not acknowledge the existing of
	multiple optima.
	I would recommend not to use the $argmin$ operator here, since $argmin$ for
	something like a neural network would for example either return a global
	optimum (which no optimizer used in practice finds, and is not ment here)
	or would take on a set value with multiple local minima for example.
	In the same context, the authors should mention the issues about uniqueness
	of optima (we are not even really finding optima when training neural networks),
	implicit functions / implicit differentiation
	In the context of using stochastic optimizers one should also at least
	mention something about the differentiability of outputs of such optimizers
	and how they potentially depend on randomness of mini-batches
	(what if different randomness is used with the same or a perturbed
	hyper parameter?)
- Page 3: You mention the potential statefulness of the optimizer.
	Why not explicitly carry it in the math notation?
	Probably things would get cluttered but saying it should be covered within
	$\varphi$ does not seem reasonable to me.
- Page 3: While this may seem like a fairly trivial formalization...
	Yes, but also nesting this in an outer loop is fairly trivial in the sense
	that it is a well known approach.
- Page 4: there exist continuous hyperparam...
	If they are not continuous then they should not even occur in this
	formalization so saying there exist... does not make much sense to me here
	$\alpha \subseteq \varphi^\text{opt}$ implies that $\varphi^\text{opt}$ is
	a set from notation although we are treating it as a vector everywhere else
- Page 4: All of section 2.4 seems somewhat trivial to me, but I guess that is
	highly subjective.
- Page 5: in the definition of stop operator perhaps use $:\Leftrightarrow$
- Page 5: Perhaps explicitly mention how your approach differs from a reverse
	mode differentiation of training or if it does not differ, say this.
- Page 14: When talking about _S_GD (instead of just GD) perhaps mention
	something about non-existence of mini-batch randomness / being deterministic


Deep reinforcement learning has achieved great success in many previously difficult reinforcement learning tasks, yet recent studies show that deep RL agents are also unavoidably susceptible to adversarial perturbations, similar to deep neural networks in classification tasks. Prior works mostly focus on model-free adversarial attacks and agents with discrete actions. In this work, we study the problem of continuous control agents in deep RL with adversarial attacks and propose the first two-step algorithm based on learned model dynamics. Extensive experiments on various MuJoCo domains (Cartpole, Fish, Walker, Humanoid) demonstrate that our proposed framework is much more effective and efficient than model-free attacks baselines in degrading agent performance as well as driving agents to unsafe states.

*Synopsis*:
  This paper looks at a new framework for adversarial attacks on deep reinforcement learning agents under continuous action spaces. They propose a model based approach which adds noise to either the observation or actions of the agent to push the agent to predefined target states. They then report results against several model-free/unlearned baselines on MuJoCo tasks using a policy learned through D4PG.
  Main contributions:
  - Adversarial attacks for Deep RL in continuous action spaces.
  *Review*
  The paper is well written, and has some interesting discussion/insight into attacking deep RL agents in continuous actions spaces. I think the authors are headed in the right direction, but compared to prior work in adversarial attacks for deep RL agents (i.e. the Huang and Lin) I have a few concerns that I feel the authors need to better explain/motivate in their paper. I am recommending this paper be rejected based on the following concerns. I am willing to raise my score if some of these are addressed by the authors in subsequent revisions
  1. This algorithm requires the pre-trained policy to plan attacks (which may be a high bar for such an adversarial attack). It would be a nice addition to include similar results with "black-box" adversarial attacks, as mentioned in the Huang.
  2. Another issue, addressed in the Lin paper, is this attack seems to require perturbation on every time step in a proposed trajectory. As mentioned by Lin, this is probably unrealistic and would cause the attacker to be detected. It would be another nice contribution to include variants that don't require perturbations on each transition.
  3. Another unfortunate requirement is a learned model (or a way to simulate trajectories). From the Model Based RL literature, we know learning such a model is quite difficult and often unrealistic given our current approaches. While this is problematic, I think the paper could systematically test this looking at what happens as the model becomes less accurate over time. This could provide some nice results showing an accurate model isn't necessarily needed and anneal concerns over having to learn such a model.
  4. It is unclear if the baselines measured against are meaningful in this setting, and I'm also a bit unclear how they are generated/implemented. Specifically, the random trajectories require you to return the generated trajectory with the smallest loss/reward. It is unclear how the adversary knows this information. Is it known through a model or some other simulation? Also the flip baseline could use a bit more explanation. I think these details can be safely placed in the appendix, but should appear somewhere in the final version.
  5. I'm not sure the comparison to sample efficiency to the Gleave or Uesato papers are meaningful. For Gleave, the threat model explored is much different where they do not have access to the agent's observation or action streams and instead learn policies to affect the other agent in game scenarios. This is very different. Also, the Uesato is not adversarially attacking the agent, but attempting to find failure cases for the agent, which I again feel is very different from what you are trying to accomplish. I would remove this discussion and the claim at the end of the conclusion.
  Other suggestions:
  S1. It would be helpful to include the score of the learned policy without any attacks, to see how well the baselines are performing (this will help readers understand if these are reasonable/meaningful baselines).
  S2. I'm unclear what figure three is adding to the paper, and am actually uncertain what the y-axis means. I don't think this is a wise use of the 9th page, and this plot could probably be relegated to the appendix.

  S3. As in prior work, it would be useful to see how well this line of attack works for multiple learning algorithms. Some potential candidates could be: PPO, TRPO, SAC, etc...


We introduce "Search with Amortized Value Estimates" (SAVE), an approach for combining model-free Q-learning with model-based Monte-Carlo Tree Search (MCTS). In SAVE, a learned prior over state-action values is used to guide MCTS, which estimates an improved set of state-action values. The new Q-estimates are then used in combination with real experience to update the prior. This effectively amortizes the value computation performed by MCTS, resulting in a cooperative relationship between model-free learning and model-based search. SAVE can be implemented on top of any Q-learning agent with access to a model, which we demonstrate by incorporating it into agents that perform challenging physical reasoning tasks and Atari. SAVE consistently achieves higher rewards with fewer training steps, and-in contrast to typical model-based search approaches-yields strong performance with very small search budgets. By combining real experience with information computed during search, SAVE demonstrates that it is possible to improve on both the performance of model-free learning and the computational cost of planning.

This paper proposes an approach, named SAVE, which combines model-free RL (e.g. Q-learning) with model-based search (e.g. MCTS). SAVE includes the value estimates obtained for all actions available in the root node in MCTS in the loss function that is used to train a value function. This is in contrast to closely-related approaches like Expert Iteration (as in AlphaZero etc.), which use the visit counts at the root node as a training signal, but discard the value estimates resulting from the search.
The paper provides intuitive explanations for two situations in which training signals based on visit counts, and discarding value estimates from search, may be expected to perform poorly in comparison to the new SAVE approach:
1) If a trained Q-function incorrectly recommends an action "A", but a search process subsequently corrects for this and deviates from "A", no experience for "A" will be generated, and the incorrect trained estimates of this action "A" will not be corrected.
2) In scenarios with extremely low search budgets and extremely high numbers of poor actions, a search algorithm may be unable to assign any of the visit count budget to high-quality actions, and then only continue recommending the poor actions that (by chance) happened to get visits assigned to them.
The paper empirically compares the performance of SAVE to that of Q-Learning, UCT, and PUCT (the approach used by AlphaZero), on a variety of environments. This includes some environments specifically constructed to test for the situations described above (with high numbers of poor actions and low search budgets), as well as standard environments (like some Atari games). These experiments demonstrate superior performance for SAVE, in particular in the case of extremely low search budgets.
I would qualify SAVE as a relatively simple (which is good), incremental but convincing improvement over the state of the art -- at least in the case of situations with extremely low search budgets. I am not sure what to expect of its performance, relative to PUCT-like approaches, when the search budget is increased. For me, an important contribution of the paper is that it explicitly exposes the two situations, or "failure modes", of visit-count-based methods, and SAVE provides improved performance in those situations. Even if SAVE doesn't outperform PUCT with higher search budgets (I don't know if it would?), it could still provide useful intuition for future research that might lead to better performance more generally across wider ranges of search budgets.
Primary comments / questions:
1) Some parts of the paper need more precise language. The text above Eq. 5 discusses the loss in Eq. 5, but does not explicitly reference the equation. The equation just suddenly appears there in between two blocks of text, without any explicit mention of what it contains. After Eq. 6, the paper states that "L_Q may be any variant of Q-learning, such as TD(0) or TD(lambda)". L_Q is a loss function though, whereas Q-learning, TD(0) and TD(lambda) are algorithms, they're not loss functions. I also don't think it's correct to refer to TD(0) and TD(lambda) as "variants of Q-learning". Q-learning is one specific instead of an off-policy temporal difference learning algorithm, TD(lambda) is a family of on-policy temporal difference learning algorithms, and TD(0) is a specific instead of the TD(lambda) family.
2) Why don't the experiments in Figures 2(a-c) include a tabular Q-learner? Since SAVE is, informally, a mix of MCTS and Q-learning, it would be nice to not only compare to MCTS and another MCTS+learning combo, but also standalone Q-learning.
3) The discussion of Tabular Results in 4.1 mentions that the state-value function in PUCT was learned from Monte-Carlo returns. But I think the value function of SAVE was trained using a mix of the standard Q-learning loss and the new amortization loss proposed in the paper. Wouldn't it be more natural to then train PUCT's value function using Q-learning, rather than Monte-Carlo returns?
4) Appendix B.2 mentions that UCT was not required to visit all actions before descending down the tree. I take it this means it's allowed to assign a second visit to a child of the root node, even if some other child does not yet have any visits? What Q-value estimate is used by nodes that have 0 visits? Some of the different schemes I'm aware of would involve setting them to 0, setting them optimistically, setting them pessimistically, or setting them to the average value of the parent. All of these result in different behaviours, and these differences can be especially important in the high-branching-factor / low-search-budget situations considered in this paper.
5) Closely related to the previous point; how does UCT select the action it takes in the "real" environment after completing its search? The standard approach would be to maximise the visit count, but when the search budget is low (perhaps even lower than the branching factor), this can perform very poorly. For example, if every single visit in the search budget led to a poor outcome, it might be preferable to select an unvisited action with an optimistically-initialised Q-value.
6) In 4.2, in the discussion of the Results of Figure 3 (a-c), it is implied that the blue lines depict performance for something that performs search on top of Q-learning? But in the figure it is solely labelled as "Q-learning"? So is it actually something else, or is the discussion text confusing?
7) The discussion of Results in 4.3 mentions that, due to using search, SAVE effectively sees 10 times as many transitions as model-free approaches, and that experiments were conducted on this rather complex Marble Run domain where the model-free approaches were given 10 times as many training steps to correct for this difference. Were experiments in the simpler domains also re-run with such a correction? Would SAVE still outperform model-free approaches in the more simple domains if we corrected for the differences in experience that it gets to see?
Minor Comments (did not impact my score):
- Second paragraph of Introduction discusses "100s or 1000s of model evaluations per action during training, and even upwards of a million simulations per action at test time". Writing "per action" could potentially be misunderstood by readers to refer to the number of legal actions in the root state. Maybe something like "per time step" would have less potential for confusion?
- When I started reading the paper, I was kind of expecting it was going to involve multi-player (adversarial) domains. I think this was because some of the paper's primary motivations involve perceived shortcomings in the Expert Iteration approaches as described by Anthony et al. (2017) and Silver et al. (2018), which were all evaluated in adversarial two-player games. Maybe it would be good to signal at an early point in the paper to the reader that this paper is going to be evaluated on single-agent domains.
- Figure 2 uses red and green, which is a difficult combination of colours for people with one of the most common variants of colour-blindness. It might be useful to use different colours (see https://usabilla.com/blog/how-to-design-for-color-blindness/ for guidelines, or use the "colorblind" palette in seaborn if you use seaborn for plots).
- The error bars in Figure 3 are completely opaque, and overlap a lot. Using transparant, shaded regions could be more easily readable.
- "... model-free approaches because is a combinatorial ..." in 4.2 does not read well.
- Appendix A.3 states that actions were sampled from pi = N / sum N in PUCT. It would be good to clarify whether this was only done when training, or also when evaluating.


Learning-based approaches for semantic segmentation have two inherent challenges. First, acquiring pixel-wise labels is expensive and time-consuming. Second, realistic segmentation datasets are highly unbalanced: some categories are much more abundant than others, biasing the performance to the most represented ones. In this paper, we are interested in focusing human labelling effort on a small subset of a larger pool of data, minimizing this effort while maximizing performance of a segmentation model on a hold-out set. We present a new active learning strategy for semantic segmentation based on deep reinforcement learning (RL). An agent learns a policy to select a subset of small informative image regions -opposed to entire images -to be labeled, from a pool of unlabeled data. The region selection decision is made based on predictions and uncertainties of the segmentation model being trained. Our method proposes a new modification of the deep Q-network (DQN) formulation for active learning, adapting it to the large-scale nature of semantic segmentation problems. We test the proof of concept in CamVid and provide results in the large-scale dataset Cityscapes. On Cityscapes, our deep RL region-based DQN approach requires roughly 30% less additional labeled data than our most competitive baseline to reach the same performance. Moreover, we find that our method asks for more labels of under-represented categories compared to the baselines, improving their performance and helping to mitigate class imbalance. 1 1 INTRODUCTION Semantic segmentation, the task of labelling an image pixel-by-pixel with the category it belongs to, is critical for a variety of applications such as autonomous driving (Müller et al., 2018;Wang & Pan, 2018), robot manipulation (Schwarz et al., 2018), embodied question answering (Yu et al., 2019) and biomedical image analysis (Ronneberger et al., 2015). Convolutional neural networks (Lecun et al.,  1998)-based methods have achieved excellent results on large-scale supervised semantic segmentation, in which we assume pixel-level annotations are available (Farabet et al., 2013; Pinheiro & Collobert,  2014; Long et al., 2015). For such models to work, however, they need a large amount of pixel-level annotations that may require costly human labor (Cordts et al., 2016;Bearman et al., 2016).
Current semantic segmentation datasets have pixel-wise annotations for each image. This standard approach has two important issues: (i) pixel-level labelling is extremely time consuming. For example, annotation and quality control required more than 1.5h per image (on average) on Cityscapes (Cordts et al., 2016), a popular dataset used for benchmarking semantic segmentation methods. (ii) Class imbalance in the data is typically extreme. Certain categories (such as 'building' or 'sky') can appear with two orders of magnitude more frequently than others (e.g. 'pedestrian' or 'bicycle'). This can lead to undesired biases and performance properties for learned models. This is specially relevant when we want to collect annotated data with a human in the loop to create a new dataset or to add more labeled data to an existing one. We can tackle the aforementioned problems by selecting, in an efficient and effective way, which regions of the images should be labeled next. Active learning (AL) is a well-established field that studies precisely this: selecting the most informative samples to label so that a learning algorithm will perform better with less data than a non-selective approach, such as labelling the entire collection of data. Active learning 1 Code will be available upon acceptance.

# Summary #
The paper works on active learning for semantic segmentation, aiming to annotate as few "blocks/patches" as possible while training a strong model. The authors proposed to learn a query policy via Q learning, and design states and actions specifically for segmentation. The experimental results show that the learned policy can attend to informative patches and rare classes to learn a model faster and efficiently.
# Strength #
S1. The paper is well-motivated; the references are quite sufficient.
S2. The paper clearly states the challenges when applying RL algorithms like Q-learning to image segmentation, which should serve as good guidance for other future work.
#Weakness/comments#
W1. The writing of the technical part can be strengthened.  The authors deferred the state and action design entirely to the supplementary, while they are the main contributions to the paper.
W2. The proposed algorithms seem to be highly time-consuming. The actions require pairwise comparison, and at every step, the models need to evaluate all the validation images to get the reward.
W3. If I understand correctly, the authors use part of the training data D_T, D_S (of an existing dataset) together with the validation data D_R to learn the policy, and then use the learned policy to select patches from the remaining training data D_V to train the segmentation model. I have two questions.
1) The labeled data involved in policy training is indeed quite large (validation plus part of the training, D_S + D_T + D_R). Does it mean that to learn a good active learning policy we indeed need a large number of labeled data?
2) Since (D_S, D_T, and D_R) are used to learn the policy, they should be treated as available training data for segmentation that all the compared algorithms can use without spending the budget. In other words, all the compared algorithms (U, H, B) should use those data to fine-tune a pre-train segmentation network before they start to acquire data from D_V. It would be great if the authors can clarify this.
W4. In applying the policy for selecting patches from D_V, do the authors update the model once on the selected patches, or do the authors train with them for multiple iterations together with other previous selected patches? Since deep neural nets are known to forget what has been learned (i.e., catastrophic forgetting), it's better if the author could clarify this.
W5. The authors include an upper bound in Fig. 4; however, I didn't find the explanation. Why the proposed methods can outperform the upper bound with all the training data, even with only 24% of data?
#Rebuttal#
Please discuss W1-W5.
- Annotating an entire image is definitely easier to annotators than annotating patches. Could the authors discuss how to design an active learning algorithm by selecting informative images to annotate, and maybe compare to such a method?
- Can the authors discuss Figure 3 more? As H is based on maximum entropy, why is it outperformed by the proposed method?
- There is no explicit mechanism to prevent that the k actions select similar patches. Can the authors provide more discussion?
# Post rebuttal
The authors responded to most of my concerns. I'd like the authors to incorporate all their responses into the manuscript or the appendix so that future readers can better understand the concepts and details.  I would like to raise the score to borderline (4 or 5). I modified my scores to weak accept (6) since there is no option in between.
One concern I still have is W3. 2). Given only 360 images are available, it might be inappropriate to use D_T + D_S (roughly 160 images?) to pre-train baseline models and then use D_R with "200" images for validation (early stopping). It will be more appropriate to use, for example, 70% of 360 images for training. This is supported by that many modern datasets use a much larger training set than the validation set. Therefore, I would highly suggest the authors redoing the baseline methods; otherwise, future work that re-splits the data (this is totally valid!) from the 360 images might easily achieve higher accuracy.
The authors must also reorganize the paper, taking W1 into account.


We study the problem of generating adversarial examples in a black-box setting, where we only have access to a zeroth order oracle, providing us with loss function evaluations. We employ Markov Random Fields (MRF) to exploit the structure of input data to systematically model the covariance structure of the gradients. The MRF structure in addition to Bayesian inference for the gradients facilitates one-step attacks akin to Fast Gradient Sign Method (FGSM) albeit in the blackbox setting. The resulting method uses fewer queries than the current state of the art to achieve comparable performance. In particular, in the regime of lower query budgets, we show that our method is particularly effective in terms of fewer average queries with high attack accuracy while employing one-step attacks.

This paper deals with the problem of finding an adversarial examples when only the output of a model can be evaluated, but not its gradient. The key idea of the paper is building a Gaussian MRF (a Gaussian with a sparse inverse covariance matrix with a special band structure) to maintain a model for the gradients for predicting search directions. The approach is sensible and uses the FFT trick applicable for diagonalizing covariance matrices with circulant structure.
- The ideas in this paper have practical utility. The paper is unfortunately not very carefully written, and the arguments require occasionally some guesswork.
- There is not a sufficient discussion of experimental findings. Why does the proposed method works better than a white box FGSM for instance?
- I don’t fully understand what ‘solving for GMRF’ means (Alg 1). I would expect it is estimating the parameters of the covariance function but this algorithm just calculates the likelihood.
- Given the simple coupling structure with parameter tying, this problem seems to be closely related to estimating AR(N) style models (alpha and beta parameters) so I am surprised to see only a general treatment.
In this problem, it seems much more natural to estimate theta and g recursively and concurrently, and there are very well known algorithms related to Kalman filtering. Please discuss.
- The evaluation of the idea is not complete While it is certainly interesting to see that such a simple heuristic can achieve comparable performance on standard datasets over other black-box attack methods in the limited query budget regime, I would expect to see some experiments that illustrate the quality of the gradient estimator more closely (not only its final effectiveness for finding a search direction inside of an optimization method) and more strong justification of the proposed model.
- As the entire contribution hinges on the observation that gradients across dimensions of an example as well as across examples in a dataset are correlated, it would have been also very informative to show estimates of autocorrelation functions of  the gradients on different datasets to justify the basic modelling choices.


Despite the recent breakthroughs in unsupervised domain adaptation (uDA), no prior work has studied the challenges of applying these methods in practical machine learning scenarios. In this paper, we highlight two significant bottlenecks for uDA, namely excessive centralization and poor support for distributed domain datasets. Our proposed framework, MDDA, is powered by a novel collaborator selection algorithm and an effective distributed adversarial training method, and allows for uDA methods to work in a decentralized and privacy-preserving way.

###Summary###
This paper tackles unsupervised domain adaptation in a decentralized setting.  The high-level observation is that the conventional unsupervised domain adaptation has two bottlenecks, namely excessive centralization and poor support for distributed domain datasets. The paper proposes Multi-Step Decentralized Domain Adaptation (MDDA) to transfer the knowledge learned from the source domain to the target domain without sharing the data.
The paper also explores explore a proposition: in addition to adapting from the labeled source, can uDA leverage the knowledge from other target domains, which themselves may have undergone domain adaptation in the past.
The proposed MMDA method contains a feature extractor (E), a domain discriminator (D) and task classifier (C) for each domain. The target domain components are initialized with the respective source components. The source domain discriminator D_s target domain discriminator D_t are synchronized by exchanging and averaging the gradients. The paper also proposes Lazy Synchronization to reduce the communication cost of the algorithm.
The paper also proposes Wasserstein distance guided collaborator selection schema to perform the domain adaptation task.
The paper performs experiments on five image and audio datasets: Rotated MNIST, Digits, and Office-Caltech, DomainNet and Mic2Mic.
The baselines used in this paper include "Labeled Source", "Random Collaborator", and "Multi-Collaborator". The experimental results demonstrate that the proposed method can outperform the baselines on some of the experimental settings. The paper also provides a detailed analysis of the model and experimental results.
### Novelty ###
This paper does not propose a new domain adaptation algorithm. However, the paper introduces some interesting tricks to solve the MMDA task such as the lazy synchronization between the source domain discriminator and the target domain discriminator.
###Clarity###
Several critical explanations are missing from the paper:
1) When training the source domain discriminator D_s and target domain discriminator D_t, if the features between the source domain and target domain cannot be shared with each other, how to train the D_s and D_t. For example, the D_s cannot get access to the features from the target domain, how to train D_s?
2) How is the target classifier C_t updated when there are no labels for the target domain?
3) As far as I understand, the domain discriminator is this paper is trained adversarially. The detailed adversarial training step is unclear.
###Pros###
1) The paper proposes an interesting transfer learning schema where the data between the source and target domain can not be shared with each other to protect the data-privacy.
2) The paper provides extensive experiments on multiple standard domain adaptation benchmarks, especially the most recent dataset such as the DomainNet.
3) The paper provides detail empirical analysis to demonstrate the effectiveness of the proposed methods.
###Cons###
1) The most critical issue of this paper is that some explanations are missing, e.g. how are D_s, D_t, C_t trained? Refer to the #Clarity.
2) The presentation and writing of this paper need polish. The author should do more relative surveys to motivate the authors. One critical relevant reference of this paper is:
"Secure Federated Transfer Learning", Yang Liu et al
https://arxiv.org/pdf/1812.03337.pdf
3) The baselines used in this paper is also trivial. It is desirable to compare the proposed method with state-of-the-art domain adaptation methods.
Based on the summary, cons, and pros, the current rating I am giving now is "reject". I would like to discuss the final rating with other reviewers, ACs.
To improve the rating, the author should explain the questions I proposed in the #Clarity


Using deep neural networks that are either invariant or equivariant to permutations in order to learn functions on unordered sets has become prevalent. The most popular, basic models are DeepSets (Zaheer et al., 2017) and PointNet (Qi et al., 2017). While known to be universal for approximating invariant functions, DeepSets and PointNet are not known to be universal when approximating equivariant set functions. On the other hand, several recent equivariant set architectures have been proven equivariant universal (Sannai et al., 2019;Keriven & Peyré, 2019), however these models either use layers that are not permutation equivariant (in the standard sense) and/or use higher order tensor variables which are less practical. There is, therefore, a gap in understanding the universality of popular equivariant set models versus theoretical ones. In this paper we close this gap by proving that: (i) PointNet is not equivariant universal; and (ii) adding a single linear transmission layer makes PointNet universal. We call this architecture PointNetST and argue it is the simplest permutation equivariant universal model known to date. Another consequence is that DeepSets is universal, and also PointNetSeg, a popular point cloud segmentation network (used e.g., in (Qi et al., 2017)) is universal. The key theoretical tool used to prove the above results is an explicit characterization of all permutation equivariant polynomial layers. Lastly, we provide numerical experiments validating the theoretical results and comparing different permutation equivariant models.

TLDR: The function these deep set networks can approximate is too limited to call these networks universal equivariant set networks. Authors should scope the paper to the specific function family these networks can approximate. No baseline comparison with GraphNets.
The paper proposes theoretical analysis on a set of networks that process features independently through MLPs + global aggregation operations. However, the function of interest is limited to a small family of affine equivariant transformations.
A more general function is
\begin{equation}
P(X)_i = Ax_i + \sum_{j \in N(x_i, X)} B_{(x_j, x_i)} x_j + c
\end{equation}
where $N(x_i, X)$ is the set of index of neighbors within the set $X$. It is trivial to show that this function is permutation equivariant.
Then, can the function family the authors used in the paper approximate this function? No.
Can the proposed permutation equivariant function represent all function the authors used in the paper? Yes.
1) If $B=0$, then the proposed function becomes MLP.
2) If $A=0, N(x_i, X) = [n]$ and $B_{(x_j, x_i)} \leftarrow B$, then this is $\mathbf{1}\mathbf{1}^TXB$, the global aggregation function.
Also, this is the actual function that a lot of people are interested in. Let me go over few more examples.
3) If $N(x_i, X) = $adjacency on a graph and $B_{(x_j, x_i)} \leftarrow B$, then this is a graph neural network "convolution" (it is not a convolution)
Example adjacency $N(x_i, X) = \{j \;| \; \|x_i - x_j\|_p < \delta, x_j \in X\}$.
\begin{equation}
\text{GraphOp}(X)_i = Ax_i + \sum_{j \in \{j \;| \; \|x_i - x_j\|_p < \delta, x_j \in X\}} Bx_j + c
\end{equation}
4) If $x_i = [r,g,b,u,v]$ where $[r,g,b]$ is the color, $[u,v]$ is the pixel coordinate and $N(x_i, X) =$ pixel neighbors within some kernel size, $B(x_j, x_i)$ to be the block diagonal matrix only for the first three dimensions and 0 for the rest, then this is the 2D convolution.
Again, the above function is a more general permutation equivariant function that can represent: a graph neural network layer, a convolution, MLP, global pooling and is one of the most widely used functions in the ML community, not MLP + global aggregation.
Regarding the experiment metrics and plots:
On the Knapsack test, the metric of interest is not the accuracy of individual prediction. Rather, whether the network has successfully predicted the optimal solution, or how close the prediction is to the solution.
For example: success rate within the epsilon radius of the optimal solution while satisfying all the constraints. Fail otherwise. If these networks can truly solve these problems, authors should report the success rate while varying the threshold, not individual accuracy of the items which can be arbitrarily high by violating constraints.
Also, the authors should compare with few more graphnet + transmission layer (GraphNetST) baselines with the graph layers: $P(X)_i = Ax_i + \sum_{j \in N(x_i, X)} Bx_j + c$ and the same single transmission layer $\mathbf{1}\mathbf{1}^TXB$ in PointNetST.
PointNet is a specialization of graphnets and GraphNetST should be added as a baseline with reasonable adjacency.
Also experiment figures are extremely compact. Try using log scale or other lines to make the gaps wider.

Minor
I am quite confused with the name PointNetST. Authors claim adding one layer of DeepSet layer to a PointNet becomes PointNetST, but I see this as a special DeepSet with a single transmission layer. The convention is B -> B', not A + B -> A'. In this case, A: PointNet, B: DeepSet
Lemma 3 is too trivial.
The paper is not very self contained. Spare few lines of equations to define what are DeepSets and PointNetSeg in the paper and point out the difference since these networks are used throughout the paper extensively without proper mathematical definition.
P.2 power sum multi-symmetric polynomials. "For a vector $x \in R^K$ and a multi-index ..." I think it was moved out of the next paragraph since  the same $x$ is defined again as $x \in R^n$ again in the next sentence.
Also, try using the consistent dimension for x throughout the paper, it confuses the reader.


Interpretability methods often measure the contribution of an input feature to an image classifier's decisions by heuristically removing it via e.g. blurring, adding noise, or graying out, which often produce unrealistic, out-of-samples. Instead, we propose to integrate a generative inpainter into three representative attribution methods to remove an input feature. Compared to the original counterparts, our methods (1) generate more plausible counterfactual samples under the true data generating process; (2) are more robust to hyperparameter settings; and (3) localize objects more accurately. Our findings were consistent across both ImageNet and Places365 datasets and two different pairs of classifiers and inpainters.

The paper is focused on perturbation-based local explanation methods; methods that only need black-box(ish) access to the model and generally seek to find a region, pixel, etc's importance score through by removing that region, pixel, etc.. The intuition is that an important region if removed, will result in a large drop in the predicted class confidence. One main issue with such methods is that the removal itself can throw the image out of the data distribution and therefore result in a drop in confidence, not because of the region's importance, but because of the network's unpredictable behavior in such unseen parts of the input space. The work is focused on giving a solution to this problem: instead of removal through blurring, graying, etc, use inpainting; i.e. replace the removed region with using given the rest of the image. The idea has already been discussed out in the literature and the novelty of the work seems to be twofold: They introduce the same method in a way that is not curated for a specific perturbation-based method and could be concatenated with "ANY" given (or future) perturbation-based local explanation method (which authors notate by calling it ${existing_method}-G, they study robustness to hyper-parameter choice.
The paper is quite well written and the experiments are comprehensive. I have two major comments/issues with the work:
1- The contribution of this work given existing work (more specifically the famous Chang et al work seems not to be enough for a venue like ICLR. If I want to list the contributions, it would be as follows (I would appreciate if the authors could correct me as the score is subject to change given more clarification on the matter):
    - This work utilizes an inpainting step in combination with several methods while previous work is focused on meaningful perturbations method. This, although useful, does not introduce a novel technical contribution. The main technical contribution has been the use of inpainting (to be more exact, using generative models to approximate P(c|x_r)) which has been done before on a few previous works.
    - The work argues that the use of inpainting in Chang et al (focused on keeping the salient object and removing background) was invalid as the inpainter model is not trained to do such a task. It could be argued that one could train another inpainting model that "is" capable of such a thing and therefore the general argument would not hold. One drawback of this approach, however, would be that training such an inpainting model might be difficult.
    - Hyperparameter robustness. Studying this question is valuable. However, given that the assumption of this work and previous works is that generative approaches are generally better (even not considering the hyperparameter robustness), I am not sure how this knowledge could be used.
2- Both this work and the previous works run on the assumption mentioned at the beginning of this review which basically says that non-generative perturbation-based methods throw the image out of data distribution and this is bad for such and such reasons. Although intuitively clear, I could not find any evidence in this work suggesting any meaningful difference using objective measures. One would assume that such phenomena would manifest itself clearly using insertion-deletion explanation metrics while as the authors report there was no significant difference. (Section 4.1 results clearly show a difference but this is not related to how the downstream explanation task is affected) For all it is know, generative methods have the drawback of being computationally more expensive than a simple blurring or replacing with random noise. (And a major elephant in the room is whether using an inpainter is actually taking the data back to the true data distribution which seems to be on an unproven assumption that these generative models are capable of learning the data manifold)
Minor comments:
    -Section 4.2 is really interesting. Thanks!
    - Fig 3 results: MP is more robust than MP-G and I couldn't find any explanation of why this method behaves specifically different than the other two in the experiments section. It might be better to move the explanation in the discussions to the experiments.
    - The task of most generative perturbation-based methods is to find a way to approximate P(c|x_\r) which is the conditional probability given the non-removed part of the image. Usually, they do the approximation by sampling several images from the conditional P(x_r|x_\r) (conditional inpainting)  using the generative model and averaging the prediction probability. This work seems not to be concerned with these specifics and directly feeds one of such samples. Could you explain this choice
    - For studying the robustness of LIME, apart from the random seed, couldn't one change the hyperparameters of the superpixel method? Tha one seems more of a practical problem.


Few-shot learning aims to handle previously unseen tasks using only a small amount of new training data. In preparing (or meta-training) a few-shot learner, however, massive labeled data are necessary. In the real world, unfortunately, labeled data are expensive and/or scarce. In this work, we propose a few-shot learner that can work well under the semi-supervised setting where a large portion of training data is unlabeled. Our method employs explicit task-conditioning in which unlabeled sample clustering for the current task takes place in a new projection space different from the embedding feature space. The conditioned clustering space is linearly constructed so as to quickly close the gap between the class centroids for the current task and the independent per-class reference vectors meta-trained across tasks. In a more general setting, our method introduces a concept of controlling the degree of task-conditioning for meta-learning: the amount of task-conditioning varies with the number of repetitive updates for the clustering space. Extensive simulation results based on the miniImageNet and tieredImageNet datasets show state-of-theart semi-supervised few-shot classification performance of the proposed method. Simulation results also indicate that the proposed task-adaptive clustering shows graceful degradation with a growing number of distractor samples, i.e., unlabeled samples coming from outside the candidate classes.

Summary
========
This paper tackles the more realistic variant of few-shot classification where a large portion of the available data is unlabeled, both at meta-training time in order to meta-learn a learner capable of fast adaptation, as well as meta-test time for adapting said learner to each individual held-out task.
Their approach is based on TapNet, a model which constructs a task-specific projection based on the support set of each episode, and then classifies each query according to its distance to each class prototype in the projected space. The projection is computed so that the class prototypes of the episode (averaged support examples) are well-aligned with a set of meta-learned references. Intuitively, those references learn to be far from each other, so that aligning the prototypes with them leads to a space where the episode’s classes are well separated, allowing for easier classification between them.
They then extend this model to incorporate unlabeled examples by performing task projection as follows: 1) the projection is computed so as to align the initial prototypes (computed only using labeled examples) to the meta-learned references. 2) In that projected space, each unlabeled example is assigned a predicted label based on its proximity to the projected prototypes. 3) Then, back at the original space, those predicted labels of the unlabeled examples are used to refine the class prototypes (weighted average as in Ren et al). 4) The projected space so that the *refined* prototypes are best aligned with the meta-learned references. 5) Possibly repeat 2-4 (at meta-test time).
Experimentally, they outperform recent approaches to semi-supervied few-shot classification on standard benchmarks, though not by far. Perhaps more interestingly, their performance degrades less than that of Ren et al as the distractor ratio increases, and they show that their method benefits from additional steps of task adaptation, whereas that of Ren et al reaches its performance limits after the first step of soft clustering.
High-level comments
==================
A) Ablation: An interesting ablation would be: instead of going back and forth between the embedding space and the projected space, the task adaptation happens only in the initially-computed projection space (i.e. the one computed based on the labeled data only). This would amount to: computing the projection space, and then performing a few steps of soft clustering, similar to Ren et al. in that space. This would help determine how beneficial it is to re-compute that projection space according to the current ‘best guess’ of where the class prototypes lie at each iteration. The way I see it, it is an empirical question whether the initially computed projection space already sufficiently separates the classes or not. I assume this would also lead to a more computationally efficient solution?
B) Handling distractors: In the case of distractors, they use an additional centroid (and a corresponding additional reference vector) for the purpose of ‘capturing’ the unlabeled examples that don’t belong to candidate classes. I find the initialization of this strange: this additional centroid is computed as the mean of all unlabeled examples, and the initial projection construction is influenced by a term that matches this centroid to a corresponding reference. This would mean, however, that even the unlabeled examples which do indeed belong to one of the candidate classes end up far from those classes in the projected space, in order to be close to the designated extra reference in that space. We know that this is not ideal, since we assume that some unlabeled examples do belong to the same classes as the labeled ones. Is there a way to quantify how severely this affects the quality of this initial projection? I would also be curious about the meta-learned location of the extra reference. Does it end up being roughly in the center of the references corresponding to the labeled classes?
C) Inference-only baselines. Ren et al. experimented with inference-only baselines: meta-learning happens only using the labeled subset, and the proposed clustering approach only takes place at meta-test time. In this case this would amount to meta-training a standard TapNet and then performing the proposed refinement only in test episodes. This is interesting as it allows to understand the importance of learning an embedding end-to-end for being more appropriate for unlabeled example refinement. It is not obvious that this is required, so I would be curious to see these results too. (This differs from the reported TapNet baseline in that at meta-test time it would make use of the proposed semi-supervised refinement).
Clarity / quality of presentation:
============================
D) A lot of emphasis is placed on the ability of the proposed method to control the degree of task conditioning. I would like to emphasize that this is not something that previous methods lack. Ren et al.’s approach could also perform multiple steps of clustering for example. Whether or not this is beneficial is an empirical question, but I wouldn’t say that the proposed method does something special to “control” how much adaptation is performed.
E) It would have been useful to have a separate section or subsection that explains TapNet, since this is the model that the proposed method is built upon. Instead, the authors blend the explanation of TapNet in the same section as their method which makes it hard to understand the approach and to separate the contribution of this method from TapNet.
G) The length of the paper is over the recommended pages, and I did feel that a few parts were too lengthy or repetitive (e.g. the last paragraph of the introduction described the model in detail. I felt that it would have been more appropriate to give the higher level intuition there and leave the detailed description for the next section).


We present a modern scalable reinforcement learning agent called SEED (Scalable, Efficient Deep-RL). By effectively utilizing modern accelerators, we show that it is not only possible to train on millions of frames per second but also to lower the cost of experiments compared to current methods. We achieve this with a simple architecture that features centralized inference and an optimized communication layer. SEED adopts two state of the art distributed algorithms, IMPALA/V-trace (policy gradients) and R2D2 (Q-learning), and is evaluated on Atari-57, DeepMind Lab and Google Research Football. We improve the state of the art on Football and are able to reach state of the art on Atari-57 three times faster in wall-time. For the scenarios we consider, a 40% to 80% cost reduction for running experiments is achieved. The implementation along with experiments is open-sourced so results can be reproduced and novel ideas tried out.
Github: anonymized.

This paper presents a scalable reinforcement learning training architecture which combines a number of modern engineering advances to address the inefficiencies of prior methods. The proposed architecture shows good performance on a wide variety of benchmarks from ALE to DeepMind Lab and Google Research Football. Important to the community, authors also open source their code and provide an estimate which shows that the proposed framework is cheaper to run on cloud platforms.
Pros:
1. This work is solid from the engineering perspective. It effectively addresses the problems with prior architectures and the accompanying source code is clear and well structured. It is also extensively tested on several RL benchmarks.
2. The proposed framework is especially suited for training large models as the model parameters are not transferred between actors and learners.
3. The paper is well written and organized.
Cons:
1. The gain of the main algorithmic improvement (SEED architecture) over the baseline (IMPALA architecture) is obscured by the usage of different hardware. TPUv3 has different characteristics than Nvidia P100/V100 GPU chips which also might contribute to the speed up.
Questions:
1. Is it possible to provide more “apple-to-apple” comparison by running SEED and IMPALA on the same hardware (TPUv3 or Nvidia P100/V100 GPU)?


The data stream model is a fundamental model for processing massive data sets with limited memory and fast processing time. Recently Hsu et al. (2019) incorporated machine learning techniques into the data stream model in order to learn relevant patterns in the input data. Such techniques led to the training of an oracle to predict item frequencies in the streaming model. In this paper we explore the full power of such an oracle, showing that it can be applied to a wide array of problems in data streams, sometimes resulting in the first optimal bounds for such problems, and sometimes bypassing known lower bounds without such an oracle. We apply the oracle to counting distinct elements on the difference of streams, estimating frequency moments, estimating cascaded aggregates, and estimating moments of geometric data streams. For the distinct elements problem, we bypass the known lower bound and obtain a new space-optimal algorithm. For estimating the p-th frequency moment for 0 < p < 2 we obtain the first algorithms with optimal space and update time. For estimating the p-th frequency moment for p > 2 we obtain a quadratic savings in memory, bypassing known lower bounds without an oracle. We empirically validate our results, demonstrating also our improvements in practice.

The paper presents algorithms for solving computational problems in a datastream model augmented with an oracle learned from data. The authors show that under this model, there exist algorithms that have significantly better time and space complexity than the current best known algorithms that do not use an oracle. The authors support their theoretical analysis with experiments in which the oracle is represented by a deep neural network and demonstrate improvement over classical algorithms that do not use machine learning.
Overall, this paper seems like a solid contribution to the literature. However, in its current state it does seem to be presented and motivated in a way that is appropriate for the audience of ML researchers at ICLR. It reads very much like a STOC theory paper, and a lot of the key ML details that would be relevant to audience at this conference seem to have been shoved under the rug in a way. Therefore my score for now is a weak reject, but I am very happy to increase the score if the authors address my presentations concerns.
Major comments:
* The oracle-augmented datasteam model needs to be contextualized better. I don't have a good sense of whether this is a reasonable theoretical model to explore and a lot of very basic questions remain unanswered for me. For example, how do I even know that the oracle in question exists? What are the particular assumptions under which it exists? What are the requirements on the training data, optimization ability, generalization error, etc. How do we know that we can create in practice ML learning models that are sufficiently accurate to serve as an oracle?
* The connections to deep learning seem arbitrary in some of the experiments. In one of the experiments, the authors train neural networks over a concatenation of IP address embeddings. Why do we need to use deep learning here? What is the benefit of using DL algorithms within the oracle-augmented datastream model? Is a simple algorithm enough? What algorithms should we ideally use in practice? What if you used simpler online learning algorithms with formal accuracy guarantees?
Minor comments:
* I thought there was a bit over-selling in intro. The authors say that they match the theoretical lower bounds for several problems. However, you are in a different computational model in which you now have access to an oracle. This needs to be made more explicitly, and language could be a bit toned down (e.g. in this model, we can obtain runtime that match or improve over lower bounds...)


In this paper, we propose a method named Dimensional reweighting Graph Convolutional Networks (DrGCNs), to tackle the problem of variance between dimensional information in the node representations of GCNs. We prove that DrGCNs have the effect of stabilizing the training process by connecting our problem to the theory of the mean field. However, practically, we find that the degrees DrGCNs help vary severely on different datasets. We revisit the problem and develop a new measure K to quantify the effect. This measure guides when we should use dimensional reweighting in GCNs and how much it can help. Moreover, it offers insights to explain the improvement obtained by the proposed DrGCNs. The dimensional reweighting block is light-weighted and highly flexible to be built on most of the GCN variants. Carefully designed experiments, including several fixes on duplicates, information leaks, and wrong labels of the well-known node classification benchmark datasets, demonstrate the superior performances of DrGCNs over the existing state-of-the-art approaches. Significant improvements can also be observed on a large scale industrial dataset.

This paper proposes a method, known as DrGCN, for reweighting the different dimensions of the node representations in graph convolutional networks (GCN). Specifically, the representation of every node is element-wise multiplied with a weight vector, which is parameterized as a function of the average input node representation, where the function is a two-layer neural network.
At a conceptual level, this is similar to various existing normalization schemes, such as batch normalization and weight normalization. While it is claimed in Section 4.3 that the difference is that “[batch normalization] reduces variance between samples, while DrGCN reduces variance between dimensions”, I am not sure if this characterization is accurate. Batch normalization actually makes each dimension have unit (sample) variance and so does not make the variance of each dimension small. What it does is to make the sample variance of each dimension equal, which is also what DrGCN tries to do, i.e.: reduce variance across dimensions (since samples in the case of GCNs are the representations of different nodes). DrGCN is also similar to weight normalization because like weight normalization, DrGCN learns a transformation on top of the vanilla representation (in the case of weight normalization, the vanilla representation is the normalized weight vector; in the case of DrGCN, the vanilla representation is the node representation before reweighting). The conceptual contribution therefore seems incremental.
Incremental conceptual contributions would be fine if (1) they result in a surprising theoretical result, or if (2) they result in a surprising improvement in empirical performance. Unfortunately, neither seems to be demonstrated in this paper.
In the theoretical analysis, there are various occurrences of unjustified leaps of logic; as a result, what is claimed to be shown by the analysis is different from what is actually shown, and it is unclear what is actually shown is substantially related to the proposed method. For example, in Section 3.1, the paper says that “GCNs are different from fully-connected networks only in \tilde{A}, and degrade to fully-connected networks when \tilde{A} = I. So, our analysis can be somehow be generalized to DrGCNs.” The first sentence is true; in other words, it says that fully-connected networks are a special case of GCNs when \tilde{A} = I. However, the second sentence does not follow - just showing a special case (which is what the subsequent analysis does) does not say much about the general case. Relatedly, the architecture that is analyzed is of the form H^l = W^l \phi(S^l H^{l-1}) + b^l for l = 1, …, k, whereas the architecture that is proposed is H^l = \phi(W^l S^l H^{l-1} \tilde{A}^{l} + b^{l}) for l = 1, …, k. The latter cannot be cast as the former unless \tilde{A} is diagonal. Also, the caveat of the mean field approximation are not stated - whatever result that is shown is only valid at the infinite width limit, which is different from what is claimed in the abstract, which says that "We prove that DrGCNs can reduce the variance of the node representations by connecting our problem to the theory of the mean field.” Additionally, the analysis is done in the case where S is directly parameterized, whereas the proposed method parameterizes S as the output of a two-layer neural network. I presume the reason why latter was done in practice because it worked better empirically. Because this is not explained by the analysis, the theory is incomplete, and so this caveat should be clearly stated in the abstract.
In the experimental results, the performance improvement of DrGCN over layer normalization is not statistically significant. Also, DrGCN is only compared to other normalization schemes (batch normalization and layer normalization) on one dataset, and so there is insufficient evidence to conclude that DrGCN generally works better than existing methods empirically,
Additionally, in section 2.1, it is claimed that “sampling-based GCNs still lie within the framework of equation (2), as we can set all unsampled edges to 0 in \tilde{A}”. This does not seem to be accurate, because in sample-based GCNs, different edges are sampled in every minibatch, and so there is no fixed choice of \tilde{A} that makes these GCNs equivalent to the formulation in eq. (2).
Also, if the goal is to reduce variance across dimensions (as the paper claims in Section 4.3), why was the average node representation fed into the two-layer neural network rather than its reciprocal?


The International Classification of Diseases (ICD) is a list of classification codes for the diagnoses. Automatic ICD coding is a multi-label text classification task with noisy clinical document inputs and extremely long-tailed label distribution, making it difficult to perform fine-grained classification on both frequent and zeroshot codes at the same time. In this paper, we propose a latent feature generation framework for generalized zero-shot ICD coding, where we aim to improve the prediction on codes that have no labeled data without compromising the performance on seen codes. Our framework generates semantically meaningful features for zero-shot codes by exploiting ICD code hierarchical structure and a novel cycle architecture that reconstructs the relevant keywords. To the best of our knowledge, this is the first adversarial generative model for the generalized zero-shot learning on multi-label text classification. Extensive experiments demonstrate the effectiveness of our approach. On the public MIMIC-III dataset, our methods improve the F1 score from nearly 0 to 20.91% for the zero-shot codes, and increase the AUC score by 3% (absolute improvement) from previous state of the art.

The paper proposes a method to do zero-shot ICD coding, which comes down to determining which elements from a set of natural language labels (ICD codes) apply to a given text (diagnostic summary). The main problem is that many codes have zero or very few examples. The proposed solution for this problem is to learn a feature-space generator for examples which can be conditioned on a code. This generator is trained using a GAN, which moves the few-/zero-shot problem to training the discriminator. Here, the tree structure of the ICD codes is used, by using examples with sibling labels as approximate examples for codes with few or on labelled data points. Additionally, a keyword reconstruction loss is used, based on the idea that the keywords of the corresponding ICD code can be reconstructed from a good feature vector.
The paper is written and executed well; the setup, which involves a fair number of components, is described and motivated clearly. The experiments include comparisons to state of the art results, finding that applying the GAN-based technique they choose (introduced in Xian et al., 2018 for computer vision problems) produces significant gains in recall of low-data classes (few or zero examples). The precision decreases, probably due to a shift from false negatives to false positives, while the AUC shows modest gains.
The main potential issue with the paper is the degree of (effective) novelty. The bulk of the gain relative to the SotA seems to be achievable by applying the GAN-based method, which is described in the Xian et al. (2018) reference, or by applying the label-distribution-aware margin, which are known techniques even if they have not been applied to this particular dataset yet. The additional elements introduced by the authors - the use of sibling codes and the keyword reconstruction loss - are good ideas, and it is worthwhile having a documented test of the benefit they provide, but they don’t seem to have a major influence on the quality of the model.
All in all, I would argue for the paper to be accepted. The work done here is valuable to have on record, and the presentation and execution are well done.
Two questions for the authors:
1. Could the model benefit from having a self-attention model, like a transformer? This applies mostly to the diagnostic text encoder, as interpreting the ICD code descriptions seems not to depend strongly on structure or context. From the text of the paper it appears that a 1D convolution was used to process the diagnostic texts, but I would expect that longer-distance links between words can be quite relevant there.
2. Could the precision-recall curves be added to the supplementary material?


The evaluation of robustness against adversarial manipulations of neural networks-based classifiers is mainly tested with empirical attacks as the methods for the exact computation, even when available, do not scale to large networks. We propose in this paper a new white-box adversarial attack wrt the l p -norms for p ∈ {1, 2, ∞} aiming at finding the minimal perturbation necessary to change the class of a given input. It has an intuitive geometric meaning, yields quickly high quality results, minimizes the size of the perturbation (so that it returns the robust accuracy at every threshold with a single run). It performs better or similarly to state-of-the-art attacks which are partially specialized to one l p -norm.

The authors propose a new gradient-based method (FAB) for constructing adversarial perturbations for deep neural networks. At a high level, the method repeatedly estimates the decision boundary based on the linearization of the classifier at a given point and projects to the closest "misclassified" example based on that estimation (similar to DeepFool). The authors build on this idea, proposing several improvements and evaluate their attack empirically against a variety of models.
I found the proposed method quite interesting and intuitive. All the improvements made to the core method are well-motivated and clearly explained, while the ablation experiments are relatively thorough.
However, I did find the presentation of experimental evidence quite misleading.
Specifically, reporting mean accuracy over models, datasets, and epsilon constraints in Table 2 does not give the full picture. Going through the appendix tables, we can see the following:
-- The step size used for PGD is quite large---eps/4 for the L2 case---which is quite uncommon when using 150 iterations. Based on prior work and my own personal experience, a step size of 2 * eps / #steps (i.e., eps / 75) would seem more suitable. I wonder if this is the reason for PGD performing worse than FAB for large epsilon values on CIFAR10. The authors mention that they chose this parameter using grid search but do not provide concrete details.
-- The adversarially trained MNIST model of Madry et al. 2018 learns to use thresholding filters as the first layer (observed in the original paper). This causes issues for most gradient-based methods (e.g., PGD performs worse than the decision-based attack of Brendel et al. 2018, also observed in other prior work). While it is encouraging that FAB is robust to such gradient obfuscation, this is arguably not the ideal setting to compare gradient based methods (especially when averaging performance over models).
-- For MNIST and Restricted IN, PGD performs comparably or even better than FAB (modulo larger epsilon values for which the large step size used could be an issue for PGD and the Linf-trained model with the thresholding filters).
-- For the L1-norm setting, EAD performs similarly or better compared to FAB (again modulo the Linf-trained model).
Based on these observations, I am not fully convinced that FAB outperforms PGD (for L2 and Linf) and EAD (for L1) by as much as Table 2 suggests.
Moreover, the runtime comparison performed in not exactly fair:
-- It is not clear how many restarts where included in the runtime of PGD. Its runtime should be in the same ballpark as FAB but the time reported is ~20x higher.
-- PGD is known to produce quite accurate estimates when run with much fewer (say 15) steps. Thus in order to make a fair comparison one would also need to look at the entire #steps vs robust accuracy curve to get a better picture of the efficiency of these two methods. Choosing an arbitrary number of steps for each method is not very enlightening.
-- It is not necessary to run PGD 5 times to evaluate the robust accuracy at 5 thresholds. One can perform binary search for each input in order to find the smallest epsilon for which a misclassification can be found. This will result in at most 3 (sometimes 2) evaluations per point (instead of 5).
Despite these shortcomings of the experimental evaluation, I still believe that the paper has merit. After all, the method is clean and well-motivated,  performs comparably to the best of PGD and EAD in a variety of settings, and is robust to a certain degree of gradient masking. In that sense, it could potentially be a valuable contribution and could be of interest to a subset of the adversarial ML community.
In the sense, while my initial stance is to recommend (weak) rejection, I would be open to increasing my score and recommending (weak) acceptance should my concerns be addressed.
UPDATE: I appreciate the response and the additional experiments performed by the authors. The authors have addressed my concerns in their response. I am increasing my score to a weak accept.
One thing that would be nice to add in the next version of the manuscript is a note inviting the reader to consider the appendix tables since average robust accuracy can be inconclusive.


Interactive Fiction games are text-based simulations in which an agent interacts with the world purely through natural language. They are ideal environments for studying how to extend reinforcement learning agents to meet the challenges of natural language understanding, partial observability, and action generation in combinatorially-large text-based action spaces. We present KG-A2C, an agent that builds a dynamic knowledge graph while exploring and generates actions using a template-based action space. We contend that the dual uses of the knowledge graph to reason about game state and to constrain natural language generation are the keys to scalable exploration of combinatorially large natural language actions. Results across a wide variety of IF games show that KG-A2C outperforms current IF agents despite the exponential increase in action space size.

This paper proposes a knowledge graph advantage actor critic (KG-A2C) model to allow an agent to do reinforcement learning in the interactive fiction game. Under the general framework of A2C, the core contribution of the paper is to apply a graph attention network on the knowledge graph to help learn better representation of the game state and reduce the action space. Experiments on Zork1 game environment are done to verify the effectiveness of the proposed method.
Overall, this paper presents a novel contribution to reinforcement learning with augmented memory/world-state. However, I do have a few concerns regarding the baselines and other details. Given these clarifications and or comparisons in an author response, I would be willing to increase the score.
Pros:
1, I like the idea of constructing the knowledge graph as the agent roll out. I think it is a better way to construct a structural representation of the world rather than assuming the agent gonna learn everything via single hidden state vector. It also permits more explainable policy in the future. Authors do make good progress along this line.
2, The paper is well written and the design of the proposed new model seems technically reasonable.
Cons & Questions:
1, The main concern I had is regarding to the baseline. I think it is more convincing to have a baseline which leverages the same entity extraction and template-action space. In particular, it should have the same model architecture except that it uses maybe a LSTM to decode the action rather than a GAT applied on knowledge graph. Note that the baseline I am referring to is different from the LSTM-A2C baseline reported in the paper as: (1) with entity extraction, although you may not get a graph mask, but you can still have a object-mask which also reduces the action space; (2) it is not clear to me that LSTM-A2C uses the same template-action mechanism as the KG-A2C, e.g., the valid action construction procedure described in section 4.1. Without such a baseline, it is hard to fully judge how helpful the knowledge graph is.
2, How do you test the generalization of the proposed models? In particular, do you use different maps during training and testing? If the model is merely trained on one map as shown in figure 5, it may just memorizes it in the knowledge graph and overfit to this map.
3, The details of the interaction fiction problem setup are sparse. It would be very helpful to explain what exactly the observations are in the example of Figure 2. For example, what are the game description, game feedback are in this case?
4, In the caption of figure 1, “Solid lines represent gradient flow” is misleading. If I understood correctly, solid lines refer to the computation flow which has gradient back-propagated in the backward pass.
5, Could you explain why KG-A2C converge slower than DRRN in figure 3?
6, Do you think having a fully differentiable mechanism of building knowledge graph would help or not? Why?
======================================================================================================
Most of my concerns are addressed by the authors' response. I increased my score.


We propose a novel score-based approach to learning a directed acyclic graph (DAG) from observational data. We adapt a recently proposed continuous constrained optimization formulation to allow for nonlinear relationships between variables using neural networks. This extension allows to model complex interactions while avoiding the combinatorial nature of the problem. In addition to comparing our method to existing continuous optimization methods, we provide missing empirical comparisons to nonlinear greedy search methods. On both synthetic and real-world data sets, this new method outperforms current continuous methods on most tasks, while being competitive with existing greedy search methods on important metrics for causal inference.

################## Response to rebuttal ##################
I would like to thank the authors for their diligent rebuttal. Overall, I consider that the issues raised in my review have been mostly addressed.
In summary, after reading the other reviews and going through the revised manuscript, I believe the paper would be a meaningful contribution to the field, both in terms of the potential applicability of GraN-DAG to certain problems and its potential to lead to future work. As a result, I have updated my score to support acceptance without reservations.
In particular, the authors now discuss clearly some of the potential limitations of the approach, such as the scaling of its asymptotic complexity with respect to the number of features or its statistical performance in the small sample size regime. The additional experimental results are, in my opinion, also a good addition to the manuscript, further clarifying the strengths and weaknesses of NOTEARS, CAM and GraN-DAG under different regimes and modelling assumptions. I believe these changes substantially improve the paper, as it will both help practitioners decide when to use GraN-DAG as well as clearly delineate avenues for future work.
# Minor points / additional questions
The new results regarding convergence in Table 4 suggest that early stopping might indeed be instrumental. I wonder the extent to which NOTEARS could benefit from early stopping, be it in terms of runtime or, less likely due to its use of explicit regularization, the quality of the structures it retrieves.
# Typos
Section 3.6: … GraN-DAG performs fewer iteration(s) than NOTEARS…
Section 5: GraN-DAGand DAG-GNN -was- were not designed with…
Appendix A.1: … graphs can be roughly -halfed- halved when executed…
Appendix A.5: … graphs were generated using the -BarabsiAlbert- Barabási-Albert model …
Appendix A.9: … to selecting hyperparameters in practice consist(s) in ...
################## Original review ##################
# High-level assessment
The manuscript presents an alternative to the method in [2] to extend the ideas of [1] to nonlinear DGMs using neural networks.
From a methodological perspective, in my opinion the proposed approach is sound and sufficiently novel to be a significant contribution. While some limitations might be present, such as being prone to overfitting without relying on additional (pre/post)-processing heuristics or perhaps being computationally intensive for high-dimensional data ($O(p^{3})$ runtime per SGD step), I believe that addressing those issues (if indeed present) could be left for future work.
The paper is written clearly, being easy-to-follow while providing enough detail to understand the proposed method in depth and attempt to reproduce the results.
In my opinion, perhaps the weakest aspect of the paper concerns the experimental analysis, which is somewhat limited in its scope. Most notably, I believe the current experimental setup focuses on cases likely to be favorable to the proposed approach relative to the baselines ($n > p$, nonlinear and non-additive ground-truth models) while presenting no results for other settings commonly encountered in structure learning applications, such as high-dimensional $n << p$ problems or ground-truth models given by linear or generalized linear SCMs.
Taking all those factors into consideration, I currently lean towards a “weak accept” rating, but would gladly modify my score after the author discussion phase as appropriate.
# Summary
In this paper, the authors propose a novel approach for learning the structure of Directed Graphical Models (DGMs) from observational data that allows to flexibly model nonlinear relationships between variables using neural networks.
The proposed method builds on the recent breakthrough in [1], which pioneered the use of a smooth characterization of acyclicity for directed graphs. In particular, the authors in [1] use this characterization to learn the structure of **linear** DGMs by solving a (nonconvex) continuous, constrained optimization problem.
The main contribution of this manuscript is an extension of the approach in [1] to the nonlinear setting by making use of neural networks, and which appears to outperform the only existing method that shares a similar high-level goal [2], presumably by allowing more flexibility in its parametrization.
In a nutshell, the authors accomplish this by:
1. Flexibly modelling the conditional distribution of each variable $X_{j}$ as a nonlinear function of (possibly) all other variables $\bf{X_{-j}}$ using an MLP, i.e. $p_{j}(X_{j} \mid \bf{X_{-j}}) = f_{j}(X_{j} ; NN_{j}(\bf{X_{-j}}))$, with $f_{j}$ denoting any probability density differentiable w.r.t. its parameters and $NN_{j}$ being a neural network with parameters $\theta_{j}$.
2. Deriving a function $C_{i,j}(\theta_{j})$ such that $C_{i,j}(\theta_{j}) = 0$ is a sufficient condition for the conditional distribution of $X_{j}$ to be independent of $X_{i}$.
3. Using $C_{i,j}(\theta_{j})$, which is differentiable w.r.t. $\theta_{j}$ for most values of $\theta_{j}$, to define a “connectivity” matrix to be directly plugged into the acyclicity criterion in [1].
4. Following [1], the model is then fit by (constrained) maximum likelihood using an augmented Lagrangian method. However, unlike [1], the unconstrained subproblem in each iteration of the augmented Lagrangian method is solved using stochastic gradient descent, rather than a batch quasi-Newton method, and regularized using early stopping on an external validation set, rather than by using explicit $L_{1}$ regularization to induce sparsity.
5. The authors also incorporate additional “heuristics”, such as borrowing the Preliminary Neighbour Selection (PNS) and feature selection pruning techniques from [3], which are reported to greatly enhance the performance of the proposed approach (Table 4), and adapt the thresholding scheme in [1] to the nonlinear setting.
The proposed method is evaluated using synthetic and semi-synthetic data, as well as one real-world dataset.  The results suggest that, in low dimensional problems ($n > p$, $p \le 100$) for which the ground-truth relationships between variables are nonlinear and non-additive, the proposed approach clearly outperforms [2] and also outperforms [3] in many settings, albeit often not by a statistically significant margin in the latter case.
# Major points / suggestions
1. I believe it could be argued that one of the main drawbacks of the smooth acyclicity constraint in [1] is its high computational complexity w.r.t. the number of features. However, [1] uses a batch quasi-Newton method to solve the unconstrained subproblems in their optimization routine, which should in principle lead to relatively few $O(p^3)$ matrix exponential evaluations being necessary.
As it relies on the same characterization of acyclicity as [1], this drawback might also limit the scalability of the proposed approach for high-dimensional data. Moreover, since the method in this manuscript uses mini-batch gradient descent instead, this problem could in fact be exacerbated by perhaps requiring substantially more $O(p^3)$ matrix exponential evaluations.
To this end, I would be glad if the authors could clarify whether this could indeed be a limitation or not. In any case, I believe it could also be helpful to include a small discussion regarding computational complexity for the proposed approach and related work in the manuscript or appendix.
2. All experimental results reported in the manuscript use only data generation models with nonlinear and non-additive relationships between variables.
This setup is well aligned with the contribution, as it emphasises its ability to flexibly model nonlinear relationships between variables learnt using global, gradient-based optimization. In contrast, most baselines are limited to either linear/generalized linear models or use some variant of greedy search.
While investigating this setting is certainly the main priority given the problem statement, I believe it would also be important to characterize the extent to which the performance of the proposed approach deteriorates when this extra flexibility is not needed, if at all. To this end, I would encourage the authors to repeat their experiments for synthetic data, but generated under linear and generalized linear SCMs.
In particular, it is remarkable that CAM [3], which suffers from model misspecification due assuming the data is generated by a generalized linear model, remains so competitive throughout all experiments shown in the current version of the manuscript. It would be interesting to see if the proposed approach can match or still outperform CAM under CAM’s “ideal scenario”.
3. Parallel to the previous point, all experimental results in the manuscript so far concern somewhat low-dimensional data $p \le 100$ and relatively abundant data $n \approx 1,000 > p$.
This setup might also be well aligned with the proposed approach, which could be more prone to overfitting than some of the baselines due to its high capacity and its lack of explicit sparsity-inducing regularization, which was found to boost performance for NOTEARS [1, Fig. 3(b)].
Also, in the small sample size regime, the reliance on an external validation set for early stopping might become a liability by (i) reducing the amount of data available for model fitting and (ii) making the fitted DGMs less stable w.r.t. changes in the random seed which, compounded with the nonconvexity of the objective, could be problematic if the DGM is to be interpreted for downstream tasks (e.g. in computational biology).
To this end, I would likewise encourage the authors to extend their experimental setup for synthetic data by (i) repeating their experiments for one “high-dimensional” setting ($n << p$) and (ii) studying the stability of the DAG w.r.t. different training runs.
4. All experimental results currently focus heavily on accuracy, while the computational aspect of the problem is largely secondary.
To this end, it would be interesting to study, using synthetic data, the scalability of the proposed approach relative to the best performing baseline approaches with respect to (i) number of features, (ii) sparsity of the ground-truth DGM and (iii) sample size, covering a sufficiently broad range in each case (low-dimensional vs high-dimensional, sparse vs dense, scarce vs abundant sample size).
# Minor points / suggestions
1. Perhaps the decision to permanently set the thresholding masks to 0 when an entry of the connectivity matrix is “small enough” could be justified further. For example, by showing that this heuristic rarely “zeros out” incorrectly any entries, or that such entries would seldom become nonzero again if the model was trained for longer without being masked.
2. I might have missed it, but I could not find the info about how the train/validation split is formed for the experiments following Section A.1.
# References
[1] Zheng, Xun, et al. "DAGs with no tears: Continuous optimization for structure learning." *Advances in Neural Information Processing Systems*. 2018.
[2] Yu, Yue, et al. "DAG-GNN: DAG Structure Learning with Graph Neural Networks." *International Conference on Machine Learning*. 2019.
[3] Bühlmann, Peter, Jonas Peters, and Jan Ernest. "CAM: Causal additive models, high-dimensional order search and penalized regression." *The Annals of Statistics* 42.6 (2014): 2526-2556.


Formal verification of neural networks is essential for their deployment in safetycritical areas. Many available formal verification methods have been shown to be instances of a unified Branch and Bound (BaB) formulation. We propose a novel framework for designing an effective branching strategy for BaB. Specifically, we learn a graph neural network (GNN) to imitate the strong branching heuristic behaviour. Our framework differs from previous methods for learning to branch in two main aspects. Firstly, our framework directly treats the neural network we want to verify as a graph input for the GNN. Secondly, we develop an intuitive forward and backward embedding update schedule. Empirically, our framework achieves roughly 50% reduction in both the number of branches and the time required for verification on various convolutional networks when compared to the best available hand-designed branching strategy. In addition, we show that our GNN model enjoys both horizontal and vertical transferability. Horizontally, the model trained on easy properties performs well on properties of increased difficulty levels. Vertically, the model trained on small neural networks achieves similar performance on large neural networks.

This paper proposes to use graph neural networks (GNNs) to replace the
splitting heuristic in branch and bound (BaB) based neural network verification
algorithms. The paper follows the general BaB framework by Bunel et al., but
considers only splitting ReLU neurons, not input domains. The GNN is built by
replacing each neuron in network to be verified as a vertex, and the
connections between neurons as edges. Each vertex has a feature vector combining
information like pre-activation bounds and primal/dual LP solutions.  A
specialized GNN training procedure is developed to exploit the structure of the
problem, and the weights of GNN are updated in a forward and backward manner.
Overall the paper proposes a novel idea of using GNN for accelerating
verification and it is demonstrated to be effective on one MNIST network as
well as its wider and deeper variants.  I feel the main weakness is that the
empirical evidence provided are not thorough and sufficient (only 1 base model
on 1 dataset). Since this paper is 10-page, I evaluate it at a higher standard
and expect more convincing empirical results.
Questions and suggestions for improvements:
1. How much time does it take to generate training examples? It seems to me
that it is a very costly process because obtaining the relative improvement (8)
of splitting at each node can be quite expensive - basically, we need to split
almost every ambiguous neuron to get their improvement values, and in normal
BaB we only need to split one each time. The paper mentioned it "minimum 5%
coverage per layer" but does not provide more details.
2. Also how much time does it take for training the GNN? It seems the GNN has
many vertices - the same number as the number of neurons in a network, which
can be quite large.  If the dataset generation and training time are much
longer comparing to the BaB time, the usefulness of the proposed method can be
limited especially it does not necessarily generalize to foreign networks
(networks with significantly different structure, or trained using different
methods).
3, An ablation study for the fail-safe strategy is needed. Without the
fail-safe strategy, is the GNN learned split better than other strong
heuristics? If the fail-safe strategy is too strong, the improvement we see can
probably come from the fail-safe strategy mostly, and GNN might not do too much
useful things. This is an important study that should be part of this paper.
4. It seems all networks in this work are trained using a single training
method, Wong & Kolter, 2018.  Does the split heuristic learned by GNN works for
networks trained using a different training strategy?  For example, interval
bound propagation (IBP) based methods [1][2] which achieve the state-of-the-art
results.  Also, adversarial training with L1 regularization is also verifiable,
as demonstrated in [3]. Running a few pretrained models by these methods should
be an easy experiment to add.
5. There have been a few strong baselines in this field that the authors do not
discuss and compare against, including [4][5][6]. They solve similar problems
as in this paper and also provide promising results. At least, the authors should
discuss them in related works, and it is strongly encouraged to add at least one
of them as a stronger baseline.
6. This paper claims that Neurify is theoretically incorrect (in Appendix D.2,
page 17). I am quite surprised and not sure if this claim is true. I am not
aware of any firm evidence that Neurify is theoretically incorrect.  It is
better to communicate with the authors of Wang et al., 2018 and make sure this
paper is making a correct claim.
Given that the idea proposed by this paper is novel and interesting, I tend to
accept this paper *under the condition* that the authors can conduct an
ablation study of the fail-safe strategy, provide generalization results on
models trained using different robust training strategies, and provide results
on at least one more dataset (like ACAS Xu, or CIFAR). Adding at least one more
baseline is also strongly encouraged.
** After author discussion, I have increased my score based on the new results
provided. The authors should make sure to include the ablation study results, and
a detailed discussion on training data generation time and training time in the
final version of the paper.
[1] Sven Gowal, Krishnamurthy Dvijotham, Robert Stanforth, Rudy Bunel, Chongli Qin, Jonathan Uesato, Timothy Mann, and Pushmeet Kohli. "On the effectiveness of interval bound propagation for training verifiably robust models." arXiv preprint arXiv:1810.12715 (2018).
[2] Huan Zhang, Hongge Chen, Chaowei Xiao, Bo Li, Duane Boning, and Cho-Jui Hsieh, "Towards Stable and Efficient Training of Verifiably Robust Neural Networks" (https://arxiv.org/abs/1906.06316)
[3] Xiao, K. Y., Tjeng, V., Shafiullah, N. M., & Madry, A. (2018). Training for faster adversarial robustness verification via inducing relu stability. arXiv preprint arXiv:1809.03008.
[4] Katz, Guy, et al. "The marabou framework for verification and analysis of deep neural networks." International Conference on Computer Aided Verification. Springer, Cham, 2019.
[5] Singh, G., Gehr, T., Püschel, M., & Vechev, M. (2018). Boosting Robustness Certification of Neural Networks.
[6] Anderson, G., Pailoor, S., Dillig, I., & Chaudhuri, S. (2019, June). Optimization and abstraction: a synergistic approach for analyzing neural network robustness. In Proceedings of the 40th ACM SIGPLAN Conference on Programming Language Design and Implementation (pp. 731-744). ACM.