Machine learning algorithms have been increasingly deployed in critical automated decision-making systems that directly affect human lives. When these algorithms are solely trained to minimize the training/test error, they could suffer from systematic discrimination against individuals based on their sensitive attributes, such as gender or race. Recently, there has been a surge in machine learning society to develop algorithms for fair machine learning. In particular, several adversarial learning procedures have been proposed to impose fairness. Unfortunately, these algorithms either can only impose fairness up to linear dependence between the variables, or they lack computational convergence guarantees. In this paper, we use Rényi correlation as a measure of fairness of machine learning models and develop a general training framework to impose fairness. In particular, we propose a min-max formulation which balances the accuracy and fairness when solved to optimality. For the case of discrete sensitive attributes, we suggest an iterative algorithm with theoretical convergence guarantee for solving the proposed min-max problem. Our algorithm and analysis are then specialized to fair classification and fair clustering problems. To demonstrate the performance of the proposed Rényi fair inference framework in practice, we compare it with wellknown existing methods on several benchmark datasets. Experiments indicate that the proposed method has favorable empirical performance against state-of-the-art approaches.

This paper proposes a method to use renyi correlation to improve fairness of ML models by reducing the dependence of outputs wrt sensitive inputs such as gender. Previous models either use a linear dependence measure or use more complex optimization objectives; this paper improves upon them. They first build a min-max objective where the goodness-of-fit and fairness are jointly optimized. This non-convex objective is difficult to optimize and the authors propose a reformulation of the renyi correlation for discrete random variable case where it reduces to finding second largest eigenvalue. Based on this, they reformulate the objective which can be optimized more efficiently. They show the performance of their model for supervised and unsupervised learning problems on 4 different dataset by comparing to standard correlations such as Pearson.
Overall the paper is clearly written and I liked the idea of using renyi correlation which also has a nice theoretical formulation allowing to be optimized more efficiently. But, the experimental results are a bit weak since the only model they experiment with is logistic regression. Given that their main motivation is to capture non-linear dependencies, I think some results with neural networks is necessary. Since their main focus is on discrete case, the authors can show if training a word embedding model with renyi regularization helps improve fairness of word embeddings.
I have several question regarding the paper:
The datasets that authors use have a predefined feature space. Can you show if the sensitive feature is really important for high accuracy? Can we get the same performance without the sensitive features?
Since the model is trained with gradient descent, how would a more simple baseline where the gradient of the sensitive feature is penalized work?
How would this algorithm generalize to larger problems such as language models since Q_{theta} is regenerated at every iteration?


We consider the Inverse Reinforcement Learning problem in Contextual Markov Decision Processes. In this setting, the reward, which is unknown to the agent, is a function of a static parameter referred to as the context. There is also an "expert" who knows this mapping and acts according to the optimal policy for each context. The goal of the agent is to learn the expert's mapping by observing demonstrations. We define an optimization problem for finding this mapping and show that when it is linear, the problem is convex. We present and analyze the sample complexity of three algorithms for solving this problem: the mirrored descent algorithm, evolution strategies, and the ellipsoid method. We also extend the first two methods to work with general reward functions, e.g., deep neural networks, but without the theoretical guarantees. Finally, we compare the different techniques empirically in driving simulation and a medical treatment regime.

This paper introduces a formulation for the contextual inverse reinforcement learning (COIRL) problem and proposed three algorithms for solving the proposed problem. Theoretical analysis of scalability and sample complexity are conducted for cases where both the feature function and the context-to-reward mapping function are linear. Experiments were conducted in both a simulated driving domain and a medical treatment domain to compare the three proposed algorithms empirically. Empirical results for using a deep network as the contextual mapping function is also provided.
As a special case of IRL for POMDPs, the contextual IRL problem (with latent contexts) is an interesting research topic that is of interest to researchers working on generalizing IRL to a greater range of real-world applications. This paper was written with clarity and detailed descriptions of experiments. The authors presented their algorithms with thorough theoretical analysis. However, further ablation study and proofs are needed to demonstrate the proposed problem formulation and algorithms outperform existing IRL frameworks.
The authors motivate the problem of COIRL from the formulation of contextual MDP by Hallak et al. (2015). However, the referenced work of Hallak et al. (2015) (published as a preprint on arXiv) does not provide a sufficient argument/evidence for why or in what cases this particular formulation, especially when the context is observed (as assumed by this paper), is better than alternative ones, such as directly modeling the context as part of the state. When a discrete number of contexts exist (as in the simulated driving domain: average vehicle v.s. ambulance), the proposed problem can be reduced to solving a set of normal IRL problems, i.e. learning a reward function under each observed context. For continuous context variables (such as age and weight for patients in the medical treatment domain), an intuitive solution is to model them as part of the state/feature and run normal IRL algorithms. However, the authors did not analyze or show in either of their experiments how existing IRL algorithms compare with the proposed algorithms. Without further explanation, the authors claim, in the fourth paragraph of section 1, that “..Apprenticeship Learning algorithms that try to mimic the expert cannot be used..”, yet it is not trivial to understand why the proposed formulation and algorithms would outperform existing methods/baselines.
Lastly, it would be more insightful to the readers if the authors can provide some discussion on how they would extend their algorithms to the more interesting/practical case where the context is not directly observed and analyze how latent contexts would affect the performance/complexity of their proposed algorithms.
Overall, this is a well-written paper on an exciting research topic but lacks sufficient analysis and experimental results to support the significance of the intended contributions.
Reference (from the paper):
Hallak, A., Di Castro, D., & Mannor, S. (2015). Contextual Markov decision processes. arXiv preprint arXiv:1502.02259.
==================================
Edit after rebuttal period:
- The updated version of the paper includes preliminary results of comparisons with alternative methods and shows that COIRL does have better scalability for modeling problems with large number of observable contexts. It would be great if the authors can show how the baseline performs in the two original experiments as well to understand the improvements gained from adopting COIRL for these problems.
- As stated before, I think COIRL is an exciting research topic on its own. While many recent works on IRL focuses on generalizing to complex problem with deep neural nets as function approximation tools, it is important to analyze the underlying problems from bottom up (in smaller domains and with assumptions that may not always hold) in order to better understand the limits of different solutions. I'd certainly love to see more follow-up works on this topic and how it can be extended to more complex problems.
- Hence, after carefully reading the authors responses and the updated version of the paper, I changed my decision to 'Weak Accept'.


Few-shot classification refers to learning a classifier for new classes given only a few examples. While a plethora of models have emerged to tackle it, we find the procedure and datasets that are used to assess their progress lacking. To address this limitation, we propose META-DATASET: a new benchmark for training and evaluating models that is large-scale, consists of diverse datasets, and presents more realistic tasks. We experiment with popular baselines and meta-learners on META-DATASET, along with a competitive method that we propose. We analyze performance as a function of various characteristics of test tasks and examine the models' ability to leverage diverse training sources for improving their generalization. We also propose a new set of baselines for quantifying the benefit of meta-learning in META-DATASET. Our extensive experimentation has uncovered important research challenges and we hope to inspire work in these directions.

The paper presents Meta-Dataset, a benchmark for few-shot classification that combines various image classification data sets, allows the number of classes and examples per class to vary, and considers the relationships between classes. It performs an empirical evaluation of six algorithms from the literature, k-NN, FineTune, Prototypical Networks, Matching Networks, Relation Networks, and MAML. A new approach combining Prototypical Networks and first-order MAML is shown to outperform those algorithms, but there is substantial room for improvement overall.
Many papers in the literature have observed that current algorithms achieve very high performance on existing few-shot classification data sets such as Omniglot, making it difficult to compare them. This paper fills the need for a more challenging data set. This data set is also more realistic in that
1. The training and test data may come from different data sets. In the real world, we often encounter locations, weather conditions, etc. that are unseen during training, and agents must be able to adapt quickly to these new situations.
2. Classes are selected based on their semantic structure. This allows control of the difficulty of the task, as it is easier to adapt to a semantically similar class, and thus enables a more nuanced comparison of few-shot classification algorithms.
Suggestions for clarifying the writing:
1. Section 3 should also discuss choosing the data set to sample from.
2. In figures 1c and 1d, it would be helpful to include standard error regions, like in the rest of figure 1.
3. Maybe the paragraph on Proto-MAML should be moved to section 3, as it is not from previous literature. In addition, steps 2b and 2c in section 3.2 overlap in content, and so it may be clearer to combine them.


Randomized classifiers have been shown to provide a promising approach for achieving certified robustness against adversarial attacks in deep learning. However, most existing methods only leverage Gaussian smoothing noise and only work for 2 perturbation. We propose a general framework of adversarial certification with non-Gaussian noise and for more general types of attacks, from a unified variational optimization perspective. Our new framework allows us to identify a key trade-off between accuracy and robustness via designing smoothing distributions, helping to design two new families of non-Gaussian smoothing distributions that work more efficiently for 2 and ∞ attacks, respectively. Our proposed methods achieve better results than previous works and provide a new perspective on randomized smoothing certification.

Summary:
This paper investigates the choice of noise distributions for smoothing an arbitrary classifier for defending against adversarial attacks.  The paper focuses on the two major adversaries: \ell_2 adversaries and \ell_\infty adversaries. Theorem 1 quantifies the tradeoff between the choice of smoothing distribution which (1) has clean accuracy close to the original classifier and (2) promotes the smoothness of smoothed classifier (and hence adversarial accuracy).  For the \ell_2 adversary, the paper argues that Gaussian distribution is not the right choice, because the distribution is concentrated on the spherical shell around the x. Instead, the authors propose using a new family of distributions, with the norm square  (p_{|z|_2^2}) following the scaled \chi^2 distribution with degree d-k (Eq. 8). This allows an extra degree of freedom, and setting k=0 recovers the Gaussian distribution. For \ell_\infty perturbations, the paper suggests another family of distributions combining the \ell_2 and \ell_\infty norm (Eq. 9), and argues that it outperforms the natural choice of \ell_\infty norm-based distributions (Eq. 10).
I think the paper should be rejected because (1) For \ell_2 perturbations, there is no major difference between this new family of distributions (d-k \chi^2) and a Gaussian with different variance. (2) For \ell_\infty distributions, the motivation of mixed norm distributions (Eq. 9) over \ell_\infty based distributions (Eq. 10) is not very clear. (3) The experimental evidence is also weak (see below).
Main arguments:
1. The distribution of the norm \|z\|_2 in Eq. (8) would be concentrated on a thin spherical shell of radius about \sqrt{d-k}\sigma. As the Gaussian distribution with standard deviation \sigma' is supported on a shell of radius about \sqrt{d} \sigma', for each (k,\sigma) in the family of Eq. 8, there is an equivalent Gaussian with appropriate \sigma' (Theorem 3 now just compares the radius of the spherical shell). Therefore, I don't see the benefit of this extra degree of freedom of k: the noise distribution is again a "soap bubble" of a different radius. Thus, a grid search over \sigma' for a Gaussian should be the same as a grid search over (k,\sigma) in Eq. 8.
Even the experimental experiments are a marginal improvement over Cohen et al.  I don't see why the value of (k,\sigma) was not provided in Table 1 and only \sigma was provided. Also, the table of Cohen et al. was only calculated for specific values of \sigma for Gaussian distributions (0.12, 0.25, 0.5, 1.00). For a fair comparison, comparable values of \sigma's must be calculated, and then the best choice should be selected.
2. In the light of previous arguments, I don't think the choice of Eq. (9) or Eq. (10) is well motivated.
Why not smooth it with a cube of appropriate radius? Also, not enough experimental details are provided for Table 3.  Salman et al. (2019) reports the accuracy of 68.2% for \ell_infty perturbations (Table 3, Salman et al. (2019)), whereas the value reported in your Table 3 for at the same radius is 58%. Is it a typo? In any case, the values reported for the proposed model in Table 3 are only a marginal improvement over Figure 1 (left) in Salman et al. (2019), just going by the trivial \ell_2 to \ell_\infty certificate.
Other areas for improvement:
1. The paper contains numerous grammatical errors, confusing statements, and nonstandard phrases.  For example: (i) more less robust, (ii) black start, (iii) pointy points, etc.  I suggest that the authors spend more time clarifying their manuscript.
2. The paragraph starting with "Trade-off between Accuracy and Robustness": I think this paragraph should be reworded for clarification.  It is not robustness but rather the lack thereof -- say, sensitivity.
3. On p.5, why was the toy classifier sphere-based? The toughest classifier for Gaussian smoothing (the one achieving the lower bound for Gaussian smoothing) is actually a linear classifier.


Few-shot classification is the task of predicting the category of an example from a set of few labeled examples. The number of labeled examples per category is called the number of shots (or shot number). Recent works tackle this task through metalearning, where a meta-learner extracts information from observed tasks during meta-training to quickly adapt to new tasks during meta-testing. In this formulation, the number of shots exploited during meta-training has an impact on the recognition performance at meta-test time. Generally, the shot number used in meta-training should match the one used in meta-testing to obtain the best performance. We introduce a theoretical analysis of the impact of the shot number on Prototypical Networks, a state-of-the-art few-shot classification method. From our analysis, we propose a simple method that is robust to the choice of shot number used during meta-training, which is a crucial hyperparameter. The performance of our model trained for an arbitrary meta-training shot number shows great performance for different values of meta-testing shot numbers. We experimentally demonstrate our approach on different few-shot classification benchmarks.

A Theoretical Analysis of the Number of Shots in Few-Shot Learning
This paper considers the problem of meta-learning a representation that is suitable for few-shot classification using the distance from class centroids. In particular, it investigates the sensitivity of this approach to the number of shots from the perspective of generalization error. This seems to be a valuable and novel analysis of the problem, and the problem is important since the number of shots might not be known a-priori. The paper proposes a transformation (a dimensionality-reducing linear projection) that is based on the covariance of the within-class and between-class variance in the training set. It is shown that this makes the procedure relatively robust to the number of shots during training. The technique is compared to an informative set of baselines: PCA, adding a fully-connected layer and mixed-shot training.
I liked the explanation for the harmfulness of training and testing with different shots: with few shots, the priority is to minimize the within-class variance, whereas with many shots, it's OK to have some examples that are further from the true mean as their effect on the empirical mean will be mitigated. Furthermore, the hypothesis that the ratio of within-class to between-class variance depends on the number of shots was empirically verified on several datasets. This was great to see. The paper additionally argues that, with fewer shots, the meta-learner in Prototypical Networks might reduce the intrinsic dimension of the embedding to improve generalization error (by effectively reducing the VC dimension). This was verified experimentally by examining the dimensionality of a subspace that approximately represents the embedding vectors.
Unfortunately, I had difficulty following the theoretical analysis. Admittedly, I don't often work with generalization bounds. Nevertheless, to make the paper accessible to a wider audience, I believe it's necessary to improve the clarity (see points below). I spent quite a lot of time trying to understand the proof of Lemma 1 and I did not have time to closely assess the remaining proofs. I have given the benefit of the doubt for now, but I might have to downgrade my rating depending on the response of the authors and the feedback of other commenters.
High-level concerns:
(1.1) The approach closely resembles (multi-class) Linear Discriminant Analysis, yet this connection was not discussed in the paper.
(1.2) The improvement of EST-ProtoNet over PCA-ProtoNet is often marginal in Table 1. It would have been better to provide an estimate of the variance over several trials. Nevertheless, it's an interesting result that simply applying PCA improves the robustness of ProtoNet to the training shots.
(1.3) I wonder whether whitening would be more effective than dimensionality reduction? This could also avoid the need to specify $d$.  To be clear, by "whitening" I mean incorporating a factor of $\Lambda^{-1/2}$ into the transform to make the covariance matrix equal to identity.
Issues with mathematical clarity:
(2.1) I could not figure out how equation 5 follows from Chebyshev's inequality. I understand that Chebyshev's inequality establishes a bound on the likelihood of a sample being some distance from the mean, and this bound depends on the variance. I do not see how this can be used to bound $\operatorname{Pr}(\alpha > 0)$. I also investigated Markov's inequality (sometimes referred to as Chebyshev's inequality) and Cantelli's inequality (sometimes referred to as the one-sided Chebyshev's inequality), but I could not see how either of these could be applied. Please clarify.
(2.2) The notation of vector inner and outer products did not seem to be consistent throughout the paper (i.e. whether vectors are treated as a row or a column). For example, in Lemma 1, it seems that $x y^T$ denotes an inner product, since $\alpha$ is a scalar. Then in equation 12, it seems that $x^T y$ denotes an inner product. This is particularly bad in the appendix. The multi-line equation at the end of page 12 seems to mix the two notations. If I have misunderstood, please correct me. I feel that the column-vector notation is more widespread ($x^T y$ for inner product and $x y^T$ for outer product), but the important thing is to be explicit and consistent.
(2.3) I found the proof of Lemma 1 difficult to follow. Maybe I am being slow, but it would be helpful to explain the steps more clearly. (Side note: for the purpose of discussion, it would have been good to enable equation numbering here.) I did not understand the step from the 1st to the 2nd line of "i = …". I also could not follow the step from the 2nd to the 3rd line of the equation for $\Sigma_{\phi(x) - \bar{\phi}(S_b)}$. Nevertheless, I feel that the "1/k" term feels plausible, because as the number of sample increases, the estimate of the mean will be more accurate. I also could not follow the step from the 2nd to the 3rd line of the equation for $\mathbb{E}_{a, b, x, S}[\alpha]$. Please clarify these points or I may need to reduce my score.
Minor errors:
(3.1) Equation 2 should probably be "arg max p(...) = y" rather than just "p(...) = y"?
(3.2) In the definition of \Sigma_c, it is not clear what it means to square a vector. I think this should be written as a vector outer product (x x^T) or at least add a note to explain the notation.
(3.3) There might be some small errors in appendix A.5 that don't affect the outcome:
- Shouldn't the expression for (ii) include a term which depends on $\mu_a - \mu_b$, like the expression for (i)?
- I wondered whether the expression for $\mathbb{E}_{x,S|a,b}[\phi(x) - \bar{\phi}(S_b)]$ should be $0.5 (\mu_a - \mu_b)$, since $x$ has a 0.5 chance of being from class $a$ and 0.5 chance of being from class $b$?
(3.4) In the statement of Theorem 4 in the appendix, there should be something after "Var"?


We explore a new domain of learning to infer user interface attributes that helps developers automate the process of user interface implementation. Concretely, given an input image created by a designer, we learn to infer its implementation which when rendered, looks visually the same as the input image. To achieve this, we take a black box rendering engine and a set of attributes it supports (e.g., colors, border radius, shadow or text properties), use it to generate a suitable synthetic training dataset, and then train specialized neural models to predict each of the attribute values. To improve pixel-level accuracy, we also use imitation learning to train a neural policy that refines the predicted attribute values by learning to compute the similarity of the original and rendered images in their attribute space, rather than based on the difference of pixel values. We instantiate our approach to the task of inferring Android Button attribute values and achieve 92.5% accuracy on a dataset consisting of real-world Google Play Store applications.

Authors proposed an algorithm to predict the attribute of GUI elements from rasterized design images. The problem is separated into two steps. The first step is to predict initial values of the attributes (border width, color, padding etc) from the image where the type of UI element and set of attributes are already known. Authors designed a typical convolutional DNN for each of the attributes. The second step is to learn a policy \pi to iteratively adjust one attribute a time until the final rendering matches input image pixel-perfect.
Authors conducted the experiment with large synthetic data set of Android buttons, and evaluate the performance with a held-out synthetic set as well as a 110 hand crafted real world buttons set sampled from apps in Google Play App Store. Several variations of the same model were compared. The result showed that network structure, padding strategy (this is a bit unexpected), background type and color selection strategy all affect the accuracy significantly.
Reviewer has concern about the scope and application value of the problem as a research paper. A number of key prior assumptions have to be made to let the algorithm work: the type of the UI element need to be known; the list of attributes and their value ranges need to be fixed beforehand and each of the attribute demands a DNN; the refinement iteration has the actual rendering in the loop which could be costly on current App development platforms.
Feedback questions.
1) The abstract mentioned vector image as input but the main body only discussed rasterized images.
2) Since the rendering process could be costly, it's useful to discuss the speed of convergence in the attribute value adjustment iterations.
3) Reviewer is interested in the nature of the error (7.5% loss) but it's not discussed.
4) In related work, authors mentioned the REINFORCE algorithm by Williams et al 1992 is expensive. It could help the reader if a brief explanation of why it's expensive is provided.
5) In Section 3 Background, authors mentioned that the attributes are not independent of each other, which is a major challenge. Reviewer would like to see some discussion or experiment data on how this affects the process and how did the current algorithm address it.
6) It's a bit surprise that color clipping method has a big impact on accuracy. Some examples could have helped the reviewer understand it.
7) In Section 4.2 first paragraph, it seems that the user of the algorithm need to set the [-c, c] clipping values manually per feature. This sounds like quite some prior knowledge and hand-tuning.
8) In Section 4.2. Second paragraph, Reviewer can understand the nessesity of additive and subtractive operations, but why multiplication?
9) In the equation to the end of page 5, do we need an extra outer bracket for the denominator? By the way, the equations should be numbered for easier reference.
10) The task of predicting Android button attributes, while practical, seems over-simplified. Reviewer suggests at least experiment with a set of common UI elements to proof the horizontal performance.
11) In Section 5.1, Reviewer respect the experiment results but doesn't understand why solid color background provides the best variety but screenshots don't. May need more analysis and explanation.
12) In Table 1, the first line for variant (C) also looks pretty good, or even better than core on the Android app store dataset.
13) In Section 5.1, the effect of color clipping selection seems very specific to applications with a fixed color palette. While this is indeed the majority, this prior knowledge need to be speciifed clearly by saying it's tailored towards such applications (or use more examples to proof that's not the case).
14) In Table 2: Pixel Sim's performance on Best Prediction Initialization seems pretty good, and Reviewer believes this is the more practical scenario. Is a more complicated Siamese Network justified?


Numerical common sense (e.g., "a person with a height of 2m is very tall") is essential when deploying artificial intelligence (AI) systems in society. To predict ranges of small and large values for a given target noun and unit, previous studies have implemented a rule-based method that processed numeric values appearing in a natural language by using template matching. To obtain numerical knowledge, crawled textual data from web pages are frequently used as the input in the above method. Although this is an important task, few studies have addressed the availability of numerical common sense extracted from corresponding textual information. To this end, we first used a crowdsourcing service to obtain sufficient data for a subjective agreement on numerical common sense. Second, to examine whether common sense is attributed to current word embedding, we examined the performance of a regressor trained on the obtained data. In comparison with humans, the performance of an automatic relevance determination regression model was good, particularly when the unit was yen (a maximum correlation coefficient of 0.57). Although all the regression approach with word embedding does not predict values with high correlation coefficients, this word-embedding method could potentially contribute to construct numerical common sense for AI deployment.

This paper attempts to study if learned word embeddings for common objects contain information about "numerical common sense". The hypothesis is that certain numerical information may co-occur with the words for certain objects/measurement units within their context windows. To verify this hypotheses, the authors have created a dataset through a crowd-sourcing service which represents "numerical common sense". Using this dataset, the authors examine the predict abilities of regressors trained on learned word embeddings and the aforementioned crowd-sourced dataset. The hypothesis is that if the regressors demonstrate good accuracy, then the word embeddings contained information relevant to "numerical common sense". To the best of my knowledge, this is the first paper that attempts to analyze learned word embeddings in the context of numerical common sense.
This paper should be rejected because (1) the NCS datasets are too small to represent "numerical common sense" (2) the NCS datasets contain faulty data points and (3) the results from the experiments conducted are not sufficient to accept or refute the hypotheses.
Main argument
The first question that we must ask is - are the NCS-50x1 and NCS-60x3 datasets reliable for experiments on "numerical common sense". No, because of two flaws:
(1) The number of samples in the dataset is too small to represent "numerical common sense". Consider the histogram for object "dog" in Figure 2. If the largest data point in this plot was absent, the average of the distribution would be smaller by several orders of magnitude. Perhaps there are other objects in the dataset which are missing samples from the tail end of the distribution that could have large effects on the mean of the collected dataset.
(2) Some data points in the dataset don't make sense to me. For example, Fig 2 represents the "small" dataset, yet I see samples like 400m long dogs, 40m long cats, 150m long monitors and 20m long mice?
Also, it is not clear how the confidence scores of the participants were taken into account when training the regressors or if they were used at all.
If the NCS dataset does not represent "numerical common sense", it invalidates all experimental results from the paper.
My second issue with the paper is that it is not possible to conclude if the experimental results support or refute the hypothesis (ignoring the issue with the dataset):
1. In tables 2 and 3, the correlation coefficients were quite low and and the MAEs were pretty large. In Table 3, rows 1 and 2, even though the correlation is 0.57 and 0.48, the MAE is 100 million yen and 7.5 million yen respectively which is quite large. To me this suggests that just because the correlation is larger we cannot conclude mean that the model is performing well.
2. It is unclear why the correlation coefficient was chosen to decide that ARD is the superior model in experiment 1. The MAE for random forests with concatenated feature vectors was an order of magnitude smaller than that of the ARD model.
3. Why are the correlation coefficients missing for the unit-only experiment in Table 2? The LS model shows very good MAE relative to the other models and perhaps the correlation should have been measured for that as well? In fact, if the correlation coefficients for this case is comparable to the case with concatenated features, it would mean that the word embedding for the object is not helping at all! Moreover, I find it surprising that the LS model with concatenated features performs worse than the unit-only features. We cannot conclude if paper's interpretation about the results is correct unless this missing information is provided.
4. It is hard to judge what a correlation coefficient of 0.57 means. Why didn't you provide a scatter plot of the predictions vs targets as well? It often happens that even noisy plots demonstrate good correlations.
5. The paper should have additional ablation studies - for example, what would happen in the concatenated feature vector experiment if you trained the regressors using randomly initialized word embeddings instead of the trained word embeddings? Do you get the same performance as learned word embeddings?


Policy gradient algorithms in reinforcement learning optimize the policy directly and rely on efficiently sampling an environment. However, while most sampling procedures are based on sampling the agent's policy directly, self-performance measures could be used to improve sampling before each policy update. Following this line of thoughts, we propose SAUNA, a method where transitions are rejected from the gradient updates if they do not meet a particular criterion, and kept otherwise. The criterion, V ex , is the fraction of variance explained by the value function V : a measure of the discrepancy between V and the returns. In this work, V ex is used to estimate the impact transitions will have on learning: it refines sampling by simplifying the underlying state space and improves policy gradient methods. In this paper: (a) We introduce V ex , the criterion used for denoising policy gradient updates. We refer to this procedure as transition dropout and explore its implications on learning. (b) We conduct experiments across a variety of benchmark environments, including continuous control problems. SAUNA clearly improves performance. (c) We investigate how V ex reliably selects samples with most positive impact on learning. (d) We study how this criterion can work as a dynamic tool to adjust the ratio between exploration and exploitation.

In their post-review revision, the authors have added a much clearer motivation for SAUNA, along with extra experiments that validate and clarify the approach.
The revision is a significantly better paper than the original, and I am updating my score accordingly.
-------------
This paper introduces 'the fraction of variance explained' V^{ex} as a measure of the ability of the value function to fit the true reward. Given the mean per-timestep reward as a baseline, V^{ex} measures the proportion of the variance in reward that is captured by the value function.
In this paper, the authors introduce a filtering condition aimed to ensure that no single transition excessively reduces the fraction of variance explained. This filtering condition relies on a prediction of the variance explained, which comes from an extra head added to the standard PPO architectures, and its parameters are updated along with all other model parameters at the end of each trajectory.
Intuitively, I can believe that learning will be more stable under the condition that no single transition leads to too great a divergence between the predicted reward and true reward. However, I do not understand the authors' assertion that this filtering procedure removes noisy samples (how can we characterize these samples as noise?). I'm also insufficiently familiar with the related literature to properly gauge the theoretical implications of this modification (see low confidence below).
The paper compares learning accuracy over time of PPO with and without the variance explained filtering. It seems that the filtering does improve learning for the MuJoCo environments, as well as low resource models for a harder task from the Roboschool environment but it is hard to state definitively that the new method is better. I commend the authors on their discussion of non-positive Atari results in the appendix and I agree that it contributes significantly to the paper.
With the caveat that this paper is quite far from the realm of my expertise. I think that the approach is intriguing but, in the absence of any theoretical justification for the approach, I'm not sure that the empirical results are sufficiently convincing for ICLR. I also believe that the paper would be easier to understand with a more thorough investigation of the effect of the filtering on the learning procedure.
Questions for the authors:
- what proportion of samples are rejected
- how does this proportion change over the course of learning
- how was the V^{ex} threshold of 0.3 chosen


Adversarial patch attacks are one of the most practical threat models against realworld computer vision systems. This paper studies the certified and empirical performance of defenses against patch attacks. We begin with a set of experiments showing that most existing defenses, which work by pre-processing input images to mitigate adversarial noise, are easily broken by simple white-box adversaries. Motivated by this finding, we present an approach for certified defense against patch attacks, and propose methods for fast training of these models. Finally, we experiment with different patch shapes for testing, and observe that robustness transfers across shapes surprisingly well.

The paper proposes a certified defense for adversarial patch attacks.
Technically, the authors use the well developed IBP based methods (Gowal et
al., 2018, Mirman et al., 2018, Zhang et al. 2019).  The technique is simple
but effective. Since the number of possible patches are quadratic w.r.t image
dimension, to reduce the number of bounds to propagate, the authors propose a
U-Net based NN to predict the worst case scenario, and only propagate "worst
case" bounds predicted by the U-net.
Empirically, the proposed method gets good results, with certified accuracy
sometimes even higher than empirical accuracy by previous methods.  The authors
also provide results for transferring robustness properties to shapes that are
not included during training.
Overall, the contribution of this paper is novel, and results are promising,
but it still has some missing components, especially the idea of combining
multiple IBP bounds into one, which can be very effective for adversarial
patches, as I will elaborate below.
Suggestions and Questions:
The core idea behind IBP is that for whatever input perturbation is given (any
Lp norm or semi-norm, or non-norm based perturbations like patches at arbitrary
locations), it converts them to per-neuron lower and upper bounds after the
first linear/conv layer.  For example, if the input perturbation is *two* patches
B_1 and B_2, after propagating them through the first layer of the network, we
got two lower bounds l_{i,1}, l_{i,2} and two upper bounds u_{i,1}, u_{i,2} for
the i-th neuron. We then take the worst case bound, l_i = min(l_{i,1},
l_{i,2}), u_i = max(u_{i,1}, u_{i,2}) and propagate only one set of bounds l_i
and u_i to the next layer. The authors should explore on this direction, as
detailed below:
1. For the exhaustive patch enumeration in (11), we can actually greatly reduce
the computation cost by combining the bounds of different patches after the
first layer of the network, as I mentioned above. At the input layer, the
number of bounds (each for one possible location of patch) are large; but after
the first linear/conv layer, we can compress them to one or a small group of
bounds by taking the worst cast of them, like l_i = min(l_{i,1}, ...,
l_{i,|L|}), u_i = max(u_{i,1}, ..., u_{i,2}). The patches close with each other
should also produces similar lower and upper bounds, so taking min or max over
them will not make the combined bounds much worse. This is better than U-net
prediction since we are guaranteed to include the worst case scenario.
2. Considering multiple patches (at different locations) on a single input. In
the simplest case, consider multiple 1x1 patches (in fact, this is equivalent
to bounded L0 norm threat model); since each patch is 1x1 (only changing one
pixel), the bounds should be relatively tight after the first layer, and after
the first linear/conv layer we got 28*28 or 32*32 bounds which will be combined
into one or very few sets of lower and upper bounds that will be propagate into
later layers.  Multiple larger patches (2x2, 5x5) can be difficult since bounds
are looser; multiple 1x1 in my opinion is both technically feasible and
practically important, and should definitely be included in this paper.
Minor issues:
1. Eq. (5) and (6) are incorrect; the second term should be \overline{z} -
\underline{z}. Also it is missing the bias term.
2. In related works (section 4.1, page 3), (Weng et al., 2018) is not a defense
method (it is a certification method, and no training is involved).
3. In Table 3, it is better if the authors can provide empirical adversarial
accuracy to IBP defended networks as well.
Overall, I think it is a good paper but the authors should explore more to
strengthen their contributions. I gave a weak reject but I will not hesitate
to recommend an accept as long as the authors can provide additional results
mentioned above.


Answering compositional questions that require multiple steps of reasoning against text is challenging, especially when they involve discrete, symbolic operations. Neural module networks (NMNs) learn to parse such questions as executable programs composed of learnable modules, performing well on synthetic visual QA domains. However, we find that it is challenging to learn these models for non-synthetic questions on open-domain text, where a model needs to deal with the diversity of natural language and perform a broader range of reasoning. We extend NMNs by: (a) introducing modules that reason over a paragraph of text, performing symbolic reasoning (such as arithmetic, sorting, counting) over numbers and dates in a probabilistic and differentiable manner; and (b) proposing an unsupervised auxiliary loss to help extract arguments associated with the events in text. Additionally, we show that a limited amount of heuristically-obtained question program and intermediate module output supervision provides sufficient inductive bias for accurate learning. Our proposed model significantly outperforms state-of-the-art models on a subset of the DROP dataset that poses a variety of reasoning challenges that are covered by our modules.

This works applies neural module network to reading comprehension that requires symbolic reasoning. There are two main contributions: (1) the authors designed a set of differentiable neural modules for different operations (for example, arithmetics, sorting, and counting) that is required to perform reasoning over a paragraph of text. These modules can be compositionally combined to perform complex reasoning. And the parameters of each module (which can be viewed as executor of each operation) are learned jointly with the parser that generates thee program composed of those modules. (2) To overcome the challenge of weak supervision, the authors proposed to use auxiliary loss (information extraction loss, parser supervision, intermediate output supervision). The model is evaluated on a subset of DROP, and outperforms the state-of-the-art models. Ablation studies supported the importance of the auxiliary losses.
Strength:
(1) The problem of applying symbolic reasoning over text is important and very challenging. This work has explored a promising direction that applies NMN, which achieved good results in VQA, to QA tasks that requires reasoning, specifically, a subset of the DROP dataset.
(2) The result, although preliminary, seems promising. The design of the modules seems intuitive and the introduction of auxiliary tasks to alleviate the problem of weak supervision is well motivated and works reasonably well.
I am leaning towards rejection because:
(1) The main concern is that the paper, in its current form, seems incomplete. It is understandable that the type of datasets that requires reasoning is not very common nowadays, so only DROP is used for evaluation. However, the current evaluation is only on a subset of DROP, which seems unsatisfying.
The paper argues that "Our model possesses diverse but limited reasoning capability; hence, we try to automatically extract questions in the scope of our model based on their first n-gram". However, results on the full dataset seems necessary for evaluating the potential of NMN approach over text. Even if the result is negative, it is still good to know the cause of the failure. For example, does the difficulty come from unstable training or does it come from insufficient coverage of the modules.
(2) There are several modules introduced in the paper, but there isn't much analysis of them during the experiments. For example, what are some good and bad samples that uses each type of operations.
(3) Since the modules are learned jointly with the parser, it is good to check whether the learned modules are indeed performing the intended operation instead of just adding more capacity to the model. For example, it might help to show a few examples that demonstrates the "compare-num-lt" is actually performing the comparisons. This can support the interpretability claim of the proposed model.
Minor issues:
The complexities of some modules seem large. For example, "compare-num-lt" needs to enumerate all the pairs of numbers, which is quadratic. And the complexity of "find-max-num" depends on the choice of n, which could be large (although it is chosen to be 3 in this work).
It is stated that "Our model performs significantly better than the baseline with less training data, showing the efficacy of explicitly modeling compositionality." However, the comparison with MTMSN using less training data seems a bit unfair since the proposed model is given more supervision (question parse supervision and intermediate module output supervision). Maybe a better argument is that by explicitly modeling compositionality, it is easier to add such extra supervisions than black box models like MTMSN.
For "count", why is the attention scaled using values [1, 2, 5, 10] first?
In summary, I do like the main idea and the paper has merits, but it requires more evaluation and analysis to be accepted. I am willing to increase my score if more contents are added and I look forward to seeing it in a more complete form.
===================================
Update after author response:
Thanks for the clarification and adding the content, I have updated my score accordingly. However, I still believe the impact of this paper will be much larger if the evaluation can be more complete, e.g., evaluating over the full DROP dataset or even include some other datasets. In the current form, it looks borderline.
Selecting a subset (~22.7% of DROP dataset) based on the design of the proposed model ("heuristically chosen based on their first n-gram such that they are covered by our designed modules"), and compare to other models, which can actually handle a broader set of questions, only on the selected subset seems incomplete and raises concerns about how generally applicable the proposed model is. For example, since the proposed model is handling some types of questions better, it would be good to show that it can be combined with other models to get a better overall result.


Adversarial attacks such as Gradient-based attacks, Fast Gradient Sign Method (FGSM) by Goodfellow et al. (2015) and DeepFool by Moosavi-Dezfooli et al. (2016) are input-dependent, small pixel-wise perturbations of images which fool state of the art neural networks into misclassifying images but are unlikely to fool any human. On the other hand a universal adversarial attack is an input-agnostic perturbation. The same perturbation is applied to all inputs and yet the neural network is fooled on a large fraction of the inputs. In this paper, we show that multiple known input-dependent pixel-wise perturbations share a common spectral property. Using this spectral property, we show that the top singular vector of input-dependent adversarial attack directions can be used as a very simple universal adversarial attack on neural networks. We evaluate the error rates and fooling rates of three universal attacks, SVD-Gradient, SVD-DeepFool and SVD-FGSM, on state of the art neural networks. We show that these universal attack vectors can be computed using a small sample of test inputs. We establish our results both theoretically and empirically. On VGG19 and VGG16, the fooling rate of SVD-DeepFool and SVD-Gradient perturbations constructed from observing less than 0.2% of the validation set of ImageNet is as good as the universal attack of Moosavi-Dezfooli et al. (2017a). To prove our theoretical results, we use matrix concentration inequalities and spectral perturbation bounds. For completeness, we also discuss another recent approach to universal adversarial perturbations based on (p, q)-singular vectors, proposed independently by Khrulkov & Oseledets (2018), and point out the simplicity and efficiency of our universal attack as the key difference.

This paper studied the problem of universal adversarial attack which is an input-agnostic perturbation. The authors proposed to use the top singular vector of input-dependent adversarial attack directions to perform universal adversarial attacks. The authors evaluated the error rates and fooling rates for three attacks on standard benchmark datasets.
- The paper is generally well-written and easy to follow. My main concern towards this paper is about the experiments part from several aspects. First, the proposed method needs quite large L2 norm (50 on ImageNet) to work, while common adversarial attack experiments on ImageNet are usually conducted with L2 perturbation strength of 5 or less. I totally understand that performing universal attack would be much more difficult, yet having such loose L2 norm constraint still seems impractical. Second, the authors did not compare with any other baselines such as  (Moosavi-Dezfooli et al. 2017a) arguing that their universal attack is different for different perturbation strength and pixels are normalized. I do not think normalized pixel will be a problem as you can simply scale the perturbation strength accordingly. And because (Moosavi-Dezfooli et al. 2017a) uses different attack vectors for different perturbation strength, some comparison between these two types of universal attacks should be presented in order to mark the difference and demonstrate your advantages. I would suggest the authors to compare with several mentioned baselines in the paper to show the superiority of the proposed method.
- Theorem 1 seems interesting, yet it needs a special assumption. The authors argue that this is a reasonable assumption in a small neighborhood of x. I wonder if the authors could conduct some demonstrative experiments to verify this? Because the definition of S_x depends on the attack function, does it mean that the assumption need to be held for any attack function? Also regarding the choice of \delta, it seems that \delta is different for different x? If so, since u is also depend on \delta, this attack vector seems not universal?
Detailed comments:
- In proof of Theorem 1, all S should be G?
- In proof of Theorem 2, how to get \|v - \hat v\|_2 \leq \epsilon/(\gamma - \epsilon)? Directly applying the Theorems seems to get \epsilon / (\gamma) only?
Depending on whether the authors can address my concerns, I may change the final rating.
======================
after the rebuttal
I thank the authors for their response but I still feel that the assumption is not well-justified and there is still a lot to improve in terms of experiments. Therefore I decided to keep my score unchanged.


We propose a reinforcement learning framework for discrete environments in which an agent makes both strategic and tactical decisions. The former manifests itself through the use of value function, while the latter is powered by a tree search planner. These tools complement each other. The planning module performs a local what-if analysis, which allows to avoid tactical pitfalls and boost backups of the value function. The value function, being global in nature, compensates for inherent locality of the planner. In order to further solidify this synergy, we introduce an exploration mechanism with two distinctive components: uncertainty modelling and risk measurement.
To model the uncertainty we use value function ensembles, and to reflect risk we use propose several functionals that summarize the implied by the ensemble. We show that our method performs well on hard exploration environments: Deep-sea, toy Montezuma's Revenge, and Sokoban. In all the cases, we obtain speed-up in learning and boost in performance.

The authors propose to combine planning methods like MCTS with an ensemble of value functions to a) estimate the value of leaf nodes of the search tree and b) use the ensemble estimate of uncertainty to guide exploration during MCTS search.
The MCTS rollouts are also used as optimization targets for the value function.
I believe this is a clear reject. On the one hand, the paper needs signficiantly more work on the writing and clarity. On the other hand I have several worries on the method and evaluation side.
Regarding the presentation of the paper:
Overall, the paper seems quite rushed. This is not a strong reason for rejection but should be improved in a future version. For example, punctuation and sentence structure is often wrong, the paper has only slighlty over 7 pages, a citation is undefined on p.7 and images and whitespace is formatted wrongly on occasion (e.g. top of page 6).
More importantly, on the content side, the experimental section is sufficiently clear and well written, however, the method description needs more detail and background information. The paper relies on several prior works which are referred to but not described (E.g. MCTS , the sampling mechanism by Osband et al. which they are using but not describing, the 'mask' from Osband et al which they are using but not describing).
Furthermore, the algorithm itself is not described in sufficient detail:
- How does the 'soft-penalization' work?
- How exactly does the mechanism "similar in fashing to" Thomson sampling work?
- Are you learning a model or do you have access to the true transition function?
Regarding the method:
I can't say anything definitive about the method as I'm not entirely clear how exactly it works. However, I have several worries that might need addressing:
- It seems to me that the method relies on access to the _true_ transition and reward function and not on a learned model. This is a big difference to much of the prior work they compare against. This also makes the comparison against any pure model free method like PPO much less meaningful.
- Similarly, manually avoiding dead-ends and loops is a very strong assumption
- Also, being able to distinguish and use a fixed ratio of "solved" and "unsolved" episodes is a strong assumption.
- The one main contribution seems to be a new way of how \phi_a(x) is defined. Their particular choice needs a clearer motivation. Furthermore, if there is more contribution and differences to prior work, highlighting them more would help the reader understand the contribution.
- As the work makes several strong assumptions regarding the environment and access to the model, significantly more work (e.g. ablation studies) is needed to clearly show which assumption and feature of the algorithm is important for performance (and ideally also why). For example (but that's just a first idea): To understand the impact of their choice of \phi vs. their planning architecture, it would be be interesting to maybe train PPO using an exploration bonus based on \phi. This would allow disentangling the contribution of: Access to the true model, "discrete-environment-tricks" like penalizing dead-ends, and exploration incentivication of \phi.
Edit:
Thank you for your response and the updated manuscript, which reads considerably better.
I also agree with your point regarding the strength of assumption regarding "solved" and "unsolved" episodes.
Consequently, I will raise my score to a "weak reject" to express that I think this is promising work.
I do believe that ablation studies would add a lot to the paper as they would allow one to see which of the (many) added components help how much, for example between the selection function $\phi$ and the various penalizations used.


In many machine learning problems, loss functions are weighted sums of several terms. A typical approach to dealing with these is to train multiple separate models with different selections of weights and then either choose the best one according to some criterion or keep multiple models if it is desirable to maintain a diverse set of solutions. This is inefficient both at training and at inference time. We propose a method that allows replacing multiple models trained on one loss function each by a single model trained on a distribution of losses. At test time a model trained this way can be conditioned to generate outputs corresponding to any loss from the training distribution of losses. We demonstrate this approach on three tasks with parametrized losses: β-VAE, learned image compression, and fast style transfer.

Due to the late rebuttal, I was not able to respond during discussion time.
Q1, Q2) sound and look convincing.
Q3) I still cannot find details on this in the paper. How the validation set was chosen? What was the size? The authors need to make all the experiments fully reproducible.
Q4, Q5) ok
My concerns were sufficiently addressed in the current revision, and I will increase my score. However, the paper still feels close to a borderline, but probably, tending to "accept".
Also, I agree with Review #4, that also wondering about the application of the proposed method to hyperparameter search (Suggestion 1 in my review). Even if this would not be a sota in hyperparameter search, it feels like missing the opportunity to make the paper much stronger, by adding one more nice property to the proposed model.
-----
Generative models often use a loss function that is a weighted sum of different terms, e.g., data-term and regularizer. Let's denote these weights as λ. The paper proposes a method for learning a single model that approximates the result produced by a generative model for a range of loss-term weights λ. The method uses the following mechanisms i) λ-conditioned layers ii) training with a stochastic loss function, that is induced by a (log-uniform) distribution over λ. The performance of the model is demonstrated on the following problems learning β-VAE, image compression, and style transfer. The models clearly demonstrate an ability to approximate problem solutions for a range of coefficients. The paper is clearly written. The experiments, however, need future discussion.
1) The beta-VAE experiments (sec. 4.1)
Q1. While models demonstrate a reasonable behavior on Shapes3d dataset. The samples and reconstructions on CIFAR10 (Figure 8) indicate that all models are not trained well. If this is the case, conclusions might be misleading, since the approximating output of undertrained models might be much simple comparing to well-trained ones. Authors may want to provide a comparison of the trained models with conventional VAEs (with β=1), the reference figures for CIFAR10 are provided, for example, in https://arxiv.org/abs/1606.04934.
Q2. Wider YOTO seems to help a lot, but, what happens to the baseline models of increased size?
"We select the fixed β so that it minimizes the average loss over all β values."
Q3. Was it done directly on a test set, or were validation-data used?
2) Image compression (sec. 4.3)
"Finally, a wider model trained with a larger batch size (“YOTO wider batch16”) closely follows the fixed weight models in the high compression regime and outperforms them in the high quality regime." (Figure 5)
Q4. How is this compared to the baseline with batch16?
Q5. Authors also may want to provide std for provided metrics. The difference does not look statistically significant.
Suggestion 1: It might also be interesting to see if we can use this technique to perform a hyperparameter search. Train the model, select one the best performing set of hyperparameters, and then train models with this best value.
Overall, the paper proposes an interesting technique, that surprisingly, can work for a range of hyperparameters, and potentially have a high practical impact. However, the empirical evaluation is half-baked, specifically has certain methodological drawbacks e.g., perhaps undertrained beta-VAE model, absence of standard deviations while comparing (close) numerical results, and comparing models with different optimization parameters -- the performance difference might be due to optimization.
I recommend to reject the paper, however, I will appreciate discussions with authors and other reviewers, and will consider changing my score in case of reasonable argumentation.


Graph Neural Networks (GNNs) are a powerful representational tool for solving problems on graph-structured inputs. In almost all cases so far, however, they have been applied to directly recovering a final solution from raw inputs, without explicit guidance on how to structure their problem-solving. Here, instead, we focus on learning in the space of algorithms: we train several state-of-the-art GNN architectures to imitate individual steps of classical graph algorithms, parallel (breadth-first search, Bellman-Ford) as well as sequential (Prim's algorithm). As graph algorithms usually rely on making discrete decisions within neighbourhoods, we hypothesise that maximisation-based message passing neural networks are bestsuited for such objectives, and validate this claim empirically. We also demonstrate how learning in the space of algorithms can yield new opportunities for positive transfer between tasks-showing how learning a shortest-path algorithm can be substantially improved when simultaneously learning a reachability algorithm.

This paper investigates using GNNs to learn graph algorithms. It proposes a model which consists of algorithm-dependent encoder and decoder, and algorithm-independent processor.
Authors try to learn BFS, Bellman-Ford and Prim algorithms on various types of random graphs.
Experimental results suggest that MPNN with max aggregator outperforms other variants significantly in terms of generalization.
Overall, this paper presents a solid contribution on learning graph algorithms using GNNs, despite the caveat for some clarifications on the model and experiments.
Given these clarifications in an author response, I would be willing to increase the score.
Pros:
1, Although it is less satisfying to learn to solve graph problems where polynomial-time algorithms exist, I still appreciate the contribution of the paper, especially the algorithm-independent processor. It is one step forward to models which could learn meta-level representation of algorithms. The findings of this paper may suggest that different types of operators used in the GNN may have different inductive bias in learning different types of algorithms.
2, Experimental comparisons are adequate and convincing. The detailed analysis of empirical results also provide good explanation.
Cons & Suggestions:
1, Given the dynamic programming nature of the most of tasks, it is not that surprising that MPNN with max aggregation could solve them pretty well. What surprises me is that MPNN-mean / MPNN-sum could not generalize well (i.e., performance drops significantly on 50 and 100 nodes settings) on the reachability task. In my opinion, the reachability task could be easily handled by any diffusion/propagation based models including, e.g., MPNN, GCN, GAT, as long as the information is spread over the input graph. Could you explain why does this happen?
2, Training details are sparse. If I understood correctly, the training of various GNNs is done by teacher forcing such that each step some supervised information collected from the underlying graph algorithms is provided to GNNs. However, it is not clear that what supervised information is exactly provided under each graph algorithm. It would be great to have a table to summarize what the input and supervised output information is for each graph algorithm.
3, The processor is trained to learn multiple graph algorithms simultaneously. A natural question to ask is how does the performance change when you train the processor with different combinations of algorithms? What is the impact of correlations between different graph algorithms? Did you explore learning with graph algorithms following some sequential schedule, e.g., curriculum learning?
4, It would be great to provide more results on larger size graphs, e.g., trained on 20 nodes but test on 1000 or more nodes. I saw one set of experiments trained on 100 and test 1000 nodes in the appendix. More results along this line would make the paper more convincing on the generalization.
5, It would be great to discuss [1] as it also studies how to use GNNs to learn a special class of graph algorithms, i.e., probabilistic inference algorithms on graphs. It is shown in [1] that belief propagation algorithm could be seen as a specially constructed GNN. Interestingly, some well-known graph algorithms could be re-formulated as probabilistic inference algorithms, e.g., graph-cut could be reformulated as max-product [2].
[1] Yoon, K., Liao, R., Xiong, Y., Zhang, L., Fetaya, E., Urtasun, R., Zemel, R. and Pitkow, X., 2018. Inference in probabilistic graphical models by graph neural networks. arXiv preprint arXiv:1803.07710.
[2] Tarlow, D., Givoni, I.E., Zemel, R.S. and Frey, B.J., 2011, July. Graph Cuts is a Max-Product Algorithm. In UAI (pp. 671-680).
======================================================================================================
Thanks for the thorough response! It resolves my concerns. I improved my score.


Black-box adversarial attacks require a large number of attempts before finding successful adversarial examples that are visually indistinguishable from the original input. Current approaches relying on substitute model training, gradient estimation or genetic algorithms often require an excessive number of queries. Therefore, they are not suitable for real-world systems where the maximum query number is limited due to cost. We propose a query-efficient black-box attack which uses Bayesian optimisation in combination with Bayesian model selection to optimise over the adversarial perturbation and the optimal degree of search space dimension reduction. We demonstrate empirically that our method can achieve comparable success rates with 2-5 times fewer queries compared to previous stateof-the-art black-box attacks.

This paper studied the problem of black-box adversarial attack generation by leveraging Bayesian optimization (BO).
Merits of this paper:
1) The combination of BO and dimension reduction, which makes BO more efficient under a low-dimensional space.
2) Good experiment results.
Comments/questions about this paper:
1) Comment on "Finally, to the best of our knowledge, the only prior work that uses Bayesian optimisation is a workshop paper by...". BO was also used for generating black-box adversarial examples at https://arxiv.org/pdf/1907.11684.pdf
This is a missing related work, and please elaborate on the differences.
2) The presentation of the proposed algorithm is not clear. Please explicitly state the acquisition function. And how to tune the hyperparameter in the acquisition function? What decoder is used in experiments? Have authors tested the sensitivity of the decoder (not reduced dimension)?
3) In experiments, the authors mentioned "we randomly select 3 correctly classified images for each class from CIFAR10 test data which sums up to 27 CIFAR10 images, and randomly select 7 correctly classified images from MNIST test data."
I feel that the number of tested images is not sufficient. How about conducting experiments on a large number of tested images for untargeted attack?
4) In Table 1, what does 0,0,0, mean in ZOO?
5) It is known that BO has itself parameter to tune, and is not computationally efficient. It might be good to show the computation efficiency of BO for different reduced dimensions together with the corresponding attack performance.
6) The convergence of BO is usually not stable. However, Figure 3 shows that BO converges very smoothly in terms of ASR. Could authors also show the loss value of using BO-attacks against iteration numbers?
Meanwhile, in Figure 3 is the best ASR  (up to the current query counts) reported or the ASR at the current query number?
Based on the aforementioned questions, my initial rating is weak reject.
############## Post-feedback ################
Thanks for the response. Most of my questions have been addressed. Thus, I increased my score to 6.
I suggested to have a clearer presentation on the possible pros and cons of BO in attack generation, e.g., making a comparison between BO and other methods in both query efficiency and computation efficiency.


Click Through Rate (CTR) prediction is a critical task in industrial applications, especially for online social and commerce applications. It is challenging to find a proper way to automatically discover the effective cross features in CTR tasks. We propose a novel framework for CTR task, called Deep neural networks with Encoder enhanced Factorization Machine (DeepEnFM). Instead of learning the cross features directly, DeepEnFM adopts the Transformer encoder as a backbone to align the feature embeddings by the clues of other fields. The embeddings generated from the encoder are beneficial for further feature interactions. Particularly, DeepEnFM utilizes a bilinear approach to generate different similarity functions with respect to different field pairs. Furthermore, the max-pooling method makes DeepEnFM feasible to capture both the supplementary and suppressing information among different attention heads. Our model is validated on the Criteo and Avazu datasets, and achieves the state-of-art performance.

The authors propose a model for Click-Through Rate Prediction using a model consisting of an embedding layer, a Transformer stack, a Factorization Machine, and a DNN.
I have several major concerns about the submission:
2. Relevance: This work is extremely application specific, the application is not relevant to this community.
1. Clarity and writing: The contributions which are relevant to the ICLR community are not explained well and the paper needs copy-editing for English grammar
4. Novelty: While seemingly showing good results on some benchmarks, the model is a mix of many components and it's not clear which components actually improve performance and would be worth further study.
Minor comments:
Applying the DNN directly on top of the embeddings, and having a parallel stack of Encoder-FM, is not well explained. What does it mean that "DNN aims at bit-wise level" if the DNN receives the same embedding features as the encoder, which supposedly "learn[s] at vector wise level"?
References to datasets are missing
Ablation study is limited, and has surprising results. E.g. even completely removing self-attention barely makes a dent in how well the method compares to other published work, moving it from rank 1 to rank 2. Otherwise only small tweaks with even more minor effects are made. What about removing e.g. the FM, other major components?
The biggest architectural innovations here are the bi-linear attention mechanism and max-pooling self attention. They are hard to interpret in this context. It's not clear how they would perform in a simpler architecture (e.g. vanilla BERT or Transformer) and in the context of a more standard benchmark. That study would have a lot more relevance to this community than the present one.


In this paper, we propose an end-to-end deep learning model, called E2Efold, for RNA secondary structure prediction which can effectively take into account the inherent constraints in the problem. The key idea of E2Efold is to directly predict the RNA base-pairing matrix, and use an unrolled algorithm for constrained programming as the template for deep architectures to enforce constraints. With comprehensive experiments on benchmark datasets, we demonstrate the superior performance of E2Efold: it predicts significantly better structures compared to previous SOTA (especially for pseudoknotted structures), while being as efficient as the fastest algorithms in terms of inference time.

RNA Secondary Structure Prediction by Learning Unrolled Algorithms
This paper proposes E2Efold, which is an RNA secondary structure prediction algorithm based on an unrolled algorithm. Previous methods rely on dynamic programming (which does not work for molecular configurations that do not factorize) or rely on energy-based models (which require a minimization step, e.g. by using MCMC to traverse the energy landscape and find minima). The former does not work for all molecules and the latter can be difficult to optimize. The method presented here is novel, shows strong SOTA performance, and would be of interest to the wider deep learning community.
The method is based on an unrolled algorithm, which is motivated by the inclusion of three inductive biases / constraints important underlying RNA folding. These constraints limit the wide RNA search space. The first component of the method is a “Deep Score Network” which uses a stack of Transformer encoders (with relative and exact positional embeddings) followed by 2D convolutional layers to output a L x L symmetric matrix describing the “scores” of base pairing. As these scores may not obey the rules of RNA folding, a second post-processing network is trained end-to-end together with the “Deep Score Network” to enforce constraints. This network starts with a transformation that symmetrizes the matrix and applies a constraint-enforcing mask. The problem is transformed into an unconstrained problem by using Lagrange multipliers; it is then solved using a proximal gradient. Finally, a recurrent cell is defined that implements this algorithm in a deep learning framework. This method is creative, could be applied to other tasks with constraints, and would be interesting to the wider deep learning community.
In addition to developing the deep score network and post-processing network, the authors also develop a differentiable F1 loss, so that the network can directly optimize for precision and recall on the task. The performance of this method significantly outperforms previous methods. There was a fruitful discussion on OpenReview regarding whether this was a result of overfitting on the task. Indeed, it is critical in deep learning applications to carefully construct train/test sets to avoid high performance by memorization alone. To address this, the authors train on RNAStralign and test on ArchiveII. As the original ArchiveII dataset contains subsequences of other RNA sequences, which can result in overfitting, the authors re-ran their experiment with that removed, and similar results were achieved. To support the hypothesis that ArchiveII and RNAStralign capture different distributions, they perform a permutation test on the unbiased empirical Maximum Mean Discrepancy estimator, finding that the distributions are different. I do wonder why they did not check if P(ArchiveII) = P(ArchiveII) as they do check if P(RNAStr_train) = P(RNAStr_train). On the specific task of pseudoknot prediction, the method also performs well (F1 is >0.23 over the baseline). On sequence length-weighted F1, the model does even better.
The paper is rich with ablations. The analysis of the number of unrolling iterations T helps support the use of an unrolled method and builds intuition for its importance - it would be useful to include this in the appendix of the paper. I also appreciated the visualizations, which are a good sanity check that the model correctly handles pseudoknots. The performance of the method is broken down by RNA family, which is also quite interesting -- the method outperforms LinearFold on all classes, besides 5S RNA, SRP, and Group I intron. Further analysis is required to better understand why the method is weaker on those datasets. Additionally, further work should explore training on one set of families and testing on a held-out set of families. This was pointed out by public comments on this paper. This is potentially a limitation of E2EFold (the authors do not seem to have tried this suggested experiment) and further exploration is required. Exploring this limitation (even if it is not overcome) would make this paper even more rich.
That said, I recommend acceptance of this paper due to the extensive experiments, polished writing, novel method, and strong results, which can inspire future research.


Developing agents that can learn to follow natural language instructions has been an emerging research direction. While being accessible and flexible, natural language instructions can sometimes be ambiguous even to humans. To address this, we propose to utilize programs, structured in a formal language, as a precise and expressive way to specify tasks. We then devise a modular framework that learns to perform a task specified by a program -as different circumstances give rise to diverse ways to accomplish the task, our framework can perceive which circumstance it is currently under, and instruct a multitask policy accordingly to fulfill each subtask of the overall task. Experimental results on a 2D Minecraft environment not only demonstrate that the proposed framework learns to reliably accomplish program instructions and achieves zero-shot generalization to more complex instructions but also verify the efficiency of the proposed modulation mechanism for learning the multitask policy. We also conduct an analysis comparing various models which learn from programs and natural language instructions in an end-to-end fashion.

This paper presents a reinforcement learning agent that learns to execute tasks specified in a form of programs with an architecture consisting of three modules. The (fixed) interpreter module interprets the program, by issuing queries to a (pre-trained) vision module and giving goals to a policy module that executes them in the environment. The paper also introduces a policy modulation technique, with the goal of modulating the current state with the expected (symbolic) goal. The model is evaluated on a 2D approximation of Minecraft, where it outperforms a set of baselines. In addition, the authors modified the program dataset and re-expressed it in terms of natural language and showed that the baselines perform better with programs than with the natural language instructions.
Though I think the general idea of the paper is worth exploring, I find the concrete contribution of the paper a bit thin to the point that I am hesitant to recommend this paper for acceptance. Allow me to explain my objections:
First and foremost, this work is very close to work by Denil et al where the execute (in a differentiable fashion) programs in an RL agent. Their work does not have a discrete interpreter per-se, but it does have a differentiable execution of commands. The major difference between these two works would be that Denil et al do not have the vision module (they do mention learning from pixels as future work).
However, that is not entirely true. The model presented here uses a pretrained vision module, which by itself is not a problem and is used in related work [1], but this vision module does not operate on visual input but the symbolic representation of the map. A crucial thing here is that if all of a sudden we want to include a new object on the map, the model won’t be able to use some learned similarity since it would require introducing a new object (slice of the input), as it would should it have been trained on pixels. So technically speaking, this is not a vision module but a symbolic state processing module.
Then, the modulation mentioned in the paper does not seem particularly novel. Sure, the exact architecture of the model is probably unique, but the idea of modulating a state with a goal is not and has been seen in other work such as [2] and [3] among others. The paper does not mention why, for example, this modulation technique is useful and why any other similar architecture would not be as successful, nor does it mention related modulation techniques in other work.
A big issue I have with the evaluation in the paper is that I do not see the benefit of having the experiments with natural language at all. The focal point of the paper are agents able to execute tasks in the form of programs. The (though manually) generated natural language instructions from those same programs cannot even be used by the proposed agent as there is no natural language interpreter, so they are a dangling part of the paper which is there just to showcase that programs should be easier for baselines to learn to execute than natural language (the hypothesis would be that a simpler and more structured/formal language is easier to learn than the natural language). Hence the seq-LSTM results on language which are a tad lower than the results on programs are expected, though the performance of transformers is the opposite---they are better on language, and that is something that is puzzling and left unexplained, as well as the unexpectedly low performance of Tree-RNNs. One would expect them to perform a bit better than LSTMs, but that might be contingent on the size of the dataset more than the structure of the inputs. However, none of these curious findings have been explained.
Moreover, the comparison to baselines is not particularly fair as these baselines had to learn the symbolic state interpretation, whereas your model did not. You could have provided the same to the baselines for a better comparison.
In addition to that, 5.4.2 goes into detail of analysing the baselines, and ignoring the proposed model. Why didn’t you include the same statistics for your agent in Figure 5 and said subsubsection?
The paper is missing some notable related work:
- S.R.K. Branavan’s work (all but one cited language-instructed agent papers are post 2017) as well as [4]
- object-oriented and hierarchical RL
- [5], where they train the neural programmer interpreter from the final reward only, which brings NPI close to this work
Questions:
- Figure 5 - There’s a mark for Tree-RNN, but Tree-RNNs are not in any figure. Why didn’t you plot the performance of your model?
- The setup naturally induces a curriculum - how does it do that if the programs are randomly sampled?
- You say that your model learns to comprehending the program flow. I’m not sure I would agree with that because what your model learns is to execute single commands. From what it seems, the interpreter is the only part of the model (which is fixed btw) which sees the control flow, whereas the policy just executes singular commands. Did you mean something else by that statement?
Minor issues:
- You say twice interpreter (i.e. compiler). Given that they’re not the same, and you’re using an interpreter, I suggest omitting the word compiler.
- Figure 2 is lacking detail. There is no difference between r and i (both being positive integers) other than their descriptions, - operators agent[_], env and is_there lack parameters (non-terminal nodes), and where’s river, bridge, etc?
[1] COBRA: Data-Efficient Model-Based RL through Unsupervised Object Discovery and Curiosity-Driven Exploration
[2] FeUdal Networks for Hierarchical Reinforcement Learning
[3] Universal value function approximators
[4] Vogel et al Learning to follow navigational directions
[5] Improving the Universality and Learnability of Neural Programmer-Interpreters with Combinator Abstraction


In this paper, we solve the arms exponential exploding issues in multivariate Multi-Armed Bandit (Multivariate-MAB) problem when the arm dimension hierarchy is considered. We propose a framework called path planning (TS-PP) which utilizes decision graph/trees to model arm reward success rate with m-way dimension interaction, and adopts Thompson sampling (TS) for heuristic search of arm selection. Naturally, it is quite straightforward to combat the curse of dimensionality using a serial processes that operates sequentially by focusing on one dimension per each process. For our best acknowledge, we are the first to solve Multivariate-MAB problem using graph path planning strategy and deploying alike Monte-Carlo tree search ideas. Our proposed method utilizing tree models has advantages comparing with traditional models such as general linear regression. Simulation studies validate our claim by achieving faster convergence speed, better efficient optimal arm allocation and lower cumulative regret.

In this paper, the authors address the curse of dimensionality in Multivariate Multi-Armed Bandits by framing the problem as a sequential planning problem.
Their contributions are the following:
1. They propose to solve the Multivariate MAB sequentially across each dimension, which I think is a reasonable and not very common approach in the literature compared to e.g. structured bandits, even though the authors themselves call this approach "natural" and "quite straightforward".
2. They propose a framework based on a Monte-Carlo Tree Search procedure to solve this sequential decision problem.
3. They introduce several approximations and heuristics to avoid traversing the entire tree and trade-off performance for computational complexity, which they evaluate empirically.
The obtained results seem to support their original claim: the tree-based approaches (e.g. FPF) perform better than the considered baselines. However, this claim is at the same time contradicted by the fact that a depth-one planning procedure with heuristic leaf evaluation (PPF2) reaches similar performances, which indicates that the underlying problem can be solved greedily and does not involve long-term (m-way) interations and limits the interest of the main idea in this paper. This is probably due to the design of experiments which only model pairwise interactions, but then this questions the relevance of the proposed empirical evaluation with respect to the claim.
Generally, though several good ideas were presented, I felt that they were not properly motivated and that the paper generally lacks rigor and clarity. The many variants and modeling decisions proposed seem quite arbitrary, and in the end fail to provide any valuable insight.
Several aspects of the proposed approach and algorithms were not clear or justified. I will list some of them, grouped by the corresponding contribution:
1. Multivariate MAB framed as a planning problem.
No intuition is provided regarding why it is expected to perform any better than classical N^D-MAB. In this absence of assumption regarding additional structure to the problem considered (such as, say, the independence assumption in D-MABS), is there any reason to think so? (for instance, all the N^D arms are present at the leaves of the tree) The authors only mention in their conclusion that they leverage a "hierarchy dimension structure", but this claimed structure was is never defined nor formalized in the paper. I can only guess that such a structure would involve a particular ordering of dimensions such that shallow ones have more influence on the payoff than the deeper ones? But then, it would be unknown in practice, and the choice of the particular ordering used in TS-PP is expected to have a crucial influence on its performance. Yet, the sensitivity of their algorithms to the ordering is never discussed by the authors -they choose it arbitrarily-  nor evaluated in their experiments.
2. The proposed TS-PP framework
- The authors propose using MCTS with TS instead of UCB as a sampling rule, which they claim is quite novel: "Applying TS as node selection policy is not well investigated from our best knowledge". However, the authors do not mention important reference (Bai et al, 2014) until the very end of the paper in their conclusion: "We notice some related work dealing with continuous rewards in this area". Yet, Bai et al. also consider Bernoulli rewards in their experiments and use conjugate Beta priors, exactly like the authors of the present work do. Moreover, the choice of TS rather than UCB is not properly motivated, and the UCT algorithm is not considered as a baseline in the experiments. Hence, there is no basis to assess whether TS is particularly relevant or not for this problem.
- They also propose to repeat S times the candidate searching procedure in Algorithm 1 in order not to be "[stuck] in sub-optimal arms". This property is normally ensured by optimistic sampling rules, it is not clear why it is required here. More importantly, the sensitivity of the Algorithm 1 to S is not evaluated.
- I did not fully understand the Candidates Construction Stage in Algorithm 1. For instance, in the case of FPF, for each search c=1..S (l3) there is a call (l4) to Full Path Finding() to generate a candidate A^c. From what I understand, a random ordering is chosen (Algorithm 2, l2), and a single rollout is performed (l2, the dimensions 1->D are iterated from root to leaf) to return a candidate. When we move to the next search c -> c+1, since a new ordering d_1:D is chosen then the previous tree storing the models for p(fd_i | fd_1...fd_i-1) is not relevant anymore and must be discarded. Does that mean that the priors at the nodes are recomputed for each search/ordering/tree given the entire history, instead of being maintained and updated? The authors do state that "FFP utilizes hidden states from the same decision tree", but I do not see how this is feasible when the dimensions ordering changes at each iteration.
- This confusion is related to the fact that the update of the Beta posterior does not appear anywhere in Algorithm 1 and 2. This is unfortunate, especially considering that contrary to TS in a normal MAB setting, the observations for an internal node are not i.i.d. since they depend on the underlying planning procedure, which induces a distributional drift as more sample are collected and probably make the derived posterior invalid. Has this issue been considered by the authors?
3. The variants of FPF
The idea behind Partial-PF-m is common in the tree-search literature: to stop at early depths and use a heuristic to estimate the value at the leaf. Here, the suggestion in the particular context of Multivariate-MAB to use TS with an independence assumption (D-MABs) is interesting and sound. However, It requires the knowledge the degree m of interactions, whereas the idea of MCTS is precisely to avoid using such heuristics for leaf evaluation by replacing them without random rollouts. Other methods such as DS start at a leaf and move locally which rather resembles Hill-climbing more closely than actual Tree-Search, and I do not really see how they fit in the TS-PP framework?
Finally, I found the manuscript difficult to read due to many language mistakes and typos.
Minor comments:
- The citation style is incorrect.
- What is the point of introducing contextual MABs in the problem formulation when the context is not going to play any part whatsoever in the presented work? This only brings unnecessary complexity and impedes clarity.
- In the experiments, H=100 replications are performed, which is appreciated, but unfortunately the corresponding confidence intervals or standard deviations are not shown, only the mean performance.
- I found the section 3.1 unclear. Is B_A^T \mu a dot product? Where do the coefficients of B_A or its length M appear in (1)? How is M chosen, how is B_A obtained?
- Figure 1.a is also unclear, the graph seems to be used to define permutations/orderings of dimensions 1..D. But all paths are not valid (eg. d1 -> d2 -> d1), ad random orderings are used in Algorithm 2, so where is this graph used exactly?
=============
After Rebuttal:
On one hand, some of the points mentioned are now clearer (e.g. 1 & 2.2, 2.3).
On the other hand, most of my concerns remain valid, and some were not addressed.
Typically, the dependency to an unknown (and undefined...) optimal ordering and the underlying optimization procedure (random search) are not properly discussed, whereas they seem central to justify the tree-based approach.
Other remaining concerns are more generally the lack of justification and clarity of the proposed framework and implemented variants, and the inadequacy of the experiments with respect to the claim (using m=2 basically means only the first depth of the tree is useful). Therefore, I will maintain my initial score.


Many applications of linguistic embedding models rely on their value as pretrained inputs for end-to-end tasks such as dialog modeling, machine translation, or question answering. This position paper presents an alternate paradigm: Rather than using learned embeddings as input features, we instead treat them as a common-sense knowledge repository that can be queried via simple mathematical operations within the embedding space. We show how linear offsets can be used to (a) identify an object given its description, (b) discover relations of an object given its label, and (c) map free-form text to a set of action primitives. Our experiments provide a valuable proof of concept that language-informed common sense reasoning, or 'reasoning in the linguistic domain', lies within the grasp of the research community. In order to attain this goal, however, we must reconsider the way neural embedding models are typically trained an evaluated. To that end, we also identify three empirically-motivated evaluation metrics for use in the training of future embedding models.

This is a position paper - it discusses overall about multi-word embedding (sentence) level and how it can be leveraged in multiple applications. Also, it discusses some open opportunities on how the embeddings can be used as a base knowledge source and some challenges in them
Pros:
1. The examples and analysis of embeddings involved in the paper is detailed
2. The authors have run lots of experiments to position their idea correctly.
Cons:
1. I find many of their intuitions well established in the literature - the intuition of geometric structures in the embedding space, the intuition of incidence angle. The ideas from section 4,5,6 seem to be rather obvious extension of what we know
2. Especially, in Section 4,5 the idea of finding indexes, is nothing but the projection of x on the mean of (A,B) - which is more of a common practice. I find both these ideas of finding indexes and extrapolation, pretty much recurrent and existing in literature.
3. What I would be really interested is, a new training algorithm for learning embeddings, that would be able to answer all linear algebra based questions in the embedding space. Rather, this paper analyses the pros and cons of 6 existing models. In my personal opinion, this is half work done for a very good paper.
4. Personally, I am not very appreciative of the use of  "Common Sense" in this paper. There is a set of papers in the literature (atleast in vision domain), where they actually try to learn the common sense from the underlying data. The use of common sense in this paper does not align with that usage and hence, is misleading for me.
5. From reading a position paper, I would ideally want to see ideas worth of atleast 2-3 PhD thesis. I do not find such compelling ideas in this paper.