The travelling salesman problem (TSP) is a well-known combinatorial optimization problem with a variety of real-life applications. We tackle TSP by incorporating machine learning methodology and leveraging the variable neighborhood search strategy. More precisely, the search process is considered as a Markov decision process (MDP), where a 2-opt local search is used to search within a small neighborhood, while a Monte Carlo tree search (MCTS) method (which iterates through simulation, selection and back-propagation steps), is used to sample a number of targeted actions within an enlarged neighborhood. This new paradigm clearly distinguishes itself from the existing machine learning (ML) based paradigms for solving the TSP, which either uses an end-to-end ML model, or simply applies traditional techniques after ML for post optimization. Experiments based on two public data sets show that, our approach clearly dominates all the existing learning based TSP algorithms in terms of performance, demonstrating its high potential on the TSP. More importantly, as a general framework without complicated hand-crafted rules, it can be readily extended to many other combinatorial optimization problems.

The paper proposes an algorithm for the Travelling Salesman Problem that starts with a random tour and iteratively improves it using 2-opt local search, followed by Monte Carlo Tree Search with k-opt moves to search a larger neighborhood of solutions. Results show that the algorithm matches the optimal objective value on a dataset of 2D Euclidean TSP instances with 20, 50, and 100 cities, and achieves better objective value than S2V DQN on a subset of instances from TSPLIB with 50 to 318 cities. It is interesting that the algorithm is fairly simple (apart from the definition of the local moves) and still able to achieve competitive objective values on small instances.
The rating is a weak reject for the following reasons:
1. Domain knowledge: The proposed algorithm has significant domain knowledge about TSP hand-coded into it via the choice of 2-opt local search as well as the k-opt moves with top-10 nearest neighbor filtering used in MCTS. These types of moves are same/similar to the expert-designed ones used in state-of-the-art local search algorithms like LKH. This can give the proposed algorithm a significant advantage over S2V-DQN which does not have such built-in knowledge. So the comparison to S2V-DQN is unfair and improved results may not be due to any differences in the learning approaches at all. Perhaps a more generic definition of local moves that is not so TSP-specific would be a fairer comparison. This issue is particularly important because the definition of “local moves” appropriate for a problem domain is one of the main challenges in applying local search successfully. Bypassing that challenge can simplify the problem.
2. Baselines: Better comparisons to baselines are needed in Table 2. S2V-DQN was published two years ago, and since then many other learning-based have been proposed (which are referenced in the paper), but they are not included in Table 2. It would also be good to discuss non-learning local search algorithms that use same/similar local moves as the proposed algorithm and how they perform.
3. Scalability to large instances: TSP experts consider problems with hundreds of cities as practically “solved” by the current state-of-the-art solvers like Concorde and LKH, and it is not at all clear what benefit learning can provide on such small instances. Much larger instances (e.g., 10^4-10^6 cities) would have to be considered for improvements over the state-of-the-art to become plausible with learning. So if the goal is to improve on the state-of-the-art with learning (even if it is a long-term goal), then it is important to consider whether the proposed approach has a plausible path for scaling up to such sizes. But like most papers on learning-for-TSP, this paper does not consider scalability.
Misc comments:
It would be useful to include the exact running times of Concorde in Table 2. The results given are “reported by Concorde within 1 hour for each instance”, but it would be good to know what the actual running time is.
Typo: “sub-decisions sequently”


The dynamics of DNNs during gradient descent is described by the so-called Neural Tangent Kernel (NTK). In this article, we show that the NTK allows one to gain precise insight into the Hessian of the cost of DNNs. When the NTK is fixed during training, we obtain a full characterization of the asymptotics of the spectrum of the Hessian, at initialization and during training. In the so-called mean-field limit, where the NTK is not fixed during training, we describe the first two moments of the Hessian at initialization.

This paper discusses the behavior of the Hessian of the loss of NN during training. There is no question that the subject is relevant, and the paper gives an interesting contribution to the problem.  I do support publication.
My main criticism is in the presentation of the paper: The title and the first pages are, I believe, really misleading. Unless I am missing an important point, all derivation of the papers are being made in the so-called NTK regime. In fact, this is explicitly stated in page 3 "In the limit we study in this paper, the NTK is fixed". While this is an interesting assumption, this is by no mean a trivial one that should be stated casually in the middle of the paper. This is an extremely strong assumption (essentially, one is either in the lazy Regime of [Chizat&Bach] with infinity small learning rate, or in the kernel regime of [Jacot&al] with astronomically large layers. This is a nice setting for mathematics, but by no means a natural one and this limits drastically the conclusion of the paper: what is studied is the behavior of the DNN in the "Kernel" regime, or the "Lazy" regime. Such a limitation should be explicitly mentioned, at least in the abstract and the introduction, if not in the title itself! It should, also, be mentioned in the conclusion ("We have given an explicit formula for the limiting moments of the Hessian of DNNs throughout training").
In this respect, with the connection to the Kernel, it make sense that the Hessian is dominated by, essentially, the equivalent of Kernel PCA! This is to be expected, since one is essentially doing a Kernel Ridge Regression with random features.
Other source of confusion (for me):
* "This is contrast to the results obtained in [Pennington&Bahri] etc...." Is the only difference the fact that the network is more than two layers?
* "This gives theoretical confirmation of a number of observations about the Hessian"... I do not believe all these references are in the NTK regimes. (e.g. Chaudhari et al., for instance).
That being said, the paper is a nice and interesting contribution to the NTK regime, that is the subject of intense studies currently. I do support acceptance if the paper clarify the range of applicability of the its results.


Granger causal structure reconstruction is an emerging topic that can uncover causal relationship behind multivariate time series data. In many real-world systems, it is common to encounter a large amount of multivariate time series data collected from heterogeneous individuals with sharing commonalities, however there are ongoing concerns regarding its applicability in such large scale complex scenarios, presenting both challenges and opportunities for Granger causal reconstruction. To bridge this gap, we propose a Granger cAusal StructurE Reconstruction (GASER) framework for inductive Granger causality learning and common causal structure detection on heterogeneous multivariate time series. In particular, we address the problem through a novel attention mechanism, called prototypical Granger causal attention. Extensive experiments, as well as an online A/B test on an E-commercial advertising platform, demonstrate the superior performances of GASER.

--
Comments after reading the reply from the authors:
Thanks for the clarification which resolves most of my concerns, and I have updated the score accordingly.  Besides, it will be more convincing to add more baselines for comparison or provide more explanation why linear Granger (which is proposed decades ago) is a strong baseline.
--
This paper investigates the important problem of inferring Granger casual structures from multi-variate time-series data, and propose the Granger casual structure reconstruction (GASER) framework with prototypical Granger causal attention.  The paper is in general well written with clear notations and is easy to follow. The proposed attention mechanism is also  intuitive. Experiments on  both simulated/synthetic and real-world datasets show the proposed approach achieved both improved casual recovery and more accurate predictions.
Here are some concerns, and the reviewer is willing to adjust the rating if these concerns are resolved.
- Baselines. Have enough strong baseline algorithms been included?
 (1) It seems that GASER outperforms state-of-the-art prediction algorithm IMV-LSTM by a large margin (Table 5) even if it is not designed for the prediction task.
(2) In Table 1-3, the other two baselines usually perform similarly or worse than the simple linear Granger baseline. Does it indicate the compared baselines are not strong enough?
(3) Why linear Granger, i.e, VAR with L1 regularization, is not included in Table 5 for prediction?
- Model design and trade-off.  According to Table 1-5, it seems that by adding the proposed attention mechanism, we can achieve both improved results (better than SOTA IMV-LSTM) and better interpretability.
(1) Is there any trade-off in the design?
(2) Is the proposed approach sensitive to different parameters, e.g., 1) the number of prototypes, $K$ (when it is not the same with the ground-truth), 2) $\alpha$ in Equation 9 (is 0.5 is a generally good default choice?), 3) $\lambda_1, \lambda_2$ in Equation 13, 4) $\gamma$ in Equation 12.
- Prototype learning. It seems that the prototype learning is used to deal with the lack of data. However, adding prototype learning increases the number of parameters to be learnt, i.e., $[p_1, … p_K]$.  It will be helpful to provide more intuition.


First-order adaptive optimization algorithms such as Adam play an important role in modern deep learning due to their super fast convergence speed in solving large scale optimization problems. However, Adam's non-convergence behavior and disturbing generalization ability make it fall into a love-hate relationship to the deep learning community. Previous studies on Adam and its variants (refer as Adam-Type algorithms) mainly rely on theoretical regret bound analysis, which overlooks the natural characteristics residing in Adam-Type algorithms and limits our understanding. In this paper, we aim at seeking a different interpretation of Adam-Type algorithms so that we can intuitively comprehend and improve them. The way we chose is based on a traditional online convex optimization algorithm scheme known as mirror descent method. By bridging Adam and mirror descent algorithm, we receive a clear map of the functionality of each term in Adam. In addition, this new angle brings us a new insight on identifying the non-convergence issue of Adam and explaining the superior convergence speed of Adam than other first-order methods. Moreover, we provide a new variant of Adam-Type algorithm, namely AdamAL which can naturally mitigate the non-convergence issue of Adam and improve its performance. We further conduct experiments on various popular deep learning tasks and models, and the results are quite promising.

Summary:
This work proposed a framework to analyze both Adam-type algorithms and SGD-type algorithms. The authors considered both of them as specialized cases of mirror descent algorithms and provided a new algorithm AdamAL. The authors showed experiments to backup their theoretical results.
Pros:
The authors provided a novel framework to analyze Adam-type algorithms by using standard FTRL framework. It provides a unified viewpoint to consider a broad class of algorithms. The authors also provided a new algorithm AdamAL to overcome some shortcomings in previous algorithms.
Cons:
- The novelty of this paper is limited. The authors provided a framework to analyze Adam-type algorithms. However, it seems that contribution is more conceptual rather than practical. I suggest the authors add more examples or theorems to show the superiority of their mirror descent framework.
- Some of the explanations in this paper may be wrong. For instance, around equation 17, the authors suggested that if the swap happens, then v_{t+1} = v_t. That is not correct since the swap happens for each coordinate. Meanwhile, the explanation about why AdamAL is better than AMSgrad is quite poor since the update rule of AMSgrad can also guarantee the coordinate decreasing of v_t. I suggest the authors explain more on the algorithm design.
- The authors should complete the proof of Theorem 3.1.
- The settings of experiments are limited. The authors should at least compare AdamAL with other baseline algorithms on some modern deep learning tasks including Imagenet.
Minor comments:
- Below equation 8, detail -> details.
- The authors should add the definition of 1:t in subscript for g_{1:t} or \phi_{1:t}.
- Page 6, the first paragraph, logt-> \log t
- This paper lacks some references in this area.
J. Chen and Q. Gu. Closing the generalization gap of adaptive gradient methods in training
deep neural networks. arXiv preprint arXiv:1806.06763, 2018.
Ward, R., Wu, X. and Bottou, L. (2018). Adagrad stepsizes: Sharp convergence over nonconvex
landscapes, from any initialization. arXiv preprint arXiv:1806.01811 .
Li, X. and Orabona, F. (2018). On the convergence of stochastic gradient descent with adaptive
stepsizes. arXiv preprint arXiv:1805.08114 .


The edit distance between two sequences is an important metric with many applications. The drawback, however, is the high computational cost of many basic problems involving this notion, such as the nearest neighbor search. A natural approach to overcoming this issue is to embed the sequences into a vector space such that the geometric distance in the target space approximates the edit distance in the original space. However, the known edit distance embedding algorithms, such as Chakraborty et al. (2016), construct embeddings that are data-independent, i.e., do not exploit any structure of embedded sets of strings. In this paper we propose an alternative approach, which learns the embedding function according to the data distribution. Our experiments show that the new algorithm has much better empirical performance than prior data-independent methods.

In this paper, the authors propose an approach for learning embeddings for strings using a three-phase approach. Each phase uses a different neural network to learn embeddings for strings such that the distance between strings in embedding space approximates the edit distance between the strings. A modest set of empirical results suggests that the proposed approach outperforms hand-crafted embeddings.
I found the presentation in this paper very disjointed and difficult to follow. While I believe (my interpretation of) the basic idea of this paper is interesting, I believe the current presentation significantly hinders readers from following the authors’ intentions.
Comments
The description of how the network structure and weights are “initialized” across the different phases is not clear. Different notation (f_1, f_2, f_3) is used for each network, but in reality, this is just the same network. However, the writing makes this very difficult to notice.
The authors introduce the CGK and CGK’ embedding algorithms, and then proceed to prove various properties about them. However, it is not clear to me how these theoretical properties are used by the neural network. From what I can tell, CGK’ is an alternative to CGK which reduces the output size relative to CGK (from 3n to at most 2n) while still ensuring exact reconstruction of the input. (I did not verify the proof in detail.) The authors then claim that this is helpful in the current context because it ensures the network parameters can be easily optimized. It is not clear to me what this means. (I guess that somehow using “CGK’ distance” makes training the model easier than using “CGK distance”.) Additionally, the experiments do not verify this claim empirically. So it is unclear whether using “CGK’ distance” helps in the context of learning embeddings.
It is really unclear to me whether the neural network outputs a continuous or a binary vector. In particular, Equations 5 - 8 all suggest that Hamming loss is defined on the outputs of the various neural networks (f_1, f_2, and f_3). The paper also refers to bits in the output of f_3. Later on, though, the paper mentions that the neural embedded strings are continuous vectors. While this could just be typos or inconsistent notation, considering that other parts of the paper do rely on binary representations, this makes the presentation very confusing.
It is unclear to me whether the can be (approximately) reconstructed from the embeddings. It seems that Theorem 1 suggests that the binary outputs of CGK’ can be decoded, but I cannot tell whether that extends to the embeddings.
It is unclear to me how positives and negatives are sampled for training in Phase 2, and also whether that impacts training.
The experimental results should include some measure of variance based on different train and/or test splits.
It seems as though the three phases could be rolled into a single multi-task learning problem in which the network is trained during a single phase.
Typos, etc.
The references are not consistently formatted.


Value-based reinforcement learning (RL) methods like Q-learning have shown success in a variety of domains. One challenge in applying Q-learning to continuous-action RL problems, however, is the continuous action maximization (max-Q) required for optimal Bellman backup. In this work, we develop CAQL, a (class of) algorithm(s) for continuous-action Q-learning that can use several plugand-play optimizers for the max-Q problem. Leveraging recent optimization results for deep neural networks, we show that max-Q can be solved optimally using mixed-integer programming (MIP). When the Q-function representation has sufficient power, MIP-based optimization gives rise to better policies and is more robust than approximate methods (e.g., gradient ascent, cross-entropy search). We further develop several techniques to accelerate inference in CAQL, which despite their approximate nature, perform well. We compare CAQL with state-of-the-art RL algorithms on benchmark continuous-control problems that have different degrees of action constraints and show that CAQL outperforms policy-based methods in heavily constrained environments, often dramatically.

Summary:
This paper targets the maximization issue in continuous value based methods, especially Q Learning. The idea is to use Mixed Integer Programming (MIP) to solve the Q maximization step by formulating the neural network structure of the Q function as a constrained mixed integer program. Further improvements are made by approximating the MIP solution in order to make training/inference faster. The method is tested on tasks from the Mujoco domain and compared with other value based methods for continuous control. I found the paper simple to follow and well structured. The problem is well motivated too and the empirical analysis is quite rigorous.
The obvious concerns are regarding scalability of the method; both in terms of 1) using other forms of neural network components (ex. Other activation functions, Convolutional networks in the case of vision based problems such as robotic manipulation) and 2) problems that are inherently less sample efficient, i.e. cases where shortening the sampling horizon is not a feasible option for learning meaningful policies.
Overall, I feel the positive aspects more or less outweigh the drawbacks and therefore my vote is for a weak accept.
Comments/Questions:
- Table 8 description says hyper-parameter sweeps were done for temperature and exploration noise decay values but the table is missing their values.
- What happens when action range is increased from default? One of the reasons mentioned for constraining the action space is to validate how well policy-based methods work. To really take this point home, I feel it might be good to check with an increased action range.
- Controlling w.r.t symmetry of the env (esp. in Mujoco domains), thus reducing the number of actions by half, might help faster MIP computation times.
- Figure 3 x-axis runs till different values, but the description says training steps are 1000. How long are the experiments run for?
- Can the authors elaborate on why the episode length is decreased from 1000 to 200?


The dominant approaches to sentence representation in natural language rely on learning embeddings on massive corpuses. The obtained embeddings have desirable properties such as compositionality and distance preservation (sentences with similar meanings have similar representations). In this paper, we develop a novel method for learning an embedding enjoying a dilation invariance property. We propose two algorithms: Orthrus, a classification algorithm, constrains the distribution of the embedded variable to be regularly varying, i.e. multivariate heavy-tail. and uses Extreme Value Theory (EVT) to tackle the classification task on two separate regions: the tail and the bulk. Hydra, a text generation algorithm for dataset augmentation, leverages the invariance property of the embedding learnt by Orthrus to generate coherent sentences with controllable attribute, e.g. positive or negative sentiment. Numerical experiments on synthetic and real text data demonstrate the relevance of the proposed framework.

The paper presented two methods for augmenting sentiment classification from the perspective of applying the Extreme Value Theory (EVT), including:
1) A classification algorithm which has an adversarial classifier to enforce the intermediate representations of a neural network to be similar to one EVT distribution, logistic distribution;
2) An encoder-decoder model that is able to generate grammatically coherent sentences with the same sentiment as the given input sentence.
Questions:
--
1) the Fisher–Tippett–Gnedenko theorem states that it is possible that the maximum value of a set of iid samples converges to one of three plausible distributions, and the chosen logistic distribution falls into the Weibull distribution category. I have a couple concerns about this choice:
1.1) In order to show that the EVT indeed helps empirically in the way that an adversarial classifier enforces the inf-norm of vectors follow the Generalised Extreme Value (GEV) distribution, at least three plausible distributions from each form of the GEV distribution needs to be checked. The logistic distribution is interesting, but the marginal improvement gained by enforcing the lengths of the produced vectors to follow the logistic distribution could be a result of hyper-param tuning, which shouldn't be a piece of supporting evidence.
1.2) From the perspective of applying the EVT, recent successful work from the best of my knowledge is on Anomaly Detection [1], where the EVT enables the system to learn from samples in only one class and also adjust the threshold for detecting the abnormal behaviour of samples. It is also theoretically grounded as the error variable of a logistic regression follows a Gumbel distribution which is one form of the GEV distribution, therefore, applying EVT for binary classification case makes sense.
1.3) From the perspective of learning representations with structured priors, there exists an interesting work on decomposing vector representations into lengths and directions and enforcing lengths to follow a uniform distribution and directions a Von Mises–Fisher (vMF) distribution as in [2]. It would be interesting to see if the proposed method is indeed better than the way that structured priors are enforced in [2].
1.4) Linguistically, given the distributional hypothesis, the length of learnt vectors tends to be highly correlated with the frequency information of available concepts and the direction of them matters more. The argument is also presented by the paper. However, in sentiment analysis, the length could contain the information about how strong the sentiment of the input sentence is, so I am not convinced that the proposed method would be applicable in fine-grained sentiment analysis, such as Stanford Sentiment Treebank [3].
--
2) A soft approximation over the inf-norm of a set of iid samples is log-sum-exp function, and it is the cdf of softmax function, which is also theoretically grounded in EVT for classifications. It could be a nicer story than the current one as the choice of the logistic distribution seems to be too intend.
--
3) The construction of the two datasets seems to be very arbitrary given that there exists a large number of sentiment analysis datasets and many with lots of samples, I am not sure that the results on the chosen constructed two datasets are sufficient enough to support the claim.
3.1) The size of the datasets is too small. Given that, the marginal improvement against the NN baseline could be a result of a specific initialisation, which doesn't generalise to other random initialisations.
3.2) The dimension of vector representations is also too small. Normally, commonly used word embeddings are of 300 dimensions, and contextualised ones are of higher than 1200 dimensions. The chosen 50 dimension could prevent the NN baseline model to perform well and IMO, it is helpful for picking a suitable logistic prior than it is in a very high dimensional space.
3.3) There are many straightforward distributions that could be applied as a prior on the lengths of vector representations, e.g. the Rayleigh distribution in 2D and the Chi-squared distribution in higher-dimension. Then again, the distribution gets flatter and becomes similar to a uniform distribution when the dim goes higher, which is a common issue. It goes back to my concern or doubt on the usability of a prior on the norm of high dimensional vectors.
--
4) I am still interested in seeing EVT being applied in various domains, but I'd be in favor of more justifiable approaches.
[1] Siffer, Alban, et al. "Anomaly detection in streams with extreme value theory." Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. ACM, 2017.
[2] Guu, Kelvin, et al. "Generating sentences by editing prototypes." Transactions of the Association for Computational Linguistics 6 (2018): 437-450.
[3] Socher, Richard, et al. "Recursive deep models for semantic compositionality over a sentiment treebank." Proceedings of the 2013 conference on empirical methods in natural language processing. 2013.


Adversarial examples of deep neural networks are receiving ever increasing attention because they help in understanding and reducing the sensitivity to their input. This is natural given the increasing applications of deep neural networks in our everyday lives. When white-box attacks are almost always successful, it is typically only the distortion of the perturbations that matters in their evaluation. In this work, we argue that speed is important as well, especially when considering that fast attacks are required by adversarial training. Given more time, iterative methods can always find better solutions. We investigate this speed-distortion trade-off in some depth and introduce a new attack called boundary projection (BP) that improves upon existing methods by a large margin. Our key idea is that the classification boundary is a manifold in the image space: we therefore quickly reach the boundary and then optimize distortion on this manifold.

This paper proposed an adversarial attack method based on optimization on the manifold. The authors claim it is a fast and effective attack even with quantization.
It would better to also evaluate the method on the state of the art robust models (such as Madry et al ICLR'18) instead of only testing it on natural models. Generating adversarial examples on natural models is rather a well-solved problem and I do not think a 0.1 decrease in L2 norm is a big contribution since it is already so small that humans cannot distinguish. A better way to prove the strength would be to test it on a robust model to achieve higher success rates given a maximum distortion.
I do not think the results in Table 3 are convincing or necessary. It is well-known that the FGSM is so weak that the adversarial examples produced by it are not strong enough for adversarial training. The state of the art adversarial training defense uses the adversarial examples obtained from PGD. Also, a popular way to evaluate model robustness would be to evaluate the attack success rate under a given upper bound of distortion (e.g. 0.3 for MNIST). If there is no constraint on the distortion, we can always achieve a 100% attack success rate by simply use an image from another class. So in Table 3, the authors may either make sure all attacks have a 100% success rate and compare the distortion, or set an upper bound of distortion and compare the success rate (just as in the operating characteristics plot). With the current results, I do not believe the robust training with BP can be any better than FGSM. Similar issues also exist in Table 2.


In anomaly detection (AD), one seeks to identify whether a test sample is abnormal, given a data set of normal samples. A recent and promising approach to AD relies on deep generative models, such as variational autoencoders (VAEs), for unsupervised learning of the normal data distribution. In semi-supervised AD (SSAD), the data also includes a small sample of labeled anomalies. In this work, we propose two variational methods for training VAEs for SSAD. The intuitive idea in both methods is to train the encoder to 'separate' between latent vectors for normal and outlier data. We show that this idea can be derived from principled probabilistic formulations of the problem, and propose simple and effective algorithms. Our methods can be applied to various data types, as we demonstrate on SSAD datasets ranging from natural images to astronomy and medicine, and can be combined with any VAE model architecture. When comparing to state-of-theart SSAD methods that are not specific to particular data types, we obtain marked improvement in outlier detection.

This paper presents two novel VAE-based methods for the (more general) semi-supervised anomaly detection (SSAD) setting where one has also access to some labeled anomalous samples in addition to mostly normal data. The first method, Max-Min Likelihood VAE (MML-VAE), extends the standard VAE objective that maximizes the log-likelihood for normal data by an additional term that in contrast minimizes the log-likelihood for labeled anomalies. To optimize the MML objective, the paper proposes to minimize the sum of the standard (negative) ELBO for normal samples and the so-called CUBO, which is a variational upper bound on the data log-likelihood, for anomalous samples. The second method, Dual Prior VAE (DP-VAE), modifies the standard VAE by introducing a second separate prior for the anomalous data, which is also Gaussian but has different mean. The DP-VAE objective then is defined as the sum of the two respective ELBOs which is optimized over shared encoder and decoder networks (with the adjustment that the outlier ELBO only updates the encoder). The anomaly score for both models then is defined as the (negative) ELBO of a test sample. Finally, the paper presents quite extensive experimental results on the benchmarks from Ruff et al. [2], CatsVsDogs, and an application of robotic motion planning which indicate a slight advantage of the proposed methods.
I am quite familiar with the recent Deep SAD paper [2] this work builds upon and very much agree that the (more general) SSAD setting is an important problem with high practical relevance for which there exists little prior work. Overall this paper is well structured/written and well placed in the literature, but I think it is not yet ready for acceptance due to the following key reasons:
(i) I think DP-VAE, the currently better performing method, is ill-posed for SSAD since it makes the assumption that anomalies are generated from one common latent prior and thus must be similar;
(ii) I think the worse performance of MML-VAE, which I find theoretically sound for SSAD, is mainly due to optimization issues that should be investigated;
(iii) The experiments do not show for the bulk of experiments how much of the improvement is due to meta-algorithms (ensemble and hyperparameter selection on a validation set with some labels).
(i) DP-VAE models anomalies to be generated from one common latent distribution (modeled as Gaussian here) which imposes the assumption that anomalies are similar, the so-called cluster assumption [2]. This assumption, however, generally does not hold for anomalies which are defined to be just different from the normal class but anomalies do not have to be similar to each other. Methodologically, DP-VAE is rather a semi-supervised classification method (essentially a VAE with Gaussian mixture prior having two components) which the paper itself points out is ill-posed for SSAD: “... the labeled information on anomalous samples is too limited to represent the variation of anomalies ... .” I suspect the slight advantage of DP-VAE might be mainly due to using meta-algorithms (ensemble, hyperparameter selection) and due to the rather structured/clustered nature of anomalies in the MNIST, F-MNIST, and CIFAR-10 benchmarks.
(ii) I find MML-VAE, unfortunately the worse performing method, to be a conceptually sound approach to SSAD following the intuitive idea that normal samples should concentrate under the normal prior whereas the latent embeddings of anomalies should have low likelihood under this prior. This approach correctly does not make any assumption on the latent structure of anomalies as DP-VAE does. I believe MML-VAE in its current formulation leads to worse results mainly to optimization issues that I suspect can be resolved and should be further investigated. I guess the major issue of the MML-VAE loss is that the log-likelihood for outlier samples has steep curvature and is unbounded from below. Deep networks might easily exploit this without learning meaningful representations as the paper also hints towards. This also results in unstable optimization. I think removing the reconstruction term for outliers, as the paper suggests, also helps for this particular reason but this is rather heuristic. These optimization flaws should be investigated and the loss adjusted if needed. Maybe simple thresholding (adding an epsilon to lower bound the loss), gradient clipping, or robust reformulations of the loss could improve optimization already?
(iii) To infer the statistical significance of the results and to assess the effect of meta-algorithms (ensemble, hyperparameter tuning) an ablation study as in Table 4 (at least on the effect of ensembling) should be included also for the major, more complex datasets. Which score is used for hyperparameter selection (ELBO, log-likelihood, AUC)? How would the competitors perform under similar tuning?
####################
*Additional Feedback*
*Positive Highlights*
1. Both proposed methods can be used with general data types and VAE network architectures (the existing Deep SAD state-of-the-art method employs restricted architectures).
2. The paper is well placed in the literature and all major and very recent relevant work that I am aware of are included.
3. This is an interesting use of the CUBO bound which I did not know before reading this work. This might be interesting for the general variational inference community to derive novel optimization schemes.
4. I found the robotic motion planning application quite cool. This also suggests that negative sampling is useful beyond the AD task.
5. I appreciate that the authors included the CatsVsDogs experiment although DADGT performs better as it demonstrates the potential of SSAD. I very much agree that employing similar self-supervised learning ideas and augmentation is a promising direction for future research.
*Ideas for Improvement*
6. Extend the semi-supervised setting to unlabeled (mostly normal), labeled normal, and labeled anomalous training data. The text currently formulates a setting with only labeled normal and labeled anomalous samples. A simple general formulation could just assign different weights to the unlabeled and labeled normal data terms.
7. There might be an interesting connection between MML-VAE and Deep SAD in the sense that MML-VAE is a probabilistic version of the latter. The $\chi_n$ distance of the CUBO loss has terms similar to the inverse squared norm penalty of Deep SAD.
8. Report the range from which hyperparameters are selected.
9. Add the recently introduced MVTec AD benchmark dataset to your experimental evaluation [1].
10. Run experiments on the full test suite of Ruff et al. [2]. At the moment only one of three scenarios are evaluated.
*Minor comments*
11. Inconsistent notation for the expected value ($\mathbb{E}$ vs $\mathbf{E}$)
12. In Section 3, the parameterization of the variational approximate $q(z | x)$ is inconsistently denoted by $\phi$ and $\theta$ (which beforehand parameterizes the decoder).
13. In Section 3.2, the current formulation first says that MC produces a biased, then an unbiased estimate of the gradients.
14. First sentence in Section 4: I would not use “classify” but rather “detect” etc. for anomaly/novelty detection since the task differs from classification.
15. In Section 4.2, there should be a minus in front of the KL-divergence terms of the $ELBO_{normal}$ and $ELBO_{outlier}$ equations.
16. In the fully unsupervised setting on CIFAR-10 (Table 5), why is the VAE performance essentially at random (~50) in comparison to CAE and Deep SVDD although they use the same network architecture?
17. Is the CUBO indeed a strictly valid bound if one considers the non-normal data-generating distribution?
18. Are there any results on the tightness of the CUBO?
####################
*References*
[1] P. Bergmann, M. Fauser, D. Sattlegger, and C. Steger. Mvtec ad–a comprehensive real-world dataset for unsupervised anomaly detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 9592–9600, 2019.
[2] L. Ruff, R. A. Vandermeulen, N. Görnitz, A. Binder, E. Müller, K.-R. Müller, and M. Kloft. Deep semi-supervised anomaly detection. arXiv preprint arXiv:1906.02694, 2019.


It has been established that diverse behaviors spanning the controllable subspace of a Markov decision process can be trained by rewarding a policy for being distinguishable from other policies (Gregor et al., 2016;Eysenbach et al., 2018;Warde-Farley et al., 2018). However, one limitation of this formulation is the difficulty to generalize beyond the finite set of behaviors being explicitly learned, as may be needed in subsequent tasks. Successor features (Dayan, 1993;Barreto et al., 2017) provide an appealing solution to this generalization problem, but require defining the reward function as linear in some grounded feature space. In this paper, we show that these two techniques can be combined, and that each method solves the other's primary limitation. To do so we introduce Variational Intrinsic Successor FeatuRes (VISR), a novel algorithm which learns controllable features that can be leveraged to provide enhanced generalization and fast task inference through the successor features framework. We empirically validate VISR on the full Atari suite, in a novel setup wherein the rewards are only exposed briefly after a long unsupervised phase. Achieving human-level performance on 12 games and beating all baselines, we believe VISR represents a step towards agents that rapidly learn from limited feedback.

The authors address the problem of finding optimal policies in reinforcement learning problems after an initial unsupervised phase in which the agent can interact with the environment without receiving rewards.  After this initial phase, the agent can again interact with the environment while having access to the reward function. To address this specific setting, the authors propose to use the successor feature representation of policies and combine it with methods that estimate policies in the unsupervised setting (without a reward function) by maximizing the mutual information of a policy-conditioning variable and the agent behaviour. The result is a method called Variational Intrinsic Successor Features (VISF) which obtains significant performance gains on the full Atari task suite in a setting of few-step RL with unsupervised pre-training. The main contribution seems to parameterize the successor features in terms of a variable specifying the policy. This variable will be the same as the linear weights in the linear model for the reward assume by the successor features representation. Finally, a discriminator aims to predict the linear weights of the policy from the observed state-feature representation.
Clarity:
I think this is one of the weaknesses of the paper. The writing and clarity could be significantly improved. How do you go from equation 8 to equation 9? How does q lower boun p? In the paragraph after equation 9 the authors mention the score function estimator. But very few details are given. After reading the paper, my impression is that the reproducibility of the results could be very hard, because of the lack of details of the specific implementation. The authors also do not mention that the code will be publicly available after acceptance.
I feel that, to better understand the method, the authors should include experiments in simple and easy to understand synthetic environments which can be more illustrative than ATARI.
What is the difference between the policy parameters theta and the conditioning variable z?
Novelty:
The proposed approach is novel up to my knowledge. I find the idea of parameterizing the successor features in terms of the policy parameters very innovative.
Quality:
The proposed approach seems well justified and the experiments performed indicate that the method can be useful in practice. However, I think it would be very useful to have results on other environments besides ATARI. For example,  DIAYN contains experiments on a wide range of tasks, including gridworld style tabular experiments to illustrate what their method does. This work would benefit from similar simpler and easier to understand synthetic environments (unlike ATARI).
Significance:
The proposed contribution seems significant as illustrated by the experimental results and the novel methodological contributions. However, the lack of clarity and the difficulty in the reproduction of the results limit this.
Some minor comments:
I recommend the authors to remove references in the abstract.
Update after the authors' rebuttal:
After looking at the response from the authors, I believe that they have successfully addressed my concerns.
Therefore, I have decided to update my rating and vote for acceptance. I am looking forward to seeing the python notebook with the implementation of the VISR algorithm.


All living organisms struggle against the forces of nature to carve out niches where they can maintain relative stasis. We propose that such a search for order amidst chaos might offer a unifying principle for the emergence of useful behaviors in artificial agents. We formalize this idea into an unsupervised reinforcement learning method called surprise minimizing RL (SMiRL). SMiRL trains an agent with the objective of maximizing the probability of observed states under a model trained on all previously seen states. The resulting agents acquire several proactive behaviors to seek and maintain stable states such as balancing and damage avoidance, that are closely tied to the environment's prevailing sources of entropy, such as winds, earthquakes, and other agents. We demonstrate that our surprise minimizing agents can successfully play Tetris, Doom, and control a humanoid to avoid falls, without any task-specific reward supervision. We further show that SMiRL can be used together with a standard task reward to accelerate reward-driven learning.

Summary
This paper proposes a novel form of surprise-minimizing intrinsic reward signal that leads to interesting behavior in the absence of an external reward signal. The proposed approach encourages an agent to visit states with high probability / density under a parametric marginal state distribution that is learned as the agent interacts with its environment. The method (dubbed SMiRL) is evaluated in visual and proprioceptive high-dimensional "entropic" benchmarks (that progress without the agent doing anything in order to prevent trivial solutions such as standing and never moving), and compared against two surprise-maximizing intrinsic motivation methods (ICM and RND) as well as to a reward-maximizing oracle. The experiments demonstrate that SMiRL can lead to more sensible behavior compared to ICM and RND in the chosen environments, and eventually recover the performance of a purely reward-maximizing agent. Also, SMiRL can be used for imitation learning by pre-training the parametric state distribution with data from a teacher. Finally, SMiRL shows the potential of speeding up reinforcement learning by using intrinsic motivation as an additional reward signal added to the external task-defining reward.
Quality
As a practical paper, this work needs to be judged based on the quality of the experiments. I find the number of benchmarks and baselines sufficient. One major issue, that currently prevents me from voting for acceptance, is that experiments have not been conducted with enough seeds. I couldn't find any information in the paper regarding how many repetitions there are for each experiment. Also, most figures do not indicate any uncertainty measures (standard deviation, percentiles or the like)---some do, e.g. Figure (4), but it is not mentioned what type of uncertainty is depicted. One seed is certainly not enough to support the claims made by the authors---especially not the one that SMiRL can help improve RL in Figure (5). Figure (5a) clearly  does not draw a clear picture under one seed, and (5b) and (5c) require additional expert demonstrations. If experiments are repeated with more seeds and still support the claims, I am happy to increase my score to acceptance (presupposing that the discussion phase does not prevent acceptance for other reasons).
Clarity
The paper is clearly written and easy to follow. However, I do not like one aspect in the way the authors motivate their approach. The problem formulation starts with an MDP formulation. MDPs rely on a stationary reward signal and RL agents aim to optimize for future cumulative rewards based on these stationary reward signals. The authors propose to optimize for a non-stationary signal since the parametric state distribution changes over time. This itself is not uncommon and nothing controversial from a practical perspective. However, the statement that SMiRL agents seek to visit states that will change the parametric state distribution to obtain higher intrinsic reward in the future is controversial (see e.g. at the end of Section 2.1), because optimizing a non-stationary signal is outside the scope of the problem formulation. The reinforcement learning problem of maximizing intrinsic rewards does not know how the intrinsic reward signal is altered in the course of the future, i.e. how the parametric state distribution is updated. These statements should therefore be either adjusted accordingly, or the claims should be backed up theoretically rather than intuitively. On a minor note, I don't think that Figure (1) is necessary and the quote at the beginning of the paper might be better suited for a book chapter (but that is just my personal opinion).
Originality
I find the simple idea presented by the authors to minimize rather than maximize surprise quite original. However, I do not have much experience in the domain of intrinsic motivation and leave the judgement of originality to the other reviewers and the area chair. Also, some references might be missing regarding learning with intrinsic reward (e.g. empowerment).
Significance
The fact that a simple intrinsic reward, as presented by the authors, can lead to interesting behavior, as demonstrated by the experiments, is quite significant. Unfortunately, the experiments are not significant from a statistical perspective which is why I do not recommend acceptance at this stage (as mentioned above, depending on how the authors address this issue and the discussion period, I might change to acceptance).
Update
The authors have addressed my main concern regarding missing seeds. I therefore change to weak accept. But I am not happy how the authors responded to my concern regarding their motivation:
1.) The formulation that tries to justify their reasoning as given in the rebuttal was absent in the first version of this paper, but something along this line would have been necessary since the argument relies on a non-standard MDP formulation.
2.) I don't think the argumentation given in the rebuttal is correct. Imagine hypothetically the agent is in exactly the same augmented state s_t, a_t, \theta_t at two different points in time, e.g. in the first episode and after multiple episodes. In both cases, the collected states seen so far are going to be different, hence the optimization objective for learning a marginal state distribution is different, hence the parameter updates are different, hence the transitions are not stationary.
I do therefore encourage the authors to attenuate their wording.


Model-based reinforcement learning (MBRL) with model-predictive control or online planning has shown great potential for locomotion control tasks in both sample efficiency and asymptotic performance. Despite the successes, the existing planning methods search from candidate sequences randomly generated in the action space, which is inefficient in complex high-dimensional environments. In this paper, we propose a novel MBRL algorithm, model-based policy planning (POPLIN), that combines policy networks with online planning. More specifically, we formulate action planning at each time-step as an optimization problem using neural networks. We experiment with both optimization w.r.t. the action sequences initialized from the policy network, and also online optimization directly w.r.t. the parameters of the policy network. We show that in the MuJoCo benchmarking environments, POPLIN is about 3x more sample efficient than the previously stateof-the-art algorithms, such as PETS, TD3 and SAC. To explain the effectiveness of our algorithm, we show that the optimization surface in parameter space is smoother than in action space. Further more, we found the distilled policy network can be effectively applied without the expansive model predictive control during test time for some environments such as Cheetah. Code is released 1 .

Summary
This work provides a novel model-based reinforcement learning algorithm for continuous domains (Mujoco) dubbed POPLIN. The presented algorithm is similar in vein to the state-of-the-art PETS algorithm, a planning algorithm that uses state-unconditioned action proposal distributions to identify good action sequences with CEM in the planning routine. The important difference compared to PETS is the incorporation of a parametric state-conditioned policy (trained on real data) in the planning routine to obtain better action-sequences (CEM is used to learn the "offset" from the parametric policy). The paper presents two different algorithmic ablations where CEM either operates in action space or parameter space (POPLIN-A and POPLIN-P respectively), in combination with different objectives to learn the parametric policy. The method is evaluated on 12 continuous benchmarks and compared against state-of-the-art model-based and model-free algorithms, indicating dominance of the newly proposed method.
Quality
The quality of the paper is high. This is an experimental study and the number of benchmarks and baselines is far above average compared to other papers in that field. One minor point is that averaging experiments over 4 seeds only is usually not optimal in these environments, but in light of the sheer amount of baselines and benchmarks excusable. While the experimental results are impressive, the authors mention that asymptotic performance in Walker2d and Humanoid might not match the asymptotic performance of model-free baselines. This could be stated more clearly. Also, there are no Humanoid experiments in the paper despite mentioned in the text (2nd paragraph in Section 5.1)?
Clarity
The clarity of the paper can be in parts improved upon. For example, how does the "policy control" ablation (mentioned in Section 4.3) work precisely, i.e. the interplay between executing the parametric policy in the real world and harnessing the environment model? I assume the policy distillation techniques in Section 4.4 are different alternatives for the second-to-last lines in the pseudocodes from the appendix? Which one is the default used in Section 5.1? On a minor note, above Equation (7), a target network is mentioned---where does the target network occur in Equation (7)? There are some plots that do not mention the name of the environment, e.g. in Figure (4), but also some in the appendix. Furthermore, it could be stated more clearly that the reward function is assumed to be known. If the authors improve the clarity of their paper significantly, I am willing to increase my score further (presupposing that no severe issues arise in the discussion phase).
Originality
Adding a parametric policy to PETS is not the most original idea, but clearly a gap in the current literature.
Significance
The experiments and the empirical results make the paper quite significant.
Update
The authors improved the clarity of the paper. I therefore increase to 8. Section 4.4 paragraph "Setting parameter average (AVG)" can still be improved---does this go together with POPLIN-P-Uni from Section 4.2?


We introduce a novel end-to-end approach for learning to cluster in the absence of labeled examples. Our clustering objective is based on optimizing normalized cuts, a criterion which measures both intra-cluster similarity as well as inter-cluster dissimilarity. We define a differentiable loss function equivalent to the expected normalized cuts. Unlike much of the work in unsupervised deep learning, our trained model directly outputs final cluster assignments, rather than embeddings that need further processing to be usable. Our approach generalizes to unseen datasets across a wide variety of domains, including text, and image. Specifically, we achieve state-of-the-art results on popular unsupervised clustering benchmarks (e.g., MNIST, Reuters, CIFAR-10, and CIFAR-100), outperforming the strongest baselines by up to 10.9%. Our generalization results are superior (by up to 21.9%) to the recent top-performing clustering approach with the ability to generalize.

This paper proposes a new clustering method, called CNC, which is composed of two-step procedures.
It first embeds an input dataset into a d-dimensional space, followed by performing relaxed normalized cut to detect clusters.
Although the contribution of introducing a new relaxed formulation of the normalized cut is interesting, I have the following concerns regarding with the clarity, significance, and evaluation of the proposed method.
- The paper is not clearly written at many points and the quality of presentation is not high, which also deteriorates the significance of the paper.
    In particular, the optimization process for clustering discussed in Section 4.1 is not clearly presented.
    Although the objection function, which is the expectation of the Ncut, is introduced in Equation (6), how to solve it is not presented.
    Since this is the key step for CNC, it should be carefully discussed.
- In the embedding step, how to choose the dimensionality d?
    This is not even reported in experiments.
- Empirical evaluation is not thorough and important evaluation is missing.
    * First, the contribution of embedding is not evaluated.
      The performance between CNC with the proposed embedding and without it should be compared.
      Moreover, the sensitivity of the performance with respect to changes in d should be examined.
    * A number of resulting scores are missing; in particular, CNC is compared to only SpectralNet for CIFAR-10 and CIFAR-100 under NMI.
      It would be more convincing if the NMI for other methods are also reported.
- Parameter sensitivity is not evaluated, while there are a number of parameters in the proposed method as reported in Section 5.7.
    Since parameter tuning is fundamentally difficult in the unsupervised setting, parameter sensitivity is crucial.
    Also how to choose such parameters is not clear.
Minor comments:
- In Algorithm 1, line 1, "X \in R^n" -> "X \subseteq R^m"?
- In Algorithm 1, the dimensionality "m" of data points and the batch size "m" are the same. Is it correct?
- At the first line in Section 4.1: "for each data point" -> "For each data point"
- P.4, L.-4: "nreast-neighbor" -> "nearest-neighbor"


Learning disentangled representations that correspond to factors of variation in real-world data is critical to interpretable and human-controllable machine learning. Recently, concerns about the viability of learning disentangled representations in a purely unsupervised manner has spurred a shift toward the incorporation of weak supervision. However, there is currently no formalism that identifies when and how weak supervision will guarantee disentanglement. To address this issue, we provide a theoretical framework to assist in analyzing the disentanglement guarantees (or lack thereof) conferred by weak supervision when coupled with learning algorithms based on distribution matching. We empirically verify the guarantees and limitations of several weak supervision methods (restricted labeling, match-pairing, and rank-pairing), demonstrating the predictive power and usefulness of our theoretical framework.

The paper tries to bring some theoretical foundation to the weakly supervised disentanglement. Overall it is a good contribution, but the message of the paper is not clear.  The authors propose two notions: consistency and restrictiveness, which they don't imply each other. However, the experiment on real data shows that they are highly correlated.   Up until the experiment section, the paper is well written (although a bit verbose). It seems that it is great but unfinished work.
The paper is well written, but in my opinion, there is too much verbosity on page 4-5 on rather trivial definitions consistency and restrictiveness and a big box in the calculus of disentanglement that steals space from the main results.  In my opinion, those sections can be reduced so that other theorem can be covered. In my opinion, the theorem nine should be part of the main text.
I understand the definition of "Sufficiency for Disentanglement " but it is not clear why it is important. Sure, it is a strong definition that says for any  $\mathcal{H}$ (and not a subset) the algorithm ($\mathcal{A}$) should be able to match the distribution of the observation but why is it a big deal according to the next paragraph?
I don't see any proof that Eq.11 should be between [0,1]. Yes, g is optimal, and if you enter suboptimal values to it, one expects the nominator to be less than dominator. However, g a function that is optimal in expectation, which does not mean for every s value it nominator is less than the denominator. In fact, some of the values in fig 3 are small negatives.
Fig 3 is not explained well: you are showing normalized consistency and restiveness. First of all, what is the dataset you tried this on? Second, why some values are negative?! These are supposed to be between [0,1]. Third, what is the take-home-message of this figure? the first two matrices from left show that the factors are consistent b/c they are almost diagonal. The third one from left shows that the algorithm you used is not restrictive? Then are you suggesting this as a metric of evaluation? I am not sure I understand the first figure from the right.
Overall, the authors perform a significant amount of experiments, but they did a poor job in summarizing the results.
Finally, the authors claim
                  "...We believe this correlation between consistency and restrictiveness to have been a general source of confusion in the disentanglement literature, causing many to either observe or believe that restricted labeling or share pairing on $S_i$ (which only guarantees consistency) is sufficient for disentangling Si ..."
Each of those methods should be analyzed separately to ensure that their algorithms do not induce restiveness. I just don't see the natural connection between your figure 4 and this conclusion that you made.
Minor:
Where is the proof for Theorem 1? In the Supp, it starts with Theorem 8, I guess you meant Lemma 8? You need to clean up the Supp so that one can find the proof easily. I suggest restructuring the Supp to less and finally proof of Thorem 1.


We introduce a new deep learning technique that builds individual and class representations based on distance estimates to contextual dimensions for different modalities. Recent works have demonstrated advantages to creating representations from probability distributions over their contexts rather than single points in a low-dimensional Euclidean vector space. These methods, however, rely on preexisting features and are limited to textual information. In this work, we obtain generic template representations that are vectors containing the average distance between the distribution of a class and that of contextual information, the latter of which consists of a selection of a thresholded number of classes from the power set of classes. These representations have the benefit of being both interpretable and composable. They are initially learned by estimating the Wasserstein distance for different data subsets with deep neural networks. Individual samples or instances can then be compared to the generic class representations, which we call templates, to determine their similarity and thus class membership. We show that this technique, which we call WDVec, delivers good results for multi-label image classification. Additionally, we illustrate the benefit of templates and their composability by performing retrieval with complex queries where we modify the information content in the representations. Our method can be used in conjunction with any existing neural network and create theoretically infinitely large feature maps.

The paper defines a representation learning strategy based upon estimation
of a matrix of Wasserstein distances.
The idea is excellent.  The ability to "solve" IPMs reliably is a recent
development in deep learning whose ramifications are still being explored.
Intuitively this line of research could plausibly result in general
methods which are theoretically intelligible and broadly applicable.
Indexing at least one side of the matrix of estimated WDs with events
(rather than classes) has interpretability properties useful for
information retrieval and also conveys benefits reminiscent of learning
with privileged information.
However, the exposition could be greatly improved by using the
standard language of probability theory.  The discussion in 3.1
was particularly painful to read.  What is the difference between
"existing in an environment" and "conditioning on a measurable event"?
Phrases like "belonging to any random subset of the dataset" suggest
a non-deterministic method of selecting an element of the power set of
the training data, but it is unclear what to do if more training data
arrives in this case.
Throughout the entire paper the word "random" is apparently used in the
colloquial sense of "arbitrary".  *Correct every instance of this.*
If you actually are referring to generating samples from a distribution,
be explicit about the generative process.
Section 3.5 was more confusing than enlightening.  In general I understand
that environments can be leveraged for intelligibility and admit manipulation
for information retrieval.  The exact strategy remains somewhat opaque.  If
you are under space constraints refer to an appendix with more explicit details.
In the experiments section phrases like "environments consist of random
combinations of classes" is also not helpful.  Do you mean something like
"uniformly selected from the set of all class pairs?"  Or something like
"uniformly selected from the power set of all classes?"  How volatile
are the experimental results with respect to the non-deterministic choice
of environments?
I want to accept this paper if the exposition is improved, which I think
is possible during the response period.
My other comments are not blocking issues, but would either improve the
current paper or inform future directions of research.
The technique bears some resemblance to Wasserstein Discriminant
Analysis.[1]  That paper seeks a projection that maximizes the ratio of
Wasserstein distance between classes vs. within classes.  Here,
although the common representation is a nonlinear mapping
analogous to a projection, we merely try to estimate all the
Wasserstein distances rather than maximize them, so it is not trained
to be discriminative per se.   That is ok since the representation is
designed to be used for a variety of tasks (modulo section 4.2), but it
does leave open the question "what if the matrix of estimated
Wasserstein distances isn't informative, e.g., due to poor choice of
environments?"  There is no attempt to assess the representation
except via utility in downstream tasks.
The common representation was justified computationally, but I suspect
is beneficial statistically.  It might facilitate safely including a
large number of environments and then spectrally compressing (i.e., SVD)
the resulting matrix without overfitting the data.  However clearly if
the capacity of this layer is too small, then all estimated WDs will
be close to zero.  If we posit a low Bayes error classifier for the
multi-class problem associated with the dataset, that might imply there
is some conditioning of the input under which the matrix of (actual) WDs
has rank equal to the number of classes, which would in turn provide a
useful diagnostic to guard against an insufficiently discriminative choice
of environments or insufficient capacity in the common representation. If
the matrix is full rank with a flat spectrum, however, that might indicate
the choice of environments is too granular and overfitting has occurred,
it's not immediately obvious to me how to guard against this.
I am curious what the results in appendix A.1. look like relative to the spectral
norm or the smallest eigenvalue of the estimated WD matrix (smallest
eigenvalue assuming number of environments < number of classes,
otherwise the k-th eigenvalue where k = number of classes).
[1] https://arxiv.org/abs/1608.08063


The worst-case training principle that minimizes the maximal adversarial loss, also known as adversarial training (AT), has shown to be a state-of-the-art approach for enhancing adversarial robustness against norm-ball bounded input perturbations. Nonetheless, min-max optimization beyond the purpose of AT has not been rigorously explored in the research of adversarial attack and defense. In particular, given a set of risk sources (domains), minimizing the maximal loss induced from the domain set can be reformulated as a general min-max problem that is fundamentally different from AT since the maximization is taken over the probability simplex of the domain set. Examples of this general formulation include attacking model ensembles, devising universal perturbation under multiple inputs or data transformations, and generalized AT over different types of attack models. We show that these problems can be solved under a unified and theoretically principled min-max optimization framework. We also show that the self-adjusted domain weights learnt from our method provide a holistic tool to explain the difficulty level of attack and defense over multiple domains. Extensive experiments show that our approach leads to substantial performance improvement over the conventional heuristic strategies 1 .

Note: I applied a higher standard to this paper given that it significantly exceeds the recommended page limit. Furthermore, important details are left out in the appendices, which make it difficult to read the main body of the paper in a self-contained fashion. Given that the main body was already over the recommended page limit, I did not read the appendices.
This paper generalizes the min max formulation of adversarial training, and proposes a formulation that encompasses adversarial training of an ensemble, robustness to universal adversarial examples, and robustness to non-adversarial transformations. This formulation is used to derive an adversarial training procedure that trains against the worst-case adversarial example among adversarial examples generated by a set of attacks. Experiments seek to demonstrate applicability of this framework to both attacks and defenses.
As far as experiments are concerned, Section 4.1 presents results on MNIST, which is known to be a poor dataset to study adversarial examples on [https://arxiv.org/abs/1902.06705]. If models C and D are more difficult to attack, could better baselines be employed than attacking the ensemble A+B+C+D? For instance, would an adversary evading models C+D only perform better? It is difficult to draw insights that are generally applicable from a single ensemble. How was the ensemble chosen? Why would a defender add models which are known to be significantly less robust to the ensemble?
When discussing universal perturbations, how are they generated? Given that the performance of the proposed approach significantly degrades average evasion across all images from all groups, what is the threat model for an adversary being interested in group-level success rather than average evasion across all images from all groups? How were the values of K chosen? This comment also 	applies to experiments over data transformations. For these experiments, what was the value of K?
As far as the defensive perspective is concerned, it is not clear whether the improvements observed are statistically significant. Were multiple runs averaged to produce Table 4? Given that without DPAR, the improvement is negligible, this is important to interpret results. It appears that most of the robustness gains in both the average and max settings stem from DPAR. This should be clearly surfaced in the introduction and presentation of contributions if DPAR is required for the proposed generalized min max formulation to improve robustness. In particular, it is not clear whether DPAR is “a beneficial supplement to adversarial training” or a required supplement to adversarial training - per the formulation in this paper.
There are issues with grammar throughout the document, which make it difficult to read. Some specific issues:
1 - Adversarial attack is a tautology (an attack is always adversarial)
7 - What does “robust” adversarial attack mean?
7 - What is CAAD-18?
7 - Define ASR_all: what does evade mean here? Is the attack targeted or untargeted?
7 - What is an “advanced” DNN?


In recent years, substantial progress has been made on graph convolutional networks (GCN). In this paper, for the first time, we theoretically analyze the connections between GCN and matrix factorization (MF), and unify GCN as matrix factorization with co-training and unitization. Moreover, under the guidance of this theoretical analysis, we propose an alternative model to GCN named Cotraining and Unitized Matrix Factorization (CUMF). The correctness of our analysis is verified by thorough experiments. The experimental results show that CUMF achieves similar or superior performances compared to GCN. In addition, CUMF inherits the benefits of MF-based methods to naturally support constructing mini-batches, and is more friendly to distributed computing comparing with GCN. The distributed CUMF on semi-supervised node classification significantly outperforms distributed GCN methods. Thus, CUMF greatly benefits large scale and complex real-world applications.

The work poses an interesting question: Are GCNs (and GNNs) just special types of matrix factorization methods? Unfortunately, the short answer is **no**, which goes against what the authors say.
Until recently I thought like the authors, but the concurrent work [1] (On the Equivalence between Node Embeddings and Structural Graph Representations) https://openreview.net/forum?id=SJxzFySKwH changed my mind.
The work of (Li et al., 2018) shows that nearby nodes tend to get similar representations. There is mounting experimental evidence of that being the case in real-world graphs (e.g., https://arxiv.org/abs/1908.08572). But [1] shows that GCNs and GNNs are fundamentally different from matrix factorization methods, regardless of the loss function used to learn the embeddings. Consider Figure 1 in [1], and it is easy to see that matrix factorization will give different embeddings to the Lynx and Orca nodes, while GCNs and GNNs must give the same embedding. Even if we connect the graphs through the Spruce and the Zooplankton nodes, their conclusion would not change. Matrix factorization (as broadly understood) will give embeddings that can even be used to cluster nodes. The eigenvectors of the symmetric Laplacian encode the diffusion of a type of random walk and nodes that are far away in the graph must have different embeddings (because through the diffusion operator, they are far away).
In GCNs, the convergence of the embeddings is better explained by the mixing of a random walk (Theorem 1 of (Xu et al., 2018)), which, in the special case of a GCN, converges to 1/sqrt(degree of node), as shown by (Li et al., 2018) in their Theorem 1 for the symmetric Laplacian. This is unrelated to what we get in matrix factorization as explained earlier.
What is wrong with the math: Equation (11) is equated with matrix factorization, but note that it does not account for nonedges, while matrix factorization accounts for nonedges. This issue is more clear in Equation (14). The problem happens when the paper jumps from Equation (14), which is correct but not MF, to Equation (15) which is MF but unrelated to Equation (14). The argument is that “negative edges sampling is used, for better convergence”… sorry, not for better convergence, it completely changes the optimization objective. Hence, GCNs are not matrix factorization methods.
I think the paper is a valiant effort, but unfortunately the core premise is incorrect. The jump from Equation (14) to equation (15) cannot be justified, and I believe showcases a fundamental flaw the argument. I do not see a way to fix the paper. I vote to reject it.
[1] On the Equivalence between Node Embeddings and Structural Graph Representations, https://openreview.net/forum?id=SJxzFySKwH
Xu, K., Li, C., Tian, Y., Sonobe, T., Kawarabayashi, K.I. and Jegelka, S., 2018. Representation learning on graphs with jumping knowledge networks. ICML 2018.
Li, Qimai, Zhichao Han, and Xiao-Ming Wu. Deeper insights into graph convolutional networks for semi-supervised learning. AAAI, 2018.
--------------
Read rebuttal. Will keep my original assessment.


Neural inductive program synthesis is a task generating instructions that can produce desired outputs from given inputs. In this paper, we focus on the generation of a chunk of assembly code that can be executed to match a state change inside the CPU and RAM. We develop a neural program synthesis algorithm, AutoAssemblet, learned via self-learning reinforcement learning that explores the large code space efficiently. Policy networks and value networks are learned to reduce the breadth and depth of the Monte Carlo Tree Search, resulting in better synthesis performance. We also propose an effective multi-entropy policy sampling technique to alleviate online update correlations. We apply AutoAssemblet to basic programming tasks and show significant higher success rates compared to several competing baselines.

[Summary]
This paper addresses the problem of synthesizing programs (x86 assembly code) from input/output (I/O) pairs. To this end, the paper proposes a framework (AutoAssemblet) that first learns a policy network and a value network using imitation learning (IL) and reinforcement learning (RL) and then leverages Monte Carlo Tree Search (MCTS) to infer programs. The experiments show that AutoAssemblet can synthesize assembly programs from I/O pairs to some extent. Ablation studies suggest the proposed IL and RL guided search is effective.
Significance: are the results significant? 2/5
Novelty: are the problems or approaches novel? 2/5
Evaluation: are claims well-supported by theoretical analysis or experimental results? 4/5
Clarity: is the paper well-organized and clearly written? 4/5
[Strengths]
*clarity*
The overall writing is clear. The authors utilize figures well to illustrate the ideas. Figure 1 clearly shows the proposed pipeline as well as the MCTS process. In general, the notations and formulations are well-explained.
*technical contribution*
- Optimizing both the imitation learning loss and the reinforcement learning loss yields better performance when more tasks are available and tasks are more difficult.
- Leveraging a learned policy network and value network for improving the efficiency of the MCTS seems effective.
*ablation study*
Ablation studies are comprehensive. The proposed framework first optimizes two losses (IL and RL) and leverages the learned policy network and the value network for improving MCTS. The provided ablation studies help analyze the effectiveness of each of them.
*experimental results*
- All the descriptions of the experiments and the presentations of the results are fairly clear.
- The results demonstrate the effectiveness of the proposed RL guided MCTS.
[Weaknesses]
*novelty*
Overall, I do not find enough novelty from any aspects while the overall effort of this paper is appreciated. The reasons are as follows.
- This  "self-learning" framework is not entirely novel since it has been proposed in [1], where the model is trained on a large number of programs that were randomly generated and tested on a real-world dataset (FlashFill).
- The hybrid objective (IL+RL) has been explored in neural program synthesis [2] (a supervised learning model is fine-tuned using RL), learning robot manipulation [3], character control [4], etc.
- Utilizing Monte Carlo Tree Search for program synthesis has been studied in many works. [5] proposes to treat the network outputs as proposals for a sequential Monte Carlo sampling scheme and [6] presents an RL guided Monte Carlo tree search framework.
- Program synthesis on assembly languages: RISC-V [6], etc.
*related work*
The descriptions of the related work are not comprehensive. While many neural synthesis works [1, 5, 7-10] have explored a wide range of settings for learning to perform program synthesis, they are not mentioned in the paper. I suggest the authors conduct a comprehensive survey on this line of works.
*baselines*
In my opinion, the baselines (imitation, REINFORCE, MCTS) presented in the paper are far from comprehensive. I believe the following baselines should also be considered:
- As the proposed model optimizes a combination of the IL loss and the RL loss, it would make sense to also evaluate a model optimizing this hybrid loss.
- Search-based program synthesis baseline (i.e. learning guided search vs heuristic search)
- Comparing the proposed framework against some neural induction baselines would confirm the importance and effectiveness of explicitly synthesizing programs instead of directly predicting the outcome/output. This has been shown in [1, 7].
*testing set*
The testing sets are extremely small (with only 50, 40, 40 programs), which makes the results less convincing. Also, how those testing sets were created is not mentioned anywhere in the paper. It only states "we designed a set of human-designed tasks".
*number of given I/O pairs*
It is not mentioned anywhere how the authors split the observed I/O pairs and the assessment I/O pairs such as what has been done in most of the works [1, 2, 7, 8]. While the observed I/O pairs are input to the program synthesis framework, the assessment I/O pairs are used to evaluate the synthesized programs. By doing so, the more observed I/O pairs are given, the more accurate the synthesized programs should be (assuming the model can find programs that fit the observed I/O pairs).
- The discovery (Figure 2d) in this paper is contradictory to what is mentioned above: the program synthesis accuracy decreases when more I/O pairs are given. I am assuming the authors do not split observed I/O pairs and assessment I/O pairs.
- With the setup where the observed I/O pairs are separated from the assessment I/O pairs, a larger number of K (the number of the observed I/O pairs) should be used so that it is more likely that the program for each task is unique and the evaluation would make more sense.
[1] "RobustFill: Neural Program Learning under Noisy I/O" in ICML 2017
[2] "Leveraging Grammar and Reinforcement Learning for Neural Program Synthesis" in ICLR 2018
[3] "Reinforcement and Imitation Learning for Diverse Visuomotor Skills" in RSS 2018
[4] "DeepMimic: Example-Guided Deep Reinforcement Learning of Physics-Based Character Skills" in SIGGRAPH 2018
[5] "Learning to Infer Graphics Programs from Hand-Drawn Images" in NeurIPS 2018
[6] "Program Synthesis Through Reinforcement Learning Guided Tree Search" arXiv 2018
[7] "Neural Program Synthesis from Diverse Demonstration Videos" in ICML 2019
[8] "Execution-Guided Neural Program Synthesis" in ICLR 2019
[9] "Learning to Describe Scenes with Programs" in ICLR 2019
[10] "Learning to Infer and Execute 3D Shape Programs" in ICLR 2019
===== After rebuttal =====
I appreciate the authors for the revision and for clarifying some points. I am still not entirely convinced by the response and the revision.
First, I mentioned several papers that are related to this work, the authors failed to discuss the difference between this submission and these works.
- The "self-learning" paradigm is used in [1, 2, 7, 8] but [1, 7] are still not mentioned. If the authors intentionally ignore these works so that they can claim the novelty, it is not acceptable.
- While the proposed hybrid objective is very similar to the one proposed in [2, 3, 4], the revision does not mention this.
- Why [5, 6] that utilizing Monte Carlo Tree Search for program synthesis are still missing from the revision? I believe these works are very relevant to this submission.
- The authors acknowledged that they did not know about [6] that works on program synthesis for an assembly language. Yet, this paper is still missing from the paper.
I spent a lot of time conducting a survey in this field to help the authors to improve this submission and trying to identify the novelty of this submission. However,  the authors just chose to ignore it, which is very disappointing.
Second, many suggestions that I made are just rejected by the authors because they believe it is "not very easy and trivial". Given this self-learning setting, I believe it would be easy to implement some program induction baselines using supervised learning. Or how about search-based program synthesis baselines?
The writing about 50, 40, 40 testing sets is very misleading. I believe all the reviewers just think the testing accuracy was computed using those testing sets.
From my point of view, "program aliasing” does not mean there are not sufficient I/O examples to fully describe a task; Instead, it means there are programs written differently can be semantically the same, which means no matter how many input examples are given, their outputs will always match.
Overall, I believe this submission requires a serious revision and I firmly recommend this paper to be rejected.


Solving long-horizon sequential decision making tasks in environments with sparse rewards is a longstanding problem in reinforcement learning (RL) research. Hierarchical Reinforcement Learning (HRL) has held the promise to enhance the capabilities of RL agents via operation on different levels of temporal abstraction. Despite the success of recent works in dealing with inherent nonstationarity and sample complexity, it remains difficult to generalize to unseen environments and to transfer different layers of the policy to other agents. In this paper, we propose a novel HRL architecture, Hierarchical Decompositional Reinforcement Learning (HiDe), which allows decomposition of the hierarchical layers into independent subtasks, yet allows for joint training of all layers in end-to-end manner. The main insight is to combine a control policy on a lower level with an image-based planning policy on a higher level. We evaluate our method on various complex continuous control tasks for navigation, demonstrating that generalization across environments and transfer of higher level policies can be achieved. See videos https://sites.google.com/view/hide-rl.

The paper proposes a neat framework for creating HRL framework that will be able to generalize its application to slightly different environment layout. This is done via an image-based top-down from as input to the high level. An intermediate layer is used to help create more fine-grained goal specification for a final goal-based control layer. These layers are trained together using HAC. Overall, the method shows promise but there needs to be more analysis to understand which parts of this combination of ideas are the most important. The results are also only shown for a single environment. Last, the generalization analysis in the paper does not appear to be overly thorough. It would be good to perform this on more than one type of environment also the random environment is not very random.
More detailed comments:
- The results seem very similar to some of the work in "Universal Planning Networks" that did not need a more complex HRL design to achieve subgoal specification via images. This should be discussed more in the paper.
- The authors point out that the use of relative goal positions "ensures generalization to new environments", this is a rather strong statement. The use of relative goal specification my help improve generalization but that can only be shown empirically.
- The demonstration to show that the method generalizes to other configuration after being trained on a fixed environment should be evaluated over many randomly generated environments so that we have a non-biased estimate of the true generalization performance. In Table 2, it is rather surprising that the HiDe trained model does better on the "Random" environment vs the "FOrward" environment it is trained on. Can more details be provided on how the "Random" environment is created? Are the locations of the walls randomized? Is the initial position of the agent and goal randomize?
- The video seems to contradict the ordering of operations for training the planning network. The video suggests that first it is learned with the Ant then transferred to the ball which is less complex to control.
- At the beginning of the prior work section, it is noted that many other methods require prior knowledge of the environment I would say this method also requires certain kinds of prior knowledge about the task. For example, a top-down view of the environment is needed which is not often feasible.
- HIRO and HAC use a more proprioceptive state space but I don't think the sharing of global states is intentional. I am not convinced that this choice, in particular, is what makes the approaches prone to overfitting.
- You show a comparison to the "windows" created from your method vs a fixed neighbourhood. Do you perform any empirical evidence that your introduced methods provide an improvement over this fixed window?
- It is mentioned at the end of section 4.1 that MVProp is differential so it can be trained with the Bellman error objective. Because many policies are being trained concurrently does the MVProp attention model need to be recomputed after every sub-policy update? Does the frequency of updates have a large effect on performance?
- Section 4.2 introduces an interface layer that is not a very common practice. It would be good to include an ablation study of the effects of this introduced layer.
- In section 4.3 it says that the control layer is the only layer with access to the agent's proprioceptive state. Would it not be good to at least include the agent facing direction or current average velocity to higher layers to improve the attention mask estimation?
- In figure 5 it says HIRO converges the fastest because it has dense rewards. Can you be more specific? Also, If different agents are using different reward signals I am not sure this evaluation is a fair comparison.
- Are tables 2 and 3 just for the HiDe algorithm? Is it possible to include data for the other algorithms?
- You perform an experiment to train HiDe with random initial and goal locations for comparison. I think running this comparison for HIRO and HAC would be a good additional point of comparison. This would help the reader know if the generalization is not biased to the particular initial environment configuration for Maze Forward.
- In the generalization analysis for the paper, how is the analysis performed? There are percentages for the success of the policy, where does the randomness come from is the agent state and goal are always fixed? Are these averaged because the agent has a stochastic policy during evaluation? If this is the case how many random trajectories are collected to compute these statistics?
- It would be very helpful to have a description of the algorithm in the paper. How the algorithm works is not very clear and some details about how the goal and states are passed to the different policies would be very helpful if anyone wanted to reimplement this work.
Updated comments:
- The addition of more results and added analysis helps show the improvement of this method over the most related baselines.


Mean field models have provided a convenient framework for understanding the training dynamics for certain neural networks in the infinite width limit. The resulting mean field equation characterizes the evolution of the time-dependent empirical distribution of the network parameters. Following this line of work, this paper first focuses on the teacher-student setting. For the two-layer networks, we derive the necessary condition of the stationary distributions of the mean field equation and explain an empirical phenomenon concerning training speed differences using the Wasserstein flow description. Second, we apply this approach to two extended ResNet models and characterize the necessary condition of stationary distributions in the teacher-student setting.

This paper studies the mean fields limit of neural networks and extends the mean field limit treatment to resnets. The paper seems to be presenting non-trivial formal results but I could not really understand what are their specific consequences of interest. What are the concrete insights that the results of this paper bring? Even as a theoretician, I did not really understand what should I take from this paper. As such I cannot recommend its acceptance. But it is well plausible that with better explanations I will understand its value and change my mind. Some more concrete questions/issues follow:
The introduction several times states: "First, for the two-layer networks in the teacher-student setting, we characterize for the first time the
necessary condition for a stationary distribution of the mean field PDE and show how it relates to the parameters of the teacher network. We also discuss how different activation functions influence the results." The outline states: "Section 2 concerns the mean field description of two-layer networks and provides necessary conditions for stationary distributions."
But where is this discussion of necessary condition? I cannot find a single mention of "necessary" in section 2 that only includes formal derivations and statements. Can the result of Section 2 be translated to somewhat more general audience? What is the take-home message there?
Where is the discussion on the role of the activation function? I see the two examples stating the formal result for two different activations, but again I am unable to understand what should one take from this?
I could understand the result of section 2.2. about stronger weight-teachers being learned first. But this is completely intuitive in the same way as independent components corresponding to larger eigenvalues would get learned first in PCA. See also
similar conclusions in works on so-called "INCREMENTAL LEARNING" in Andrew M Saxe, James L McClelland, and Surya Ganguli. Exact solutions to the nonlinear dynamics of learning in deep linear neural networks. arXiv preprint arXiv:1312.6120, 2013. And subsequent results. So this alone does not seem very novel result.
As I had hard time understanding the section two on the simpler feedforward case, Section 3 was even less clear to me. How does what is done here compare to what is known on resnets and what should somebody interested in resnets (but not specifically in the mean field equations) take out from this. Some non-technical summary of the findings is seriously missing in this paper.
Comments that are less fundamental to the overall understanding:
** The mean field treatment was also extended to the multi-layer case in: https://arxiv.org/abs/1906.00193
** The paper present nice account on previous results involving the mean field limit. The manuscript should also discuss the long line of papers analyzing the teacher student setting on one-hidden-layer neural networks. After-all this seems to be the main object of study here so the results only make sence presented against what was previously known. Notably the case of eq. (1) where the 2nd layer weights are fixed to one is a model called the committee machine widely considered in previous literature (in the case of finitely wide one-hidden-layer network).
The (non-exhaustive) list of paper on the teacher-student setting is (more references are in those papers):
-- The teacher-student setting was introduced in Garder, Derrida'83 (model B, https://iopscience.iop.org/article/10.1088/0305-4470/22/12/004/pdf).
-- In the classical textbook on neural networks Engel, Andreas, and Christian Van den Broeck. Statistical mechanics of learning. Cambridge University Press, 2001, there is a rather detailed account of many results on the setting from 80s and 90s.
-- Notably the plain SGD was analyzed via ODEs for the teacher-student setting in classical papers: David Saad and Sara A Solla. On-line learning in soft committee machines. Physical Review E, 52 (4):4225, 1995.
David Saad and Sara A Solla. Dynamics of on-line gradient descent learning for multilayer neural networks. In Advances in neural information processing systems, pp. 302–308, 1996.
-- This line of work was recently extended with a focus on the overparametrized regime (but not infinitely wide) in: Dynamics of stochastic gradient descent for two-layer neural networks in the teacher-student setup, Sebastian Goldt, Madhu S. Advani, Andrew M. Saxe, Florent Krzakala, Lenka Zdeborová, NeurIPS'19.
-- There even in analysis with more than one hidden layer in Zeyuan Allen-Zhu, Yuanzhi Li, and Yingyu Liang. Learning and generalization in overparameterized
neural networks, going beyond two layers. NeurIPS, 2019a.
It would be really interesting to see a discussion of what is similar and different between these works and results and the present work. And what is the added value of the present one about understanding the teacher-student setting.