We propose a novel approach to compute a stable and generalizing metric (LNSM) with convolutional neural networks (CNN) to compare field data from a variety of numerical simulation sources. Our method employs a Siamese network architecture that is motivated by the mathematical properties of a metric and is known to work well for finding similarities of other data modalities. We leverage a controllable data generation setup with partial differential equation (PDE) solvers to create increasingly different outputs from a reference simulation. In addition, the data generation allows for adjusting the difficulty of the resulting learning task. A central component of our learned metric is a specialized loss function, that introduces knowledge about the correlation between single data samples into the training process. To demonstrate that the proposed approach outperforms existing simple metrics for vector spaces and other learned, image based metrics we evaluate the different methods on a large range of test data. Additionally, we analyze generalization benefits of using the proposed correlation loss and the impact of an adjustable training data difficulty.

This is a well-written paper which looks into options for learning similarity metrics on data derived from PDE models common in the sciences. In comparison to other metric learning settings, here a type of ground-truth distance information is available (rather than, say, triplets), and it is possible to attempt to directly target an objective function which aims to match the learned distance to the ground-truth distance. The model architecture follows a fairly standard siamese-network setup.
Quite a bit of space is devoted to ensuring that the learned metric actually satisfies pseudo-metric axioms. This is all very clearly presented, with justifications for different modeling choices and how they preserve the axioms; my only criticism here is that many aspects of this are fairly obvious (i.e. an architecture which shares weights in computing the embeddings of both data points, followed by computing a squared L2 distance, will quite clearly get us in the ballpark of a pseudometric), but in my opinion "excess" clarity is much better than the opposite.
I think the more important contribution of the paper is in sections 4 and 5, which outlines a specific data generation process, including means of injecting noise, and compares options for loss functions. I would have expected pearson correlation to work quite well in this context, and it is interesting to note that performance notably improved by also adding an MSE term. I am curious about the "distance" prediction, as described just before equation 5, where it is stated that d \in R^n — is this really R^n? The target distance c is in [0,1]^n, and it seems like a simple modification to the distance prediction network would be capable of ensuring that the predicted values d also fall in this range. Such normalization could reduce the need for the MSE loss term, which presumably helps keep the overall relative scales of the two distances in check.
The empirical testing is also thorough, and I particularly appreciate the use of the random-weight networks as a baseline — I think it is good to note that these are actually fairly competitive on many of the test data sets (in fact, I believe it should be in bold for "TID" in table 1).
I think the main weakness of this paper is that it falls slightly short of actually presenting the real use cases and needs for a similarity metric on PDE outputs — in my opinion, this comes to play when matching the output of a PDE-based model with real data. It would be nice to see a discussion of how this could be useful for parameter inference in PDE models. If there are other important applications of a distance learned in this way, I think the paper could benefit *greatly* by pointing them out. Otherwise, this risks being perceived as adding little value, since for individual PDE runs with known parameters, there is a ground-truth distance available — in which case, why bother using deep learning to estimate the distance, if the parameters are known? I think relevance to applications should to be clearly addressed.
The supplemental material is long, but complete and clearly presented.


Outlier detection and novelty detection are two important topics for anomaly detection. Suppose the majority of a dataset are drawn from a certain distribution, outlier detection and novelty detection both aim to detect data samples that do not fit the distribution. Outliers refer to data samples within this dataset, while novelties refer to new samples. In the meantime, backdoor poisoning attacks for machine learning models are achieved through injecting poisoning samples into the training dataset, which could be regarded as "outliers" that are intentionally added by attackers. Differential privacy has been proposed to avoid leaking any individual's information, when aggregated analysis is performed on a given dataset. It is typically achieved by adding random noise, either directly to the input dataset, or to intermediate results of the aggregation mechanism. In this paper, we demonstrate that applying differential privacy could improve the utility of outlier detection and novelty detection, with an extension to detect poisoning samples in backdoor attacks. We first present a theoretical analysis on how differential privacy helps with the detection, and then conduct extensive experiments to validate the effectiveness of differential privacy in improving outlier detection, novelty detection, and backdoor attack detection.

This paper leverages differential privacy’s stability properties to investigate its use for improved anomaly and backdoor attack detection. Under an assumption (called “uniformly asymptotic empirical risk minimization”), the authors show that difference between the expected loss of a differentially private learning algorithm on an outlier (where the expectation is taken over the randomness of the learning algorithm) and the expected loss of the same algorithm on data from the underlying distribution (expectation taken over data & randomness of the algorithm) is lower bounded by a (possibly/hopefully) non-negative quantity with high probability. The authors then conduct a set of experiments to show that differential privacy improves the performance of outliers, novel examples, and backdoor attack detection.
Overall, the paper is very well written and easy to read. The paper also tackles an important and timely problem that is relevant to the ICLR community. While there has been some recent work on connecting differential privacy to robustness & attacks, this paper investigates the use of differential private model training as a means to improve novelty detection at inference time.
A few points that need attention from the authors:
1. The theory developed is insightful in general but has very little (to no) practical value. For starters, it assumes that differentially private model training is uniformly asymptotic to empirical risk minimization. This is not necessarily true for highly non-convex models trained with SGD. Further, it cannot be verify via experimentations (despite the authors’ attempt to sanity check it using Figure 1). More importantly, the theory developed in Section 3 is not used in any meaningful way in the experiments section — the anomaly detection schemes are agnostic to it.
2. The authors make no attempt to co-optimize the performance of the model with its ability to be used for better anomaly detection. For instance, the authors choose an l2-clipping-norm C of 1 and do not consider trading off C with the noise variance.
When the training set contains anomalies, this work can be viewed as “what is the impact of differential privacy” on a training sets with a majority group (training examples from a given distribution) and a minority group (training examples from a different distribution). Under this view, this paper essentially says that “differential privacy leads to disparate impact on model accuracy/loss”. This has been recently investigated in the following NeurIPS19 paper: https://arxiv.org/abs/1905.12101. Thus the contributions of the paper are not substantial.


Attention to semi-supervised learning grows in machine learning as the price to expertly label data increases. Like most previous works in the area, we focus on improving an algorithm's ability to discover the inherent property of the entire dataset from a few expertly labelled samples. In this paper we introduce Boosting via Self Labelling (BSL), a solution to semi-supervised boosting when there is only limited access to labelled instances. Our goal is to learn a classifier that is trained on a data set that is generated by combining the generalization of different algorithms which have been trained with a limited amount of supervised training samples. Our method builds upon a combination of several different components. First, an inference aided ensemble algorithm developed on a set of weak classifiers will offer the initial noisy labels. Second, an agreement based estimation approach will return the average error rates of the noisy labels. Third and finally, a noiseresistant boosting algorithm will train over the noisy labels and their error rates to describe the underlying structure as closely as possible. We provide both analytical justifications and experimental results to back the performance of our model. Based on several benchmark datasets, our results demonstrate that BSL is able to outperform state-of-the-art semi-supervised methods consistently, achieving over 90% test accuracy with only 10% of the data being labelled.

In this paper, the authors present an approach for semi-supervised learning which combines noisy labels with boosting. In a first step, the labeled instances are used to train a set of classifiers, and these are used to create noisy labels for the unlabeled instances. Then, an EM procedure is used to estimate the noise level of each instance. Finally, a version of AdaBoost which accounts for instance noise levels is proposed to create a final classifier. A limited set of experiments suggests the proposed approach is competitive with existing approaches.
Major Comments
As a non-expert in this area, I had trouble identifying the novel contributions of this work. For example, many of the results in Section 3 (noise-resistant AdaBoost) seem to replicate, or follow closely, the results of [Natarajan et al., 2013]. Similarly, using EM to assign pseudo-labels has been extensively studied in the literature [Lee, WREPL 2013; Chapelle and Zien, AISTATS 2005; Kang et al., ECCV 2018; Rottman et al., ICMLA 2018].
The experiments are very poorly described, so it is difficult to gauge if they are valid:
Most importantly, the authors point out that the estimated error rates do not always match the actual error rates. Since this seems to be one of the most important factors of the proposed approach, further investigation should be performed to answer questions like: why is the error rate not estimated well? on what type of datasets? can more/better supervised learners help? In some cases (Diabetes, Thyroid, Heart), the actual noise rate increases with more labeled samples. What does that mean?
Second, the proposed approach seems to have a number of important hyperparameters, including the number of supervised models trained and their hyperparameters, the parameters of the Beta distribution used as a prior on the noise estimation, and the hyperparameters of the AdaBoost algorithm. Likewise, all of the competing algorithms also have hyperparameters which are known to affect performance (e.g., learning rate for NNs). The paper does not mention how (or if) a validation set was used to select these.
Third, while the caption of Table 1 mentions that 20 trials were used, it is not clear if this was some sort of k-fold cross validation, Monte Carlo, cross validation, the same splits but with different random seeds, etc. Additionally, the variance across the different trials should be given; otherwise, it is not possible to tell if any of the empirical results are significant.
Minor Comments:
The references are not consistently formatted.
This paper is very notation heavy. It would be helpful to include a “table of symbols” for the reader in an appendix.
Additionally, the notation in the paper is not consistent. For example, both “$M$” and “$\mathcal{M}$ are used to indicate the number of models trained on the labeled data. Later on, “$\mathcal{M}$” is also used to refer to the set of trained classifiers. The first bullet point in Step 3 of the pseudocode seems to suggest that each classifier is trained on a single labeled data point. The equation at the bottom of Page 5 used \theta, but it does not seem to be defined.
The discussion on experts, spammers, and adversaries could be helpful if this terminology were used throughout the paper; however, it is used in only one paragraph.
The main body of the paper should mention that proofs are given in the appendices.
For context, if may be helpful to mention that graph convolutional networks and other representation learning techniques are commonly used for semi-supervised learning (e.g., [Kipf and Welling, ICML 2016]). Those approaches are quite different (and lack any sort of theoretical guarantees, for the most part), though, so empirical comparisons may not be so meaningful.
It would be helpful to give a sentence or two on the intuition behind what the proofs are showing. For a non-expert, they are very difficult to follow.
Do the various proofs still hold when the datasets are artificially balanced (with respect to the last paragraph in Section 4)?
It would be helpful to include the performance using the complete labeled dataset for comparison.
Stratified sampling could be used to ensure both classes are present in the training data. Also, “0.99%” -> “99%”.
Besides accuracy, some measure like AuROC or the F1 score which account for class imbalance should be given.
Typos, etc.
“Logitboost tested against” -> “Logitboost were tested against”
“therefore is not” -> “therefore, having a lot of labeled data is not”


In this work we present an approach that combines deep learning together with laws of Newton's physics for accurate trajectory predictions in physical games. Our model learns to estimate physical properties and forces that generated given observations, learns the relationships between available player's actions and estimated physical properties and uses these extracted forces for predictions. We show the advantages of using physical laws together with deep learning by evaluating it against two baseline models that automatically discover features from the data without such a knowledge. We evaluate our model abilities to extract physical properties and to generalize to unseen trajectories in two games with a shooting mechanism. We also evaluate our model capabilities to transfer learned knowledge from a 2D game for predictions in a 3D game with a similar physics. We show that by using physical laws together with deep learning we achieve a better human-interpretability of learned physical properties, transfer of knowledge to a game with similar physics and very accurate predictions for previously unseen data.

The problem addressed by this paper is the estimation of trajectories of moving objects thrown / launched by a user, in particular in computer games like angry birds or basketball simulation games. A deep neural network is trained on a small dataset of ~ 300 trajectories and estimates the underlying physical properties of the trajectory (initial position, direction and strength of initial force etc.). A new variant of deep network is introduced, which is based on an encoder-decoder model, the decoder being a fully handcrafted module using known physics (projectile motion).
I have several objections, which can be summarized by the simplicity of the task (parabolic trajectories without any object/object or object/environment collisions / interactions), the interest of the task for the community (how does this generalize to other problems?), and the writing and structuring of the paper. I will detail these objections further in the rest of the review.
Learning physical interactions is a problem which has received considerable attention in the computer vision and ML communities. The problem is certainly interesting, but I think we should be clear on what kind of scientific knowledge we want to gain by studying a certain problem and by proposing solutions. The tasks studied by the community are mostly quite complex physical phenomena including multiple objects of different shapes and properties and which interact with each other. All these phenomena can be simulated with almost arbitrary precision with physics engines, and these engines are mostly also used for generating the data. In other words, the simulation itself is solved and is not the goal of this body of work. The goal is to learn differentiable models, which can be used as inductive bias in larger models targeting more general tasks in AI.
Compared to this goal, the proposed goal is far too easy: learning projectile motion is very easy, as these trajectories can be described by simple functions with a small number of parameters, which also have a clear and interpretable meaning. The simplicity of the task is also further corroborated by the small number of samples used to estimate these parameters (in the order of 300). A further indication is the fact, that the decoder in the model is fully hardcoded. No noise modelling was even necessary, which further corroborates that a very simple problems is addressed.
In order words, I am not really sure what kind of scientific problem is solved by this work, and how this knowledge can help us to solve other problems, harder problems.
My second objection is with the written form of the paper. The paper is not well enough structured and written, many things are left unsaid. First of all, the problem has never been formally introduced, we don’t know exactly what needs to be estimated. What are the inputs, outputs? Is computer vision used anywhere? How are the positions of the objects determined if not with computer vision? How are the user forces gathered? What are “in game variables” mentioned multiple times in the document? No notation has been introduced, no symbols have been introduced (or too late in the document). For instance, there is no notation for the latent space of the encoder-decoder model.
The figures are not very helpful, as the labelling of the blocks and labels is very fuzzy. As an example, For InferNet, inputs and trajectories are “Trajectories”, so what is the difference? Of course we can guess that (inputs are measured trajectories, outputs are reconstructed trajectories), but we should not guess things when reading papers.
The figure for encoder-decoder model is very confusing, as the different arrows have different functional meanings and we have no idea what they mean. The outputs of the encoder and the MLP both point to the latent space and at a first glance the reader might think that they are concatenated, which raises several questions. Reading the text, we infer that first a model is trained using on one of the arrows (the one coming from the encoder) and ignoring the other one, and then the MLP is learned to reconstruct the latent space using the other arrow (the one coming from the MLP), but this is absolutely impossible to understand looking at the figure, which does not make much sense. We can infer all this from the text around equations (1) to (3), which is itself quite fuzzy and difficult to understand, in spite of the simplicity of the underlying maths.
The relationship of RelateNet and InferNet is not clear. While the task of InferNet is clear, the role of InferNet in the underlying problem is not clear and it has not been described how it interacts with RelateNet.
It is unclear how the transfer between science birds and basketball has been performed and what exactly has been done there.
As mentioned above, the role of “in game variables” is unclear. What are those? I suggest to more clearly define their roles early in the document and use terms from well-defined fields like control (are they “control inputs”) or HCI (are they “user actions”?).
In the evaluation section, we have baseline models BM1 and BM2, but they have never been introduced. We need to guess which of the models described in the paper correspond to these.
The related work section is very short and mostly consists of an enumeration of references. The work should be properly described and related to the proposed work. How does the proposed work address topics which have not yet been solved by existing work?


Neural language models have achieved great success in many NLP tasks, to a large extent, due to the ability to capture contextual dependencies among terms in a text. While many efforts have been devoted to empirically explain the connection between the network hyperparameters and the ability to represent the contextual dependency, the theoretical analysis is relatively insufficient. Inspired by the recent research on the use of tensor space to explain the neural network architecture, we explore the interpretable mechanism for neural language models. Specifically, we define the concept of separation rank in the language modeling process, in order to theoretically measure the degree of contextual dependencies in a sentence. Then, we show that the lower bound of such a separation rank can reveal the quantitative relation between the network structure (e.g. depth/width) and the modeling ability for the contextual dependency. Especially, increasing the depth of the neural network can be more effective to improve the ability of modeling contextual dependency. Therefore, it is important to design an adaptive network to compute the adaptive depth in a task. Inspired by Adaptive Computation Time (ACT), we design an adaptive recurrent network based on the separation rank to model contextual dependency. Experiments on various NLP tasks have verified the proposed theoretical analysis. We also test our adaptive recurrent neural network in the sentence classification task, and the experiments show that it can achieve better results than the traditional bidirectional LSTM.

This paper first provides a theoretical interpretation of separation rank as a measure of a recurrent network's ability to capture contextual dependencies in natural language text. The analysis is primarily done in the context of tensor space language models (TSLM) that was demonstrated to be a generalisation of n-gram language models in earlier work. The theoretical derivations suggest that increasing a network's depth increases its separation rank exponentially, while increasing its width only increases its separation rank linearly. Based on this finding, the paper proposes a bidirectional adaptive recurrent neural network that adapts the network depth for each task using dynamic halting. Experiments on six NLP tasks demonstrate that increasing network depth helps more for tasks that generally require many long-range dependencies, while increasing network width is generally sufficient for tasks that require mostly short-term dependencies, although I disagree with the paper's dichotomy of tasks with long-term and short-term dependencies (see point 3 below).
Overall, this paper suffers from several serious issues, as listed below. I am therefore recommending a rating of "Reject" ahead of the authors' response.
1. This paper suffers from substantial clarity issues beyond simple grammatical errors. Some examples of the most serious clarity issues are as follows: (i) Section 6.1.1 lists "sentiment analysis" as one of the tasks, but Figure 2 has no entry for "sentiment analysis", yet instead features experimental results for "semantic classifier" which was never mentioned or introduced before; and (ii) Figure 2 shows "1st layer LSTM", "2nd layer LSTM", and "3rd layer LSTM", while (based on my reading) what the paper means are "1-layer LSTM", "2-layer LSTM", and "3-layer LSTM", hence highly confusing for the reader.
2. The proposed explanation about the proposed bidirectional adaptive RNN is really sparse, despite being a central part of the paper. The explanation of the proposed model is only contained in two paragraphs (Sections 5.1 and 5.2), and are not sufficient for the readers to understand the model. A more extensive explanation and intuition about what the model is like, and how it is similar or different to TSLM and standard RNNs, is required to improve this.
3. In the experiments, the paper assumes a false dichotomy of tasks that only require short-range dependencies (NER, POS tagging, and constituency parsing), and tasks that require long-range dependencies (WSD, sentiment analysis, and coreference resolution). This dichotomy is overly simplistic and ultimately false. For instance, constituency parsers often need to identify spans that are very long-distance in nature (for a recent investigation of this, see the work by Fried et al. (2019)), while sentiment analysis often only requires the model to identify a few salient words that are indicative of the sentiment, e.g. "excellent" or "terrible", hence not requiring much contextual dependencies.
4. Related to point 3, a better evaluation is to examine the cases that require long-range dependencies within each task, rather than assuming which tasks require long-range dependencies and which ones do not. An example of this is reporting e.g. constituency parsing performance for long-range spans and coreference resolution performance for long-distance entity chains.
5. The paper mostly fails to report performance comparison with existing numbers from prior work. For instance, the coreference performance (Fig. 2) are far below the result from Lee et al. (2017) that this paper is based on (Section 6.1.1), while the constituency parsing numbers (also Fig. 2) are also far below the reconciled span parser (Joshi et al., 2018) that this paper is also based on. This discrepancy calls into question the strength of the model implementation used in this paper.
6. Some missing citations, e.g. the use of adaptive computation time and dynamic halting in Universal Transformers (Dehghani et al., 2019).
7. This paper can benefit from careful copy-editing. Some examples of grammatical errors: (i) "Eq. 16" in Section 5.2 should be "Eq. 6", (ii) "to verify our theoretical, respectively" in Section 6 should be "to verify our theoretical [findings/derivations]", (iii) "ourself" on the caption of Table 1 should be "ourselves", (iv) "The model Ling et al. (2015) is used, which using bidirectional LSTMs" should be "The model [of] Ling et al. (2015) is used, which [used] bidirectional LSTMs", etc.
References
Daniel Fried, Nikita Kitaev, and Dan Klein. "Cross-domain generalization of neural constituency parsers". In Proc. of ACL 2019.
Kenton Lee, Luheng He, Mike Lewis, and Luke Zettlemoyer. "End-to-end neural coreference resolution". In Proc. of EMNLP 2017.
Vidur Joshi, Matthew E. Peters, and Mark Hopkins. "Extending a Parser to Distant Domains Using a Few Dozen Partially Annotated Examples". In Proc. of ACL 2018.
Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, and Lukasz Kaiser. "Universal Transformers". In Proc. of ICLR 2019.


Convolutional networks are not aware of an object's geometric variations, which leads to inefficient utilization of model and data capacity. To overcome this issue, recent works on deformation modeling seek to spatially reconfigure the data towards a common arrangement such that semantic recognition suffers less from deformation. This is typically done by augmenting static operators with learned free-form sampling grids in the image space, dynamically tuned to the data and task for adapting the receptive field. Yet adapting the receptive field does not quite reach the actual goal -what really matters to the network is the effective receptive field (ERF), which reflects how much each pixel contributes. It is thus natural to design other approaches to adapt the ERF directly during runtime. In this work, we instantiate one possible solution as Deformable Kernels (DKs), a family of novel and generic convolutional operators for handling object deformations by directly adapting the ERF while leaving the receptive field untouched. At the heart of our method is the ability to resample the original kernel space towards recovering the deformation of objects. This approach is justified with theoretical insights that the ERF is strictly determined by data sampling locations and kernel values. We implement DKs as generic drop-in replacements of rigid kernels and conduct a series of empirical studies whose results conform with our theories. Over several tasks and standard base models, our approach compares favorably against prior works that adapt during runtime. In addition, further experiments suggest a working mechanism orthogonal and complementary to previous works.

This work presents the idea of deformable kernels (DKs). As opposed to rigid kernels in standard convolutional networks, DKs allow each of their grid locations to be moved around in a larger kernel field. The offset by which a DK grid cell is moved is computed conditioned on the input to the network. To motivate the idea of DKs, the authors give some background on convolution, receptive and effective receptive fields (ERFs). The authors argue that since ERFs are spatially porous and irregularly distributed, one way to model them is to convolve square grids of input with DKs, which are composed of samples drawn from larger kernels. The authors define the concept of global and local DKs. They further contrast DKs with spatial sampling (deformable convolutions) and argue that although conceptually similar, both approaches are complementary to each other and can be used in combination in practice. Numerical experiments show competitive performance of DKs on image classification and object detection tasks. In the end empirical analysis is performed to analyze the characteristics of DKs.
I am unfamiliar with prior work in this direction, but the idea of DKs seems to be conceptually appealing and as the authors point-out, their approach can be seen as an alternative to spatial sampling for modeling deformations. Unfortunately the authors get hand-wavy when in Sec 3.2, they claim that the idea of subsampling kernels "roughly generalizes" to non-linear networks. I don't see how they can generalize what they present beyond piece-wise linear networks. I appreciate the effort to give a background on ERFs and describe (local and global) DKs, but in my opinion, technical sections of the paper partly very obscure. For instance it is not entirely clear how the offset predictors are trained, how exactly the sampling is used, details of architecture etc.
Empirically the method does not seem to offer a significant performance boost. Also, while the authors sell the idea of subsampling kernels, but the finding that kernel sizes beyond 4x4 don't seem to offer any benefit make the idea practically questionable.
The idea of DKs seems relevant, but both conceptually and empirically it seems very close to deformable convolutions. The authors need to clearly present their work, including its shortcomings (i.e., generalization or not beyond linear networks).


This paper presents a method to solve the city metro network expansion problem using reinforcement learning (RL). In this method, we formulate the metro expansion as a process of sequential station selection, and design feasibility rules based on the selected station sequence to ensure the reasonable connection patterns of the metro line. Following this formulation, we train an actor critic model to design the next metro line. The actor is a seq2seq network with attention mechanism to generate the parameterized policy which is the probability distribution over feasible stations. The critic is used to estimate the expected reward, which is determined by the output station sequences generated by the actor during training, in order to reduce the training variance. The learning procedure only requires the reward calculation. Thus our general method can be extended to multi-factor cases easily. Considering origin-destination (OD) trips and social equity, we expand the current metro network in Xi'an, China, based on the real mobility information of 24,770,715 mobile phone users in the whole city. The results demonstrate the effectiveness of our method.

This paper introduces a method for solving the problem of network expansion, in particular, considers the city metro network and its expansion within the new metro line. Since this problem was previously represented as the non-linear integer problem, having an exponential number of constraints or requiring expert knowledge, authors had an initial motivation to appeal to learnable algorithms from the reinforcement learning paradigm. Authors proposed to consider the problem as Markov Decision Process, so that the environment is represented as the city grid, with constructed metro network, before some timestamp conditioned by additional transportation constraints, while each new state in the expansion process of construction one separate metro line is the state of the metro network as graph with new station added to the metro line, considered as the agent’s action. The custom reward function was represented based on metrics like OD trips used in the traditional approaches and the first time used in this problem specific social equity score. The permissibility constraints of action, i.e. adding a new station, were incorporated into policy function, while the attention mechanism was used here as in other seq2seq models, to query the most suitable station among candidates. Authors use the actor-critic approach, widely exploited in reinforcement learning problems, and with all the above-mentioned modification reached the baseline results on the real-world data, obtained from Xi’an city metro network, Shaanxi province, China.
Main arguments
Apart from the interesting approach, this paper should be rejected due to several reasons: (1) the specific application field, considered in this paper, have to be generalized to more general cases, in order to be valuable for Machine Learning community, (2) from RL algorithms perspective, the novelty of proposed method is questionable due to lack of literature review and advanced approach to deep reinforcement learning, (3) requirement of explainable AI is essential for deployment and in spite of the quite sufficient explanations of the algorithm’s works, this paper does not well justify its superiority over the traditional methods either by theory or practice, due to experiments suffering from the lack of variability and missing the main point of improvement, which leads to generally insufficient contribution (4) the paper is unpolished and lacks specific formulations from the existing literature on related subjects.

In detail
The paper does a great job of justifying neither the novelty of its method nor the guarantees of increased performance or efficiency compared to existing methods. Although the proposed method discusses the specific problem and there could be a lack of existing sufficiently good methods in this specific topic, the reinforcement learning paradigm is utilized by authors in a very superficial and general manner.
Firstly, the paper excludes any comparison to similar problems and already existing methods of RL used by similar fields, which can be expressed as planning with constraints (for example, path planning in mobile robot navigation), thus will be more valuable to ML community than specific application of graph expansion problem, limited to line-by-line addition.
Secondly, except general notice of combinatorial problems solved by RL algorithms, the literature review lacks mentioning graph representation methods or any seq2seq algorithms (which efficiently uses attention), which should serve as an initial guess for the well-performing model in this particular problem. Regarding the RL approach, there are plenty of already solved planning problems, papers comparing RL algorithms and their performances on grid-based, discrete environment problems in order to mention here and ideally, comparing to proposed one within the same specific problem formulation. Because of this reason, there is no justification of concrete architecture construction and choice of actor-critic model for learning this policy in this specific example and so there is no intuition, why this model will work better than existing methods. Besides, this leads to the doubtful novelty of the proposed algorithm, although used first time in this specific field, but composed totally from the same components and sometimes in the same combination as in other works related to RL methods in planning.
Moreover, the explanation of the methods’ architecture and basic components’ influence on the system is insufficient but might serve as an initial explanation of how this algorithm works. However, the number of experiments and their variations leaves much to be desired to justify its practical advantages and its utilization purposes as explainable AI. (1) there is no ablation study conducted to justify the choice of the proposed model over existing reinforcement learning methods, like actor-critic methods and their several variations with baseline, including modifications like attention model, graph representation, encoder and decoder architecture. The mentioning of these methods in the literature review is enough to answer this issue. (2) Using only one, hardly tuned baseline method on one dataset (without any reference to existing benchmark results on it) does not prove the efficiency and good performance of the model. Additional experiments can on different environments structure (city grids, metro network) of complexity of the environments (the width of the cell in the grid) to measure the performance of the algorithm based on hyperparameters (3) Real-world data brings more application-based matter into the subject, but requires more thorough investigation on the optimality of the solutions, proposed by RL method. This means that the comparison with the groundtruth metro network expansion (built after October 15) cannot be used as it is without using the expert evaluation. As a solution and habit in the machine learning literature, without the expert knowledge on the dataset, authors may propose a comparison of the real-world solution with the algorithm’s results, based on the optimized metrics (OD trips score and social equity) included into reward function.

Finally, the proposed paper is imprecise in several ways. In terms of RL formulation and usage of RL terms (commonly introduced in literature), it reveals inappropriate usage of MDP terms and insufficient description policy-gradient approach in actor-critic training algorithm. While not appealing to this issue, authors missed the opportunity to formulate this problem as discrete MDP with high-dimensional action space and sparse rewards, thus had less opportunity to research different model-based reinforcement learning problems in discrete environments to conduct better model selection and experiments. Besides, the paper lacks details in some parts, like the usage of 3G cellular mobile data, which was not mentioned in this work, although it is emphasized as an essential part if not a significant contribution of this research.

Questions to answer
Introduction
	What is the reason by constructing the city metro networks line by line, not iteratively adding stations to the existing network on the grid? This would make your problem closer to general grid-based planning problems. Do traditional methods expand metro networks only line-by-line and can this be there a limitation?
	How does a “good solution” is represented in the literature with traditional approaches? What are the general measures and constraints in practice?
	Is incorporating social equity concern your contribution? Then why does the maximization of OD trips is not enough (there is no mention of the preferability of social equity metric based on the results of experiments)?
	The effectiveness of the method due to experimental results is still questionable.
Related work
	If the main concern of the paper is still the limitation of other methods which use expert knowledge then it is better to state the usage of additional data (development index) in social equity metric as a reward design part in Appendix to justify this reward engineering
	“We believe that RL has the ability to solve the metro expansion problem” is the statement, which should be substituted by extensive literature review on RL methods used for planning with constraints or specifically graph-based expansion methods.
Problem formulation
	Is the network graph undirected or directed (imprecise inclusion of “direct links”)? How does this the OD trips are measured (no information here or in Appendix A).
	What is the reason behind 4 constraints provided in problem formulation? Are they used or defined in the literature review papers?

Method
	RL and deep learning methods included in the literature review should be the reference of choosing one or another component of the algorithm, including the formula for policy and critic update.
	It is essential to use notation based on RL formulation, for example, use r(s_t,a_t) = 0 if s_t  is not terminal state and r(s_t,a_t) = w(Z|G) in order to state the sparsity of the problem. S_t here is Z_t sequence of chosen stations during the episode (line expansion process) and so on.
	Use precise formulation, for example, the proposed P(Z|G) probability distribution is the trajectory distribution and not policy function, which could be denoted as π(z_t | S,G,M(Z_t))
	How does the choice of filter design is justified, based on similar works in the transportation field?
	What is the reason by choosing the 1-dimensional CNN layer to represent candidate stations and what is the concrete input information? There is no formal specification of the input data, regarding the existing graph structure, only a short mention of it.
	Is the attention mechanism architecture, used here, similar to other models used in seq2seq problems? (Answer: Yes) What is the motivation to use this concrete architecture?
	Why does the permissibility mapping using filter is used in policy function? Are there reasons to use it in policy function, rather than include in reward design? For example, the commonly used approach is to penalize unfeasible actions (stations) with a very low reward, during the episode learning?
	Training:
	How does the sparsity of reward influence the performance? Is there a way to better design reward, so that there will not be a necessity to update actor and critic only after the end of the episode (termination step)?
	What is the reason for learning critic as V(Z) = w(Z|G)? Is there a need for baseline if the state and the sequence of actions have bijective mapping, meaning that one sequence of stations can generate the unique state of the environment – line expansion, and vice versa? The intuition of the baseline is to measure the value of the state as the average among all possible actions, which lead to this state.
	What are the batches B used in the actor-critic training procedure?
	How do we generalize the environment training? Do we need to retrain the reinforcement learning algorithm for each different initial metro city network configuration, or it is generalizable to other grids, using the same weights?
Experiments
	Please, include the training time of the RL algorithm and its inference time, and compare it to the performance of the baseline algorithm, which should be one of the most essential contributions, due to the high time-complexity of traditional methods.
	How does the choice of corridor width influence the performance of the baseline method?
	Is there the baseline performance on the same city metro network (Xi’an city metro) mentioned in other literature, to directly compare with? This is necessary to fill the gap in justification of proposed results optimality (for the baseline case).
	How does the partial similarity of the 2 times line expansion by RL method and 6 real-world lines of the city metro network can justify the optimality of the proposed method? Can you provide the measurement based on OD trips and social equity? Can you provide a truly optimal solution based on the grid granularity, initial network graph, and constraints, to compare with as an optimal solution?


Bayesian optimization (BO) is a popular approach to optimize expensive-toevaluate black-box functions. A significant challenge in BO is to scale to highdimensional parameter spaces while retaining sample efficiency. A solution considered in previous literature is to embed the high-dimensional parameter space into a lower-dimensional manifold, often a random linear embedding. In this paper, we identify several crucial issues and misconceptions about the use of linear embeddings for BO. We thoroughly study and analyze the consequences of using linear embeddings and show that some of the design choices in current approaches adversely impact their performance. Based on this new theoretical understanding we propose ALEBO, a new algorithm for high-dimensional BO via linear embeddings that outperforms state-of-the-art methods on a range of problems, including learning a gait policy for robot locomotion.

This paper criticizes existing High Dimensional BO (HDBO) via linear embedding literature for the following reasons:
- Points in the embedded space projected mostly to the facet of the bounding box in the original space.
- The projection induces a distorted space which is not fit to be modeled by a GP.
- Linear projections do not preserve product kernels.
- Linear embeddings have low probability of containing an optimum.
The paper then proposes ALEBO which supposedly improves these aspects over other linear embedding BO techniques.
I have the following concerns regarding the authors' criticisms above:
1. What is wrong with the points being projected mostly to the facet of the original bounding box?
If I understand correctly, Theorem 3 of the REMBO paper proved that there exists y* \in R^{d_e} such that f(Ay*) = f(x*)
(i.e., the projected space contains the optimum) with high probability so to me, it does not really matter if the projection does not include the interior of the bounding box. Fig. 1 seems to make a point that most projections seem to indeed land on the facet of the bounding box (to be thorough, how many points did the authors sample to make this plot) but given what I said above, I do not think there is much of a point in Fig. 1 here.
2. The authors claimed that it is not appropriate to model the distorted space with GP, but ended up using GP
for ALEBO anyway (although with a different kernel). I understand that the authors did not project Ay back to B
like REMBO did, but the authors also gave me no reason to believe that this will improve things either. In fact, I think
the authors should show the space induced by ALEBO embedding as a comparison. I suspect that with the imposed
constraint -1 <= (B^T)y <= 1 the space will have discontinuous regions and is also not fit to be modeled
with any GP.
3. The authors stated that "A product kernel in the true subspace will not produce a product kernel in the embedding;
we will see this more explicitly in Sec. 5.1" but I did not see it explicitly in Sec. 5.1. At the very
least, I do not see how REMBO fails to do the same thing. Also, the authors claimed that "Inside the embedding,
moving along a single dimension will move across all dimensions of the true subspace, at rates depending
on the angles between the embedding and the true subspace". This seems like a very qualitative claim.
Can the authors formally define what this statement means, and prove it or at least provide some backup citations?
Other comments:
4. Why do the authors use conjugate transpose (if B^T means what i think it means) instead of normal transpose
when B is drawn from R^{d_e \times d}? Shouldn't they be the same?
5. Please explain the choice of \Epsilon(B) = {x: B^T B x = x}
6. The experiments provided are very limited. There is only one set of experiments showing performance of
ALEBO against other methods and it was done on a very small extrinsic dimension too (D = 100). I would like to see how
ALEBO scales with truly large dimension (REMBO also claimed that it could scale up to much higher extrinsic dimension). What about other important properties like does it guarantee that an optimum lies in the constraint space? What about rotational invariance? There are so many elements missing from the analysis.
Overall conclusion:
This paper is largely empirical and lacks technical depth. It is not at all convincing that the problem it
addresses is real, much less important. It also does not offer strong empirical evidence (too few experiments).
Given these reasons, I do not think the paper is not ready to be published as it is.


Asynchronous distributed methods are a popular way to reduce the communication and synchronization costs of large-scale optimization. Yet, for all their success, little is known about their convergence guarantees in the challenging case of general non-smooth, non-convex objectives, beyond cases where closed-form proximal operator solutions are available. This is all the more surprising since these objectives are the ones appearing in the training of deep neural networks.
In this paper, we introduce the first convergence analysis covering asynchronous methods in the case of general non-smooth, non-convex objectives. Our analysis applies to stochastic sub-gradient descent methods both with and without block variable partitioning, and both with and without momentum. It is phrased in the context of a general probabilistic model of asynchronous scheduling accurately adapted to modern hardware properties. We validate our analysis experimentally in the context of training deep neural network architectures. We show their overall successful asymptotic convergence as well as exploring how momentum, synchronization, and partitioning all affect performance.

This paper proposes a model to study asynchronous stochastic subgradient methods for minimizing a nonsmooth nonconvex function. Studying stochastic subgradient methods in the nonsmooth nonconvex setting is already very challenging. Throwing asynchronous updates into the mix further complicates the analysis. This is overcome by proposing a model for asynchrony which captures the salient features of computation platforms, while being amenable to analysis.
The main drawback of this paper is that the experiments seem to suggest that, although asynchronous methods do a good job optimizing training metrics (loss, accuracy), the models they train do not generalize as well as using synchronous SGD. This demotivates the need for the theory developed in this paper - rather, the theory is ahead of it's time. The paper could be significantly strengthened by reporting an example application where using an asynchronous stochastic subgradient method is advantageous for training deep networks.
The notation and discussion in Sec 3.1 is pretty heavy. Can you explain how this model differs from other models of asynchrony, e.g., as put forward in the well-known book by Bertsekas and Tsitsiklis on Parallel and Distributed Computation?
The main convergence result (Theorem 1) holds for a continuous-time process, following the typical analysis of stochastic approximation algorithms. However, practitioners rarely use a diminishing step size, especially for training deep networks. Is it possible to quantify the effect of using a constant step size? (E.g., in stochastic approximation analyses for smooth functions, one typically gets a "law of large numbers" which ensures convergence of the average, and a "central limit theorem" which bounds the distance of any realization of the process to its mean. Is there an analogous CLT for this setting?)
In the experiments, the SGD with DataParallel is effectively using a mini-batch size that is 3-5x larger than that of a single worker. In this case one would expect to be able to use a larger learning rate too (since there is less noise). Evidence for this is provided in Goyal et al., "ImageNet in 1 hour". Did you try tuning the learning rate separately for each method?
How were the parameters of the deep CNNs used in the experiments divided into blocks? This choice should affect the average time to evaluate a block gradient, right? Computing the gradient of parameters in the lowest layer requires doing a full forward and backward, while computing the gradient of higher layers should be faster since it doesn't require a full backward pass. Did you measure the average time to compute a block gradient, and could you report it in the paper? Did you divide into blocks based on layer? Are all parameters within the same block part of the same filter at a given layer?
The last paragraph of Sec 4 talks about the necessity of momentum, but it isn't clear what evidence this claim is based on.


We propose a novel approach for image segmentation that combines Neural Ordinary Differential Equations (NODEs) and the Level Set method. Our approach parametrizes the evolution of an initial contour with a NODE that implicitly learns from data a forcing function describing the evolution. In cases where an initial contour is not available or to alleviate the need for careful choice or design of contour embedding functions, we propose using NODEs to directly evolve the embedding of an input image into a pixel-wise dense semantic label. We evaluate our methods on kidney segmentation (KiTS19) and on salient object detection (PASCAL-S, ECSSD and HKU-IS). In addition to improving initial contours provided by deep learning models while using a fraction of their number of parameters, our approach achieves F β scores that are higher than several state-of-the-art deep learning algorithms.

This paper proposes to apply the Neural ODE framework (Chen et al 2018) for image segmentation. The method relies on contour delineation through Level Sets. Since contour estimation requires to solve an ODE, this naturally allows to apply the work presented in (Chen et al 2018). The method is here applied in two segmentation tasks: kidney segmentation and salient object detection.
The concept underlying the paper is interesting, and leverages on very recent advances in the field. The idea of learning the dynamics required to evolve segmentation contours is original and certainly appealing. Unfortunately the content of this work seems quite preliminary in terms of presentation and experiments.
First, the methodology is only sketched, while the motivation underlying the modelling rationale is often missing. For example, it is not clear what is the difference between "image evolution" and "contour evolution" models, besides the implementation details, and what motivates the definition of these two different modelling approaches in parallel.
Second, the experimental paradigm is controversial. The application on medical imaging is overly simplistic, as the authors do not consider the original 3D image stack, but rather the set of corresponding 2D slices modelled independently. The paper seems to ignore the large variety of body organ segmentation methods already available to the community, most of them working in 3D (e.g. [1-4]). The paper should necessarily compare with respect to these approaches and, even more importantly, with respect to standard level sets methods.
From the practical perspective, the proposed method builds upon the results obtained with the UNet, and therefore is characterised by an additional computational burden. Given that the the training of neural ODE is not straightforward and computational expensive, the use of this model for achieving a tiny accuracy improvement seems overkill for this kind of application. Moreover, the segmentation accuracy is still computed slice-by-slice in the 2D images, and no information is available for the consistency of the reconstruction in 3D (regularity over the vertical axis).
Finally, the results reported in Table 3 are not clear, why the metrics of the competing methods are approximated (e.g. ~ 0.7), while for the proposed methods are given up to the 3rd decimal term?
[1] 3D Kidney Segmentation from Abdominal Images Using Spatial-Appearance Models. Fahmi Khalifa, Ahmed Soliman, Adel Elmaghraby, Georgy Gimel'farb, and Ayman El-Baz 1.
[2] Automatic Detection and Segmentation of Kidneys in 3D CT Images Using Random Forests. Rémi Cuingnet, Raphael Prevost, David Lesage, Laurent D. Cohen, Benoit Mory, Roberto Ardon. Medical Image Computing and Computer-Assisted Intervention – MICCAI 2012: 15th International Conference, Nice, France, October 1-5, 2012, Proceedings, Part III
[3] Multi-organ localization with cascaded global-to-local regression and shape prior. Medical image analysis. Gauriau, R., Cuingnet, R., Lesage, D., & Bloch, I. (2015), 23(1), 70-83.
[4] Joint Classification-Regression Forests for Spatially Structured Multi-object Segmentation. Ben Glocker, Olivier Pauly, Ender Konukoglu, Antonio Criminisi. ECCV 2012 pp 870-881


Generative models have achieved impressive results in many domains including image and text generation. In the natural sciences, generative models have lead to rapid progress in automated drug discovery. Many of the current methods focus on either 1-D or 2-D representations of typically small, drug-like molecules. However, many molecules require 3-D descriptors and exceed the chemical complexity of commonly used datasets. We present a method to encode and decode the position of atoms in 3-D molecules along with a dataset of nearly 50,000 stable crystal unit cells that vary from containing 1 to over 100 atoms. We construct a smooth and continuous 3-D density representation of each crystal based on the positions of different atoms. Two different neural networks were trained on a dataset of over 120,000 three-dimensional samples of single and repeating crystal structures. The first, an Encoder-Decoder pair, constructs a compressed latent space representation of each molecule and then decodes this description into an accurate reconstruction of the input. The second network segments the resulting output into atoms and assigns each atom an atomic number. By generating compressed, continuous latent spaces representations of molecules we are able to decode random samples, interpolate between two molecules, and alter known molecules.

PAPER SUMMARY: This paper addresses the problem of encoding and decoding 3D chemical structures, with the ultimate goal of generating 3D crystal structures. The authors propose an auto-encoder framework for encoding the 3D locations of atoms in the crystal to a latent representation and then decoding that representation back into 3D structure. The paper's contributions can be summarized as follows:
1) A data representation that converts the 3D atom locations to a 3D voxel density map, so that they can be encoded by a standard 3D convolutional network.
2) A decoder network that first estimates a 3D density map from the latent vector using upsampling and convolutions and then classifies the atom type (atomic number) per voxel using a 3D segmentation.
3) The network is applied on unit cells of crystals or repeated unit cells from a dataset of crystal structures and the network is shown to be able to accurately reconstruct the 3D density maps, predict the number of atoms in the cell and perform fairly well in their classification into atomic types.
I appreciate that the paper addresses an interesting problem that is not sufficiently explored and can motivate the development of novel methods that generate 3D molecules with particular structure and multiple types of atoms, however the current work combines existing methods, without any architectural modifications that exploit the new domain.  If the presentation of results and experiments is improved, this could be a good application paper in material science with an interesting combination of techniques from machine learning and computer vision. It would be more appropriate to submit this paper to a domain-specific venue rather than to ICLR. Therefore, I cannot justify its acceptance to ICLR in its current format.
Strengths
-----------------------
1.	New domain: The paper addresses an interesting problem in a new domain. Previous work on generative models for molecules have concentrated on molecules with 1D structure (which can be represented as strings) or 2D structure (which can be represented as planar graphs). Generating 3D molecule structures is a challenging and interesting problem.
2.	Efficient data representation: The authors bypass the difficulties associated with modeling sets of 3D points with arbitrary structure by proposing a canonical, voxel-based density representation.
3.	Joint VAE + UNet training: The joint training of the encoder-decoder VAE and the 3D segmentation network results in a decoder that can reconstruct atom locations and atom types, being robust to mistakes in the density map reconstruction.
4.	Qualitative results: There are nice visualizations of the reconstructed molecules and of the effect of the latent variable z. Figure 3 in particular does a great job at elucidating the outputs of the network and the reconstruction quality.
Weaknesses
------------------------
1.	Limited methodological novelty: The methods used in the paper, i.e. the data representation (see detailed comments), the encoder network, the decoder network and the segmentation network are all existing methods without any (or with only minor) modifications.  Their combination also seems straightforward, except for the joint training of the VAE and the segmentation network, which has not been tried before to the best of my knowledge. Networks have not been modified to exploit the intricacies of the new domain/task, such as symmetries, repetition of unit cells, large number of atoms.
2.	Lack of comparison with other methods/baselines: There is no quantitative comparison with alternative baselines or methods or even discussion of such alternatives. It is understandable that the paper addresses a relatively new problem, however the method could be compared to CrystalGAN. Also, there is no justification of the advantages of using a voxel-based density map representation along with 3D convolutions vs, for instance, a graph representation along with graph convolutions (Xie & Grossman, [3]) or a point cloud GAN (Achlioptas).
3.	No quantitative evaluation of the generative capabilities of the network: In the case of drug molecule generation, logP and QED scores  [4,5] are used to evaluate their drug-likeness. Without such scores in the crystal generation domain, it is not easy to judge how good a generated crystal is.  The need for some type of quantitative evaluation is especially important, since it is not as easy to qualitatively judge the quality of crystals obtained by sampling random latent vectors (Fig 5B) as in the case of generated images or text. The only such evaluation presented in the paper is related to the distribution of distances between atoms which seems to be close to their actual distribution in nature.
4.	Presentation of quantitative results: The quantitative results are scattered in text throughout the results section without being summarized in a table. Results about the distance of generated atoms w.r.t to their true location, the predicted atom counts, predicted atom types etc. for the cases of single unit cells and repeated unit cells should be added to a table to complete a rigorous experimental evaluation.
Additional Comments
-------------------------
1.	Lack of novelty: The data representation, which is presented as one of the core contributions of the paper, is not new. Radial basis functions have been used to generate 3D density maps (e.g. [2]) and 3D Convolutional VAEs (Brock2016, [1]) have been employed on density maps (e.g. occupancy maps) on computer vision tasks.
2.	Confusing description of the repeated unit cells case: It is not clear, especially for a general audience, how the repeated unit cells data are generated (in which directions are the unit cells repeated, how many times). It is also not clear how the training routine (conditioning) and the output post-processing (connected components + majority voting) is modified.
3.	After Eq. (2), q is the --approximate-- posterior …
Suggested References
--------------------------
1.	VoxNet: A 3D Convolutional Neural Network for Real-Time Object Recognition
2.	VV-NET: Voxel VAE Net with Group Convolutions for Point Cloud Segmentation
3.	Dynamic Edge-Conditioned Filters in Convolutional Neural Networks on Graphs
4.	Graph Convolutional Policy Network forGoal-Directed Molecular Graph Generation
5.	Grammar Variational Autoencoder
6.	3D U-Net: Learning Dense VolumetricSegmentation from Sparse Annotation
7.	PointConv: Deep Convolutional Networks on 3D Point Clouds


Over the last few years deep multi-agent reinforcement learning (DMARL) has become an increasingly active area of research with hundreds of papers submitted to top machine learning conferences every year. However, so far there have been few real world use cases that benefited from the progress in the field. We use DMARL to develop a flexible computational model of fake news on social networks in which agents act according to learned best response functions. We achieve this by extending an information aggregation game to allow for fake news and by representing agents as recurrent deep Q-networks (DQN). In the game, agents repeatedly guess whether a claim is true or false taking into account an informative private signal and observations of actions of their neighbors on the social network in the previous period. We incorporate fake news into the model by adding an adversarial agent, the attacker, that either provides biased private signals to, or takes over, a subset of agents. The attacker can follow either a hand-tuned or trained policy. Our model allows us to tackle questions that are analytically intractable in fully rational models, while ensuring that agents follow reasonable best response functions. Our results highlight the importance of awareness, privacy and social connectivity in curbing the adverse effects of fake news and open an entire new real world application area for DMARL.

Update: I thank the authors for their response and for improving the paper. However, I maintain my position that the paper lacks a significant technical contribution to learning algorithms and that the applicability of the proposed approach is remains questionable in the current state.
Summary:
The paper proposes the  use of deep multi agent reinforcement learning (DMARL) for modelling fake news propagation and detection in social networks. The agents observe an informative yet noisy private signal and the actions of their neighbors (in the social network graph) and have to guess whether a claim (related to the received signal) is true or false. The fake news is modelled as an adversarial attack to the graph that either provides a hand-coded biased private signal to one of the agents or replaces one of  the agents with an RL policy trained to minimize the total reward of the agents in the graph (i.e. social network).
Main Comments:
I lean towards rejecting this paper because I do not find the methodological contribution to be significant enough to be published at ICLR, given that the main contribution is applying current techniques to a novel toy domain. While this paper attempts to apply DMARL to a new domain with real-world relevance, the authors only consider a toy example and make strong assumptions that are likely to break in the real world. Hence, it is not at all clear whether or how the conclusions of this paper would translate to more realistic scenarios of fake news in social networks. While strong assumptions and toy examples are reasonable for showing algorithmic improvements, this paper does not propose any improvement to core DMARL algorithms, but merely applies current methods to a new toy domain.
I am also concerned about the lack of comparison with other approaches to information aggregation in social networks. While I admit I am not familiar with that literature, I would still find it useful to provide some comparisons with non-DMARL (e.g. heuristic or game theoretic) approaches or at least some motivation for not comparing against those methods. The authors qualitatively describe those methods and their shortcomings, but the experimental section does not support those claims due to the lack of comparison. Despite all these concerns, the paper does indeed open-up numerous research directions and I can imagine follow-up papers being written that relax some of the current assumptions.
Other Questions / Comments:
1. Can you provide  some motivation for choosing to model the social network as a Barabsi Albert graph and why this is a reasonable modelling choice?
2. What happens if instead of a clustered or balanced graph, you have some combination of the two? It seems to me like that would be a more realistic scenario (i.e. a large graph containing subgraphs with different structures). Can the framework generalize to that? How would the conclusions change?
3. Is there any evidence that the conclusions supported by the experiments in this paper hold in real-world social networks and model some realistic aspects of social network dynamics?  Without such evidence, it is difficult to assess the relevance of this work for the  real-world application. Since the behavior of the agents in the graph is not guaranteed to be optimal / a best-response or even stable, is it at least a good approximation to human behavior in social networks? It would also be useful to show more comparisons against best-response or heuristic agents.
4. What is  the motivation behind considering a fixed budget of bias? Why not instead have a fixed number of agents that you will be biased? I think it would be informative to compare against applying beta = 3 to two citizens,  along with the focused and spread scenarios.
The legend in Figure 2 A & B is slightly confusing. I’d suggest using different styles for “private signal optimal” and “all signals optimal”.
5. Can you include error bars in Figure 2 D?
6. Can you provide results with a heuristic attacker that always lies about the claim? I read your intuition of why  you believe this wouldn’t be stronger than an attacker that is trained with RL together with the other agents, but is it actually true in practice, do the agents really learn to easily detect the “lying” attacker and distrust it? Based on your results, doesn’t it mean that one can find a heuristic attacker that has an equivalent behavior to the learned one ? Can one build even stronger hand-tuner attackers  based on heuristics?


We tackle the challenge of using machine learning to find algorithms with strong worst-case guarantees for online combinatorial optimization problems. Whereas the previous approach along this direction (Kong et al., 2018) relies on significant domain expertise to provide hard distributions over input instances at training, we ask whether this can be accomplished from first principles, i.e., without any human-provided data beyond specifying the objective of the optimization problem. To answer this question, we draw insights from classic results in game theory, analysis of algorithms, and online learning to introduce a novel framework. At the high level, similar to a generative adversarial network (GAN), our framework has two components whose respective goals are to learn the optimal algorithm as well as a set of input instances that captures the essential difficulty of the given optimization problem. The two components are trained against each other and evolved simultaneously. We test our ideas on the ski rental problem and the fractional AdWords problem. For these well-studied problems, our preliminary results demonstrate that the framework is capable of finding algorithms as well as difficult input instances that are consistent with known optimal results. We believe our new framework points to a promising direction which can facilitate the research of algorithm design by leveraging ML to improve the state of the art both in theory and in practice.

Update to the Review after the rebuttal from the Authors:
After carefully reviewing the responses by the authors especially on my concerns about the significance of solving an instance of a given problem and the improvement in the exposition of the ideas I would like to amend my earlier decision and recommend to accept. For completeness below is the original review.

This paper introduces a framework to learn to generate solutions to online combinatorial optimization problems with worst case guarantees. The framework as the authors claim eliminates the need for manual hard to solve instance/data creation, which is necessary to teach the model to provide the aforementioned worst case guarantees.  Therefore the main contribution of the paper can be said that this framework shows that it is possible to train a machine learning model, which can learn an algorithm to solve hard online combinatorial optimization problems and this training can be done without knowing much about the actual optimization problem domain. The only input required is the way to calculate the objective function of the actual problem. This contribution is demonstrated on two classes of problems: Ski-Rental and Fractional AdWords. The framework requires two neural networks one for solution generation agent and one for problem instance generation. These two networks are trained jointly from scratch and the underlying algorithm for the training is provided.
Although a generic framework that learns to solve online combinatorial optimization problems without domain knowledge is by itself a very motivational goal neither the paper successfully demonstrates that the framework the authors propose achieves this goal nor it explains well enough why one would take the machine learning approach to find good algorithms to such problems. Is it because the ML solution would be faster to compute with big instances? Is it because with the proposed approach one can curate sophisticated heuristic solutions when provable optimality is out of reach?
This paper should be rejected because proposed method demonstrates that an instance of one class of problems, Fractional Adwords, can be learned to solve without domain expertise, however fails to prove that the approach would be beneficial for any other instances of the same problem. Although they show that the Ski Rental problem can also be learned to solve though it is trivial and does not even use the framework the authors propose in its full extent, ie. problem instances are not generated by use of a machine learning model, which is one of main claims the authors are making. Therefore I do not find being able to solve this problem as a supporting evidence for the contributions claimed. In particular there is not any theoretical not experimental evidence that the approach would scale to any instances where a pure optimization approach would be slow to provide any meaningful solutions. I find this important because for combinatorial optimization usually scale matters a lot. While a small instance of a problem can be solve by a general purpose solver quickly a small increase in the problem size can turn out to be intractable. When proposing a machine learning approach to such problems I would expect the model to scale better than pure optimization approach so that there would be demonstrable benefit. Although the paper proposes an interesting framework I would argue that it is a “green apple” in the sense that authors need to motivate the approach better and expand the contribution beyond solving a particular instance. Authors acknowledge the fact that their experimental setup is rather limited in Appendix C.1, which I agree with and they also claim that there is a representation for a uniform algorithm for any number of advertisers for the AdWords problem, however they leave this as a future work, which I find unfortunate. I would recommend taking this direction rigorously and expand the contribution, which would prove to be a very sound contribution.
In order to clarify the exposition the following are some questions:
1. Authors call the approach YaoGAN due to its structural similarity to GANs. I understand the fact that they are training two neural networks in an alternating scheme, which is similar to the GAN training. How can one evaluate the solutions generated by this framework similar to how GAN generators are evaluated? Can one walk the latent distribution of the algorithm agent and draw insights, which might lead into tailoring some algorithms that would be appropriate for some input distribution although in general inferior in terms of worst case guarantees?
2. The main technical contribution claim needs to be elaborated.  I understand how the game theoretic framework is established but how does this manifest itself in the algorithm described in Section 3.1 needs more explanation.
3. Authors claim there are two shortcomings of the previous method proposed in Kong et. al 2018. They need to elaborate how their method overcomes these issues better.
4. Authors state that fractional relaxation of combinatorial mainly integer optimization problems, which is accurate. Yet their approach is only able to solve the fractional version of the AdWords problem. In addition I agree with the fact that although continuous relaxations to integer optimization problems might provide insightful directions they usually employed to to prove bounds on the heuristic approaches. Yet the authors stop at only solving this version with a machine learning approach, which does not hit the bar for me. I would have expected the authors to at least elaborate on why the current framework is not suitable for the non-relaxed problem. What are the shortcomings?
5.In Appendix A authors talk about no-regret dynamics, which are relevant. However, they state they loosely follow this approach. What does that entail? What kind of theoretical guarantees are given up due to not following this, a better exposition on this topic would help to support the claims.
6. In appendix C.2 authors provide additional plots for the Fractional AdWords problem. However, they retain from providing any intuition about them. In particular what is the conclusion to be drawn from Figure 5.  This needs more elaboration. Is this way of training results expected? What is the lesson learned?
7.In Figure 8 they provide example data from experience array. What are the significance of these examples? How they help us understand the problem instance generation was actually able to find interesting instances? What kind of dynamics are under covered? These are not directly revealed by only looking at the pictures one needs more explanation to support the claims.


Model-based reinforcement learning methods typically learn models for highdimensional state spaces by aiming to reconstruct and predict the original observations. However, drawing inspiration from model-free reinforcement learning, we propose learning a latent dynamics model directly from rewards. In this work, we introduce a model-based planning framework which learns a latent reward prediction model and then plan in the latent state-space. The latent representation is learned exclusively from multi-step reward prediction which we show to be the only necessary information for successful planning. With this framework, we are able to benefit from the concise model-free representation, while still enjoying the data-efficiency of model-based algorithms. We demonstrate our framework in multi-pendulum and multi-cheetah environments where several pendulums or cheetahs are shown to the agent but only one of them produces rewards. In these environments, it is important for the agent to construct a concise latent representation to filter out irrelevant observations. We find that our method can successfully learn an accurate latent reward prediction model in the presence of the irrelevant information while existing model-based methods fail. Planning in the learned latent state-space shows strong performance and high sample efficiency over model-free and model-based baselines.

This paper presents a technique for model based RL/planning with latent dynamics models, which learns the latent model only using reward prediction. This is in contrast to existing work which generally use a combination of reward prediction and state reconstruction to learn the latent model. The paper suggests that by removing the state reconstruction loss, the agent can learn to ignore irrelevant parts of the state, which should enable better performance in settings where state reconstruction is challenging.
Overall the motivation for this work is good, and the idea is promising. Difficulty in reconstructing high dimensional states is a challenge for learning latent dynamics models. The paper is also very well written and easy to follow.
My concerns are centered around the experimental evaluation. Specifically, I see the following issues: (1) the experimental environments seem artificial, and hand tailored for this method, (2) given that the proposed method is a minor modification to the PlaNet paper, it seems that PlaNet should be included as a comparison (especially because it has been shown to work on high dimensional states), and (3) the proposed method seems very prone to overfitting to the given task, and there should be an analysis of how the proposed change affects generalization and robustness.
(1): The testing environments contain many distractor pendulums/cheetahs, which makes state reconstruction especially challenging. While this does seem to be the point the authors are trying to show, the environments are an extreme, almost artificial, case of difficult state reconstruction. Would the same results hold in more realistic settings, for example, visual robot manipulation in a cluttered scene? Model based RL with video prediction models has been shown to work in such real cluttered robot manipulation environments. Showing that the proposed method can outperform such approaches in robot manipulation settings would be a powerful result. The results on images in the appendix seem to show a delta between the true and predicted reward, suggesting that the proposed method does not yet work on images. Why might this be the case?
(2): From what I can see, the proposed method is very similar to the PlaNet algorithm with state reconstruction loss removed. Given the similarity, PlaNet should be included as a comparison in both the pendulum and cheetah environments. Similarly why was DeepMDP performance not shown in the Cheetah environment?
(3): One of the strengths of model based reinforcement learning is the ability to plan to reach unseen goals with a model trained via self-supervision or different goals. Does the proposed approach lose some of this, by overfitting to only the task reward? I suspect that in generalizing to unseen tasks, a model trained with state prediction would potentially perform much better. If trained on many tasks, could this method achieve similar generalization?
Due to some of these questions which remain unanswered by the experimental evaluation my current rating is Weak Reject. If the authors are able to clarify some of the questions above I may adjust my score.


Convolutional Neural Networks (CNNs) have emerged as highly successful tools for image generation, recovery, and restoration. This success is often attributed to large amounts of training data. However, recent experimental findings challenge this view and instead suggest that a major contributing factor to this success is that convolutional networks impose strong prior assumptions about natural images. A surprising experiment that highlights this architectural bias towards natural images is that one can remove noise and corruptions from a natural image without using any training data, by simply fitting (via gradient descent) a randomly initialized, over-parameterized convolutional generator to the single corrupted image. While this over-parameterized network can fit the corrupted image perfectly, surprisingly after a few iterations of gradient descent one obtains the uncorrupted image. This intriguing phenomenon enables state-of-the-art CNN-based denoising and regularization of linear inverse problems such as compressive sensing. In this paper, we take a step towards demystifying this experimental phenomenon by attributing this effect to particular architectural choices of convolutional networks, namely convolutions with fixed interpolating filters. We then formally characterize the dynamics of fitting a two-layer convolutional generator to a noisy signal and prove that early-stopped gradient descent denoises/regularizes. This result relies on showing that convolutional generators fit the structured part of an image significantly faster than the corrupted portion.

This paper studies the situation in which a two-layer CNN with RELU nonlinearity is fit to a single image and the observation that it is able to fit a "natural" image in fewer iterations than a "noisy" image. Theorems on the convergence of this fitting are discussed and proven in the appendices. Intermediate results study the convergence of fitting a linear model to an image plus noise and fitting a single-layer CNN. Denoising is demonstrated on two images with additive white Gaussian noise and the approach under study is shown to provide a better signal-to-noise ratio than BM3D, another untrained denoising approach. The main result is that the use of upsampling via a fixed interpolation filter provides an inductive bias towards "natural" images.
The bibliography does a good job of positioning this paper within the recent set of articles exploring the curious behavior of fitting a CNN to a single image. The problem is interesting, timely, and surprising. The analysis does provide some insight into what is going on.
But, the paper would do well to embrace the Fourier domain and first discuss what is meant by "natural images" and "noise" in terms of their frequency content. In particular, something like Simoncelli and Olshausen (2001) could be used as a description of the spectra of natural images. Additive white Gaussian noise has a flat spectrum, which is never mentioned in the paper. Thus, the theoretical result mainly highlights the fact that natural images have a low-pass spectrum, while white noise has a flat spectrum, and CNNs using interpolation also have a low-pass spectrum in some sense. This makes sense from a frequency perspective, because interpolation increases the sampling rate of a signal without changing its frequency content, i.e., it adds high frequencies with no energy.
The "trigonometric basis" used in the paper, which consists of sines and cosines at each frequency, could be more cleanly described as a basis of complex exponentials, i.e., the Fourier basis, which doesn't require partitioning the basis into a cosine half and a sine half. The discussion of triangular and Gaussian smoothing functions has been very well explored in the signal processing literature discussing windowing functions and their Fourier transforms, e.g., Harris (1978). I don't think Figure 3 showing some sinusoids is necessary.
The main body of the paper goes into the 10th page, but the appendices make up another 18 pages. The main body of the paper does not include any of the proofs of the provided theorems. This seems rather excessive.
Harris, F. J. (1978). On the use of windows for harmonic analysis with the discrete Fourier transform. Proceedings of the IEEE, 66(1), 51-83.
Simoncelli, E. P., & Olshausen, B. A. (2001). Natural image statistics and neural representation. Annual review of neuroscience, 24(1), 1193-1216.
Minor comments:
Several citations are in the wrong form (\citet instead of \citep) throughout the paper.
There is a link to a figure in the appendix that is broken on page 5
In figure 7, one one set of y-axis labels is shown, but it appears that each subplot uses an independent y-axis, just without labels. Please plot them all on the same y-axis.
Similarly, figures 1, 4, and 6 show similar things on different y-axes when they could be plotted at the same scale on the y-axis to make them more easily comparable visually.
Typos: "over-paramtrized" and "spacial" both on page 1.
After discussion:
The authors have addressed my concerns, so I am changing my decision from Weak Reject to Weak Accept.


Identifying computational mechanisms for memorization and retrieval is a longstanding problem at the intersection of machine learning and neuroscience. In this work, we demonstrate empirically that overparameterized deep neural networks trained using standard optimization methods provide a mechanism for memorization and retrieval of real-valued data. In particular, we show that overparameterized autoencoders store training examples as attractors, and thus, can be viewed as implementations of associative memory with the retrieval mechanism given by iterating the map. We study this phenomenon under a variety of common architectures and optimization methods and construct a network that can recall 500 real-valued images without any apparent spurious attractor states. Lastly, we demonstrate how the same mechanism allows encoding sequences, including movies and audio, instead of individual examples. Interestingly, this appears to provide an even more efficient mechanism for storage and retrieval than autoencoding single instances.

This paper empirically demonstrates that DNNs can be trained to be identity mappings for small quantities of samples. It also demonstrates that for many parameterizations, these identity DNNs also have a small number of attractors, iterative fixed points, and can also learn short circular sequences of examples.
The paper is well written and easy to understand, although the presentation could be improved a bit (see comments). Its contents aren't particularly novel in terms of ideas, but they investigate memorization and attractors much further than previous studies.
In a way, the memorization results are unsurprising. We know that DNNs can memorize perfectly, including sequences, so it is natural that by increasing capacity, at some point they should be able to memorize entire images (in fact this is what Zhang et al. (2019)'s Figure 1 appears to be showing).
The more novel and surprising aspect of this is that DNNs would learn such strong (and so few) attractor basins. The fact that deep autoencoders could have attractors centered on the training points has been postulated before (see [1]), but this work makes a stronger case for it.
A crucial aspect that is missing from this paper in order for me to give in an accept is that there is very little about how this paper positions itself in the current literature. There could be much more discussion about related work, and much more discussion about the impacts of these findings.
I have given this paper a 'weak reject' mark but I think with some work this paper could be of interest to many. To reiterate, I am unable to see anything wrong with this paper, but at the same time I am unable to see how impactful these findings are.
Detailed comments:
- It's interesting that DNNs can implement associative memory, but what is the cost of doing that? Should we be using that in practice? Since there is no sense of how costly the presented experiments are, it is hard to tell.
- Again, these results are interesting, but after some time pondering about it, I can't really convince myself that knowing the results of this paper will be beneficial to future research. That being said, there are many areas of Machine Learning that I am unfamiliar with. It should be part of the paper to familiarize readers with areas where these results could be impactful.
- "the function interpolates the training images" not sure what this means. Interpolation means making a prediction for a point `u` that is "between" two points `x,y` with known values
- "black and white" should be "grayscale" if values are in [0,1]
- Figure 2b is interesting, but I wonder what happens if e.g. a perturbed version of e.g. Example 6 is fed. Presumably since Example 6 is not an attractor (Jacbian with an eigeinvalue > 1), it should converge to another example.
- Figure 2b's caption numbers, which say you use 1k example, to not correspond to numbers earlier in the text, which say you use 10k examples.
- "Since overparameterized autoencoders interpolate the training data", again this is a fairly important assumption and it needs to be defined very clearly, because it could mean many things.
- "it is essential that we interpolate to numerical precision", I don't think you are using the word "interpolate" correctly, do you mean "inference"? "train"?
- Adam citation should be "Adam: A Method for Stochastic Optimization, Diederik P. Kingma, Jimmy Ba", not Goodfellow et al., RMSprop should also have a citation, Hinton et al. 2012
- ReLU citation should be "Rectified linear units improve restricted Boltzmann machines, Nair & Hinton", Leaky ReLU should be Maas et al 2013, SELU should be Klambauer et al. 2017.
- The combination of section 3.1 and Figure 3 doesn't make it clear if models trained with Adam and RMSprop have weight decay or not. Can you clarify?
- "Note that a minimum width of 100 is needed to allow for interpolation." Again I think you mean "learning" rather than "interpolation".
- You say that you trained black and white images, but all the images of CIFAR10 in the figures are colored, including the inputs and outputs. Can you clarify why?
- You might be interested in [2], which is much older work about perceptrons, but still relevant to what is studied here.
- The linked supplemental material gives a 404 for me. I replicated the MNIST Figure 6 experiment. I was unable to replicate exactly your results but they were similar enough. In particular, the activation function choice seems to be critical.
[1] The Potential Energy of an Autoencoder, Hanna Kamyshanska, Roland Memisevic
[2] Basins of Attraction in a Perceptron-like Neural Network, Werner Krauth, Marc Mezard, Jean-Pierre Nadal