We present Decentralized Distributed Proximal Policy Optimization (DD-PPO), a method for distributed reinforcement learning in resource-intensive simulated environments. DD-PPO is distributed (uses multiple machines), decentralized (lacks a centralized server), and synchronous (no computation is ever 'stale'), making it conceptually simple and easy to implement. In our experiments on training virtual robots to navigate in Habitat-Sim (Savva et al., 2019), DD-PPO exhibits near-linear scaling -achieving a speedup of 107x on 128 GPUs over a serial implementation. We leverage this scaling to train an agent for 2.5 Billion steps of experience (the equivalent of 80 years of human experience) -over 6 months of GPU-time training in under 3 days of wall-clock time with 64 GPUs. This massive-scale training not only sets the state of art on Habitat Autonomous Navigation Challenge 2019, but essentially 'solves' the task -near-perfect autonomous navigation in an unseen environment without access to a map, directly from an RGB-D camera and a GPS+Compass sensor. Fortuitously, error vs computation exhibits a power-law-like distribution; thus, 90% of peak performance is obtained relatively early (at 100 million steps) and relatively cheaply (under 1 day with 8 GPUs). Finally, we show that the scene understanding and navigation policies learned can be transferred to other navigation tasks -the analog of 'ImageNet pre-training + task-specific fine-tuning' for embodied AI. Our model outperforms ImageNet pre-trained CNNs on these transfer tasks and can serve as a universal resource (all models + code will be publicly available).

The contribution of this paper is twofold. First the authors propose and implement and new scalable RL training procedure they call 'Decentralized Distributed Proximal Policy Optimization' (DD-PPO). This method extends PPO in a way that is 'synchronous' and 'distributed': no parameter server exists to asynchronously collect experience from worker agents and instead an 'explicit communication state' exists during which all workers communicate gradient updates between one another to update parameters based on their experiences. However, naively implementing this approach is limited by 'stragglers', the slowest of the workers that all other agents need to wait for. To overcome this limitation, the authors add a 'preemption threshold' which halts worker rollouts after a certain percentage has completed. The authors use this procedure to tackle the task of PointGoal navigation, in which an agent tries to reach a point in space specified by its relative location to the agent. They achieve state-of-the-art performance (arguably 'mastery') on this task. Finally, the authors show that their learned policy transfers reasonably well to two other navigation tasks, 'flee' and 'exploration'. Code is also provided.
This is an impressive paper. The algorithmic contribution is useful and the results are state-of-the-art. The paper is written very clearly, and it was a pleasure to read. The investigation of the task of PointGoal Navigation, and the impact of various changes on the overall performance of the system . The transfer tasks, to the 'flee' and 'exploration' tasks, were interesting, and further reinforced the results in the paper.
My only high-level 'complaint' is that the title might be tempered: 'Mastering PointGoal Navigation' is a very strong claim. The results in the paper are very impressive, but 'Mastering', in my mind, implies (among other things) that the agent is guaranteed to reach its goal, though I understand that it is perhaps a minor point, considering the failure rate is systematically <3%. I also make a comment later about how the Matterport 3D environments may be a biased set, and that there are almost certainly environments that *would* make this system fail, for one reason or another. I am willing to accept the title as-is (and could be overwritten by other reviewers), but I would like some discussion on a more appropriate tile, if possible.
I recommend this paper for acceptance.
Minor comments:
- Introduction: "Thus, there is a need to develop a different distributed architecture." It is unclear how this sentence logically follows from the previous sentence. I am on board with the vision in general, and it is quite clear to me: having a single parameter server is limiting. However, that the move from CPU- to GPU-based agents is what reveals this limitation is not quite so clear. Reworking this paragraph (or perhaps adding a sentence) to make this clearer would be helpful.
- Introduction: "This means there is no scope for mistakes of any kind --- no wrong turn at a crossroad, ..." This is a fascinating point, though I feel reflects a bias in the types of environments that appear in the training data set. It is a defining behavior of intelligent embodied agents that they are capable of recovering from their mistakes; in general, real-world environments and navigation tasks have inherent ambiguity that cannot be resolved without exploration (a 'maze' is an extreme example of this). One question comes to mind: if the environments were ambiguous in this way, *could* the agent recover from its mistakes. There is not much evidence in the paper one way or the other. Since there are few examples in the data that seem to bring about such scenarios, it is not obvious how well this agent would perform in the face of this uncertainty. However, the results later in the paper about how "No GPS+Compass remains unsolved" is related to this point. A comment (or a figure in the Appendix) from the authors describing some of the failure cases would be instructive: do these failures occur in environments with such an ambiguity or, conversely, are there some examples in which the agent succeeds in reaching the goal but must overcome failure during its travel?
- The acronym 'SL' (for Supervised Learning) is not defined and is used only once. Prefer using 'supervised learning'.


We study the roots of algorithmic progress in deep policy gradient algorithms through a case study on two popular algorithms, Proximal Policy Optimization and Trust Region Policy Optimization. We investigate the consequences of "codelevel optimizations:" algorithm augmentations found only in implementations or described as auxiliary details to the core algorithm. Seemingly of secondary importance, such optimizations have a major impact on agent behavior. Our results show that they (a) are responsible for most of PPO's gain in cumulative reward over TRPO, and (b)  fundamentally change how RL methods function. These insights show the difficulty, and importance, of attributing performance gains in deep reinforcement learning.

Summary
This paper calls to attention the importance of specifying all performance altering implementation details that are current inherent in the state-of-the-art deep policy gradient community. Specifically, this paper builds very closely on the work started by  Henderson et al. 2017, building a conversation around the importance of more rigorous and careful scientific study of published algorithms. This paper identifies many "code-level optimizations" that account for the differences between the popular TRPO and PPO deep policy gradient algorithms. The paper then subselects four of these optimizations and carefully investigates their impact on the final performance of each algorithm. The clear conclusion from the paper is that the touted algorithmic improvement of PPO over TRPO has negligible effect on performance, and any previously reported differences are due only to what were considered unimportant implementation details.
Review
This paper investigates the claims made by Schulman et al. 2017 carefully, by investigating the impact of PPO's clipping mechanism on maintaining a valid trust-region; the central claim made by PPO's originating paper. The empirical results suggest that PPO is not sufficient for maintaining a valid trust-region, however the "code-level optimizations" that differ between the TRPO implementation the PPO implementation are sufficient. The ablation study of the four optimizations studied by the paper shows dramatic and clear results suggesting that annealing stepsizes and normalize rewards make very strong differences in learning performance; much more effect than demonstrated by the differences between TRPO and PPO's core algorithmic contribution as demonstrated in Figure 2 and even more strongly in Figure 3. I find the work included in this paper to be novel and a valuable contribution to the field.
For the above reasons, I recommend to accept this paper for publication at ICLR. In the following paragraphs I will discuss why I only recommend a weak accept instead of a strong accept.
My primary concern with the empirical study is the use of only three random seeds. As demonstrated in Henderson et al. 2017 (which is heavily cited in this paper), using such a small number of random seeds can have very misleading results. Although the effects appear very strong in the empirical studies in this paper, the effects likewise appear strong in Henderson et al.'s Figure 6 where 10 random seeds were split into two groups for the same algorithm. For this paper to make such strong claims about the negligence of the careful scientific study on TRPO and PPO, it would be best if this paper included far more random seeds in its investigation.
My second concern is with the discussion and conclusions drawn from Tables 1 and 2. It appears that the inclusion of clipping plays a strong role in the variance of each algorithm on every domain except Hopper. Specifically, the algorithms that include clipping appear to be much lower variance than the algorithms including clipping. Admittedly using only 3 seeds means that investigating the variance appropriately is near impossible (see the above paragraph), however variance should be considered and discussed in a conversation about the effects of the core contribution of PPO. If clipping leads to more consistent results across runs, even if those results are a little worse, it is still a valid and important contribution.
The paper cites Henderson et al. 2017 in several places. I would point out (perhaps in the introduction) that this paper builds on work already done in Henderson et al. 2017. Specifically, Henderson et al. 2017 investigates the effects of using different codebases for TRPO and shows that these different codebases result in dramatically different performance. The similarity to the investigation in this paper to too close to be unreported. However, I find that the investigation in this paper is much more complete and insightful than that of Henderson et al. 2017 (this paper has a more narrow focus), thus contributes significantly and meaningfully to this ongoing conversation.
Additional Comments (do not affect score)
It might be worthwhile to move the related work section to the beginning of the paper, either merged with the introduction or immediately after. This section is of critical importance to understanding the scope of this paper and for understanding why you are studying what you study. In fact, there is already a bit of duplication between the related works and introduction sections, so the paper could likely gain some additional real-estate by combining these.
I disagree with the terminology "code-level optimizations" and I find that it is misleading. This caused a bit of confusion on my first pass reading the paper, as I originally was expected the code differences to be more akin to using Tensorflow vs PyTorch or switching hash table functions, etc. Instead the changes focused on in this paper are changes to the problem specification and algorithm implementation. These are not simply implementation details as "code-level optimizations" suggests, but are rather details that necessarily must be included in peer-reviewed works. I don't have a suggested name to switch to, but felt strongly enough to mention it.


In this work, we first propose ParaNet, a non-autoregressive seq2seq model that converts text to spectrogram. It is fully convolutional and obtains 46.7× speed-up over Deep Voice 3 (Ping et al., 2018b) at synthesis while maintaining comparable speech quality using a WaveNet vocoder. ParaNet also produces stable alignment between text and speech on the challenging test sentences by iteratively improving the attention in a layer-by-layer manner. Based on ParaNet, we build the first fully parallel neural text-to-speech system using parallel neural vocoders, which can synthesize speech from text through a single feed-forward pass. We investigate several parallel vocoders within the TTS system, including variants of IAF vocoders and bipartite flow vocoder.

This paper proposes improvements to the TTS architecture in DeepVoice3.
For the most part (as far as I am aware), neural architectures for TTS are encoder-decoder based. The backbone can either be an RNN or a wavenet style model (autoregressive). Synthesizing audio from the feature representation (mel spectrogram) by a neural vocoder is usually an autoregressive model from the wavenet family.
In the current work, the main novelties seem to be the following:
1) Replace the autoregressive component in the decoder (seemingly inspired by DeepVoice3) with a non-autoregressive model. This could be very advantageous because synthesis in autoregressive models can be quite slow owing to the sample level generations (and to overcome this defect, faster sampling with inverse autoregressive flows has been used in parallel wavenet, clarinet, etc. and probability density distillation). Here, if I can interpret the paper correctly, the authors use a trained attention model (autoregressive), to distill attention for the non-autoregressive setup used, which would otherwise have difficulties learning alignment. The paper claims speed up of ~50X over DeepVoice 3 which is very significant.
2) In the mel to audio converter (vocoder), the proposal is a VAE Wavenet, with appropriate modifications for the sequence modeling. The authors mention that there are similarities to the approach used in Clarinet (this has a closed form KLD between the distilled distributions, which makes things easy).
Experiments: It is mentioned that poor attention alignments are the cause of many issues in these architectures (repeats, mispronunciations, skipped words, etc.), and they go on to show that their architecture fares well as regards these metrics, while maintaining a comparable MOS score with DeepVoice3+Wavenet.
Evaluations: I am reasonably convinced by the audio demos provided.
My thoughts:
The paper is generally a good addition to the TTS literature. These are difficult to implement, brittle setups, and a practitioner could spend a lot of time debugging their broken attention curves. To that end, I feel that any technique that throws light into the modeling process would be useful for the practitioner. It is suggested that we can use a non-autoregressive model with speedups. Likewise, they also use a wavenet VAE, which they claim can be trained without distillation (could the authors please clarify this point?)
I do, however, feel that the paper has a few drawbacks.
1) The presentation is not at all clear. This is a very subjective comment, but I feel that this work might be unreadable to someone who hasn't studied the literature (starting from Tacotron, DeepVoice 1, 2, 3, wavenet, parallel wavenet, transformer, distillation, etc.). Furthermore, the paper does not seem to make things any clearer with succinct architecture diagrams. Just as a comparison, I would like to draw attention to Tacotron (1, 2) in which I think the details can actually be worked out with some effort.
2) Why are we using the WaveVAE - is this just to do away with the probability distillation in wavenet? Are we also using the  IAF setup as described in Kingma's work?
3) I really feel that we need more architecture diagrams for this work to be useful. That being said, the authors do provide a list of hyperparameters for potential use.
4) Distilling attention seems to require a previously trained autoregressive attention model. What if we don't have one ready at hand?
In summary, while I see that the work will definitely be useful to the Speech Synthesis practitioner, the clarity of the paper could be improved and we need a few more diagrams (maybe even code) to make it implementable.


For multi-valued functions-such as when the conditional distribution on targets given the inputs is multi-modal-standard regression approaches are not always desirable because they provide the conditional mean. Modal regression approaches aim to instead find the conditional mode, but are restricted to nonparametric approaches. Such approaches can be difficult to scale, and make it difficult to benefit from parametric function approximation, like neural networks, which can learn complex relationships between inputs and targets. In this work, we propose a parametric modal regression algorithm, by using the implicit function theorem to develop an objective for learning a joint parameterized function over inputs and targets. We empirically demonstrate on several synthetic problems that our method (i) can learn multi-valued functions and produce the conditional modes, (ii) scales well to high-dimensional inputs and (iii) is even more effective for certain uni-modal problems, particularly for high frequency data where the joint function over inputs and targets can better capture the complex relationship between them. We conclude by showing that our method provides small improvements on two regression datasets that have asymmetric distributions over the targets.

This paper considers the regression problem in scenarios in which the conditional distribution of the response variable y given the input x is multimodal. For artificially constructed datasets, in which the the conditional modes are known, performance is assessed by the RMS distance to the mode closest to each prediction.  For real-world datasets, performance is in terms of RMSE and MAE.
The authors propose to learn a function f(x,y) that takes both an input x and a proposed output y and produces a value of 0 when the output y is correct for the input (i.e a conditional mode of p(y|x)), and hopefully a nonzero output when y is incorrect.  To predict on a point x, one searches for a y that makes f(x,y)=0.  For this to work, we will need f(x,y)≠0 for incorrect y. To achieve this, the authors take inspiration from the implicit function theorem and additionally seek an f with ∂f/∂y=-1 at the modes for each x.  For implementation, they seeks an f that minimizes the objective f²(xᵢ,yᵢ) + ( ∂f(xᵢ,yᵢ)/∂y - 1 )² over the training data.  To predict y for a given x, they find argmin_y f²(x,y) + ( ∂f(x,y)/∂y - 1 )² by grid search over the range of y, which they assume to be known.
I think this is a very interesting and novel approach to regression.  It seems to work well in artificial examples and competitively on some real world examples.
My main concern is that, although the objective function is fairly intuitive, it is not  clear to me what the properties are of the population minimizer (i.e. in the infinite data case).  Equation (1) defines modes as local maxima of the density, and for evaluation purposes on the artificial data, we look at deviation from the closest conditional mode.  With infinite data, do we expect the f to converge to something that's 0 at all the conditional modes, or just at the ones with the largest density?  What do we even want to happen?  Can we use this approach to predict all the modes for each x?  While I consider all of these interesting questions, after some consideration, I don't think that missing answers to these questions should delay publication.
I recommend accepting this paper, assuming reasonable responses can be provided to the questions and issues below.  It's a novel and promising approach to an interesting problem in regression.  It leaves some open questions for further theoretical analysis, but that seems ok to me.
Specific questions:
- In section 3, second sentence, I think you want to write either the minimum of f^2 or a zero of f?
- Page 3, not sure what is meant by "should be described by the same mapping g_j(x)."
- In the implicit function theorem, it ends with a capital F, but I think you want f.
- When you say "this condition is only local and may not encourage a minimal set of conditional modes" -- I guess you mean a minimum set of zeros? Or predicted conditional modes?
- In Equation 5, you write a full l_2 norm for what I believe is just a scalar -- how about parenthesis instead?
- When you make your y prediction using the argmin... it seems like you could also minimize without the partial derivative term, as in your original explanation of f.  Did you compare the performance with and without the partial derivative term?
- In comparing to the MDN network, it's interesting that MDN needs more mixture components than the true distribution, but that doesn't really seem like a point of criticism.  Neural networks themselves seem to work better with far more parameters than we think they should need.  To be fair, I'd suggest hyperparameter searching over a broader range of mixture components for MDNs, maybe up to 10.
- In your section "verify the learned error distribution", you say that "it is obvious that the error function indeed looks like a Gaussian distribution and the one trained without noise 4(a) shows an extremely small variance."  Two comments: 1) It's definitely not obvious that 4(a) has a Gaussian distribution. 2) Why would you expect 2a to be Gaussian?  The variance should be entirely due to the randomness in the selection of training data, which has nothing to do with Gaussians in the 0 noise case, as far as I can tell.
- In Figure 5, are the error bands the standard error of the mean, or the standard deviation across trials, or something else?
- I don't at all understand the setup of the 'Examining the neural network representation'.  I understand what the embedding of x is for the L2 model but what does that mean for the Implicit model, in which an x cannot be embedded without a y?


Named entity recognition (NER) and relation extraction (RE) are two important tasks in information extraction and retrieval (IE & IR). Recent work has demonstrated that it is beneficial to learn these tasks jointly, which avoids the propagation of error inherent in pipeline-based systems and improves performance. However, state-of-the-art joint models typically rely on external natural language processing (NLP) tools, such as dependency parsers, limiting their usefulness to domains (e.g. news) where those tools perform well. The few neural, end-to-end models that have been proposed are trained almost completely from scratch. In this paper, we propose a neural, end-to-end model for jointly extracting entities and their relations which does not rely on external NLP tools and which integrates a large, pre-trained language model. Because the bulk of our model's parameters are pre-trained and we eschew recurrence for self-attention, our model is fast to train. On 5 datasets across 3 domains, our model matches or exceeds state-of-the-art performance, sometimes by a large margin.

The paper proposes a new joint learning algorithm that works for two tasks, NER and RE. The model is based on a pre-trained BERT model, which provides the word vectors of the input word sequence. Then it solves two tasks with two network branches: the first branch minimizes the loss for NER, and the second branch minimizes the loss for RE. The second branch uses entity labels predicted by the first branch, so joint learning may benefit both tasks.
The design of the architecture is novel, but it is also not groundbreaking. Each network branch is from known structures, but the combination is not proposed before.
The submission has evaluated the proposed algorithms on four datasets and improved SOTA performances. The ablation study justifies the design details.
The writing is generally clear.
Now critics:
Ablation study:
1. As pointed by one public comment, the ablation study should show how much improvement is from BERT vectors.
2. I'd like to see another ablation study of whether RE helps NER. If you remove the RE component, does the NER performance suffer?
Writing:
3. how are predicted labels embedded? Do you learn a vector of each tag of BIOES and then take a weighted sum of these vectors with predicted probabilities as weights?


The ability to design biological structures such as DNA or proteins would have considerable medical and industrial impact. Doing so presents a challenging black-box optimization problem characterized by the large-batch, low round setting due to the need for labor-intensive wet lab evaluations. In response, we propose using reinforcement learning (RL) based on proximal-policy optimization (PPO) for biological sequence design. RL provides a flexible framework for optimization generative sequence models to achieve specific criteria, such as diversity among the high-quality sequences discovered. We propose a model-based variant of PPO, DyNA-PPO, to improve sample efficiency, where the policy for a new round is trained offline using a simulator fit on functional measurements from prior rounds. To accommodate the growing number of observations across rounds, the simulator model is automatically selected at each round from a pool of diverse models of varying capacity. On the tasks of designing DNA transcription factor binding sites, designing antimicrobial proteins, and optimizing the energy of Ising models based on protein structure, we find that DyNA-PPO performs significantly better than existing methods in settings in which modeling is feasible, while still not performing worse in situations in which a reliable model cannot be learned.

Contribution
This paper apply a model-based RL algorithm, DyNA-PPO for designing biological sequences. By being model-based, this algorithm is sample efficiency compared to model-free RL algorithms. This advantage is attractive and important in the context of biological sequence design since the designed is constrained to be done in the large batch / low round settings. To further improves model efficiency, the authors reduce learning bias by quantifying the reliability and automatically selecting models of appropriate complexity via cross validation. To encourage diversity in the target distribution they also penalize the reward using a visitation-based strategy.
Clarity
Overall, the paper is well written, well motivated and well structured. The technical content is also very clear and good.
Novelty
The novelty in this work seems to be more on the applicative side (RL to optimizing DNA and protein sequences) than the method itself. I agree with the authors that most existing optimization methods are ill equipped for the large batch / low round settings and as sample efficiency becomes critically important as the number of round gets lower and their method is a good solution in such settings.
The technical novelty seems incremental as cross-validating and using a set of models under particular performance constraints does not constitutes a novel contribution. Penalizing the reward if the same sequence is seen multiple times seems decent and works well compared to the entropy regularization but it is still questionable if it is the best solution for biological sequences. Showing results when the reward is penalized using the hamming loss or the biological similarity  with previous sequences could go a long way to convince of your choice.
Experiments:
The experiments are overall well presented and seems robust given the number of replicates that was made each time.
Analyzing the model performances across different metrics: diversity, fraction of optimals, cumulative maximum helped to  understand the method and its advantages.
However, I would like to see other RL algorithms that were shown in the appendix for all those comparisons.
Including MCMC methods in the experiments will also allow to see how RL methods compared to  the sota in bioinformatics.
For the Icing dataset, you mention that it is a contribution but do not provide enough details regarding it to allow further research with it.
In a real life biological setting, the data obtained at each batch will be more likely different both in term of sequences but also in term of labels. How all your method (and the others) perform with changing distribution of data from one batch to another. Ex (e.g. 0 < y batch 1 < 100, 100 <= y batch 2 < 500, etc) ?
You tried two values for R^2 in one experiment (one positive and one negative). What happens for any other positive values (e.g 0.1. 0.2, 0.3, etc) ?
Points of improvement
Given the applicative nature of the paper and the proposed method there are few small experiments that could have been done to strengthen the manuscript (see questions and comments above).
Preliminary rating:
* weak accept *


In reinforcement learning, we can learn a model of future observations and rewards, and use it to plan the agent's next actions. However, jointly modeling future observations can be computationally expensive or even intractable if the observations are high-dimensional (e.g. images). For this reason, previous works have considered partial models, which model only part of the observation. In this paper, we show that partial models can be causally incorrect: they are confounded by the observations they don't model, and can therefore lead to incorrect planning. To address this, we introduce a general family of partial models that are provably causally correct, yet remain fast because they do not need to fully model future observations.

This paper tackles the issue of identifying the causal reasoning behind why partial models in MBRL settings fail to make correct predictions under a new policy. The novel contribution was a framework for learning better partial models based on models learning an interventional conditional, rather than an observational conditional. The paper tried to provide both theoretical and experimental reasoning for this framework.
I vote to (weak) reject the paper due to the major issues with section 2. Furthermore, the paper hard or at times almost impossible to understand as too many assumptions are made and too little is explained.
Recommendations
Because your graphs are not MDPs, you are not framing your example as an RL problem. This is causing a number of issues with notation and lack of clarity in the argument you're making.
1. It is unclear to me that the FuzzyBear example is correctly constructed as a RL example, reasons being that:
- Figure 1 (a) & (b) do not correspond to an MDP as two different states, teddy vs grizzly, are both designated as s_1 and similarly, the two possible actions, hug or run, are both designated as a_1 and thus are not distinct.
- Note your terminal state for the episodes
- Have a reward for (s0, a0) as every s-a pair should have a reward
- It would be helpful to note that the environments in Figure 1 are stochastic
2. Clarify notation. There are a number of assumptions about what background knowledg the reader should have. Given the bridging of disciplines in the paper, it would be useful to provide more detail on notation in Section 3.
3. Add a section on reinforcement learning in Section 3. If it's the last subsection in section 3, you could describe the relationship between the various causal reasoning and RL principles. This would further clarify how you're bridging these subtopics.
4. For sentence,
"Fundamentally, the problem is due to causally incorrect reasoning: the model learns the observational condi- tional p(r|s0, a0, a1) instead of the interventional conditional given by p(r|s0, do(a0), do(a1)) = s1 p(s1|s0, a0)p(r|s1, a1)."
As you don't cover the meaning of the do() operator until a later paragraph, provide a quick description of it as it is not common knowledge to a general AI audience, e.g., where do() indicates that the action was taken.
5. Correct the following sentences,
"Mathematically, the model with learn the following conditional probability:"
"In Section 3, we review relevant concepts from causal reasoning based on which we propose solutions that address the problem."
6. I recommend putting the interventional conditional equation, p(r|s0, do(a0), do(a1)) = 􏰀s1 p(s1|s0, a0)p(r|s1, a1), on its own line as the reader is doing a comparison of it with the previous equation, p(r|s0, a0, a1), given on page 2.
7. Strengthen your abstract by aligning more with claims you make in your conclusion.
8. The experiments in Figure 5 are averaged over 5 seeds. This is not enough to be statistically significant - furthermore, there are no error bars in the Figure.
Question(s):
1. You've indicated two policies for Figure 1 (a):
- pi1: the agent knows it is encountering a teddy bear, so it will hug
- pi2: the agent knows it is encountering a grizzly bear, so it will run
Is this the "change in the behavior policy" that you're referring to? If so, make this clearer, this currently requires a lot of work by the reader to make sense of it.
2. What are the partially observable parts of the environments in Figure 1 (a) & (b)? Make this clear.


Many computer vision applications require solving multiple tasks in real-time. A neural network can be trained to solve multiple tasks simultaneously using multitask learning. This saves computation at inference time as only a single network needs to be evaluated. Unfortunately, this often leads to inferior overall performance as task objectives can compete, which consequently poses the question: which tasks should and should not be learned together in one network when employing multi-task learning? We systematically study task cooperation and competition and propose a framework for assigning tasks to a few neural networks such that cooperating tasks are computed by the same neural network, while competing tasks are computed by different networks. Our framework offers a timeaccuracy trade-off and can produce better accuracy using less inference time than not only a single large multi-task neural network but also many single-task networks.

This paper works on the problem if training a set of networks to solve a set of tasks. The authors try to discover an optimal task split into the networks so that the test performances are maximized given a fixed testing resource budget. By default, this requires searching over the entire task combination space and is too slow. The authors propose two strategies for fast approximating the enumerative search. Experiments show their searched combinations give better performance in the fixed-budget testing setting than several alternatives.
+ This paper works on a new and interesting problem. Training more than one networks for a few tasks is definitely a valid idea in real applications and related to broad research fields.
+ The baseline setup is comprehensive. The difference between optimal, random, and worst clearly shows this problem worths effort for research.
- I believe this problem setup requires a larger task set. In the paper the authors manually picked 5 tasks. It seems straightforward for a human to manually group them together: segmentation and edge/ surface + depth/ keypoint for 3 networks. It is unclear how better a network can do than a human in one minute or should we expect learning the task split is better than manual design.
- Both technical contributions in Section 3.3 look straightforward. Given the good performance in Figure3, it is fine.
- I am confused by the comparison to Sener and Koltun. How do you change the inference budget for them? If it is changing the number of channels for a single network, I believe it can be improved more.
Overall I believe this is a good paper to open interesting research direction with solid baselines. I am happy to accept this paper and see more exciting future works in this direction.


We study the following three fundamental problems about ridge regression: (1) what is the structure of the estimator? (2) how to correctly use cross-validation to choose the regularization parameter? and (3) how to accelerate computation without losing too much accuracy? We consider the three problems in a unified large-data linear model. We give a precise representation of ridge regression as a covariance matrix-dependent linear combination of the true parameter and the noise. We study the bias of K-fold cross-validation for choosing the regularization parameter, and propose a simple bias-correction. We analyze the accuracy of primal and dual sketching for ridge regression, showing they are surprisingly accurate. Our results are illustrated by simulations and by analyzing empirical data.

This paper presents a theoretical study of ridge regression, focusing on the practical problems of correcting for the bias of the cross-validation based estimate of the optimal regularisation parameter, and quantification of the asymptotic risk of sketching algorithms for ridge regression, both in the p / n -> gamma in (0, 1) regime (n = # data points, p = # dimensions). The authors derive most of their results exploiting their (AFAICT) new asymptotic characterisation of the ridge regression estimator which may be of independent interest. The whole study is complemented by a series of numerical experiments.
I am recommending this paper to be accepted for publication at ICLR. The paper is clearly written, makes several solid theoretical contributions, and recommends a simple and practical bias correction for CV-based estimates of the optimal ridge regulariser. While ICLR is biased towards deep learning focused publications, the work of Belkin, Hsu, Ma, Bartlett, Hastie, Montanari, Rakhlin, Liang and others (apologies to everyone whose name was omitted---it is for the sole reason of brevity) has recently shown that we can learn non-negligible amount about neural networks from study of linear models.
Comments:
- Most of the examples in the paper focus on the regime gamma < 1. Would you expect the observed behaviours to be significantly different when gamma > 1? I am asking specifically because of [1] which has found that the bias of the risk estimate obtained via cross-validation is often most extreme when p >> n, which makes me wonder about what would an experiment like those in fig.2 look like in the p >> n regime?
- I was somewhat confused when I first read the statement of thm.2.1. In particular, the definition of asymptotic equivalence requires (roughly speaking) that any series of projections of the difference between the two random vectors converges to zero a.s. However, within the theorem, you introduce Z without much explanation which confused me because not every Z ~ standard normal would be asymptotically equivalent. I needed to look at the proof to understand how Z is “coupled” with hat(beta), which I think should not be necessary. If possible, I would either say that there exists a (series of) Z (all standard normal) s.t. the asymptotic equivalence holds, or add some other clarification (possibly in the form of a footnote).
- When you are citing a book, please consider citing exact pages or at least chapters/sections (e.g., when citing the exact shortcut from Hastie et al. (2019)).
- In the definition of asymptotic equivalence (starting at the bottom of p.2), did you mean to assume limsup ||w|| < \infty a.s. (or is limsup not needed here)?
References:
[1] Tibshirani, R. J., & Tibshirani, R. (2009). A bias correction for the minimum error rate in cross-validation. The Annals of Applied Statistics, 822-829.


Image segmentation aims at grouping pixels that belong to the same object or region. At the heart of image segmentation lies the problem of determining whether a pixel is inside or outside a region, which we denote as the "insideness" problem. Many Deep Neural Networks (DNNs) variants excel in segmentation benchmarks, but regarding insideness, they have not been well visualized or understood: What representations do DNNs use to address the long-range relationships of insideness? How do architectural choices affect the learning of these representations? In this paper, we take the reductionist approach by analyzing DNNs solving the insideness problem in isolation, ie. determining the inside of closed (Jordan) curves. We demonstrate analytically that state-of-the-art feed-forward and recurrent architectures can implement solutions of the insideness problem for any given curve. Yet, only recurrent networks could learn these general solutions when the training enforced a specific "routine" capable of breaking down the long-range relationships. Our results highlights the need for new training strategies that decompose the learning into appropriate stages, and that lead to the general class of solutions necessary for DNNs to understand insideness.

This paper investigates the problem of modeling insideness using neural networks. To this end, the authors carefully designed both feedforward and recurrent neural networks, which are, in principle, able to learn the insideness in its global optima. For evaluation, these methods are trained to predict the insideness in synthetically generated Jordan curves and tested under various settings such as generalization to the different configuration of curves or even different types of curves. The experiment results showed that the tested models are able to learn insideness, but it is not generalizable due to the severe overfitting. Authors also demonstrated that injecting step-wise supervision in coloring routine in recurrent networks can help the model to learn generalizable insideness.
This paper presents an interesting problem of learning to predict insideness using neural networks, and experiments are well-executed. However, I believe that the paper requires more justifications and analyses to convince some claims and observations presented in the paper. More detailed comments are described below.
1. Regarding the usefulness of learning insideness to improve segmentation
The authors motivated the importance of learning insideness in terms of improving segmentation (e.g., instance-wise segmentation). However, I believe that this claim is highly arguable and needs clear evidence to be convincing. Although I appreciate the experiments in the supplementary file showing that some off-the-shelf segmentation models fail to predict insideness, I believe that these two are very different tasks (one is filling the region inside the closed curve and the other is predicting the labels given the object texture and prior knowledge on shapes; please also note that segmentation masks also can be in very complex shapes, where the prior on insideness may not be helpful). It is still weak to support the claim that learning to predict insideness is useful to improve segmentation.
2. More analyses for experiment results
In the experiment, the authors concluded that both feedforward and recurrent neural networks are not generalized to predict insideness in fairly different types of curves. However, it is hard to find further insights in the experiments, such as what makes it hard to generalize this fairly simple task. Improving generalization using step-wise supervision in a recurrent neural network is interesting but not surprising since we simply force it to learn the procedure of predicting insideness using additional supervision.
To summarize, although the problem and some experiment results presented in the paper are interesting, I feel that the paper lacks justifications on the importance of the problem and insights/discussions of the results.


Artificial neural networks have revolutionized many areas of computer science in recent years, providing solutions to a number of previously unsolved problems. On the other hand, for many problems, classic algorithms exist, which typically exceed the accuracy and stability of neural networks. To combine these two concepts, we present a new kind of neural networks-algorithmic neural networks (AlgoNets). These networks integrate smooth versions of classic algorithms into the topology of neural networks. A forward AlgoNet includes algorithmic layers into existing architectures to enhance performance and explainability while a backward AlgoNet enables solving inverse problems without or with only weak supervision. In addition, we present the algonet package, a PyTorch based library that includes, inter alia, a smoothly evaluated programming language, a smooth 3D mesh renderer, and smooth sorting algorithms.

The paper conceptualizes a neural network architecture that integrates smoothed versions of traditional algorithms into the network topology. The AlgoNet concept employs smoothed versions of algorithms implemented in the WHILE language, with options for different levels of differentiability. The paper outlines a forward version of AlgoNet based on traditional, skip, and residual connections, and a backward version for solving inverse problems in a reconstructive, autoencoder-like fashion. As smoothing introduces a form of domain shift, the paper the reconstructive adversarial network that that employs "domain translators" as well as a discriminator that are trained in an adversarial fashion. The paper concludes by describing versions of AlgoNet for various different algorithms.
The general idea of being able to better control the behavior of neural networks by better leveraging known structure (e.g., algorithms) is appealing. However, the paper does not go beyond conceptualizing how this might be done in a hand wavy manner. It is difficult to see what if anything can be learned from the paper, let alone what practical utility it has, which is important given that the paper claims to propose a new neural network architecture.
The paper would benefit from a discussion of empirical results in the main text, with baseline comparisons. With the exception of a single figure, the details are relegated to the appendices.
The related work discussion is surprisingly short, given the attention that has been paid to designing/optimizing different network topologies. The work that is discussed is very narrow in scope (see below).
The  paper devotes too much discussion to the challenges introduced by non-differentiable layers and the advantages of increasing degrees of differentiability (this is about half of the intro and most of the related work).
Despite the lack of experimental demonstrations, the paper is one page over the suggested 8 page limit.


As distributed approaches to natural language semantics have developed and diversified, embedders for linguistic units larger than words (e.g., sentences) have come to play an increasingly important role. To date, such embedders have been evaluated using benchmark tasks (e.g., GLUE) and linguistic probes. We propose a comparative approach, nearest neighbor overlap (N2O), that quantifies similarity between embedders in a task-agnostic manner. N2O requires only a collection of examples and is simple to understand: two embedders are more similar if, for the same set of inputs, there is greater overlap between the inputs' nearest neighbors. We use N2O to compare 21 sentence embedders and show the effects of different design choices and architectures.

The paper proposes N2O, a tool for probing the similarity among sentence embedders. Given two sentence embedders, N2O measures the amount of overlap of the k-nearest neighbor sets reported by the two embedders, averaged over a sample of probing queries. Cosine similarity is used as the similarity metric. The paper computes all-pair N2O scores for common sentence embedders and analyzes the results.
Overall, while the idea of probing similarity between embedders is interesting, the paper has the following weaknesses:
- The use of neighbor overlap to compare embedders has precedence in the context of word embeddings [https://www.aclweb.org/anthology/C16-1262/ | https://hal.archives-ouvertes.fr/hal-01806468/ | https://www.aclweb.org/anthology/Q18-1008/]. The methods in these works are mostly identical to N2O with some minor variations (e.g., using Jaccard distance).
- Using an existing method is justified if it provides new insights for the new setting. However, the results do not offer a lot of new insights. The fact that static embeddings, ELMo, and BERT give different neighbors is unsurprising. (The paper even admitted this.) Moreover, since sentence embedders are usually used as features when fine-tuned on downstream tasks, the importance of observations based on the pre-fine-tuned models is unclear. Contrast this with the work on word embedding similarity. Since some research areas (e.g., social science) directly use the similarity of word embeddings to conclude findings, the insights of how different word embeddings behave are more directly applicable.
- The paper does address some design choices, such as the number of neighbors and the number of probing queries, showing that different choices have little effects on the scores. However, the use of cosine similarity is problematic. As noted in the paper, some embedders such as ELMo were not trained on similarity objectives. BERT does have the next-sentence task, but the sparse attention-based architecture does not necessarily push similar sentences to have low cosine similarity.
Additional comments and questions:
- Instead of using the amount of overlap for a fixed k, it would be nice to have a metric that captures the whole distribution. For instance, maybe two embedders disagree on the first 20 neighbors, but end up retrieving the same set when considering 50 neighbors. The current overlap-based metric cannot capture such a phenomenon.
- Page 2: The method is technically not task-agnostic --- the task is sentence similarity with respect to a specific corpus.
- The paper tests variants of the same embedders by training on different corpora, with the conclusion that mismatched corpora give lower N2O scores. What is the N2O between two embedders of the same type trained on the same corpora but with different seeds?


Modern deep learning methods provide effective means to learn good representations. However, is a good representation itself sufficient for efficient reinforcement learning? This question is largely unexplored, and the extant body of literature mainly focuses on conditions which permit efficient reinforcement learning with little understanding of what are necessary conditions for efficient reinforcement learning. This work provides strong negative results for reinforcement learning methods with function approximation for which a good representation (feature extractor) is known to the agent, focusing on natural representational conditions relevant to value-based learning and policy-based learning. For value-based learning, we show that even if the agent has a highly accurate linear representation, the agent still needs to sample an exponential number trajectories in order to find a near-optimal policy. For policy-based learning, we show even if the agent's linear representation is capable of perfectly predicting the optimal action at any state, the agent still needs to sample an exponential number of trajectories in order to find a near-optimal policy. These lower bounds highlight the fact that having a good (value-based or policybased) representation in and of itself is insufficient for efficient reinforcement learning and that additional assumptions are needed. In particular, these results provide new insights into why the analysis of existing provably efficient reinforcement learning methods make assumptions which are partly model-based in nature. Furthermore, our lower bounds also imply exponential separations on the sample complexity between 1) value-based learning with perfect representation and valuebased learning with a good-but-not-perfect representation, 2) value-based learning and policy-based learning, 3) policy-based learning and supervised learning and 4) reinforcement learning and imitation learning.

This paper's contribution is a sample complexity lower bound for linear value-based learning and policy-based learning methods. The bound being exponential in the planning horizon is bad news, and has some implications with respect to further analysing sample complexity in RL.
The gist of this paper is that one can craft a hard MDP which requires visiting every state at least once, and that since this MDP's state space is exponential in the MDP's horizon, then there exists a set of MDPs which require an exponential (in the horizon) number of trajectories to be solved. As a consequence, further analysis of sample complexity in RL may need some much stronger assumptions.
The writing of the paper is good, I was able to understand everything (I think). As far as I can tell, this is novel work. Unfortunately I am currently unable to see why this contribution is valuable. I have set my score to weak reject but I am very open to having my mind changed, as I feel I may have missed some critical element.
I have two criticisms:
A- I don't understand why this bound is significantly different than previous bounds.
B- I don't understand why this is bad news for representation learning, nor how this failure mode of linear features translates to the "deep" case.
In the same spirit, I find rather odd the way the paper is introduced. Discussions of representations usually involve some discussion of generalization, but that's not what this paper is about. Deep neural networks/representation learning are only useful if there is an opportunity for generalization.
With respect to A, I am either grossly misunderstanding past bounds and/or your bounds, or something is wrong with the way complexities are compared:
- In Wen & Van Roy, the "polynomial" sample complexity is in the number of states, it is related to |S|x|A|xH^2 (Theorem 3 of Wen & Van Roy)
- In this paper, Theorem 4.1 states that the sample complexity is exponential because it is of the form 2^H. One *critical* assumption for this bound is precisely that |S| >= 2^H. Thus the bound that you propose is still polynomial in |S|.
I am thus puzzled, how is this bound significantly different?
With respect to B, I don't see how this bound has much to do with good representations, or even representations at all.
In Lemma A.1, you essentially craft a set of features that, being mutually orthogonal, are in some sense "mutually linearly separable", making learning the mapping from those features to a value function "trivial" once data is obtained. This is barely different from saying that you assume there is a magical learner that learns in O(1) given the data, because in either case, you need to visit _every_ of the 2^H state in order to solve the MDP, because by construction of your problem, there is _no hope_ of generalization*. Since learning features or creating "good" features has everything to do with generalization (otherwise we'd just to tabular), I don't see how this bound is relevant to representations. (We already have Wolpert's no free lunch theorem to tell us that there are always some problems that ML just can't be general enough to solve efficiently. What is more interesting is understanding how we can efficiently learn where there _is_ structure to a problem.)
* There is no hope of generalization, unless something about the observation space (which is left undiscussed in the paper) contains *information* about the agent being to the unique path to the reward. In such a case, I can see a probabilistic argument being made where in the worst case the agent needs to visit all 2^H states, but in the average case, the agent may learn to ignore paths where it can generalize that there is no reward. This is not entirely unreasonable, think of e.g. AlphaGo, where very few states end in victory, where there is an exponential number of states in the horizon, yet learning is totally reasonable because of structure in observation. This is where I don't agree with a statement like: "Since the class of linear functions is a strict subset of many more complicated function classes, including neural networks in particular, our negative results imply lower bounds for these more complex function classes as well."


The text generated by neural language models is not as good as the real text. This means that their distributions are different. Generative Adversarial Nets (GAN) are used to alleviate it. However, some researchers argue that GAN variants do not work at all. When both sample quality (such as Bleu) and sample diversity (such as self-Bleu) are taken into account, the GAN variants even are worse than a well-adjusted language model. But, Bleu and self-Bleu can not precisely measure this distributional discrepancy. In fact, how to measure the distributional discrepancy between real text and generated text is still an open problem. In this paper, we theoretically propose two metric functions to measure the distributional difference between real text and generated text. Besides that, a method is put forward to estimate them. First, we evaluate language model with these two functions and find the difference is huge. Then, we try several methods to use the detected discrepancy signal to improve the generator. However the difference becomes even bigger than before. Experimenting on two existing language GANs, the distributional discrepancy between real text and generated text increases with more adversarial learning rounds. It demonstrates both of these language GANs fail.

This paper proposes an estimator to quantify the difference in distributions between real and generated text based on a classifier that discriminates between real vs generated text.  The methodology is however not particularly well motivated and the experiments do not convince me that this proposed measure is superior to other reasonable choices.  Overall, the writing also contains many grammatical errors and confusing at places.
Major Comments:
- There are tons of other existing measures of distributional discrepancy that could be applied to this same problem.  Some would be classical approaches (eg. Kullback-Leibler or other f-divergence based on estimated densities, Maximum Mean Discrepancy based on a specific text kernel, etc) while others would be highly related to this work through their use of a classifier.  Here's just a few examples:
i) Lopez-Paz & Oquab (2018). "Revisiting Classifier Two-Sample Tests
": https://arxiv.org/abs/1610.06545
ii) the Wasserstein critic in Wasserstein-GAN
iii) Sugiyama et al (2012). "Density Ratio Estimation in Machine Learning"
Given all these existing methods (I am sure there are many more), it is unclear to me why the estimator proposed in this paper should be better. The authors need to clarify this both intuitively and empirically via comparison experiments (theoretical comparisons would be nice to see as well).
- The authors are proposing a measure of discrepancy, which is essentially useful as a two-sample statistical test.  As such, the authors should demonstrate a power analysis of their test to detect differences between real vs generated text and show this new test is better than tests based on existing discrepancy measures.
- The authors claim training a generator to minimize their proposed divergence is superior to a standard language GAN. However, the method to achieve this is quite convoluted, and straightforward generator training to minimize D_phi does not appear to work (the authors do not say why either).
Minor Comments:
- x needs to be defined before equation (1).
- It is mathematically incorrect to talk about probability density functions when dealing with discrete text. Rather these should be referred to as probability mass functions, likelihoods, or distributions (not "distributional function" either).


Graph Neural Network (GNN) is a popular architecture for the analysis of chemical molecules, and it has numerous applications in material and medicinal science. Current lines of GNNs developed for molecular analysis, however, do not fit well on the training set, and their performance does not scale well with the complexity of the network. In this paper, we propose an auxiliary module to be attached to a GNN that can boost the representation power of the model without hindering the original GNN architecture. Our auxiliary module can improve the representation power and the generalization ability of a wide variety of GNNs, including those that are used commonly in biochemical applications.

= Summary
A Graph Neural Network extension integrating a global supernode explicitly into the message passing process. Concretely, message passing along the graph edges is alternated with message passing to/from a fresh super-node. Experiments show that this improves results of a number of common GNN architectures on four datasets.
= Strong/Weak Points
+ Simple but useful extension of the existing super-node idea
+ Experiments on a number of datasets and baseline GNN architectures, providing ample experimental evidence of the usefulness of the method.
- Writing is overcomplicated and uses a lot of jargon ("transmitter unit", "warp gate", "intermodule hyperspace"). I found the text entirely impenetrable and instead simply focused on Fig. 3 + the actual equations.
= Recommendation
This is a nice contribution of minor novelty, with empirical evidence of its usefulness. I believe the paper should be accepted to a large conference such as ICLR.
= Minor Comments
- Fig. 3: Inconsistent "intra-module" (top) vs. "intra module" (bottom)
- Concurrent work in https://openreview.net/forum?id=B1lnbRNtwr discusses a "sandwich" model which alternates graph message passing with (essentially) a Transformer layer applied to all nodes. This idea seems related (in that it alternates local and global information exchange).


Graph convolutional neural networks have demonstrated promising performance in attributed graph learning, thanks to the use of graph convolution that effectively combines graph structures and node features for learning node representations. However, one intrinsic limitation of the commonly adopted 1-D graph convolution is that it only exploits graph connectivity for feature smoothing, which may lead to inferior performance on sparse and noisy real-world attributed networks. To address this problem, we propose to explore relational information among node attributes to complement node relations for representation learning. In particular, we propose to use 2-D graph convolution to jointly model the two kinds of relations and develop a computationally efficient dimensionwise separable 2-D graph convolution (DSGC). Theoretically, we show that DSGC can reduce intra-class variance of node features on both the node dimension and the attribute dimension to facilitate learning. Empirically, we demonstrate that by incorporating attribute relations, DSGC achieves significant performance gain over state-of-the-art methods on node classification and clustering on several real-world attributed networks.

The authors propose a method that incorporates two types of graphs into a graph convolutional networks:
(1) the given graph, which the authors refer to as node affinity graph, and
(2) the graph that models an affinity between node attribute values.
The main contribution of their work is a type of graph convolution that combines convolutions operating on these two graphs.
The paper is densely written and provides several mathematical derivations that are unnecessary to convey the proposed method. Personally, I don't see any benefits in having sections 3.1-3.4 in the main paper. The method actually proposed and evaluated in the paper is described in section 3.5. Sections 3.1-3.4 could be moved to an appendix. They confuse the reader more than they help. (They demonstrate knowledge of graph signal processing on the parts of the authors but little more.)
Trying to provide some theoretical analysis of the proposed method (and standard graph convolutions) by showing that the intra-class variance is reduced is laudable. The theorems, however, only hold under strong assumptions and could, in my opinion, also be moved to an appendix. In the end, they don't have any bearing on the performance of the methods using real-world datasets. Adding some experiments to analyse to what extent the assumptions made by the theorems are met in the given datasets would be an interesting addition to the paper.
The authors discuss related work sufficiently with one exception: there has been recent work on learning the structure of graph neural networks. See for example [1]. The structure is derived/bootstrapped using node attribute similarities and it is shown that augmenting the graph with these new edges improves accuracy significantly. I would like to point the authors specifically to Figure 2 and Table 1 in said paper, where the authors show that adding edges (e.g., based on some node attribute affinity before or during training) is beneficial and improves accuracy. It would therefore be interesting to see how the authors proposed 2-D convolution would compare to a baseline where the edges based on attribute affinity are added to the original (node affinity) graph. It is a (somewhat simpler) alternative way to combine node affinity and node attribute graphs.
[1] https://arxiv.org/pdf/1903.11960.pdf
The empirical results are mixed. Due to the numerous different variations of DSGC for which experiments were conducted, the difference between DSGC and existing methods is probably not statistically significant (a bonferroni correction was not performed to counteract the multiple comparisons).
Overall this an interesting paper that introduces a way to incorporate node attribute affinity graphs. It is too densely written and could benefit from moving the theoretical parts to an appendix. They don't really add much to the core of the paper. Moreover, the authors do not consider approaches that also add edges to the graph (based, e.g., on attribute value similarity or during learning, see e.g. [1]) showing that that improves performance even when using a vanilla GCN. A comparison to a baseline that simply adds edges based on attribute affinity to the graph and applied a vanilla GCN should be part of the evaluation. The empirical results are mixed and don't show a clear advantage of the proposed method.


Challenges in natural sciences can often be phrased as optimization problems. Machine learning techniques have recently been applied to solve such problems. One example in chemistry is the design of tailor-made organic materials and molecules, which requires efficient methods to explore the chemical space. We present a genetic algorithm (GA) that is enhanced with a neural network (DNN) based discriminator model to improve the diversity of generated molecules and at the same time steer the GA. We show that our algorithm outperforms other generative models in optimization tasks. We furthermore present a way to increase interpretability of genetic algorithms, which helped us to derive design principles.

The paper describes a model for the generation and optimization of molecules (de novo molecular design), which combines a genetic algorithm based on the SELFIES string representation and a trained discriminator.
The performance of the method is then shown on a small number of optimization tasks, and analyzed in detail.
Overall, this reviewer thinks this is a very interesting approach with a lot of potential, however, the experimental validation could be stronger.
Embedding in previous work:
Several key references should be added. In particular, this reviewer would suggest that the paper from the AZ group (https://arxiv.org/abs/1701.01329) is added,
 which first introduced the language model (LM)-based class of molecule generation models, and the RL paradigm of a molecule-generating agent, which takes actions in an environment to construct a molecule, receiving reward from an external scoring function.
Also, the previous paper by Liu et al (“Constrained Graph Variational Autoencoders for
Molecule Design”, NeurIPS 2018,  https://arxiv.org/pdf/1805.09076.pdf ) should be cited, in particular, because it highlights the competitiveness of the LM based baselines.
In Table 2, it would be great to include a reference to the DEFactor paper from Yoshua Bengio’s lab (https://arxiv.org/abs/1811.09766), which achieves comparable performance to the strongest baseline.
In addition, the authors should cite earlier work on using GAs for de novo design, which have a rich tradition in chemoinformatics since the 1990ies.
Experimental:
Re 4.1. In the unconstrained optimization task, several baselines are missing. In particular, a language model + RL baseline should be considered. This model has also shown strong performance (in terms of optimization and quality) in the state of the art benchmark set (Guacamol https://pubs.acs.org/doi/10.1021/acs.jcim.8b00839 ) -  the code is available on github
Can the authors explain why they believe the physchem only function ”log(P) - SA(m) - RingPenalty(m)” is a relevant scoring function for the tasks that are faced in real life drug and materials discovery? It would be great if the authors would consider using the Guacamol benchmarks which have been designed by domain experts to allow a more fine grained assessment on tasks relevant to practical drug and materials design, while still being tractable computationally. Also, the guacamol paper provides a benchmark to assess the quality of molecules, which is currently missing in the present work.
Re 4.2. And 4.3.
The long term experiment and the molecule class exploration analysis are great, and well presented. I like the analysis of how the different classes emerge. However, are you sure that the GA is not just exploited the deficiencies of the rdkit logP calculator (which itself is just a model) by creating the molecules with the long sulfur chains? Also, I am not sure that it is likely that one would be able to synthesize these molecules in the lab.
Re 4.4
The results on the constrained optimization are impressive, however, it is not really clear if the GA is not simply exploiting deficiencies in the scoring function again.
Overall, this reviewer believes the proposed model is very promising (in particular also because GA’s are cool), however, the experimental validation could be more rigorous, which can be addressed by considering the guacamol benchmarks. If this results can provided, this reviewer will raise their recommendation to 'accept' (regardless of outcome of the benchmark)
#####
After response:
Considering the authors' response, I adapt my recommendation to accept. I acknowledge the short time the authors have  to run additional experiments. nevertheless, I would suggest that the authors at least run more challenging benchmarks later and report the results e.g. as a blogpost or in a leaderboard.


Many real-world sequential decision-making problems can be formulated as optimal control with high-dimensional observations and unknown dynamics. A promising approach is to embed the high-dimensional observations into a lowerdimensional latent representation space, estimate the latent dynamics model, then utilize this model for control in the latent space. An important open question is how to learn a representation that is amenable to existing control algorithms? In this paper, we focus on learning representations for locally-linear control algorithms, such as iterative LQR (iLQR). By formulating and analyzing the representation learning problem from an optimal control perspective, we establish three underlying principles that the learned representation should comprise: 1) accurate prediction in the observation space, 2) consistency between latent and observation space dynamics, and 3) low curvature in the latent space transitions. These principles naturally correspond to a loss function that consists of three terms: prediction, consistency, and curvature (PCC). Crucially, to make PCC tractable, we derive an amortized variational bound for the PCC loss function. Extensive experiments on benchmark domains demonstrate that the new variational-PCC learning algorithm benefits from significantly more stable and reproducible training, and leads to superior control performance. Further ablation studies give support to the importance of all three PCC components for learning a good latent space for control.

This paper considers learning low-dimensional representations from high-dimensional observations for control purposes. The authors extend the E2C framework by introducing the new PCC-Loss function. This new loss function aims to reflect the prediction in the observation space, the consistency between latent and observation dynamics, and the low curvature in the latent dynamics. The low curvature term is used to bias the latent dynamics towards models that can be better approximated as locally linear models. The authors provide theory (error bounds) to justify their proposed PCC-Loss function. Then variational PCC is developed to make the algorithm tractable. The proposed method is evaluated in 5 different simulated tasks and compared with the original E2C method and the RCE method.
The paper is well-written.
Pros:
- The idea in this paper is quite original. The three principles used to formulate the loss function provide some new insights.
- The authors have proposed a theory to justify the use of their loss function. The technical quality of this part seems solid.
- Simulations have been used to show that the proposed PCC method outperforms E2C and RCE.
-The paper is well written.
Cons:
- The tasks in this paper are not that complicated. It is unclear whether the proposed method outperforms other model-based RL methods such as Solar and DSAE for practical robotic applications. More comparisons are needed.
- It is also not that clear why one wants to SOC3 to be close to SOC1 in the first place. It seems the true optimization problem should be posed on the space of the original state s. SOC1 is just a surrogate problem for the original problem.
- There seems to be a gap between the proposed theory  and the algorithm implementation. This makes the theory part less useful.
Overall, I think the idea in this paper is interesting. The authors have made a serious effort in coming up principles for model-based RL control. But at this moment it is not that convincing the proposed method will be the best model-based RL method for practical robotic applications. If the authors can address my comments, I will be willing to increase my score.
Minor Comments:
- It seems that for the task the authors have tested their method, it is not that difficult to directly estimate the state. Am I correct here? Can the authors make a comment on this? How to compare their approach and a more direct control approach using estimation of state s?
- I have never seen the curvature principle in any control papers. Any control reference on why this is a good principle? It seems that the linearization works well when the control inputs are around the reference points. Does the curvature really matter that much for ILQR to work?
- How to justify the Markovian assumption on x? Just by observation or there is a more principle way to test this assumption on the buffered images?
====================================================
Post-Rebuttal:
After reading the authors' response, I am changing my score to weak accept. Lemma 4 is nice. I have not seen anything similar to this in the controls literature before. The authors have addressed most of my concerns. I still have a few comments for preparing the final version of this paper.
1. I still don't see why SOC1 is the "original problem." Yes, it is assumed that the true state cannot be directly observed. But if the observations are Markov eventually, then some estimated version of the states can be obtained, right? I think treating SOC1 as the original problem is one possible way of doing things and clearly the authors have built a principled framework for doing things in this way. But treating SOC1 as the original problem seems not the only way of doing things. I hope the authors can clarify this and do not oversell the proposed approach.
2. I think it is still worth comparing SOLAR and PCC empirically. This will help the readers to choose algorithms when they need.
3. The comment on the verification of the Markov assumption is hand-waving.  The authors said "A simple test would be to see if a control algorithm with the Markovian assumption works well with our representation or not." Does this mean that the users will not be able to verify this assumption before using the proposed approach to obtain controllers? It will be helpful if the authors can explain this step for one specific example in details.


In this paper, we demonstrate that the parameters of Deep Neural Networks (DNNs) do not satisfy the i.i.d. prior assumption and activations being i.i.d. is not valid for all the hidden layers of DNNs, thus the Gaussian process cannot correctly model the distribution of the hidden layers. Alternatively, we propose a novel probabilistic representation for the hidden layers of DNNs in two aspects: (i) a hidden layer formulates a Gibbs distribution, of which the energy function is determined by the activations, and (ii) the statistical connection between two adjacent layers can be formulated as a product of experts model. Based on the proposed probabilistic representation, we further confirm the equivalence between the variational inference and the stochastic gradient descent, and prove the later is also equivalent to the energy minimization optimization. As a result, the entire architecture of DNNs can be viewed as a Bayesian network, in which the hidden layers close to the input formulate the prior distributions of the training dataset. In addition, we propose a new regularization approach to improve the generalization performance of DNNs by pre-training the hidden layers corresponding to the prior distributions. Simulation results validate the proposed theories.

Main contribution of the paper
- The paper argues that the base assumption, the i.i.d. of the activated elements (activations) in the hidden layers, the existing methods (lee.et.al 2018) hold is not convincible.
- Instead, the author proposes a new way to probabilistically model the hidden layers, activations, and layer/layer connections.
- Based on the probabilistic model, the paper proposes a new regularizer.
Methods
- The author argues that the activation is not iid by empirically showing that the trained MLP (in most cases) does not un-correlated.
- The author proposes a new probabilistic model for MLP, and CNN assuming the Gibbs distribution to each activation and also assuming the product of expert (poE) model to explain the layer/layer relationship.
- And according to their model, CNN will be explained by the MRF model.
- The author proposes a regularization term regarding layer/layer connection.
- They argue that the SGD training can be seen as a first-order approximation of the inference of the hidden activations in MLP.
Questions
- See the Concerns
Strongpoints
- The probabilistic explanation of the MLP and the CNN seems novel and was interesting to the reviewer
- The proposed explanation assumes a weaker condition compared to the existing methods.
Concerns
- The main concern is that the reviewer cannot fully convince that i.i.d. assumption is wrong.
Even though the trained MLP does not support the i.i.d. condition, one can suppose that the reason would be the typical training method (SGD), just finding the local minima in a deterministic way.
Maybe the proof in Appendix.G. supports the argument of the author, but the reviewer failed to clearly agree with the argument.
A clear explanation regarding the issue would be required.
- As far as the author understands, the paper proposes a probabilistic (Bayesian) model for explaining MLP, but it seems that they just used SGD for training the model.
In that case, the reviewer is little suspicious of the role of the proposed regularization in that the regularization comes from Bayesian formulation, but the model was trained in a deterministic way.
The reviewer wants to ask the author that
(1) is it possible to infer the model in a Bayesian manner such as sampling?
(2) Is there any justification for using SGD when conducting the experiments regarding the regularization? If it is related to Appendix.G, clearer explanation would be appreciated.
- As far as the reviewer understands, the regularization deals with the practical part of the paper. It would be better to see the effect of the regularization of widely used networks such as small-layered ResNet or others.
If the proposed formulation has other practical strongpoints, it would be nice to clarify them.
- The explanation using Gibbs distribution and PoE looks similar to RBM. The reviewer strongly wants a clear explanation of the difference and the strongpoints compared to RBM.
Conclusion
- The author proposed a new probabilistic explanation of the neural network, which seems novel and worth reporting.
- However, the reviewer failed to fully agree on some steps in the process of the paper.
Therefore, the reviewer temporary rates the paper as weak-reject, but this can be adjusted after seeing the answers of the author.

Inquiries
- See the concerns parts.


Universal approximation property of neural networks is one of the motivations to use these models in various real-world problems. However, this property is not the only characteristic that makes neural networks unique as there is a wide range of other approaches with similar property. Another characteristic which makes these models interesting is that they can be trained with the backpropagation algorithm which allows an efficient gradient computation and gives these universal approximators the ability to efficiently learn complex manifolds from a large amount of data in different domains. Despite their abundant use in practice, neural networks are still not well understood and a broad range of ongoing research is to study the interpretability of neural networks. On the other hand, topological data analysis (TDA) relies on strong theoretical framework of (algebraic) topology along with other mathematical tools for analyzing possibly complex datasets. In this work, we leverage a universal approximation theorem originating from algebraic topology to build a connection between TDA and common neural network training framework. We introduce the notion of automatic subdivisioning and devise a particular type of neural networks for regression tasks: Simplicial Complex Networks (SCNs). SCN's architecture is defined with a set of bias functions along with a particular policy during the forward pass which alternates the common architecture search framework in neural networks. We believe the view of SCNs can be used as a step towards building interpretable deep learning models. Finally, we verify its performance on a set of regression problems.

1. Summary of the paper
This paper introduces *simplicial complex networks*, a new class of
neural networks based on the idea of the subdivision of a simplicial
complex. Using the simplicial approximation, a classical theorem in
algebraic topology, the paper demonstrates that the network is capable
of learning a suitable function approximation that in turn can be used
for regression tasks.
2. Summary of the review
This paper brings an interesting perspective to the table, viz. the use
of concepts from algebraic topology to develop new neural network
architectures with improved approximation capabilities. I am
appreciative and excited of any work in this direction, yet I deem the
current version of this paper not acceptable for publication because of
the following issues:
- The paper suffers from a lack of clarity that makes assessing the
  results and the proposed architecture harder than it needs to be.
  In particular, numerous aspects such as scalability or the
  generalization capabilities are glossed over. Even though I am
  familiar with the underlying concepts and techniques, I found the
  exposition not always easy to follow.
- The experiments in the paper appear to be highly preliminary and need
  more refinement. I understand that it is tough to develop a new
  technique and 'compete'  against so many already existing ones, but
  nevertheless I would suggest to at least extend one of the experiments
  to a larger-scale comparison. Personally, I found the function
  approximation experiment to be very interesting, so to me, this would
  be a natural starting point for a more in-depth comparison.
Again, I want to stress that I think this paper has the potential to
become a strong contribution but for this to happen, it requires an
additional revision cycle. I will outline some suggestions below.
3. Clarity & technical correctness
It is my understanding that the paper is technically correct on all
levels. At times, imprecise language is used that might mislead or
confuse readers.
My overall recommendation would be to start the paper with a brief
intuition of _why_ BCS is required and how to make use of the
approximation theorem. After this, I would propose a high-level overview
of the new architecture and its individual components. Only _then_ would
I suggest describing each of these components in more detail.
Here are some detailed suggestions:
- The flow of the abstract could be improved; it is not clear why TDA
  methods are suitable here.
- Topological data analysis is not directly a 'geometric approach' for
  data analysis. It incorporates specific attributes of geometry by
  means of filtrations, for example, to 'escape' the coarseness of
  unweighted simplicial complexes
- TDA is also not _just_ restricted to point clouds
- I find the difference between 'deterministic' and 'statistical'
  methods incorrect in the introduction; the kernel-based approaches
  that were developed to compare persistence diagrams, for example, are
  _not_ the opposite of 'deterministic'. I would rather make
  a distinction between 'distance-based methods' that are hard to
  compute, and 'kernel-based methods' with improved scalability.
- When analysing point clouds, TDA does not use 'an inducing' distance;
  it can make use of the induced distance of a graph, for example, but
  typically, the distances are pre-defined and evaluated between all
  pairs of points.
- I find the statement that hidden units are conceptually well-defined
  only in SCNs to be highly problematic; I fully agree that
  interpretability of architectures is relevant, but another expression
  should be used here---FCN architectures (or in fact, all other
  architectures in deep learning) are perfectly well-defined on the
  conceptual level. Maybe the paper should be more clear about what it
  expects the 'role' of a hidden unit to be; in fact, this is also not
  discussed satisfactorily at present.
- Introduce the concept of a *simplex* briefly before discussing
  simplicial complexes.
- Figure 1 cannot be understood in the context in which it is placed
  currently; either move it back or create another figure that is more
  high-level and explains the concepts
- The requirement that all data points are within the standard
  probability simplex seems vacuous to me; this can always be achieved
  by scaling. Am I misunderstanding this? I would rewrite this and make
  it clear that this is _not_ a limiting assumption.
- Definition 2 is lacking formality; please add a brief description of
  the function itself
- Definition 3 is could use some more intuition; I understand that it
  is technically correct, but a brief illustration of the iterative
  nature of subdivision would be helpful.
- After introducing the simplicial approximation theorem, please give
  a brief connection on its relevance to the problem at hand.
- The brief introduction of the Vietoris--Rips complex is relatively
  hard to understand; please rephrase it. It is the complex in which
  a set of points is a simplex if and only if the pairwise distances
  of the points are less than or equal to the pre-defined threshold.
- Section 2.3 is highly confusing at first because it requires readers
  to understand Algorithm 1; please rephrase the introduction to this
  section to make it clear what the _purpose_ is. To my understanding,
  you want to use the BCS in order to obtain a situation as described
  by the approximation theorem---but at this point, the utility of the
  subdivision is not clear yet.
- What does it mean that weights are 'specified to the BCS'?
- When discussing the 'data-driven' approach, please add a motivation
  for this. I understand that such a subdivision is required in order
  to be computationally feasible, but this needs to be motivated.
- Lemma 1 appears to be a direct consequence of the definition of
  simplicial subdivision. I would suggest moving its proof to the
  appendix.
- Please give an improved explanation and intuition of the parameters of
  the subdivision and why they require updating. It took me a while to
  understand the relevance of the weights and what is actually updated
  in the algorithm.
- When talking about SCNs in Section 2.4, the salient points (learning
  a subdivision with trainable vertex function values etc.) need to be
  mentioned well before this section. When reading the paper for the
  first time, I started understanding the method better while reading
  the section, so I think that this intuition should become as early as
  possible.
- Please add intuition why _finding_ the position of a point is
  required; I understand that this is necessary in order to obtain
  weights and neighbours, from which to obtain a mapping, but at first,
  this 'location problem' struck me as artificial.
- When talking about the 'network policy', more details are needed. How
  relevant is this choice for the network?
- When discussing related work, please look at the follow-up
  publication from Hofer et al., which discuss more architectures and
  use cases:
      Hofer et al., Connectivity-Optimized Representation Learning via Persistent Homology
      ICML 2019
- 'Topological geometry' on p. 7 is an unfamiliar term to me. What is
  meant by this?
- What about the computational complexity? This is somewhat glossed over
  in the description of the BCS algorithm. Is this currently
  a bottleneck for the deploying the architecture?
- How easy would it be to extend this to _classification_ problems? The
  title suggests that the architecture is highly generic, yet the paper
  only discusses regression tasks. At the very least, this should be
  briefly discussed.
4. Experiments
- The experiments are lacking a brief explanation: what are they
  supposed to show or analyse? I find the function approximation
  example to be very interesting, but somewhat preliminary insofar
  they only compare *two* functions. Why not generate sequences or
  families of functions and make this into a large-scale comparison?
- I found the discussion of storage to be somewhat surprising; is this
  a property that the paper wants to stress for SCNs? If so, more
  experiments in that direction are required.
- The observations on bias function choice are not reflected in the
  experiments that are presented in the main text. I understand that it
  works for the toy examples, but a more in-depth comparison is required
  for this sort of experiment to be compelling.
- Is it possible to _tune_ the number of subdivisions or provide an
  upper bound? It would be interesting to see how such a bound might
  change the results---this is just a suggestion, but I feel that it
  might be interesting to discuss how much performance or approximation
  qualities are gained by _not_ restricting the BCS algorithm.
5. Minor style comments
The presentation quality of the paper could be improved. Here are some
suggestions.
- Please use `\citep` when citing works in the text; currently, citations are intermingled with sentences and make it harder to read the paper.
- 'Universal approximation property' --> 'The universal approximation property'
- 'with similar property' --> 'with similar properties'
- 'alternates the common architecture' --> 'changes the common architecture'
- the citation 'Bernardo et al.' should be fixed; it is missing information
- I suggest removing the sentences 'The rest of this paper is organized as follows'...
- 'Let f denotes a mapping' --> 'Let f denote a mapping'
- 'As theorem 2' --> 'As theorem 1'
- 'we can subdivision' --> 'we can subdivide'
- 'a same' --> 'the same'
- 'that how the position' --> 'how the position'
- 'used to fed' --> 'used to feed'
- 'the appendix F' --> 'appendix F'
- 'may outputs' --> 'may output'
- 'this phenomena' --> 'this phenomenon'
- 'bias function observed to be' -->' bias function was observed to be'
' 'can be continued' --> 'can be extended'