Human reasoning involves recognising common underlying principles across many examples by utilising variables. The by-products of such reasoning are invariants that capture patterns across examples such as "if someone went somewhere then they are there" without mentioning specific people or places. Humans learn what variables are and how to use them at a young age, and the question this paper addresses is whether machines can also learn and use variables solely from examples without requiring human pre-engineering. We propose Unification Networks that incorporate soft unification into neural networks to learn variables and by doing so lift examples into invariants that can then be used to solve a given task. We evaluate our approach on four datasets to demonstrate that learning invariants captures patterns in the data and can improve performance over baselines.

The authors propose a neural network approach to variable unification and
reasoning by example as a way to mimic the human ability to identify invariant
patterns in examples and then apply them more generally in practice.
This general idea of identifying invariates and mapping new instances to
them is well motivated by the authors, citing work in philosophy of mind,
cognitive science, and developmental psychology.
The authors go on to propose MLP, CNN and Memory Network models
of unification for sequence, grid, and story reasoning tasks respectively.
Experiments on the sequence and grid datasets demonstrate the data efficiency
of this approach. MLP and CNN models with unification achieve near perfect
performance in fewer iterations (an order of magnitude fewer in the MLP case!)
 than their non-unification enabled counter parts.
Unification enabled models also demonstrate high performance in a reduced
training set setting (using only 50 training examples).
While this is encouraging, these are very simple toy tasks.
I also am in doubt as to whether the representation of these problems
causes some issues. In the sequence task, one question the models are
trying to solve is what symbol is the head or tail of the sequence.
Modeling variables over the sequence of symbols here is, in a sense, the
wrong object of study. The position of the symbols would need to be
represented, e.g.
a b c d
1 4 3 1
where I've represented positions as a-d, and the learned invariant about
head questions would be:
X:a b c d
Y:1 4 3 1
As is, by mapping symbols and not positions to variables, one cannot,
at the variable level distinguish between the two 1s in the sequence above.
My guess is that in practice the bi-GRU model that produces embedding
features of the symbols in sequence is implicitly representing head/tail
positioning.
Similar arguments could be made about the grid example.
I don't find the experiments/analysis on the bAbI dataset very convincing.
For instance, in the example given in Figure 4b (reproduced below)
is shown as an example of
temporal reasoning, where a symbol Z is mapped to the
word morning (a symbol distinguishing a time), and the question asked is
where was Bill before school.
If logical reasoning is being used to solve this question, surely the
symbol 'before' must also be represented as a variable. Its possible that
the model is instead learning a trick about mutual exclusivity, i.e. that
Y:school is the only location symbol not mentioned in question but this
could fail as a general strategy.
this Z:morning X:bill went to the Y:school
yesterday X:bill journeyed to the A:park
where was X:bill before the Y:school
A:park
Figure 4b
It would make for a much more interesting paper if the authors took
examples such as these and formed counter-factuals to probe the way
the models are answering the questions. E.g., transforming the question
in 4b to "where was X:bill today" or "where was X:bill after school."
Because the authors use soft unification, interpretability is difficult
to assess. Interpretability is crucial here because to claim that unification and reasoning by logical induction
is being used to solve tasks, it becomes important to show how the neural networks
make their decisions. Given the instances of extra variables and one to many
mappings on the bAbI dataset it seems very likely that the models are not
solving many tasks
using unification as it would be possible to learn to use the symbols directly
to learn to answer. As such, I think these issues are not addressed in the
paper sufficiently to warrant acceptance.
Minor Notes
- In definition 1, the definition of Variable is a little confusing because there are two different senses of the word in use. I understand them to be (1) Variable (X) in the logical template that is intended to be learned and used in problem solving, and
(2) variable (x) in the neural network model that is a soft asignment of
the Variable to a default symbol s. It would be nice if this distinction could
be noted or made clearer.
- In the definition 2, in the phrase "is the invariant example such as a tokenized story" it might be worth stating that the tokens are the symbols in S.
- My understanding is that each unique symbol in the invariate is a potential
variable. Does this mean there are no co-referent symbols in the invariate?
Would be helpful to state whether babi contains co-referent expressions
and how these might affect the model.
- It might be interesting to see how model architecture affects variable
learning. For example, does a CNN result in more sensible variable
assignments  than the mlp on a flattened representation of the grid problem?
- What is the strongly supervised case? These are token level annotations I think (at least for babi) but it might be good to specify in more detail what
they  are.
- The figure and explanation of the UMN are not very clear. From the figure
is does not seem that the variables interact with the memory at all. More
space could be devoted to this section.
Possibly Relevant Related Work
Brenden Lake. Compositional generalization through metasequence-to-sequence learning. NeurIPS 2019.


Training recurrent neural networks (RNNs) on long sequence tasks is plagued with difficulties arising from the exponential explosion or vanishing of signals as they propagate forward or backward through the network. Many techniques have been proposed to ameliorate these issues, including various algorithmic and architectural modifications. Two of the most successful RNN architectures, the LSTM and the GRU, do exhibit modest improvements over vanilla RNN cells, but they still suffer from instabilities when trained on very long sequences. In this work, we develop a mean field theory of signal propagation in LSTMs and GRUs that enables us to calculate the time scales for signal propagation as well as the spectral properties of the state-to-state Jacobians. By optimizing these quantities in terms of the initialization hyperparameters, we derive a novel initialization scheme that eliminates or reduces training instabilities. We demonstrate the efficacy of our initialization scheme on multiple sequence tasks, on which it enables successful training while a standard initialization either fails completely or is orders of magnitude slower. We also observe a beneficial effect on generalization performance using this new initialization.

The aim of this paper is to suggest randomized initializations for the various weights of a recurrent neural network (GRUs and various LSTMs are covered), such that training these networks gets to a successful start, when the model is trained on long sequences. Instead of being heuristic, their approach follows first principles of analyzing signal propagation through time, using ideas from statistical thermodynamics (mean field approximations). Some experiments, on toy datasets, validate their approach.
I am quite intrigued by this paper. It is using interesting theory, shapes it to a practically highly relevant and difficult applied problem, and in the end comes up with a computable criterion of how to choose hyperparameters (means and variances of Gaussians to sample initial weights from). While the results in practice are still not too convincing, I am strongly in favour of giving this approach the benefit of doubt, as it could lead to practically very useful downstream work.
The main direction of improvement for this paper (given that experiments are what they are -- somewhat limited to toy situations right now) is to better explain the methodology to researchers not familiar with mean field methods. Most importantly, it is not explained in the main text how hyperparameters are really chosen in the end. Looking at Appendix E, I find some pretty basic choices, and no other alternatives considered. It is not explained why these choices satisfy the theory, why they'd be the only ones, etc. This creates a disconnect between the very nice (and seemingly useful) theory and its implications (they are not really well spelled out).
Here is what I understood (and I am not specifically an expert on stat mech). The authors assume that the dimension of latent states (N) grows large. They assume that weights are sampled independently, and identical distributed in groups k (different cell types, weight vs bias), and that inputs are correlated with each other in each dimension. Based on these assumption, they follow Gaussians statistics through a number of time steps. In the limit, one gets a deterministic dynamical system, and as t -> infty, this may converge to a fixed point. In a very nice argument (which they could explain better), they state that such rapid convergence is bad news, because then information cannot spread across long time scales, so one has to find hyperparameters for which the system behaves "critically". A second arguments tries to keep gradient sizes (under MF assumptions) of O(1), so neither -> 0, not -> infty, which is again some "critical" range. Under their assumption, these critical conditions can be computed depending on the hyperparameters.
Unfortunately, this is where the paper somewhat stops, it does not give specific methods for finding hyperpars that satisfy the criteria, at least not in the sense of characterising the whole space of such hyperpars (instead, in Appendix E, they just state some few settings that do). As a direction for future work, this would be very important. Another side question is whether for what the authors call "trainability", the only point that matters is whether for the initial weights, signals can spread and gradients are O(1). It is certainly necessary, I see that.
Detailed comments:
- Please fix Table 1, the expressions seem broken. What does "r2" mean in the GRU column?
- At least for me, (1a) to (1c) really was too short. At least in the Appendix, please do explain how this gives GRU and LSTM
- Please explain the untied weight assumption somewhere. s^t is a map of s^(t-1) and W_k, so how can W_k be independent
   of all s^t? What are you really assuming here?
- It took some repeated reading until I understood why the expressions in (2a) to (4b) do not depend on i, j, a, b (except
  whether a = b or a != b). Explain that properly
- The core of the whole approach seems to be first half of page 5. This seems like a very nice argument, but hard to understand. Try making it more crisp. I kind of get the rough idea why fast convergence over t would be bad, but would total divergence over t not also be bad?
- In (12a-c), do you mean "equal" or "approximately equal"?
- In 4.4: "This motivates the general form of the initializations used in the experiments": You have to make this more explicit. Why are your choices the only ones? Could there not be other choices satisfying (12a-c) approx, and be better?
- Value of Sigma_z = 1: This seems odd to me, then your covariances are degenerate (rank 1 instead of 2). Please explain
- Standard LSTM harder than GRU or peephole LSTM: Again, this sounds real interesting, but I did not get it from the explanation
- I did not understand Figure 2. How are Theta_0, Theta_1 chosen?
- As said above, the experiments are interesting, but somewhat artificial. Please do at least comment on real-world applications, and whether (and how) the ideas here would apply
- Discussion: "there is no clear principled way...": Well, but practitioners need something. I'd disagree, at least one could attempt to navigate this space by global optimization techniques...
ADDITIONAL COMMENT:
I tried to append the following as comment, but the (pretty broken) system would not let me, insisting that "reader is not valid" (???). Anyway, here it is. I hope I am allowed to add to my own review.
I've seen the argument in reviews that assumptions made by this paper about independencies between weights and inner states are wrong, and therefore conclusions are not valid.
First, such assumptions are indeed pretty common in such statistical mech analyses of learning methods. Second, you have to distinguish between weights after (random) initialization and after training. Of course, LSTM represents long term dependencies after training, but initialization is a different story.
If I was the AC for this paper, I'd ask somebody with at least some background in statistical mech to provide some additional opinions, as the reviewers (including myself) are not fully qualified.


People ask questions that are far richer, more informative, and more creative than current AI systems. We propose a neural program generation framework for modeling human question asking, which represents questions as formal programs and generates programs with an encoder-decoder based deep neural network. From extensive experiments using an information-search game, we show that our method can ask optimal questions in synthetic settings, and predict which questions humans are likely to ask in unconstrained settings. We also propose a novel grammar-based question generation framework trained with reinforcement learning, which is able to generate creative questions without supervised data.

[Summary]
This paper studies a very interesting problem - if machines can learn to ask the right questions to address or gain crucial information about tasks. I believe this is a very important yet under-explored problem. Specifically, this paper considers a setting where a neural network model is trained to ask questions by predicting a formal program. The network consists of a CNN encoder which encodes a partially observable state and a Transformer decoder that generates a program as a sequence of tokens. The experiments on a Battleship task show some promising results.
Significance: are the results significant? 3/5
Novelty: are the problems or approaches novel? 3/5
Evaluation: are claims well-supported by theoretical analysis or experimental results? 4/5
Clarity: is the paper well-organized and clearly written? 4.5/5
[Strengths]
*motivation*
The motivation for investigating machines' ability to generating questions to gain important information is convincing.
*novelty*
The idea of utilizing learning models for generating questions and modeling human-generated questions intuitive and convincing. This paper presents an effective way to implement this idea.
*technical contribution*
Leveraging a context-free grammar (CFG) and reinforcement learning (RL) for question generation seems effective especially when it comes to generating unique and novel questions.
*clarity*
The overall writing is clear and the authors utilize figures well to illustrate the ideas. Figure 1 illustrates the Battleship task and Figure 2 presents a clear overview of the proposed framework.
*ablation study*
Ablation studies are comprehensive. The proposed framework consists of multiple components. The provided ablation studies (Table 2 and Table 4) help analyze the effectiveness of each of them.
*experimental results*
- The presentations of the results are clear. The conclusions are fairly convincing:
- Experiment 1: learning to generate programmatic questions encourages learning rules for composition
- Experiment 2: it is possible to capture the human-generated questions distribution as a conditioned language model.
- Experiment 3: CFG+RL can produce more diverse and novel questions compared to supervised learning or RL.
[Weaknesses]
*novelty & contribution*
Overall, I do not find enough novelty from any aspects while the overall effort of this paper is appreciated.
- The problem is not entirely novel as [1-3] have explored asking questions with neural networks learning to generate programmatic questions.
- The proposed framework leverages a CFG and learns with RL, while the former (program synthesis with a CFG) has been studied in [4-5] and the latter (program synthesis using RL) has been discussed in [4, 6-7].
- Many setups considered in this paper have been explored in several neural program synthesis papers [8-12], which are neglected from this paper.
- The conclusions presented in the experiment section are more or less expected (i.e. other works have presented similar results on slightly different problems).
*oversell the motivation*
I believe the authors oversell the key motivation (i.e. to enable learning agents to ask questions) a little bit. To be more specific, it would be more interesting if an agent is allowed to ask a question, take actions based on the answer, and then ask the next question. However, this paper investigates the case where a significant amount of human-generated questions are given, which could limit the learning in my opinion. Also, it is not clear to me how the setup evaluated in this paper can be extended to a setting that allows an agent to iteratively ask questions.
*experiment setup*
Do the models learning using RL get the reward based on just the ground truth questions (a sequence of tokens) or the execution results?
*format*
The title & reference format looks wrong
*reference*
[1] Question Asking as Program Generation in NIPS 2017
[2] Neural-Symbolic VQA: Disentangling Reasoning from Vision and Language Understanding in NeurIPS 2018
[3] The Neuro-Symbolic Concept Learner: Interpreting Scenes, Words, and Sentences From Natural Supervision in ICLR 2019
[4] Leveraging grammar and reinforcement learning for neural program synthesis in ICLR 2018
[5] Learning a Meta-Solver for Syntax-Guided Program Synthesis in ICLR 2019
[6] Neural Scene De-rendering in CVPR 2017
[7] Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learning
[8] RobustFill: Neural Program Learning under Noisy I/O in ICML 2017
[9] Execution-Guided Neural Program Synthesis in ICLR 2019
[10] Neural Program Synthesis from Diverse Demonstration Videos
 in ICML 2018
[11] Learning to Describe Scenes with Programs in ICLR 2019
[12] Learning to Infer and Execute 3D Shape Programs in ICLR 2019


Unsupervised feature selection involves finding a small number of highly informative features, in the absence of a specific supervised learning task. Selecting a small number of features is an important problem in many scientific domains with high-dimensional observations. Here, we propose the restricted autoencoder (RAE) framework for selecting features that can accurately reconstruct the rest of the features. We justify our approach through a novel proof that the reconstruction ability of a set of features bounds its performance in downstream supervised learning tasks. Based on this theory, we present a learning algorithm for RAEs that iteratively eliminates features using learned per-feature corruption rates. We apply the RAE framework to two high-dimensional biological datasets-single cell RNA sequencing and microarray gene expression data, which pose important problems in cell biology and precision medicine-and demonstrate that RAEs outperform nine baseline methods, often by a large margin.

Authors of this paper propose the restricted autoencoder (RAE) framework for selecting features that can accurately reconstruct the rest of features. Authors justify the proposed method via the proof that the reconstruction ability of a set of features bounds its performance in downstream supervised learning tasks. The algorithm that iteratively eliminates features using learned per-feature corruption rates is proposed.
The fundamental of this paper is built on the argument that the optimal approach is to select a set of features that can accurately reconstruct all the remaining features for the settings where they will be used in downstream prediction tasks. Authors studied the performance losses of linear and nonlinear models by using the defined imputation losses.  Some concerns are listed as follows:
1. the theorem based on strong assumptions that all learned models are optimal. The applicability of the theoretical results to general prediction model is still questionable.
2. only the prediction problems of least square (linear or nonlinear) are studied. It is not equivalent to the downstream supervised learning tasks. It is just a special case study.
3. It is unclear how to get the conclusion from Theorem 1 that the linear imputation loss is equal to the sum of eigenvalues. Please clarify it in details.
The RFE-like algorithm is used to solve (7). However, the sensitivity measures used in Algorithm 1 seems to take the different optimization problems since additional regularization terms are added. This is different from RFE where a single SVM optimization problem is used and the ranking score is solely based on the learned SVM classifier. The discussion on the inconsistency of learning h_{\theta} and the sensitivity measures could be interesting.
In the experiments, authors did not mention the parameter settings of all compared methods. It is known that the unsupervised feature selection methods incorporate priors with usually various parameters. For fair comparisons, it is better to report the properly tuned results since these parameters are often data-dependent.


In this work, we develop quantitative results to the learnability of a two-layers Graph Convolutional Network (GCN). Instead of analyzing GCN under some classes of functions, our approach provides a quantitative gap between a two-layers GCN and a two-layers MLP model. Our analysis is based on the graph signal processing (GSP) approach, which can provide much more useful insights than the message-passing computational model. Interestingly, based on our analysis, we have been able to empirically demonstrate a few cases when GCN and other state-of-the-art models cannot learn even when true vertex features are extremely low-dimensional. To demonstrate our theoretical findings and propose a solution to the aforementioned adversarial cases, we build a proof of concept graph neural network model with stacked filters named Graph Filters Neural Network (gfNN).

Summary
Several theoretical analyses have been conducted on MPNN-type NNs. On the other hand, although many graph neural network models take the approach of graph filtering, there is little theoretical analysis on graph NNs that take the filtering approach. This paper analyzed the learnability of filtering type graph NNs.
First, the authors hypothesized that the data used practical tasks consisted of true low-frequency information and high-frequency noise (Assumption 2), and verified the correctness of the assumption.
Next, the authors quantitatively evaluated the difference between the case where we feed noisy data to a graph NN and the case where we feed noiseless data to an MLP. The authors showed that a two-layer GCN approximately behaves as a noise-filtering + a two-layered MLP (Theorem 8). Based on this observation, the authors proposed gfNN, which directly models the latter architecture. We can interpret gfNN as an SGC whose final linear transformation is replaced with an MLP.
Finally, the authors empirically showed that gfNN achieved the same level of accuracy as existing GNNs in the citation network tasks. Furthermore, the authors created a dataset that has meaningful information in the high-frequency area (BA-High). It was shown that the existing GNN, including gfNN, did not perform well, while a GNN designed to pass high frequency (gfNN-high) can predict accurately.
Decision
I judge that this paper contributes to deepning the understanding of graph NNs and is worth to be accepted based on the following three points.
First, it enabled us to understand what causes the oversmoothing phenomena. Several studies have shown that Laplacian-type graph convolution works as a low pass filter. However, most of them considered a linear setting and did not explain how the graph convolution behaves when a graph NN has non-linear activation functions. Compared to them, their result admits non-linearity.
Second, by showing that we can approximate a GCN by a noise filter followed by an MLP, this paper has made the relationship between a GCN and an SGC clearer.
Finally, the authors experimentally showed that existing graph NNs do not have predictive power when useful information is in high-frequency domains. It gives insights on what graph NNs can and cannot solve.
For these reasons, I think this paper has contributed to a deeper understanding of graph NNs and sufficiently significant.
Suggestions
- If I understand correctly, the title of Section 4 reflects the content of Lemma 5 in Section 5 rather than the content of Theorem 3 and (5). I recommend the authors to reconsider the titles of Sections 4 and 5.
- Could you add more explanations to the proof of Theorem 8 in Appendix A. Especially, I could not understand how the term $\tilde{O}(\epsilon^{1/4})$ is derived.
Questions
- I could not find implementations of graph NNs in the notebook to the code (I only found the ls result of dataset directories). Do you plan to release the experiment code?
- At the beginning of Section 6, the authors wrote that they set $k$ to $2$ (hence two-layered GNNs are in consideration). But Theorems 7 and 8 considered the optimal $k^\ast$ in Corollary 6. Which is correct?
- In Section 3, the authors claimed that the performance of MLPs is relatively more robust to the Gaussian noise in the low-frequency regime compared to the high-frequency regime. Certainly, the decrease in performance at $\sigma = 0.01$ is massive in the high-frequency setting for the CiteSeer dataset. However, it is hard to see this trend in Cora and Pubmed. Therefore, I think it is a little too aggressive to conclude that claim.


We propose the Fixed Grouping Layer (FGL); a novel feedforward layer designed to incorporate the inductive bias of structured smoothness into a deep learning model. FGL achieves this goal by connecting nodes across layers based on spatial similarity. The use of structured smoothness, as implemented by FGL, is motivated by applications to structured spatial data, which is, in turn, motivated by domain knowledge. The proposed model architecture outperforms conventional neural network architectures across a variety of simulated and real datasets with structured smoothness.

"Towards a Deep Network Architecture for Structured Smoothness" proposes a new layer, dubbed FGL, specifically focused on structured smoothness.
The paper clearly discusses the proposed layer, introduces the necessary formulation, and demonstrates the effectiveness of this layer in a suite of practical applications in fMRI analysis. The writing is clear, the method is well formulated, and performance compared to relevant and recent benchmarks is strong. Thus most of the questions remaining are about the finer details of the proposed layer and method.
What is the sensitivity of FGL to non-optimal group proposals for A? Given the authors reference Aydore et. al., do randomized (or perhaps meta-optimized) groupings perform poorly? Are there any potential ways to jointly-or-iteratively learn the groupings? Is Ward clustering the primary practical method for getting the groupings?
Can the authors discuss a bit the relationship and potential tradeoffs between "early" segmentation architectures (such as FGL), and those often favored in semantic segmentation ("late" segmentation)? Is there a particular reason why "late" segmentation is particularly a poor choice in brain imaging?
My primary concerns come in relation to other methods for structured smoothness such as CRF/MRF, and the trade-offs and drawbacks of imperfect or downright wrong A, and alternative methods for practically finding good A matrices (potentially with an eye to applications outside fMRI). Addressing some or all of these in a section or two of the text body would potentially raise my score.
Some small errors:
Optimg -> Opting Introduction
Guassian -> Gaussian section 3.2
Potential useful references:
CRF as RNN https://arxiv.org/abs/1502.03240
Efficient Piecewise Training https://www.cv-foundation.org/openaccess/content_cvpr_2016/html/Lin_Efficient_Piecewise_Training_CVPR_2016_paper.html
Deep Learning Markov Random Field for Semantic Segmentation https://arxiv.org/abs/1606.07230
Pixel Adaptive Convolutional Neural Networks http://openaccess.thecvf.com/content_CVPR_2019/html/Su_Pixel-Adaptive_Convolutional_Neural_Networks_CVPR_2019_paper.html


We present a novel generative model that combines state-of-the-art neural textto-speech (TTS) with semi-supervised probabilistic latent variable models. By providing partial supervision to some of the latent variables, we are able to force them to take on consistent and interpretable purposes, which previously hasn't been possible with purely unsupervised TTS models. We demonstrate that our model is able to reliably discover and control important but rarely labelled attributes of speech, such as affect and speaking rate, with as little as 30 minutes supervision. Even at such low supervision levels we do not observe a degradation of synthesis quality compared to a state-of-the-art baseline. Audio samples are available on the web 1 .

Overview:
This paper proposes to use semi-supervised learning to enforce interpretability on latent variables corresponding to properties like affect and speaking rate for text-to-speech synthesis. During training, only a few training items are annotated for these types of properties; for items where these labels are not given, the variables are marginalised out. TTS experiments are performed and the approach is evaluated objectively by training classifiers on top of the synthesised speech and subjectively in terms of mean opinion score.
I should note that, although I am a speech researchers, I am not a TTS expert, and my review can be weighed accordingly.
Strengths:
The proposed approach is interesting. I think it differs from standard semi-supervised training in that at test time we aren't explicitly interested in predicting labels from the semi-supervised labelled classes; rather, we feed in these labels as input to affect the generated model output. I agree that this is a principled way to impart interpretability on latent spaces which are obtained through unsupervised modelling aiming to disentangle properties like affect and speaking rate.
Weaknesses:
This work misses some essential baselines, specifically a baseline that only makes use of the (small number of) labelled instances. In the experiments, the best performance is achieved when gamma is set very high, which (I think) correspond to the purely supervised case (I might be wrong). Nevertheless, I think a model that uses only the small amount of labelled data (i.e. without semi-supervised learning incorporating unlabelled data) should also be considered.
As a minor weakness, the evaluation seems lacking in that human evaluations are only performed on the audio quality, not any of the target properties that are being changed. For affect specifically, it would be helpful to know whether the changes can be perceived by humans. As a second minor weakness, some aspects of the paper's presentation can be improved (see below).
Overall assessment:
The paper currently does not contain some very relevant baselines, and I therefore assign a "weak reject".
Questions, suggestions, typos, grammar and style:
- p. 1: "control high level attributes *of of* speech"
- p. 2: It would be more helpful to state the absolute amount of labelled data (since 1% is somewhat meaningless).
- p. 2: I am not a TTS expert, but I believe the last of your contributions have already been achieved in other work.
- Figure 2: It would be helpful if these figures are vectorised.
- p. 4: "*where* summation would again ..."
- Figure 4: Is there a reason for the gamma=1000 experiment, which performs best in (a), not to be included in (b) to (d)?
- Section 5: Table 1 is not references in the text.
- Section 5.1: "P(x|y,z_s,z_u)" -> "p(x|y,z_s,z_u)"
- In a number of places, I think the paper meant to cite [1] but instead cited the older Kingma & Welling (2013) paper; for instance before equation (6) (this additional loss did not appear in the original VAE paper).
References:
[1] https://arxiv.org/abs/1406.5298
Edit: Based on the author's response, I am changing my rating from a 'weak reject' to a 'weak accept'.


Regularization and normalization have become indispensable components in training deep neural networks, resulting in faster training and improved generalization performance. We propose the projected error function regularization loss (PER) that encourages activations to follow the standard normal distribution. PER randomly projects activations onto one-dimensional space and computes the regularization loss in the projected space. PER is similar to the Pseudo-Huber loss in the projected space, thus taking advantage of both L 1 and L 2 regularization losses. Besides, PER can capture the interaction between hidden units by projection vector drawn from a unit sphere. By doing so, PER minimizes the upper bound of the Wasserstein distance of order one between an empirical distribution of activations and the standard normal distribution. To the best of the authors' knowledge, this is the first work to regularize activations via distribution matching in the probability distribution space. We evaluate the proposed method on the image classification task and the word-level language modeling task.

This submission belongs to the general field of neural networks and sub-field of activation regularisation. In particular, this submission proposes a novel approach for activation regularisation whereby a distribution of activations within minibatch are regularised to have standard normal distribution. The approach, projected error function regularisation (PER), accomplishes that by minimising an upper-bound on 1-Wasserstein distance between empirical and standard normal distributions.
I think the idea described in this submission is interesting. Unfortunately, I have issues with 1) presentation, 2) experimental results, 3) English.
The PER is presented as an objective function that minimises an upperbound on 1-Wasserstein. I believe I have seen no evidence to the origin of PER other than it is the upper-bound on 1-Wasserstein. Therefore, I find it strange to see a presentation where first an objective function is introduced, then 1-Wasserstein is described, and after applying standard inequality you obtain an expression that is PER. The current presentation seems to indicate that before this derivation has been done no one new the connection between PER and the upper bound on 1-Wasserstein. I disagree and say that you obtained the upper bound on 1-Wasserstein and called it PER. For unknown reasons you decided to present first PER, then upper bound and finally claim connection. This is a mistake as it is not a connection but merely a consequence.
Simply looking up CIFAR-10 best numbers on any search engine I can find significantly better numbers. It is therefore unclear why did you decide to use sub-optimal configuration without commenting on that. The same applies to PTB and possibly to WikiText2.
There are numerous places where English is not adequate. For instance, "new perspective of concerning the target distribution".
Following the rebuttal stage where the authors have made significant changes to the manuscript I have decided to increase my assessment score.


Conventional out-of-distribution (OOD) detection schemes based on variational autoencoder or Random Network Distillation (RND) have been observed to assign lower uncertainty to the OOD than the target distribution. In this work, we discover that such conventional novelty detection schemes are also vulnerable to the blurred images. Based on the observation, we construct a novel RND-based OOD detector, SVD-RND, that utilizes blurred images during training. Our detector is simple, efficient at test time, and outperforms baseline OOD detectors in various domains. Further results show that SVD-RND learns better target distribution representation than the baseline RND algorithm. Finally, SVD-RND combined with geometric transform achieves near-perfect detection accuracy on the CelebA dataset.

UPDATE:
I acknowledge that I‘ve read the author responses as well as the other reviews.
I appreciate the clarifications and improvements made to the paper. I‘ve updated my score to 6 Weak Accept.
####################
This paper presents the idea to use blurred images as regularizing examples to improve out-of-distribution (OOD) detection performance based on Random Network Distillation (RND). The paper proposes to generate sets of such blurred images via Singular Value Decomposition (SVD) on the training images by pruning the lowest K non-zero singular values. The proposed method, SVD-RND, then extends the standard RND objective, which is to train a predictor network f to minimize the L2 loss to the output of some randomly initialized network over the (original) training data, with an additional regularization term that minimizes the L2 loss to the outputs of further multiple randomly initialized networks over the sets of blurred images. In OOD experiments on CIFAR-10, SVHN, TinyImageNet, LSUN, and CelebA, the proposed SVD-RND consistently outperforms baselines and recent competitors which are demonstrated to be vulnerable to blurred images.
I find it hard to make a definitive evaluation for this work at this point and would like to take the authors' responses into account for my final recommendation. The main idea of the paper to generate adversarial examples for training via blurring is rather simple and thus the novelty of this work is somewhat minor. I think the quality of the paper also suffers from many statements in the text that draw too general and too bold conclusions at this point in my mind. The presentation overall is rather unpolished (see comments below). However, I find the empirical results itself quite strong and convincing and think they would make a relevant and significant contribution to the community. I do have questions left open though that need clarification:
(i) I don’t see why different techniques for blurring (SVD, DCT, GB) should lead to such different results as the approach remains conceptually similar. Do you have a reason/intuition why SVD gives the best results? Might SVD just be the easiest to tune method?
(ii) What do you think is the key reason that SVD-RND also appears to generalize too non-blurry OOD samples? Could you elaborate more on the two reasons you give in the paper? (1. RND performance on samples orthogonal to the data; 2. Discrimination between data and its low-rank projection)
(iii) The generation and tuning of multiple sets of blurred images (how many samples per set?) may get quite extensive for large datasets. Could you be specific on the computational cost?
(iv) Might the deep generative models (e.g. GPND) fail to detect blurred images due to insufficient model capacity of the decoder which results in blurry reconstructions? Have you varied the network capacity or latent space dimensionality of such models?
(v) What is the idea behind choosing the log effective rank in such an equidistant manner as proposed?
####################
*Additional Feedback*
*Positive Highlights*
1. SVD-RND shows strong OOD detection performance in OOD experiments on a variety of image dataset combinations (CIFAR-10, SVHN, TinyImageNet, LSUN, and CelebA).
2. The related work includes major works from all the related lines of research (Deep anomaly detection; OOD detection using side information; Adversarial examples/training)
3. Useful hyperparameter selection criterion based on effective rank if no OOD validation data is available.
*Ideas for Improvement*
4. I think the tone of the paper would greatly benefit from not drawing too general conclusions and too bold implications. Keep statements precise and evidence-based. Declare hypotheses as such.
5. I would appreciate a plot showing samples before and after blurring in the appendix to see the most effective degree of blurring. Maybe also compare the different blurring baselines here to see differences.
6. Report std. devs. with your performance results to infer statistical significance.
7. Compare to the specific geometric transforms method as proposed in the paper [3] and not only using those transformations within your RND approach.
8. Add missing deep anomaly detection related work [6, 2, 5, 1].
9. Expand the sensitivity analysis in Figure 3 (left) over a greater range of K. Especially, I would like to also see K = 0 (unblurred, original images) as a sanity check which may only improve over RND due to ensembling over multiple randomly initialized networks.
10. Consider the one-vs-rest anomaly detection evaluation benchmarks from Ruff et al. [4] or Golan and El-Yaniv [3] to further infer the generalization performance of the proposed method.
11. I think the paper spends too much time on introducing previous work. Section 2 Related Work and Section 3 Background might be combined into one section.
*Minor comments*
12. In the abstract: “... VAE or RND are known to assign lower uncertainty to the OOD data than the target distribution.” is a bit strong. Rather “have been observed” etc. This is a working hypothesis in the community, but there is recent work (https://openreview.net/forum?id=Skg7VAEKDS) indicating (at least for VAEs) those are effects of poor model design.
13. In the abstract: “... efficient in test time ...” » “... efficient at test time ...”
14. In the abstract: “... in CelebA domain.” » “... on the CelebA dataset.” or just “... on CelebA.”
15. Section 1: “However, such models show underwhelming performance on detecting OOD, such as detecting SVHN from CIFAR-10. Specifically, generative models assign a higher likelihood to the OOD data than the training data.”. I think those are way too general conclusions at the moment. Rather something like “OOD detection performance of deep generative models has been called into question” and “have been observed to assign ...”.
16. Section 1: “Such results clearly support the degeneracy of deep OOD detection schemes”. Again, I find this way too bold of a statement at this point in time.
17. Section 2: “... a recently proposed paper ...” » “... a recent paper ...”
18. Section 2: “... outlier data independent of OOD data.”. What would outlier data not being out-of- distribution be?
19. Section 2: “Golan et al. (2018) design geometrically transformed data and regularized the classifier ...”. Not regularized. They trained a classifier on labels identified with these transformations.
20. Section 2: “..., resulting in OOD detection in each labeled data” » “..., resulting in OOD detection on labeled data”
21. A subsection title directly following a section title is bad style. A major section should be introduced with a few sentences on what this section is about.
22. Figure 2, right plot: These loss curves are rather strange... Increasing, then sharply decreasing again. Is there a drop in learning rate at epoch 80?
23. Many axis labels are too small and hard to read.
24. In Section 3.3.: “.. in Section 7.2.” » “.. in Section 4.2.”?
25. The definition of the log effective rank in Eq.~2 is weird. It's the entropy over the singular value distribution, i.e.~$H\left(\sigma_1 / \sum_j \sigma_j, \ldots, \sigma_N / \sum_j \sigma_j \right)$. Also, the parameters/notation involved are not introduced.
26. Make clear you apply SVD on single images. The paper alters formulations between data matrix and images...
27. Several instances where citet is used instead of citep.
28. In Table 1: Separate the target column from the three OOD columns more clearly. (e.g. vertical separator, center OOD, target in bold, etc.)
29. In Section 5.1: “area of the region under the ... curve” » “area under the ... curve”.
30. Figure 3: Better explain the plots. Are the three curves the respective target classes? What is the OOD set?
31. Figure 4: Maybe use subfigures with individual titles/captions.
32. Section 6: “For evidence, we fine-tune the classifier ...” » “For evidence, we fine-tune a classifier ...”
33. Plots and Figures are somewhat scattered and not referenced chronologically.
34. Introduce the effective rank at the point where it is used (Section 6.2). Somewhat unclear why to introduce this in Section 3 already.
35. Visually speaking, I find the RND examples in Figure 4 actually more anomalous than the top-anomalous SVD-RND samples (flashy colors, weird angles, borders, high contrast, ...)
####################
*References*
[1] R. Chalapathy and S. Chawla. Deep learning for anomaly detection: A survey. arXiv preprint arXiv:1901.03407, 2019.
[2] H. Choi, E. Jang, and A. A. Alemi. Waic, but why? generative ensembles for robust anomaly detection. arXiv preprint arXiv:1810.01392, 2018.
[3] I. Golan and R. El-Yaniv. Deep anomaly detection using geometric transformations. In NIPS, 2018.
[4] L. Ruff, R. A. Vandermeulen, N. Görnitz, L. Deecke, S. A. Siddiqui, A. Binder, E. Müller, and M. Kloft. Deep one-class classification. In International Conference on Machine Learning, pages 4393–4402, 2018.
[5] L. Ruff, R. A. Vandermeulen, N. Görnitz, A. Binder, E. Müller, K.-R. Müller, and M. Kloft. Deep semi-supervised anomaly detection. arXiv preprint arXiv:1906.02694, 2019.
[6] T. Schlegl, P. Seeböck, S. M. Waldstein, U. Schmidt-Erfurth, and G. Langs. Unsupervised anomaly detection with generative adversarial networks to guide marker discovery. In Proceedings International Conference on Information Processing in Medical Imaging, pages 146–157. Springer, 2017.


Many domains in machine learning have datasets with a large number of related but different tasks. Those domains are challenging because task-specific labels are often scarce and test examples can be distributionally different from examples seen during training. An effective solution to these challenges is to pre-train a model on related tasks where data is abundant, and then fine-tune it on a downstream task of interest. While pre-training has been effective for improving many language and vision domains, pre-training on graph datasets remains an open question. Here, we develop a strategy for pre-training Graph Neural Networks (GNNs). Crucial to the success of our strategy is to pre-train an expressive GNN at the level of individual nodes as well as entire graphs. We systematically study different pre-training strategies on multiple datasets and find that when ad-hoc strategies are applied, pre-trained GNNs often exhibit negative transfer and perform worse than non-pretrained GNNs on many downstream tasks. In contrast, our proposed strategy is effective and avoids negative transfer across downstream tasks, leading up to 11.7% absolute improvements in ROC-AUC over non-pre-trained models and achieving state of the art performance.

The paper proposes pre-training strategies (PT) for graph neural networks (GNN) from both node and graph levels. Two new large-scale pre-training datasets are created and extensive experiments are conducted to demonstrate the benefits of PT upon different GNN architectures. I am relative positive for this work. Detail review of different aspects and questions are as follows.
Novelty: As far as I know, this work is among the earliest works to think about GNN pre-training. The most similar paper at the same period is [Z Hu, arXiv:1905.13728]. I read both papers and found they have similar idea about PT although they have different designs. This paper leverages graph structure (e.g., context neighbors) and supervised labels/attributes (e.g., node attributes, graph labels) for PT. These strategies are not surprising for me and the novelty is incremental.
Experiment: The experiments are overall good. The authors created two new large scale pre-training graph datasets. Experimental results of different GNN architectures w/o different PT for different tasks are provided. Comparing to non-pretraining GNN, the improvements are significant for most cases.
Writing: The writing is good and easy to follow.
Questions: I would like to see more discussion about difference between this work and [Z Hu, arXiv:1905.13728]. Comparing to the other work, what are strengths of this work? In addition, have the authors compared the performances of their work and [Z Hu, arXiv:1905.13728] using the same data?


Applying resolution-preserving blocks is a common practice to maximize information preservation in video prediction, yet their high memory consumption greatly limits their application scenarios. We propose CrevNet, a Conditionally Reversible Network that uses reversible architectures to build a bijective two-way autoencoder and its complementary recurrent predictor. Our model enjoys the theoretically guaranteed property of no information loss during the feature extraction, much lower memory consumption and computational efficiency. The lightweight nature of our model enables us to incorporate 3D convolutions without concern of memory bottleneck, enhancing the model's ability to capture both short-term and long-term temporal dependencies. Our proposed approach achieves state-of-the-art results on Moving MNIST, Traffic4cast and KITTI datasets. We further demonstrate the transferability of our self-supervised learning method by exploiting its learnt features for object detection on KITTI. Our competitive results indicate the potential of using CrevNet as a generative pre-training strategy to guide downstream tasks.

This paper introduces Conditionally Reversible Network (CrevNet) that consists of the invertible autoencoder and a reversible predictive module (RPM). The two-way autoencoder is an invertible network that preserves the volume with no information loss while reducing memory consumption by using bijective downsampling. The RPM is a recurrent extension of two-way autoencoder that provides the reversiblity in temporal domain. The experiments on Moving MNIST, Traffic4cast, KITTI, and 2D object detection on KITTI show the improvement compare to other state-of-the-art models.
The paper is well-written and the contributions are clear. The experiments on diverse tasks and datasets are provided. Especially, the use of pretrained generative model for the object detection task shows the benefit of the model.
However, it is not clear the necessity of the proposed model and how the memory demand and computation cost are reduced.
To have a better support for the contributions, I have following suggestions.
- The benefit or necessity of each module is not clear. The comparisons with 1) only the invertible two-way autoencoder (without Section 2.2 and 2.3)", 2) only RPM (without Section 2.1 and/or 2.3), and 3)two-way autoencoder + reversible predictive model without 3D convolutions (without Section 2.3) are required.
- Model size/memory comparison is provided only for the MNIST dataset. Since the main benefit of the proposed model is low memory demand and computation cost, I suggest to provide the the comparisons of the memory demand with other models as well.
1) Other models on on Caltech Pedestrian dataset, Especially, the performance gap on Caltech Pedestrian dataset between the proposed method and CycleGAN is very small.
2) Some of the compared models on 2D KITTI detection are also lightweighted e.g., SqeeqeDet. The comparison of the model size/memery/test time with these models are necessary.
----
<After rebuttal>
Authors have mostly addressed my concerns, and the contributions are clearer. I adjusted my rating accordingly.


Although Reinforcement Learning (RL) algorithms have found tremendous success in simulated domains, they often cannot directly be applied to physical systems, especially in cases where there are hard constraints to satisfy (e.g. on safety or resources). In standard RL, the agent is incentivized to explore any behavior as long as it maximizes rewards, but in the real world undesired behavior can damage either the system or the agent in a way that breaks the learning process itself. In this work, we model the problem of learning with constraints as a Constrained Markov Decision Process, and provide a new on-policy formulation for solving it. A key contribution of our approach is to translate cumulative cost constraints into state-based constraints. Through this, we define a safe policy improvement method which maximizes returns while ensuring that the constraints are satisfied at every step. We provide theoretical guarantees under which the agent converges while ensuring safety over the course of training. We also highlight computational advantages of this approach. The effectiveness of our approach is demonstrated on safe navigation tasks and in safety-constrained versions of MuJoCo environments, with deep neural networks.

In this work, the authors studied solving the CMDP problem, in particular to model the constraint propagation using the idea of the backward value functions. Using the time-reversed probability property, they first established Bellman optimality conditions for the backward value function, which accounts for the tail expected return, conditioned on the final state.
Utilizing  the notion of backward value function, they further use that to model the constraint in the CMDPs, and proposed several safe policy iteration algorithms for both cases with discrete actions and with continuous actions. Experiments show that this method has better constraint guarantees than the state-of-the-art algorithms, such as the Lyapunov approach.
The idea of using backward value functions to model constraints in CMDP is interesting and so far I have not seen it in other places. The algorithms developed with this approach also appear to work reasonably well (especially in constraint guarantees) on benchmark domains such as gridworld and mujoco control tasks. The authors also provide several properties such as consistent feasibility and policy improvement, similar to the Lyapunov method,  and derive several versions of safe policy optimization algorithms. However, I found the intuition of using backward value function rather unclear. Unlike the Lyapunov function, which attempts to estimate a near-optimal "remaining constraint budget" w.r.t. CMDP constraint, what is the motivation behind using the backward value. Does the backward probability have any connections to occupation measures? Without similar motivations it is unclear how close the performance of the policy computed from this backward value approach with that of an optimal CMDP policy. Also unlike the Lyapunov approach in Chow'18, the consistent feasibility  property in this work appears to be more restricted as it is only limited to a slow policy update. Finally the experimental results look promising, but it would be great to also compare with other state-of-the-art results such as Lagrangian method, and conservative policy improvement (such as CPO).


Neural networks have a reputation for being better at solving statistical or approximate problems than at performing calculations or working with symbolic data. In this paper, we show that they can be surprisingly good at more elaborated tasks in mathematics, such as symbolic integration and solving differential equations. We propose a syntax for representing these mathematical problems, and methods for generating large datasets that can be used to train sequence-to-sequence models. We achieve results that outperform commercial Computer Algebra Systems such as Matlab or Mathematica.

It is rather interesting for a humble academic to review this paper. It already has a discussion, which I find very valuable, and many tweets and social media exposure and endorsements. It is onerous to review in this setting.
The paper makes a valuable contribution. The adversarial discussions in this website and the unhelpful hype can in this case be addressed to some extent by the authors. I will start with discussing this. Clearly, the title is too broad. This is not deep learning for symbolic mathematics. In no way does this paper address the essence of what is understood by "symbolic mathematics". What the authors address is mapping sequences of discrete quantities to other sequences of discrete quantities. The sequences in this paper correspond to function-integral i/o sequences, and 1st/2nd ODEs-function i/o sequences. I will leave it to the authors to come up with a more informative title, but something like deep learning or transformers for symbolic (1d) integration and simple ODEs with be far more accurate.
To hammer this point, note that Section 3 discusses removing "invalid" expressions: log(0) or sqrt(-2). However, it is the manipulation of infinity and imaginary numbers that could be considered to be one of the greatest achievements of symbolic mathematics over the last couple of hundred years. It is reasonable to expect neural nets to do this one day, because humans can, but this should come with results. It's too early to make the claim in the paper title.
Sentences such as "This suggest (sic) that some deeper understanding of mathematics has been achieved by the model." and "These results are surprising given the incapacity of neural models to perform simpler tasks ..." are speculative, potentially inaccurate and likely to increase hype. This hype is not needed.
Hype and over-claiming aside, I did enjoy reading this paper. The public commenters have already asked important questions about methodology and related work on neural programming that the authors have addressed in comments. I look forward to these being incorporated in the revised pdf.
A big part of the paper is about generating the datasets, and I therefore sympathise with the comment about requesting either a dataset release or the generating code. I see no obvious ethical concerns in this case, and the authors have already kindly offered to do this. This is a commendable and important service to our community and for this alone I would be inclined to vote for acceptance at ICLR.
The paper is clear and well written. However (i) it would be good to show several examples of input and output sequences (as done already in this website) and (ii) the Experiments section needs work. I'll expand on this next.
The seq2seq transformer with 8 heads, 6 layers and dimensionality 512 is a sensible choice. The authors should however explain why they expect this architecture to be able to map the sequences they adopt. That is, it is well known that a deep neural network is just a skeleton for an algorithm. By estimating the parameters, we are coming up with (fitting) the algorithm for the given datasets. What is the resulting algorithm? Why are 6 layers enough? Here some visualization would be helpful. See for example https://arxiv.org/pdf/1904.02679.pdf and https://arxiv.org/pdf/1906.04341.pdf For greater understanding of the problem, it may be useful to also try sparse transformers eg https://arxiv.org/abs/1805.08241
Beam search is a crucial component of the current solution. However, the authors simply cite Koehn 2004 for this. First, that work used language models to compute probabilities for beam search. I assume no language models are used in this case. What I'm getting to is that there are not enough details about the beam search in this paper. The authors should include pseudocode for the beam search and give a few examples. The paper (even better thesis) of Koehn is a good template for what should be included. This is important and should be explained.
For Mathematica, it would be useful to state it does other things and has not been optimized for the two tasks addressed in this paper only. It would also be useful, now that you have more time, to run it for a week or two and get answers not only for 30s but also for 60s. How often does it take longer than 30s? How do you score it then?
Please do include train and test curves. This would be helpful too. I will of course consider revising my score once the paper is updated.
Thanks for constructing this dataset and writing this paper. It is very interesting and promising.


It has been shown that instead of learning actual object features, deep networks tend to exploit non-robust (spurious) discriminative features that are shared between training and test sets. Therefore, while they achieve state of the art performance on such test sets, they achieve poor generalization on out of distribution (OOD) samples where the IID (independent, identical distribution) assumption breaks and the distribution of non-robust features shifts. Through theoretical and empirical analysis, we show that this happens because maximum likelihood training (without appropriate regularization) leads the model to depend on all the correlations (including spurious ones) present between inputs and targets in the dataset. We then show evidence that the information bottleneck (IB) principle can address this problem. To do so, we propose a regularization approach based on IB called Entropy Penalty, that reduces the model's dependence on spurious featuresfeatures corresponding to such spurious correlations. This allows deep networks trained with Entropy Penalty to generalize well even under distribution shift of spurious features. As a controlled test-bed for evaluating our claim, we train deep networks with Entropy Penalty on a colored MNIST (C-MNIST) dataset and show that it is able to generalize well on vanilla MNIST, MNIST-M and SVHN datasets in addition to an OOD version of C-MNIST itself. The baseline regularization methods we compare against fail to generalize on this test-bed.

The paper proposes to combat domain shift via the information bottleneck principle, where the representations are encouraged to learn features that have high mutual information with the labels while having low mutual information with the original data. The paper proposes a specific approach to enforcing this information bottleneck, called "entropy penalty", which tries to minimize the L2 distance between the representation at the first layer and the mean representation of the corresponding class.
The paper proceeds to demonstrate theoretically that some variant of the proposed entropy regularization works for synthetic, binary cases in the sense that it decreases the effect of features that are not highly correlated with the label.
The paper finally demonstrates superior performance on synthetic experiments and classifying digits in an out-of-distribution setting where the OOD distribution is changing foreground and background color.
My greatest concern of the proposed approach is the generality of the method. While the information bottleneck is a very general principle, the way it is implemented in this paper is very specific, which only works for the first hidden layer representation of an input x (even before non-linearity). Combining this with the experiment in which the domain shift is changing colors of in the images (which applies linear transforms to background and foreground pixels), it is hard for me to believe that the same procedure is not tailored for this specific type of domain shift (although this is still an interesting type of domain shift that is important in sim-to-real applications in robotics).
In fact, based on the information bottleneck principle, one could essentially learn that the color is also a feature that also has high MI with label and low MI with inputs, thus the top-left pixel would be indicative of the label. If I use this as the feature for entropy penalty (assuming that there is 1 color per class), this gives a R_{EP} loss of zero. I wonder if this is the reason why the experiments specifically asked for two foreground and two background colors for each class.
Based on there concerns, it might seem more helpful to consider other types of "unknown" domain shifts, such as CIFAR vs. CINIC, in which the method proposed might introduce fewer inductive biases than in the case of being invariant to color.
Minor comments:
	- It would seem like L1 regularization would achieve a similar effect to what is shown in the theory here? It would be interesting to see how sensitivity changes with some existing regularization methods, because they can be easily implemented.
	- Entropy is smaller for lower layers?
	- Minimizing entropy for higher layers at least minimizes an upper bound to the entropy of lower layers (if we consider discrete RVs).
	- Eq 6 is false for continuous RV and differential entropy. One could simply have h_2 = 2 * h_1 to get larger entropy. Eq 5 is true from data processing but the conditional entropy H(h_l | x) is not fixed.
	- It seems that from the discussion about entropy / MI of different layers does not justify the EP approach for the first layer. Even if the final layer retains all the information, you can still technically apply EP to remove redundant information? You can use leaky relu activation to make sure information is not lost in activation functions.
	- Typo for definition of \mu_k
	- The first layer is a convolution layer, so maybe it could be helpful to visualize the learned features with or without EP?


Hierarchical reinforcement learning has demonstrated significant success at solving difficult reinforcement learning (RL) tasks. Previous works have motivated the use of hierarchy by appealing to a number of intuitive benefits, including learning over temporally extended transitions, exploring over temporally extended periods, and training and exploring in a more semantically meaningful action space, among others. However, in fully observed, Markovian settings, it is not immediately clear why hierarchical RL should provide benefits over standard "shallow" RL architectures. In this work, we isolate and evaluate the claimed benefits of hierarchical RL on a suite of tasks encompassing locomotion, navigation, and manipulation. Surprisingly, we find that most of the observed benefits of hierarchy can be attributed to improved exploration, as opposed to easier policy learning or imposed hierarchical structures. Given this insight, we present exploration techniques inspired by hierarchy that achieve performance competitive with hierarchical RL while at the same time being much simpler to use and implement.

This is an interesting paper, as it tries to understand the role of hierarchical methods (such as Options, higher level controllers etc) in RL. The core contribution of the paper is understand and evaluate the claimed benefits often proposed by hierarchical methods, and finds that the core benefit in fact comes from exploration. The paper studies hierarchical methods to eventually draw the conclusion that HRL in fact leads to better exploration based behaviour in complex tasks.
While the conclusion suggesting HRL leading to better exploration seems interesting, I am not sure whether this is in fact too surprising? For example, the Options framework has been fundamentally proposed and shows benefits in terms of transfer learning and faster exploration. Options has been fundamentally argued to lead to faster exploration (for example, when the goal changes in the four rooms task). Therefore, isn't the conclusion that this paper draws already known? Maybe the paper studies HRL methods in different experimental settings, not considerered in existing HRL or Options based frameworks before, but I would imagine similar conclusions can be drawn if a pure HRL system is studied?
The paper mainly does an experimental ablation study for HRL systems, and draws the conclusion of faster exploration led by HRL methods. However, I am not sure whether this conclusion, while assummably true as argued by previous methods too, is convincing enough based on the proposed suite of tasks?
It might have been more useful if there were similar theoretical contributions or analysis were also made for a stronger claim for HRL methods. Without lack of theoretical analysis or proofs, and simply based on the ablation studies as in this paper, I am not sure whether this paper is yet ready for acceptance in a venue like ICLR.
In the tasks considered, also there is no termination condition being learnt, and in fact the HRL systems studied here terminates every c time steps (as in HIRO and most of other HRL papers). This might be a significant limitation though. If we learn the termination condition, as in Option-Critic, would we expect similar behaviour in performance and can attribute the benefits of HRL only to exploration?
There are a vast amount of approaches based on identifying bottleneck states, and often they have shown better performance in terms of transfer learning. While in these approaches, often the benefits are claimed to be in terms of both exploration and transfer learning, this paper seems to contradict that?
Experimentally, there are only few specific tasks considered, like the variations of Ant tasks like AntPush and AntMaze. I am not convinced from the set of experimental results that they are sufficient enough to draw the conclusion that HRL methods only excel due to better exploration.
Why is the conventional Four Rooms domain ignored? What if we take the Four Rooms domain, change the goal states - I would expect the paper to do such analysis on the range of HRL methods, and then propose a convincing argument.
I think overall the paper needs more work, before such conclusion can be drawn overall. It seems to me like a strong claim that the benefits of HRL is only due to exploration. The paper does not do enough experimental abltation studies to strengthen the claim, mainly lacking fundamental HRL task setups. It does not do a theoretical analysis studying HRL methods and their benefits either. Overall, I think with these contributions, if similar behaviour persists, then it would make a more convincing argument.
There are a lot of theoretical papers studying provably sufficient exploration methods. Perhaps such approaches can also be taken here to study HRL methods and whether they provably lead to faster exploration?
Experimentally, I would expect a more wide range of task setups and domains to be studied - since this is mainly a paper based on experimental studies and trying to draw conclusion for existing HRL methods without proposing new approaches. I think without these carefully studied experiments, the conclusions are over-claimed.


Identifying algorithms that flexibly and efficiently discover temporally-extended multi-phase plans is an essential next step for the advancement of robotics and model-based reinforcement learning. The core problem of long-range planning is finding an efficient way to search through the tree of possible action sequenceswhich, if left unchecked, grows exponentially with the length of the plan. Existing non-learned planning solutions from the Task and Motion Planning (TAMP) literature rely on the existence of logical descriptions for the effects and preconditions for actions. This constraint allows TAMP methods to efficiently reduce the tree search problem but limits their ability to generalize to unseen and complex physical environments. In contrast, deep reinforcement learning (DRL) methods use flexible neural-network-based function approximators to discover policies that generalize naturally to unseen circumstances. However, DRL methods have had trouble dealing with the very sparse reward landscapes inherent to long-range multi-step planning situations. Here, we propose the Curious Sample Planner (CSP), which fuses elements of TAMP and DRL by using a curiosity-guided sampling strategy to learn to efficiently explore the tree of action effects. We show that CSP can efficiently discover interesting and complex temporally-extended plans for solving a wide range of physically realistic 3D tasks. In contrast, standard DRL and random sampling methods often fail to solve these tasks at all or do so only with a huge and highly variable number of training samples. We explore the use of a variety of curiosity metrics with CSP and analyze the types of solutions that CSP discovers. Finally, we show that CSP supports task transfer so that the exploration policies learned during experience with one task can help improve efficiency on related tasks.

===== Summary =====
The paper introduces Curious Sample Planner (CSP) a long-horizon motion planning method that combines task and motion planning with deep reinforcement learning in order to solve simulated robotic tasks with sparse rewards. The CSP algorithm considers two different hierarchies of actions: primitive actions, which control the rotation of several joints in a robotic arm, and macro-actions corresponding to complex behaviours  such as moving from one position to another or linking two objects together. Macro-actions are selected using the actor-critic architecture PPO and then turned into primitive actions using geometric motion planning and inverse kinematics. Specifically, RRT-Connect is used for motion planning with the recursive Newton-Euler algorithm for inverse kinematics on a perfect model of the environment to determine the specific sequence of primitive actions necessary to execute the macro-action. As CSP is interacting with the environment, it also builds a tree of states in the environment connected by the macro-actions leading to each of them. Each vertex of the tree is assigned a curiosity score, which is used as an exploration bonus for PPO and to determine the probability with which each vertex is sampled from the tree for future exploration. The whole process is repeated until a feasible path from the initial state to the goal state is found. The paper provides empirical evaluations in four different tasks where it compares the performance of CSP with three different curiosity measures to the performance of PPO and A2C. The results show that CSP accomplishes each task while using significantly less samples. Moreover, a second set of experiments is presented that show the potential for transfer learning across tasks using CSP.
Contributions:
1. The paper introduces CSP, a successful combination of task and motion planning and deep reinforcement learning that can discover temporally extended plans.
2. The paper demonstrates a statistically significant improvement in performance over PPO and A2C in the four robotic tasks that the paper studies.
3. The paper shows evidence that CSP might facilitate transfer learning across similar tasks.
===== Decision =====
The paper represents a significant contribution to the reinforcement learning and task and motion planning literature. The main algorithm is well motivated from previous literature and demonstrate a significant improvement over previously proposed deep reinforcement learning methods. Moreover, the ideas are presented clearly and logically throughout the paper and the empirical evaluations clearly support the claims about the performance of CSP. However, I have concerns about the reproducibility of the results because of the little amount of details provided about the hyper-parameter selection and settings and about the network architectures and loss functions. Thus, I consider that the paper should be rejected, but I am willing to increase my score if my comments are properly addressed.
===== Questions and Comments =====
1. Although the ideas in the paper are presented clearly, the algorithms and methods are presented mostly at a very high level. There are no details about the losses used in lines 11 and 12 of Algorithm 1 and there are no specifications about the network architectures and the hyperparameter settings and selection for each algorithm. This raises two concerns. First, this hinders reproducibility and future work by other authors that might be interested in building upon the ideas presented in the paper; this would also decrease the impact of the paper. Two, it is difficult to determine if the comparisons against A2C and PPO were fair without any information about the hyper-parameter selection. Thus, I consider these details should be included and I would consider increasing my score to accept if this was properly addressed. Specifically, I think the paper should include this:
- The hyper-parameter settings for each different algorithm and an explanation about how they were selected.
- A detailed description of the network architectures used in the experiments.
- A definition for the loss functions used for the policy network, the value network, and the curiosity module.
2. As mentioned in the Decision section above, the paper clearly demonstrates an improvement over previously proposed deep reinforcement learning algorithm. However, there are no comparisons to any previously proposed Task and Motion Planning Methods. What motivated this decision?
3. An alternative to using separate networks for the policy and the value function, it could be possible to use a two-headed network with one head for the policy and another one for the value. What was the reason for using two separate networks over this alternative?
4. Were any other alternatives tested for the activation functions of the network?
5. CSP was tested with three different curiosity and novelty metrics, none of which dominated over all the other ones. However, PPO was only tested with one of the measures and A2C had no curiosity measure added to it. Where there any preliminary results that justified this decision? In terms of computation and time, how difficult would it be to include this in the paper?
6. The paper already hinted at this, but macro-actions could be framed within the option framework from Sutton, Precup, & Singh (1999). This would open up the opportunity to apply some of the already proposed methods for option discovery such as the option-critic architecture from Bacon, Harb, & Precup (2016) or the Laplacian framework for option discovery from Machado, Bellemare, & Bowling (2017), which is cited on the paper. Could the authors provide more comments about this line of future work?
===== References =====
Bacon, P., Harb, J., & Precup, D. (2016). The Option-Critic Architecture. Retrieved 17 October 2019, from https://arxiv.org/abs/1609.05140
Marlos C. Machado, Marc G. Bellemare, and Michael H. Bowling. A laplacian framework for option discovery in reinforcement learning. CoRR, abs/1703.00956, 2017. URL http://arxiv.org/abs/1703.00956.
Sutton, R., Precup, D., & Singh, S. (1999). Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning. Artificial Intelligence, 112(1-2), 181-211. doi: 10.1016/s0004-3702(99)00052-1


Origin-Destination (OD) flow data is an important instrument in transportation studies. Precise prediction of customer demands from each original location to a destination given a series of previous snapshots helps ride-sharing platforms to better understand their market mechanism. However, most existing prediction methods ignore the network structure of OD flow data and fail to utilize the topological dependencies among related OD pairs. In this paper, we propose a latent spatial-temporal origin-destination (LSTOD) model, with a novel convolutional neural network (CNN) filter to learn the spatial features of OD pairs from a graph perspective and an attention structure to capture their long-term periodicity. Experiments on a real customer request dataset with available OD information from a ride-sharing platform demonstrate the advantage of LSTOD in achieving at least 7.94% improvement in prediction accuracy over the second best model.

Review Summary
--------------
While I think there are some interesting innovations here, I don't think this is ready for ICLR. I have concerns that the evaluation doesn't focus enough on application-relevant scenarios (why exclude cells that are not in the top 50? why not use 7-day windows to capture day-of-week effects?), and that the evaluations don't quantify uncertainty (and thus the claimed improvements due to attention may not be significant). The presentation quality of the method details in Sec. 3 needs a significant rewrite to improve clarity. I do think the non-standard convolution operator is a nice idea.
Paper Summary
-------------
This paper addresses the forecasting of origin-destination demand data, with a focus on ride-sharing transportation applications. We assume an urban area has been divided up into N cells (known in advance). The problem is to forecast demand for *directed* rides from cell i to cell j, using historical demand data.
The paper's first contribution is the design of a convolutional neural net architecture that uses non-rectangular receptive fields more suitable to origin-destination data. Instead of assuming that cells in a given 2D euclidean neighborhood are correlated, it assumes that all journeys that overlap with the one of the cells in the current origin-destination pair are similar. They call this the "SACN" architecture.
The second contribution is developing an "ST-Conv" block that combines the above SACN convolution with gated temporal convolutions, so that both space and time can be summarized via convolution operators. A periodically-shifting attention mechanism is further used to measure long-term data's similarities to short-term data and weight accordingly.
Overall, these two ideas (SACN convolutions and ST-Conv blocks with attention) are combined into what they call their Latent Spatio-Temporal Origin-Destination or "LSTOD" architecture, which is claimed to be the first to use both short-term and long-term features in prediction.
Evaluation examines demand for ride-sharing in two major cities, A and B (presumably masked because they are proprietary). RMSE error comparing to several classic (e.g. ARIMA) and deep feature learning (e.g. LSTMs and SRCN) baselines, with primary results in Table 1. A few side experiments examine superiority over standard convolutions (Fig 3) and experiments without the attention and long-term bits of the model (Table 2).
Novelty & Significance
-----------------------
The current paper is quite niche when it comes to significance; I think it may be a bit too specialized for most ICLR readers. Solving this kind of forecasting problem seems important to ride-sharing applications, but is highly specialized for origin-destination demand data. Would be nice to see the paper attempt to connect to other problems beyond ride-sharing (maybe animal migration? maybe package logistics?).
That said, I think there's sufficient novelty. The architectural design contributions here do appear new to me (though I don't follow this kind of data closely).
Method Concerns
------------------
## M1: Complexity analysis missing, scaling could be a problem
When comparing the current square receptive field architecture for CNNs with the proposed SACN, I felt there was an opportunity to clarify how both scale with N and other key problem size parameters in terms of number of parameters or execution time. I'm concerned that it will be non-scalable given the O(N) cost in terms of number of parameters and the O(N^2) cost of runtime (you need to execute Eq. 2 for each of the N^2 entries at each layer). I'd like to see some careful breakdown of this compared to other approaches.
Experimental Concerns
---------------------
## E1: Why not use a 7-day past history?
Shouldn't ride share demand have a day-of-week trend? Why wouldn't we use last Sunday's demand to predict this Sunday's demand? I find it quite odd that for the Historical Average baseline, only the last 5 days (rather than last 7) are used, and for the presented method, only the last 3 days are used. I'd be happy to be proven wrong, but I'd guess including a 7-day window would lead to noticeably better results.
## E2: Lack of error bars / uncertainty quantification
A natural question is, can we reliably tell that the difference between (for example) RMSE 2.49 and 2.54 is significant and not noise?  I'd like to see some attempt at quantifying the uncertainty for measurements in Table 1 (perhaps taking each full day in test set as its own "mini" test set that produces one RMSE score, then reporting average as well as 2.5th and 97.5th percentiles or something across each day). Without this, I think the claim that attention is useful here is unproven, since the change in performance is so small (less than 0.1 RMSE).
## E3: Why focus only on the N=50 most common cells?
I would think to really assess demand forecasting, you want to know when to task drivers to visit less-common cells. The focus on the cells with only the top 80% of journeys means that 1 of every 5 rides would not be covered by this prediction system. I wonder if part of the reason is that using N much larger than 50 is problematic due to the scaling mentioned earlier.
Presentation Concerns
---------------------
## P1: Need to simplify notation
I found the descriptions of the neural net architecture throughout Sec. 3 quite hard to parse. I think there's an overreliance on math notation and there could be more simple description of high level intent and motivation.
For example, Eq. 2 could be simplified to avoid the channel "m" and layer "l" notation and thus let the reader focus on what matters, which is how any part of input A that involves the origin node i or the destination node j is included in the weighted sum.  Words can be used around this to clarify this same operation can happen across channels and layers.
Similarly, Eq. 3 and Eq. 4 could perhaps be replaced by a good diagram. Currently, Eq. 3 both P and Q are undefined and quite confusing.
## P2: Name of the convolution operation
I'm not sure "SACN" is the best name. The receptive field of a standard CNN looks much more "spatially adjacent" to me (a compact rectangle surrounding the target cell). Instead, you might call it "topologically adjacent" or even something like "vertex adjacent". You want to emphasize that you are getting strength by finding all journeys whose start or end overlaps with one of the endpoints of the current journey.


Score-function-based text generation approaches such as REINFORCE, in general, suffer from high computational complexity and training instability problems. This is mainly due to the non-differentiable nature of the discrete space sampling and thus these methods have to treat the discriminator as a reward function and ignore the gradient information. In this paper, we propose a novel approach, CaptainGAN, which adopts the straight-through gradient estimator and introduces a "re-centered" gradient estimation technique to steer the generator toward better text tokens through the embedding space. Our method is stable to train and converges quickly without maximum likelihood pre-training. On multiple metrics of text quality and diversity, our method outperforms existing GAN-based methods on natural language generation.

The submission proposes to train a GAN on discrete sequences using the straight-through Gumbel estimator introduced in Jang et al. (2016) in combination with gradient centering. The proposed approach is evaluated on COCO and EMNLP News in terms of BLEU and Self-BLEU scores, Fréchet Embedding Distance, Language Model Score, and Reverse Language Model Score.
My assessment is that the submission is below the acceptance bar, mainly due to clarity and novelty concerns. The proposed approach does have empirical backing, but I would argue that it is a very straightforward application of the straight-through Gumbel estimator to GANs, which is itself similar to existing work on applying the Gumbel-softmax estimator to GANs (Kusner & Hernández-Lobato, 2016). Detailed comments can be found below.
The submission does not feel self-contained. For instance, it borrows notation from Jang et al. (2016) without explicitly acknowledging it, and my personal experience is that reading Jang et al. (2016) beforehand makes a big difference in terms of clarity in Section 2.2.
The notation is inconsistent and confusing, and gets in the way of understanding the proposed approach. Here’s a (non-exhaustive) list of examples:
- The reward function is first introduced as f_\phi(\mathbf{x}) above Equation 3, but all subsequent mentions of the reward function use f_\phi(\hat{\mathbf{x}}).
- The \mathbf{m}_\theta variable is introduced in Equation 5 and is immediately replaced with \mathbf{p}_\theta, which adds notational overhead without any benefit.
- The difference between \hat{\mathbf{x}} and \hat{x} is not explained in the text. From the context I understand that \hat{x} is a categorical scalar in {1, …, V}; is this correct?
- In Equation 6, x_1, …, x_V are used to denote the *values* that \hat{x} can take. This clashes with the previous convention that \mathbf{x} is a sequence sampled from p_{data} (Equation 1). Given that convention and the difference between bolded and non-bolded variables discussed above, I would have expected that x_1, …, x_V would correspond to the categorical values of elements of the \mathbf{x} sequence. That contributes to confusion in Equation 9, where \mathbf{e}_{x_t} and p_\theta(x_t) are *not* time-dependent.
- Equation 8 sums over time steps, but the first summation that appears in Equation 8 does not make use of the temporal index. There is also a symbol collision for T, which is used both as the sequence length and as the "transpose" symbol.
As a result, the proposed centering method and the rationale for it is still not entirely clear to me. In particular, is the gradient centering approach necessary to avoid the drawback of score function-based approaches (i.e. the generator is only given feedback on the tokens it samples), or does the non-centered, straight-through variant of the proposed approach also avoid this drawback?
I’m also not convinced that the centering heuristic is a crucial component of the proposed approach when the biggest improvement observed over the straight-through baseline is obtained by adding spectral normalization. I would argue that the proposed approach is a straightforward application of the straight-through Gumbel gradient estimator to GAN training, which is similar in spirit to work by Kusner & Hernández-Lobato (2016) (not cited in the submission) -- the main difference being that the latter uses the Gumbel-softmax distribution directly and anneals the temperature parameter over the course of training. A comparison between the two would be warranted.
References:
- Kusner, M. J., & Hernández-Lobato, J. M. (2016). GANs for sequences of discrete elements with the Gumbel-softmax distribution. arXiv:1611.04051.


Regularization is known to be an inexpensive and reasonable solution to alleviate the overfitting problem arising in inference models, including deep neural networks. In this paper, we propose hierarchical regularization which preserves the semantic structure of a sample distribution. At the same time, this regularization promotes diversity by imposing a distance between parameter vectors enlarged within semantic structures. To generate evenly distributed parameters which is considered less redundant, we constrain them to lie on hierarchical hyperspheres. To define a hierarchical parameter space, we propose to reformulate the topology space with multiple hyperspheres. On each hypersphere, a projection is parameterized by two individual parameters. Since maximizing a groupwise and pairwise distance between points on hypersphere is nontrivial (generalized Thomson problem), we propose a new discrete metric integrated with a continuous metric. The proposed method shows improved generalization performance on extensive experiments using publicly available datasets (CIFAR-10, CIFAR-100, CUB200-2011, and Stanford Cars), especially when the number of super-classes is large.

The paper proposes a hierarchical regularization framework based on hierarchical hyperspheres. In particular, the paper tackles the problem of diversity promoting learning. Following (Liu et al., 2018), pairwise distances between parameters on hyperspheres are used in the regularization framework.
The topology of the parameter space is reformulated with multiple hypersphere spaces which are each defined by two parameters: the centroid of a sphere and its surface vector. Multiple strategies involving hierarchical hyperspherical structures are proposed in Section 3 (continuous and discrete).
The relevance of the proposed method is experimentally demonstrated on different computer vision datasets.
I vote for reject for the following reasons:
- The paper is hard to read in general. Although the method section is understandable, its readability could be improved because each method currently just looks like a succession of equations. The paper also does not really give an intuition of why (or what contexts) one of the proposed regularizers would be better than the others.
- The reported (test accuracy) scores do not seem significantly better than the l2 baseline: none of the reported scores beats the l2 baseline by at least 1 percent, and it is unclear how that difference is measurable. How many splits/different initalizations were used? Why not give standard deviation over different test splits? etc... Given the fact that the improvements do not seem significant compared to a single baseline, a proper evaluation with standard deviation should be provided.
- Although the paper cites (Liu et al., 2018) as motivation for their framework, why does the proposed method does not compare to the other related work (i.e. works by Xie)?
As a side note, the paper seems to motivate the use of multiple spherical spaces to represent hierarchies. Recent work in machine learning has shown the advantage of using hyperbolic geometry [A,B] to represent trees, hence hierarchies.
[A] Nickel and Kiela, Poincaré Embeddings for Learning Hierarchical Representations, NIPS 2017
[B] Ganea et al., Hyperbolic Neural Networks, NeurIPS 2018
======= after the rebuttal
I have read carefully the updated manuscript, other reviews and rebuttal.
My score does not change since the proposed method does not seem to improve much compared to a simple weight decay (baseline + l2 regularization). The motivation of using the method for a very small improvement is not convincing. The "well-defined hierarchical information which is categorized by an expert as mentioned in the manuscript" can also be exploited by hyperbolic representations and should then also be compared (as baseline).