Training neural-networks is computationally hard. However, in practice they are trained efficiently using gradient-based algorithms, achieving remarkable performance on natural data. To bridge this gap, we observe the property of local correlation: correlation between small patterns of the input and the target label. We focus on learning deep neural-networks with a variant of gradient-descent, when the target function is a tree-structured Boolean circuit. We show that in this case, the existence of correlation between the gates of the circuit and the target label determines whether the optimization succeeds or fails. Using this result, we show that neural-networks can learn the (log n)-parity problem for most product distributions. These results hint that local correlation may play an important role in differentiating between distributions that are hard or easy to learn.

Summary:
The paper proposes a layer-wise method for training the weights of a binary-tree-structured neural network such that it correctly reproduces certain classes of Boolean functions defined by binary-tree-structured Boolean circuits. Specifically, this paper shows analytically that if a circuit satisfies a property termed “local correlation” where there is sufficient correlation between every gate in the circuit and the true output label of the circuit, then this circuit can be learned by a neural network with the same structure as the circuit by training it one layer at a time from the input to the output. The paper motivates this by showing empirically that the k-parity problem with some bias to the labels can be learned by a neural network, but that this does not work when there is no bias in the labels, implying that this bias is necessary for successful learning. The paper shows formally that instances of the k-parity problem satisfy the local correlation assumption and can thus be learned, and also shows that there exists at least one distribution given by a simple generative model that satisfies this assumption and is thus also learnable in this manner.
Overall:
Weak reject. My main concerns are as follows.
First, if you define a circuit such that each gate’s output is correlated with the actual label, and then simply copy that structure and learn each gating function independently to predict the value that it is correlated with, then it seems likely that each additional gate should improve the representation. And because the network mimics exactly the circuit structure, the capacity of the network will not be a problem either. Thus, while it’s not trivial to say that the function can be recovered exactly, it is not exactly surprising either, and I don’t see what this offers in terms of novel intuition.
Second, the definitions and assumptions are not very reflective of what is actually done in deep learning, and I do not see a clear connection that would make this result useful in the field. It doesn’t connect well enough to actual datasets, problems, models, or learning algorithms that are used, so I do not see what insights can be taken from it.
However, I do not have a problem with the quality of the paper, and think it would find a more appropriate audience at a different venue.
Clarity: The paper is quite well written and intelligible, although the notation is fairly dense and not always intuitive (e.g., numbering layers of the network from output to input).
Significance: I do not think the results are particularly significant.
Detailed comments:
Section 1.
- The claims made in this section come off overly strong for the remainder of the paper. For example, training a deep neural network is not necessarily computationally hard because training is almost always taken to mean using SGD to locally minimize some loss function, and not taken to mean global optimization. This should be made more clear. Further, in my opinion, the “holy grail” of theoretical deep learning right now is the work on understanding why these networks generalize, not on global optimization or exact function learning (although there is some overlap between these two goals).
- Can you better explain why correlation between the input bits and the label being necessary is not obviously necessary for all standard correlation-based learning methods? I agree that it would be nice if our methods were not so limited but this seems to simply reiterate the fact that they are. (e.g., I would not expect a deep network to learn on ImageNet if there was no correlation between some pixels and the output).
- The ImageNet example is not particularly clear. First, the appendix implies you are taking center crops from each image, not random crops as stated in the intro. Second, I assume that you take a center crop per image, but the text is written to sound like only a single crop for all of ImageNet is used, which doesn’t make sense.
- I do not think that the ImageNet experiment implies that local correlation exists in ImageNet any more than is already well known. It simply shows that an even smaller center crop from ImageNet images is still mildly predictive of the output classes; however, since ImageNet images are generally object-centered already, this is not particularly surprising.
Section 2.
- This is a well-written section that does a good job of situating this work.
Section 3.
- The assumption that the circuit must be a binary tree is quite restrictive and excludes pretty much all common deep learning architectures. Could this be relaxed empirically, even if not analytically?
- In neural networks, it is common to define layer 1 as the input and layer d as the output, and it wasn’t immediately clear from the notation here that the opposite was being done here. Please make this more clear early on (or flip the ordering).
- In the neural-gate definition, should v_i be v_l? Otherwise what is i indexing? Further, since v is not actually learned later, can it just be removed? This would simplify the notation somewhat.
- How are v and w initialized?
Section 4.
- While it is strong to assume that every gate’s output should correlate with the label, it’s natural to think that many of each layer’s output in an actual neural network will correlate with the output, since earlier layers can be thought of as an input corresponding to a transformed representation of the actual input. This is also implied by the Belilovsky et al. layer-wise training work. Making these points more clearly and providing some empirical evidence could strengthen this paper’s claims. Perhaps measure the correlation between outputs in different layers and the labels?
- Assumption 2 is at odds with the fact that datasets are typically designed to be as balanced as possible. Can you reconcile these?
Appendix.
- It would be nice if this read as well as the main paper.


Learned joint representations of images and text form the backbone of several important cross-domain tasks such as image captioning. Prior work mostly maps both domains into a common latent representation in a purely supervised fashion. This is rather restrictive, however, as the two domains follow distinct generative processes. Therefore, we propose a novel semi-supervised framework, which models shared information between domains and domain-specific information separately. The information shared between the domains is aligned with an invertible neural network. Our model integrates normalising flow-based priors for the domain-specific information, which allows us to learn diverse many-to-many mappings between the two domains. We demonstrate the effectiveness of our model on diverse tasks, including image captioning and text-to-image synthesis.

Summary:
The paper proposes a model for joint image-text representations
The paper proposes a model fro cross-domain generative tasks, specifically image captioning and text-to-image synthesis. The proposed Latent Normalizing Flows for Many-to-Many Mappings uses normalizing flows to model complex joint distributions. The latent representation consist of domain-specific representation and cross-domain information shared across image and text using invertible metrics.
Novelty:
- The paper explores an interesting area of learning representations for cross-domain tasks such as image captioning and text-to-image synthesis.
- The model is well explained.  Section 3 explains the model formulation nicely, has consistent notation and slowly  builds up to the final formulation by explaining each component concisely.
- Recent methods like VQ-VAE [1]  have shown promising results for image generation. The related work doesn't provide any discussion regarding that.
Experiments / Analysis:
- The model contains exhaustive experiments for both image-captioning and image generation. The model performs on-par or beat state of the art methods on both perceptual and diversity metrics. On diversity metrics, the model performs much better than other recent methods like Seq-CVAE (which arrived on ArXiv only a few weeks prior to the submission deadline) and POS.
- The model also shows results on text-to-image synthesis comparing with multiple baselines and various diversity metrics and inception score.
- While the proposed methods beats existing diversity and perceptual metrics, it'd be good to also run a human study since these metrics are only a proxy to human judgement.
- Apart from showing empirical result, the paper can benefit from providing Insights what the domain specific representation has learnt and what the cross-domain representation learnt.
- Can the model benefit from training on unaligned image and textual data to learn better domain specific representations?
Clarity:
- Ablations not clear: It's not a 100% clear from the paper what LNFMM-MSE and LNFMM-TXT mean. "LNFMM-TXT contains unsupervised dimensions only for the text distribution and
all encoded image features are used for supervision, i.e.without fφv" What does this sentence mean?  Similarly, it's not clear what "LNFMM (semi-supervised, 30% labeled)" mean?
- It's also not clear why the authors call the approach a semi supervised setup? For instance,  the paper relies on supervision from  paired image-caption data to train the model.
[1] Neural Discrete Representation Learning; Aaron van den Oord, Oriol Vinyals, Koray Kavukcuoglu
**Update after rebuttal**
Thank you for clarifications to my questions regarding the ablations and  using unaligned image and textual data to learn better domain specific representations. The visualizations in Appendix A.4 are also somewhat helpful in understanding what the representations have learnt. After reading the rebuttal, and considering that they will run the human study to further validate their approach in the final manuscript, I am happy to raise the score.


Recurrent Neural Networks (RNNs) facilitate prediction and generation of structured temporal data such as text and sound. However, training RNNs is hard. Vanishing gradients cause difficulties for learning long-range dependencies. Hidden states can explode for long sequences and send unbounded gradients to model parameters, even when hidden-to-hidden Jacobians are bounded. Models like the LSTM and GRU use gates to bound their hidden state, but most choices of gating functions lead to saturating gradients that contribute to, instead of alleviate, vanishing gradients. Moreover, performance of these models is not robust across random initializations. In this work, we specify desiderata for sequence models. We develop one model that satisfies them and that is capable of learning long-term dependencies, called GATO. GATO is constructed so that part of its hidden state does not have vanishing gradients, regardless of sequence length, and so that its state does not explode. We study GATO on copying and arithmetic tasks with long dependencies and on modeling intensive care unit and language data. Training GATO is more stable across random seeds and learning rates than GRUs and LSTMs.

The paper proposes a new RNN architecture designed to overcome vanishing/exploding gradient problems and to improve long-term memory for sequence modelling. The main ideas are (i) to split the hidden state into two parts, one of which does not influence the recurrence relation, and can therefore not blow up or contract by self-feedback; and (ii) to use periodic functions, in particular the cosine, as non-linearity in the decoder, so that the output is bounded but does not saturate.
The paper puts forward a fairly systematic analysis of the gradients in RNNs. The analysis appears correct, and is in fact quite similar to considerations in earlier RNN work (which is correctly cited), and forms the basis for the proposed GATO unit. There are two loose ends in this part:
1) the cosine non-linearity results from a purely negative selection - the function should be bounded, but not saturating. The paper does not even ask the question which periodic function might be a good choice.
2) While the method is presented as a grand theory, with the only constraint that a part of the hidden state does not influence the recurrence function; the actual implementation and experiments are limited to the narrow special case of "non-interacting" GATO, where the "passive" variables make up exactly half of the hidden vector, and the update of each individual hidden variable is influenced only by a single variable from the previous state. So there are in fact no empirical results, not even on toy data, for the general case that the paper claims to introduce.
In the experiments, there are two artificial problems (copying, adding) for sequences of symbols. These are illustrative and sensible to verify and analyse  the behaviour of GATO in a controlled setting, but rather far from most real sequence modelling tasks. In a third experiment the task is to classify whether or not patients will stay in intensive care for >1 week, based on a (seemingly private) database of time series of vital parameters. Unfortunately, the results for that experiment do not go beyond the usual "ours is better". While the numbers clearly support GATO, it would have been nice to look a bit closer and pinpoint what makes the difference. Also, the comparison is a bit loose. It would have been better to add additional baselines where also LSTM and GRU are restricted to "non-interacting", element-wise recurrence (as far as technically feasible). As it stands, it is unfair to claim "we can do it with much fewer parameters" - perhaps LSTM / GRU could, too. In fact, it could even be that the task is just simple, so that more restricted model with fewer parameters generally perform better - I do not claim this is the case, but the experiments do not rule it out and, hence, do not confirm that the clever GATO recurrence makes the difference.
A small gap is also that even the "real" experiment might not be completely realistic. Nowadays it is a popular strategy to use deep LSTMS / GRUs, i.e., stack multiple levels of recurrence, possibly with temporal sub-sampling, to better capture long-term relations. While this is potentially even more brittle, because of the additional gradient flow across layers, it does seem to work. But deep RNNs are not tested (in fact, not even mentioned) in the paper.
Overall, in spite of a few loose ends, I find both the design and the practical performance of the proposed GATO unit very interesting, and potentially valuable for the ICLR crowd. This is one of the more convincing RNN papers I have recently read.


Conventionally, model-based reinforcement learning (MBRL) aims to learn a global model for the dynamics of the environment. A good model can potentially enable planning algorithms to generate a large variety of behaviors and solve diverse tasks. However, learning an accurate model for complex dynamical systems is difficult, and even then, the model might not generalize well outside the distribution of states on which it was trained. In this work, we combine model-based learning with model-free learning of primitives that make modelbased planning easy. To that end, we aim to answer the question: how can we discover skills whose outcomes are easy to predict? We propose an unsupervised learning algorithm, Dynamics-Aware Discovery of Skills (DADS), which simultaneously discovers predictable behaviors and learns their dynamics. Our method can leverage continuous skill spaces, theoretically, allowing us to learn infinitely many behaviors even for high-dimensional state-spaces. We demonstrate that zero-shot planning in the learned latent space significantly outperforms standard MBRL and model-free goal-conditioned RL, can handle sparse-reward tasks, and substantially improves over prior hierarchical RL methods for unsupervised skill discovery. Video demonstration of our results are available at: https://sites.google.com/view/dads-skill Figure 1: A humanoid agent discovers diverse locomotion primitives without any reward using DADS. We show zero-shot generalization to downstream tasks by composing the learned primitives using model predictive control, enabling the agent to follow an online sequence of goals (green markers) without any additional training.

This paper proposes a novel approach to learn a continuous set of skills (where a skill is associated with a latent vector and the skill policy network takes that vector as an extra input) by pure unsupervised exploration using as intrinsic reward a proxy for the mutual information  between next states and the skill (given the previous state). These skills can be used in a model-based planning (model-predictive control) with zero 'supervised' training data (for which the rewards are given), but using calls to the reward function to evaluate candidate sequences of skills and actions. The proposed approach is convincingly compared in several ways to both previous model-based approaches and model-free approaches.
This is a very interesting approach, and although I can already think of ways to improve, it seems like an exciting step in the right direction to develop more autonomous and sample efficient learning systems. I suggest to rate this submission as 'accept'.
Regarding the comparison to model-free RL: although it is true that no task-specific training is needed, a possibly large number of calls to the reward functions are needed during planning. It would be good to compare those numbers with the number of rewarded trajectories needed for the model-free approaches.
My main concern with the proposed method is how it would scale when the state-space becomes substantially larger (than the 2 dimensions x and y used in the experiments). The reason I am concerned is that the proposed method uses brutal sampling to search for good trajectories in z-space and action-space. It looks like the curse of dimensionality will quickly make this approach unfeasible. Also, it would be nice to have the learning system discover the important dimensions in which to plan (the x and y in the experiments), rather than having to provide them by hand.
A minor concern is the following: is it possible that the optimization could end up discovering a large number of highly predictable (and diverse) but useless skills?
In the related work section, 1st paragraph, in the list of citations, it might be good to also include the work on maximizing mutual information between representation of the next state and representation of the skill (Thomas et al, arXiv:1802.09484).
The definition of Delta (page 9) is strange: it is said that Delta should be minimized but Delta is defined as proportional to the rewards (which should be maximized). Maybe a sign is missing. Also, why not simply define the rewards as being normalized in the first place, so that the metric IS the accumulated reward rather than this unusual normalized version of it.


It has been noted in existing literature that over-parameterization in ReLU networks generally improves performance. While there could be several factors involved behind this, we prove some desirable theoretical properties at initialization which may be enjoyed by ReLU networks. Specifically, it is known that He initialization in deep ReLU networks asymptotically preserves variance of activations in the forward pass and variance of gradients in the backward pass for infinitely wide networks, thus preserving the flow of information in both directions. Our paper goes beyond these results and shows novel properties that hold under He initialization: i) the norm of hidden activation of each layer is equal to the norm of the input, and, ii) the norm of weight gradient of each layer is equal to the product of norm of the input vector and the error at output layer. These results are derived using the PAC analysis framework, and hold true for finitely sized datasets such that the width of the ReLU network only needs to be larger than a certain finite lower bound. As we show, this lower bound depends on the depth of the network and the number of samples, and by the virtue of being a lower bound, over-parameterized ReLU networks are endowed with these desirable properties. For the aforementioned hidden activation norm property under He initialization, we further extend our theory and show that this property holds for a finite width network even when the number of data samples is infinite. Thus we overcome several limitations of existing papers, and show new properties of deep ReLU networks at initialization.

This work considers random parameter initialization in neural networks (In particular the initialization presented in He et al.) and develops non-asymptotic bounds for the norms and gradients of neural networks during initialization. The authors show that the norms of the outputs and gradients (for gradients, under a different assumption on the dimension of the matrix) remain constant through the different layers. The results presented differ from previous work in that they give nice concentration bounds for such output and gradient norms. In addition the authors prove results in the case of infinite samples under the assumption that they arise from a finite dimensional space.
Overall the presentation is clean, but the results presented have the following issues.
1.Very similar in nature to previous results, for example the results presented in [1] Theorem 5.4 give very similar concentration results and use very similar mechanisms.
2. All the results for the infinite stream coming from a finite dimensional subspace do not address the fact that the training points usually do not lay in a linear subspace of dimension d << n. Further, there is a stringent requirement on d in such results and they fail to hold for even "fairly" small d.
3. In Theorem 2, the argument that the output through a layer of a rank-d linear subspace remains within a rank-d linear subspace does not seem correct, is it possible that it remains in the union of subsets each of which lies in a subspace of rank d?
4. The proofs for the gradients make assumptions that deviate from the initialization previously introduced. The work of Glorot et al. for example discusses the tradeoffs between the two assumptions and suggests the balancing between maintaining the input and output variance distributions.
Due to the following issues I choose to reject this work at this time.
Below are additional minor typos or issues in the paper.
On page 6, the sentence that starts with: "it is beneficial for the width": Suggesting to use neural networks of constant width seems a bit impractical.
The citation of Arpit et al. from 2017 seems possibly wrong since the paper cited discusses memorization and does not focus at the initialization of neural networks.
TYPOS:
page 1 last sentence before equation 1:  back backward?
page 1 last sentence: repeated words: as as the the
page 1 last sentence: property --> properties?
page 2 section 2 paragraph 1: both these papers --> both papers, both of these papers?
page 4 sentence after the proof sketch paragraph: the sentence is difficult to read
[1] "Stochastic gradient descent optimizes over-parameterized deep relu networks", 2018, Difan Zou, Yuan Cao, Dongruo Zhou, Quanquan Gu


Learning high-level representations for graphs is of great importance for graph analysis tasks. In addition to graph convolution, graph pooling is an important but less explored research area. In particular, most of existing graph pooling techniques do not consider the graph structural information explicitly. We argue that such information is important and develop a novel graph pooling technique, know as the STRUCTPOOL, in this work. We consider the graph pooling as a node clustering problem, which requires the learning of a cluster assignment matrix. We propose to formulate it as a structured prediction problem and employ conditional random fields to capture the relationships among assignments of different nodes. We also generalize our method to incorporate graph topological information in designing the Gibbs energy function. Experimental results on multiple datasets demonstrate the effectiveness of our proposed STRUCTPOOL.

The authors here introduces a novel  graph pooling technique called StructPool that uses the underlying graph’s structural information to behave as a node clustering algorithm and learns a node clustering matrix.
Graph level classification requires learning good graph level representation, especially for  aggregating low level information for high . Recent work in pooling does not take advantage of important structural information of the relationship between different  nodes. Here, the authors formulate  graph pooling as a structured prediction problem, control clique set in the CRFs and use mean filed approximation to calculate assignments.
A cluster assignment matrix assigns each node in the original graph to a cluster in the new graph. The assignment not only depends on the node features but also on the cluster assignment of the other nodes. The authors therefore draw connection with finding the optimal assignment to minimizing the Gibbs energy. The authors propose to learn clustering assignment via CRF conditioned on the global feature representation of the nodes.
The unary potentials of the cliques are computed used the GCN to measure energy of each node. The novelty in accommodating topology information is in using l hop connectivity based on adjacency A to define pairwise cliques thus building pairwise relationship between pairs of nodes thus allowing the Gibbs energy formulation of the cluster assignment thereby using GCN to also compute this pairwise energy.
I have  a few questions as below:
I think the authors can better elucidate the motivation for using  the attention matrix over Gaussian kernels to measure pairwise energy in section 3.3; an  empirical experiment for drawing comparison wrt to the computational time and number of feature dimensions on a toy problem seems important.
How is the computation  of the unary potential and pairwise energy influenced by the connectivity of the graph G for the datasets considered? It would be interesting to see how the pairwise energy, unary energy varies over different layers of GCNs.
Further, how is the cluster assignment affected by the l-hop connectivity?
Is there a notion of the minimum value of ‘k’ in the context of convergence?
What happens in case of very different graph features, or structural assumptions where the cliques are not enforced?
Is there a notion of how the method performs on datasets with a high percentage of isomorphism bias: repeating instances or repeating instances with different labels?
It will be interesting to see a discussion on how the performance varies with respect to the depth of the overall architecture,  positioning of the structpool and some results on how effective they are on  hierarchical features and multiple pooling ops as in architectures such as Graph UNet.
Avoid repetition in 2.2 Related work section and in other sections throughout. Otherwise, the paper is rather well written and has clarity.


In natural language inference, the semantics of some words do not affect the inference. Such information is considered superficial and brings overfitting. How can we represent and discard such superficial information? In this paper, we use first order logic (FOL) -a classic technique from meaning representation language -to explain what information is superficial for a given sentence pair. Such explanation also suggests two inductive biases according to its properties. We proposed a neural network-based approach that utilizes the two inductive biases. We obtain substantial improvements over extensive experiments.

This paper presents an approach to treat natural language inference using first-order logic, and to infuse neural NLI models with logical information to be more robust at inference. However, the paper does not contain a single reference to the computational semantics literature, where logical approaches towards semantics were the dominant trend for many years (see e.g. [1, 2]). Indeed, 'neuralising' first order logic has been an active area of recent research ([3] or indeed much of the recent work coming from Sebastian Riedel's group). This is a glaring oversight.
The paper starts by introducing background on first-order logic, and then gives a definition of a 'superficial' predicate, namely one whose extension is not necessary to prove an implication for any collection of background facts. However, by extension, this makes s_1 -> s_2 a tautology, which is the 'true' notion that the authors are looking for. Indeed, if |- (s_1 -> s_2), then for any collection of formulae \Delta then \Delta |- (s_1 -> s_2) (by monotonicity of entailment) and clearly if for any \Delta we have \Delta |- (s_1 -> s_2), we can take \Delta to be the empty set. Finally, the authors show that tautologies are still tautologies under change of predicates (i.e. if we only require logical rules to prove one statement from another, then the extensions of predicates in those statements do not matter).
The authors then use this to motivate two extensions to inference models. One is to 'drop out' word information, and the other is to treat different occurrences of the same word as reflecting the same underlying predicate. The first somewhat transparently forces the model to care less about the exact meaning (i.e. extension in the logical world) of words (indeed, word vectors have been shown to capture extensional information [4, 5]), and so may force the inference model to learn more 'logical' inference rules. Further, the word dropout calculation includes whether the word is in both sentences, which is a strong signal that its extension may not be necessary. However, the second only forces the intuition that different mentions of the same word are likely to be coreferent, which is a weak assumption that models may already pick up. Indeed, it is noticeable that this component seems to be less necessary in the authors' ablation study.
In summary, while I am sympathetic to the aim of grounding neural models in explicit notions of semantics, this paper shows such a lack of awareness of previous literature that I cannot recommend acceptance.
[1] The Meaning Factory: Formal Semantics for Recognizing Textual Entailment and Determining Semantic Similarity, Bjerva et al. 2014
[2] Natural Logic for Textual Inference, MacCartney and Manning 2009
[3] End-to-end Differentiable Proving, Rocktaschel and Riedel 2017
[4] Building a shared world: mapping distributional to model-theoretic semantic spaces, Vecchi and Herbelot 2015
[5] Deriving Boolean structures from distributional vectors, Kreuzewski et al 2015


Previous studies have found that an adversary attacker can often infer unintended input information from intermediate-layer features. We study the possibility of preventing such adversarial inference, yet without too much accuracy degradation. We propose a generic method to revise the neural network to boost the challenge of inferring input attributes from features, while maintaining highly accurate outputs. In particular, the method transforms real-valued features into complex-valued ones, in which the input is hidden in a randomized phase of the transformed features. The knowledge of the phase acts like a key, with which any party can easily recover the output from the processing result, but without which the party can neither recover the output nor distinguish the original input. Preliminary experiments on various datasets and network structures have shown that our method significantly diminishes the adversary's ability in inferring about the input while largely preserves the resulting accuracy.

Abstract:
In this paper, the authors propose to hide information in phase of the input features. They proposed that if each layer of processing layer, sitting outside of the local unit, is phase preserving, then they can recover the phase back. They propose a modification to the most popular layers in DNN to satisfy that property.
I think the general idea of the paper is interesting, but overall, the paper is very poorly written. It appears that it is written in a rush.
*The abstract and introduction are poorly written. There are poorly written sentences in the text. Here are some examples:
    + "... We propose a generic method to revise a conventional neural network to boost the challenge of adversarially inferring about the input but still yields useful outputs. ...."
    + "... given a transformed feature, an adversary can at best recover a number of features which contain at least another k − 1 features which are different but cannot be distinguished from the real feature.  ..." -- This is so central to the whole paper and it is not well-written. Personally, didn't understand this attack.
Please proofread your paper.
*Why do you need both theta and b ?  It seems to me if \| b \| is comparable to \| a \|, it can destroy the information in original the a arbitrary bad, so it makes sense to keep the norm of b small. However, in the paper, it is suggested to use a random sample (I') and set b = g(I'). I don't see any ablation study in the paper. There is no free lunch, if you provide stronger identity preservation, there should be a compromise in the accuracy, and it seems to be the magnitude of the b is that compromise.
* The whole idea of the GAN encoder is not well justified. What does it mean that the fake feature should contain information "beyond" a ? This very vague.
* "Inference attack 1 " and "Inference attack 4" are the same; only the inference models used in each attack are different. I don't know why the author has separated them.


Noisy labels often occur in vision datasets, especially when they are obtained from crowdsourcing or Web scraping. We propose a new regularization method, which enables learning robust classifiers in presence of noisy data. To achieve this goal, we propose a new adversarial regularization scheme based on the Wasserstein distance. Using this distance allows taking into account specific relations between classes by leveraging the geometric properties of the labels space. Our Wasserstein Adversarial Regularization (WAR) encodes a selective regularization, which promotes smoothness of the classifier between some classes, while preserving sufficient complexity of the decision boundary between others. We first discuss how and why adversarial regularization can be used in the context of label noise and then show the effectiveness of our method on five datasets corrupted with noisy labels: in both benchmarks and real datasets, WAR outperforms the state-of-the-art competitors.

Update after rebuttal:
The rebuttal addresses my concerns. The point raised by reviewer 2 regarding other results on Clothing1M is concerning, but the authors' response is reasonable. Moreover, the proposed method is novel and could be complementary with other methods that achieve good results on Clothing1M. I think this paper would generate good discussion, and I recommend acceptance.
----------------------------------------------------------------------------------------
This paper presents a novel approach for dealing with asymmetric label noise. There are in fact two methods proposed, Adversarial Regularization (AR) and Wasserstein Adversarial Regularization (WAR), both of which derive from the intuition that model smoothing (in the sense of virtual adversarial training) acts as label smoothing when there are noisy labels. The authors prove two statements formalizing this notion and demonstrate the efficacy of AR and WAR in an extensive evaluation.
WAR is an iteration on AR that uses the optimal transport distance between categorical distributions to allow incorporating a cost matrix that encodes class similarities. The authors obtain these similarities from word embeddings of class names.
The evaluation demonstrates that the proposed WAR method achieves state-of-the-art results with the largest gains at high noise levels. In no case does it perform substantially worse than competing methods. Moreover, the AR method proposed as an intermediary to WAR also outperforms all prior work at non-zero noise levels, demonstrating the promise of the underlying approach.
The authors also evaluate on Clothing1M and evaluate on a semantic segmentation dataset for which they synthesize structured, realistic label noise. WAR outperforms prior work on these tasks. Moreover, the semantic segmentation task could act as a useful benchmark for future work.
For these reasons, I recommend acceptance.
Minor points:
Comparing the second and third columns in Figure 1, it looks like lambda has a large effect, so it would be good to know how much tuning lambda requires for the main tasks.
In equation 9, shouldn’t there be a beta in front of the entropy term? This does not break the result.
In equation 10, it looks like you’re multiplying both sides by (1 – epsilon), but you drop the (1 – epsilon) on the left side. This does not break the result.


Deep metric learning (DML) has received much attention in deep learning due to its wide applications in computer vision. Previous studies have focused on designing complicated losses and hard example mining methods, which are mostly heuristic and lack of theoretical understanding. In this paper, we cast DML as a simple pairwise binary classification problem that classifies a pair of examples as similar or dissimilar. It identifies the most critical issue in this problemimbalanced data pairs. To tackle this issue, we propose a simple and effective framework to sample pairs in a batch of data for updating the model. The key to this framework is to define a robust loss for all pairs over a mini-batch of data, which is formulated by distributionally robust optimization. The flexibility in constructing the uncertainty decision set of the dual variable allows us to recover state-of-the-art complicated losses and also to induce novel variants. Empirical studies on several benchmark data sets demonstrate that our simple and effective method outperforms the state-of-the-art results.

This paper casts deep metric learning (DML) as a pairwise binary classification problem such that pairs of examples need to be classified as similar or dissimilar. The authors propose an objective function that computes a weighted sum over the pairwise losses in a mini-batch. The weight vector is selected to maximize the objective from a decision set encoding constraints. This formulation is called the distributionally robust optimization (DRO) framework.
The authors argue that the DRO framework is theoretically justified by showing how certain decision sets result in existing machine learning loss functions. This portion of the paper seemed hand-wavy. It is not clear what is the purpose of including the theorem from Namkoon & Duchi. It would be more clear in my view to just make the short point that a certain decision set recovers the DRO with f-divergence as would be expected. The claims with regard to learning theory are over-stated in the paper.
The authors proposed three variants of the general framework. They include a top-K formulation, a variance-regularized version, and a top-K version using a balance between positive and negative examples. The DRO framework and the variants are the main contributions in terms of methodology in this paper. It is also shown that the framework generalizes more complicated recently proposed losses.
The experiments demonstrate the DRO framework consistently outperforms state of the art deep metric learning methods on benchmark datasets by small margins. There is also a computational speed advantage that is shown.
Overall, this paper shows that the ideas from distributionally robust optimization work well in deep metric learning. In particular, the paper shows that by combining the DRO framework with simple loss functions, performance comparable with complicated loss functions can be obtained. This aspect, along with the generality are the main strong suits. That being said, I do not see this paper to be that significant of a contribution. The main idea in the paper seems like a rather direct application of the DRO modeling framework and it does not provide too significant of improvement over the MS loss.  The paper was not written super clearly and was too long. Reviewers were instructed to apply a higher standard to papers in excess of 8 pages and this paper would have been presented more effectively if it was shorter. For these reasons, I recommended a weak reject.


This paper aims to analyze knowledge consistency between pre-trained deep neural networks. We propose a generic definition for knowledge consistency between neural networks at different fuzziness levels. A task-agnostic method is designed to disentangle feature components, which represent the consistent knowledge, from raw intermediate-layer features of each neural network. As a generic tool, our method can be broadly used for different applications. In preliminary experiments, we have used knowledge consistency as a tool to diagnose representations of neural networks. Knowledge consistency provides new insights to explain the success of existing deep-learning techniques, such as knowledge distillation and network compression. More crucially, knowledge consistency can also be used to refine pre-trained networks and boost performance. The code will be released when the paper is accepted.

The submission proposes a method for extracting "knowledge consistency" in neural networks and using that toward analyzing different aspects of them, eg understanding the representations, explaining knowledge distillation, and analyzing network compression. What's defined as consistent knowledge is essentially the stable parts of representations of different networks trained for the same task (stable-->consistent). I found the submission insightful, well executed, and backed up by many experimental results.
Strengths:
A) the proposed concept, extracting the *consistency* among representation of different networks, is interested and using that towards understanding what's going on under-the-hood of neural networks makes sense. I'm not aware of a similar existing method and didn't see one among the citations, so my current assumption is that this proposal is pretty novel.
B) the extracted representational consistency quantity appears to be meaningful and strong, as authors were able to use it toward explaining several existing phenomena (e.g. knowledge distillation, network compression, etc).
C) Authors perform extensive experimental studies on various aspects of neural networks in relation to the proposed representational consistency quantity. I applaud the efforts made by authors.
Improvements/questions:
D) The submission uses a few different phrases in close connection or interchangeably, eg "fuzziness", "order", "linear/nonlinear transformation", "easy/hard to guess/estimation" (see the last 2 paragraphs of page 2). While I understand what the authors are trying to convey, those concepts are not in principle necessarily the same, so some clarification/unification would make the presentation more solid. For instance: nonlinear<--> higher order<-->fuzzy<-->hard to guess. Also guess<-->estimate.
E) Sec 3 is the most important part of the paper and the technical meat. However, I found it harder to follow compared to the rest of the paper. It probably deserves more than 1 page. Authors could consider smoothing the presentation and adding details even at the cost of slightly extending the length.
F) I think section 4.2 could walk the reader through more details to make sure the basics are understood, as this is the first time results of the method are being presented. E.g. fig 3 could be analyzed further and the color maps should be defined.
G) I found the post hoc explanation in the last paragraph of page 6 somewhat dubious.
H) In the feature refinement experiment (paragraph 2 of sec 4.3), is the process done in a recursive manner (first layer 1, then layer 2, and so on)? Or as a one time process? The former seems stronger.
I) The consistency quantity is based on comparing different networks trained for the *same task*. Have you considered doing the same among different networks trained for *different tasks*? Would that say something about similarities among tasks and multi-task learning in a fashion similar to the analysis of Taskonomy 2018?
J) I did not understand "and Beyond" in the title. I'd consider a more directly descriptive title.
K) Seems like authors define "knowledge" and "visual concept" to be the invariant part of a feature (see paragraph 3 of intro). I don't see a particular problem with that, but a direct statement like 'invariant features' would have resonated with me just fine, while whether we can call that "knowledge" is a matter of (unnecessary) semantics in my opinion.
=====
Post rebuttal comments:
Thanks to authors for the detailed response and additions. I read through the comments and skimmed the revised PDF. Overall I preserve my positive rating toward this paper. The updates did improve the paper, but I would recommend the authors to use the camera ready opportunity to further address the original comments especially with respect to the delivery.
I find the added experiments in multi task learning interesting especially given the time constraints of rebuttal. However the adopted setting is not the strongest case as what is defined to be "two tasks" is basically a superclass classification of a fine grained classifier. More concrete datasets and tasks specifically targeting challenging multi task learning are available now (see the full discussion in the original review). I would recommend taking those into consideration either for more experimentation or calibrating the conclusion made out of the current experiment.
I recommend acceptance.


We re-think the Two-Player Reinforcement Learning (RL) as an instance of a distribution sampling problem in infinite dimensions. Using the powerful Stochastic Gradient Langevin Dynamics, we propose a new two-player RL algorithm, which is a sampling variant of the two-player policy gradient method. Our new algorithm consistently outperforms existing baselines, in terms of generalization across differing training and testing conditions, on several MuJoCo environments.

Summary:
The paper considers the task of learning robust policies. Specifically, they focus on the noisy-robust (NR) variant of the action robust framework proposed in [1]. As noted in [1], as strong duality does not hold in the NR variant it can not be solved using deterministic policies (pure strategies).
The authors propose to combine SGLD with DDPG, to maintain a distribution over deterministic policies (essentially a mixed strategy) - hence overcoming these issues.
The results look promising, outperforming [1] in all tested domains.
Review:
Overall this seems like the first approach of using Bayesian learning in order to reach mixed Nash equilibrium in RL. This is super-important, since many problems can be formulated as zero-sum games for which the solution is not necessarily a pure strategy.
As [2] have already shown the ability of this concept to find mixed equilibria in GANs, the main novelty is the introduction to RL.
As this work does not introduce new theory or a dramatic new concept, I feel that the acceptance of this work lies mainly on the empirical side. In my opinion, the experiments need to be more convincing - for instance, include additional domains (such as the Inverted Pendulum which was a failure case in [1]) and additional forms of robustness (e.g., the probabilistic-robust variant from [1] which is based on deterministic strategies and was shown to work better, and RARL [3] which test robustness to external disturbances).
An addition option is to build a low-dimensional toy problem in which the exact solution is known. This will enable you to show that while the naive solution in [1] either does not converge or converges to sub-optimal solutions, the SGLD approach is capable of finding superior solutions (hopefully the global optimum).
The idea is in the right direction. However, in my opinion, it is not there yet and is thus not ready for ICLR.
--- Post Rebuttal ---
I stand by my original assessment. I feel that such work needs to be more convincing. I for one would feel more confident had the authors provided simpler experiments in which they show that their approach indeed converges to the mixed Nash equilibria while the NR-DDPG approach from [1] does not.
I did overall like this direction and I believe Robustness in RL is very important.
[1] Action Robust Reinforcement Learning and Applications in Continuous Control - Tessler et al. 2019
[2] Finding mixed nash equilibria of generative adversarial networks - Hsieh et al. 2019
[3] Robust Adversarial Reinforcement Learning - Pinto et al. 2017


These days deep neural networks are ubiquitously used in a wide range of tasks, from image classification and machine translation to face identification and selfdriving cars. In many applications, a single model error can lead to devastating financial, reputational and even life-threatening consequences. Therefore, it is crucially important to correct model mistakes quickly as they appear. In this work, we investigate the problem of neural network editing -how one can efficiently patch a mistake of the model on a particular sample, without influencing the model behavior on other samples. Namely, we propose Editable Training, a model-agnostic training technique that encourages fast editing of the trained model. We empirically demonstrate the effectiveness of this method on large-scale image classification and machine translation tasks.

-----------
Update after the authors' response: the authors addressed some of my concerns and presented some new results that improve the paper. I am therefore upgrading my score to "Weak Accept".
-----------
This paper proposes a way to effectively "patch" and edit a pre-trained neural network's predictions on problematic data points (e.g. where mistakes on these data points can lead to serious repercussions), without necessarily re-training the network on the entire training set plus the problematic samples. More concretely, the edit operation on the problematic samples are done using a few steps of stochastic gradient descent.
Furthermore, the paper modifies the training procedure to make it easier for the model to perform effective patching, based on three criteria: (i) reliability (i.e. obeying the constraints specified in the edit), (ii) locality (i.e. only minimally changing the model parameters to account for the problematic samples, while ensuring the model's performance on unrelated samples remains consistent and does not degrade), and (iii) efficiency in terms of runtime and memory. To this end, the paper uses a loss term that incorporates these criteria, as weighted by interpolation coefficient hyper-parameters. The edit operation is differentiable with respect to the model parameters, which takes a similar form as the MAML approach and similarly necessitates computing second-order derivatives with respect to the model parameters. Experiments are done on CIFAR-10 toy experiments, large-scale image classification with adversarial examples, and machine translation.
Overall, while the idea of patching neural network predictions is really interesting, I have several concerns regarding the paper in its current form. I am therefore recommending a "Weak Reject". I have listed the pros and cons of this paper below, and look forward to the authors' response to the concerns I have raised.
Pros:
1. The paper is well-written, and clearly motivates the problem and why it is important.
2. The proposed approach is explained very clearly and is easy to understand.
3. The paper correctly identifies the use of distillation loss (Table 4) as a potential confound, and runs a distillation baseline without editable training as an additional baseline. This improves the thoroughness of the experiments, and ensures that the gains can really be attributed to the proposed editable training objective.
4. The paper makes interesting connections to other problems where the editable training approach can potentially be useful, such as catastrophic forgetting and adversarial training.
Cons:
1. I still have my doubts about the locality constraint. If the model makes a mistake on some problematic samples, what we want is not just fixing the model's predictions only on these problematic samples, but also on other samples that share a substantial degree of similarity to the problematic samples, so that the model can avoid making the same broader type of mistakes in the future. On the surface, the locality constraint seems to do the opposite, since it confines the impact of the edit operation only to the problematic samples. In contrast, the alternative of re-training the neural network on the full training set augmented with the problematic samples can potentially overcome this problem (since the entire model parameters are updated, correcting the model's mistakes on the problematic samples would help the model avoid making the same mistakes on other similar samples), although of course computationally much more expensive to do.
2. Related to Point 1 above, the equation defining the locality constraint (Eq. 4) seems to require an expectation of x that is drawn from p(x). How is this quantity computed? Does the model assume that x comes from the empirical distribution? Also, the "control set" (bottom of page 2) is not well defined. This is very important, since the control set defines what examples should (or should not) be changed by the editing operations. These two points should be clarified further.
3. Related to Points 1 and 2 above, I am not sure whether lower "Drawdown" is an indication of better editable training. As mentioned before, ideally the model should not only "patch" its mistakes on problematic samples, but also on samples that are substantially similar to these problematic samples. Naturally this might result in more differences from the initial model, which also means less locality (i.e. higher locality/lower drawdown may not necessarily be a good evaluation metric to evaluate edit effectiveness).
4. The paper contains a substantial set of experiments and analysis based on toy experiments on CIFAR-10, where the labels of some examples are randomly swapped (i.e. random noise). I think this setup is dangerous, since the paper is drawing conclusions based on how well editable training can fit random noise! Instead, all the experiments should be done on real use cases (e.g. adversarial training or catastrophic forgetting). It is unfortunate that the paper has more content for the toy experiments (2.5 pages) than for the adversarial example experiment in image recognition (1.5 pages) or machine translation experiment (0.5 page).
5. The hyper-parameter experiments for selecting the learning rates can be put in the Appendix instead, rather than taking up nearly half a page in the main text (e.g. Table 1). This would leave more space for experiments, more explanation, intuition, analysis, etc.
6. The improvements (in terms of both image classification error rates or machine translation BLEU score) afforded by editable training are quite small. For instance, in the machine translation experiment, Table 6 indicates that editable training only leads to a 0.04 BLEU improvement. This is really small by standard machine translation literature, and can very well be explained by randomness in hyper-parameter initialisation, etc., rather than a better training procedure. It is hard to draw any conclusions based on such small gains.
More minor points:
1. The use of bold in the Tables are not very consistent. For instance, in Table 3, for column "Test Error Rate", 6.31% is in bold, even though baseline training has an accuracy of 6.3%, which is lower (and thus better). Also, in the "test error drawdown" column, 0.86% is in bold, even though there is a lower entry of 0.65%.
2. Missing citation to the work of Furlanello et al. (2018) for self-distillation.
References
Tommaso Furlanello, Zachary C. Lipton, Michael Tschannen, Laurent Itti, and Anima Anandkumar. Born Again Neural Networks. In Proc. of ICML 2018.


Multi-step greedy policies have been extensively used in model-based Reinforcement Learning (RL) and in the case when a model of the environment is available (e.g., in the game of Go). In this work, we explore the benefits of multi-step greedy policies in model-free RL when employed in the framework of multi-step Dynamic Programming (DP): multi-step Policy and Value Iteration. These algorithms iteratively solve short-horizon decision problems and converge to the optimal solution of the original one. By using model-free algorithms as solvers of the short-horizon problems we derive fully model-free algorithms which are instances of the multi-step DP framework. As model-free algorithms are prone to instabilities w.r.t. the decision problem horizon, this simple approach can help in mitigating these instabilities and results in an improved model-free algorithms. We test this approach and show results on both discrete and continuous control problems.

===== Summary =====
The paper proposes an extension of multi-step dynamic programming algorithms from Efroni, Dalal, Scherrer, and Mannor (2018a, 2018b) to the reinforcement learning setting with function approximation. The multi-step dynamic programming algorithms proposed by Efroni et. al. (2018a)  find the solution of the h-step optimal Bellman operator, which applies the maximum over the next h sequence of actions. Moreover, Efroni et. al. (2018a) also showed an equivalence between h-step optimal Bellman operators and k-Policy Iteration (k-PI) and k-Value Iteration (k-VI) algorithms, which, similar to TD( 𝜆 ) but for policy improvement, take a geometric average of all future h-step returns weighted by k. The paper extends the work from Efroni et. al. (2018a, 2018b) to the deep reinforcement learning setting by proposing an approximate k-PI and k-VI algorithm based on DQN and TRPO. Finally, the paper provides empirical evaluations of k-PI and k-VI with DQN in several Atari games and of k-PI and k-VI with TRPO in several MuJoCo environments with continuous actions paces.
Contributions:
1. The paper proposes a non-trivial extension for k-PI and k-VI to use function approximation via the DQN algorithm.
2. Similarly, the paper proposes a non-trivial extension for k-PI and k-VI to use function approximation with continuous action spaces via the TRPO algorithm.
3. The paper provides empirical evaluations of the four proposed algorithms and, at least for the k-PI algorithm with DQN and TRPO, demonstrates an improvement over the baselines.
===== Decision =====
The paper represents a natural next step to the work of Efroni et. al. (2018a, 2018b). The paper extends the applicability of multi-step greedy policies to more complex environments and shows a statistically significant improvement in performance compared to the methods that it builds upon. Additionally, the ideas are presented clearly and incrementally throughout the paper, which makes it flow nicely until the part where k-PI and k-VI DQN and TRPO are introduced. This is my main complaint about the paper, the lack of simple and intuitive understanding about k-PI and k-VI with function approximation due to the complicated architectures associated with DQN and TRPO. For this reason, my rating of the paper is weak accept.
===== Detailed Comments about Decision =====
All of these are comments for which I would consider increasing my score if they were addressed.
=== Empirical Evaluations ===
First, my main complaint is the complicated architectures and complex domains used to gain insights about k-PI and k-VI with function approximation. Big demonstrations in Atari and MuJoCo are important, but in the case of very new algorithms such as these ones, I consider it to be more important to gain insight through small domains that allow us to dig deep into the algorithms. Any small domain that would allow for big sample sizes for ablation and parameter studies would be more insightful than big demonstrations with very small sample sizes. I do not mean to be dismissive about what has been done in the paper, but it would be a great source of insight and a big improvement to what has already been done if a simple demonstration was presented in the paper.
My suggestion would be to use a simple approximation method, such as Tile Coding with linear function approximation, in small a domain such as mountain car. This would allow for a bigger sample size and a parameter study that could provide more insight about the role of the parameters k and C_{FA} on the performance of k-PI and k-VI.
Additionally, one of the claims in the conclusions was never emphasized in the results: “importantly, the performance of the algorithms was shown to be ‘smooth’  in the parameter k.” This was not completely obvious until I spent some time looking closely at the graph. It eventually became clear, but I think a simpler way to emphasize this is to show a plot of the cumulative reward over the whole training period with the values of k on the x-axis. Based on the top right pane of FIgure 1, this type of plot would show a smooth increase from k=0.99 to k=0.68 followed by a smooth decrease from k=0.68 to k=0.
Finally, I have some questions about some of the choices made in the experiments and results sections:
1. Why choose 50% confidence intervals? 50% confidence intervals with a sample size of 4 in the case of DQN and 5 in the case of TRPO is equivalent to multiplying the standard error by a factor of approximately 0.7, which is narrower than using the standard error on its own. Thus, it seems that some of the conclusions would change based on using a 95% confidence interval compared to a 50% confidence interval in Tables 1 and 2. I insist in showing the performance in a small domain with a simple form of function approximation. This would complement the Atari and MuJoCo experiments by showing improvements in performance with a higher confidence.
2. In remark one, it is pointed out that another target network \tilde Q should be used to obtain \pi_{t-1}, but this was not done to reduce the space complexity of the algorithm. How big were the networks that you used for k-PI DQN? If the network was not prohibitively big, why not implement \tilde Q instead of using an alternative that further deviates from the original k-PI algorithm?
3. Line 19 of Algorithm 5 in Appendix A.1 is supposed to be the off-policy TD(0) update. However, it is not clear how this update is off-policy TD(0) since it based on Q and it does not have any importance sampling to correct for the difference in policies. Am I missing something? It seems that it should be off-policy Sarsa(0), but even then it would still be missing an importance sampling term (see Sutton & Barto, 2018, Equation 7.11, or Algorithm 1 of Precup, Sutton, and Singh, 2000, for more information).
=== Contradictory Claims in the Results ===
There are a few claims that contradict with what is shown in Table 1 and 2.
In the last paragraph of Section 5.1.1 it says that “[the table 1] show[s] that setting N(k) = T leads to a clear degradation of the final training performance on all the domains except Enduro.” This is only true in two out of four games presented in Table 1. In Seaquest the lower confidence bound of the performance of k-PI with k=0.68 is 4643, whereas the upper confidence bound of the performance of k-PI with N(k) = T is 4837; the intervals clearly overlap. Similarly, in the game of Enduro, where k-PI with N(k) = T is said to have better performance, the lower confidence bound of k-PI with N(k) =T is 530, whereas for k-PI with k=0.84 the upper confidence bound is 575; again, the confidence intervals overlap. Hence, neither of these two claims are fully justified, and it is certainly not a “clear degradation of the final training performance.”
Similarly, in Section 5.2.2, k-PI is said to have a better performance than N(k) = T based on the results of Table 2. However, similar calculations show that this is only true for the Ant domain.
===== Minor Comments =====
1. I believe there is a typo in the last column of Table 1, it should be a \kappa instead of a 𝜆.
2. In the second paragraph above Equation 7, the convergence of PI and VI are said to converge to the optimal value with linear rate, but the rate of convergence is O( \gamma^N ), i.e., exponential. Similarly, for the k-PI and k-VI their rate of convergence is O( \ksi ( \kappa )^{N( \kappa )} ), which is also exponential.
===== References =====
Precup, Doina; Sutton, Richard S.; and Singh, Satinder, "Eligibility Traces for Off-Policy Policy Evaluation" (2000).ICML '00 Proceedings of the Seventeenth International Conference on Machine Learning. 80.Retrieved fromhttps://scholarworks.umass.edu/cs_faculty_pubs/80
R. Sutton and A. Barto. Reinforcement learning: An introduction. 2018.
Y. Efroni, G. Dalal, B. Scherrer, and S. Mannor. Beyond the one step greedy approach in reinforcement learning. In Proceedings of the 35th International Conference on Machine Learning, 2018a.
Y. Efroni, G. Dalal, B. Scherrer, and S. Mannor. Multiple-step greedy policies in approximate and online reinforcement learning. In Advances in Neural Information Processing Systems, pp. 5238–5247, 2018b.


Recurrent neural networks (RNN) are powerful time series modeling tools in machine learning. It has been successfully applied in a variety of fields such as natural language processing (Mikolov et al.

The paper attempts to establish the asymptotic accuracy of the "RNN" (but not the RNN models that are well-known in the literature - see the below comments) as a universal functional approximator. It considers a general state space model and uses feedforward neural nets (FNN) to learn the filtering and forecast distributions. Based on the well-known universal approximation property of FNNs, the paper shows that their RNN-based filter can approximate arbitrarily well the optimal filter.
The paper targets an important problem in statistical learning and offers some interesting insights. However, their so-called RNN-based filter formulation is not anywhere close to the usual RNN models, such as Elman's basic RNN model or LSTM, that are currently known in the literature. Hence the paper's title "RNNs are universal filters" and the way it is presenting are confusing. I think what exactly the paper is about is as follows. They consider a general state space model, then use FNNs to approximate the so-called transition equation and observation equation of that SS model. Then, the resulting RNN-based filter is shown to be able to approximate well the optimal filter of the original SS model. I didn't check the proof carefully but I guess it's intuitively straightforward given the available results from the approximation capacity of FNNs.
Therefore, I suggest the authors re-write the paper carefully to reflect better the problem that it actually targets. The property of the RNN-based filter is interesting, but using it is, I believe, very difficult from a practical point of view. It's well-known that doing particle filters is computationally expensive, especially in high dimensions, and the RNN-based filter might have millions of parameters! Could the authors please give some comments/discussion about this issue?
Some minor points:
1)  what exactly does "synthesized" mean?
2) Page 5: "Note that the probability space ... with finite moment..." doesn't read well. What is the moment of probability space?


Compositional structures between parts and objects are inherent in natural scenes. Modeling such compositional hierarchies via unsupervised learning can bring various benefits such as interpretability and transferability, which are important in many downstream tasks. In this paper, we propose the first deep latent variable model, called RICH, for learning Representation of Interpretable Compositional Hierarchies. At the core of RICH is a latent scene graph representation that organizes the entities of a scene into a tree structure according to their compositional relationships. During inference, taking top-down approach, RICH is able to use higher-level representation to guide lower-level decomposition. This avoids the difficult problem of routing between parts and objects that is faced by bottom-up approaches. In experiments on images containing multiple objects with different part compositions, we demonstrate that RICH is able to learn the latent compositional hierarchy and generate imaginary scenes.

Update: I thank the auhors for the detailed response. The updated paper does look more convincing, but my main concern remains - all considered datasets and tasks are synthetic and specifically made for the method. I think experiments on more real data would be crucial to show the potential of the method. Some examples I can imagine would be part segmentation in images or 3D models, or robotic control.
---
The paper proposes an unsupervised approach to learning objects and their parts from images. The method is based on the "Attend, Infer, Repeat" (AIR) line of work and adds a new hierarchy level to the approach, corresponding to object parts. The method is evaluated on two custom synthetic dataset: one composed of simple 2D geometric shapes and another one with 3D geometric shapes. On these datasets the method successfully infers the object-part structure and can parse and reconstruct provided images.
While the general topic of the paper is interesting, I do not think it is fit for publication. First and foremost, the experiments are very incomplete: the method is only evaluated on two custom synthetic datasets and not compared against any baselines or ablated versions of the method. Moreover, the proposed approach seems like a relatively minor modification of SPAIR (Crawford & Pineau).
Pros:
1) The topic is interesting and the method generally makes sense.
2) The method works on the tasks studied in the paper, including relatively challenging scenarios with 3D occlusions or ambiguity in assigning parts to objects.
3) The paper shows generalization to a number of objects different from that seen during training.
Cons:
1) Experiments are limited:
1a) The method is evaluated on custom and relatively toy datasets. It is unclear if it would apply to more practical situations. While I agree that at the high level the question of decomposition of object into parts is interesting, it is still important to connect research in this direction to potential practical applications. Where could the proposed method be used? Perhaps in some control tasks, such as robotics? Could it be applied to more realistic data, such as for instance ShapeNet objects?
1b) There are no comparisons to baselines. One might argue that baselines do not exist since there are no methods addressing the same task. Generally, it is the job of the authors to come up with relevant baselines to show that the proposed model actually improves upon some simpler methods. One variant of obtaining baselines is via ablating the proposed model. Another is by taking prior approaches, for instance object-based ones without the part decomposition, and showing that the part decomposition allows to improve upon those in some sense.
2) The novelty is somewhat limited: the method seems like a relatively straightforward extension of SPAIR by adding another hierarchy layer. It might be sufficient if the experimental results would be strong, but given that they are not, becomes somewhat concerning. If the method does indeed include significant technical innovation, it might be helpful to better highlight it.
3) The related work overview is extremly limited. It is authors' job to provide a comprehensive ovreview of prior literature and I cannot do it for them here, but below are a few papers that come to mind. This is by no means a complete list.
[1] Zhenjia Xu, Zhijian Liu, Chen Sun, Kevin Murphy, William T. Freeman, Joshua B. Tenenbaum, Jiajun Wu. Unsupervised Discovery of Parts, Structure, and Dynamics. ICLR 2019
[2] Shubham Tulsiani, Hao Su, Leonidas J. Guibas, Alexei A. Efros, Jitendra Malik. Learning Shape Abstractions by Assembling Volumetric Primitives. CVPR 2017
[3] Jun Li, Kai Xu, Siddhartha Chaudhuri, Ersin Yumer, Hao Zhang, Leonidas Guibas. GRASS: Generative Recursive Autoencoders for Shape Structures. SIGGRAPH 2017.
[4] Gopal Sharma, Rishabh Goyal, Difan Liu, Evangelos Kalogerakis, Subhransu Maji. CSGNet: Neural Shape Parser for Constructive Solid Geometry. CVPR 2018.
[5] Adam R. Kosiorek, Hyunjik Kim, Ingmar Posner, Yee Whye Teh. Sequential Attend, Infer, Repeat: Generative Modelling of Moving Objects. NeurIPS 2018.
4) Presentation is at times suboptimal:
4a) It would be very helpful to have more visuals and intuitions about the functioning of the method, as opposed to equations. Equations are definitely good to have, but they are not the easiest to parse, especially by those not intimately familiar with this specific line of work.
4b) It is quite unclear to me how and why is the memory used in the model. If this is described in another paper, it would be useful to point there, but still briefly summarize in this paper to make it self-contained.


Learning long-term dependencies is a key long-standing challenge of recurrent neural networks (RNNs). Hierarchical recurrent neural networks (HRNNs) have been considered a promising approach as long-term dependencies are resolved through shortcuts up and down the hierarchy. Yet, the memory requirements of Truncated Backpropagation Through Time (TBPTT) still prevent training them on very long sequences. In this paper, we empirically show that in (deep) HRNNs, propagating gradients back from higher to lower levels can be replaced by locally computable losses, without harming the learning capability of the network, over a wide range of tasks. This decoupling by local losses reduces the memory requirements of training by a factor exponential in the depth of the hierarchy in comparison to standard TBPTT.

Claim: Backpropagation of gradients from a higher to lower level in a HRNN can be removed and replaced with auxiliary losses predicting input tokens at the lower level without affecting performance.
Significance: The significance of the claim hinges on whether HRNNs are more effective than other methods designed to help RNNs capture long-term dependencies (e.g. stacking RNNs or using different architectures). I think the authors could make a more substantive argument why this would be the case in the introduction, but they do a nice job of situating their work in the context of the present literature.
Novelty: The proposed method is not very original, since augmenting RNNs with auxiliary losses in order to better capture long-term dependencies has been used in many previous papers. The authors mention some of these papers in the related work section.
Clarity: The paper's description of the proposed method is well-written. Some parts of the experiment section could be made clearer.
--  I encourage the authors to invent a new acronym to refer to "our model" (perhaps aux-HRNN?). In the description of the mr-HRNN (pg. 5), I find the sentence "trained using only as much memory as our model requires for training" confusing.  I initially thought our model referred to the mr-HRNN in the setence.
-- Training settings (e.g. the number of ticks of the upper RNN) should be described at the beginning of each section.
-- A seeming contradiction is made when discussing the results in 4.3. First, it said that because short term dependencies dominate long term dependencies it is expected that the proposed method will suffer greatly (pg. 6, bottom). In the next paragraph, it is claimed that all three models perform similarly due to the same reason. Which is it?
Supporting evidence: The claim is empirical and the supporting evidence is experimental. As such, I find the comprehensiveness of the experiments wanting. There are several ways the experiments could be improved.
-- Results for each \beta value should be included, to see how placing increasing significance on the auxiliary loss impacts the results.
-- Include all relevant details necessary to reproduce the results, such as the length of training or stopping criterion used.
-- Additional results when varying the number of ticks.
-- More results with deeper hierarchies, since the ability to capture salient information at different levels of coarseness is a key selling point of HRNNs.
-- Results on larger scale tasks besides character level language modelling on Penn TreeBank.
Other comments:
-- In the intro, I think some mention of parallel architectures such as transformers or convolutional architectures is warranted here, since parallelizability of training is a significant reason why these architectures are becoming preferred over RNNs.
-- Citations are mishandled throughout the paper. Citations should be enclosed in parentheses unless used as a subject in the sentence (e.g. "Sordoni et al. make the case that..."). There is no need to refer to a citation twice in a sentence, like you do in "More recently, Koutnik et al. introduced the Clockwork RNN Koutnik et al. (2014)..."
-- I don't understand why the permuted accuracy of the gr-HRNN is so much higher than the non-permuted accuracy. One possible explanation is that the important pixels ended up at the end in each of the three trials, hence the gr-HRNN did not have to remember much information from the past. This should be addressed in the paper.
-- I would welcome some theoretical analysis as to why replacing the gradient path with this particular auxiliary loss does not impact results. I also think some discussion of what this means HRNNs are actually doing might be nice as well.


Due to the wider availability of modern electronic health records (EHR), patient care data is often being stored in the form of time-series. Clustering such timeseries data is crucial for patient phenotyping, anticipating patients' prognoses by identifying "similar" patients, and designing treatment guidelines that are tailored to homogeneous patient subgroups. In this paper, we develop a deep learning approach for clustering time-series data, where each cluster comprises patients who share similar future outcomes of interest (e.g., adverse events, the onset of comorbidities, etc.). The clustering is carried out by using our novel loss functions that encourage each cluster to have homogeneous future outcomes. We adopt actorcritic models to allow "back-propagation" through the sampling process that is required for assigning clusters to time-series inputs. Experiments on two realworld datasets show that our model achieves superior clustering performance over state-of-the-art benchmarks and identifies meaningful clusters that can be translated into actionable information for clinical decision-making.

This paper proposes a temporal clustering algorithm for the medical domain. The main advantage of the proposed method is that it uses supervised information for temporal clustering. The proposed method is evaluated on two real-world datasets and showed improvements against a few other temporal clustering methods.
Detailed Comments:
Methodology:
The actor-critic part of the loss is really just a type of policy gradient. There is no estimation of a value function in Eqn. 7 or anywhere to be found in Appendix B. This is very misleading on the part of the authors because policy grad and AC are very different algorithms. AC type of algorithms provide some variance reduction mechanisms for the classical policy gradient (not done here), but they require additional work to estimate a value function for future trajectories that incur bias. If the authors claim a real AC algorithm for the predictive clustering loss, then some justification for bias-variance tradeoffs should be mentioned instead of attributing it to tuning the hyperparameters between the losses.
Perhaps a bigger issue is that there is a general tradeoff between reconstructing the time-series (the unsupervised learning portion) vs. predictive performance that is common to predictive clustering problems -- it is not addressed here. One can increase the performance in one (in this case prediction accuracy) while sacrificing the other.
In the extreme case, one can just tune the hyperparameter of the loss function to turn this into a pure supervised learning problem while sacrificing the capacity of the embedding representations to have any meaning (e.g., to reconstruct time-series [SOM-VAE] or predict future ones). The authors did not really propose a systematic way to control this tradeoff, nor did they provide some experiments to show how well the embeddings can be used to recover the temporal patterns.
Experiments:
The baseline experiments are rather weak. For example, 3 out of the 4 models (DTW, DCN extensions and SOM-VAE) were all unsupervised learning techniques that are not designed to take into account label information. This is especially true for comparison against SOM-VAE (which the authors called state of the art for the proposed problem). SOM-VAE is actually less intended for prognostication than the likes of Baytas et al. (2017) and Madiraju et al. (2018) and provides less informative baselines. The authors should also provide more details regarding how these ``extensions'' are done for things like DCN and K-means clustering on deep neural networks.


State of the art sequence-to-sequence models for large scale tasks perform a fixed number of computations for each input sequence regardless of whether it is easy or hard to process. In this paper, we train Transformer models which can make output predictions at different stages of the network and we investigate different ways to predict how much computation is required for a particular sequence. Unlike dynamic computation in Universal Transformers, which applies the same set of layers iteratively, we apply different layers at every step to adjust both the amount of computation as well as the model capacity. Experiments on machine translation benchmarks show that this approach can match the accuracy of a baseline Transformer while using only half the number of decoder layers.

[Post-reponse EDIT]: After reading the authors' response, and taking into account the other reviewers' opinions, I am willing to suggest acceptance of this paper on the condition that ***the authors will add the experimental results to address issue #7 and #8 of my original review (as they promised in their rebuttal)***. Before that, I'm keeping the current score, while noting that a more accurate rating should be "5 - Marginally below the acceptance threshold" (which is not available in ICLR this year). I want to thank the authors for addressing the other issues/questions I raised.
----------------------------------------- Original Review Below -----------------------------------------
Summary: The author proposes a new way to improve the supervision of training encoder-decoder Transformers so that it can make flexible, depth-adaptive predictions at inference time. The paper compares multiple dynamic computation schemes to investigate the effectiveness of these different approaches (e.g., at both sequence and token levels). Relatively extensive experiments were conducted in different settings (on a small and a large MT dataset).
While the dynamic halting on Transformers has been previously explored in the Universal Transformers paper, this technique has not been applied to standard (encoder-)decoder Transformers. I think this is an interesting and exciting topic to pursue, as the authors seem to demonstrate (at least on the small IWSLT'14 DE-EN dataset) that we may not need as many layers at the decoding phase as we thought (and thus # of FLOPs).
However, I think the paper can be further improved in terms of both its organization/clarity and its experimental study (see details following).
--------------------------------------------
Questions/comments:
1. Please add a Related Work section. While using dynamic halting on standard Transformers may be novel, general dynamic routing on deep networks (which also aims to reduce the computation) is not. For example, SkipNet [1] uses an input-based dynamic skipping to bypass certain layers of a DNN. It's a good scholarship to give proper credit to prior related work(s).
2. I think Poisson binomial (PB) may be a misnomer for the second qt modeling technique. PB is a probability distribution on the __sum__ of N (unequal) Bernoulli trials [2], which doesn't make sense here. What you are looking for, I think, is a learned "coin flip" at every layer of the network to decide whether to continue to the next layer (i.e., you are looking for the first "head" in the coin flips, instead of the sum of heads). And indeed, in Eq. (9), the definition for qt(n|...) is a "geometric-alike" distribution with success probability \chi_{t,n} at the n^th flip. (Which brings to a side point: for the "otherwise" case in Eq. (9), I think you mean \prod_n (1-\chi_{t, n}); otherwise you can't guarantee \sum_n qt(n|...) = 1 :-) )
3. Why and how did you pick the exit threshold \chi_{t, n} > 0.5 for your experiments? According to your definition of \chi_{t,n} and q_t in Eq. (9), isn't the correct thing to do to sample from \chi_{t, n}? My major concern is that, given different \sigma (which you didn't specify, but I guess is something like a sigmoid), your \chi_{t, n} may land in very different ranges. What if you have the following case: \chi_{t, 1} = 0.501, \chi_{t, 2} = 0.1, \chi_{t,3} = 0.999, \chi_{t,4} = 0.95? Doesn't this mean it may worth it to stop at the 3rd layer instead? (Using the coin flip analogy, having a coin with head probability 0.501 doesn't necessarily mean you "must" get a "head" at the first flip.) Moreover, if I use $\sigma = sigmoid(x)/2 + 0.5$, won't I always get a \chi > 0.5?
4. In the token-specific likelihood-based method, you used an RBF kernel to model the influence of a time step t on its neighboring time steps. Two questions: 1) When you do \sum_{t'}, did you also include t'<t? 2) Have you tried any other kernels, and how does this choice affect the performance?
5. The aligned training is actually a type of intermediate auxiliary loss (or deep supervision, as the computer vision community probably more often calls it), which makes it not surprising that the unaligned Transformer could perform slightly better than the baseline. I find it interesting that the mixed training strategy doesn't work.
6. Why did you use different training schemes for WMT'14 and IWSLT'14 (e.g., freezing the parameters, etc.)?
7. One of the major concerns I have is that the empirical results don't seem that impressive. First, to demonstrate the effectiveness of the proposed adaptive depth estimation methodology, besides comparing to the baseline (black line) in Figures 3 & 4, you should also compare with the blue line (which adds these auxiliary losses to the baseline, but doesn't use the adaptive strategy that involves qt(n|...) at all) for fairness. From Figures 3 & 4, it seems that the adaptive-depth predictions are usually in the close neighborhood of the blue lines when at the same value of AE (instead of substantial, consistent improvement). In addition, while Tok-LL Poisson did well in Figure 4(b), it didn't seem to make a difference in Figure 4(a), which is where one is supposed to tune the hyperparameters on (such as \lambda, \sigma; see #8 below). This makes me a bit dubious about how much help the adaptive module brought, compared to just using the aligned model.
8. For the WMT'14 experiments (Figure 4), why is there only one run of the "Tok-LL Poisson"? My understanding is that, if you are to use the proposed approach on a new dataset, you would want to try different settings of (\sigma, \lambda) on the validation set and pick the best one to use for testing. In Figure 4(a), for instance, for Seq-LL I would probably pick the (\sigma, \lambda) setting corresponding to the top-right yellow square--- which turns out to be slightly worse than the blue line on the test set (Figure 4(b)). It'd be useful to plot more "red triangles" in Figure 4 to evaluate how much the adaptive-depth methodology contributes to the performance.
========================
Some minor errors that don't have much impact on the score:
9. The beginning sentence of the 4th paragraph of Section 1 is a bit strange (grammatically).
10. Not all equations are numbered! See section 2.
11. In the first equation of Section 2, do you mean h_{\leq t}^{n-1} rather than h_{< t}^{n-1}? I think h_t from the previous layer is also used?
12. In the 3rd equation of Eq. (12), you should have m somewhere within the summation.
13. Equation (7) missing a right parenthesis.
14. In Eq. (8) W_h is a matrix, whereas in Eq. (9) W_h is a vector (if I'm not mistaken)? Maybe use a lowercase w.
15. Inconsistent notations. For instance, as I described in #3 above, you didn't say in the paper what \sigma means in Eq. (9), but later re-used the same letter for a different meaning in Eq. (11) for the RBF kernel. Another example is the usage of \theta_n in the "Confidence thresholding" paragraph for the threshold value; you used the same letter again in the "Gradient scaling" section of the appendix but with a different meaning (learnable parameters).
16. In the last sentence of the Likelihood-based token specific depth, I'd suggest \sigma \rightarrow 0 instead of \sigma=0, which would otherwise make \frac{...}{\sigma} in Eq. (11) undefined.
17. I don't think you specified the \theta_n used in your experiments (and how you tune them).
18. What is the FS in Appendix B? Also, for "ffn" in Table 4, it's best to write its full name before referring to it with acronyms.
19. Just curious: how did you implement the gradient scaling described in Appendix A (e.g., ensuring the \gamma_n only applies to block n, but not blocks < n)?
========================
I think the current shape of the paper is marginally below the acceptance threshold. (Note that while the rating is "3 - Weak reject", I don't mean the score, but only the "weak reject" part. I'm still excited about the idea of applying dynamic computation in Transformer.) But I'm happy to consider adjusting the score if my concerns above can be satisfactorily addressed.
[1] https://arxiv.org/abs/1711.09485
[2] https://en.wikipedia.org/wiki/Poisson_binomial_distribution


The classification of images taken in special imaging environments except air is the first challenge in extending the applications of deep learning. We report on an UW-Net (Underwater Network), a new convolutional neural network (CNN) based network for underwater image classification. In this model, we simulate the visual correlation of background attention with image understanding for special environments, such as fog and underwater by constructing an inception-attention (I-A) module. The experimental results demonstrate that the proposed UW-Net achieves an accuracy of 99.3% on underwater image classification, which is significantly better than other image classification networks, such as AlexNet, InceptionV3, ResNet and Se-ResNet. Moreover, we demonstrate the proposed I-A module can be used to boost the performance of the existing object recognition networks. By substituting the inception module with the I-A module, the Inception-ResnetV2 network achieves a 10.7% top1 error rate and a 0% top5 error rate on the subset of ILSVRC-2012, which further illustrates the function of the background attention in the image classifications.

This paper shows a method to classify between underwater and non-underwater images (binary). For this, they proposed a new module in a CNN network, called 'Inception-Attention' module. It combines the inception module (multiple sizes of kernels) with an attention module (learn jointly some masks in order to focus the network to part(s) on the image). Also, their network proposes 2 classifications, by separating the network in 2 branches at the end - with the goal of less overfitting. Their method was compared against other state-of-the-art networks, usually designed for more complex tasks (multiclass classification). Lastly, they showed how their new 'IA module' could be useful in other tasks/networks, by replacing the standard inception network with theirs.
Overall, I found the paper confusing: while some technical aspects are interesting, I still don't get the motivation of this new network for this task. Moreover, I did not understand the goal of the paper until the middle of the paper: I thought it was to do a multi-class classification from underwater images, while it is just a binary classification of underwater/ non-underwater images. I still think there are some valuable arguments, and the last part, 4.4, showing the usefulness of the method for more complex classifications, is to me the most interesting part. I could change my grade if a better motivation or a reorienting of the paper was made.
Remarks/ questions:
- The writing should be improved as it is even hard to understand some sentences. Some particular help will be given below.
- My main concern is: have you tested easier classifications methods? It seems that classifying underwater vs. not underwater images would be easy. In fact, even the mean color should be identifiable... do you have a simple baseline to compare to? You are saying that one of the main problems is that salient objects are less visible underwater, there is blurring, ect. : these are for me all arguments why it would be easy to detect underwater images, because standard convolutions will behave differently.
- Have you tested a simpler CNN? You are right in saying that the state-of-the art methods for multi-class classification are too large for this task. So why to you want to complexify it, and why not use a simpler network?
- There is no related work on background classification (or 'context' classification); but I am sure that there might be works on this. It would be more interesting than the general image classification models.
- in 3.1 and 3.2, you are saying 2 opposite things. a) 'the features extracted by a conv. kernel with large size tend to describe the global information [..] and will result in a certain degree of waste of computational resource' and b) 'we adopt convolution kernels and the average pooling to reduce the impact of local features of the image'. Do you want or not want large kernels?
- 'Moreover, human recognition of underwater images is often based on the background..'. Why? Are you sure? I think it is more based on the color of the image, the texture, ect.
- Auxiliary classification branch: If you have overfitting problems; why don't you use standard methods for treating overfitting (the first one is to have a smaller and simpler network..)?
-I don't see a real improvement between the auxiliary classification branch and no auxiliary classifier in the Figure 4. Such a small difference, compared to the large oscillations, is not enough. Why don't you use different runs of your model with different initializations, in order to take the mean?
- 3.3: not clear: 'the channel of the input image is increased to 32'. I think you want to talk about the number of channels, and not 'the channel'. Same error few lines below.
- How did you select the 5000 non-underwater images? Randomly? Or just outside views?
- 4.2: I don't see a validation set. It is important to have 3 sets, one training, one validation, one test. All the tuning of parameters/architectures must be done on the validation, with the test kept hidden until the publication.
- 4.3: results on the training set should not even be shown...if you want, you can show the validation error, but not the training. An accuracy of 99.3% means that it is an easy task. Yes, it's true that your method works, but i) since you are not using a validation set, you could have just tuned your model until you have a good accuracy on the test set; ii) it might be better to prove the usefulness of your method on a more complex task.
- 4.4 : this is the important part of the paper I think, you should develop it :)!