Artificial neural networks (ANNs) lack in biological plausibility, chiefly because backpropagation requires a variant of plasticity (precise changes of the synaptic weights informed by neural events that occur downstream in the neural circuit) that is profoundly incompatible with the current understanding of the animal brain. Here we propose that backpropagation can happen in evolutionary time, instead of lifetime, in what we call neural net evolution (NNE). In NNE the weights of the links of the neural net are sparse linear functions of the animal's genes, where each gene has two alleles, 0 and 1. In each generation, a population is generated at random based on current allele frequencies, and it is tested in the learning task. The relative performance of the two alleles of each gene over the whole population is determined, and the allele frequencies are updated via the standard population genetics equations for the weak selection regime. We prove that, under assumptions, NNE succeeds in learning simple labeling functions with high probability, and with polynomially many generations and individuals per generation. We test the NNE concept, with only one hidden layer, on MNIST with encouraging results. Finally, we explore a further version of biologically plausible ANNs inspired by the recent discovery in animals of dopaminergic plasticity: the increase of the strength of a synapse that fired if dopamine was released soon after the firing.

First of all, I must confess that my knowledge is quite limited to read this paper. Perhap the authors present something that I can not catch up at the present.
I conjecture the paper would like to bring the evolution in genetics and perhap brain cirecuits as well to define a novel neural net model, called NNE by the authors.
The paper is somewhat cumbersome in the introduction that makes the reader (here is myself) can not understand the main idea. At first the authors introduce about evolution in genetics and genomics that is a bit different from what I known. Then the authors claim that they can show that the brain circuits can evolved in their model.
There are so many mistake and/or typos in sentences and in mathematical formulation. These make me can not finish reading the paper. I have to stop reading at the end of Section 2.
Here are my concerns and questions:
1). What is "STDP" in the 2nd papragraph in Introduction?
2). in the 3rd papragraph in Introduction: "Suppose that the brain circuitry for a particular classiﬁcation task, such as “food/not food”,is encoded in the animal’s genes, assuming each gene to have two alleles 0 and 1". This is realistics, the allels in animal is 0 1 or 2 if encoded.
3). The authors use the words "gene, genotype, phenotype" in a special way that is different to what I known in genomics (in GWAS).
4).  in the 3rd papragraph in Introduction: "At each generation, a gene is an independent binary variable with ﬁxed probability of 1". What do you mean by fixed probability of 1? I can not understand in anysense that I know.
5). In Section 2, What is n? the authors start the mathematical formulation but I can not find out what is n? Is it the sample size?
6). In Section 2, paragraph 2, you define y~ \mathcal{D}, BUT then in all formulas later you denote y ~ D. What is D??? I can not understand.
7).  In Section 2, paragraph 2, you define a label of y as \ell(y), BUT then in the 1st sentence of the 3rd paragraph in Section 2 you wrote    L(NNE_x(t), l(y)). What is l(.) here ???
8). In the equations (1) and (2), what is p^t  ???  You have NOT defined it.
9). Right after equations (1) and (2), What is \epsilon ???? Can NOt understand.
10). The sentence right after the equation (3): "This is the standard update rule in population genetics under the weak selection assumption." This is NOT trivial to me, and even the machine learning comunity, we do not know this rule, it is not obvious. PLEASE provide exact reference.
11). the first equation in the PROOF of LEMMA 1 wrong  \mathcal{L} (p^t)  should be equal to E_{x~p^t}  NOT p.
12). in the PROOF of Theorem 1, I can NOT find out where \gamma has been defined?
13). What is d  in Theorem 2? is it the dimension? I make too many guesses !!!


We explore the idea of compositional set embeddings that can be used to infer not just a single class, but the set of classes associated with the input data (e.g., image, video, audio signal). This can be useful, for example, in multi-object detection in images, or multi-speaker diarization (one-shot learning) in audio. In particular, we devise and implement two novel models consisting of (1) an embedding function f trained jointly with a "composite" function g that computes set union operations between the classes encoded in two embedding vectors; and (2) embedding f trained jointly with a "query" function h that computes whether the classes encoded in one embedding subsume the classes encoded in another embedding. In contrast to prior work, these models must both perceive the classes associated with the input examples, and also encode the relationships between different class label sets. In experiments conducted on simulated data, OmniGlot, and COCO datasets, the proposed composite embedding models outperform baselines based on traditional embedding approaches.

This paper describes a way to train functions that are able to represent the union of classes as well as to query if the classes in an image subsume the classes in another image. This is done throughly jointly training embedding functions, a set union function and a query function. The paper reads well.
While the approach is reasonable, the experiments seem to be quite incomplete and no explanation is given why a trivial solution cannot be used instead of the learnt functions.
The paper argues for learning a set union function however much of the evaluation focuses on quite small sets of 2 or 3 items. On the evaluation that utilises larger sets, e.g. COCO, there isn't any analysis of how performance of the technique scales with the size of the set since that would be one of the defining characteristics of a set union function. The COCO experiment is also lacking in detail, for example, how many items are there in the positive and negative sets and how the test set is balanced. Finally, it seems that f, g and h could be trivial non-learnt functions. For example, f could be a function that maps an image to a binary representation of its classes (this could be a typical ResNet image classifier), g could be a function that does a binary OR of its two arguments and h could be a function that uses a binary AND and equality test on its two arguments. In this case, g and h don't need to be learnt at all. This may not be possible in the COCO experiment where the individual labels are not known but it seems quite unrealistic to have a dataset where only pairwise subset relationships are known.
It also seems that the f is always different between that used with g and that used with h, is this the case? SimRef also doesn't do data augmentation but there's no explanation why it is done for the proposed method and not for this baseline. The MF baseline in experiment 1 seems to be a straw man especially since the baselines in experiment 2 are much stronger.
================================================================================
Update after rebuttal:
Thanks for answering my questions and performing the additional experiments with a ResNet baseline and performing an additional analysis based on the number of subclasses in figure 5. I think these provide a substantially better analysis of the algorithm so I've increased my score correspondingly. For the final paper, I think it would be good to add TradEmb/ResNet to figure 5 as well to understand how those methods scale worse/better with the number of subclasses.


State-of-the-art results in imitation learning are currently held by adversarial methods that iteratively estimate the divergence between student and expert policies and then minimize this divergence to bring the imitation policy closer to expert behavior. Analogous techniques for imitation learning from observations alone (without expert action labels), however, have not enjoyed the same ubiquitous successes. Recent work in adversarial methods for generative models has shown that the measure used to judge the discrepancy between real and synthetic samples is an algorithmic design choice, and that different choices can result in significant differences in model performance. Choices including Wasserstein distance and various f -divergences have already been explored in the adversarial networks literature, while more recently the latter class has been investigated for imitation learning . Unfortunately, we find that in practice this existing imitation-learning framework for using f -divergences suffers from numerical instabilities stemming from the combination of function approximation and policygradient reinforcement learning. In this work, we alleviate these challenges and offer a reparameterization of adversarial imitation learning as f -divergence minimization before further extending the framework to handle the problem of imitation from observations only. Empirically, we demonstrate that our design choices for coupling imitation learning and f -divergences are critical to recovering successful imitation policies. Moreover, we find that with the appropriate choice of fdivergence, we can obtain imitation-from-observation algorithms that outperform baseline approaches and more closely match expert performance in continouscontrol tasks with low-dimensional observation spaces. With high-dimensional observations, we still observe a significant gap with and without action labels, offering an interesting avenue for future work.

* Summary:
The paper proposes an IL method based on the f-divergence. Specifically, the paper extends f-VIM (Ke et al., 2019), which uses the f-divergence for IL, by using a sigmoid function for discriminator output’s activation function. This choice of activation function yields an alternative objective function, where the reward function for an RL agent does not directly depend on the convex conjugate of the function f; the paper claims that this independency improves stability of policy learning. This proposed method is named f-VIM-sigmoid. The paper extends f-VIM-sigmoid to the setting of IL with observation and proposes f-VIMO-sigmoid. Experiments on Mujoco locomotion tasks show that f-VIM-sigmoid and f-VIMO-sigmoid perform better than existing methods.
* Rating:
The paper proposes a simple but interesting approach to improve stability of adversarial IL. However, the paper has issues regarding baseline methods, motivation, supports of the claim, and experiments (see below). These issues should be addressed. At the present, I vote for rejection.
* Major comments:
- Discussion and comparing against a simple baseline method based on swapping distributions:
To make the reward function be independent of the convex conjugate f*, it is possible to simply swapping the distributions P and Q in the definition of the f-divergence. More specifically, instead of minimizing D_f(P||Q), we can minimize D_f(Q||P), where P is a data distribution and Q is a generator. In this case, pi* and pi_theta in Eq. (7) swap, and the RL agent minimizes the cost function g_f(V_w(s,a)). This cost function does not directly depend on f*, similarly to the reward function r(V_w(s,a)) in Eq. (8). This swapping is simpler and more flexible than re-parameterizing, while achieving the same goal as f-VIM-sigmoid. This swapping method should be discussed and compared against the proposed methods.
- Need stronger baseline methods for ILfO:
The paper should evaluate f-VIMO-sigmoid against stronger baselines, e.g., forward adversarial IL (Sun et al., 2019) which outperforms GAIL-based methods in the ILfO setting.
[1] Wen Sun, Anirudh Vemula, Byron Boots, and J Andrew Bagnell. Provably efficient imitation learning from observation alone. ICML, 2019.
- Using the f-divergence for ILfO is not well motivated:
The paper does not provide good motivations for using f-divergence in ILfO. This makes the paper quite difficult to follow, since there is no connection between f-divergence and ILfO.
- The experiments focus on evaluating existing methods rather than the proposed methods:
Specifically, the proposed methods are evaluated with only one choice of the divergence (TV) in Figure 2. Meanwhile, most of Section 6 and results (Figure 3 and 4, and additional results in the appendix) focus on evaluating the existing methods (f-VIM and f-VIMO) with different choices of divergence.
- The experiments in Figure 2 do not support the claim regarding stability:
The paper claims to improve stability of IL by using the proposed re-parameterization. However, the experimental results do not support this claim, and the questions asked in Section 5 are not related to this claimed. Instead, it seems that re-parameterization helps avoiding local optima (possibly due to a biased reward function, see below), while stability is improved by regularizing the discriminator. I could not see how the re-parameterization improves the policy stability as claimed.
- The experiments in Figure 2 seem unfair, since TV-VIM-sigmoid incorporates priors about survival bonuses:
Specifically, TV-VIM-sigmoid uses sigmoid which yields strictly positive rewards, while TV-VIM uses tanh which yields positive and negative rewards. As discussed by Kostrikov et al., 2019, using strictly positive rewards incorporate strong priors about the survival bonuses, which exist in the locomotion task used in the experiments. Therefore, TV-VIM-sigmoid uses strong priors while TV-VIM does not. In order to make the comparison fairer, I suggest the authors to evaluate TV-VIM with sigmoid reward output, or include environments that do not have survival bonuses.
[2] Ilya Kostrikov, Kumar Krishna Agrawal, Debidatta Dwibedi, Sergey Levine, and Jonathan Tompson. Discriminator-actor-critic: Addressing sample inefficiency and reward bias in adversarial imitation learning. ICLR, 2019
* Minor comments:
- The abstract is long and could be shorten.
- Figures are too small and difficult to see, especially the legends.
- Table 1 should describe the form of f in addition to its conjugate.
- The title of the Algorithm 1 should be f-VIMO-sigmoid instead of f-VIMO.
** Update after response.
I read the response. I thank the authors for clarifying the claims as well as the new experiments with the swap formulation. However, improving clarity of the claims is considered a major revision. I still keep the vote of rejection.
Regarding reward bias. As the authors acknowledge, the improvement achieved by using reparameterization+sigmoid can be explained by two equally-plausible reasons: 1) reparameterization+sigmoid improves stability (as claimed) and 2) sigmoid gives biased rewards. The issue here is that we do not know which is the actual reason, given the current experiments in the paper. As I commented, evaluating TV-VIM with sigmoid but without reparameterization will help address this issue.


Infinite horizon off-policy policy evaluation is a highly challenging task due to the excessively large variance of typical importance sampling (IS) estimators. Recently, Liu et al. (2018a) proposed an approach that substantially reduces the variance of infinite-horizon off-policy evaluation by estimating the stationary density ratio, but at the cost of introducing potentially high biases due to the error in density ratio estimation. In this paper, we develop a bias-reduced augmentation of their method, which can take advantage of a learned value function to improve accuracy. Our method is doubly robust in that the bias vanishes when either the density ratio or value function estimation is perfect. In general, when either of them is accurate, the bias can also be reduced. Both theoretical and empirical results show that our method yields significant advantages over previous methods.

Comments :
This paper provides an approach for reducing bias in long horizon off-policy evaluation (OPE) problems, extending recent work from Liu et al., 2018 that estimates the ratio of the stationary state distributions in off-policy evaluation for reducing variance. The paper provides a doubly robust method for reducing bias, since it requires separately estimating a value function. The key idea of the paper is to provide low variance, low bias OPE since their approach relies on accurately estimating the state distribution ratio and also the estimation of the value function.
The paper provides a follow up on Liu et al., 2018 and other recent works in off-policy learning that tries to estimate the state distribution ratio directly, but can introduce high inaccuracy if the obtained ratio estimates are inaccurate. In line of that, this approach seems useful as it tries to reduce the error from inaccurate ratio estimates by incorporating prior works with an additional value function estimation.
Furthermore, the paper introduces a new perspective to doubly robust estimation that tries to reduce bias, instead of variance in previously known OPE literature (Jiang et al., 2016; Thomas et al., 2016). It is interesting to see how doubly robust can be related to primal-dual relations and the connections between these approaches, which is a novel contribution to the best of my knowledge.
- The fundamental connection comes from observing OPE withdoubly robust estimator that estimates \hat{V} and equation 6 that incorporates the ratio of the state density. This is clearly written in equations 7 and 8, that are two ways of evaluating the value function with off-policy samples known in the literature.
- It uses the property of the Bellman recursive expression for the estimated value function of doubly robust OPE estimator and uses it in the OPE with state density ratio, leading to equation 9, and obtaining the bridge estimator that now relies on both estimates of value function \hat{V} and the state density ratio. Although initially this does not seem useful, but the authors clarify this and how a bias reducting method can be achieved as in equation 11.
- It is a nice and elegant trick, exploiting the connections between OPE estimators leading to a bias reduction method that seems quite interesting.
- The most interesting part of the paper comes from section 4 that exploits the connection between doubly robust approaches with Lagrangian duality, and that their approach is equivalent to a primal dual formulation of policy evaluation. This stands out in itself as a novel contribution of the paper. Equations 14 and 15 best writes down the connections, as to how policy evaluation can be formualted as a Lagrangian function for optimization.
- Although the authors point out how equation 15 is equivalent to equation 11 - this does not seem straight forward at first unless carefully looked at. I would encourate the authors to perhaps add more clarity that exploits this connection, to make the paper more self-contained.
- In terms of experiments, the paper compares the doubly robust approach with previous works that estimates the density ratio, along with other baseline comparisons such as weighted DR. In both the two evaluated problems, their approach seems useful in terms of obtaining better accuracy (lower MSE).
- However, I am not sure to what extent this approach can be scaled to more complicated tasks for OPE? Are there any example domains where the proposed method fails, or is difficult to scale up to more complicated tasks?
- The current set of experiment results seems adequate, given the theoretical contribution and that most OPE papers evaluate on such domains. But overall, it might be useful to analyse the significance of this approach when scaling to more complicated domains.
Overall, I think the paper has a neat and elegant theoretical contribution, exploiting the connection of OPE with primal-dual frameworks that seems quite novel to me. Experimental set of results are properly presented too showing significance of the approach compared to previously known baselines. I think such papers exploting connections with other literature is useful for the community in general, and this paper has significantly novel theoretical contribution. Hence, I would recommend for acceptance of this paper.


Overparameterized neural networks trained to minimize average loss can be highly accurate on average on an i.i.d. test set, yet consistently fail on atypical groups of the data (e.g., by learning spurious correlations that hold on average but not in such groups). Distributionally robust optimization (DRO) provides an approach for learning models that instead minimize worst-case training loss over a set of pre-defined groups. We find that naively applying group DRO to overparameterized neural networks fails: these models can perfectly fit the training data, and any model with vanishing average training loss will also already have vanishing worst-case training loss. Instead, their poor worst-case performance arises from poor generalization on some groups. By coupling group DRO models with increased regularization-stronger-than-typical 2 regularization or early stopping-we achieve substantially higher worst-group accuracies, with 10-40 percentage point improvements over standard models on a natural language inference task and two image tasks, while maintaining high average accuracies. Our results suggest that regularization is critical for worst-group generalization in the overparameterized regime, even if it is not needed for average generalization. Finally, we introduce and provide convergence guarantees for a stochastic optimizer for this group DRO setting, underpinning the empirical study above.

This paper describes a method of training neural networks to be robust to a worse case mixture of a set of predefined example attributes. This is done with a loss in accuracy in the average case but improvements in the worse case. The proposed algorithm is relatively simple and convergence rates are also given for this new algorithm.
Building neural networks that perform well in the face of group-level worse case test-set distributions is a very important problem particularly in areas such as health and safety-critical applications as past work points out. This paper shows good results in the worse case and additionally shows that the common technique of importance reweighting cannot arrive at the same solution. The convergence analyses also yield additional insight into this new algorithm. The paper is well written and relatively easy to understand with good details on the experimental setup. The algorithm has a downside in that the groups must be known a priori, is it possible for these groups to be learnt? Also, can using a hinge loss also improve robustness to the worse case examples?
However, there are some unanswered questions in the paper. What is the effect on the training time of this algorithm? Is it just the time for an additional forward prop? What is the effect on the worse-case examples of weight decay on the Bert model? Even though it hurts the average performance does it improve the worse case at all? In the last line of algorithm 1 why is q_G used instead of q_g?
Other comments:
At the bottom of P5: The ordering of 93.4% and 97.1% seem to be reversed.
Above eq. 5 , \delta_g seems to be overloaded. In the paragraph, it first refers to the generalization gap and then later to a heuristic.
Table 2: The drop in average accuracy for waterbirds does not seem 'small'.
Bottom of P8: 'background is more unique', it seems this is supposed to mean the background appears less often?
======================================================================================================
Update after rebuttal:
Thanks for the detailed answers to my comments and the additional experiments done with a hinge loss. I will keep my rating.


The validation curve is widely used for model selection and hyper-parameter search with the curve usually summarized over all the training tasks. However, this summarization tends to lose the intricacies of the per-task curves and it isn't able to reflect if all the tasks are at their validation optimum even if the summarized curve might be. In this work, we explore this loss of information, how it affects the model at testing and how to detect it using interval plots. We propose two techniques as a proof-of-concept of the potential gain in the test performance when per-task validation curves are accounted for. Our experiments on three large datasets show up to a 2.5% increase (averaged over multiple trials) in the test accuracy rate when model selection uses the per-task validation maximums instead of the summarized validation maximum. This potential increase is not a result of any modification to the model but rather at what point of training the weights were selected from. This presents an exciting direction for new training and model selection techniques that rely on more than just averaged metrics.

The paper examines the common practice of performing model selection by choosing the model that maximizes validation accuracy. In a setting where there are multiple tasks, the average validation error hides performance on individual tasks, which may be relevant. The paper casts multi-class image classification as a multi-task problem, where identifying each different class is a different task.
At different points of training, a model's performance on a task will vary. To make it easier to examine this, they propose displaying an interval plot. The validation accuracy for a single task will be largest at a specific epoch. For each single task, display the epochs where performance is within some threshold of the best validation-set accuracy for that task. Plots on CIFAR-100 demonstrate how some tasks are learned quickly then forgotten, while other tasks are only learned late.
To examine the difference in performance, they compare the single best CIFAR-100 model, to the classifier defined by "for class k, forward input to the model with best validation accuracy for class k". This gives about +2.5% in Top-1 performance, a similar story appears on the other 2 datasets they try (Tiny ImageNet, PadChest). This requires N models, where N is the number of classes. They also examine a clustering approach, where they cluster the N models into K models (using K-means), then forward class k to the best model out of those K models.
Overall, it isn't an especially large contribution but I felt it had some interesting points. Some comments:
* Citation list feels short. The catastrophic forgetting literature seems relevant here. Rich Caruana's multitask learning thesis (Caruana et al 1997) is also relevant.
* It's unclear how the K-means is carried out. I assume the features used for each model checkpoint are tied to its performance on individual task but it's never spell out what distance metric is used in the K-means.
* It feels strange to have no comparisons to ensemble-based baselines. The baseline here would be, for some parameter K, run K independent training runs, take the top average validation accuracy from each run, and average them together, then compare this to the K models found from K-means clustering. For small enough K (like 5 or 10) this seems like a feasible experiment, computation-wise.
* In a similar vein, I wonder how this compares to the Snapshot Ensembles paper, which has a similar guarantee of doing 1 training run and giving M models for an ensemble. Is the gain from ensembling, or from directly examining which models are better at which classes?
* A natural follow-up here is to use model distillation to distill the N best models for each individual class into a single model checkpoint - does this give us a better single model checkpoint?
As-is, I give this paper a weak reject, but would be willing to increase if some of these experiments were tried.
Edit: having read the other reviews and author comments, I still maintain a weak reject rating. I believe this paper needs further work.


This paper presents a novel physics-inspired deep learning approach for point cloud processing motivated by the natural flow phenomena in fluid mechanics. Our learning architecture jointly defines data in an Eulerian world space, using a static background grid, and a Lagrangian material space, using moving particles. By introducing this Eulerian-Lagrangian representation, we are able to naturally evolve and accumulate particle features using flow velocities generated from a generalized, high-dimensional force field. We demonstrate the efficacy of this system by solving various point cloud classification and segmentation problems with state-of-the-art performance. The entire geometric reservoir and data flow mimics the pipeline of the classic PIC/FLIP scheme in modeling natural flow, bridging the disciplines of geometric machine learning and physical simulation.

The paper is about using classical PIC/FLIP scheme in Computational Fluid Dynamics for solving the learning problem of 3D object detection and segmentation. In general, there are extrinsic CNNs like the Vox net etc. which look for global features which the authors refer to as Eulerian formulation of the data representation, and there are intrinsic CNNs like the GCN(graph convolutions), Point nets etc. which look for localized neighborhood information which the authors refer to as Lagrangian formulation. The authors acknowledge that hybridizing the extrinsic CNNs and intrinsic CNNs is not new and several works are cited. The key contribution is to look at this problem from the perspective of PIC/FLIP scheme which has been used in CFD for decades.
The idea is very nice, well describes and quite novel in my opinion. I really liked the adoption of classical CFD approaches in learning. This provides a very interesting perspective to 3D deep learning.
However, the papers struggles to demonstrate why the 3D deep learning community would adopt this approach. The results are not that conclusive. The algorithm works (understood well from the ablation study). However, the performance of the proposed approach is at best comparable to some of the state-of-the-art methods such as PointCNN or SE-Net. The authors need to clarify what potential advantages could there be other than accuracy (if any) such that the community uses the proposed method.
Also, the grids used in the study are too low to make any conclusive remarks on what happens when dealing with higher resolutions of grid. The authors themselves acknowledge the limitation of not being able to go higher in resolution of grid. Interestingly, such limitations of CFD has recently motivated the community to explore deep learning based fast and agile surrogates for computationally tractable approaches.
Some of the new works in 3D object recognition and segmentation such as  Deep SDF(https://arxiv.org/abs/1901.05103), AtlasNet(https://arxiv.org/abs/1802.05384),  Deep Level Sets (https://arxiv.org/abs/1901.06802), occupancy networks (https://arxiv.org/pdf/1812.03828v1.pdf), http://openaccess.thecvf.com/content_cvpr_2018/html/Yu_PU-Net_Point_Cloud_CVPR_2018_paper.html, Adaptive O-CNN (https://dl.acm.org/citation.cfm?id=3275050), https://arxiv.org/abs/1805.12254, 3D Point Capsule Networks, http://t.cvlibs.net/publications/Niemeyer2019ICCV.pdf etc. can be compared with or at least contrasted in the related works.
In summary, I really liked the algorithmic idea, but skeptical about its practical relevance from the results.


Learning useful and reusable skill, or sub-task primitives, is a long-standing problem in sensorimotor control. This is challenging because it's hard to define what constitutes a useful skill. Instead of direct manual supervision which is tedious and prone to bias, in this work, our goal is to extract reusable skills from a collection of human demonstrations collected directly for several end-tasks. We propose a weakly-supervised approach for trajectory segmentation following the classic work on multiple instance learning. Our approach is end-to-end trainable, works directly from high-dimensional input (e.g., images) and only requires the knowledge of what skill primitives are present at training, without any need of segmentation or ordering of primitives. We evaluate our approach via rigorous experimentation across four environments ranging from simulation to real world robots, procedurally generated to human collected demonstrations and discrete to continuous action space. Finally, we leverage the generated skill segmentation to demonstrate preliminary evidence of zero-shot transfer to new combinations of skills. Result videos at https: //sites.google.com/view/trajectory-segmentation/.

The paper presents a weakly supervised method for segmentation of trajectories into sub-skills inspired by multi-instance learning (MIL) in image classification by Andrews et al. (2002). This is done via training a classifier to label each observation per time-step with the probability of skills corresponding to that observation. These predictions are then accumulated throughout the trajectory to compute the probability of the skill in that trajectory. There is only a trajectory level supervision provided which specifies which skills are present with no specification of the order in which they appear. They empirically show that their model can achieve decent skill level classification scores on multiple environments provided that there is a large variety of demonstrations provided.
In its current form, I would recommend this paper to be rejected because 1) the framing and motivation of the paper does not correspond to the results and experiments reported and hence seems misleading 2) the paper is limited in scope 3) further experiments and comparisons to relevant baselines are needed to support the claims made in the paper.
The problem that they are proposing is interesting and is of clear value, however, the paper falls short in addressing this problem. In particular, the paper is framed as a way to learn re-usable useful sub-skills that can help generalize to new situations in control. However, the method presented provides a per time-step labelling of each observation with the associated most likely sub-skill. Having the per time-step labelling of the trajectory provides no indication that the data could be useful for learning reusable skills for downstream tasks later. One very basic experiment could be to train a behaviour cloning (BC) agent on the observation-action pairs conditioned on the sub-skill (or a separate network per sub-skill) and show that the learned policies can be leveraged in solving the tasks presented. For instance, one can train a meta-controller that can switch between these learned sub-skills to successfully perform the task. Training such sub-skills from weakly supervised skill annotations has been successfully done by Shiarlis et al. (2018). It should be noted that in their setting, the annotations are ordered which simplifies the problem to some extent.
Ignoring the motivation and focusing only on the problem setting addressed which is annotation of trajectories with skill labels, the experiments seem very restrictive to me. To my understanding there are at most 4-6 primitive skills present in each environment investigated. Looking at the videos linked, it feels like there is some overfitting to the trajectories provided as for example in the case of the video with the red bowl (Reach and Stir inside Cup), the classifier is predicting 'Stirring' from the first frames where there is not much information present in the scene and 'reach to object' could also be plausible for example. It would have been nice to see the logits of predictions per classes for these examples to understand the confidence of the model particularly when some of the skills could be equally likely (e.g. first few frames).
I found the experimental setup unclear. Particularly, details regarding the task setup, how many demonstrations are needed per task/skill, architectural choices and hyper-parameters are lacking and makes experiments hard to follow and understand. I would have also liked to see more analysis regarding the segmented trajectories, particularly how consistent these predictions are through time. To my understanding there is nothing that keeps a skill annotation consistent over some period of time (e.g. the model could keep switching its prediction every time-step). This would be quite unsatisfactory if one would like to use this to actually segment sub-trajectories associated with a given skill that could be useful for training policies. They report applying Gaussian smoothing to filter out noisy predictions but there are no details provided on how this is tuned and how much this affects the quality of segmentations.
Overall, the paper seems to me very limited in its scope and experimental results. The claims made throughout the paper are not supported empirically or theoretically. There is not enough evidence for me to assess the significance of the proposed method and know whether this is indeed useful in practice.


Deep neural networks (DNNs) have achieved great success in various applications due to their strong expressive power. However, recent studies have shown that DNNs are vulnerable to adversarial examples which are manipulated instances targeting to mislead DNNs to make incorrect predictions. Currently, most such adversarial examples try to guarantee "subtle perturbation" by limiting the L p norm of the perturbation. In this paper, we aim to explore the impact of semantic manipulation on DNNs predictions by manipulating the semantic attributes of images and generate "unrestricted adversarial examples". In particular, we propose an algorithm SemanticAdv which leverages disentangled semantic factors to generate adversarial perturbation by altering controlled semantic attributes to fool the learner towards various "adversarial" targets. We conduct extensive experiments to show that the semantic based adversarial examples can not only fool different learning tasks such as face verification and landmark detection, but also achieve high targeted attack success rate against real-world black-box services such as Azure face verification service based on transferability. To further demonstrate the applicability of SemanticAdv beyond face recognition domain, we also generate semantic perturbations on street-view images. Such adversarial examples with controlled semantic manipulation can shed light on further understanding about vulnerabilities of DNNs as well as potential defensive approaches.

The authors describe a method for adversarially modifying a given (test) example that 1) still retains the correct label on the example, but 2) causes a model to make an incorrect prediction on it. The novelty of their proposed method is that their adversarial modifications are along a provided semantic axis (e.g., changing the color of someone's skin in a face recognition task) instead of the standard $L_p$ perturbations that the existing literature has focused on (e.g., making a very small change to each individual pixel). The adversarial examples that the authors construct, experimentally, are impressive and striking. I'd especially like to acknowledge the work that the authors put in to construct an anonymous link where they showcase results from their experiments. Thank you!
Overall, I think that this is interesting work that can help to broaden the study of adversarial examples and make them more applicable even in non-adversarial settings (e.g., by making models more robust to the changes in semantic attributes that the authors consider). There has been quite a bit of interest in the community in adversarial examples that are not just $L_p$ perturbations, and I believe that the authors' approach will encourage a good deal of follow-up research.
However, my main concern with the paper is that in my opinion, it does not sufficiently address why it is important to generate adversarial examples in the way they do. For example:
1) Is the argument that this is a more powerful attack surface, so adversaries should take note (and defenders should figure out how to defend against this)? If that is the case, what is the attack model under which these attacks are realistic? For example, the original $L_\infty$ attacks are motivated in the sense that the adversarial examples are visually imperceptible, so they might not be noticed by the end-user. What is the equivalent argument for these semantic attacks?
2) Is the argument that these semantic attacks somehow capture a more realistic part of the data distribution over all natural images, and therefore it is good to have models that perform well on these semantic adversarial examples even if we're not concerned about an adversary (e.g., because the model might generalize better to other tasks or be more causally correct)? If that's the case, then I think this needs to be explored more. For example, what about the following straw man baseline: use a controllable semantic-attribute-based generator to generate semantically different images without any notion of an adversarial attack, and then do standard $L_p$ attacks on that generated image? How would that be better or worse than the proposed method?
3) Or is the argument that it is just good to be able to generate examples that models get wrong? If so, why, and why is this method better than other methods?
I think the paper would be significantly stronger if the importance and implications of their work were explicated along the above lines. For this reason, my current assessment is a weak reject, though I'd be open to changing this assessment.
=== Less critical comments, no need to respond or fix right away ===
While the overall concept and approach was clear, I generally found the notation and mathematical exposition difficult to follow. Please be more precise. Here is a non-exhaustive list of examples from section 3:
a) I'm not sure what's the difference between $x^\text{tgt}$ and $x^\text{adv}$, or between $x^\text{new}$ and  $x^*$. These seem to be used somewhat interchangeably?
b) Equation 3 is the central optimization problem in the paper, and should be written out explicitly using $\alpha$ as the optimization variable, instead of referring to equations 1 and 2 (in which $x^*$ doesn't even appear).
c) I didn't understand equation 4. What does assuming $M(x^\text{tgt}) = y^\text{tgt}$ mean? What happens when that is not true?
d) Equation 5: Why is $y$ in the right hand side by not in the left?
e) Equation 6: $L_\text{smooth}$ is missing an argument.


Improving sample efficiency has been a longstanding goal in reinforcement learning. In this paper, we propose the VRMPO: a sample efficient policy gradient method with stochastic mirror descent. A novel variance reduced policy gradient estimator is the key of VRMPO to improve sample efficiency. Our VRMPO needs only O( −3 ) sample trajectories to achieve an -approximate first-order stationary point, which matches the best-known sample complexity. We conduct extensive experiments to show our algorithm outperforms state-of-the-art policy gradient methods in various settings.

This paper proposed a variant of policy gradient algorithm with mirror descent update, which is a natural generalization of projected policy gradient descent. The authors also proposed a variance reduced policy gradient algorithm following the variance reduction techniques in optimization. The authors further proved the convergence of the proposed algorithms and some experiments are conducted to show the effectiveness of their algorithms. The paper is not written in a very good way due to many typos and misleading notations. Moreover, there seem to be some technical issues in the proofs of both main theorems.
The notations are inconsistent. In eq (14) the trajectory index is k, while in eq (16) the index changed to n. In eq (28), the trajectory is denoted by $\tau_j^t$, while in the algorithm, there is no index $t$ in $\tau_j$.
Definition of P in eq (36) seems to overlap with that of $\mathcal{G}$ in eq (10), which means the same quantity was defined twice using two different notations.
Section 3.1 is almost the same as in Ghadimi et al., (2016), both the theorem and remarks. Since this is not a new finding, I suggest the authors to simplify the current statement in this subsection.
In Theorem 1 & 2, the authors used the criteria $\mathbb{E}[\|\mathcal{G}(\theta_n)\|_2^2]$ to establish the convergence result. However, they used $\mathbb{E}[\|\mathcal{G}(\theta_n)\|_2]$ to establish the convergence result in Theorem 3. This is kind of confusing because in different criteria, the complexity result will be different.
It seems that eq (32) is the same as eq (28). Can the authors elaborate the differences discussed in the paragraph after eq (32) in more detail?
In eq (40), $\hat g_k$ is defined as the average gradient over all the iterates up to $k$, which means $\hat g_k$ is a function of $\theta_1,\ldots,\theta_k$. But $g_k$, according to eq (21) in Algorithm 1, is just defined based on the current policy parameter $\theta_k$. I am skeptical why these two terms will be equal. This seems to be a technical issue of the whole proof of Theorem 2.
The derivation in (55) seems incorrect. In particular, the authors called Lemma 3, which is the result in Fang et al. (2018). However, the recursive gradient is defined differently in the current paper and in Fang’s paper. In Fang’s paper, at different iterations, the data are sampled from an unknown but fixed data distribution. But in this paper, since the data are sampled from the current policy which varies at each iteration, the data distribution is changing all the time. Therefore, Lemma 3 does not hold in the setting of this paper.
In all the experiments, it is strange that the proposed VRMPO was not compared with any of the other variance reduced algorithms listed in Table 1. I think it would be more convincing to demonstrate the improvement in Table 1 by comparing the proposed VRMPO with at least one other variance reduced algorithm.
Other comments:
1. On page 15, the first sentence of the proof: “trjecories” -> “trajectories”
2. What is $\mathcal{J^*}$ in eq (39)? It was not defined in the paper.
3. In the statement of Lemma 3, the authors said “telescoping Eq. (3) ...” However, Eq (3) in this paper is the definition of policy gradient. I assume this is a typo?
4. What is $\epsilon_1$ in eq (50)? It seems that this term does not appear in Lemma 4.
====after rebuttal====
Thanks for the authors' response. I am not satisfied with their response since they did not directly answer many of my questions about the technical flaws in this paper.
The response of the authors on the difference between estimator in Shen et al (2019) and the estimator in this paper is incorrect. The estimator in Shen et al. (2019) can also be constructed recursively based on G_0.
The authors also claimed that equation (40) is correct. However, this is not true. Although the authors removed $\hat g_k$ from equation (40), they still used the property that $\hat g_k$ is an unbiased estimator in the equation after (40). Note that $\epsilon_k$ is defined based on $\hat g_k$. Therefore, the proof is still problematic. The authors did not show how to fix it.
A more important issue of the direct use of Lemmas from Fang et al. (2018) is not fixed either. In their response, the authors simply changed the data generation in the inner loop of their algorithm such that all the iterates generate data from the initial policy $\pi_{\theta_k,0}$. However, this will cause the expectation of all the stochastic gradients used in their proof to be equal to the true policy gradient at the initial point, i.e., $\nabla J(\theta_{k,0})$. In other words, they can never obtain the full gradient $\nabla J(\theta_{k,t})$ even though their goal is to bound $\|\theta_{k,t+1}-\theta_{k,t}\|^2$ which definitely requires an approximation of $\nabla J(\theta_{k,t})$. Therefore, the issue cannot be fixed by simply using the same distribution in all the inner loop iterates. I suggest the authors to read more carefully the paper by Fang et al. (2018) and Papini et al. (2018) to understand the difference in the two settings.


Deep reinforcement learning requires a heavy price in terms of sample efficiency and overparameterization in the neural networks used for function approximation. In this work, we employ tensor factorization in order to learn more compact representations for reinforcement learning policies. We show empirically that in the low-data regime, it is possible to learn online policies with 2 to 10 times less total coefficients, with little to no loss of performance. We also leverage progress in second order optimization, and use the theory of wavelet scattering to further reduce the number of learned coefficients, by foregoing learning the topmost convolutional layer filters altogether. We evaluate our results on the Atari suite against recent baseline algorithms that represent the state-of-the-art in data efficiency, and get comparable results with an order of magnitude gain in weight parsimony.

This paper investigates a list of methods to reduce the number of weights for deep RL architecture under the low-data regime. These methods include tensor regression, wavelet scattering, as well as second-order optimization (K-FAC). The experiments on the Atari games shows that by using tensor regression to replace the dense layer of the neural nets and using K-FAC for the optimization, one can reduce around 10 times of parameters without losing too much of performance.
However, I have some concerns on the novelty of this work and therefore I’m giving this paper a weak reject. Here are the reasons:
To begin with, leveraging tensor structure of the neural nets to reduce number of parameters while maintaining similar level or getting even better results have been done before, for example: Tensorizing Neural Networks (Novikov et al, 2015), Learning compact recurrent neural networks with block-term tensor decomposition (Ye et al., 2018) etc. Although the use of tensor regression might be new, the core idea is still to leverage the low rank property of the tensor and obtain a compression of the weight tensors. Moreover, why use Tucker decomposition specifically for the tensor regression? It has been proposed that using tensor train (TT) decomposition can also get very good results (see Garipov et. al. Ultimate tensorization: compressing convolutional and FC layers alike). Is it possible to investigate the use of TT decomposition for the dense layer of the deep RL architecture? Therefore the novelty for this aspect seems a bit weak for me.
The second method the authors have attempted is to swap the convolution layer of the deep RL architecture with wavelet scattering. For one particular game (demon_attack), this approach seems to outperform every other methods by a large margin. However the experiment shows that for the rest of the Atari games, there is a huge drop (45%) of performance. Therefore the significance of this approach is rather thin for me. Maybe some further investigation of the game demon_attack is needed to understand why using scattering in this game in particular gives such a huge performance boost.
Thirdly, as an approximation of the second order optimization, K-FAC does not really concern with the main theme of the paper, which is an investigation of potential weights reduction methods. It is great that the authors applied this techniques and seems to have great results. However, as the authors pointed out, K-FAC has been wildly applied in the deep RL literature, and the authors did not propose new extension for the K-FAC method, therefore the contribution of this matter is also quite thin.
Last but not least, the writing of the paper is a bit clumsy, and I was having a hard time to figure out what exactly is the proposed method. I think this paper might need some rework on the writing to describe the idea of the authors in a more clear way for the publication. Due to these reasons, I’m giving this paper a weak reject.
Some writing comments and potential writing errors (did not affect the decision):
Page 3, first line of “Tensor regression layer”, the shape of the tensor X seems to be a typo.
Also here, the definition of <X, Y>_N in the paper is to sum over the dimension of I_1…I_N, then the shape of <X, Y>_N should be K_1*…*K_x*L_1*…*L_y, without the I_N in the middle.
Also in this section, the authors mentioned Tucker decomposition for the tensor regression. However the phrasing of this sentence needs a bit rework. The usage of “For instance here”, gives the readers a feeling that Tucker is just one possible way of doing this decomposition, but not necessarily the actual decomposition for the reported experiments.
In 2.3, there is a lack of definition for  \Lambda_1 and \Lambda_2. In addition, it would be better for the general readers to add a few definitions for the terminologies in this section. For example, “circular convolution”, “wavelet filter banks” etc. I guess people with corresponding background will understand it with no problem, however I do find myself a bit lost in this section with these terminologies.
2.4 line 6, “with A and. B smaller, architecture-dependent matrices”. I think it should be “with A and B being…“
3.1, line 5, “This is all the more pressing that….”, I did not understand this sentence.
In page 6, line 3, there is a lack of definition for “compression rate”. Is it the compression rate w.r.t only the last dense layer, or w.r.t the whole network?
Figure 4 is lacking y-axis and x-axis labels.
4.2, last bullet point, “However, one must not forget that the conv layers one learns must be somehow be well adapted…”, I get what you are saying, but the sentence is a bit clumsy.
Table 1 and 2, the row name “Average” is lacking definition.
 Overall it is a good attempt to reduce the number of weights in the deep RL architecture, but I do think the novelty of this work is a bit thin and the three contributions were not tied together with the main theme of the paper. Therefore, I’m giving this work a weak reject.


Our work offers a new method for domain translation from semantic label maps and Computer Graphic (CG) simulation edge map images to photo-realistic images. We train a Generative Adversarial Network (GAN) in a conditional way to generate a photo-realistic version of a given CG scene. Existing architectures of GANs still lack the photo-realism capabilities needed to train DNNs for computer vision tasks, we address this issue by embedding edge maps, and training it in an adversarial mode 1. We also offer an extension to our model that uses our GAN architecture to create visually appealing and temporally coherent videos.

SUMMARY: Use HED to convert semantic maps to edge maps, then use GAN to generate realistic images, by adding several losses together. Also add a flow loss while generating videos.
Paper could be written much better, it is slightly confusing to follow.
The effort behind the work is commendable, they used good baselines to compare. However, the idea does not seem different from already existing ones. Although the authors mention this as a CG2real problem, as far as I understand this is a semantic maps to image problem. Maybe I'm missing something.
The only innovation is to have a middle step of converting these semantic maps to edge maps, and then from edge maps to realistic images. This is not altogether a big change, I was hoping to see it goes in a good direction. As far as I understand, the DNED model the authors use is a slight change to the HED model the authors cite, in that they sample different weights to combine to form different outputs.
The results look good from the examples shown. However, there is a lot more work to be done. Since architecturally there is not much difference in the already existing models, maybe focus on more use cases of edge maps. What is the advantage of using edge maps as an intermediate step, as opposed to some other representation? How do we exploit this fact better? These are good questions to explore.
The video results look good, and this is one example of exploiting the fact that edge maps are an intermediate step. However, more work needs to be done to support claims of diversity.
Overall, it seems intuitive to throw in a lot of losses and let the network optimize over all of them in an adversarial way, but a lot more effort needs to be put into thinking about any innovation there.


Analyzing deep neural networks (DNNs) via information plane (IP) theory has gained tremendous attention recently as a tool to gain insight into, among others, their generalization ability. However, it is by no means obvious how to estimate mutual information (MI) between each hidden layer and the input/desired output, to construct the IP. For instance, hidden layers with many neurons require MI estimators with robustness towards the high dimensionality associated with such layers. MI estimators should also be able to naturally handle convolutional layers, while at the same time being computationally tractable to scale to large networks. None of the existing IP methods to date have been able to study truly deep Convolutional Neural Networks (CNNs), such as the e.g. VGG-16. In this paper, we propose an IP analysis using the new matrix-based Rényi's entropy coupled with tensor kernels over convolutional layers, leveraging the power of kernel methods to represent properties of the probability distribution independently of the dimensionality of the data. The obtained results shed new light on the previous literature concerning small-scale DNNs, however using a completely new approach. Importantly, the new framework enables us to provide the first comprehensive IP analysis of contemporary large-scale DNNs and CNNs, investigating the different training phases and providing new insights into the training dynamics of largescale neural networks.

This paper concerns the "information plane" view of visualizing and understanding neural net training. Kernel-based estimators of Renyi entropy-like quantities are applied to study moderate-sized DNNs. A straightforward extension to CNNs is presented.
Major comments:
I found this paper extremely hard to follow. I think a lot of my difficulty was a lack of familiarity with the papers this work was building on, but I also felt the main line of reasoning was unnecessarily opaque. I've tried to indicate where some clarifying wording might help readers such as myself.
In general I am very skeptical that reasonable entropy-like quantities can be estimated reliably for high-dimensional data using anything along the lines of kernel density estimation, especially based on a single minibatch or small collection of minibatches! The authors provide no experimental evidence that these estimates are even close to being accurate (for example on synthetic datasets where the true entropy / mutual information is known). Clearly the estimated quantities evolve during training, and that may be interesting in itself, but calling the estimated quantities "mutual information" seems like a leap that's not justified in the paper.
Throughout the paper, CNNs are referred to as having some special difficulty for entropy estimates because of the channel dimension. This is misleading. For DNNs the activations are vectors, and so a kernel defined over rank 1 tensors is needed. Even without the channel dimension, CNN activations would be rank 2 tensors, and so a kernel would need to be defined over rank 2 tensors. Going from rank 2 to rank 3 doesn't pose any special difficulties. Indeed the most obvious way of defining a kernel over higher rank tensors is to flatten them and use a kernel defined over vectors, which is exactly what the paper does in practice.
Minor comments:
In the introduction, it would be helpful to include some relevant citations after the sentence "Several works have demonstrated that one may unveil interesting properties... IP".
In section 3, the multivariate extension proposed by Yu et al. (2019) seems like an interesting side note (since it was used in a previous attempt to estimate information plane for CNNs), but it doesn't seem central to the paper, and I personally found it unnecessarily confusing to have it presented in the main text. What about moving sections 3.2 and 4.2 to the appendix for clarity?
In section 3.1, "over a finite set" is probably incorrect ("probability density function" implies a continuous space for X, as does the integral in (1)).
In section 3.1 (and the appendix), "The ones developed for Shannon" seems imprecise. "Certain approaches developed for estimating the Shannon entropy"?
It's not clear what "have the same functional form of the statistical quantity" means. Which statistical quantity? What aspects of the functional form are similar? Please elaborate in the paper.
I think "marginal distribution" is incorrect. It's representing a whole probability distribution, not just a marginal distribution. Which marginal would it be in any case?
Section 3.2 states "The matrix-based... entropy functional is not suitable for estimating... convolutional layer... as the output consists of C feature maps", but as discussed above, there is no special difficulty caused by the channel dimension. Even if the channel dimension were not present the difficulties would be the same. (Also, it seems like defining a kernel over the rank-3 tensors is an extremely natural / unsurprising thing to try given the set-up so far.)
In section 4.1, "can not include tensor data without modifications" seems misleading for a similar reason. One of the great things about kernels is that they can be defined on lots of objects, including things like rank 3 tensors!
Near (8), it would be very helpful to state explicitly what is done for the joint entropy term in (5). It sounds like this term, which involves the Hadamard product, in practice amounts to summing the Euclidean distances between x's and between y's, and it might be helpful to the new reader to point this out. (It also highlights that the method is easy to implement in practice).
The discussion in section 4.2 is only valid for the RBF kernel, but the first paragraph of that section makes it sound like it is true more generally.
At the bottom of section 4.2, if the proposed approach is equivalent to the multivariate approach, then how can one suffer from numerical instability while the other doesn't? Also, numerical stability makes it sound like an artifact of floating point arithmetic, whereas I think the point that's being made is more mathematical? Please clarify in the paper.
In "enabling training of complex neural networks", shouldn't "training" be "analysis"?
In section 5, under "increasing DNN size", I wasn't clear on the meaning of "deterministic" in "neither .... is deterministic for our proposed estimator". Random variables can be deterministic or not, but how can a mutual information be deterministic?
Under "effect of early stopping", isn't looking at the test set entropies (as is done elsewhere in the paper) much more relevant to overfitting than considering different "patience" values?
In the bibliography, two different papers are "Yu et al. (2019)".
In appendix A, "the same functional form of the statistical quantity", "marginal", etc don't seem quite correct, as mentioned above. Also the first equation should not have a comma between f(X) and g(Y) (which if I understand correctly are being multiplied).


Training neural machine translation models (NMT) requires a large amount of parallel corpus, which is scarce for many language pairs. However, raw non-parallel corpora are often easy to obtain. Existing approaches have not exploited the full potential of non-parallel bilingual data either in training or decoding. In this paper, we propose the mirror-generative NMT (MGNMT), a single unified architecture that simultaneously integrates the source to target translation model, the target to source translation model, and two language models. Both translation models and language models share the same latent semantic space, therefore both translation directions can learn from non-parallel data more effectively. Besides, the translation models and language models can collaborate together during decoding. Our experiments show that the proposed MGNMT consistently outperforms existing approaches in all a variety of scenarios and language pairs, including resourcerich and low-resource languages.

This paper proposes an approach to neural MT in which the joint (source, target) distribution is modeled as an average over two different factorizations: target given source and source given target. This gives rise to four distributions - two language models and two translation models - which are parameterized separately but conditioned on a common variational latent variable.  The model is trained on parallel data using a standard VAE approach. It can additionally be trained on non-parallel data in an approach similar to iterated back-translation at sentence-level granularity, but with language and translation model probabilities for observed sentences coupled by the latent variable. Inference iterates between sampling a latent variable given the current best hypothesis, and using beam search plus rescoring to find a new best hypothesis given the current latent variable. The approach is evaluated in several different scenarios (low- and high-resource, domain adaptation - trained on parallel data only or parallel plus monolingual data) and found to generally outperform previous work on generative NMT and iterated back-translation.
Strengths: clearly written, well motivated, very comprehensive experiments comparing to relevant baselines.
Weaknesses: somewhat incremental relative to Shah and Barber (Neurips 2018), results are only marginally positive, framework is probably too cumbersome to justify widespread adoption based on the results.
I think the paper should be accepted. Although it’s not highly original, it ties together three strands of work in a principled way: joint models, variational approaches, and back-translation / dual learning. The increment over Shah and Barber is bolstered by the addition of back-translation, which gives substantial improvements when using non-parallel data; and to a lesser extent by the argument about the advantage of separate models for distant language pairs. Using all possible LMs and TMs coupled with a latent variable feels like an area that was inevitably going to get explored, and this paper does a good job of it. Although the gains over the baselines are not overly compelling, they are quite systematic, indicating that the advantage is probably real, albeit slight. The authors are also to be commended on their use of not just the Shah and Barber baseline, but also the back-translation-based techniques, which are generally stronger competitors when monolingual data is incorporated.
Further comments/questions:
Why are there no results for Transformer+Dual in table 4? This omission looks odd, since Transformer+Dual was the strongest baseline in table 3.
Please add implementation details for the Transformer+{BT,JBT,Dual} baselines.
It was surprising not to see robustness experiments like Shah and Barber’s dropped source words, since robustness to source noise could be one of the advantages of having an explicit model of the source sentence.
A few additional suggestions for related work: noisy channel approaches (eg, The Neural Noisy Channel, Yu et al, ICLR 2017); decipherment (eg, Beyond parallel data: Joint word alignment and decipherment improves machine translation, EMNLP 2014 - yes, from SMT days, but still); other joint modeling work (KERMIT: Generative Insertion-Based Modeling for Sequences, Chan et al, 2019).
Consider dropping the “1+1 > 2” metaphor. It’s not clear to me exactly what it means, or what it adds to the paper.
“deviation” is used in a couple places where you probably meant “derivation”?
Line 6 in algorithm 2 should use both forward and backward scores for rescoring.


Unsupervised Domain Translation (UDT) consists in finding meaningful correspondences between two domains, without access to explicit pairings between them. Following the seminal work of CycleGAN, many variants and extensions of this model have been applied successfully to a wide range of applications. However, these methods remain poorly understood, and lack convincing theoretical guarantees. In this work, we define UDT in a rigorous, non-ambiguous manner, explore the implicit biases present in the approach and demonstrate the limits of theses approaches. Specifically, we show that mappings produced by these methods are biased towards low energy transformations, leading us to cast UDT into an Optimal Transport (OT) framework by making this implicit bias explicit. This not only allows us to provide theoretical guarantees for existing methods, but also to solve UDT problems where previous methods fail. Finally, making the link between the dynamic formulation of OT and CycleGAN, we propose a simple approach to solve UDT, and illustrate its properties in two distinct settings.
Given pairs of elements from two different domains, domain translation consists in learning a mapping from one domain to another, linking these paired elements together. If we consider a photograph of a given scene, and an artistic painting of the same scene, we may want to learn the map associating a photograph to paintings describing the same scene, and conversely, for paintings to photographs. A wide range of problems can be formulated as translation, including image-toimage (Isola et al. (2016)), or video-to-video (Wang et al. (2018), image captioning ), natural language translation (Bahdanau et al. (2015)), etc... However, obtaining paired examples can be difficult thus motivating the unpaired setting where only samples from both domains are available which allows us to tackle a wider range of problems. A seminal work in this direction has been the CycleGAN model proposed in Zhu et al. (2017a), which has led to extensions for many applications and has given impressive results.
The starting point of this work is to understand and study this successful approach as there remains little theoretical understanding of why these models work. ; Yang et al. (2018) have observed that the approach first introduced in Zhu et al. (2017a) is ill-posed: in most cases, any pairing between samples of both domains -many of which are unwanted pairings -satisfies their objective function. This is in contradiction with the empirical results of the model and shows that there must be an implicit bias towards well-behaved mappings.  made a first step in this direction by demonstrating that desirable mappings are obtained by networks of minimal complexity, relating this notion of complexity to the number of hidden layers of the neural network implementing the mapping. They then show that the number of mappings satisfying the objective is expected to be small 1 . This is indeed a first step but some ambiguity remains: For example, when translating photographs to paintings, instead of preserving the content of the input, we may want to preserve the same color palette. Here, both tasks are in contradiction and no mapping can solve both problems: What task will be solved by the CycleGAN model? What is the right notion of complexity to explain this convergence? Can we steer the model toward a certain task?
In this work, we aim to answer these questions and further bridge the gap between practice and theory, by first rigorously and unambiguously defining the UDT problem. Optimal Transport theory then appears as a natural tool to regularize UDT models in this context and its dynamical formulation allows us to build a model which overcomes the shortcomings of CycleGAN-like models.
Our main contributions are the following:
• Highlighting the need for a more rigorous treatment of the problem of UDT, we reformulate it in a rigorous, non-ambiguous and generic way, showing the theoretical limits of the CycleGAN approach.
• We provide an explanation for the success of CycleGAN, bridging the gap between theory and empirical findings and shedding light on the implicit biases in those models.
• More specifically, this allows us to link UDT and Optimal Transport. We show that any UDT problem can be cast into the Optimal Transport framework and prove that any UDT Task can be solved by selecting the appropriate transportation cost.
• Exploiting the connection between residual networks and dynamical systems, we present a general and flexible approach to solve UDT, based on the Dynamical formulation of Optimal Transport. This allows us to train a steerable one-sided mapping, and obtain the inverse mapping for free after training, along with smooth interpolations between both domains.
• We empirically highlight the implicit bias and limitations of the CycleGAN approach, and demonstrate the potential of the proposed approach in several different scenarios.

The paper proposes an analysis of CycleGAN methods. Observing that the class of solutions (mappings) covered by those methods is very large, they propose to restrict the set of solutions by looking at low energy mappings (energy being defined wrt. a dedicated cost). A natural formulation of the associated problem is found in optimal transport (OT) theory. They examine the underlying problem in its dynamical formulation, for which a direct connection can be made with ResNet architecture that are commonly used in cycleGANs. They illustrate these results on simple examples, involving pairing swapped digits from MNIST and celebA male to female examples. As a matter of facts, results presented with the OT formulation are more constant. The main proposition of the paper is that the task at hand can be efficiently coded through the distance (cost) function of OT.
Overall the paper is well written and the proposition is reasonable. Yet some parts seem unnecessary long to me, or bring little information, notably the formalisation of 2.1 and 2.2. The fact that cycleGANs are severely ill-posed problems is well known from the computer vision community. Variants that can include a few paired samples can be found (not exhaustive):
Tripathy, S., Kannala, J., & Rahtu, E. (2018, December). Learning image-to-image translation using paired and unpaired training samples. In Asian Conference on Computer Vision (pp. 51-66). Springer, Cham.
Oe that try to regularize the associated flow:
DLOW: Domain Flow for Adaptation and Generalization Rui Gong, Wen Li, Yuhua Chen, Luc Van Gool; The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019, pp. 2477-2486
In this spirit, I wonder if a comparison with only the vanilla cycleGAN is sufficient to really assess the interest of using the OT formulation of the problem. Notably, in the example of digit swaps, a cost is learnt by finding a representation of the digits that eliminates the importance of position in the representation. Training such a classifier assumes having some labelled data, that could theoretically be paired, and thus making amenable variants of cycleGans that use a few paired samples. In this sense, I think that the paper fails in giving convincing arguments that advocate the use of OT here. As the dynamical formulation is known and already used to learn mappings (  see Trigila, G., & Tabak, E. G. (2016). Data‐driven optimal transport. Communications on Pure and Applied Mathematics, 69(4), 613-648. For instance). Also variants of OT that estimate a Monge mapping could have been included (e.g. V. Seguy, B. B. Damodaran, R. Flamary, N. Courty, A. Rolet, M. Blondel, Large-Scale Optimal Transport and Mapping Estimation, International Conference on Learning Representations (ICLR), 2018.)
As a summary:
Pros:
A nice interpretation of CycleGAN with OT
The paper is fairly well written
Cons:
Overall the quantity of novelties is, in the eyes of the reviewer, somehow limited. At least the contributions should be clarified;
The experimental section is not convincing in explaining why the OT formulation is better than variants of cycleGAN or also other schemes for computing OT than the dynamical formulation
Minor remark:
A reference to Benamou, Brenier 2000 could have been given regarding section 3.2 and the dynamical formulation of OT.


A deployable machine learning model relies on a good representation. A desirable criteria for a good representation is to be interpretable by humans. We propose a technique termed perceptual regularization that enables visualization of the latent representation. By visualizing the learned representation, we are also able to understand the attention of a model, obtaining visual evidence that supervised networks learn task-specific representations. Moreover our method provides a direct visualization of the effect that adversarial attacks have on the internal representation of a deep network. We show how this can be used to systematically perform latent space interpolation to modify semantic features.

Summary: This paper discusses the use of a reconstruction network which is fed an internal representation of a deep network and is trained to reconstruct the input. The classification and reconstruction networks are trained simultaneously with a combined loss function. Various experiments are carried out to test whether this approach can help investigate adversarial examples and the quality of learned features for transfer learning.
Review: The idea presented in this paper is simple and easy to understand and the paper includes experiments testing its effectiveness in various settings. However, there are various issues with the paper in its current form: First, the idea of reconstructing hidden activations is not new and the paper fails to cite future work. "Dynamic Routing Between Capsules" by Sabour et al. consider adding a similar reconstruction network to Capsule networks. "Detecting and Diagnosing Adversarial Images with Class-Conditional Capsule Reconstructions" considers the use of this reconstruction network for detecting adverarsarial examples (like this paper) and also includes an unconditional reconstruction network which is added to a convnet (like this paper). I imagine there are other examples, but those are ones I know of. Second, the experiments are limited - they only include simple datasets (SVHN, MNIST, CelebA) which are normalized so that the object appears in the center of the image. This makes reconstructions difficult. The experiments also lack various important details like the model architectures and other hyperparameters, and overall are mostly "proof of concept" experiments rather than exhaustively testing out the proposed method. Finally, some of the claims made in the paper (particularly in the introduction) are dubious, for example that adversarial examples are a security concern and that transfer learning has been unsuccessful because it can harm performance. Based on these issues, I won't recommend acceptance. I think if the authors better situated the novelty of their work, included more exhaustive and difficult experiments, and/or focused on one specific application area it would be a stronger submission.
Specific comments:
- "to be understandable" - can you be more specific? Internal representations of deep neural networks are not understandable whatsoever on their own, we usually use some post-hoc method (like your reconstruction network) to try to interpret them somehow.
- "[adversarial examples] raise many security concerns, such as the reliability of driverless cars, or the trustworthiness of facial recognition systems" The field is moving towards a consensus that adversarial examples do not pose a realistic security threat in any of the settings they have been tested in. The most significant description of this viewpoint is in "Motivating the Rules of the Game for Adversarial Example Research" by Gilmer et al.
- "but has made the transferability of features to new tasks difficult, sometimes even harming performance" This statement completely omits a huge body of work showing how effective transfer learning is, not only in computer vision (typically going from an ImageNet classifier to some downstream task with a small training set) but also in NLP (where transfer learning-based methods have state-of-the-art in virtually every benchmark). A good discussion of when transfer learning works and doesn't in CV is provided in "Do Better ImageNet Models Transfer Better?" by Kornblith et al.
- The variables X and Y (without mathcal) in the "(Perceptual regularization)" definition equation are not defined.
- The classification and reconstruction models are not described in Section 3. This is incredibly important. The quality of the reconstructions is likely extremely dependent on these architectural choices as they provide an implicit prior over the image space. This detail needs to be filled in. E.g. the pooling/downsampling in your classification network  will have a strong impact on the reconstruction quality, as will the size, depth, and configuration of the reconstruction network. Furthermore, I imagine the regularization parameter lambda is extremely important (as shown in section 4.2) but its value is not mentioned.
- The experiments are on SVHN, MNIST, and CelebA which are all "normalized" datasets, i.e. all of their constituent images are normalized so that the object of interest appears in the same position in the center of the image. This makes reconstruction much, much easier. Results should be presented on non-normalized datasets (e.g. CIFAR-10, ImageNet, etc) in order to be convincing.
- "But, as the network gets more complicated it becomes computationally difficult to decode the feature map." Can you quantify this claim or back it up with experiments?
- While the results on transferring to different CelebA classification tasks are interesting, it is much more common to pre-train on a diverse task like ImageNet than an extremely fine-grained task like classifying smiling or not smiling. I think these results would be more impressive if you followed modern experimental practice more closely.
- "The cost is that our method performs slightly poorer than the standard training method when S = T" I would not call a difference of upwards of 8% absolute small.


Recent advances in deep learning theory have evoked the study of generalizability across different local minima of deep neural networks (DNNs). While current work focused on either discovering properties of good local minima or developing regularization techniques to induce good local minima, no approach exists that can tackle both problems. We achieve these two goals successfully in a unified manner. Specifically, based on the Fisher information we propose a metric both strongly indicative of generalizability of local minima and effectively applied as a practical regularizer. We provide theoretical analysis including a generalization bound and empirically demonstrate the success of our approach in both capturing and improving the generalizability of DNNs. Experiments are performed on CIFAR-10 and CIFAR-100 for various network architectures.

Post-rebuttal update: I have just noticed the authors modified their summary post below and claimed "[my concerns] are all minor or resolved". This is not true. Here is my summary of unresolved concerns written after the discussion period.
This work has been substantially improved during the rebuttal process, and some of my concerns are addressed. But there are still major issues, as raised in my [last comment]( https://openreview.net/forum?id=BJlXgkHYvS&noteId=r1xAnokijS ), that remains unanswered. Specifically,
(A) the relation between this work and information theory
In the revision, the authors have make it very clear that the relation between FIA and their proposed regularized objective is very vague, relying on the crude approximation of expected Fisher information with observed Fisher information. Therefore the "information-theoretic" part in the title seems awkward and to some extent, misleading.
As Reviewer 1 has pointed out, it would have been better if the authors relate their theory and method to the observed FIM, instead of information theory, from the beginning. Since the observed FIM and the neural tangent kernel (NTK) share the same eigenspectrum, it would also be interesting to relate this work to the NTK.
(B) the different behavior of the proposed regularization (log det(I)) and its bound that is actually implemented (log tr(I))
This is the more important issue. My concern is that the observed FIM (or the NTK) is known to have fast decaying spectrum; (Karakida et al) has shown empirically that the decay can be exponential. Thus log det(I) would be dominated by the long tail (since after taking logarithm it is the sum (or average) of an arithmetic sequence), while log tr(I) would be dominated by the first largest few values.
The authors claim that this is not an issue since they replaced the observed FIM with a subsampled, low-rank (<=10), version. It corresponds to consider a small submatrix of (the gram matrix of) the NTK. Denote this matrix as .
(a) This does not help with the problem, since we now have no chance of recovering the smaller eigenvalues that would have dominated log(det(I)), and it is impossible that the proposed regularizer has a similar behavior to log(det(I)).
(b) One could verify easily, using small feed-forward networks (or even simpler, computing a gram matrix using RBF kernels, since the FIM shares its eigenspectrum with NTK which is a p.d. kernel), that the new matrix  still has a fast-decaying eigenspectrum, so the behavior of  and \tilde{I} are still significantly different, even though this cannot be established by concentration bounds as the authors argue. While FFN and modern deep architectures can have different behaviors, I believe the above evidence suggests that a numerical experiment comparing the behavior of the two bounds is a must.
Following this argument we can see *another issue* of this work, namely the proposed generalization bound will be vacuous given the fast-decaying spectrum of the FIM, since it contains gamma=log(det(I)).
Reviewer 1 mentioned this work could enlighten future discussions on this subject. While I agree this paper presents interesting empirical observations (namely its final algorithm, which is vaguely connected to the proposed objective, leads to improved performance on CV tasks), I think this submission in its current form is a bit too misleading to serve this purpose well, and overall I believe it would be better to go through another round of revision.
Original Review
============================================
This paper presents a generalization bound based on Fisher information at a local optima, and proposes to optimize (an approximation to) it to get better generalization guarantees. There are issues in both parts, and I don't think it should be accepted. Specifically,
1. The definition of Fisher information is incorrect (for almost every parameter). The expectation should be taken w.r.t the model distribution p(c_x|x;w), instead of the data distribution S.
2. Assumption (1) (loss locally quadratic) is not reasonable for DNNs, since local optimas will not be unique in their neighborhoods. See e.g. Section 12.2.2, "Information Geometry and Its Applications".
3. Regarding the approximation to the bound, approximating log det(I) with log trace(I) is not a good idea: adding a very small eigenvalue will lead to noticeable change in the former, but negligible change in the latter. This is particularly problematic for DNNs, since the spectrum of their Fisher information matrix varies in a wide range: see "Universal Statistics of Fisher Information in Deep Neural Networks: Mean Field Approach".
(Edit 11.8:
* regarding point (1), there is a quantity called observed Fisher information in e.g. Grunwald (2007) that coincide with Eq (1) in the paper, but it is a function of the dataset instead of the model parameter, and can only used to study model parameters near the *global optima* (as it is applied in Grunwald (2007)); it cannot help with choosing between different local optimas as this work claims. Additionally, the FIA criterion, which is used in this paper to devleop the generalization bound, is defined using the standard form of Fisher information (i.e. taking expectation w.r.t model distribution), see Rissanen (1996). These facts lead me to believe this is a confusion on the authors' part.
* in point (2) I was referring to the authors' argument " Since L(S,w) is analytic and w_0 is the *only local minimum* of L(S,w) in M(w_0)", which is incorrect.)


The ability to autonomously explore and navigate a physical space is a fundamental requirement for virtually any mobile autonomous agent, from household robotic vacuums to autonomous vehicles. Traditional SLAM-based approaches for exploration and navigation largely focus on leveraging scene geometry, but fail to model dynamic objects (such as other agents) or semantic constraints (such as wet floors or doorways). Learning-based RL agents are an attractive alternative because they can incorporate both semantic and geometric information, but are notoriously sample inefficient, difficult to generalize to novel settings, and are difficult to interpret. In this paper, we combine the best of both worlds with a modular approach that learns a spatial representation of a scene that is trained to be effective when coupled with traditional geometric planners. Specifically, we design an agent that learns to predict a spatial affordance map that elucidates what parts of a scene are navigable through active self-supervised experience gathering. In contrast to most simulation environments that assume a static world, we evaluate our approach in the VizDoom simulator, using large-scale randomly-generated maps containing a variety of dynamic actors and hazards. We show that learned affordance maps can be used to augment traditional approaches for both exploration and navigation, providing significant improvements in performance.

The paper proposes an interesting, and to the best of my knowledge novel, pipeline for learning a semantic map of the environment with respect to navigability, and simultaneously uses it for further exploring the environment.
The pipeline can be summarized as follows: Navigate somewhere using some heuristic. When navigation "works", as well as when encountering something "negative", back-project that into past frames, and label the corresponding pixels as such: either positive or negative. This generates a collection of partially densely labelled images, on which a segmentation network can be learned that learns which part of the RGBD input are navigable and which should be avoided. For navigation, navigability of the current frame is predicted, and that prediction is down-projected into an "affordance map" that is used for navigation. One experiment confirms the usefulness of such an affordance map.
I am marking weak reject currently because of the following concerns, which might be me just missing something. On the one hand, I am glad to see something that is not just blind "end to end RL with exploration bonus", sounds reasonable, and works well. On the other hand, I do have several major concerns about the method, outlined as follows:
1. How can this approach work for moving obstacles? Let's say a monster walks from point A to point B, and collides with the agent at point B. Then, point B is marked as a hazard, but in the previous frames, the monster is not located at point B, and thus an image region that does not contain the monster is marked as hazard. Am I missing something here?
2. The method does not seem practical for actual mobile robots, only for in-game or in-simulation agents. The reason being that in order to learn "robot should not bump into baby", the robot actually needs to bump into multiple babies in order to collect data about that hazard. To be fair, blind "PPO+exploration bonus" suffers from the same problem, but in this paper, the whole motivation is about mobile robots (at least that was my impression after reading it).
Furthermore, I do not think I would be able to reproduce any of the experiments, as many details are missing. Will code be released?
###### Post-rebuttal update
I am happy with the author's response to my concerns, and they have included corresponding discussions in their paper. Thus, I am improving my rating to recommend acceptance of this paper to ICRL2020.