System identification of complex and nonlinear systems is a central problem for model predictive control and model-based reinforcement learning. Despite their complexity, such systems can often be approximated well by a set of linear dynamical systems if broken into appropriate subsequences. This mechanism not only helps us find good approximations of dynamics, but also gives us deeper insight into the underlying system. Leveraging Bayesian inference and Variational Autoencoders, we show how to learn a richer and more meaningful state space, e.g. encoding joint constraints and collisions with walls in a maze, from partial and high-dimensional observations. This representation translates into a gain of accuracy of the learned dynamics which we showcase on various simulated tasks.

Thank you for the detailed reply and for updating the draft
The authors have added in a sentence about the SLDS-VAE from Johnson et al and I agree that reproducing their results from the open source code is difficult. I think my concerns about similarities have been sufficiently addressed.
My main concerns about the paper still stem from the complexity of the inference procedure. Although the inference section is still a bit dense, I think the restructuring helped quite a bit. I am changing my score to a 6 to reflect the authors' efforts to improve the clarity of the paper. The discussion in the comments has been helpful in better understanding the paper but there is still room for improvement in the paper itself.
=============
Summary: The authors present an SLDS + neural network observation model for the purpose of fitting complex dynamical systems. They introduce an RNN-based inference procedure and evaluate how well this model fits various systems. (I’ll refer to the paper as SLDVBF for the rest of the review.)
Writing: The paper is well-written and explains its ideas clearly
Major Comments:
There are many similarities between SLDVBF and the SLDS-VAE model in Johnson et al [1] and I think the authors need to address them, or at least properly compare the models and justify their choices:
- The first is that the proposed latent SLDS generative models are very similar: both papers connect an SLDS with a neural network observation model. Johnson et al [1] present a slightly simpler SLDS (with no edges from z_t -> s_{t + 1} or s_t -> x_t) whereas LDVBF uses the “augmented SLDS” from Barber et al. It is unclear what exactly  z_t -> s_{t + 1} is in the LDVBF model, as there is no stated form for p(s_t | s_{t -1}, z_{t - 1}).
- When performing inference, Johnson et al use a recognition network that outputs potentials used for Kalman filtering for z_t and then do conjugate message passing for s_t. I see this as a simpler alternative to the inference algorithm proposed in SLDVBF. SLDVBF proposes relaxing the discrete random variables using Concrete distributions and using LSTMs to output potentials used in computing variational posteriors. There are few additional tricks used, such as having these networks output parameters that gate potentials from other sources. The authors state that this strategy allows reconstruction signal to backpropagate through transitions, but Johnson et al accomplish this (in theory) by backpropagating through the message passing fixed-point iteration itself. I think the authors need to better motivate the use of RNNs over the message-passing ideas presented in Johnson et al.
- Although SLDVBF provides more experiments evaluating the SLDS than Johnson, there is an overlap. Johnson et al successfully simulates dynamics in toy image systems in an image-based ball-bouncing task (in 1d, not 2d). I find that the results from SLDVBF, on their own, are not quite convincing enough to distinguish their methods from those from Johnson et al and a direct comparison is necessary.
Despite these similarities, I think this paper is a step in the right direction, though it needs to far more to differentiate it from Johnson et al. The paper draws on many ideas from recent literature for inference, and incorporating these ideas is a good start.
Minor Comments:
- Structurally, I found it odd that the authors present the inference algorithm before fully defining the generative model. I think it would be clearer if the authors provided a clear description of the model before describing variational approximations and inference strategies.
- The authors do not justify setting $\beta = 0.1$ when training the model. Is there a particular reason you need to downweight the KL term as opposed to annealing?
[1] Johnson, Matthew, et al. "Composing graphical models with neural networks for structured representations and fast inference." Advances in neural information processing systems. 2016.


META-LEARNING WITH LATENT EMBEDDING OPTIMIZATION
Gradient-based meta-learning techniques are both widely applicable and proficient at solving challenging few-shot learning and fast adaptation problems. However, they have practical difficulties when operating on high-dimensional parameter spaces in extreme low-data regimes. We show that it is possible to bypass these limitations by learning a data-dependent latent generative representation of model parameters, and performing gradient-based meta-learning in this lowdimensional latent space. The resulting approach, latent embedding optimization (LEO), decouples the gradient-based adaptation procedure from the underlying high-dimensional space of model parameters. Our evaluation shows that LEO can achieve state-of-the-art performance on the competitive miniImageNet and tieredImageNet few-shot classification tasks. Further analysis indicates LEO is able to capture uncertainty in the data, and can perform adaptation more effectively by optimizing in latent space.

This work presents an extension of the MAML framework for "learning to learn." This extension changes the space in which "inner-loop" gradient steps are taken to adapt the model to a new task, and also introduces stochasticity. The authors validate their proposed method with regression experiments in a toy setting and few-shot classification experiments on mini- and tiered-Imagenet. The latter are well known and competitive benchmarks in few-shot learning.
The primary innovations that distinguish this work from previous gradient-based approaches to meta-learning (namely MAML) are that (i) the initial set of parameters is data-dependent and drawn from a generative distribution; and (ii) the adaptation of model parameters proceeds in a lower-dimensional latent space rather than in the higher-dimensional parameter space. Specifically, model parameters are generated from a distribution parameterized by an adapted latent code at each adaptation step. I find both of these innovations novel.
The experimental results, in which LEO outperforms the state of the art on two benchmarks derived from ImageNet by "comfortable margins," and the ablation study demonstrate convincingly that these innovations are also significant. I also found the curvature analysis and embedding visualization illuminating of the model's function. My one suggestion would be to test the model on realistic data from beyond the image domain, perhaps on something sequential like language (consider the few-shot PTB setting from Vinyals et al. (2016)). I'm aware anecdotally that MAML struggles with adapting RNNs and I wonder if LEO overcomes that weakness.
The paper is clearly written and I had little difficulty in following the algorithmic details, although I'm sure it helped to be familiar with the convoluted meta-learning and inner-/outer- loop frameworks. I recommend it for publication.
Pros:
- Natural, novel extension to gradient-based meta-learning
- state of the art results on two competitive few-shot benchmarks
- good analysis
- clear writing
Cons:
- realistic, high-dim data is only from the image domain
Minor questions for the authors:
- Relation Networks are computationally intensive, although in few-shot learning the sets encoded are fairly small. Can you discuss the computational cost and training time of the full framework?
- What happens empirically when you generate parameters for more than just the output layer in, eg, your convolutional networks?
- What happens if you try to learn networks from scratch through the meta-learning process rather than pre-training and fine-tuning them? Some of the methods you compare against do so, to my understanding.


Driven by the need for parallelizable hyperparameter optimization methods, this paper studies open loop search methods: sequences that are predetermined and can be generated before a single configuration is evaluated. Examples include grid search, uniform random search, low discrepancy sequences, and other sampling distributions. In particular, we propose the use of k-determinantal point processes in hyperparameter optimization via random search. Compared to conventional uniform random search where hyperparameter settings are sampled independently, a k-DPP promotes diversity. We describe an approach that transforms hyperparameter search spaces for efficient use with a k-DPP. In addition, we introduce a novel Metropolis-Hastings algorithm which can sample from k-DPPs defined over any space from which uniform samples can be drawn, including spaces with a mixture of discrete and continuous dimensions or tree structure. Our experiments show significant benefits in realistic scenarios with a limited budget for training supervised learners, whether in serial or parallel.

I reviewed the same paper last year. I am appending a few lines based on the changes made by authors.
The authors propose k-DPP as an open loop (oblivious to the evaluation of configurations) method for hyperparameter optimization and provide its empirical study and comparison with other methods such as grid search, uniform random search, low-discrepancy Sobol sequences, BO-TPE (Bayesian optimization using tree-structured Parzen estimator) by Bergstra et al. (2011). The k-DPP sampling algorithm and the concept of k-DPP-RBF over hyperparameters are not new, so the main contribution here is the empirical study.
The first experiment by the authors shows that k-DPP-RBF gives better star discrepancy than uniform random search while being comparable to low-discrepancy Sobol sequences in other metrics such as distance from the center or an arbitrary corner (Fig. 1).
The second experiment shows surprisingly that for the hard learning rate range, k-DPP-RBF performs better than uniform random search, and moreover, both of these outperform BO-TPE (Fig. 2, column 1).
The third experiment shows that on good or stable ranges, k-DPP-RBF and its discrete analog slightly outperform uniform random search and its discrete analog, respectively.
I have a few reservations. First, I do not find these outcomes very surprising or informative, except for the second experiment (Fig. 2, column 1). Second, their study only applies to a small number like 3-6 hyperparameters with a small k=20. The real challenge lies in scaling up to many hyperparameters or even k-DPP sampling for larger k. Third, the authors do not compare against some relevant, recent work, e.g., Springenberg et al. (http://aad.informatik.uni-freiburg.de/papers/16-NIPS-BOHamiANN.pdf) and Snoek et al. (https://arxiv.org/pdf/1502.05700.pdf) that is essential for this kind of empirical study.
COMMENTS ON THE CHANGES SINCE THE LAST YEAR
I am not convinced by the comparison with Spearmint added by the authors since the previous version. It is unclear to me if the comparison of wall clock time and accuracy holds for larger number of hyperparameters or against Spearmint with more parallelization.
In addition the authors do not compare against more recent work, e.g.,
@INPROCEEDINGS{falkner-bayesopt17,
 author    = {S. Falkner and A. Klein and F. Hutter},
 title     = {Combining Hyperband and Bayesian Optimization},
 booktitle = {NIPS 2017 Bayesian Optimization Workshop},
 year      = {2017},
 month     = dec,
}
@InProceedings{falkner-icml-18,
  title =        {{BOHB}: Robust and Efficient Hyperparameter Optimization at Scale},
  author =       {Falkner, Stefan and Klein, Aaron and Hutter, Frank},
  booktitle =    {Proceedings of the 35th International Conference on Machine Learning (ICML 2018)},
  pages =        {1436--1445},
  year =         {2018},
  month =        jul,
}


This paper introduces NSGA-Net, an evolutionary approach for neural architecture search (NAS). NSGA-Net is designed with three goals in mind: (1) a NAS procedure for multiple, possibly conflicting, objectives, (2) efficient exploration and exploitation of the space of potential neural network architectures, and (3) output of a diverse set of network architectures spanning a trade-off frontier of the objectives in a single run. NSGA-Net is a population-based search algorithm that explores a space of potential neural network architectures in three steps, namely, a population initialization step that is based on prior-knowledge from hand-crafted architectures, an exploration step comprising crossover and mutation of architectures and finally an exploitation step that applies the entire history of evaluated neural architectures in the form of a Bayesian Network prior. Experimental results suggest that combining the objectives of minimizing both an error metric and computational complexity, as measured by FLOPS, allows NSGA-Net to find competitive neural architectures near the Pareto front of both objectives on two different tasks, object classification and object alignment. NSGA-Net obtains networks that achieve 3.72% (at 4.5 million FLOP) error on CIFAR-10 classification and 8.64% (at 26.6 million FLOP) error on the CMU-Car alignment task.

After rebuttal, I adapted the score. See below for original review.
--------------------------------------------
The authors implement a two-stage multi-objective optimization scheme to optimize neural network architectures with several conflicting goals.
I can not accept the paper in its current form.
In short, I have the following main criticisms:
1. use of crowding distance(CD) instead of hypervolume-contribution.
CD is not consistent with the HV estimator, especially CD might remove solutions that have a large HV-contribution and thus HV will not increase monotonically. The effect is even visible in Figure 8c) as in iteration 22, HV is decreasing as crowding distance removes a good offspring. In short: Crowding distance should not be used as long as the number of objectives does not prohibit computing the HV-contribution.
2. No good justification of BN. It is unclear to me why BN should be used instead of more iterations at stage 1. In 4.4 BN is only compared to the uniform initialization, but this comparison has no meaning given that we already have an optimized front that improved on the uniformly sampled distribution. To be honest, the samples shown from BN do not look very convincing as a lot of very poor architectures are created.
A proper comparison would be comparing the 2-step approach with only the first step and the same budget. Then we could compare samples from both distributions (either sampling from the front using mutation/crossover or sampling from BN). Also we would have a fair comparison of the obtained fronts and HV-values.
3. Ablation study cross-over
I am not convinced by the results presented. The paper says this is a "small scale" study but does not give the number of iterations/samples. It is clear that in the setup of the mutation operator cross-over might help, simply because it can change many more connections in a single iteration than mutation alone, which is limited to max 1 change. Allowing up to two mutations and no crossover could already proof to be better (orsmaller size of offspring population, see below)
Smaller concerns:
1. The results suggest that the uniform distribution might not be tuned well, as it only covers the "expensive" networks but not the "cheap" networks. A better initialization scheme that covers the x-axis better might already show vastly different results. As the Flop-objective is cheap to compute and does not require simulation, one could expect to tune this offline before initialization.
2. No handling of Noise.
During optimization, the chosen starting point and SGD algorithm will introduce noise into the process. Thus, the final test accuracy will be noisy. As an elitist dominance scheme is used, one might easily end up with an architecture that has a large variance when trained, i.e. when performing a final training pass on the full dataset, the performance might be very different. Moreover, the algorithm might stop convergence towards the true pareto front as it is held back by noisy "good" results. This should be discussed in the paper
3. A single-offspring approach might be better than sampling a full population (or offspring size in the order of parallel instances one can expend to run). 40 sounds excessive given that the sampling distribution is only improved through selection and given that the pareto front approximation appears to include less than 40 elements. This might also affect the results in the ablation study for cross-over: more iterations with reduced offspring size allows for more mutations of successful offspring.
4. Some unclear or wrong wordings:
page 4: "As a consequence[...] the best solution encountered [...] will always be present in the final population. " What do you consider "best" in a 2-objective problem? Do you mean: the best in each objective?
page 6, footnote1: this is not true. even without crossover the selection operator ties the solutions together, an offspring has to beat any point in the population, not necessarily its direct parent.
5. Figure 8a) does not include the state of the art result for CIFAR10, see for example
http://rodrigob.github.io/are_we_there_yet/build/classification_datasets_results.html#43494641522d3130


Discrete latent-variable models, while applicable in a variety of settings, can often be difficult to learn. Sampling discrete latent variables can result in highvariance gradient estimators for two primary reasons: 1. branching on the samples within the model, and 2. the lack of a pathwise derivative for the samples. While current state-of-the-art methods employ control-variate schemes for the former and continuous-relaxation methods for the latter, their utility is limited by the complexities of implementing and training effective control-variate schemes and the necessity of evaluating (potentially exponentially) many branch paths in the model. Here, we revisit the reweighted wake-sleep (RWS) (Bornschein & Bengio, 2015) algorithm, and through extensive evaluations, show that it circumvents both these issues, outperforming current state-of-the-art methods in learning discrete latent-variable models. Moreover, we observe that, unlike the importance weighted autoencoder, RWS learns better models and inference networks with increasing numbers of particles, and that its benefits extend to continuous latentvariable models as well. Our results suggest that RWS is a competitive, often preferable, alternative for learning deep generative models.

Main idea:
This paper studies a problem of the importance weighted autoencoder (IWAE) pointed out by  Rainforth 18, that is, tighter lower bounds arising from increasing the number of particles improve the learning of the generative model, but worsen the learning of the inference network. The authors show that the reweighted wake-sleep algorithm (RWS) doesn't suffer from this issue. Moreover, as an alternative to control variate scheme and reparameterization trick, RWS doesn't suffer from high variance gradients, thus it is particularly useful for discrete latent variable models.
To support the claim, they conduct three experiments: 1) on ATTEND, INFER, REPEAT, a generative model with both discrete and continuous latent variables; 2) on MNIST with a continuous latent variable model; 3) on a synthetic GMM.
Clarity issues:
1. "branching" has been used many times, but AFAIK, this seems not a standard terminology. What do "branching on the samples", "conditional branching", "branching paths" mean?
2. zero-forcing failure mode and delta-WW: I find this part difficult to follow. For example, the following sentence
"the inference network q(z|x) becomes the posterior for this model which, in this model, also has support at most {0, . . . , 9} for all x".
However, this failure mode seems an interesting finding, and since delta-WW outperforms other methods, it deserves a better introduction.
Questions:
1. In Fig 1 (right), how do you estimate KL(q(z|x) || p(z|x))?
2. In Sec 4.2, why do you say IWAE learns a better model only up to a point (K = 128) and suffers from diminishing returns afterwards?
3. In Fig 4, why WS doesn't achieve a better performance when K increasing?
Experiments:
1. Since the motivating story is about discrete latent variable models, better baselines should be compared, e.g. RBM, DVAE, DVAE++, VQ-VAE etc.
2. All experiments were on either on MNIST or synthetic data, at least one large scale experiment on discrete data should be made to verify the performance of RWS.


AGGREGATED MOMENTUM: STABILITY THROUGH PASSIVE DAMPING
Momentum is a simple and widely used trick which allows gradient-based optimizers to pick up speed along low curvature directions. Its performance depends crucially on a damping coefficient β. Large β values can potentially deliver much larger speedups, but are prone to oscillations and instability; hence one typically resorts to small values such as 0.5 or 0.9. We propose Aggregated Momentum (AggMo), a variant of momentum which combines multiple velocity vectors with different β parameters. AggMo is trivial to implement, but significantly dampens oscillations, enabling it to remain stable even for aggressive β values such as 0.999. We reinterpret Nesterov's accelerated gradient descent as a special case of AggMo and analyze rates of convergence for quadratic objectives. Empirically, we find that AggMo is a suitable drop-in replacement for other momentum methods, and frequently delivers faster convergence with little to no tuning.

The paper introduces a variant of momentum that aggregates several velocities with different dampening coefficients. The proposed optimization algorithm can significantly decrease oscillation thus one can use much larger dampening coefficient to achieve faster convergence.
The paper does a good job of motivating the algorithm, both in terms of intuitive reasoning from physics, and some demonstrative examples. The paper also has nice analysis in the simple quadratic case where it tries to make equivalence to the Nesterov's accelerated gradients.
The experiments are also thorough and convincing since it contains various network architectures and different datasets. From the comparison, it seems AggMo consistently achieves the best or comparable performance even in test error/accuracies.
The connection to Nesterov's accelerated gradient and extragradient methods can be discussed in more details. It'll be nice to put the theoretical results in the main text as well.


A VARIATIONAL INEQUALITY PERSPECTIVE ON GENERATIVE ADVERSARIAL NETWORKS
Generative adversarial networks (GANs) form a generative modeling approach known for producing appealing samples, but they are notably difficult to train. One common way to tackle this issue has been to propose new formulations of the GAN objective. Yet, surprisingly few studies have looked at optimization methods designed for this adversarial training. In this work, we cast GAN optimization problems in the general variational inequality framework. Tapping into the mathematical programming literature, we counter some common misconceptions about the difficulties of saddle point optimization and propose to extend techniques designed for variational inequalities to the training of GANs. We apply averaging, extrapolation and a computationally cheaper variant that we call extrapolation from the past to the stochastic gradient method (SGD) and Adam.

This paper looks at solving optimization problems that arise in GANs, via a variational inequality perspective (VIP). VIP entails solving an optimization problem that is related to the first order condition of the optimization problem that we wish to solve. VIP have been very successful in solving min-max style problems. Given that, GAN formulations tend to be min-max style problems (though not necessarily 0 sum) the VIP perspective is very natural, though under-explored in machine learning. Two techniques that have been widely used to solve VIP problems are averaging and extragradient methods. The authors look at a simple GAN setup where both the generator and the discriminator are linear models. In this case two kinds of gradient updates can be derived. First are simultaneous updates, and the other is alternated updates. The authors show that simultaneous updates are not even bounded and diverge to infinity, whereas alternated updates are more stable and stay bounded, but need not necessarily converge. However, I think this behaviour is limited to only linear discriminator/generator and might not extend beyond the linear case. The second key idea is the use of extra-gradient updates. Extra-gradient updates perform an "extra" or fake gradient step to get to a new point, and then kind of retracks back and perform a gradient step using the gradient step obtained from the "extra step".  This extra-gradient method is a close approximation to Euler's method, though far more computationally efficient.  However, the extragradient step requires one to calculate gradient twice, which can be expensive in large models. For this reason, the authors suggest using gradients from past as the "extragradient" in the extragradient method.
For strongly-monotone operators (a generalization of strongly-convex functions) extrapolation updates are shown to have linear convergence.  Furthermore, the authors show that using extrapolation and averaging under the assumption that the operator is monotonic, and using constant step size SGD the rates of convergence are better than the rates obtained using plain SGD with averaging but without extrapolation. Authors also show how one can use these ideas using other first order methods such as ADAM instead of SGD. Experiments are shown on the DCGAN architecture.
On the whole this is a really nice paper, that shows how standard ideas from VIP can be useful for training GANs. I recommend acceptance


Gated recurrent networks such as those composed of Long Short-Term Memory (LSTM) nodes have recently been used to improve state of the art in many sequential processing tasks such as speech recognition and machine translation. However, the basic structure of the LSTM node is essentially the same as when it was first conceived 25 years ago. Recently, evolutionary and reinforcement learning mechanisms have been employed to create new variations of this structure. This paper proposes a new method, evolution of a tree-based encoding of the gated memory nodes, and shows that it makes it possible to explore new variations more effectively than other methods. The method discovers nodes with multiple recurrent paths and multiple memory cells, which lead to significant improvement in the standard language modeling benchmark task. Remarkably, this node did not perform well in another task, music modeling, but it was possible to evolve a different node that did, demonstrating that the approach discovers customized structure for each task. The paper also shows how the search process can be speeded up by training an LSTM network to estimate performance of candidate structures, and by encouraging exploration of novel solutions. Thus, evolutionary design of complex neural network structures promises to improve performance of deep learning architectures beyond human ability to do so.

The authors apply (tree-based) genetic programming (GP) to RNN search, or more specifically RNNs with memory cells, with the foremost example of this being the LSTM. GP provide a structured search that seems appropriate for designing NN modules, and has previously been applied successfully to evolving CNNs. However, the authors fail to mention that (tree-based) GP has been applied to evolving RNN topologies as far back as 2 decades ago, with even multiple cells in a single RNN unit [1]. The selection of more advanced techniques is good though - use of Modi for allowing multiple outputs, and neat-GP for more effective search (though a reference to the "hall of fame" [2] is lacking).
The authors claim that their method finds more complex, better performing structures than NAS, but allow their method to find architectures with more depth (max 15 vs. the max 10 of NAS), so this is an unfair comparison. It may be the case that GP scales better than the RL-based NAS method, but this is an unfair comparison as the max depth of NAS is not in principle limited to 10.
The second contribution of allowing heterogeneity in the layers of the network is rather minimal, but OK. Certainly, GP probably would have an advantage when searching at this level, as compared to other methods (like NAS). Performance prediction in architecture search has been done before, as noted by the authors (but see also [3]), so the particular form of training an LSTM on partial validation curves is also a minor contribution. Thirdly, concepts of archives have been in use for a long time [2], and the comparison to novelty search, which optimises for a hand-engineered novelty criteria, reaches beyond what is necessary. There are methods based on archives, such as MAP-Elites [4], which would make for a fairer comparison. However, I realise that novelty search is better known in the wider ML community, so from that perspective it is reasonable to keep this comparison in as well.
Finally, it is not surprising that GP applied to searching for an architecture for one task does not transfer well to another task - this is not specific to GP but ML methods in general, or more specifically any priors used and the training/testing scheme. That said, prior work has explicitly discussed problems with generalisation in GP [5].
[1] Esparcia-Alcazar, A. I., & Sharman, K. (1997). Evolving recurrent neural network architectures by genetic programming. Genetic Programming, 89-94.
[2] Rosin, C. D., & Belew, R. K. (1995, July). Methods for Competitive Co-Evolution: Finding Opponents Worth Beating. In ICGA (pp. 373-381).
[3] Zhou, Y., & Diamos, G. (2018). Neural Architect: A Multi-objective Neural Architecture Search with Performance Prediction. In SysML.
[4] Mouret, J. B., & Clune, J. (2015). Illuminating search spaces by mapping elites. arXiv preprint arXiv:1504.04909.
[5] Kushchu, I. (2002). An evaluation of evolutionary generalisation in genetic programming. Artificial Intelligence Review, 18(1), 3-14.


It is clear that users should own and control their data and privacy. Utility providers are also becoming more interested in guaranteeing data privacy. Therefore, users and providers can and should collaborate in privacy protecting challenges, and this paper addresses this new paradigm. We propose a framework where the user controls what characteristics of the data they want to share (utility)   and what they want to keep private (secret), without necessarily asking the utility provider to change its existing machine learning algorithms. We first analyze the space of privacy-preserving representations and derive natural informationtheoretic bounds on the utility-privacy trade-off when disclosing a sanitized version of the data X. We present explicit learning architectures to learn privacypreserving representations that approach this bound in a data-driven fashion. We describe important use-case scenarios where the utility providers are willing to collaborate with the sanitization process. We study space-preserving transformations where the utility provider can use the same algorithm on original and sanitized data, a critical and novel attribute to help service providers accommodate varying privacy requirements with a single set of utility algorithms. We illustrate this framework through the implementation of three use cases; subject-withinsubject, where we tackle the problem of having a face identity detector that works only on a consenting subset of users, an important application, for example, for mobile devices activated by face recognition; gender-and-subject, where we preserve facial verification while hiding the gender attribute for users who choose to do so; and emotion-and-gender, where we hide independent variables, as is the case of hiding gender while preserving emotion detection.
will make such devices not to understand the data until the visual or sound trigger is detected, with the capability to do this at the sensor level and without modifying the existing recognition system. This new paradigm of collaborative privacy environment is critical since it has also been shown that algorithmic or data augmentation and unpredictable correlations can break privacy Israel et al. (2014); Narayanan & Shmatikov (2008); Oh et al. (2016); Reuben et al. (2016). The impossibility of universal privacy protection has been studied extensively in the domain of differential privacy Dwork (2008), where a number of authors have shown that assumptions about the data or the adversary must be made in order to be able to provide utility Dwork & Naor (2010); Hardt et al. (2016); Kifer & Machanavajjhala (2011;. We can, however, minimize the amount of privacy we are willing to sacrifice for a given level of utility. Other recent data-driven privacy approaches like Wu et al. (2018) have also explored this notion, but do not integrate the additional collaborative constraints.
Therefore, it is important to design collaborative systems where each user shares a sanitized version of their data with the service provider in such a way that user-defined non-sensitive tasks can be performed but user-defined sensitive ones cannot, without the service provider requiring to change any data processing pipeline otherwise.
Contributions-We consider a scenario where a user wants to share a sanitized representation of data X in a way that a latent variable U can be inferred, but a sensitive latent variable S remains hidden. We formalize this notion using privacy and transparency definitions. We derive an informationtheoretic bound on privacy-preserving representations. The metrics induced by this bound are used to learn such a representation directly from data, without prior knowledge of the joint distribution of the observed data X and the latent variables U and S. This process can accommodate for several user-specific privacy requirements, and can be modified to incorporate constraints about the service provider's existing utility inference algorithms enabling several privacy constraints to be satisfied in parallel for a given utility task.
We apply this framework to challenging use cases such as hiding gender information from a facial image (a relatively easy task) while preserving subject verification (a much harder task), or designing a sanitization function that preserves subject identification on a consenting subset of users, while disallowing it on the general population. Blocking a simpler task while preserving a harder one and blocking a device from constantly listening out-of-sample data are new applications in this work, here addressed with theoretical foundations and respecting the provider's existing algorithms, which can simultaneously handle sanitized and non-sanitized data.
The problem statement is detailed in Section 2, and the information-theoretic bounds are derived in Section 3. Section 4 defines a trainable adversarial game that directly attempts to achieve this bound; the section also discusses how service-provider specific requirements can be incorporated. Examples of this framework are shown in Section 5. The paper is concluded in Section 6. Complementary information and proofs are presented in the Supplementary Material .

[Second update] I'd like to thank the authors for their detailed response. The authors have made changes that I believe improve the overall quality of the submission. I now lean towards accepting the paper, and have increased my rating from a 5 to a 6.
Most notably: (i) they clarified that their secret-detection model was retrained on sanitized data in their experiments, (ii) they added details about their experimental setup and the algorithms used for their experimental evaluation, and (iii) they added experiments to the appendix of the submission that evaluated their framework on synthetic data. I do, however, still have some concerns about how well the privacy guarantees of the proposed algorithm would hold up in practice against a motivated adversary (since formal privacy guarantees appear to be relatively weak right now).
As a minor comment, there may be a typo in Equation 20 of Section 7.2: the case (u, s) = (1, 0) is handled twice, whereas the case (u, s) = (0, 0) is never handled at all.
[First update] I find the authors' problem statement appealing, but share concerns with Reviewer 1 about the privacy guarantees offered by the proposed method, and with Reviewer 3 about need to clarify the experimental evaluation. No author response was provided; I've left my score for the paper unchanged. (Note: this update was posted a few days before the end of the rebuttal period; the submission was subsequently updated.)
[Summary] The authors consider a problem related to de-identification, where the goal is to perturb a dataset X in a way that makes it possible to infer some useful information U about each example in the dataset while obscuring some sensitive information S. For example, the authors consider the problem of perturbing pictures of people's faces to obfuscate the subjects' emotions while making it possible to infer their genders. The concrete approach explored in the paper's experimental evaluation ensures that an existing model trained model on the original dataset will continue to work when applied to the perturbed data.
On the theory side, the authors derive information-theoretic lower bounds on the extent to which one can disclose useful information about a dataset without leaking sensitive information, and propose concrete minimization problems that can perturb the data to trade off between the two objectives. On the practical side, the authors evaluate the minimization setup on three different problems.
[Key Comments] I'm of two minds about this paper. On the whole, I found the problem statement compelling. However, I had serious reservations about the implementation. First: I had trouble understanding the experimental setup based on the limited information provided in Section 5, and the results seem difficult to reproduce from the information in the paper. Second and more seriously: the security guarantees provided in practice seem very weak. At the very least, the authors should check whether their perturbations are robust against an adversary who retrains their model from scratch on perturbed data. This experiment would significantly strengthen the submission, but would still leave open the possibility that a clever adversary could extract more sensitive information than expected from the perturbed data.
[Details]
[Pro #1] The idea of perturbing an input in order to optimize bounds on how much "useful" versus "secret" information is disclosed by the output seems intuitively appealing. In that context, the theory from Sections 2 and 3 seems well-motivated. Section 3.2 ("defining a trainable loss metric") is especially well-motivated. It provides a concrete objective function which, when minimized, can obfuscate data in a way that trades off between utility and secrecy.
[Pro #2] The idea of perturbing a dataset in a way that allows existing useful algorithms to continue working without modifications seems like an interesting and novel contribution. I found the following excerpt from the introduction especially compelling: "it is important to design collaborative systems where each user shares a sanitized version of their data with the service provider in such a way that user-defined non-sensitive tasks can be performed but user-defined sensitive ones cannot, without the service provider requiring to change any data processing pipeline otherwise."
[Pro #3] The paper combines theoretical results with empirical case studies on three different problems. Based on visual inspection, the outputs of the perturbation heuristics shown in Section 5 / Figure 3 and Figure 4 seem reasonable.
[Con #1] Few details are provided about the experimental setup used in Section 5, and it was difficult for me to understand how the theoretical results in Section 4 were actually being applied. There's typically a lot of work that goes into turning a theoretical objective function (e.g., Equation 10 in Section 4.2) into a practical experimental setup. This could be a major contribution of the paper. But right now, I feel like there aren't enough details about the implementation for me to reproduce the experiments.
[Con #2] I had trouble understanding the motivation for the Subject within Subject case study in Section 5.1. The authors describe the problem as follows: "Imagine a subset of users wish to unlock their phone using facial identification, while others opt instead to verify their right to access the phone using other methods; in this setting, we would wish the face identification service to work only on the consenting subset of users, but to respect the privacy of the remaining users." The proposed solution (Figure 3) applies minor perturbations to the pictures of consenting subjects while editing the photos of the non-consenting users to leave only their silhouettes. A simple baseline would be to remove the photos of the non-consenting users from the dataset entirely. The case study would greatly benefit from a discussion of why the baseline is insufficient. It's also perfectly reasonable to say that the section is meant as a way to check whether the objective function from Section 4 can lead to reasonable behavior in practice, but if so, the intent should be clarified.
[Con #3] As far as I can tell, the practical experiments in Section 5 assume that the party who perturbs the dataset knows exactly what algorithm an attacker will use to infer secret information. They also seem to assume that the attacker cannot switch to a different algorithm -- or even retrain an existing machine-learned model -- to try and counter the perturbation heuristics. From the beginning of Section 5: "Initially, we assume that the secret algorithm is not specifically tailored to attack the proposed privatization, but instead is a robust commonly used algorithm trained on raw data to infer the secret." Unless I missed something, it seems like this assumption is used throughout the experimental evaluation.
To the authors' credit, the submission states this assumption explicitly in Section 5. From a security perspective, however, this seems like a dangerous assumption to rely on, as it leaves "sanitized" data vulnerable to attacks. For example, an attacker with knowledge of the perturbation algorithm can retrain the model they use to extract sensitive information, using perturbed images in place of the original images in their training dataset.
My main practical concern is that the security guarantees provided by the submission seem fragile. It may be much easier to build a perturbation algorithm that is resistant to a single (known) attack than to remove the sensitive information from the dataset entirely. Right now, the empirical results in the submission seem to focus on the former.


Separating mixed distributions is a long standing challenge for machine learning and signal processing. Applications include: single-channel multi-speaker separation (cocktail party problem), singing voice separation and separating reflections from images. Most current methods either rely on making strong assumptions on the source distributions (e.g. sparsity, low rank, repetitiveness) or rely on having training samples of each source in the mixture. In this work, we tackle the scenario of extracting an unobserved distribution additively mixed with a signal from an observed (arbitrary) distribution. We introduce a new method: Neural Egg Separation -an iterative method that learns to separate the known distribution from progressively finer estimates of the unknown distribution. In some settings, Neural Egg Separation is initialization sensitive, we therefore introduce GLO Masking which ensures a good initialization. Extensive experiments show that our method outperforms current methods that use the same level of supervision and often achieves similar performance to full supervision.

This paper describes a signal separation method called neural egg separation (NES).
The separation problem is tackled in a semi-supervised setting where the observed mixture contains a target signal and a background noise, with access to the distributions of target and mixture signals.
The strength of the paper is that it describes the importance of the problem setup for practical use with some motivating examples.
However, some unclear notations weaken the claim of the paper.
Specific comments follow.
* The loss in (1) is unclear.
Assuming latex grammar, \| \| is usually used to denote a vector norm, but (1) has two values inside.
I would write \ell(T(y_i), b_i) to show a loss function, instead of the \| \| style.
More importantly, the loss should be explicitly defined. Does this mean the l2 error?
* The iterative separation process of (2) is even unclear.
Does T^m(b_j + x_i^m) share the parameter of that from previous iterations like T^{m-1}?
Or are the parameters fixed throughout the iterations?
* Use of \cdot.
There may be a confusion between the inner product and element-wise product with the \cdot operator.
Right after (5), there is an inequality z \cdot z \leq 1, which is meant to be the inner product.
On the other hand, the use of \cdot in (8) looks like the element-wise product to describe a masking operation.
Clarifying the objective and overall procedures is necessary for presenting the proposed method.
=================================
EDIT: I confirmed the revisions regarding the notation issues, but there still have confusing parts.
* Definitions of norm operator \| \| is unclear.
  * L_1 is mentioned below (1), and used other parts (3) or Algorithm 1. Equation (12) in Appendix uses |W1|_1^2, which looks like the l1 norm as well. Use consistent notations.
  * Equations (12, 13, 14) uses \|\|_2 or \|\|_1 to specify the type of norm, whereas (5), (6), (7) and other parts after (15) use \|\|. This confuses me. What do you mean by \|\| without subscript?
  * \|\| operator taking to symbols is a weird notation for me. Usually, norm is defined for a single vector (or a matrix). For example in (5), I would write \| b - G(z_b) \|, if you want to measure the difference between b and G(z_b).
The experimental result is impressive, as the other reviewers mention. I strongly recommend clarifying the notation to better deliver the method.


We propose the factorized action variational autoencoder (FAVAE), a state-of-theart generative model for learning disentangled and interpretable representations from sequential data via the information bottleneck without supervision. The purpose of disentangled representation learning is to obtain interpretable and transferable representations from data. We focused on the disentangled representation of sequential data since there is a wide range of potential applications if disentanglement representation is extended to sequential data such as video, speech, and stock price data. Sequential data is characterized by dynamic factors and static factors: dynamic factors are time dependent, and static factors are independent of time. Previous works succeed in disentangling static factors and dynamic factors by explicit modeling the priors of latent variables to distinguish between static and dynamic factors. However, this model cannot disentangle representations between dynamic factors, such as disentangling "picking" and "throwing" in robotic tasks. In this paper, we propose new model that can disentangle multiple dynamic factors. Since our method does not require modeling priors, it is capable of disentangling "between" dynamic factors. In experiments, we show that FAVAE can extract the disentangled dynamic factors.

This paper presents a new approach to learning disentangled representations of sequential data. The model, FAVAE, is based on the information bottleneck framework (Alemi et al, 2016; Achille et al, 2016) and extends the recent beta-VAE (Higgins et al, 2017) and CCI-VAE (Burgess et al, 2017) work by changing the encoder/decoder to a Quasi-Recurrent Neural Network (QRNN) and adding multiple latents through a ladder VAE approach (Zhao et al, 2017). The authors demonstrate that their approach is able to learn a more disentangled representation than the limited set of baselines on three toy datasets.
The authors address a very important problem, since the ability to learn disentangled representations of sequential data can have far reaching consequences, as the authors rightfully point out in their introduction. I also like the approach that the authors are taking, which appears to be very general and does not seem to involve the need to have any domain knowledge. However, the paper could be improved by clarifying certain key parts and extending the experimental section, which is currently not quite as convincing as I would have hoped.
The summary of my concerns is the following:
I would like to see more baseline comparisons 1) to an FAVAE with a different recurrent encoder/decoder; 2) to at least one other approach to disentangled representation learning on sequences; 3) an FAVAE without the capacity increase. I would also like to see all the baselines run on a non toy dataset of video data. Finally, I would like to see an expanded discussion of what the different latent variables at the different levels of the ladder architecture are learning. I recommend that the authors remove the MIE metric and shorten Section 3 to make space for the expanded experiments section.
I do hope that the authors are able to address my concerns, because their method has a lot of potential and I am excited to see where they take it during the rebuttal period. Please see the more detailed comments below:
1) Section 4.2 should be expanded to include a more detailed description of QRNN. This is one of the key modifications of FAVAE compared to CCI-VAE or beta-VAE, and it is currently not clear how QRNN works unless one reads the original paper referenced. The current paper needs to be self contained. It would also be nice to get a better understanding why QRNN was chosen over an LSTM or a GRU. It would be useful to see the results of the baseline experiments with an LSTM compared to the equivalent QRNN-based version of FAVAE.
2) Why do the authors introduce the new MIE metric? The reported results do not show a significant problem with the MIG metric, and the need for the new MIE metric is not well motivated. If the authors insist on introducing this new metric, it is important to demonstrate cases where MIG fails and MIE performs better. Otherwise I would advise removing the new metric and using the space to expand section 4.2 instead.
Another point on the metric, Eq. 16 seems to be missing a term that goes over latents z_j. I assume there should be either a max_j or a mean_j in front of the I(z_j; v_k) term in the first part of Eq. 16?
3) The related work section should be re-worded. Currently it reads as if the other approaches do not optimise a loss function at all. It would also be good to include one of the mentioned models as a baseline, and to run both FAVAE and one of the previously introduced models for disentangled sequential representation learning on a more complex dataset of video data.
4) In section 7.1, it would be good to expand the discussion of how latent traversal plots are obtained. In particular, I do not understand how the different latent variables in the ladder architecture of FAVAE are traversed. In general, it would also be nice to expand the discussion of what the different latents at the different ladder stages learn, and how the number of ladder stages affects the nature of learnt representations.
5) In terms of the baselines, it would be good to see the full ablation study. The way I see it, FAVAE has three modifications on the standard variational approaches: 1) the use of a recurrent encoder/decoder (R); 2) the use of the ladder architecture (L); and 3) the use of the capacity constraint objective (C). Currently the authors show the results of the R--, R-C, and RLC conditions. I would also like to see the results of the RL- condition (where beta=1 and C=0).
6) In terms of the results presented in Tbl. 1, it would be nice to include the hyperparameters that the authors have swept over in the appendix, as well as a table of the architecture parameters and the best hyperparameters found for the models presented in the Experiments section. In Tbl.1 it is currently unclear how many seeds the authors used to calculate the standard deviations. The units of the standard deviations presented are also not clear. Finally, it is not clear whether the differences presented in the table are significant.
7) It would be useful to include some details of the 2D Wavy Reaching dataset in the main text, even if it is just listing the nature of the 5 factors.
8) It would be useful to expand the section that talks about the different settings of C explored (page 7, paragraph 2). Currently the point that the authors are trying to make in that paragraph and Fig. 4 is not clear. I would also recommend having beta in Fig. 4 on linear rather than log scale, as the log scale seems to be somewhat confusing.

Minor points:
-- Page 2/paragraph 2: used disentangle -> used to disentangle
-- P4/p5: FAVAE is disentangled representation -> is for(?) disentangled representation
-- P6/p1: the priors time dependency -> the priors' time dependency
-- P7/p1: scores for2D -> scores for 2D
-- P7/p4: MIE score (gray curve) -> MIE score (green(?) curve)


Learning representations for counterfactual inference from observational data is of high practical relevance for many domains, such as healthcare, public policy and economics. Counterfactual inference enables one to answer "What if...?" questions, such as "What would be the outcome if we gave this patient treatment t 1 ?". However, current methods for training neural networks for counterfactual inference on observational data are either overly complex, limited to settings with only two available treatment options, or both. Here, we present Perfect Match (PM), a method for training neural networks for counterfactual inference that is easy to implement, compatible with any architecture, does not add computational complexity or hyperparameters, and extends to any number of treatments. PM is based on the idea of augmenting samples within a minibatch with their propensity-matched nearest neighbours. Our experiments demonstrate that PM outperforms a number of more complex state-of-the-art methods in inferring counterfactual outcomes across several real-world and semi-synthetic datasets.

========= Summary =========
The authors propose a novel method for counterfactual inference (i.e. individual/heterogeneous treatment effect, as well as average treatment effect) with neural networks. They perform propensity score matching within each minibatch in order to match the covariate distributions during training, which leads to a doubly robust model.
PM is evaluated on several standard semi-synthetic datasets (jobs, IHDP, TCGA) and PM shows state-of-the-art performance on some datasets, and overall looks quite promising.
======= Comments =======
The paper is well-written, presents a novel method of some interest to the community, and shows quite good performance across a range of relevant benchmarks.
I have one major issue with this work: I don't see why propensity-score matching *within* a minibatch should provide a substantial improvement over propensity-score matching across the dataset (Ho et al 2011). I find the cursory explanation given ("it ensures that every gradient step is done in a way that is approximately unbiased") unconvincing, since (a) proper SGD training should be robust to per-batch biases during training (the expected loss is identical for both methods, correct?), and (b) biases should go away in the limit of large batch sizes. If indeed SGD required unbiased *minibatches* then standard minibatch SGD wouldn't work at all.
Looking at the experimental details in the appendix, it appears that the MatchIt package was used to do PSM, rather than a careful comparison under the same conditions. Are the exact matching procedure, PS estimator model, choosing "one of 6 closest  matches by propensity score", batch size, etc. the same between your PM implementation and MatchIt? I'd be very curious to see the results of a controlled comparison between Alg S1 and S2 under the same conditions (i.e. run your PM implementation on the whole dataset), and perhaps even some more clever experiments illustrating why matching within a minibatch is important.
Another hypothesis for why PM is better than PSM is that the matching distribution for PM changes at each epoch (at least due to the randomization among the 6 closest matches). Could it be that the advantage of PM is that it actually provides a randomized rather than constant distribution of matched points?
Can the authors provide more motivation for why PM should outperform PSM? Or some more careful comparison of these methods isolating the benefits of PM? I think a convincing justification and comparison here could change my opinion, as I like the paper otherwise. Thanks!
Detailed Comments:
- There is insufficient explanation of the PM method in the main text. The method is only mentioned in a single sentence buried in the middle of a long paragraph "In PM, we match every sample within a minibatch...". This should be made more clear, e.g. by moving Algorithm S1 to the main text.
- The discussion on Model Selection and the argument for nearest-neighbor PEHE is clever and well-supported by the experiments.
- In Table 3 and 4, it's not clear which numbers are reported by the original authors and which were replicated by the authors.


COARSE-GRAIN FINE-GRAIN COATTENTION NET- WORK FOR MULTI-EVIDENCE QUESTION ANSWERING
End-to-end neural models have made significant progress in question answering, however recent studies show that these models implicitly assume that the answer and evidence appear close together in a single document. In this work, we propose the Coarse-grain Fine-grain Coattention Network (CFC), a new question answering model that combines information from evidence across multiple documents. The CFC consists of a coarse-grain module that interprets documents with respect to the query then finds a relevant answer, and a fine-grain module which scores each candidate answer by comparing its occurrences across all of the documents with the query. We design these modules using hierarchies of coattention and selfattention, which learn to emphasize different parts of the input. On the Qangaroo WikiHop multi-evidence question answering task, the CFC obtains a new stateof-the-art result of 70.6% on the blind test set, outperforming the previous best by 3% accuracy despite not using pretrained contextual encoders.

This paper proposes an interesting coarse-grain fine-grain coattention network architecture to address multi-evidence question answering and achieves the new state-of-the-art results on the Qangaroo WikiHop dataset.  The main idea is to divide the task across the coarse-grain and fine-grain modules in a complimentary manner such that the coarse-grain module learns from efficient modeling of support documents and the query whereas the fine-grain module learns from associations of candidate mentions in the support documents with the query.
The major strength of the model is observed with learning effective representations of larger numbers of long support documents and the state-of-the-art results are achieved without the use of pretrained contextualized embeddings. The main novelty lies in how the coattention and self-attention strategies are combined hierarchically to learn relevant representations in a complimentary fashion (rather than serial). Overall, the paper is very well-written and presents solid results with meaningful ablation study, quantitative and qualitative analyses. I have a few comments/suggestions:
- It would be interesting to see how the inclusion of pretrained contextualized embeddings such as ELMo, ULMFit, BERT would help the current model.
- "This is likely because coreference resolution captures intra-document and inter-document dependencies more accurately than hierarchical attention." --> Please clarify why this is the case.
- "We hypothesize that ways to reduce this type of error include using more robust pretrained contextual encoders (McCann et al., 2017; Peters et al., 2018) and coreference resolution." --> I agree; also, it would be worth considering some commonsense knowledge to alleviate this issue, because the fact that Scotland is a part of UK and has a border with England should be learned. Here is a relevant work: "Knowledgeable Reader: Enhancing Cloze-Style Reading Comprehension with External Commonsense Knowledge" by Mihaylov and Frank, 2018.
- "The second type (28% of errors) results from questions that are not answerable. For example, the support documents do not provide the narrative location of the play “The Beloved Vagabond” for the query narrative location the beloved vagabond." --> It would be great if you could release the set of unanswerable questions for the community.
- Please include the memory network-based QA works in the related work section because they involve some forms of reasoning. Also, I would suggest to cover the query-focused multi-document summarization area in the related work section because they also require evidence synthesis from multiple documents to address a query. It would be very interesting if authors can apply their model for the query-focused multi-document summarization task as well, as this would further validate the effectiveness of the proposed architecture for reasoning across multiple documents.


Recent advances in Generative Adversarial Networks facilitated by improvements to the framework and successful application to various problems has resulted in extensions to multiple domains. IRGAN attempts to leverage the framework for Information-Retrieval (IR), a task that can be described as modeling the correct conditional probability distribution p(d|q) over the documents (d), given the query (q). The work that proposes IRGAN claims that optimizing their minimax loss function will result in a generator which can learn the distribution, but their setup and baseline term steer the model away from an exact adversarial formulation, and this work attempts to point out certain inaccuracies in their formulation. Analyzing their loss curves gives insight into possible mistakes in the loss functions and better performance can be obtained by using the co-training like setup we propose, where two models are trained in a co-operative rather than an adversarial fashion.

This paper tries to argue that the formulation of IRGAN (a method from 2017 that aimed to use GANs for the standard IR task of estimation query-document relevance) is now well-founded and has inherent weaknesses. Specifically the paper claims that (unlike regular GANs and what was likely intended by the authors or IRGAN) the discriminator and generator are working against each other. The paper then aims to show a couple of more well-founded different (generator-free) setups that perform about as well (if not better) as the original IRGAN work.
Overall I found the work to be quite interesting and the findings to be illuminating. That said I think the paper notably lacked rigor and depth which definitely hurt the quality of the paper.
Below are my thoughts on the different facets as well as more detailed strengths / weaknesses breakdown:
Quality: Above average
As mentioned I think some of the findings are illuminating and thus overage the paper scores well on this aspect.
Clarity: Slightly above average
While the paper is largely easy to follow, there are certain key sections that are not well explained / have fundamental errors.
Originality: Strong
Significance: Little below average
My (main) concern here with this work is that the missing rigor and depth of the work is what is needed for readers to have a deeper understanding of the fundamental issue so as to be able to rectify it in future works.
---
Strengths / Things I liked about the work:
+ The topic / theme of the work: I believe as a community we should encourage more such works that take a critical deep dive into recently proposed methods that may have some inherent weaknesses. As the authors noted the IRGAN work has become quite popular despite some of these (previously unknown) issues.
+ The experimental results in general do a fair job illustrating the likely issue (though I would have liked to see more rigor and depth here as well as detailed below)
Weaknesses / Things that concerned me:
- (W1) Lacking rigor / depth: One of my big concerns with this work is that the analysis to demonstrate the inherent flaws of IRGAN if fairly shallow and not detailed enough. For example, Section 5 (which should have been the key section of the work) is quite poorly written and not rigorous enough. Claiming that log(1-z) can be replaced with - log(z) is incorrect -- how can this substitution be made as is?
Overall my sense after reading the work is that I understand that the IRGAN formulation is not completely well-formed in terms of discriminator/generator synergy (the pairwise formulation has the additional issue of separating real pairs rather than higher rank-lower rank pairs). However I do not buy that the generator and discriminator directly oppose each other as is claimed in the work (I believe this arises only due to the incorrect claim that log(1-z) can be replaced with -log(z)).
Thus at the end of the day I feel the reader is willing to buy there is a issue with the formulation, but they do not fully understand it not do they understand enough to understand how to rectify the underlying issues. To me that was unfortunate as the paper would have been an excellent work if it had done so.
- (W2) Missing some experimental results / deeper insights : There were some notable empirical results that were missing or not provided, that raised some concerns in my mind. For instance I don't see the co-training approach listed for the MovieLens dataset .. Why so? The authors make a secondary claim that they are able to improve upon IRGANs via their proposed approach but then they do not substantiate these on all the datasets which seems like a notable oversight.
 - (W3) Missing details: To add to the above I think the authors can clearly be more detailed in describing for instance the models for D, G, p_\psi etc .. Right now I can speculate what they are but I don't think a reader should be expected to speculate in such cases. Likewise empirical details about the datasets and their sizes could easily have been added.
Also the paper presents the IRGAN pairwise approach and mentions pairs in a couple of places but I don't see an approach that can learn from pairs among the ones proposed.
Another example is the two proposed models Fig 2a (Only discriminator) vs Fig 2b (cotraining). I don't see an explanation or intuition for why 2b is expected to be better than 2a. Given the claims of the work I would have wanted to understand this better.
- (W4) Significance testing: This is an important experimental process to understand the validity of some of the claims. While I understand it is not the main claim of the paper, understand the significance of these differences helps put things in perspective. I would strongly urge the authors to add this for all of their experimental results not just the ones the proposed models are outperformed by IRGANs.
- Lastly I would urge the authors to be rigid and clear in their notations. For example in the equation in section 4.5, "o'" occurs out of nowhere.


ADVERSARIAL IMITATION VIA VARIATIONAL INVERSE REINFORCEMENT LEARNING
We consider a problem of learning the reward and policy from expert examples under unknown dynamics. Our proposed method builds on the framework of generative adversarial networks and introduces the empowerment-regularized maximum-entropy inverse reinforcement learning to learn near-optimal rewards and policies. Empowerment-based regularization prevents the policy from overfitting to expert demonstrations, which advantageously leads to more generalized behaviors that result in learning near-optimal rewards. Our method simultaneously learns empowerment through variational information maximization along with the reward and policy under the adversarial learning formulation. We evaluate our approach on various high-dimensional complex control tasks. We also test our learned rewards in challenging transfer learning problems where training and testing environments are made to be different from each other in terms of dynamics or structure. The results show that our proposed method not only learns nearoptimal rewards and policies that are matching expert behavior but also performs significantly better than state-of-the-art inverse reinforcement learning algorithms.

The paper proposes a method for inverse reinforcement learning based on AIRL. It's main contribution is that the shaping function is not learned while training the discriminator, but separately as an approximation of the empowerment (maximum mutual information). This shaping term aims to learn disentangled rewards without being restricted to learning state-only reward functions, which is a major restriction of AIRL.
The main weakness of the paper is, that it does not justify or motivate the main deviations compared to AIRL. The new objective for updating the policy is especially problematic because it does no longer correspond to the RL objective but includes an additional term that biases the policy towards actions that increase its empowerment. Although both terms of the update can be derived independently from an IRL and Empowerment perspective respectively, optimizing the sum was not derived from a common problem formulation. By combining these objectives, the learned reward function may lead to policies that fail to match the expert demonstration without such bias. This does not imply that the approach is not sound per se, however, simply presenting such update without any discussion is insufficient--especially given that it constitutes the main novelty of the approach. I think the paper would be much stronger if the update was derived from an empowerment-regularized IRL formulation. And even then, the implications of such bias/regularization would need to be properly discussed and evaluated, in particular with respect to the trade-off lambda, which--again--is hardly mentioned in the submission. I'm also not sure if the story of the paper works out; when we simply want to use empowerment as shaping term, why not use two separate policies for computing the empowerment and reward function respectively. Is the bias in the policy update maybe more important than the shaping term in the discriminator update for learning disentangled rewards?
Keeping these issues aside, I actually like the paper. It tackles the main drawback of AIRL and the idea seems quite nice. Having a reward function that does not actively induce actions that can be explained by empowerment, may not always be appropriate, but often enough it may be a sensible approach to get more generalizable reward functions. The paper is also well written with few typos. The parts that are discussed are clear and the experimental results seem fine as well (although more experiments on the reward transfer would be nice).
Minor notes:
I think there is a sign error in the policy update
Typo in the theorem, grantee should be guarantee
Question:
Please confirm that the reward transfer was learned with a standard RL formulation. Does the learned policy change, when we use the empowerment objective as well?
Update (22.11)
I think that the revised version is much better than the original submission because it now correctly attributes the improved generalization to an inductive bias in the policy update.  However, the submission still seems borderline to me.
- The proposed method uses the empowerment both for regularization as well as for reward shaping, but it is not clear whether the latter improves generalization. If the reward shaping was not necessary, it would be cleaner to use empowerment only for regularization. If the reward shaping is beneficial, this should be shown in an ablative experiment.
- The benefit of using empowerment (whether for reward shaping or for regularization) should be discussed. Empowerment for generalization is currently hardly motivated.
- The derivation could be a bit more rigorous.
As the presentation is now much more sound, I slightly increased my rating.


We propose the Information Maximization Autoencoder (IMAE), an information theoretic approach to simultaneously learn continuous and discrete representations in an unsupervised setting. Unlike the Variational Autoencoder framework, IMAE starts from a stochastic encoder that seeks to map each input data to a hybrid discrete and continuous representation with the objective of maximizing the mutual information between the data and their representations. A decoder is included to approximate the posterior distribution of the data given their representations, where a high fidelity approximation can be achieved by leveraging the informative representations. We show that the proposed objective is theoretically valid and provides a principled framework for understanding the tradeoffs regarding informativeness of each representation factor, disentanglement of representations, and decoding quality.

Summary: the paper proposes a method for unsupervised disentangling of both discrete and continuous factors of variation in image data. It uses an autoencoder learned by optimising an additive loss composed of Mutual Information (MI) I(x;y,z) between the image x and the discrete+cts latents (y,z) and the reconstruction error. The mutual information is shown to decompose into I(x,y), I(x,z) and TC(y;z), and the I(x,z) is treated in a different manner to I(x,y). With Gaussian p(z|x), and it is shown that I(x,z_k) is maximal when p(z_k) is Gaussian. So KL(p(z_k)||N(0,1)) is optimised in lieu of optimising I(x,z), and I(x,y) (and TC(y;z)) is optimised by using mini-batch estimates of marginal distributions of y (and z). The paper claims improved disentangling of discrete and continuous latents compared to methods such as JointVAE and InfoVAE.
Pros:
- The derivation of the loss shows a nice link between Mutual information and total correlation in the latents.
- It is a sensible idea to treat the MI terms of the discrete latents differently to the continuous latents
- The mathematical and quantitative analysis of MI and its relation to decoder means and variances are informative.
Cons:
- There is not enough quantitative comparison of the quality of disentanglement across the different methods. The only values for this are the accuracy scores of the discrete factor, but for the continuous latents there are only qualitative latent traversals of single models, and I think these aren’t enough for comparing different disentangling methods - this is too prone to cherry-picking. I think it’s definitely necessary to report some metrics for disentangling that are averaged across multiple models trained with different random seeds. I understand that there are no ground truth cts factors for Mnist/FashionMnist, but this makes me think that a dataset such as dSprites (aka 2D Shapes) where the factors are known and has a mix of discrete and continuous factors would have been more suitable. Here you can use various metrics proposed in Eastwood et al, Kim et al, Chen et al for a quantitative comparison of the disentangled representations.
- In figure 4, it says beta=lamda=5 for all models. Shouldn’t you be doing a hyperparameter sweep for each model and choose the best value of hyperparameters for each? It could well be that beta=5 works best for IMAE but other values of beta/lambda can work better for the other models.
- When comparing against JointVAE, the authors point out that the accuracy for JointVAE is worse than that of IMAE, a sign of overfitting. You also say that VAT helps maintain local smoothness so as to prevent overfitting. Then shouldn’t you also be comparing against JointVAE + VAT? Looking at Appendix D, it seems like VAT makes a big difference in terms of I(y;y_true), so I’m guessing it will also have a big impact on the accuracy. Thus JointVAE + VAT might beat IMAE in terms of accuracy as well, at which point it will be hard to argue that IMAE is superior in learning the discrete factor.
- In the first paragraph of Section 4, the authors claim results on CelebA, but these are missing from the paper. Testing the approach on datasets more complex than (Fashion)Mnist would have been desirable.
- There aren’t any latent traversals for the discrete latents - this would be a useful visualisation to complement the accuracy plots in Figure 3.
Qs and comments:
- It’s not clear why posterior approximation quality (used as a starting point for motivating the loss) is an important quantity for disentangling.
- I see that the upper bound to I(x;z_k) in (4) and the objective in (6) have the same optimum at p(z_k) being Gaussian, but it’s not clear that increasing one leads to increasing the other. Using (6) to replace (4) seems to require further justification, whether it be mathematical or empirical.
- In proposition 2, I’m sceptical as to how meaningful the derived bound is, especially when you set N to be the size of the minibatch (B) in practice. It also seems that for small delta (i.e. to ensure high probability on the bound) and large K_2 (less restrictive conditions on p(y) and \hat{p}(y)), the bound can be quite big.
- \mathcal{L}_theta(y) in equation (10) hasn’t been introduced yet.
- The z dimension indices in the latent traversal plots of Figure 2 don’t seem to match the x-axis of the left figure. It’s not clear which are the estimates of I(x;z_k) for k=8,3,1 in the figure.


NEURAL TTS STYLIZATION WITH ADVERSARIAL AND COLLABORATIVE GAMES
The modeling of style when synthesizing natural human speech from text has been the focus of significant attention. Some state-of-the-art approaches train an encoder-decoder network on paired text and audio samples x txt , x aud by encouraging its output to reconstruct x aud . The synthesized audio waveform is expected to contain the verbal content of x txt and the auditory style of x aud . Unfortunately, modeling style in TTS is somewhat under-determined and training models with a reconstruction loss alone is insufficient to disentangle content and style from other factors of variation. In this work, we introduce an end-to-end TTS model that offers enhanced content-style disentanglement ability and controllability. We achieve this by combining a pairwise training procedure, an adversarial game, and a collaborative game into one training scheme. The adversarial game concentrates the true data distribution, and the collaborative game minimizes the distance between real samples and generated samples in both the original space and the latent space. As a result, our model delivers a highly controllable generator with disentangled representation. Benefiting from the separate modeling of style and content, our model can generate human fidelity speech that satisfies the desired style conditions. Our model achieves start-of-the-art results across multiple tasks, including style transfer (content and style swapping), emotion modeling, and identity transfer (fitting a new speaker's voice).

This paper proposes a method to synthesize speech from text input, with the style of an input voice provided with the text. Thus, we provide content - text - and style - voice. It leverages recent - phenomenal - progress in TTS with Deep Neural Networks as seen from exemplar works such as Tacotron (and derivatives), DeepVoice, which use seq2seq RNNs and Wavenet families of models. The work is extremely relevant in that audio data is hard to generate (expensive) and content-style modeling could be useful in a number of practical areas in synthetic voice generation. It is also quite applicable in the related problem of voice conversion. The work also uses some quite complex - (and very interesting!) - proposals to abstract style, and paste with content using generative modeling. I am VERY excited by this effort in that it puts together a number of sophisticated pieces together, in what I think is a very sensible way to implement a solution to this very difficult problem. However, I would like clarifications and explanations, especially in regards to the architecture.
Description of problem: The paper proposes a fairly elaborate setup to inject voice style (speech) into text. At train time it takes in text samples $x_{txt}$, paired voice samples (utterances that have $x_{txt}$ as content) $s+$ and unpaired voice samples $s-$, and produces two voice samples $x+$ (for paired  <txt, utterance>) and $x-$ (for unpaired txt/utterance). The idea is that at test time, we pass in a text sample $x_{txt}$ and an UNPAIRED voice sample $x_{aud}$ and the setup produces voice in the style of $x_{aud}$ but whose content is $x_{txt}$, in other words it generates synthetic speech saying $x_{txt}$. The paper goes on to show performance metrics based on an autoencoder loss, WER and t-SNE embeddings for various attributes.
Context:  The setup seems to be built upon the earlier work by Taigman et al (2016) which has the extremely interesting conception of using a {\it ternary} discriminator loss to carry out domain adaptation between images. This previous work was prior to the seminal CycleGAN work for image translation, which many speech works have since used. Interestingly, the Taigman work also hints at a 'common' latent representation a la UNIT using coupled VAE-GANs with cycle consistency (also extremely pertinent), but done differently. In addition to the GAN framework by Taigman et al, since this work is built upon Tacotron and the GST (Global Style Tokens) work that followed it, the generative setup is a sophisticated recurrent attention based seq2seq model.
Formulation:
A conditional formulation is used wherein the content c (encoding generated by text) is passed along with other inputs in the generator and discriminator. The formulation in Taigman assumes that there is an invariant representation in both (image) domains with shared features. To this, style embeddings (audio) gets added on and then gets passed into the generator to generate the speech. Both c and s seem to be encoder outputs in the formulation. The loss components of what they call ‘adversarial’, ‘collaborative’ and ‘style’ losses.
Adversarial losses
The ternary loss for D consists of
Discriminator output from ‘paired’ style embedding (i.e. text matching the content of paired audio sample)
Discriminator output from ‘unpaired’ style embedding (i.e text paired with random sample of some style)
Discriminator output from target ground truth style. The paper uses x_+, so I would think that it uses the paired sample (i.e. from the source) style.
Generator loss (also analogous to Taigman et al) consists of generations from paired and unpaired audio, possibly a loose analogue to source and target domains, although in this case we can’t as such think of ‘+’ as the source domain, since the input is text.
Collaborative losses
This has two components, one for style (Gatys et al 2016) and a reconstruction component. The reconstruction component again has two terms, one to reconstruct the paired audio output ‘x+=x_audio+’ - so that the input content is reproduced -  and the other to encourage reconstruction of the latent code.
Datasets and Results:
They use two datasets: one, an internal ‘EMT-4’ dataset with 20k+ English speakers, and the other, the VCTK corpus. Comparisons are made with a few good baselines in Tacotron2, GST and DeepVoice2.
One comparison technique to test disentanglement ability is to compare autoencoder reconstructions with the idea that a setup that has learnt to disentangle would produce higher reconstruction error because it has learnt to separate style and content.
t-SNE embeddings are presented to show visualizations of various emotion styles (neutral, angry, sad and happy), and separation of male and female voices. A WER metric is also presented so that generations are passed into a classifier (an ASR system trained on Wavenet). All the metrics above seem to compare excellently (better than?) with the others.
Questions and clarifications:
(Minor) There’s a typo in page 2, line 2. x_{aud}^+ should be x_{aud}^-.
Clarification on formulation: Making the analogy (is that even the right way of looking at this?) that the ‘source’ domain is ‘+’, and the target domain is ‘-’, in equation (5), the last term of the ternary discriminator has the source domain (x_{aud}^+) in it, while the Taigman et al paper uses the target term. Does this matter? I would think ‘no’, because we have a large number of terms here and each individual term in and of itself might not be relevant, nor is the current work a direct translation of the Taigman et al work. Nevertheless, I would like clarification, if possible, on the discrepancy and why we use the ‘+’ samples.
Clarification on reconstruction loss: I think the way it is presented, equation (8) is misleading. Apparently, we are sampling from the latent space of style and content embeddings for paired data. The notation seems to be quite consistent with that of the VAE, where we have a reconstruction and a recognition model, and in effect the equation (8) is sampling from the latent space in a stochastic way. However, as far as I can see, the latent space here produces deterministic embeddings, in that c = f(x_{txt}) and s = g(x_{aud}^+), with the distribution itself being a delta function. Also, the notation q used in this equation most definitely indicates a variational distribution, which I would think is misleading (unless I have misinterpreted what the style tokens mean). At any rate, it would help to show how the style token is computed and why it is not deterministic.
Clarification on latent reconstruction loss: In equation (9), how is the latent representation ‘l’ computed? While I can intuitively see that the latent space ‘l’ (or z, in more common notation) would be the ‘same’ between real audio samples and the ‘+’, ‘-’ fake samples, it seems to me that they would be related to s (as the paper says, ‘C’ and ‘Enc_s’ share all conv layers) and the text. But what, in physical terms is it producing? Is it like the shared latent space in the UNIT work, or the invariant representation in Taigman? This could be made clearer with an block diagram for the architecture.
(Major) Clarification on network architecture
The work references Tacotron’s GST work (Wang et al 2018) and the related Skerry-Ryan work as the stem architecture with separate networks for style embeddings and for content (text). While the architecture itself might be available in the stem work by Wang et al, I think we need some diagrams for the current work as well for a high level picture. Although it is mentioned in words in section 3.3, I do not get a clear idea of what the encoder/decoder architectures look like. I was also surprised in not seeing attention plots which are ubiquitous in this kind of work. Furthermore, in the notes to the ‘inference’ network ‘C’ it is stated that C and Enc_s share all conv layers. Again, a diagram might be helpful - this also applies for the discriminator.
Clarification on stability/mode collapse: Could the authors clarify how easily this setup trained in this adversarial setup?
Note on latent representation: To put the above points in perspective, a small note on what this architecture does in regards to the meaning of the latent codes would be useful. The Taigman et al 2016 paper talks about the f-constancy condition (and 'invariance'). Likewise, in the UNIT paper by Ming-Yu Liu - which is basically a set of coupled VAEs + cycle consistency losses, there is the notion of a shared latent space. A little discussion on these aspects would make the paper much more insightful to the domain adaptation practitioner.
Reference: This reference - Adversarial feature matching for text generation - (https://arxiv.org/abs/1706.03850) contains a reconstruction stream (as perhaps many other papers) and might be useful for instruction.
Other relevant works in speech and voice conversion: This work comes to mind, using the StarGAN setup, also containing a survey of relevant approach in voice conversion. Although the current work is for TTS, I think it would be useful to include speech papers carrying out domain adaptation for other tasks.
StarGAN-VC: Non-parallel many-to-many voice conversion with star generative adversarial networks.
https://arxiv.org/abs/1806.02169
I would rate this paper as being acceptable if the authors clarify my concerns, and in particular, about the architecture. It is also hard to hard to assess reproducibility in a complex architecture such as this.


LEARNING TO ADAPT IN DYNAMIC, REAL-WORLD ENVIRONMENTS THROUGH META-REINFORCEMENT LEARNING
Although reinforcement learning methods can achieve impressive results in simulation, the real world presents two major challenges: generating samples is exceedingly expensive, and unexpected perturbations or unseen situations cause proficient but specialized policies to fail at test time. Given that it is impractical to train separate policies to accommodate all situations the agent may see in the real world, this work proposes to learn how to quickly and effectively adapt online to new tasks. To enable sample-efficient learning, we consider learning online adaptation in the context of model-based reinforcement learning. Our approach uses meta-learning to train a dynamics model prior such that, when combined with recent data, this prior can be rapidly adapted to the local context. Our experiments demonstrate online adaptation for continuous control tasks on both simulated and real-world agents. We first show simulated agents adapting their behavior online to novel terrains, crippled body parts, and highly-dynamic environments. We also illustrate the importance of incorporating online adaptation into autonomous agents that operate in the real world by applying our method to a real dynamic legged millirobot. We demonstrate the agent's learned ability to quickly adapt online to a missing leg, adjust to novel terrains and slopes, account for miscalibration or errors in pose estimation, and compensate for pulling payloads.

This work addresses the problem of online adapting dynamics models in the context of model-based RL. Learning globally accurate dynamics model is impossible if we consider that environments are dynamic and we can't observe every possible environment state at initial training time. Thus learning dynamics models that can be adapted online fast, to deal with unexpected und never seen before events is an important research problem.
This paper proposes to use meta-learning to train an update policy that can update the dynamics model at test time in a sample efficient manner. Two methods are proposed
- GrBAL: this method uses MAML for meta-learning
- ReBAL: this method trains a recurrent network during meta-training such that it can update the dynamics effectively at test time when the dynamics  change
Both methods are evaluated on several simulation environments, which show that GrBAL outperforms ReBAL (on average). GrBAL is then evaluated on a real system.
The strengths of this paper are:
- this work addresses an important problem and is well motivated
- experiments on both simulated and on a real system are performed
The weaknesses:
- the related work section is biased towards the ML community. There is a ton of work on adapting (inverse) dynamics models in the robotics community. This line of work is almost entirely ignored in this paper. Furthermore some important recent references for model-based RL are not provided in the related work section (PETS [3] and MPPI [2]), although MPPI is the controller that is used in this work as a framework for model-based RL. Additionally, existing work on model-based RL with meta-learning [1] has not been cited. This is unacceptable.
- There is no significant technical contribution - the "contribution" is that existing meta-learning methods have been applied to the model-based RL setting. Even if no-one has had that idea before - it would be a minor contribution, but given that there is prior work on meta-learning in the context of model-based RL, this idea itself is not novel anymore.
- Two methods are provided, without much analysis. Often authors refer to "our approach" - but it's actually not clear what they mean by our approach. The authors can't claim "model-based meta RL" as their approach.
- While I commend the authors for performing both simulation and real-world experiments, I find the that experiments lack a principled evaluation. More details below.
Feedback on experiments:
Section 6.2 (sample efficiency)
You compare apples to oranges here. I have no idea whether your improvements in terms of sample-efficiency are due to using a model-based RL approach or because your deploying meta-learning. It is well known that model-based RL is more sample efficient, but often cannot achieve the same asymptotic performance as model-free RL. Since MPPI is your choice of model-based RL framework, you would have to include an evaluation that shows results on MPPI with model bootstrapping (as presented in [2]) to give us an idea of how much more sample-efficient your approach is.
Section 6.3 (fast adaptation and generalization)
While in theory one can choose the meta-learning approach independently from the choice of model-based controller, in practice the choice of the MPC method is very important. MPPI can handle model inaccuracies very well - almost to the point where sometimes adaptation is not necessary. You CANNOT evaluate MPPI with online adaptation to another MPC approach with another model-learning approach. This does not give me any information of how your meta-learning improves model-adaptation. In essence these comparisons are meaningless. To make your results more meaningful you need to use the same controller setup (let's say MPPI) and then compare the following:
1. MPPI with your meta-trained online adaptation
2. MPPI results with a fixed learned dynamics model - this shows us whether online adaptation helps
3. results of MPPI with the initial dynamics model (trained in the meta-training phase) -without online adaptation. This will tell us whether the meta-training phase provides a dynamics model that generalizes better (even without online adaptation)
4. MPPI with model bootstrapping (as presented in [2]). This will show whether your meta-trained online adaptation actually outperforms simple online model bootstrapping in terms of sample-efficiency
The key here is that you need to use the same model-based control setup (whether its MPPI or some other method). Otherwise you cannot detangle the effect of controller choice from your meta-learned online adaptation.
6.4 Real-world: same comments as above, comparisons are not meaningful
[1] Meta Reinforcement Learning with Latent Variable Gaussian Processes, UAI 2018
[2] MPPI with model-bootstrapping: Information Theoretic MPC for Model-Based Reinforcement Learning , ICRA 2017
[3] Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models, NIPS 2018