SCALABLE UNBALANCED OPTIMAL TRANSPORT USING GENERATIVE ADVERSARIAL NETWORKS
Generative adversarial networks (GANs) are an expressive class of neural generative models with tremendous success in modeling high-dimensional continuous measures. In this paper, we present a scalable method for unbalanced optimal transport (OT) based on the generativeadversarial framework. We formulate unbalanced OT as a problem of simultaneously learning a transport map and a scaling factor that push a source measure to a target measure in a cost-optimal manner. We provide theoretical justification for this formulation, showing that it is closely related to an existing static formulation by Liero et al. (2018). We then propose an algorithm for solving this problem based on stochastic alternating gradient updates, similar in practice to GANs, and perform numerical experiments demonstrating how this methodology can be applied to population modeling.

In this paper the authors consider the unbalanced optimal transport problem between two measures with different total mass. The authors introduce first the now standard Kantorovich-like formulation, which considers a coupling whose marginals are penalized to look like the two target measures. The authors introduce a second formulation in (2), somewhat a Kantorovich/Monge hybrid that involves a "random" Monge map where the target point T(x) of a point x now depends also on an additional random variable z, to desribe T(x,z). The authors also consider a local mass creation term (\xi) to weight the initial measure \mu.
The authors emphasize the interest of the 2nd formulation, which, much like the original Monge problem, has an intractable push-forward constraint. This formulation is similar to recent work on Wasserstein Autoencoders (to which is added the scaling parameter). As with WAE, this constraint is relaxed to penalize the deviation between the "random" push-forward and the desired marginal.
The authors show then that the resulting problem, which involves a transportation cost integrated both on the random variable z and on the input domain x, weighted by xi + a simple penalization for xi + a divergence penalizing the deviation between push-forward and desired marginal, can be optimized altogether by using three NN: 1 for the parameterization of T, 1 for the parameterization of \xi, and one to optimize using a function f a variational bound on the divergence. 2 gradient descents (T,\xi), 1 gradient ascent (f, variational bound).
The authors then make a link between that penalize formulation and something that resembles unbalanced transport (I say resembles because there is some assymetry, and that the type of couplings is restricted). Finally the authors show that by letting increase the penalty in front of the divergence in (6) they recover something that looks like the solution of (2).
For the sake of completeness, the authors provide in the appendix an implementation of a simple dual ascent scheme to approximate unbalanced OT inspired from previous work by Seguy'17, and show that, unlike that work, their implicit parameterization of the scaling factor \xi can help, and illustrate this numerically.
I give credit to the authors for addressing a new problem and providing an algorithmic formulation to do so. That algorithm is itself recovered from an alternative formulation of unbalanced OT, and is therefore interesting in its own right. Unfortunately, I have found the presentation rushed. I really believe the paper would deserve an extensive re-write. Everything is fairly clear until Section 3. Then, the authors introduce their main contribution.  Basically the section tries to prove two things at the same time, without really completing its job. One is to prove that "dualizing" the scaling+ random push-forward equality constraint is ok if one uses big enough regularizers (intuitive), the other that this scaled + random push-forward formulation is closely related to W_{ub}. This is less clear to me (see below).
The experiments are underwhelming. For faces they happen in latent spaces, and therefore one recovers transport between latent spaces later re-visualized through a decoder. For digits, all is fairly simple. They do not clearly mention whether this alternative UOT approach approximates UOT at all. Despite the title, there's no generation. Therefore my grade is really split between a 5 and a 6.
minor comments and questions:
- Is the reference to a local scaling (\xi) for unbalanced transport entirely new? your paper is not clear on that, and it seems to me this idea already appears in the OT literature.
- I do not understand the connexion you make with GANs. In what sense can you interpret any of your networks as generators? To me it just feels like a simultaneous optimization of various networks, yet without a clear generative purpose. Technically there may be several similarities (as we optimize on networks), but I am not sure this justifies referencing GANs in the title. Additionally, and almost mechanically, putting GAN in your paper, the reader will expect some generation results..
- Numerical benchmarks: Is the technique you propose supposed to approximate the optimal value of Unbalanced OT at all? If yes, is there a way you could compare yourselves with Chizat's approach?
- Somewhere in Lemma 3.2 the fact that you had to use an alternative definition \tilde{W} (by restricting the class of couplings) is not really clarified to the reader. Qualitatively, what does it mean that you restrict the class of couplings to have the same support as \mu? In which situations would \tilde{W} be very different from W_{ub} ? (which, if I understand correctly, only appears in (2) but not elsewhere in the paper?)
- I think it would help for the simple sake of readability to add integration domains under your \int symbols.
- T is used as a subset in Lemma 3.1, while it is used after and before as a map of (x,z)
- T(x,z) looks intuitively like a noisy encoder as in Wasserstein AEs (with, of course, the addition of your term \xi). Could you elaborate?
- I have scanned the paper but did not see how you set lambda.


iclr2019_conference
There are time series that are amenable to recurrent neural network (RNN) solutions when treated as sequences, but some series, e.g. asynchronous time series, provide a richer variation of feature types than current RNN cells take into account. In order to address such situations, we introduce a unified RNN that handles five different feature types, each in a different manner. Our RNN framework separates sequential features into two groups dependent on their frequency, which we call sparse and dense features, and which affect cell updates differently. Further, we also incorporate time features at the sequential level that relate to the time between specified events in the sequence and are used to modify the cell's memory state. We also include two types of static (whole sequence level) features, one related to time and one not, which are combined with the encoder output. The experiments show that the proposed modeling framework does increase performance compared to standard cells.

[Relevance] Is this paper relevant to the ICLR audience? yes
[Significance] Are the results significant? somewhat
[Novelty] Are the problems or approaches novel? reasonably
[Soundness] Is the paper technically sound? yes, I think. I did not thoroughly check the equations.
[Evaluation] Are claims well-supported by theoretical analysis or experimental results? somewhat
[Clarity] Is the paper well-organized and clearly written? yes, except the experiments
Confidence: 2/5
Seen submission posted elsewhere: No (but I did find it on arXiv after writing the review)
Detailed comments:
In this work, the authors propose a new type of memory cell for RNNs which account for multiple types of time series. In particular, the work combines uniformly sampled observations (“normal” RNN input), time-decaying observations, non-decaying observations which may change, and static features which are not time-dependent. Empirically, the proposed approach outperforms an RNN with TLSTM cells in some cases.
=== Comments
I found the proposed approach for incorporating the different types of time series reasonable. This work definitely leans heavily on ideas from TLSTM and others, but, to the best of my knowledge, the specific combination and formulation is novel, especially concerning the “non-decaying time series” observations.
 However, I had difficult coming up with an example of a “static decay feature”. It would be helpful to give a concrete example of one of these in the main text. (It is also not clear to me why the difference in time between the time of the last event in a sequence and the prediction time for that sequence would be considered a “static decay” feature rather than just a “static” feature.)
My main concern with the paper is that the experimental design and results are not especially easy to follow; consequently, they are not as convincing as they might be. First, the sparsity mechanism is rather simple. In many domains (e.g., the medical domain considered in several of the cited papers), missingness is non-uniform and is often meaningful. While “meaningfulness” may be difficult to simulate, burstiness (non-uniformity) could be simulated. Second, for the groups, it is not clear whether all combinations of, e.g., 2 (informative) feature were sparsely sampled or if only one group of 2 was chosen. If the former, then some measure of variance should be given to help estimate statistical significance. Third, the particular classification task here is, essentially, forecasting one of the input variables. While that is certainly a relevant problem, many other time series classification or regression problems are not tied so directly to observations. It is not clear if these results are relevant in that setting.
=== Typos, etc.
The plots and figures in the paper are very difficult to read. Larger versions, or at least versions with increased fonts, should be used.


GANSYNTH: ADVERSARIAL NEURAL AUDIO SYNTHESIS
Efficient audio synthesis is an inherently difficult machine learning task, as human perception is sensitive to both global structure and fine-scale waveform coherence. Autoregressive models, such as WaveNet, model local structure but have slow iterative sampling and lack global latent structure. In contrast, Generative Adversarial Networks (GANs) have global latent conditioning and efficient parallel sampling, but struggle to generate locally-coherent audio waveforms. Herein, we demonstrate that GANs can in fact generate high-fidelity and locally-coherent audio by modeling log magnitudes and instantaneous frequencies with sufficient frequency resolution in the spectral domain. Through extensive empirical investigations on the NSynth dataset, we demonstrate that GANs are able to outperform strong WaveNet baselines on automated and human evaluation metrics, and efficiently generate audio several orders of magnitude faster than their autoregressive counterparts.

This paper proposes a strategy to generate audio samples from noise with GANs. The treatment is analogous to image generation with GANs, with the emphasis being the changes to the architecture and representation necessary to make it possible to generate convincing audio that contains an interpretable latent code and is much faster than an autoregressive Wavenet based model ("Neural Audio Synthesis of Musical Notes with WaveNet AutoEncoders" - Engel et al (2017)). Like the other two related works (WaveGAN - "Adversarial Audio Synthesis" - Donahue et al 2018) and the Wavenet model above, it uses the NSynth dataset for its experiments.
Much of the discussion is on the representation itself - in that, it is argued that using audio (WaveGAN) and log magnitude/phase spectrograms  (PhaseGAN) produce poorer results as compared with the version with the unrolled phase that they call 'IF' GANs, with high frequency resolution and log scaling to separate scales.
The architecture of the network is similar to the recently published paper  (Donahue et al 2018), with convolutions and transpose convolutions adapted for audio. However, there seem to be two important developments. The current paper uses progressive growing of GANs (the current state of the art for producing high resolution images), and pitch conditioning (Odena et al, where labels are used to help training dynamics).
For validation, the paper presents several metrics, with the recently proposed "NDB" metric figuring in the evaluations, which I think is interesting. The IF-Mel + high frequency resolution model seems to outperform the others in most of the evaluations, with good phase coherence and interpolation between latent codes.
My thoughts:
Overall, it seems that the paper's contributions are centered around the representation (with "IF-Mel" being the best). The architecture itself is not very different from commonly used DCGAN variants - the authors say that using PGGAN is desirable, but not critical, and the use of labels from Odena et al.
Many of my own experiments with GANs were plagued by instability (especially at higher resolution) and mode collapse problems without special treatment (largely documented, such as adding noise, adjusting learning rates and so forth). To this end, what do the authors see as 'high' resolution vis a vis audio signals?
I am curious if we can adapt these ideas for recurrent generators as might appear in TTS problems.
I rate this paper as an accept since this is one of the few existing works that demonstrate successful audio generation from noise using GANs, and  owing to its novelty in exploring representation for audio.


Deep learning (DL) is having a revolutionary impact in image processing, with DL-based approaches now holding the state of the art in many tasks, including image compression. However, video compression has so far resisted the DL revolution, with the very few proposed approaches being based on complex and impractical architectures with multiple networks. This paper proposes what we believe is the first approach to end-to-end learning of a single network for video compression. We tackle the problem in a novel way, avoiding explicit motion estimation/prediction, by formalizing it as the rate-distortion optimization of a single spatio-temporal autoencoder; i.e., we jointly learn a latent-space projection transform and a synthesis transform for low bitrate video compression. The quantizer uses a rounding scheme, which is relaxed during training, and an entropy estimation technique to enforce an information bottleneck, inspired by recent advances in image compression. We compare the obtained video compression networks with standard widely-used codecs, showing better performance than the MPEG-4 standard, being competitive with H.264/AVC for low bitrates.

This paper presents a spatiotemporal convolutional autoencoder trained for video compression. The basic model follows the logic of traditional autoencoders, with an intermediate quantizer:
input -> convolutional neural network with a skip connection as an encoder -> quantizer -> transposed convolutional neural network with a skip connection as a decoder.
As the quantizer is a non-differentiable operation, the paper proposes to follow (Toderici et al  2016, Balle et al, 2018) and cast quantization as adding uniform noise to the latent variables. The pre-quantized variables are modelled as Gaussians with variance that is predicted by a second "hyperprior" network dedicated to this task. The final model is trained to minimize three losses. The first loss minimizes the difference between the true frame pixel values and the predicted pixel values. The second loss minimizes the entropy of the latent codes. The third loss minimizes the difference between neighboring pixels in subsequent frames, ignoring those pixels that are not linked between frames. The model is trained on 10,000 videos from the Youtub-8M dataset and tested on 10 videos from the MCL-V database, with rather ok results.
Generally, parts of the proposed approach sound logical: an autoencoder like architecture makes sense for this problem. Also, the idea of using uniform noise to emulate quantization is interesting.  However, the paper has also weaknesses.
- The added novelty is limited and unclear.  Conceptually, the paper is overclaiming. Quoting verbatim from the conclusion: "Our work is, as far as we are aware, the first end-to-end learned video compression architecture using DL.", while already citing few works that also rely on deep networks (Wu et al., 2018, Chen et al., 2016). In the related work section it is noted that these works are computationally heavy. However, this doesn't mean they are not end-to-end. The claims appear to be contradicting.
- The technical novelty is also limited. What is new is the combination of existing components for the task of video compression. However, each component in isolation is not novel, or it is not explained as such.
- Parts of the model are unclear. How is the mask M computed in equation (7)? Is M literally the optical flow between frames? If yes, what is the percentage of pixels that is zeroed out? Furthermore, can one claim the model is fully end to end, since a non-differentiable optical flow algorithm is used?
- The purpose of the hyperprior network is unclear. Why not use a VAE that also returns the variance per data point?
- Most importantly, it is not clear whether the model is trained as a generative one, e.g., with using a variational framework to compute the approximate posterior. If the model is not generative, how can the model be used for generation? Isn't it then that the decoder simply works for reconstruction of already seen frames? Is there any guarantee that the model generalizes well to unknown inputs? The fact that the model is evaluated only on 10 video sequences does not help with convincing with the generalization.
- The evaluation is rather weak. The method is tested on a single, extremely small dataset of just 10 videos. In this small dataset the proposed method seems to perform worse in the majority of compression ratios (bits per pixel). The method does seem to perform a bit better on the very low bits per pixel regime. However, given the small size of the dataset, it is not clear whether these results suffice.
- Only two baselines are considered, both hand-crafted codecs: H.264/AVC and MPEG-4. However, in the related work section there are works that could also be applied to the task, e.g., the aforementioned ones. Why aren't these included in the comparison?
- Although it is explained that the focus is on the very low bitrates, it is not clear what part of the model is designed with that focus in mind. Is this just a statement just so to focus on the part of the curve in the experiment where the proposed method is better than the reported baselines? Is there some intrinsic model hypothesis that makes the model suitable for low bit rates?
In general, the paper needs to clarify the model and especially explain if it is (or not a generative one) and why. Also, a more extensitve arrays of experiments need to be executed to give a better outline of the methods capabilities and limitations.


We study the problem of alleviating the instability issue in the GAN training procedure via new architecture design. The discrepancy between the minimax and maximin objective values could serve as a proxy for the difficulties that the alternating gradient descent encounters in the optimization of GANs. In this work, we give new results on the benefits of multi-generator architecture of GANs. We show that the minimax gap shrinks to as the number of generators increases with rate O(1/ ). This improves over the best-known result of O(1/ 2 ). At the core of our techniques is a novel application of Shapley-Folkman lemma to the generic minimax problem, where in the literature the technique was only known to work when the objective function is restricted to the Lagrangian function of a constraint optimization problem. Our proposed Stackelberg GAN performs well experimentally in both synthetic and real-world datasets, improving Fréchet Inception Distance by 14.61% over the previous multi-generator GANs on the benchmark datasets.

A Stackelberg competition is a nonzero-sum game where 1) each player has their own objective, which do not sum up to a constant, and 2) there is an order at which the players interact. The proposed formulation only assumes that parameters of one player (data generator) partition in I tuples \gamma_i of parameters, where each tuple parameterizes a different data generator component (e.g., a separate neural network). Further, each of those components is assumed to contribute a term to the game's objective that only depends on the corresponding parameter tuple \gamma_i, and the other player's parameters \theta (e.g., weights of the discriminator). From a game theoretic perspective, this still yields a 2-player zero-sum game where the action space of the data generator is the product space of the I tuple spaces. Hence, I have doubts about the general finding that more data generating components decreases the duality gap.
The gap between the a maximin and minimax solution is determined by the shape of the objective \phi(\gamma,\theta) and is zero, for example, if \phi is (quasi) convex in \gamma=[\gamma_1, ..., \gamma_I], and (quasi) concave in \theta. The authors bound the violation of this property w.r.t. the data generator components' parameters \gamma_i, and argue that this degree of violation is the same for the whole data generator parametrized by \gamma=[\gamma_1, ..., \gamma_I] if the data generator components are from the same family of mappings (e.g., having the same network architecture). While this conclusion is true under worst cast assumption, e.g., the globally maximal possible gap, this would also imply that all data generator components find the same global best solution, that is, yield the same mapping, in which case the gap would be identical to just using one of those components.
Intuitively, the only reason to have multiple data generator components is to learn different mappings such that the joint data generator -- mixing the outputs of the different components -- is more expressiv than just a single mapping. If the different mappings only result from the inability of finding the global best solution, a worst case argument is not very insightful; in this case, one should study the duality gap in the neighborhood of the starting solutions. On the other hand, if we assume a different family of mappings for each component, the convexity violation of the joint data generator is higher than for each component; hence, the gap does not necessarily decrease with more components.
So why do multiple data generator components help in practice, and why does the proposed model outperform single-component GANs and the multi-branch GAN in the experiments? Solving a maximin/minimax problem for highly non-convex-concave functions is challenging; there is an infinity of saddle point solutions which yield different "performances". The multi-branch GAN can be seen as a model averaging approach giving more stable results, whereas the proposed GAN seems more of an ensemble approach to stabilize the result. Though, this is speculative and I would encourage the authors to study this in-depth; the reasoning in Remark 1 is not convincing to me.
UPDATE:
I read the revision and stick to my vote. In the discussion, I wasn't able to get my points across, e.g., that bounding the worst case duality gap is not enough to conclude that the observed duality gap does not grow for multiple local optimal GANs, where the duality gap is expected to be much smaller. A simple experiment could be to actually measure the duality gap (flip the order of the players and measure the difference of the objectives, when starting with the same initialization). If the authors were right, the maximum of those gap should stay constant when adding more data generators. To justify a Stackelberg setting, the authors may provide an example instantiation that cannot be cast into a standard zero-sum game with minimax solution. I can't see such an example but I'm happy to be proven wrong.


Prefrontal cortex (PFC) is a part of the brain which is responsible for behavior repertoire. Inspired by PFC functionality and connectivity, as well as human behavior formation process, we propose a novel modular architecture of neural networks with a Behavioral Module (BM) and corresponding end-to-end training strategy. This approach allows efficient learning of behaviors and preferences representation. This property is particularly useful for user modeling (as for dialog agents) and recommendation tasks, as allows learning personalized representations of different user states. In the experiment with video games playing, the results show that the proposed method allows separation of main task's objectives and behaviors between different BMs. The experiments also show network extendability through independent learning of new behavior patterns. Moreover, we demonstrate a strategy for an efficient transfer of newly learned BMs to unseen tasks.

This paper introduces a "behavior module" (BM) which is a small network that encodes preferences over actions and runs parallel to the fully connected layers of a policy. The paper shows this architecture working in Atari games, where the same policy can be used to achieve different action preferences over a game while still playing well. It also includes a thorough recap of past modular approaches.
The motivation for the BM is that we may want deep networks to be able to decompose "strategy" and "behavior", where behavior may influence decisions without affecting the performance. In this framework, the BM is trained on a reward of correctness + personalized “satisfaction”.
The experiments model behavior as preferences over how many actions to play simultaneously. The trained BMs can be transferred to new tasks without finetuning. The ideas here also have some similarity to the few shot learning literature.
Comments on the experiments:
1. The Table 2  do not show a smooth interpolation between reward scaling and AMSR vs BD. This is surprising because the performance on the game should be highest went it is weighted to the most. This indicates to me that the results are actually high variance, the 0.8 vs 0.88 in stage 2 of 0.25r vs 0.5r means that is probably at least +/- 0.08 standard deviation. Adding standard deviations to these numbers is important for scientific interpretability.
2. I expect some BMs should perform much better than others (as they have been defined by number of actions to play at once). I would like to see (maybe in the appendix) a table similar to table 2 for for individual BMs. I currently assume the numbers are averaged over all BMs.
3. Similarly, I would like to see the BD for BM0 (e.g., if a policy is not optimized for any behavior, how close does it get to the other behaviors on average). This is an important lower bound that we can compare the other BD to.
4. An obvious baseline missing is to directly weight the Q values of the action outputs  (instead of having an additional network)  by the designed behavior rewards. There is an optimal way to do this because of experimental choices.
Questions:
1.For BM2, you write " Up and Down (or Right and Left)" did you mean "Up and Right"? How can Up and Down be played at the same time?
Overall, this paper uses neuroscience to motivate a behavior module. However, the particular application and problem settings falls short of these abstract "behaviors". Currently, the results are just showing that RL optimizes whatever reward function is provided, and that architectural decomposition allows for transfer, which was already showed in (Devin 2017). An experiment which would better highlight the behavior part of the BM architecture is the following:
1. Collect datasets of demonstrations (e.g. on atari) from different humans.
2. Train a policy to accomplish the task (with RL)
3. Train BMs on each human to accomplish the task in the style of each human.
This would show that the BMs can capture actual behavior.
The dialog examples discussed in the abstract would also be very exciting.
In conclusion, I find the idea interesting, but the experiments do not show that this architecture can do anything new. The abstract and introduction discuss applications that would be much more convincing. I hope to see experiments with a more complex definition of "behavior" that cannot be handcoded into the Q function.


In this paper, we propose a general formulation to cope with a family of reinforcement learning tasks in observational settings, that is, learning good policies solely from the historical data produced by real environments with confounders (i.e., the factors affecting both actions and rewards). Based on the proposed approach, we extend one representative of reinforcement learning algorithms: the ActorCritic method, to its deconfounding variant, which is also straightforward to be applied to other algorithms. In addition, due to lack of datasets in this direction, a benchmark is developed for deconfounding reinforcement learning algorithms by revising OpenAI Gym and MNIST. We demonstrate that the proposed algorithms are superior to traditional reinforcement learning algorithms in confounded environments. To the best of our knowledge, this is the first time that confounders are taken into consideration for addressing full reinforcement learning problems.

The paper addresses an important and often overlooked issue in off-policy reinforcement learning - the possibility of confounding between the agent's actions and the rewards. This is a subject which has been exhaustively explored in the causal inference literature, and the authors are very correct in suggesting that it should be incorporated into the world of reinforcement learning.  Specifically they propose a generative model with a global latent confounder that is inferred using a variational autoencoder architecture.
The paper is generally well-written, though some points could be made clearer in my opinion, as detailed below. The experiments are constructed by introducing confounding into existing datasets; performance seems to be good, but I am not entirely sure whether the given architecture is necessary, see comments below.
High-level comments:
(1) Classic RL deals with confounders all the time. The state is a confounder between the action and the reward. The issue of confounding becomes less trivial when one is performing off-policy RL when the original policy is *unknown*. This is exactly the case that the authors mention when they cite the recent work by Gottesman et al. (2018) who deal with using RL to learn from the actions of physicians in a hospital.  While I am sure the authors are aware of these distinctions, I think the paper would be better if this is spelled out very explicitly. This includes explaining why this issue doesn't come up in classic RL.
(2) Assuming the case above - off-policy RL with unknown confounders - one would usually assume "no unmeasured confounding", i.e. that the observed actions are an unknown but learnable function of the observed states. That is basically the scenario of most off-policy RL.
(3) However, the authors strive to go one step beyond the case (2), to a situation where there is an *unmeasured* confounder affecting both observed actions and rewards. If nothing is known about this unmeasured confounder, then it is generally impossible to learn effective policies, as the causal effects of actions are not identifiable from the observed data. In this paper, the authors make an implicit assumption that while the confounder is unmeasured, it can still be inferred from the data. This is an intermediate step between "no unmeasured confounding" and "complete unmeasured confounding". This is related to work on using proxy variables e.g. Kuroki & Pearl (2014) and even more closely related to the work cited by Louizos et al. (2017).
Again, I think the paper would be much improved if all this is addressed explicitly.
(4) An important consequence of point (3) above is that in fact adding the single global latent-confounder U is not, in itself, very important from a causal perspective. The sequence of variables Z_1... Z_T are already latent confounders that are assumed to be inferrable from data. It is true that the addition of the global U might change the statistical and optimization properties of the model. This leads to a very important conclusion: the authors should test their model with and without U. I think this specific ablation experiment is crucial. In many cases I am sure that the assumption of a global latent confounder is a good one and is especially useful in the VAE case where it will make optimization more stable. However, in principle, all of U's roles could be taken within the sequence of Z's, and I am curious to see in practice how big of an effect it has.
(5) I wish to add that even if the U variable turns out to not add much empirically, this work is still valid since the sequence of Z's can themselves be considered inferred latent confounders.
Specific comments:
(1) 2.3: there are more than 2 ways of computing the do-operator. RCTs and backdoor are the best known approaches, but not the only ones, e.g. there is frontdoor adjustment.
(2) I think the paper would be easier to follow if there was one concrete example used throughout. This will make it easier to understand and possibly verify/criticize the assumptions of the generative model.
(3) Related to "higher-level point (4)" above, in eqs. 17 & 18 note that Z_t is unknown, same as U. Both are inferred. This also leads to the question which Z_t is actually used in practice? Is it the mean, or is it also sampled from the approximate posterior q?
(4) Below eq. 19, it would be very useful for the readers if you could explain exactly when would there be a difference between the two versions p(r_{t+1}|z_t,a_t) and p(r_{t+1}|z_t, do(a_t=a))
(5) In the description of all the experiments I was missing a crucial point: how does the introduced confounder affect the reward? Is it only through the different actions? The way it is currently explained, it seems like the added variable introduces lack of *overlap*, but not strictly confounding.
(6) The description of the experiment in 4.3 could be more detailed. What exactly was the training and test? What RL method was used? What did the baseline optimize for? I would like to see an ablation experiment where U is not included in the model.
(7) In 4.5, what is the "vanilla" method? And as mentioned above, I would like to see an ablation experiment where U is not included in the model.


Deep reinforcement learning has obtained significant breakthroughs in recent years. Most methods in deep-RL achieve good results via the maximization of the reward signal provided by the environment, typically in the form of discounted cumulative returns. Such reward signals represent the immediate feedback of a particular action performed by an agent. However, tasks with sparse reward signals are still challenging to on-policy methods. In this paper, we introduce an effective characterization of past reward statistics (which can be seen as long-term feedback signals) to supplement this immediate reward feedback. In particular, value functions are learned with multi-critics supervision, enabling complex value functions to be more easily approximated in on-policy learning, even when the reward signals are sparse. We also introduce a novel exploration mechanism called "hot-wiring" that can give a boost to seemingly trapped agents. We demonstrate the effectiveness of our advantage actor multi-critic (A2MC) method across the discrete domains in Atari games as well as continuous domains in the MuJoCo environments. A video demo is provided at https://youtu.be/zBmpf3Yz8tc and source codes will be made available upon paper acceptance.

Recommendation: Weak reject
Summary:
The paper proposes a variant of deep reinforcement learning (A2MC) for environments with sparse rewards.  The approach replaces the standard environment reward function with a combination of the current reward and the variability of rewards in the last T timesteps, with a goal of decreasing variability.  The authors further propose a “hot-wiring” exploration strategy to bootstrap agents by taking either random actions or actions that have been done in the recent history.  Empirical evaluations in standard benchmarks including several sparse reward Atari games show empirical improvement of this approach over a baseline (ACKTR).
Review:
The paper has strong empirical results that show the A2MC outperforming or reaching the same performance as the baselines in a large number of Atari and MuJoCo domains.  The authors also provide results with and without the hot-wiring feature, which helps isolate its contribution.  However, overall the paper lacks theoretical rigor and most of the proposed changes are done without principled reasons or convergence guarantees.   There is no way of telling from the current paper whether these changes could lead to divergence or suboptimal behavior in other domains.  Examples of such changes include:
* The averaging of the reward terms at different timescales in Equation 4 is the core of the algorithm but is derived ad-hoc.  Why is this a good equation?  Is lack of variability really a desired property and may it lead to a suboptimal policy?  Can anything be said about how it changes behavior in a tabular representation?
* The exponential equation with a constant of 100 appears out of nowhere in equation 5.  Is this a general equation that will really work in different domains and reward scales?
* The variability weights in equations 6 and 7 are never tested empirically – what happens if they are left out?  Where did this equation come from?
* Overall, it is unclear if the combination of rewards at different time scales in equation 10 is stable and leads to convergence.  The terms show resemblance to the eligibility trace equations but lack their theoretical properties.
To make the paper ready for publication, the authors need to justify which of these changes are “safe” in that they guarantee the behavior of the algorithm cannot become much worse, or need to point directly to other methods in the literature that have used such changes and cite the pros and cons that were seen with those changes.
Related to the theme above, the paper does not properly cite other methods used with sparse rewards in traditional RL or Deep RL, especially eligibility traces, which seem highly related to the current approach.  The following related work edits are needed:
* The overall approach is thematically similar to eligibility traces (see the standard Sutton and Barto textbook), except that the authors here use variability rather than combining the reward terms directly.  Eligibility traces are built exactly for these sorts of sparse reward problems and combine short and long-term rewards in a TD update. But there has been substantial investigation of their theoretical properties in the tabular and function approximation cases. The current method needs to compare and contrast to this long-standing method both theoretically and empirically.
* Two other methods that should have been considered in the experiments are experience replay and reward shaping, both of which are beneficial with deep RL in sparse domains.  Experience replay is mentioned in the paper but not implemented as a competitor.  I realize ER is not as computationally efficient as the new approach but it is an important (and stable) baseline.  Reward shaping is not mentioned at all, but is again an important and stable baseline that has been used in such problems – see “Playing FPS Games with Deep Reinforcement Learning” (AAAI, 2017).
* Finally, the related work section mentions a lot of competitive algorithms but does not implement any of them in comparison, which makes it hard to claim the current approach is the best yet proposed.


CBOW IS NOT ALL YOU NEED: COMBINING CBOW WITH THE COMPOSITIONAL MATRIX SPACE MODEL
Continuous Bag of Words (CBOW) is a powerful text embedding method. Due to its strong capabilities to encode word content, CBOW embeddings perform well on a wide range of downstream tasks while being efficient to compute. However, CBOW is not capable of capturing the word order. The reason is that the computation of CBOW's word embeddings is commutative, i.e., embeddings of XYZ and ZYX are the same. In order to address this shortcoming, we propose a learning algorithm for the Continuous Matrix Space Model (Rudolph & Giesbrecht, 2010), which we call Continual Multiplication of Words (CMOW). Our algorithm is an adaptation of word2vec (Mikolov et al., 2013a), so that it can be trained on large quantities of unlabeled text. We empirically show that CMOW better captures linguistic properties, but it is inferior to CBOW in memorizing word content. Motivated by these findings, we propose a hybrid model that combines the strengths of CBOW and CMOW. Our results show that the hybrid CBOW-CMOW-model retains CBOW's strong ability to memorize word content while at the same time substantially improving its ability to encode other linguistic information by 8%. As a result, the hybrid also performs better on 8 out of 11 supervised downstream tasks with an average improvement of 1.2%.

The authors propose CMOW, an extension of the CBOW model that allows the model to capture word order. Instead of each word being represented as a vector, words are represented by matrices. They extend the CBOW objective to take into account word order by replacing the averaging of vectors to create the context with matrix multiplication (a non-commutative operation). This is the first time this model has been applied in a large scale unsupervised setting. They are able to do this using their objective and an initialization strategy where the matrix embeddings are set to the identity matrix with some Gaussian noise added.
The results of this paper are its main weakness. I did enjoy reading the paper, and it is nice to see some results using matrices as embeddings and matrix multiplication as a compositional function. They include a nice analysis of how word order is captured by these CMOW embeddings while CBOW embeddings capture the word content, but it doesn't seem to make much of a difference on the downstream tasks where CBOW is better than CMOW and close to the performance of the hybrid combination of CBOW and CMOW.
I think it's clear that their model is able to capture word information to some extent, but other models  (RNNs etc.) can do this as well, that admittedly are more expensive, but also have better performance on downstream tasks. I think a stronger motivation for their method besides an analysis of some phenomena it captures and a slight improvement on some downstream tasks when combined with CBOW is needed though for acceptance. Could it be used in other settings besides these downstream transfer tasks?
PROS:
- introduced an efficient and stable approach for training CMSM models
- Show that their model CMOW is able to capture word order information
- Show that CMOW compliments CBOW and a hybrid model leads to improved results on downstream tasks.
CONS
- The results on the hybrid model are only slightly better than CBOW. CMOW alone is mostly worse than CBOW.


We propose a novel autoencoding model called Pairwise Augmented GANs. We train a generator and an encoder jointly and in an adversarial manner. The generator network learns to sample realistic objects. In turn, the encoder network at the same time is trained to map the true data distribution to the prior in latent space. To ensure good reconstructions, we introduce an augmented adversarial reconstruction loss. Here we train a discriminator to distinguish two types of pairs: an object with its augmentation and the one with its reconstruction. We show that such adversarial loss compares objects based on the content rather than on the exact match. We experimentally demonstrate that our model generates samples and reconstructions of quality competitive with state-of-the-art on datasets MNIST, CIFAR10, CelebA and achieves good quantitative results on CIFAR10.

Thank you for an interesting read.
The paper proposes adding an adversarial loss to improve the reconstruction quality of an auto-encoder. To do so, the authors define an auxiliary variable y, and then derive a GAN loss to discriminate between (x, y) and (x, decoder(encoder(x))). The algorithm is completed by combining this adversarial "reconstruction" loss with adversarial loss functions that encourages the matching of marginal distributions for both the observed variable x and the latent variable z.
Experiments present quite a lot of comparisons to existing methods as well as an ablation study on the proposed "reconstruction" loss. Improvements has been shown on reconstructing input images with significant numbers.
Overall I think the idea is new and useful, but is quite straight-forward and has some theoretical issues (see below). The propositions presented in the paper are quite standard results derived from the original GAN paper, so for that part the contribution is incremental and less interesting. The paper is overall well written, although the description of the augmented distribution r(y|x) is very rush and unclear to me.
There is one theoretical issue for the defined "reconstruction" loss (for JS and f-divergences). Because decoder(encoder(x)) is a deterministic function of x, this means p(y|x) is a delta function. With r(y|x) another delta function (even that is not delta(y=x)), with probability 1 we will have mismatched supports between p(y|x) and r(y|x).
This is also the problem of the original GAN, although in practice the original GAN with very careful tuning seem to be OK... Also it can be addressed by say instance noise or convolving the two distributions with a Gaussian, see [1][2].
I think another big issue for the paper is the lack of discussion on how to choose r(y|x), or equivalently, a(x).
1. Indeed matching p_{\theta}(x) to p^*(x) and q(z) to p(z) does not necessarily returns a good auto-encoder that makes x \approx decoder(encoder(x)). Therefore the augmented distribution r(y|x) also guides the learning of p(y|x) and with appropriately chosen r(y|x) the auto-encoder can be further improved.
2. The authors mentioned that picking r(y|x) = \delta(y = x) will result in unstable training. But there's no discussion on how to choose r(y|x), apart from a short sentence in experimental section "...we used a combination of reflecting 10% pad and the random crop to the same image size...". Why this specific choice? Since I would imagine the distribution r(y|x) has significant impact on the results of PAGAN, I would actually prefer to see an in-depth study of the choice of this distribution, either theoretically or empirically.
In summary, the proposed idea is new but straight-forward. The experimental section contains lots of results, but the ablation study by just removing the augmentation cannot fully justify the optimality of the chosen a(x). I would encourage the authors to consider the questions I raised and conduct extra study on them. I believe it will be a significant contribution to the community (e.g. in the sense of connecting GAN literature and denoising methods literature).
[1] Sonderby et al. Amortised MAP Inference for Image Super-resolution. ICLR 2017.
[2] Roth et al. Stabilizing Training of Generative Adversarial Networks through Regularization. NIPS 2017.


Deep reinforcement learning (RL) algorithms have made great strides in recent years. An important remaining challenge is the ability to quickly transfer existing skills to novel tasks, and to combine existing skills with newly acquired ones. In domains where tasks are solved by composing skills this capacity holds the promise of dramatically reducing the data requirements of deep RL algorithms, and hence increasing their applicability. Recent work has studied ways of composing behaviors represented in the form of action-value functions. We analyze these methods to highlight their strengths and weaknesses, and point out situations where each of them is susceptible to poor performance. To perform this analysis we extend generalized policy improvement to the max-entropy framework and introduce a method for the practical implementation of successor features in continuous action spaces. Then we propose a novel approach which, in principle, recovers the optimal policy during transfer. This method works by explicitly learning the (discounted, future) divergence between policies. We study this approach in the tabular case and propose a scalable variant that is applicable in multi-dimensional continuous action spaces. We compare our approach with existing ones on a range of non-trivial continuous control problems with compositional structure, and demonstrate qualitatively better performance despite not requiring simultaneous observation of all task rewards.

The authors introduce Divergence Correction (DC) for the problem of transfer learning by composing policies. There approach builds on GPI with a maximum entropy objective. They also prove that DC solves for the max-entropy optimal interpolation between two policies and derive a practical approximation for this algorithm. They provide experimental results in a gridworld problem and study their approximate algorithm in two continuous control problems.
While this paper has some interesting ideas (combining GPI with a Max-Entropy objective and DC), these ideas are not properly motivated. The main problem seems to be clarity. One big problem is that the paper never defines the notion of a notion of optimality (or near-optimality). Also, considering that the DC algorithm is one of the main contributions of the paper it is barely motivated. Theorem 3.2 is presented with almost no explanation about how DC was derived. Why do the authors believe that DC is a good idea on a conceptual level? It's very interesting that the paper presents cases where previous approaches (Optimistic and GPI) don't perform well. But the authors don't explain why they believe DC should perform well in these cases.
The authors make the unjustified claim in the abstract that their approach has "near-optimal performance and requires less information". I say this is unjustified because they only try this approach on three benchmarks. In addition, there should be situations where DC also performs poorly since there are known hardness results for solving MDPs. Admittedly, those results may not apply if the authors are making assumptions that are not being clearly discussed in the paper.
Minor Comments:
1. In the abstract, "requiring less information" is very imprecise. Are you referring to sample complexity?
2. In the introduction, "can consistently achieve good performance" is imprecise. What is the notion of near-optimality? What does consistent mean? Having experimental results on 3 tasks doesn't seem to be enough to me to justify this claim.
3. In the introduction (and rest of the paper), please don't call Haarnoja et al.'s approach optimistic. Optimism already has another widely used meaning in RL literature. Maybe call it "Uncorrected".
4. In section 2.2, the authors introduce \pi_1, \pi_2, ... , \pi_n but never actually use that notation. This section does not clearly explain how GPI works.
5. In Theorem 3.1, the authors should introduce Q^1, Q^2, ... , Q^n and define the policies in terms of the action-value functions. Also, the statement of this theorem is not self contained, what is the reward function of the MDP? The proof below should be called a proof sketch.
6. The paper mentions that extending to multiple tasks is possible. Is it trivial? What is the basic idea? It seems straightforward but it might be helpful to explicitly state the idea.
7. In Theorem 3.2, how was C derived? Please add some commentary explaining the conceptual idea.
8. In Table 1, what is f(s, a|b)? I don't see where this was defined?
9. CondQ is usually referred to as UVFA in the literature.
10. Section 3 really needs a conclusion statement.
11. Section 4 is very unclear and hard to follow.
12. In figure 1f, what is LTD? It's never defined. I'm guessing it's DC.
13. All of the figures are too small and some are not clear in black and white.


LEARNING MIXED-CURVATURE REPRESENTATIONS IN PRODUCTS OF MODEL SPACES
The quality of the representations achieved by embeddings is determined by how well the geometry of the embedding space matches the structure of the data. Euclidean space has been the workhorse for embeddings; recently hyperbolic and spherical spaces have gained popularity due to their ability to better embed new types of structured data-such as hierarchical data-but most data is not structured so uniformly. We address this problem by proposing learning embeddings in a product manifold combining multiple copies of these model spaces (spherical, hyperbolic, Euclidean), providing a space of heterogeneous curvature suitable for a wide variety of structures. We introduce a heuristic to estimate the sectional curvature of graph data and directly determine an appropriate signature-the number of component spaces and their dimensions-of the product manifold. Empirically, we jointly learn the curvature and the embedding in the product space via Riemannian optimization. We discuss how to define and compute intrinsic quantities such as means-a challenging notion for product manifolds-and provably learnable optimization functions. On a range of datasets and reconstruction tasks, our product space embeddings outperform single Euclidean or hyperbolic spaces used in previous works, reducing distortion by 32.55% on a Facebook social network dataset. We learn word embeddings and find that a product of hyperbolic spaces in 50 dimensions consistently improves on baseline Euclidean and hyperbolic embeddings, by 2.6 points in Spearman rank correlation on similarity tasks and 3.4 points on analogy accuracy.

Page 2: What are p_i, i=1,2,...,n, their set T and \mathcal{P}?
What is | | used to compute distortion between a and b?
Please fix the definition of the Riemannian manifold, such that M is not just any manifold, but should be a smooth manifold or a particular differentiable manifold. Please update your definition more precisely, by checking page 328 in J.M. Lee, Introduction to Smooth Manifolds, 2012, or Page 38 in do Cormo, Riemannian Geometry, 1992.
Please define \mathcal{P} in equation (1).
Define K used in the definition of the hyperboloid more precisely.
Please provide proofs of these statements for product of manifolds with nonnegative and nonpositive curvatures: “In particular, the squared distance in the product decomposes via (1). In other words, dP is simply the l2 norm of the component distances dMi.”
Please explain what you mean by “without the need for optimization” in “These distances provide simple and interpretable embedding spaces using P, enabling us to introduce combinatorial constructions that allow for embeddings without the need for optimization.” In addition, how can you compute geodesic etc. if you use l1 distance for the embedded space?
By equation (2), the paper focuses on embedding graphs, which is indeed the main goal of the paper. Therefore, first, the novelty and claims of the paper should be revised for graph embedding. Second, three particular spaces are considered in this work, which are the sphere, hyperbolic manifold, and Euclidean space. Therefore, you cannot simply state your novelty for a general class of product spaces. Thus, the title, novelty, claims and other parts of the paper should be revised and updated according to the particular input and output spaces of embeddings considered in the paper.
Please explain how you compute the metric tensor  g_P and apply the Riemannian correction (multiply by the inverse of the metric tensor g_P) to determine the Riemannian gradient in the Algorithm 1, more precisely.
Step (9) of the Algorithm 1 is either wrong, or you compute v_i without projecting the Riemannian gradient. Please check your theoretical/experimental results and code according to this step.
What is h_i used in the Algorithm 1? Can we suppose that it is the ith component of h?
In step (6) and step (8), do you project individual components of the Riemannian gradient to the product manifold? Since their dimensions are different, how do you perform these projections, since definitions of the projections given on Page 5 cannot be applied? Please check your theoretical/experimental results and code accordingly.
Please define exp_{x^(t)_i}(vi) and Exp(U) more precisely. I suppose that they denote exponential maps.
How do you initialize x^(0) randomly?
The notation is pretty confusing and ambiguous. First, does x belong to an embedded Riemannian manifold P or a point on the graph, which will be embedded? According to equation (2), they are on the graph and they will be embedded. According to Algorithm 1, x^0 belongs to P, which is a Riemannian manifold as defined before. So, if x^(0) belongs to P, then L is already defined from P to R (in input of the Algorithm 1). Thereby, gradient \nabla L(x) is already a Riemannian gradient, not the Euclidean gradient, while you claim that \nabla L(x) is the Euclidean gradient in the text.
Overall, Algorithm 1 just performs a projection of Riemannian or Euclidean gradient  \nabla L(x) onto a point v_i for each ith individual manifold. Then, each v_i is projected back to a point on an individual component of the product manifold by an exponential map.
What do you mean by “sectional curvature, which is a function of a point p and two directions x; y from p”? Are x and y not points on a manifold?

You define \xi_G(m;b,c) for curvature estimation for a graph G. However, the goal was to map G to a Riemannian manifold. Then, do you also consider that G is itself a Riemannian manifold, or a submanifold?
What is P in the statement “the components of
the points in P” in Lemma 2?
What is \epsilon in Lemma 2?
How do you optimize positive w_i, i=1,2,...,n?
What is the “gradient descent” refered to in Lemma 2?
Please provide computational complexity and running time of the methods.
Please define \mathbb{I}_r.
At the third line of the first equation of the proof of Lemma 1, there is no x_2. Is this equation correct?
If at least of two of x1, y1, x2 and y2 are linearly dependents, then how does the result of Lemma 1 change?
Statements and results given in Lemma 1 are confusing. According to the result, e.g. for K=1, curvature of product manifold of sphere S and Euclidean space E is 1, and that of E and hyperbolic  H is 0. Then, could you please explain this result for the product of S, E and H, that is, explain the statement “The last case (one negative, one positive space) follows along the same lines.”? If the curvature of the product manifold is non-negative, then does it mean that the curvature of H is ignored in the computations?
What is \gamma more precisely? Is it a distribution or density function? If it is, then what does (\gamma+1)/2 denote?
The statements related to use of Algorithm 1 and SGD to optimize equation (2) are confusing. Please explain how you employed them together in detail.
Could you please clarify estimation of K_1 and K_2, if they are unknown. More precisely, the following statements are not clear;
- “Furthermore, without knowing K1, K2 a priori, an estimate for these curvatures can be found by matching the distribution of sectional curvature from Algorithm 2 to the empirical curvature computed from Algorithm 3. In particular, Algorithm 2 can be used to generate distributions, and K1, K2 can then be found by matching moments.” Please explain how in more detail? What is matching moments?
- “we find the distribution via sampling (Algorithm 3) in the calculations for Table 3, before being fed into Algorithm 2 to estimate Ki” How do you estimation K_1 and K_2 using Algorithm 3?
- Please define, “random (V)”, “random neighbor m” and “\delta_K/s” used in Algorithm 3 more precisely.


LEARNING MULTIMODAL GRAPH-TO-GRAPH TRANS- LATION FOR MOLECULAR OPTIMIZATION
We view molecular optimization as a graph-to-graph translation problem. The goal is to learn to map from one molecular graph to another with better properties based on an available corpus of paired molecules. Since molecules can be optimized in different ways, there are multiple viable translations for each input graph. A key challenge is therefore to model diverse translation outputs. Our primary contributions include a junction tree encoder-decoder for learning diverse graph translations along with a novel adversarial training method for aligning distributions of molecules. Diverse output distributions in our model are explicitly realized by low-dimensional latent vectors that modulate the translation process. We evaluate our model on multiple molecular optimization tasks and show that our model outperforms previous state-of-the-art baselines.

Update:
The score has been updated to reflect the authors' great efforts in improving the manuscript. This reviewer would suggest to accept the paper now.
Old Review Below:
The paper describes a graph-to-graph translation model for molecule optimization inspired from matched molecular pair analysis, which is an established approach for optimizing the properties of molecules. The model extends a chemistry-specific variational autoencoder architecture, and is assessed on a set of three benchmark tasks.
While the idea of manuscript is interesting and promising for bioinformatics, there are several outstanding problems, which have to be addressed before it can be considered to be an acceptable submission. This referee is willing to adjust their rating if the raised points are addressed. Overall, the paper might also be more suited at a domain-specific bioinformatics conference.
Most importantly, the paper makes several claims that are currently not backed up by experiments and/or data.
First, the authors claim that MMPs “only covers the most simple and common transformation patterns”. This is not correct, since these MMP patterns can be as complex as desired. Also, it is claimed that the presented model is able to “learn far more complex transformations than hard-coded rules”. The authors will need to provide compelling evidence to back up these claims. At least, a comparison with a traditional MMPA method needs to be performed, and added as a baseline. Also, it has to be kept in mind that the reason MMPA was introduced was to provide an easily interpretable method, which performs only local transformations at one part of the molecule. “Far more complex transformations” may thus not be desirable in the context of MMPA. Can the authors comment on that?
Second, the authors state that they “sidestep” the problem of non-generalizing property predictors in reinforcement learning, by “unifying graph generation and property estimation in one model”. How does the authors’ model not suffer from the same problem? Can they provide evidence that their model is better in property estimation than other models?
In the first benchmark (logP) the GCPN baseline is shown, but in the second benchmark table, the GCPN baseline is missing. Why? The GCPN baseline will need to be added there. Can the authors also comment on how they ensure the comparison to the GPCN and VSeq2Seq is fair? Also, can the authors comment on why they think the penalized logP task is a good benchmark?
Also, the authors write that Jin et al ICML 2018 (JTVAE) is a state of the model. However, also Liu et al NIPS 2018 (CGVAE) state that their model is state of the art. Unfortunately, both JTVAE and CGVAE were never compared against the strongest literature method so far, by Popova et al, which was evaluated on a much more challenging set of tasks than JT-VAE and CGVAE. The authors cite this paper but do not compare against it, which should to be rectified. This referee understands it is more compelling to invent new models, but currently, the literature of generative models for molecules is in a state of anarchy due to lack of solid comparison studies, which is not doing the community a great service.
Furthermore, the training details are not described in enough detail.
How exactly are the pairs selected? Where do the properties for the molecules come from? Were they calculated using the logP, QED and DRD2 models? How many molecules are used in total in each of these tasks?


Recent image style transferring methods achieved arbitrary stylization with input content and style images. To transfer the style of an arbitrary image to a content image, these methods used a feed-forward network with a lowest-scaled feature transformer or a cascade of the networks with a feature transformer of a corresponding scale. However, their approaches did not consider either multiscaled style in their single-scale feature transformer or dependency between the transformed feature statistics across the cascade networks. This shortcoming resulted in generating partially and inexactly transferred style in the generated images. To overcome this limitation of partial style transfer, we propose a total style transferring method which transfers multi-scaled feature statistics through a single feed-forward process. First, our method transforms multi-scaled feature maps of a content image into those of a target style image by considering both interchannel correlations in each single scaled feature map and inter-scale correlations between multi-scaled feature maps. Second, each transformed feature map is inserted into the decoder layer of the corresponding scale using skip-connection. Finally, the skip-connected multi-scaled feature maps are decoded into a stylized image through our trained decoder network.

SUMMARY
The paper is concerned with the problem of arbitrary feed-forward style transfer, where a feed-forward model receives a content image and a style image as input, and must produce as output an image matching the content of the former and the style of the latter. The approach roughly follows that of [Li et al, NIPS 2017]: An encoder network (VGG pretrained on ImageNet) is used to extract features from both the style and content image; the features from the content image are adjusted to match the statistics from the style image, and the adjusted features are passed to a decoder network which generates the output image.
Compared to prior work, the main innovations are:
- Considering correlations between features from different encoder layers rather than only correlations within a single layer in both the feature adjustment step as well as in the loss
- An improved encoder / decoder architecture which uses skip connections in the decoder, allowing for a single encoder / decoder pair rather than a cascade of encoder / decoder pairs for each layer.
PROS
- Considering correlations between features from different encoder layers is a good idea
- The improved encoder / decoder architecture is significantly more efficient than the cascaded approach of [Li et al, NIPS 2017]
CONS
- Somewhat incremental
- Limited experimental evaluation
- Qualitative results not clearly better than existing methods
- Missing citation for multi-scale losses
LIMITED EXPERIMENTAL EVALUATION
One of the key claims of the paper is that “our method with inter-scale
(fig.7(f)) or intra-scale feature transform (fig.7(g)) are more similar to the target style than those of single-scale style transfer without considering inter-channel correlation” (Figure 7 caption); this claim is substantiated primarily by qualitative results in Figures 4, 7, and 8. Personally I don’t find the results with inter-feature correlations to be much better than those with only intra-feature correlations or the results from prior work. All recent style transfer methods depend on a host of hyperparameters like style and content weight, learning schedule, etc; in my experience differences in these hyperparameter settings can have large effects on the qualitative appearance of the generated images, and by varying these hyperparameters it is common to see qualitative differences similar to those shown in Figure 4 and 5. From the small number of qualitative results presented I do not think that the benefits of inter-scale correlations have been clearly demonstrated.
I appreciate that the authors tried to quantify their results by comparing loss values (Table 1) but unfortunately it’s hard to know how much the values of different losses correlate with human judgement of style quality.
In addition to selected qualitative results I would have liked to see a user study demonstrating human preference for images generated using inter-scale correlation losses, ideally across a range of different hyperparameter settings for each method.
GATYS BASELINE
From Table 1, the proposed method with intra-scale features achieves lower content loss than the direct optimization baseline (a) from [Gatys et al.] This is very surprising to me - typically direct optimization leads to much lower losses than any feedforward methods. From Section 4.4.1 this baseline uses Adam; in my experience using L-BFGS tends to achieve lower losses which may explain the results from Table 1.
MISSING CITATION FOR MULTI-SCALE STYLE TRANSFER
Instead of computing correlations between features from different encoder layers, [Wang et al, CVPR 2017] define a loss that considers the generated and style image at multiple spatial scales. This should be discussed in relation to the proposed method.
ENCODER / DECODER DETAILS
There are some missing details about exactly how the encoder and decoder are initialized and trained. I assume that the encoder was pretrained on ImageNet; is it updated during training or kept fixed? Is the decoder initialized randomly or does it mirror the pretrained ImageNet weights?
TYPOS / FORMATTING
There are minor typos throughout, e.g. “verity”, “sinlge” in Section 4.3, Paragrah 1. I also found the citation style to be somewhat jarring, especially in the introduction; parenthetical citations and better spacing may improve readability.
OVERALL
On the whole I feel that the paper is somewhat incremental. The inter-scale loss seems intuitively like a good idea, but I don’t think the paper presents sufficient experimental evidence to justify it. On the other hand the proposed decoder architecture seems like a clear improvement over the cascaded approach from [Li et al, NIPS 2017], as it is significantly more efficient without sacrificing quality. However I don’t feel that this alone is enough novelty for ICLR, so I lean slightly toward rejection.


CLARINET: PARALLEL WAVE GENERATION IN END-TO-END TEXT-TO-SPEECH
In this work, we propose a new solution for parallel wave generation by WaveNet. In contrast to parallel WaveNet (van den Oord et al., 2018), we distill a Gaussian inverse autoregressive flow from the autoregressive WaveNet by minimizing a regularized KL divergence between their highly-peaked output distributions. Our method computes the KL divergence in closed-form, which simplifies the training algorithm and provides very efficient distillation. In addition, we introduce the first text-to-wave neural architecture for speech synthesis, which is fully convolutional and enables fast end-to-end training from scratch. It significantly outperforms the previous pipeline that connects a text-to-spectrogram model to a separately trained WaveNet (Ping et al., 2018). We also successfully distill a parallel waveform synthesizer conditioned on the hidden representation in this end-to-end model.

This paper proposes some modifications to established procedures for neural speech synthesis and investigates their effect experimentally. The proposed modifications are mostly fairly straightforward conceptually, but appear to work well, and this reviewer feels the paper has huge value in its experimental contributions extending and clarifying certain aspects of WaveNet training and distillation. The paper is well-written and fairly concise, with a short-and-sweet experimental results section.
Major comments:
The conceptual novelty seems a little overstated in the abstract. For example, the value seems to not really be in the "proposing" a text-to-wave neural architecture for speech synthesis (which, aside from important experimental tweaks, is essentially Tacotron 2 training all parameters from scratch) but in showing that it works well experimentally. Conceptually the paper is extremely close to the parallel wavenet paper, the main differences being slightly different component distributions (Gaussian instead of logistic), a different set of loss terms in addition to the reverse KL, and joint training of the spectral synthesis and waveform synthesis parts of the model.
It would be super insightful to include log probabilities on the test set (everywhere MOS results have been reported) in the experimental results. This would help tease apart the effects of architecture inductive bias, different divergences, distillation, etc. One of the really nice things about flow-based models is the ability to compute the log probability tractably.
Minor comments:
Perhaps mention that teacher forcing is maximum likelihood in the introduction? Currently it almost sounds like the paper is contrasting teacher forcing for WaveNet (paragraph 2) and MLE (list item 1).
At the end of paragraph 3 in the introduction, it would be helpful to mention that the intractable KL divergence being referred to is the frame-level one-step-ahead predictions, not the entire sequence-level prediction. Also, for 1D distributions isn't taking a large number of samples quite effective in practice?
In introduction list item 3, suggest mentioning Tacotron 2 (Shen et al) and contrasting with the present work for clarity.
In section 3.1, it surprises me slightly that clipping at -7 is essential. It would be helpful to state what exactly goes wrong if this is not done. Does it lead to overfitting and so bad test log likelihoods? What effect is noticeable in the generated samples?
Equation (6) is incorrect. It should be conditioned on < t, not <= t. Conditioning on z <= t would make x_t deterministic.
Equation (7) is technically true as written, but only because all the distributions involved are deterministic. If <= t is replaced with < t (which based on the mistake in (6) is what I suspect the authors intended) then it is no longer true. This equation is not used anywhere as far as I can tell. It seems to me like the property that enables non-recursive-over-time ("parallel") sampling is (5), not (7). Incidentally, when multiple one-step-ahead samples are taken per frame for parallel wavenet, the samples viewed at the sequence level are highly correlated, and do not obey anything like (7), but it doesn't affect the correctness of the expected value.
The IAF doesn't really "infers its output x at all time steps". Maybe "models" instead of "infers"?
Learning an "IAF directly through maximum likelihood" doesn't seem all that impractical. People train networks with recursive dependence such as RNNs (which is essentially what would be required to train certain forms of IAF with MLE) as opposed to non-recursive dependence such as CNNs all the time, after all. It seems like this claim depends on the details of the transform $f$.
Out of interest, did the authors consider reversing the sequence being generated in time between successive IAF blocks? This would limit the ability to do low latency synthesis but might improve performance considerably.
The first paragraph in section 3.3 seems like it should probably be part of section 3.3.1 (it's not related to other losses such as spectrogram frame loss, for example). It would be helpful to state explicitly that: (a) the goal is to minimize the sequence-level reverse KL; (b) this can be approximated by taking a single sample z, but this may have high variance; (c) the variance of this estimate can be reduced by marginalizing over the one-step-ahead predictions for each frame; (d) parallel wavenet's mixture of logistics means it has to use a separate Monte Carlo sampling at the frame-level, whereas the proposed Gaussian allows this one-step-ahead marginalization to be performed analytically. This one-step-ahead marginalization is an example of Rao-Blackwellization.
It didn't seem clear from section 3.3 and 3.3.1 that parallel wavenet also uses the one-step-ahead marginalization trick to reduce the variance.
It might be helpful to mention that using the reverse KL would be expected to have mode-fitting behavior, making samples sound better but log probability on the test set worse.
It was not clear to me what difference or similarity was being demonstrated in Figure 1.
Small point, but "Oord et al" should be "van Oord et al" throughout (it's a surname).
In section 3.3.2, can the authors give any insight as to why training with reverse KL alone leads to whispering, and why adding the STFT term fixes this? (If it's only something that's been noticed empirically, "will lead" -> "empirically we found"?)
I noticed quite a large qualitative perceptual difference between the student and teacher samples, particularly in the speech synthesis case (experiment 3), even though I think I'd rate the quality on a linear scale as fairly similar (in line with the MOS results). The teacher sounds noticeably "harsher" but "clearer" Do the authors have any insight as to why this perceptual difference occurs (if they also perceive a qualitative difference)? Is it probably a difference in inductive bias between an AF (which WaveNet can be seen as) and IAF?
I found it fascinating that reverse KL and forward KL lead to roughly the same MOS for spectrum-to-waveform. I assumed reverse KL would be better due to its preference for high-quality samples due to mode fitting.
Out of curiosity, what is responsible for the pops at the start of the spectrogram-conditioned distilled models? Also why are the synthesized samples shorter than the ground truth (less initial silence)?


Although generative adversarial networks (GANs) have enabled us to convert images from one domain to another similar one, converting between different sensory modalities, such as images and sounds, has been difficult. This study aims to propose a network that reconstructs images from sounds. First, video data with both images and sounds are labeled with pre-trained classifiers. Second, image and sound features are extracted from the data using pre-trained classifiers. Third, multimodal layers are introduced to extract features that are common to both the images and sounds. These layers are trained to extract similar features regardless of the input modality, such as images only, sounds only, and both images and sounds. Once the multimodal layers have been trained, features are extracted from input sounds and converted into image features using a feature-to-feature GAN. Finally, the generated image features are used to reconstruct images. Experimental results show that this method can successfully convert from the sound domain into the image domain. When we applied a pre-trained classifier to both the generated and original images, 31.9% of the examples had at least one of their top 10 labels in common, suggesting reasonably good image generation. Our results suggest that common representations can be learned for different modalities, and that proposed method can be applied not only to sound-to-image conversion but also to other conversions, such as from images to sounds.

Summary:
This paper addresses the problem of generating images from sound. The general idea is to use conditional GANs. In particular, two stacked conditional autoencoder GANs, where the autoencoders have a U-Net architecture. First, sound features are mapped into multimodal features that contain image feature/class information. Such multimodal features condition the generation of image features with an initial GAN, and the image features condition the generation of the output image with a second GAN. Although the problem itself is rather difficult, the solution is almost entirely based on previous work. The most novel part of the paper is the learning of the multimodal features. The final results are not very compelling, the literature review is very limited. There is a very high-level description of the approach with very few details, which leave the reader with a lot of unanswered questions. There is no attempt to compare with previous work, the architecture is not studied in depth with an ablation study, or compared with more interesting baselines, other than verifying that images can be generated from features (something not very surprising, given StackGUN!). Probably the more important verification is that the multimodal feature can embody some class information. Overall it seems to be a limited contribution.
Comments on quality, clarity, originality and significance:
This paper provides a high-level description for an approach to a problem that is relatively difficult to address. The paper is not very well motivated, and therefore lacks clarity and leaves the reader with a lot of unanswered questions. The literature review is limited as it is the set of results and comparison with previous works.
The paper addresses an important problem; however, I feel the work is not very significant because it does not reveal new techniques, nor produces compelling results, nor performs a deep analysis.
I think the problem is interesting, but a deeper analysis is needed to lift this contribution up to a significant one.
Below is a summary of some of the additional questions gathered while reading the paper:
Why there is the need for the multimodal features at all? Why can’t the sound be converted into class labels and then StackGAN can generate images?
Why do you need a feature-to-feature GAN? Why not generating images directly from the generated multimodal features? Motivations are not provided clearly.
Unclear architecture from Figure 1. The Feature-to-Feature GAN and the Feature-to-Image GAN have the same architecture? What does the encoder and decoder do? How are they organized?
Looks like every piece is trained alone, no end-to-end learning, right? Please clarify that point.
No attempt to compare with other approaches has been made. Also, no effort to formulate a baseline model. What would happen if one were to use solely the features generated by SoundNet?
Would you be able to compare your multimodal features with those generated by Ngiam et al. (2011), for instance?
Section 3 refers to a 90/10 training/evaluation split but then it is unclear in what experiments that exact split is used.
No description on hyperparameters.
No complexity, no architecture details, (also no equations that could provide more details).
It should be clarified what it means one-to-one conversion. It is brought up in several points in the paper, but it is never clear what it means and therefore how it relates to what the Author intends to stress.
It is unclear why by performing first a feature-to-feature mapping and later a feature-to-image mapping the one-to-one conversion problem should be addressed. In StackGAN the problem addressed is the resolution increase. The problem addressed by this paper is unclear.
Unclear what is the “ensemble effect”, and what is the motivation for upsampling a feature vector in two dimensions.
What image features were used to generate the images in Figure 3? Which architecture was used and how was it trained? Where these the same multimodal features used in the full architecture?
The paragraph motivating the need for multimodal features is unclear.
How are the three type of losses weighted for learning the multimodal layers? No discussion provided on that.
Unclear why a multimodal vector should be upsampled in 2 dimensions.
The training procedure and loss for training the feature-to-feature conditional GAN is not explained.
Despite the difficulty of the problem, the generated images do not look compelling.
16 references do not seem enough by todays’ high-quality standard conferences.


Most classifiers operate by selecting the maximum of an estimate of the conditional distribution p(y|x) where x stands for the features of the instance to be classified and y denotes its label. This often results in a hubristic bias: overconfidence in the assignment of a definite label. Usually, the observations are concentrated on a small volume but the classifier provides definite predictions for the entire space. We propose constructing conformal prediction setsVovk et al. (2005) which contain a set of labels rather than a single label. These conformal prediction sets contain the true label with probability 1 − α. Our construction is based on p(x|y) rather than p(y|x) which results in a classifier that is very cautious: it outputs the null set -meaning "I don't know" -when the object does not resemble the training examples. An important property of our approach is that classes can be added or removed without having to retrain the classifier. We demonstrate the performance on the ImageNet ILSVRC dataset and the CelebA and IMDB-Wiki facial datasets using high dimensional features obtained from state of the art convolutional neural networks.

The paper proposes an approach to construct conformal prediction sets. The idea is to estimate p(x|y) and then construct a conformal set based on the p(x|y) instead of p(y|x). The paper claims that such a method produces cautious classifiers that can produce "I don't know" answers in the face of uncertainty.
However,
A] Although the paper is titled is "Cautious Deep Learning", the method is broadly applicable to any classifier, there is nothing in the method that restricts it to the domain of deep learning. A broad spectrum evaluation could have been done on standard multi-class classifiers.
B] The paper provides multiple qualitative evaluation results. While it gets the point across, I still would have liked to see a quantitative evaluation, for e.g., there have been several papers that proposed generating adversarial examples for deep learning. The author could have taken any of those methods, generated adversarial examples for deep learning and compared the original classifier with the conformal prediction set. Also, such comparison would have made the paper more connected with deep learning.
C] The paper uses Xception network as a feature extractor and then compares its result with Inception-v4. Honestly, I would have preferred if the comparison was between 1] Xception Feature Extractor + Conformal Set Prediction, 2] Xception network prediction, and 3] Inception-v4. The reason being that it is very difficult to understand how much of the cautious-ness is because of the proposed approach and how much is due to the Xception network being good. For example, in Figure 3b, does Xception network generate high probability values for the top classes or does it generate low probability values? Unless we can understand this difference, it is very difficult to appreciate what this approach is giving us.
D] Another analysis that could have been done is to apply this approach and use several different pre-trained networks as feature extractor and check whether there is a decrease in false positives across all the networks, that would suggest that the method can truly make deep learning cautious across a wide variety of networks.
E] Another analysis that could have been done is understand the impact of the quality of feature extractor. For example, take a deep network (of sufficient depth) and use the proposed approach but instead of using just the penultimate layer for feature extraction, one can keep on removing layers from the end and use the remaining as the feature extractor. Then analyze the quality of conformal predictions as each layer gets removed. One can understand the robustness of this method.
Even though doing all these evaluations may be a tad too much, but definitely, quite a few of those could have been done to make the approach look convincing and enticing. I think bulking this paper with such analysis could make for a very good submission. However, as it stands, it still quite lacks.


NON-VACUOUS GENERALIZATION BOUNDS AT THE IM- AGENET SCALE: A PAC-BAYESIAN COMPRESSION APPROACH
Modern neural networks are highly overparameterized, with capacity to substantially overfit to training data. Nevertheless, these networks often generalize well in practice. It has also been observed that trained networks can often be "compressed" to much smaller representations. The purpose of this paper is to connect these two empirical observations. Our main technical result is a generalization bound for compressed networks based on the compressed size that, combined with off-theshelf compression algorithms, leads to state-of-the-art generalization guarantees. In particular, we provide the first non-vacuous generalization guarantees for realistic architectures applied to the ImageNet classification problem. Additionally, we show that compressibility of models that tend to overfit is limited. Empirical results show that an increase in overfitting increases the number of bits required to describe a trained network.

This paper tries to push forward in important directions the seemingly increasingly powerful approach of using PAC-Bayesian formalism to explain low risks of training neural nets on real-life data. They take an interesting approach to evaluate these bounds by setting up a prior distribution as a mixture of Gaussians centered on possible heuristic compressions of the net  and this prior's variances are obtained by doing a layerwise grid search. This seems to give good risk bounds on certain known compressible nets using image data sets.
Let me list out a bunch of issues that seem to be somewhat confusing in this paper (some of these were in the comment thread I had with the authors but I am repeating nonetheless for completeness)
0.
Firstly this form of the PAC-Bayes formula used here (Theorem 2.1) is of a more complicated form than what has been previously used in say these papers, https://arxiv.org/abs/1707.09564 Given this I strongly feel that there is a need for an explanation connecting this formalism to the usual one - particularly something that proves how this is stronger than the one in the paper I referred to earlier.
1.
In the statement of Theorem 2.1 there is a \lambda parameter over which the infimum is being taken. If I understand right in the experiments one is substituting the upperbound on KL from Theorem 4.3 into this RHS of Theorem 2.1 and evaluating this. Now there is also a \lambda parameter in Theorem 4.3. Is this the same \lambda as in Theorem 2.1 and when a grid-search is being done over \lambda is the "whole" thing (theorem 2.1 upperbound with theorem 4.3 substituted) being minimized by choosing a good \lambda?
If the two \lambda s are different then is the choice of the 2 \lambda s being optimized separately?
(...the authors had earlier clarified that this is so and I strongly feel this is a very important clarification should be updated into the paper..)
2.
How is the \sigma of Theorem 4.3 chosen in the experiments? Am I right in thinking that this \sigma is the posterior variance about which it is being said towards the end of page 6 that "We add Gaussian noise with standard deviation equal to 5% of the difference between the largest and smallest weight in the filter." ?
So am I to understand that this is an arbitrary choice? Or is this choice dictated by some need to ensure that the posterior variance sigma is chosen so that under this distribution the sampled nets approximately compute the same function on the training data? (If yes, then what in the theory is motivating this?).
To the best of my understanding the results are highly dependent on this choice of sigma but there is virtually no explanation for this choice which was not even found by grid search. (As of now this is merely reflective of the fact that trained nets often have some noise resilience but its not a priori clear as to why that should be important to the PAC-Bayes formalism here.)
3.
The code based compression seems a bit mysterious to me given that I do not have enough familiarity with the algorithm that is being referred to. Hence it seems a bit weird as to why there is a sum over codebooks in the proof of Theorems 4.3. Naively I would have thought that there is a fixed codebook for a given compression scheme but here it feels that the compression scheme is a randomized algorithm which also generates a new codebook in every run of it. This seems unusual and seems to need more explanation and at the very least a detailed pseudocode explaining exactly how this compression is working.
This point ties in with a somewhat larger issue I describe next...
4.
In the previous reply to my comment the authors had shared their anonymized code and l had a look through the code. Its pretty evident from the code there are an enormous number of tweaks and hyperparameter tunings to make this work. There is very little insight otherwise as to why "Dynamic Network Surgery' should work and its great that the authors have found an implementation that works on their image data.
But then the question arises that there should have been a cleanly abstracted out pseudocode explaining how the compression was done and how the dynamic network surgery was done. To my mind this implementation is the main contribution of the paper and giving the pseudocode for it in the paper seems not only important for essential completeness of the current paper but that could also then act as a springboard for many future attempts at trying to come up with theory for these mysterious procedures.


ATTENTIVE NEURAL PROCESSES
Neural Processes (NPs) (Garnelo et al., 2018a;b) approach regression by learning to map a context set of observed input-output pairs to a distribution over regression functions. Each function models the distribution of the output given an input, conditioned on the context. NPs have the benefit of fitting observed data efficiently with linear complexity in the number of context input-output pairs, and can learn a wide family of conditional distributions; they learn predictive distributions conditioned on context sets of arbitrary size. Nonetheless, we show that NPs suffer a fundamental drawback of underfitting, giving inaccurate predictions at the inputs of the observed data they condition on. We address this issue by incorporating attention into NPs, allowing each input location to attend to the relevant context points for the prediction. We show that this greatly improves the accuracy of predictions, results in noticeably faster training, and expands the range of functions that can be modelled.

Neural process (NP) is a recent probablistic method for modeling distributions of functions. The authors claim that one substantial weakness of NP is the tendency of under-fitting. The authors give a hypoethesize: the under-fitting behaviour of NP is because the mean-aggregation step in the encoder acts as a bottleneck, as a result, it is difficult for the decoder to learn the relevant information for a give target prediction. This paper proposes to resolve this issue by adding an attention mechanism to the deterministic path. The experimental results show that the proposed method converge faster and give better results on various tasks.
One major concern about the paper is the lack of analysis of the true cause of under-fitting in NP. The authors give the hypoethesize about the potential cause of the under-fitting issue and proposes to resolve it with attention, however, without theoretical or empirical analyses, it is hard to understand the true cause of the under-fitting issue. Although the proposed method give better performance, it is not clear whether the better performance is due to the added complexity to the model (the attention mechanism) or truely resolving the under-fitting issue. Some analyses along this line can make the paper clearer and more convincing.
A lot of technical details are missing in the paper, which makes the method not reproducible. Please add more details about the proposed attention mechanism and how they are implemented into NP.
In the GP literature, there are also methods tackling meta-learning or multi-task learning or few shot learning. These works are known as multi-output / multi-tasks Gaussian processes. A few works on this topic are listed:
* Z Dai, MA Álvarez, ND Lawrence, Efficient Modeling of Latent Information in Supervised Learning using Gaussian Processes, NIPS 2017
* MA Alvarez, L Rosasco, ND Lawrence, Kernels for vector-valued functions: A review, Foundations and Trends® in Machine Learning 2012
* EV Bonilla, KM Chai, C Williams, Multi-task Gaussian process prediction, NIPS 2007
For the 1D regression experiments (Figure 1left, Figure 3right), it is not clear which fitting is better. It largely depends on the prior of kerel parameters. As the data points are generated from a GP, plotting the Gaussian process fit with the ground truth parameters can show what a ground truth fitting would look like.
The Bayesian optimization experiment is very nice and gives some good insights about the quality of the uncertainty of prediction. Maybe consider it to include it in the main text.