We introduce a new approach for comparing reinforcement learning policies, using Wasserstein distances (WDs) in a newly defined latent behavioral space. We show that by utilizing the dual formulation of the WD, we can learn score functions over trajectories that can be in turn used to lead policy optimization towards (or away from) (un)desired behaviors. Combined with smoothed WDs, the dual formulation allows us to devise efficient algorithms that take stochastic gradient descent steps through WD regularizers. We incorporate these regularizers into two novel on-policy algorithms, Behavior-Guided Policy Gradient and Behavior-Guided Evolution Strategies, which we demonstrate can outperform existing methods in a variety of challenging environments. We also provide an open source demo 1 .

Summary
This paper proposes using Wasserstein Distances to measure the difference between higher-level functions of policies, which this paper terms as "behaviors". For example, one such behavior could be the distribution over final states given the policy, or the distribution over returns given policy. Through the lens of these behavioral embeddings, this paper recovers a few important special cases that are well-known in the literature including WD-based TRPO and distributional RL. This paper shows that the dual formulation of the Wasserstein Distance gives the ability to score individual policies based on a given "behavioral mapping".
Review
The idea of generalizing the trust-region of TRPO by using the WD measure and behavior maps is intriguing. The paper introduces many choices of behavior maps that could lead towards interesting algorithms that merit additional study. I think the connections between the proposed family of algorithms and WD-based TRPO and distributional RL is highly motivating.
There are, however, a several key issues with the empirical study of the proposed methods that make it challenging to assess the value of introducing another partially understood deep policy gradient algorithm. The results show only behavioral studies of the algorithm, not investigating the effects of the WD based regularizer or the effects of the choice of behavioral map. The results are also significantly limited in their statistical significance, making distinguishing between algorithms difficult in most cases. And the comparison of wall-clock time is particularly difficult to assess due to the uncountably many possible sources of noise when comparing wall-clock time of highly complex algorithms. In the following paragraphs, I will expand upon each of these points.
One of the key contributions of this paper is the ability to define regularizers based on the definition of the behavior map. The paper introduces many such behavior maps, those over state visitation, actions, and returns; however, the paper does not investigate the effects this novel choice has on the results. It is unclear if introducing each of these behavior maps leads towards different results or if they each induce roughly the same final performance of the agent. It would be highly valuable for me to see a more careful study of the effect of choosing each of these behavior maps on a single simple environment, clarifying that this formulation leads to a family of useful algorithms. As it stands, BGPG may only beat TRPO on these domains because it had more meta-parameters to choose between (BGPG introduces many new meta-parameters: choice of behavior map, kernel for produce RKHS, the meta-parameters of that kernel, the meta-parameters of the behavior map like trajectory length, the entropy regularization term in the WD, etc.). Without a careful study, or intuitive explanation of any of these parameter choices, it is unclear if BGPG won simply through overfitting to the problem.
The statistical significance of the proposed algorithm is impossible to assess in the given form. The comparisons are made using only five random seeds and the standard error bars are frequently quite large. I refer to Henderson et al. 2017 for further explanation as to why five random seeds is simply too few to provide a meaningful comparison between algorithms. Instead, I'll discuss a few of the results in particular. In figure 3a, I notice that the proposed method has high variance until it plateaus around -300 reward; why? In each of the remaining plots of Figure 3, I notice that the propose method is significantly higher variance than any of its competitors. This greatly leads me to suspect that the proposed method would have much lower average performance if studied across a greater number of random seeds. In the walker domain (Figure 3d), why do BGPG and TRPO both plateau at the same point for many timesteps, then eventually BGPG starts improving again? From Figure 5, the paper somewhat misleadingly states that "BGES is the only method that drives the agent to the goal in both settings." However Figure 5 on the left clearly shows that BGES and NSR-ES have indistinguishable performance, and the error bars indicate that neither significantly outperform NoisyNet-TRPO. In fact, due to the high variance of BGES it is unclear if it would drive the agent towards the goal on average if run over more random seeds. On the right, NoisyNet-TRPO noticeably outperforms BGES and is significantly lower variance. The language used to describe Figure 6 is strong, stating that Figure 6 "proves that the benefits come here not just from introducing the regularizer, but from its particular form." However, I tend to disagree that Figure 6 reliably proves anything. Although it appears that BGES is significantly outperforming other methods, the number of meta-parameters over which it gets to optimize makes it difficult to say if the performance gain is from overfitting to the problem or the form of the regularizer. Additionally, good performance demonstrated across a single problem is hardly proof especially using only five random seeds (Henderson et al. 2017 Figure 5).
Although this plays comparatively little role in my scoring of the paper, I feel it is necessary to discuss briefly. Because it is impossible to determine the source of the speed differences between the particle approximator and BGPG, the results in Figure 4 are extremely difficult to interpret. It simply could be that BGPG uses slightly more optimized code than the particle approximator, or it could be that there is in fact a significant performance difference between the algorithms. The only true way to discuss performance differences between algorithms generally is through computational complexity for these reasons. In the case that these statistical algorithms share the same computational complexity, then further analysis including convergence rates would be able to shed light on the speed difference. However, wall-clock time vs. performance has so many confounding factors, it is a fairly meaningless unit of measure. For an empirical investigation of speed, I would suggest giving each algorithm similar number of learning steps or similar number of updates to their weights and compare performance in this way.
Additional Comments (do not affect score)
I'm curious if this work could be extended to the off-policy case. It does seem like a minor disadvantage that distributional RL is well-defined in both the on-policy and off-policy cases, but the proposed family of methods is not. From my reading of the paper, it appears that there is little preventing this extension. Is this true?
--------
Edit after reading other reviews and rebuttals.
I appreciate the response noting that the meta-parameters of this algorithm were not tuned. I still feel that understanding the sensitivity to these parameters is important to understand how the choice of these parameters impacts the performance. While it is certainly likely that extensive tuning of these parameters could yield even more improved results, it is also quite possible that the chosen parameters have biased the results. If a more intuitive discussion of the choice of meta-parameters could be included in the paper, I would be more willing to concede this point.
I recognize that 3 or 5 random seeds are a standard in the deep RL community, but I think Henderson et al. 2017 make this point better than I could: this is not a standard that we should hold to. I think that using small domains and smaller networks would allow running across more random seeds and providing a much more careful scientific study. This paper does not convincingly show state of the art performance, a metric that is nearly impossibly to define in a field that moves as quickly as ours, so should not strive to follow the same demonstration study design that SOTA papers follow. A paper with strong theoretical motivations (such as this) should compliment with strong empirical understanding of how that theory translates to practice. Instead, having strong theoretical motivations backed by a demonstration that the algorithm still works is much less convincing (to me). I think a more convincing set of experiments that match well with the intended purpose of the paper (theoretical contribution of behavioral embeddings) would be an investigation to _how_ the optimization affects the learned policies or representations. Instead of performance benchmarks, this would provide understanding for the various effects of the new framework.
After the rebuttal phase, I now have an additional new concern about the clarity of the writing and the completeness of the empirical details. Many of the results do not mention most of the important details necessary to replicate or even interpret the results. I had assumed that details were consistent across figures (e.g. figure 3 uses 5 random seeds, so I assumed the same to be true for all other figures that did not specify), but the author response made clear that this is not true. In additional, little to no details are provided in the paper referring to the choices of meta-parameters for any algorithm or the methodology used to chose these meta-parameters. After these details were given in the rebuttal, my concerns were greatly alleviated, but not completely removed. It is unclear if the authors intend to update the paper before the final version, but based on the language of the rebuttal (e.g. indicating it is purely a misunderstanding of the paper), I am forced to assume that the paper will not include these important details.
To summarize. I think the theoretical underpinnings of these paper are extremely motivating and interesting. I do not want to see these innovations lost. However, the current state of the empirical section of the paper leaves too much open for me to be able to recommend an accept at this time. I do not believe a paper should solve everything in one pass (e.g. careful parameter studies, SOTA demonstrations, real-world demonstrations, etc.), but I do not believe this paper demonstrates any one of these convincingly.


We present a novel statistical certification method that generalizes prior work based on smoothing to handle richer perturbations. Concretely, our method produces a provable classifier which can establish statistical robustness against geometric perturbations (e.g., rotations, translations) as well as volume changes and pitch shifts on audio data. The generalization is non-trivial and requires careful handling of operations such as interpolation. Our method is agnostic to the choice of classifier and scales to modern architectures such as ResNet-50 on ImageNet.

Summary:
This paper introduces a new smoothing algorithm which produces classifiers with certified robustness against adversarial perturbations. In particular, the authors are interested in settings where the input vectors need not be robust simply to \ell_2 perturbations, for which previous authors (e.g., Cohen et al.) have already introduced a "randomized smoothing" classifier which possesses the desired properties. Instead, motivated by the desire to certify robustness to other families of transformations, the authors propose to introduce a smoothing transform that operates in an additive way on the underlying parameter space (rather than on the raw input vectors, as in the case of \ell_2 randomized smoothing). The authors provide details for a few examples from image and audio data processing, and then show experimental results on ImageNet data.
Comments:
My assessment of this paper is that the level of novelty is relatively low. On the technical side, I do not believe this paper makes any significant contributions. The authors themselves note that the proof of the main result, Theorem 3.2, closely parallels the proof of Cohen et al. Algorithmically, the proposals of the authors are also minor tweaks of the algorithms in Cohen et al., since the types of perturbations considered in this paper are essentially single-parameter problems. (As a question: The authors prove Theorem 3.2 where the randomized smoothing operation adds possible nonisotropic Gaussian noise in the parameter space. However, it seems that in the examples, the noise added is isotropic -- it is not clear how the authors are choosing \sigma_i in relation to \sigma_r for the interpolation example. Can the authors clarify this, and give examples where smoothing with nonisotropic noise would actually be useful?)
The only real tweak that the authors introduce when presenting their smoothing algorithm is to introduce a small correction term in order to deal with interpolation/rounding errors. However, I do not think that this change is particularly novel.
As a final comment, I think it would have been interesting for the authors to include some simulations with audio data as well, since this constitutes one of the main motivating applications discussed by the authors.


Homotopy methods, also known as continuation methods, are a powerful mathematical tool to efficiently solve various problems in numerical analysis. In this work, we propose a novel homotopy-based numerical method that can be used to gradually transfer optimized weights of a neural network across different data distributions. This method generalizes the widely-used heuristic of pre-training weights on one dataset and then fine-tuning them on another dataset of interest. We conduct a theoretical analysis showing that, under some assumptions, the homotopy method combined with Stochastic Gradient Descent (SGD) is guaranteed to converge to an r θ -optimal solution for a target task when started from an r θ -optimal solution on a source task. Empirical evaluations on a toy regression dataset and for transferring optimized weights from MNIST to Fashion-MNIST and CIFAR-10 show up to two orders of magnitude speedup over random initialization and substantial improvements over pre-training.

Contribution
This paper proposes algorithm for transferring knowledge from easy -to-solved to complex tasks or from already solve to new tasks. It relies on homotopy functions and sequentially solves a sequence of optimization problems where the task distribution is gradually deformed from a source task to the target task. Theoretical guarantees are provided and proven in a strongly convex setting. The main results from the theory show that the distance between the final solution and its optimal are less or equal to  relative to the distance of the initial source solution to its optimum. So a near optimal solution for the source task will lead to near optimal solution for the target task. Regression and Classification experimentations show competitive results compared to  random and warm-start initialization schemes.
Clarity
Overall, the paper is well written, well motivated and well structured. The technical content is also very clear and excellent.
 Minor point: Seems that there is a notation error in proposition G.1 and its proof (i instead of i+1).
Novelty
The novelty in this work seems to be the application of homotopy methods to the transfer learning settings. The mathematical guarantees are also new and may even offer new ways to interpret fine tuning methods that have been so successful in recent literature.
However, given the  non-convexity of DNNs, it seems like the analysis in the non-convex settings and its implications  should be part of the main text.
Experiments:
Overall, the experiments are very insightful but limited since you only show the training loss and the validation performance is not evaluated at all. Other things that would could be  beneficial in better assessing the quality of your method are: comparison to Curriculum learning methods, more in depth analysis of the impact of k, and gamma in both regression and classification settings, and solving  toy convex optimization problems to bridge the gap between theory and application.
Preliminary rating:
* Accept *


We present a new approach to 3D object representation where the geometry of an object is encoded directly into the weights and biases of a second 'mapping' network. This mapping network can be used to reconstruct an object by applying its encoded transformation to points randomly sampled from a simple geometric space, such as the unit sphere. Our experiments examine the effectiveness of our method on a subset of the ShapeNet dataset. We find that this representation can reconstruct objects with accuracy equal to or exceeding state-of-the-art methods with orders of magnitude fewer parameters. Our smallest reconstruction network has only about 7000 parameters and shows reconstruction quality on par with stateof-the-art object representation architectures with millions of parameters. Further experiments show that the space of functions learned meaningfully captures the features of the encoded objects, enabling feature mixing through the composition of these functions.

This work is focused on learning 3D object representations (decoders) that can be computed more efficiently than existing methods.  The computational inefficiency of these methods is that you learn a (big) fixed decoder for all objects (all z latents), and then need to apply it individually on either each point cloud point you want to produce, or each voxel in the output (this problem exists for both the class of methods that deform a uniform distribution R^3 -> R^3 a la FoldingNet, or directly predict the 3D function R^3 -> R e.g. DeepSDF). The authors propose that the encoder directly predict the weights and biases of a decoder network that, since it is specific to the particular object being reconstructed, can be much smaller and thus much cheaper to compute.
The authors then note the fact that their method lacks a continuous latent space that allows for interpolation, as provided by existing (VAE-like) methods. They propose to solve this by learning an MLP that produces the output by recurrent application, and then composing subapplications of different networks as a type of interpolation.
-------------------
I like this work, it addresses a real problem in a number of models for 3D representation learning (similar models are also used for e.g. cryo-EM reconstruction). While the fast weights approach is not totally original, its application to this problem is novel and very well-suited to it. I was a bit surprised by just how much the decoder network could be shrunk by using fast weights.
The paper is also quite well written. I especially like how Section 2 synthesizes existing work into model categories which make it easier to think about their relationships. I also think the explanation in Sec. 3.2, while kind of obvious, is a nice way think about decoder vs. fast weights.
I like that the authors are straightforward about the deficiency of the method (i.e. that you can't interpolate in latent space). Their proposed solution of functional composition is exceedingly clever but in my opinion too impractical to really be useful. It adds extra complexity, requires you to do function composition which may be less expressive and takes more coomputation, etc. And to what end? The purpose of generative models is not to interpolate per se; the interpolation is really a sanity check that the model is capturing the underlying distribution rather than just memorizing training examples. The function composition doesn't capture that. I think the authors should just acknowledge that you can't soundly *sample* from their generative model the way e.g. VAE or GAN allows (their function composition is not a sampling method). But I think there are lots of useful things you can do without that capability, e.g. do 3D point cloud completion, go image -> structure, etc. I think this function composition angle should be deemphasized in the title/abstract, but I think the paper stands  reasonably on its own without that.
Nits:
- In Figure 2 it's pretty hard to see the differences between the methods. What exactly is being visualized here? DeepSDF shold be visualizing surface normals vs. HOF which is point clouds, right?
- For predicting a deformation R^3 -> R^3 function composition sort of makes sense, but how generalizable is this approach e.g. to directly predicting a function R^3 -> R (a la DeepSDF)? I think there are ways this function composition approach could generalize, e.g. using skip connections and layer dropout (which encourages layers to be composable).


In this paper, a new intrinsic reward generation method for sparse-reward reinforcement learning is proposed based on an ensemble of dynamics models. In the proposed method, the mixture of multiple dynamics models is used to approximate the true unknown transition probability, and the intrinsic reward is designed as the minimum of the surprise seen from each dynamics model to the mixture of the dynamics models. In order to show the effectiveness of the proposed intrinsic reward generation method, a working algorithm is constructed by combining the proposed intrinsic reward generation method with the proximal policy optimization (PPO) algorithm. Numerical results show that for representative locomotion tasks, the proposed model-ensemble-based intrinsic reward generation method outperforms the previous methods based on a single dynamics model.

Summary
This paper proposes a model-based method for intrinsic rewards based on probabilistic neural network ensembles. For a particular ensemble element, an intrinsic reward is defined as a log-term that measures the deviation in prediction between a Gaussian mixture over all ensemble elements and the particular ensemble element (at the previous update period). The intrinsic reward signal applied in practice is the minimum of the aforementioned quantity w.r.t. all ensemble elements. The authors provide some theoretical motivation for their approach and validate their method in sparse-reward continuous robotics domains (Mujoco), using PPO as reference algorithm. Experiments are averaged over 10 seeds and compared against various other intrinsic reward baselines including PPO without intrinsic motivation. In these experiments, the method provided by the authors seems to dominate over competing approaches.
Quality
I find the quality low. First, while I appreciate that the authors try to provide some theoretical motivation for their method, there is quite a gap from theory to practice. The theory assumes that the world model is known and considers a stationary reward signal. In practice, the world model is approximated with the current network ensemble, a quantity that is changing over time yielding a non-stationary signal. Second, the experiments are conducted in a sparse-reward modification of Mujoco-type environments that are non-sparse by construction. Sparsity is introduced by accumulating instantaneous rewards over some time window before "releasing" them. This way of sparsifying rewards yields weird partial observability issues, e.g. the same state-action pair observed at the right moment in time yields significant reward whereas at the wrong moment in time yields no reward at all. Additionally, it is always guaranteed that there will be a reward signal at a certain frequency. There are probably better environments for studying the approach, like Atari for example. I do understand that the authors cite Oh et al. 2018 who apply the same technique of sparsification, but Oh et al. also conduct additional experiments in ALE.
Clarity
The paper is clearly written and easy to follow. On a side note, it could be stated more clearly that the presented approach is not model-based because no forward prediction is required for constructing intrinsic rewards (merely probability values are queried for observed transitions). There is one question I have though regarding the experiments where different intrinsic reward approaches are compared against each other. The paper states that all approaches normalize intrinsic rewards according to Equation (17)---why do all of them then need a different weighting factor \beta as stated in the second paragraph of Section 3.3? Furthermore, since \beta is fine-tuned for each intrinsic reward approach and each environment, can the authors please elaborate in detail how exactly this is accomplished (to ensure correct interpretability of the results)?
Originality
The originality is low. As stated by the authors, the proposed method is an incremental extension of Achiam and Sastry 2017 who proposed a similar method for non-ensemble methods.
Significance
The significance is low as well. The method's improvement over other intrinsic reward approaches is minor (the environments chosen by the authors are also not ideal). I feel the significance is reduced further by the fact that there are other model-based approaches that use models of the proposed kind for increasing sample-efficiency and performance considerably in non-sparse environments (e.g. Kurutach et al.2018).
Update
After reading the other reviews and the rebuttal, the concerns I have raised remain. I still feel this paper shouldn't be accepted to ICLR. However, given the extensive experimental analysis conducted by the authors, I feel a score of 1 from my side was too harsh. I therefore increase to 3.


Distributional robust risk (DRR) minimisation has arisen as a flexible and effective framework for machine learning. Approximate solutions based on dualisation have become particularly favorable in addressing the semi-infinite optimisation, and they also provide a certificate of the robustness for the worst-case population loss. However existing methods are restricted to either linear models or very small perturbations, and cannot find the globally optimal solution for restricted nonlinear models such as kernel machines. In this paper we resolve these limitations for a general class of kernel space, and our approach is based on a new upper bound of DRRs using an empirical risk regularised by the Lipschitz constant of the model, e.g., deep neural networks and kernel methods. As an application, we showed that it also provides a certificate for adversarial training, and global solutions can be achieved on product kernel machines in polynomial time.

Summary:
This paper uses results from distributional robustness to provide bounds of p-norm-constrained adversarial risk which depend on the Lipschitz constant of the underlying classifier. The bulk of the paper focuses on sample-efficient mechanisms to approximate the Lipschitz constants of kernel methods so that a constraint on this Lip constant can be enforced during training. Empirically, the kernel methods are compared to existing deep learning approaches and are shown to be competitive at this scale.
Overview:
This is a technical paper which draws on theoretical results in Kernel methods, distributional robustness, and numerical approximation. I have been unable to verify the correctness of results from all of these areas but to the best of my knowledge the results seem reasonable. Despite the technical depth, the paper is well-written and largely easy to follow.
1) In section 3.1 you should emphasize the threat model more clearly. Distributional robustness may provide tools to evaluate very general forms of adversarial risk but in this work only the p-norm ball threat model is considered. This is a fine restriction and theoretical statements are made correctly but this does not encompass all forms of adversarial risk.
2) I had a hard time understanding Figure 1. The adversarial risk (Eqn 3) is upper-bounded by the variable-radius risk (Eqn 4 LHS) but Figure 1 shows that the adversarial risk of radius $r$ may permit larger perturbations than the variable-radius risk. Could you please explain how this Figure should be interpreted? The caption and main-text description of the Figure are very slim.
3) I like Figures 2 and 3 as they clearly highlight the benefits of both DRR and the proposed approximations in this work. This paper already presents thorough theoretical analysis but a further analysis into the conditions required for $\lambda_{max}(G^T G)$ to guarantee tighter bounds on the Lipschitz constant would be a valuable addition.
4) Unfortunately I was unable to follow the analysis in Section 4.1 due to a lack of prior knowledge. To my understanding, the per-dimensional product factorization allows cheap exact computation of the off-diagonal elements of $G^T G$. This means that only the diagonal elements need be computed by the Nystr\"om approximation. However, I cannot see intuitively how this prevents the sample complexity from scaling with the dimensionality (the number of diagonal elements still grows with $d$).
5) The experimental set-up is mostly standard with the exception that the CIFAR-10 robustness is measured with respect to a learned representation. This is a reasonable setting for validating the theoretical contributions of this work but is not a realistic interpretation of the adversarial threat model considered. The Lipschitz constant of the embedding function (the pre-trained ResNet) is not known and will likely lead to vacuous bounds (for robustness with respect to the input-space).
6) The deep learning experiments utilize the Parseval networks regularization scheme. What is the regularization constant ($\beta$) used for this method? Typically, a small $\beta << 1$ value is chosen which enables easier learning but prevents the Lipschitz constant from being tightly enforced during training. Is the final Lipschitz constant upper-bound computed from the learned weight matrices to ensure that the orthogonality constraint is not violated?
7) There is some disparity between the experimental results presented and the theory in the paper. In particular, this paper explores methods to provide robustness certificates to Lipschitz bounded classifiers but does not compute and validate these certificates empirically.
8) The gradient visualizations were surprising to me. The Par-MaxMin model has interpretable gradients in that the target class is visible through the gradients. However, for the Gauss-Lip model the gradients seem to only depend on the images current class and are suggestive of a form of gradient obfuscation. It may be useful to visualize "large perturbation" adversarial examples for the Gauss-Lip classifier --- what characteristics do these images have? Similarly, one could generate so-called "Distal Adversarials" which are generated by sampling random noise and optimizing them towards a target class. Does the Gauss-Lip model generate realistic looking images or is noise classified accurately?
9) Overall, the experiments seem sufficient to conclude that the Gauss-Lip model is a promising approach to adversarial learning. It is not clear how this method could be scaled to high dimensional data (while there are more obvious avenues for the deep learning alternatives).
Minor:
- In Theorem 1 statement, "If additionally if $\ell_f$ is convex".


In sequential tasks, planning-based agents have a number of advantages over model-free agents, including sample efficiency and interpretability. Recurrent action-conditional latent dynamics models trained from pixel-level observations have been shown to predict future observations conditioned on agent actions accurately enough for planning in some pixel-based control tasks. Typically, models of this type are trained to reconstruct sequences of ground-truth observations, given ground-truth actions. However, an action-conditional model can take input actions and states other than the ground truth, to generate predictions of unobserved counterfactual states. Because counterfactual state predictions are generated by differentiable networks, relationships among counterfactual states can be included in a training objective. We explore the possibilities of counterfactual regularization terms applicable during training of action-conditional sequence models. We evaluate their effect on pixel-level prediction accuracy and model-based agent performance, and we show that counterfactual regularization improves the performance of model-based agents in test-time environments that differ from training.

The paper presents regularization techniques for model based reinforcement learning which attempt to build counterfactual reasoning into the model. In particular, they present auxiliary loss terms which can be used in "what if" scenarios where the actual state is unknown. Given certain assumptions, they show that this added regularization can improve generalization to unseen problem settings. Specifically they propose two forms of regularization: (1) enforcing that for different actions the predicted next state should be different (action-control) and (2) enforcing that when certain parts of the low dimensional state are perturbed, over a model rollout the perturbation should only affect the perturbed parts of the state, essentially encouraging the latent space features to be independent (disentanglement).
Overall the idea is well motivated - incorporating counterfactual reasoning into model based RL has potential to to improve generalization. Also, while the assumptions needed for the regularization to be correct are not always true, they do seem to hold in many cases. Lastly, the results do seem to indicate that generalization is slightly improved when using the proposed forms of regularization.
My criticisms are:
(1) As mentioned in the paper Action-Control assumes that at every single timestep the agent has potential to change the state. However there may be settings where the agent can always change state, but only a small component of the state. In these cases the states should be quite similar. For example a robot only moving a single object when the state consists of many objects. Also as mentioned in the paper Disentanglement will not work in stochastic environments. One concern I have is that since different environments can violate the assumptions to varying degrees, it seems like actually using the regularization and picking the correct hyperparameter to weight it will be very challenging.
(2) The current results are only demonstrated in a single, custom environment. Additionally performance is shown on only 2 test tasks, and in all cases in Table 2 it is unclear how to interpret the reward. Does this performance constitute completing the task? What is the best possible cumulative reward in this case? The performance improvement seems small, but it is difficult to judge without knowing the details of the task.
I think the paper would be significantly improved by (1) adding experiments in more environments, especially standard model based RL environments where the performance of many existing methods is known and (2) adding comparisons to other forms of model regularization, for example using an ensemble of models. My current rating is Weak Accept.
Some other questions:
- In Table 2 does MPC amount to PlaNet?
- How sensitive are the current numbers to planning parameters (horizon, num samples)?
- Can you provide error bars for the numbers in the tables?
______________________________________________
After author responses and closer examination of the paper I have some additional concerns about experimental details.  Changing my score from 'Weak Accept' to 'Weak Reject'


We consider the problem of translating, in an unsupervised manner, between two domains where one contains some additional information compared to the other. The proposed method disentangles the common and separate parts of these domains and, through the generation of a mask, focuses the attention of the underlying network to the desired augmentation alone, without wastefully reconstructing the entire target. This enables state-of-the-art quality and variety of content translation, as demonstrated through extensive quantitative and qualitative evaluation. Our method is also capable of adding the separate content of different guide images and domains as well as remove existing separate content. Furthermore, our method enables weakly-supervised semantic segmentation of the separate part of each domain, where only class labels are provided. Our code is available anonymously at http://bit.ly/2mXTizX.

This work proposed a mask based approach for instance-level unsupervised content transfer, which is an extension of the disentanglement work in (Press et al., 2019) and the attention guided translation (Chen et al., 2018, Mejjati et al., 2018). Unlike the disentanglement work, the introduced mask allows the adaptation to focus on the relevant content which substantially reduce the complexity of the generation. On the other hand, the proposed method extends the attention guided translation from the domain level to the instance level which allows more specific and diverse translations. Experiments on benchmark data shows both improved qualitative and quantitative results comparing to existing methods. It is really nice that the authors also considered the situation of generalization to out of domain images.
However, I would encourage the authors to spend more discussion on the "Method" and "Ablation Analysis" sections to give a better illustration. First is the choice of the L2 norm in all the reconstruction losses, which is different from L1 norm used in both (Press et al., 2019) and (Mejjati et al., 2018). What is the advantage of using L2 instead of L1 norm here? Does it work better with the mask generation? Second, the domain confusion loss. The presence of both equation (3) and (4) are quite confusing and the domain confusion loss (3) seems different from traditional ones. In Table 7, it shows that the learned mask is empty without any of the losses (3), (5), (7). But only loss (7) is directly related to the mask generation. How does the loss (3) or (5) impact the mask learning? It is also unclear why the losses introduced in (8) would encourage the mask to be minimal despite the quantitative results shown in Table 7. Actually, I am very curious about the performance of the loss introduced in (Press et al., 2019) on top of the network introduced in Figure 2.
Other comments:
- It would be nice to see the out of domain transfer in the "attribute" domain. Ideally, the network should be able to detect "difference" in the image from domain B and apply it to the image from domain A. For example, the model is trained on faces without and with glasses, but applied to faces without and with facial hair. Indeed, the introduction of mask alleviates the decoder to learn the attribute itself, and provides the ability to locate the place of difference.
- Please unify the citation style: there are both Press et al. (2019) and (Press et al., 2019) used.
- In Section 2 under "Mask Based Approaches", the authors argued that the existing attention guided translation "does not allow for the adaptation of the image information in the masked area". I do not think this is the case. The existing work also introduced adaptation of the image information in the masked area. For example, the equation (1) in (Mejjati et al., 2018).
- How is the binarized mask generated in inference? Specifically, how to determine the threshold?
- In Section 4.1, the authors argued that "without L_{Cycle} the masks produced include larger portions of the face". But this actually produces the second smallest mask in Table 7.
- What is the "L2 reg" in Table 7?
- It would be good to show the sensitivity of the lambdas in the overall loss.


Transformer has become ubiquitous in natural language processing (e.g., machine translation, question answering); however, it requires enormous amount of computations to achieve high performance, which makes it not suitable for real-world mobile applications since mobile phones are tightly constrained by the hardware resources and battery. In this paper, we investigate the mobile setting (under 500M Mult-Adds) for NLP tasks to facilitate the deployment on the edge devices. We present Long-Short Range Attention (LSRA), where some heads specialize in the local context modeling (by convolution) while the others capture the long-distance relationship (by attention). Based on this primitive, we design Mobile Transformer (MBT) that is tailored for the mobile NLP application. Our MBT demonstrates consistent improvement over the transformer on three well-established language tasks: IWSLT 2014 German-English, WMT 2014 English-German and WMT 2014 English-French. It outperforms the transformer by 0.9 BLEU under 500M Mult-Adds and 1.2 BLEU under 100M Mult-Adds on WMT'14 English-German. On WMT'14 English-French, our MBT reduces the computation of the transformer by 2.5× with negligible BLEU degradation. Without the costly architecture search that requires more than 250 GPU years, our MBT achieves 0.5 higher BLEU than the AutoML-based Evolved Transformer under the mobile setting.

This paper presents a new technique (LSRA) improving Transformer for constrained scenarios (e.g., mobile settings). It combines two attention modules to provide both global and local information separately for a translation task. In this manner, the authors place the attention and the convolutional module side by side, thus having different perspectives (globally and locally) of the sentence. They test their approach to 2 common translation benchmarks.
Enhancing deep learning model efficiency is very important, and the authors succeeded in reducing the computation costs and consequently, the CO2 emissions. But the evaluation results are not so impressive and go in line with other previous efficient deep learning approaches for different domains. I’m not an expert in NLP, but overall results of 10-1000x or more wall clock time reduction with <1-5% loss in accuracy are usually obtained for domains that have seen more optimization for mobile deployment (especially mobile-optimized CNNs like MobileNet). LSRA-based appraoch is slightly better than the original version of Transformer and its evolved version. From the latter (ET) authors seem to take the idea of parallel branches for their architecture. Also, adaptive attention span in Transformer models and all-attention layers have already been investigated to make networks more efficient and simpler for longer sentences. Include clearer ablation studies would be also interesting to support their findings and superior performance.
To summarize, the paper is addressing an important and interesting idea. It is, in general terms a nice engineering paper, but I am not sure about whether the developments and results are relevant/novel enough yet at this point to publish at ICLR.
-------------------------------------------------
Looking at the other comments and the feedback provided by the authors, I have a more positive feeling about the contributions of the paper which are now sufficiently demonstrated. Therefore, I increase my original recommendation to "Weak Accept".


Person re-identification (re-ID) aims at identifying the same persons' images across different cameras. However, domain diversities between different datasets pose an evident challenge for adapting the re-ID model trained on one dataset to another one. State-of-the-art unsupervised domain adaptation methods for person re-ID transferred the learned knowledge from the source domain by optimizing with pseudo labels created by clustering algorithms on the target domain. Although they achieved state-of-the-art performances, the inevitable label noise caused by the clustering procedure was ignored. Such noisy pseudo labels substantially hinders the model's capability on further improving feature representations on the target domain. In order to mitigate the effects of noisy pseudo labels, we propose to softly refine the pseudo labels in the target domain by proposing an unsupervised framework, Mutual Mean-Teaching (MMT), to learn better features from the target domain via off-line refined hard pseudo labels and on-line refined soft pseudo labels in an alternative training manner. In addition, the common practice is to adopt both the classification loss and the triplet loss jointly for achieving optimal performances in person re-ID models. However, conventional triplet loss cannot work with softly refined labels. To solve this problem, a novel soft softmax-triplet loss is proposed to support learning with soft pseudo triplet labels for achieving the optimal domain adaptation performance. The proposed MMT framework achieves considerable improvements of 14.4%, 18.2%, 13.4% and 16.4% mAP on Market-to-Duke, Duke-to-Market, Market-to-MSMT and Duke-to-MSMT unsupervised domain adaptation tasks.

After reading the reviews and the comments, I confirm my rating.
=================
The paper proposes an unsupervised framework to address the problem of noisy pseudo labels in clustering-based unsupervised domain adaptation (UDA) for person re-identification. The noise derives from the limited transferability of source-domain features, the unknown number of target-domain identities, and the imperfect results of the clustering algorithm.
The proposed framework, Mutual Mean-Teaching (MMT), performs pseudo label reﬁnery by optimizing the neural networks under the joint supervisions of off-line reﬁned hard pseudo labels and on-line reﬁned soft pseudo labels. Inspired by the teacher-student approaches (Reference: Tarvainen & Valpola, 2017; Reference: Zhang et al., 2018b), the proposed MMT framework provides robust soft pseudo labels in an on-line peer-teaching manner to simultaneously train two same networks. The networks gradually capture target-domain data distributions and thus reﬁne pseudo labels for better feature learning.
The main contribution is proposing an unsupervised framework (MMT) capable of tackling the noise problem in state-of-art UDA methods for person re-identification, via producing reliable soft labels in order to achieve better performance. Since the conventional triplet loss cannot properly work with soft labels, a softmax-triplet loss is proposed to enable training with soft triplet labels for mitigating the pseudo label noise.
The proposed MMT is evaluated on Market1501, DukeMTMC-reID, and MSMT17 datasets with four adaptation tasks: Market-to-Duke, Duke-to-Market, Market-to-MSMT, and Duke-to-MSMT. It outperforms the state-of-the-art methods with significant improvements in terms of Mean average precision (mAP) and Cumulative matching characteristic (CMC). In addition to that, ablation studies conducted to evaluate each component in the proposed MMT framework.


In the past few years, off-policy reinforcement learning methods have shown promising results in their application for robot control. Deep Q-learning, however, still suffers from poor data-efficiency which is limiting with regard to realworld applications. We follow the idea of multi-step TD-learning to enhance data-efficiency while remaining off-policy by proposing two novel TemporalDifference formulations: (1) Truncated Q-functions which represent the return for the first n steps of a target-policy rollout w.r.t. the full action-value and (2) Shifted Q-functions, acting as the farsighted return after this truncated rollout. We prove that the combination of these short-and long-term predictions is a representation of the full return, leading to the Composite Q-learning algorithm. We show the efficacy of Composite Q-learning in the tabular case and compare our approach in the function-approximation setting with TD3, Model-based Value Expansion and TD3(∆), which we introduce as an off-policy variant of TD(∆). We show on three simulated robot tasks that Composite TD3 outperforms TD3 as well as state-of-the-art off-policy multi-step approaches in terms of data-efficiency.

This paper proposes the Composite Q-learning algorithm, which combines the algorithmic ideas of using compositional TD methods to truncate the horizon of the return, as well as shift a return in time. They claim that this approach will improve the method's data efficiency relative to standard Q-learning. They demonstrate its performance relative to Q-learning in a tabular domain, as well as in deep RL domains which use the compositional idea as an off-policy critic.
Overall, the paper has interesting algorithmic ideas, but there are critical issues in the evaluation and resulting claims being made. Based on this, I am recommending rejection of the paper. I do think there is value in the compositional idea, but for different reasons outlined in the suggestions.
Issues:
1) The truncation of the horizon is not a novel TD formulation, as claimed in the paper. This algorithm is described in the original TD paper (Sutton, 1988) as "prediction by a fixed interval." Sutton's group further has a recent paper following up on the fixed-horizon TD (FHTD) idea (De Asis et al., 2019), introducing an off-policy control variant of it.
2) Based on Theorem 1 of the TD(\Delta) paper (Romoff et al., 2019), as well as the sample-complexity arguments from the FHTD paper, this compositional algorithm is *exactly* equivalent to standard TD in the tabular setting (and function approximation if value functions don't share parameters), update for update, assuming that: (1) each value function is initialized identically, and (2) the same step size is used for each value function. An intuition for why is because the accuracy of the shifted action-values depends on the accuracy of the standard TD estimate, and the TD errors can be shown to exactly decompose that of standard TD. Under this, there is no ready improvement in data efficiency due to the fixed-horizon value functions converging quicker.
3) The results in the tabular setting seem to contradict what I described in Issue 2, because compositional Q-learning as presented did converge quicker than standard Q-learning. However, this is misleading in that the other methods used a step size of 1e-3, but the step size of the shifted value functions used, without explanation, a larger step size of 1e-2. The reason for the improved performance is that these values had a step size an order of magnitude larger than the remaining ones, and if one were to use the same step size across all value functions, it would have matched Q-learning exactly. This exact decomposition is supported by how the fixed-horizon value estimates follow Q-learning's curves exactly for the first h - 1 updates (and will converge to Q-learning's curve if h approaches infinity), and can further be verified by running the provided code with a step size of 1e-3 for the shifted value functions. Without acknowledging the equivalence when using a consistent step size across value functions, as well as sweeping over step sizes for each method, the results don't present a fair comparison and significantly misrepresent compositional TD methods.
4) On this observation that it is an exact decomposition of TD, it is particularly an exact decomposition of *one-step* TD, as one-step TD errors are used in the fixed-horizon and shifted value function estimates. This makes it equally biased to a one-step method, and is inconsistent with the use of "multi-step" learning in the literature where information across several time steps is included in the estimate of the return. Truncating and shifting things in time can be contextualized as a form of time-dependent discounting, and adjusting the discount rate isn't generally viewed as performing multi-step TD.
5) Based on the above, the benefit in the deep RL setting is not convincingly due to what is claimed (as parameters are shared, and a consistent step size is used in the optimizer). Some possible reasons might include the architectural choices in how the network represented the decomposition, as well as the representation learning benefits of predicting many relevant outputs to a task.
Suggestions:
1) The precise novelty of the work can be clarified, as the fixed-horizon TD formulation dates back to Sutton (1988), and has been extensively studied in De Asis et al. (2019). As far as I'm aware, there's novelty in the idea of shifting value functions, reconstructing the full return from decomposed value functions, and introducing a penalty to the loss based on inconsistencies in the value estimates.
2) The motivation and claims of the paper should be revised, as the claimed data efficiency from fixed-horizon values converging quicker isn't readily true. The resulting deep RL results may need more careful experiments to tease apart why the composition might be helping. For example, it might be useful to compare a different neural network architecture, like having all of the compositional components as outputs from the same, final hidden layer (in comparison with outputting them from intermediate hidden layers).
3) The tabular example needs to be re-worked to ensure a fair comparison between each algorithm. For example, the curves can be presented under the best step size (in terms of some metric, like area under the curve) for each algorithm. While there is an exact equivalence to standard one-step TD methods, a real benefit of the approach is that strictly more information is present to the agent, and the flexibility of being able to use separate step sizes for each value function can be favorable if it can be shown to be better after fairly tuning each algorithm. Shifting the focus toward showing that certain types of value functions are less sensitive to step sizes or work better operating at different time-scales from other components (because this seems to be what's actually happening in the results) would be a huge plus for this.
4) Because it is using one-step TD errors to estimate each of these components, and is equally biased to one-step TD, it isn't really a multi-step method. I think it would be better to emphasize the compositional aspect and its increased flexibility, than frame it as a multi-step off-policy method.
----------
Post-rebuttal:
I think the additional results post-discussion are good, and are on the right track of the claimed goal of analyzing the algorithm. However, the new results might be contradictory to some of the claims made earlier in the paper, and so a more involved revision seems to be needed. I do believe the algorithm has promise for the reasons teased apart in our discussion, and encourage the authors to improve their paper with these results.
To detail a few things:
1) The new results, which now empirically demonstrate the exact equivalence with one-step Q-learning, contradicts some claims about improved data efficiency due to truncated value-functions converging quicker. While meta-parameter selection isn't the focus, if the choice of meta-parameter is what can make it differ from vanilla Q-learning, and is the key explanation for the improvements, then the analysis should focus on this.
2) Mention of the equivalence only comes up in the experimental results, when it's a key property of the algorithm. If analysis of the composition is the paper's focus, acknowledging this property is foundational to any analysis of the method. It could have been shown analytically following the algorithm's derivation, and would have better justified some of the choices made in the experiments.
3) Being equivalent to running *one-step* Q-learning still makes the "multi-step" learning emphasis appear incorrect, especially when the algorithm can trivially be extended to use actual multi-step TD methods. The title seems to come from interpreting what the composite values represent, but the horizon isn't what makes a method multi-step, and the compositional components add up to exactly one-step Q-learning's update.
Minor:
1) Arguably one of the most prevalent explanations in the deep RL literature for why one might expect improvements is the multi-task/auxiliary task hypothesis (Jaderberg et al., 2016).


Super Resolution (SR) is a fundamental and important low-level computer vision (CV) task. Different from traditional SR models, this study concentrates on a specific but realistic SR issue: How can we obtain satisfied SR results from compressed JPG (C-JPG) image, which widely exists on the Internet. In general, C-JPG can release storage space while keeping considerable quality in visual. However, further image processing operations, e.g., SR, will suffer from enlarging inner artificial details and result in unacceptable outputs. To address this problem, we propose a novel SR structure with two specifically designed components, as well as a cycle loss. In short, there are mainly three contributions to this paper. First, our research can generate high-qualified SR images for prevalent C-JPG images. Second, we propose a functional sub-model to recover information for C-JPG images, instead of the perspective of noise elimination in traditional SR approaches. Third, we further integrate cycle loss into SR solver to build a hybrid loss function for better SR generation. Experiments show that our approach achieves outstanding performance among state-of-the-art methods.

In this article, the authors propose a single image super-resolution network that can generate high-resolution images from the corresponding C-JPG images. The method contains two main parts, namely a JPG recovering step, which recovers the information from low-quality JPG images, and an SR generation step, which generates SR images from the images achieved by the recovering step. Moreover, the authors leverage a cycle loss to generate better results.
The main contribution of this work is only the integration of existing models. The authors claimed that the proposed method is lossless, while there is no evidence to demonstrate it. The authors should show more evidence about the JPG recovering step, like how much information it can recover. Moreover, the SR generation step only incorporates s-LWSR without any improvement. It makes SR in this manuscript more like an application for the JPG recovering method rather than a contribution to the SISR field.
In the experimental section, Figure 5 makes readers confused. Does the image entitled “s-LWSR(Training)” mean the “STRAIGHT TRAINING” described in section 4.2.1? If yes, is there any perceptual difference between the result of s-LWSR and the result of ours? It is suggested that the authors should reorganize the results and provide more instructions. In Table 1, the results derived from s-LWSR32(C-JPG) and the proposed are very similar. The authors should more convincingly show the advantages of the proposed method. From the results, I observe that the SR images of the proposed model are blurry and lack much information about textures. At the same time, there are some other SISR studies, especially GAN based models like ESR-GAN, which show visual quality with more realistic and natural textures. I hope the authors can conduct more comparisons with these methods.
There are still some issues as follows:
1.	The authors should carefully check the format of the references in the whole article. For instance, in section 4.1, almost all references are in the wrong format. The same mistake happens in the caption of Figure 6. Please check the full article before submission.
2.	(Page 1, line 2 from bottom) Please add a reference to “bicubic”.
3.	(Section 3, line 1) Please add a full stop after “Challenge Formulation”.
4.	(Figure 2) Please enlarge the arrow of the red lines. They are hard to read right now.
5.	(Figure 4) The figure seems to miss the skip connect of the former five layers, which should be a part of the input added to the latter four layers (Li et al., 2019). In addition, please enlarge the arrow of the blue lines.
6.	(Section 4.2.1, line 4) “Both of the PSNR …” Does figure 5 can reflect this? Please add data instruction.


Neural Combinatorial Optimization (NCO) has shown the potential to solve traditional NP-hard problems recently. Previous studies have shown that NCO outperforms heuristic algorithms in many combinatorial optimization problems such as the routing problems. However, it is less efficient for more complicated problems such as packing, one type of optimization problem that faces mutual conditioned action space. In this paper, we propose a Conditional Query Learning (CQL) method to handle the packing problem for both 2D and 3D settings. By embedding previous actions as a conditional query to the attention model, we design a fully end-to-end model and train it for 2D and 3D packing via reinforcement learning respectively. Through extensive experiments, the results show that our method could achieve lower bin gap ratio and variance for both 2D and 3D packing. Our model improves 7.2% space utilization ratio compared with genetic algorithm for 3D packing (30 boxes case), and reduces more than 10% bin gap ratio in almost every case compared with extant learning approaches. In addition, our model shows great scalability to packing box number. Furthermore, we provide a general test environment of 2D and 3D packing for learning algorithms. All source code of the model and the test environment is released.

This paper aims at solving geometric bin packing (2D or 3D) problems using a deep reinforcement learning framework. Namely, the framework is based on the actor-critic paradigm, and uses a conditional query learning model for performing composite actions (selections, rotations) in geometric bin packing. Experiments are performed on several instances of 2D-BPP and 3D-BPP,
Overall, bin packing problems are challenging tasks for DRL, and I would encourage the authors to pursue this research topic. Unfortunately, I believe that the current manuscript is at a too early stage for being accepted at ICLR, due to the following reasons:
(a) The paper is littered with spelling/grammar mistakes (just take the second sentence: “With the developing” -> “development”). For the next versions of the manuscript, I would recommend using a spell/grammar checker.
(b) In the related work section, very little is said about Bin Packing Problems. There are various classes of BPPs, and it would be relevant to briefly present them. Moreover, BPPs have been extensively studied in theoretical computer science, with various approximation results. Again, a brief discussion about those results would be relevant. Notably, several classes of geometric bin packing problems admit polynomial-time approximation algorithms (for extended surveys about this topic, see e.g.  Arindam Khan’s Ph.D. thesis 2015; Christensen et. al. Computer Science Review 2017).
(c) According to the problem formulation and the experiments, it seems that the authors are studying a restricted subclass of 2D/3D bin packing problems: there is only “one” bin, so (it seems that) the authors are dealing with geometric knapsack problems (with rotations). Note that the 2D Knapsack problem with rotations admits a 3/2 + \epsilon - approximation algorithm (Galvez et. al., FOCS 2017). A. Khan has also found approximation algorithms for the 3D Knapsack problem with rotations. So, even if those results do not preclude the use of sophisticated DRL techniques for solving geometric knapsack problems, it would be legitimate to empirically compare these techniques with the polytime asymptotic approximation algorithms already found in the literature.
(d) The problem formulation is very unclear. Namely, the state representation is ambiguous: $s_p$ is obviously not a boolean variable, but a boolean vector (where each component is associated with an item). Nothing is said about actions and transitions and rewards (we have to read the AC framework in order to get a clue of these components). We don’t know if it is an episodic MDP (which is usually the case in DRL approaches to combinatorial optimization tasks). Also, it seems that the MDP is specified for a single instance of 3D-BPP. But this looks wrong since it should include the distribution of all instances of 3D-BPP.
(e) The Actor-Critic framework, coupled with a conditional query learning algorithm, is unfortunately unintelligible due to the fact that many notations are left unspecified. For example, in Eq (1) what are the dimensions K and V? In Eq (2) what is d_i? In the algorithm what is n_{gae}? Also in the algorithm, what are l’_i, w’_i and h’_i? Etc.
(f) Even if the aforementioned issues are fixed, it seems that the framework is using many hyper-parameters (\gamma, \beta, \alpha_t, etc.) which are left unspecified. Under such circumstances, it is quite impossible to reproduce experiments.


Recently, the paradigm of unfolding iterative algorithms into finite-length feedforward neural networks has achieved a great success in the area of sparse recovery. Benefit from available training data, the learned networks have achieved state-of-the-art performance in respect of both speed and accuracy. However, the structure behind sparsity, imposing constraint on the support of sparse signals, is often an essential prior knowledge but seldom considered in the existing networks. In this paper, we aim at bridging this gap. Specifically, exploiting the iterative reweighted 1 minimization (IRL1) algorithm, we propose to learn the cluster structured sparsity (CSS) by rewegihting adaptively. In particular, we first unfold the Reweighted Iterative Shrinkage Algorithm (RwISTA) into an end-to-end trainable deep architecture termed as RW-LISTA. Then instead of the element-wise reweighting, the global and local reweighting manner are proposed for the cluster structured sparse learning. Numerical experiments further show the superiority of our algorithm against both classical algorithms and learning-based networks on different tasks.

The paper combines deep learning and compressed sensing. Specifically the RW-LISTA algorithm is proposed for cluster-structured sparse recovery, building upon two existing methods: the Reweighted Iterative Shrinkage Algorithm and the LISTA algorithm. The reweighing process is employed to infer the dependencies between coefficients and encourage cluster structure. Strategies for local and global dependence are presented. The approach is evaluated on synthetic and real datasets.
The paper considers and important topic. However the proposed approach is too incremental and the empirical evaluation could also be improved. In addition the presentation needs more work.  Specifically:
- the use of unsupervised vs supervised is misleading. Traditional CS approaches are in fact supervised as they map to a regression problem when both input sensing matrix and response vector are available. The distinction has more to do with the ability (or lack of ability) to learn representations.
- the proposed approach is a straightforward combination of LISTA and RwISTA and the section of global/local dependence regarding cluster-sparse structures is unsurprising.
- Even though signals may exhibit cluster-sparsity, the size of such cluster might vary widely and it is therefore questionable if such patterns can be best captured via connections in the proposed reweighing blocks. Indeed for some blocks wider or lower neighborhoods might be needed to capture various radii of dependence.
- As an alternative to adopting RwISTA it might be pertinent to compare against a counterpart using fused lasso penalty (Tibshirani et al 2005).
- Experiments are limited: a wider variety of block structure with more or less variability in block size etc should be considered. In addition it would be important to compare against vanilla  Rw-ISTA, as one of the classical CSS solvers.
- It is somewhat disappointing that the proposed approach should be better the higher the SNR, where other approaches can do well enough. Ideally we are looking to improve in less favorable conditions of low SNR.


We address the following question: How redundant is the parameterisation of ReLU networks? Specifically, we consider transformations of the weight space which leave the function implemented by the network intact. Two such transformations are known for feedforward architectures: permutation of neurons within a layer, and positive scaling of all incoming weights of a neuron coupled with inverse scaling of its outgoing weights. In this work, we show for architectures with non-increasing widths that permutation and scaling are in fact the only function-preserving weight transformations. For any eligible architecture we give an explicit construction of a neural network such that any other network that implements the same function can be obtained from the original one by the application of permutations and rescaling. The proof relies on a geometric understanding of boundaries between linear regions of ReLU networks, and we hope the developed mathematical tools are of independent interest.

In this paper, the authors studied the equivalence class of ReLU networks with non-increasing weights, and proved that permutation and scaling are the only function preserving weight transformations. The proof technique is novel, and provides some insights in the geometry space of the loss surface. I think the proof technique could have its general implications to some other research, however, the direct value of this paper is not very clear.
1)	The paper starts with the discussions on the redundancy introduced by over-parametrization, as one of its motivations. However, what is discussed in this paper is actually far distant from over-parametrization. The redundancy in over-parametrized networks and the redundancy in ReLU networks are different concepts and the connection between them is not well established. The over-parametrization is talking about the smooth information flow brought by wide intermediate layers, however, the equivalence class in this paper is more about the mathematical properties of the activation functions and the topological connections of the neural networks. I feel that the authors have some confusing understanding on these two concepts.
2)	The theory was established regarding restrictive types of ReLU networks (feedforward, with non-increasing width). However, many widely used networks are not of this kind. Without extensions to other shapes of the networks, convolutional and recurrent structures, and many kind of normalization transformations (e.g., batch norm and layer norm), the practical value of this paper is limited.
3)	Some important references are missing. The following paper has in-depth theoretical analysis on ReLU networks, and characterizes the redundancy brought by positive-scale invariant properties of ReLU networks. It is strange that the authors did not cite it.  It would be necessary for the authors to discuss their additional technical contributions given this ICLR 2019 paper.
-  Q. Meng, et al. G-SGD: Optimizing ReLU Neural Networks in its Positively Scale-Invariant Space, ICLR 2019
4)	The practical implication of the theories in this paper is not clearly discussed. What if we know there are only these two kinds of redundancies? What kind of new algorithms and practices can be inspired by such theoretical understandings.
** I read the author rebuttal. Different people may have different criteria on evaluating a paper and my criterion is not only about "what", but more importantly about "so what". I still think the authors should think harder about the implications of their work, either theoretically or practically. Furthermore, I understand that some published papers also use restricted settings to ease their proofs, however, I personally do not think this is well justified. I think for top conference like ICLR, more solid and less restrictive works are preferred.  I could at most adjust my score to weak reject.


Temporal information is crucial for recommendation problems because user preferences are naturally dynamic in the real world. Recent advances in deep learning, especially the discovery of various attention mechanisms and newer architectures in addition to widely used RNN and CNN in natural language processing, have allowed for better use of the temporal ordering of items that each user has engaged with. In particular, the SASRec model, inspired by the popular Transformer model in natural languages processing, has achieved state-of-the-art results. However, SASRec, just like the original Transformer model, is inherently an un-personalized model and does not include personalized user embeddings. To overcome this limitation, we propose a Personalized Transformer (SSE-PT) model, outperforming SASRec by almost 5% in terms of NDCG@10 on 5 real-world datasets. Furthermore, after examining some random users' engagement history, we find our model not only more interpretable but also able to focus on recent engagement patterns for each user. Moreover, our SSE-PT model with a slight modification, which we call SSE-PT++, can handle extremely long sequences and outperform SASRec in ranking results with comparable training speed, striking a balance between performance and speed requirements. Our novel application of the Stochastic Shared Embeddings (SSE) regularization is essential to the success of personalization. Code and data are open-sourced at https://github.com/SSE-PT/SSE-PT.

The paper proposes SSE-PT for sequential recommendation, which is an extension of previous work SASRec by adding user embedding with SSE  regularization [Wu et al. 2019] . They further extend SSE-PT to SSE-PT++ to handle longer sequence. Experiments on five datasets show that the SSE-PT and SSE-PT++ outperform several baseline approaches.
Detailed comments:
1)	The technical contribution seems to be scattered: user embedding is introduced, effect of different types of regularization is studied and sampling based approach is added to address long sequence. It could be better if the author could make clear what the major contribution of this paper is. Also, SSE [Wu et al. 2019] is existing technique and simply applying it to sequential recommendation is a bit incremental.
2)    In addition to SASRec, there are some other transformer based model (e.g., [1]) for sequential recommendation and the paper discuss how the proposed method differ from them.
3)	In SSE-PT++, would sampling start index v based on the recency (e.g., with exponential decay) make more sense than uniform probability?
4） Overall, experiments look comprehensive: The baseline methods include both non-deep-learning methods and recent deep learning based methods for sequential recommendation; ablation study is conducted; case study is performed on MovieLens to show how the attention weights differ from SASRec; running time is compared against baselines and sensitivity analysis on hyper-parameters are also provided.
To summarize, the paper is a bit incremental/scattered in terms of technical contribution but the execution of this paper looks solid. I would give a “weak accept” to this paper given the reasons listed above.
[1] F. Sun et. al. BERT4Rec: Sequential Recommendation with Bidirectional Encoder Representations from Transformer


Image-to-image (I2I) translation aims to translate images from one domain to another. To tackle the multi-modal version of I2I translation, where input and output domains have a one-to-many relation, an extra latent input is provided to the generator to specify a particular output. Recent works propose involved training objectives to learn a latent embedding, jointly with the generator, that models the distribution of possible outputs. Alternatively, we study a simple, yet powerful pre-training strategy for multi-modal I2I translation. We first pre-train an encoder, using a proxy task, to encode the style of an image, such as color and texture, into a low-dimensional latent style vector. Then we train a generator to transform an input image along with a style-code to the output domain. Our generator achieves stateof-the-art results on several benchmarks with a training objective that includes just a GAN loss and a reconstruction loss, which simplifies and speeds up the training significantly compared to competing approaches. We further study the contribution of different loss terms to learning the task of multi-modal I2I translation, and finally we show that the learned style embedding is not dependent on the target domain and generalizes well to other domains.

In this paper, the authors tackle the problem of multi-modal image-to-image translation by pre-training a style-based encoder. The style-based encoder is trained with a triplet loss that encourages similarity between images with similar styles and dissimilarity between images with different styles. The output of the encoder is a style embedding that helps differentiates different modes of image synthesis.  When training the generator for image synthesis, the input combines an image in the source and a style embedding, and the loss is essentially the sum of image conditional GAN loss and perceptual loss. Additionally, the authors propose a mapping function to sample styles from a unit Gaussian distribution.
I think the idea of pre-training a style-based encoder is straightforward. I am mainly concerned about the performance of the presented approach. First, there are no many visual comparisons in the paper. The only visual comparison is in Figure 8, but results are only limited to faces. The visual results in Figure 5 do not look appealing to me. The change in the style mainly comes from the global change in color: no much change in the texture or local color. The "night2day" results look poor to me.  I am concerned about the diversity of the styles learned in the model.
On the other hand, I am convinced that the proposed model is better than BicycleGAN, and the approach is somehow novel. The user study in Table 5 suggests that the proposed method is somehow better than BicyleGAN in visual quality on one task. My overall rating is borderline.
Minor comments:
- In the first sentence of Section 3.2, I do not think "one-to-one correspondence" is the right description. The encoder is not expected to be invertible.
- In Equation (3), "e_i" is a little bit misleading. It does not mean the i-th element in {"e_j"}. You may want to replace "e_i" with "s_i" to avoid confusion.
- The explanation of ours v1, v2, v3, v4 is not clear. It is also difficult to find its definition.


Evolutionary Strategies (ES) are known to be an effective black-box optimization technique for deep neural networks when the true gradients cannot be computed, such as in Reinforcement Learning. We continue a recent line of research that uses surrogate gradients to improve the gradient estimation of ES. We propose a novel method to optimally incorporate surrogate gradient information. Our approach, unlike previous work, needs no information about the quality of the surrogate gradients and is always guaranteed to find a descent direction that is better than the surrogate gradient. This allows to iteratively use the previous gradient estimate as surrogate gradient for the current search point. We theoretically prove that this yields fast convergence to the true gradient for linear functions and show under simplifying assumptions that it significantly improves gradient estimates for general functions. Finally, we evaluate our approach empirically on MNIST and reinforcement learning tasks and show that it considerably improves the gradient estimation of ES at no extra computational cost.

This paper addresses the issue of noisy gradient estimation in a type of evolution strategies popularized by the open AI's reinforcement learning paper. It is a follow-up paper of reference [14], and try to analyze the optimality of the gradient estimation. The goal of the paper is well stated and well motivated. The paper itself is well-organized. However, the novelty of this work is not sufficiently high and its usefulness is questionable.
Theorem 1 is trivial under the assumption stated above the theorem---the numerical approximation of the directional derivative admits the true directional derivative. In other words, the assumption is too strong to claim the goodness of the proposed scheme.
Let's just sample k+P random normal vectors and orthogonalize and normalize them. Let them denoted by the same symbols hat zeta and hat epsilon. The theorem statements holds for this case. Therefore I failed to understand the essential claim of Theorem 1.
About Theorem 2 and 3, again, the assumption that he numerical approximation of the directional derivative matches the true directional derivative is too strong to make the claim relevant. Moreover, the effect of P somehow disappear from the analysis.
All the analysis is done assuming the above mentioned strong assumption. However, in one of the experiments and the existing works, ranking based fitness shaping has been applied to make the algorithm robust. This replaces the function value differences in the gradient estimator with some predefined values depending on the ranking of f-values of each trial vector. This definitely violates the assumption, and it may result in some vector far away from the true gradient, yet the algorithm still works well. Therefore, the hypothesis underlying in this paper---better estimation of the gradient will lead to a better performance---may not be true. At least the numerical experiments provided in this paper do justify this hypothesis.
The numerical experiments have been conducted to compare the proposed algorithm with the baseline ES algorithm. In a sense it is reasonable to evaluate the effect of the proposed modification in the baseline ES. However, since the baseline ES algorithm is not really efficient on tasks such as the one conducted in Figure 2, the usefulness of the proposed approach is not tested. At least one should compare with the "canonical" ES, where the learning rate fixed and sigma is adapted. See https://arxiv.org/pdf/1802.08842.pdf.
Use of the search history proposed in this paper is not really new in ES community. A sort of momentum terms appears in the standard CMA-ES [18] even in two parts of the algorithm and its effectiveness is well-studied empirically. This paper addresses the theoretical aspect of the momentum and this may be new. However, as mentioned above, the assumption is too strong to describe the reality.
"Linear time approximations of CMA-ES like diagonal approximations of the covariance matrix (19) often do not work well." Please specify in what sense the linear time version of the CMA-ES do not work well and provide the evidence (references).


A well-known issue of Batch Normalization is its significantly reduced effectiveness in the case of small mini-batch sizes. When a mini-batch contains few examples, the statistics upon which the normalization is defined cannot be reliably estimated from it during a training iteration. To address this problem, we present Cross-Iteration Batch Normalization (CBN), in which examples from multiple recent iterations are jointly utilized to enhance estimation quality. A challenge of computing statistics over multiple iterations is that the network activations from different iterations are not comparable to each other due to changes in network weights. We thus compensate for the network weight changes via a proposed technique based on Taylor polynomials, so that the statistics can be accurately estimated and batch normalization can be effectively applied. On object detection and image classification with small mini-batch sizes, CBN is found to outperform the original batch normalization and a direct calculation of statistics over previous iterations without the proposed compensation technique.

Paper summary: This paper proposes a new normalization technique specially designed for settings with small mini-batch sizes (where previous methods like BatchNorm are known to suffer). The approach aggregates mean/variance statistics from previous iterations, weighted based on the Taylor expansion, to get a better estimate of population statistics. The authors evaluate their approach on ImageNet classification, and object detection and instance segmentation on the COCO dataset.
The paper is clearly written and explores an interesting idea---aggregating mini-batch statistics across iterations. That being said, I am not convinced by the utility of the proposed approach since it doesn’t offer over significant benefits over prior approaches designed for the small mini-batch setting (e.g., Group Normalization)---either in terms of empirical performance, implementation complexity, or lower computational/memory requirements.
Specific comments/questions:
1. Why do the authors not include the Kalman Normalization baseline in the paper? Based on my understanding, it is also designed for the low-sample regime (and the original paper also conducts experiments on ImageNet/COCO). Also the BRN baseline is included in Table 3 for ImageNet but is missing from the COCO experiments. It is important to thoroughly compare to other normalization techniques specifically designed for this regime to clearly highlight relative benefits of the proposed approach.
2. The authors mention that they average performance across 5 trials but omit confidence intervals in their Figures/Tables. Since the difference in performance between the different approaches compared is small, I think confidence intervals should be reported to see whether improvements are within statistical error margins. In fact, for CIFAR-10 based on Table 7 in the Appendix, performance of batch renormalization, group normalization and CBN is essentially the same.
3. In Table 3, how many iterations was the BRN aggregation performed over? Was it also chosen to ensure effective number of samples was not less than 16 (like CBN). What do Figures 3 and 4 look like for BRN?
4. One thing I find interesting (the authors do not discuss this specifically) about Figure 5 in the Appendix is that both for CIFAR and COCO (the two datasets for which plots are provided), CBN helps as long as it is used before the final learning rate drop. Specifically, choosing a burn-in period as large as 120 for CIFAR and 8 for COCO is fine (in fact, probably the best) as long as you turn on CBN before the final learning rate drop. This makes me curious whether BN in the low-sample regime only suffers in terms of generalization performance in the final stages of training (compared to low-sample alternatives like GN/CBN, etc).
The authors should include other recent benchmarks (Kalman normalization and BRN in the omitted Tables) and error bars to make the change in performance clearer. It would also be interesting to see whether (and probably make the paper stronger if) alternatives like GN/KN benefit from being combined with the proposed scheme to aggregate statistics over time.
Overall, while the proposed approach is novel, its performance is comparable to prior approaches, with the disadvantage of an additional computational/memory footprint. Thus, I am not yet convinced about how useful/interesting this approach would be to the community.


In this work, we address the semi-supervised classification of graph data, where the categories of those unlabeled nodes are inferred from labeled nodes as well as graph structures. Recent works often solve this problem with the advanced graph convolution in a conventional supervised manner, but the performance could be heavily affected when labeled data is scarce. Here we propose a Graph Inference Learning (GIL) framework to boost the performance of node classification by learning the inference of node labels on graph topology. To bridge the connection of two nodes, we formally define a structure relation by encapsulating node attributes, between-node paths and local topological structures together, which can make inference conveniently deduced from one node to another node. For learning the inference process, we further introduce meta-optimization on structure relations from training nodes to validation nodes, such that the learnt graph inference capability can be better self-adapted into test nodes. Comprehensive evaluations on four benchmark datasets (including Cora, Citeseer, Pubmed and NELL) demonstrate the superiority of our GIL when compared with other state-of-the-art methods in the semi-supervised node classification task.

This paper proposes to leverage the between-node-path information into the inference of conventional graph neural network methods. Specifically, the proposed method treats the nodes in training set as a reference corpus and, when infering the label of a specific node, makes this node "attend" to the reference corpus, where the "attention" weights are calculated based on the node representations and the between-node paths. (The paper used different terms about the "attention".)
In general, this paper shares similar insights like those in memory networks, which use a reference corpus to assist the inference on an individual sample. The major technical novelty of this paper seems to lie in the introduction of this idea on graphs and the use of between-node-path information for the "attention mechanism".
The paper is also well-organized and relatively easy to follow.
One of my major concerns is the unfair experimental comparison. The proposed method uses both the training set and validation set to train the model. And it's well-known that the offical split of the Cora, Citeseer, and Pubmed datasets has a validation set larger than the training set. As most of the baseline performances in Table 1 are obtained by using the training set only, the comparison of Table 1 is unfair and totally meaningless. A valid comparison is Table 2, where the baseline GCN is trained on both training set and validation set. But only the results on Cora is given and results on all other datasets are missing. Actually, I tried a quick run of GCN, without much tuning of hyper-parameters, on Citeseer with both training set and validation set as training data. And it can easily achieve a test accuracy of 0.75+, which is better than the performance of the proposed method reported in the paper.
Another concern is that, while the proposed method is intuitively sound, it is unclear how this method compares to naively increasing the number of layers of the GNN. Increasing the number of layers of the GNN can also eventually make the test node able to interact with all the training nodes, and the model should be able to implicitly distinguish the distances between nodes. Why incorporating between-node paths, and particularly in this way, helps?
Some minor points:
	- Could you clarify how the figure 2 is generated? By grouping the test nodes with differen between-nodes steps? It is confusing why performance on the nodes far away from the training nodes is better?
	- Why figure 3 is on PubMed while all other ablation analyses are done on Cora?
In summary, this paper shows some technical novelty but the motivation behind the proposed method is not strong enough. The experiment design is questionable and especially the main experiment comparison Table 1 is meaningless. Therefore I think this work is not ready for publish yet.


Generative adversarial networks (GANs) are capable of producing high quality samples, but they suffer from numerous issues such as instability and mode collapse during training. To combat this, we propose to model the generator and discriminator as agents acting under local information, uncertainty, and awareness of their opponent. By doing so we achieve stable convergence, even when the underlying game has no Nash equilibria. We call this mechanism implicit competitive regularization (ICR) and show that it is present in the recently proposed competitive gradient descent (CGD). When comparing CGD to Adam using a variety of loss functions and regularizers on CIFAR10, CGD shows a much more consistent performance, which we attribute to ICR. In our experiments, we achieve the highest inception score when using the WGAN loss (without gradient penalty or weight clipping) together with CGD. This can be interpreted as minimizing a form of integral probability metric based on ICR.

https://openreview.net/pdf?id=SkxaueHFPB
The paper has some interesting ideas but I don’t think any of them are fully fleshed out.
I find the reporting of Inception Score highly suspect. The authors choose WGAN-GP as a baseline and report scores of ~4.5 vs ~5.5 with their modification. However the WGAN-GP paper reports an IS of 7.86 on CIFAR. Furthermore, current GAN SOTA on CIFAR is approaching IS=9. I am not making the argument that the authors ought to demonstrate SOTA results, however they should at least present results which are consistent with the published results of their chosen baseline.
The authors then make this statement:
“Thus, its superior performance supports the claim that ICR is the appropriate form of regularization for GANs. We emphasize that in our experiments we did not perform any architecture or hyperparameter tuning, and instead use a model intended to be used with WGAN gradient penalty”
This does not hold, since the numbers reported are far below the actual baseline.
Besides this major point, I am unconvinced by some of the mathematical statements in the paper. Much of the mathematical details are deferred to the original CGD paper. It is not really particularly reader-friendly to defer that to the CGD paper since they are seemingly crucial to the discussion here. Relative to the CGD paper some signs have been flipped and some definitions appear to be used in subtly different ways which makes for a very difficult read. I feel that far too much has been left as an exercise to the reader.
Concretely my concerns refer to the main discussion of the effect of the CGD as a regularizer:
The authors state:
“If some of the singular values of Dxy are very large, this amounts to approximately restricting the update to the orthogonal complement of the corresponding singular vectors”
I don’t see how this is the case. The terms Dxy/Dyx aren’t really introduced or defined anywhere in this work. Assuming is the transpose of the other (?) then the update direction is:
A + B where A=inv(S) grad_x and B = inv(S)Dxy grad_y (and S = I + Dxy Dyx). So we have a term which is being affected by the smallest singular values of S and a term which is the orthogonal projection of grad_y onto Dxy, alternatively the ridge-regression fit of grad_y on Dxy which would attenuate directions corresponding to the *small* singular values (as is well known from the theory of ridge regularizers). I feel like there is much more to say here than what is discussed in the paper in very vague terms.
Of course the effective rank of S, or the rate of decay of its singular values is crucially important. In practise I would assume the smaller SVs of Dxy to be difficult to estimate or the matrix to be rank deficient in which case they would simply be unity in the inverse whereas the directions corresponding to large singular values would be attenuated. So in this case it is the regularized orthogonal complement but its not clear (if the matrix is not full rank) that it is a meaningful direction (and again this is all highly dependent on the effective rank, too).
Further on it is mentioned: “For smoothly varying singular vectors, this can be though of as approximately constraining the trajectories to a manifold of robust play”.
First it is not at all clear to me what “smoothly varying singular vectors” are. Varying with respect to what? Secondly, the “manifold of robust play” has not been defined anywhere.
Finally, figure 3 is quite bizarre to me. None of the quantities have been rigorously defined and so it seems like the relative effect of each of the arrows and the manifold have been drawn arbitrarily in order to fit the story, rather than to actually illuminate the true behaviour in an intuitive manner.
B (defined above) has a very clear interpretation as a least-squares fit so I figure that any geometric interpretation of the CGD update direction could start from there.