A successful application of convolutional architectures is to increase the resolution of single low-resolution images -a image restoration task called super-resolution (SR). Naturally, SR is of value to resource constrained devices like mobile phones, electronic photograph frames and televisions to enhance image quality. However, SR demands perhaps the most extreme amounts of memory and compute operations of any mainstream vision task known today, preventing SR from being deployed to devices that require them. In this paper, we perform a early systematic study of system resource efficiency for SR, within the context of a variety of architectural and low-precision approaches originally developed for discriminative neural networks. We present a rich set of insights, representative SR architectures, and efficiency trade-offs; for example, the prioritization of ways to compress models to reach a specific memory and computation target and techniques to compact SR models so that they are suitable for DSPs and FPGAs. As a result of doing so, we manage to achieve better and comparable performance with previous models in the existing literature, highlighting the practicality of using existing efficiency techniques in SR tasks. Collectively, we believe these results provides the foundation for further research into the little explored area of resource efficiency for SR.

The paper proposes a detailed empirical evaluation of the trade-offs achieved by various convolutional neural networks on the super resolution problem. The paper provides an extensive evaluation of different architectural changes and the trade-off between savings in terms of memory and computational cost and performance, measured in terms of PSNR and SSIM.
This is an empirical paper, thus it does not provide technical contributions. I do think that the insights obtained from such an empirical evaluation could be of interest for practitioners and researchers working on the problem. My main concern is the method only evaluates the trade-offs between model efficiency (in terms of memory and/or computation) and performance measured using metrics that are known to not be well correlated with perceptual quality. Thus it is not obvious to me that the insights obtained in this work would translate to the other case.
It is well known that PSNR favors blurry solutions over perceptually more appealing solutions. This comes from the fact that there is no information in the low resolution image to produce the missing high resolution details. Filling up plausible details in a way that is different from the original image would lead to high PSNR. Models that treat the super resolution problem as a regression task using similarity in pixel space, tend to produce blurry solutions and require very large models to improve the score.
In recent years, many works have been studying the use of perceptual losses to mitigate this issue or simply treating the super resolution problem as conditional generative modeling.  For instance, models using L2 losses in a perceptually more relevant (or learned) feature spaces [A, B], or including GAN losses [C, D] (to list a few). To my knowledge, these models are the current state of the art in terms of perceptual quality. This has been evaluated empirically via perceptual tests [D].
This line of work needs to be cited. In my view, the paper needs to provide a detailed justification on why models using these losses are not considered. Would the conclusions drawn on this work transfer to that setting? Furthermore, it would be good to perform perceptual tests to perform this evaluation. It would be good to provide some canonical examples in the appendix.
The overall writing of the paper could be improved. Several sentences are difficult to read, due to typos or the construction of the sentences. The paper evaluates many architectural modifications proposed by other works. It would be good to add an appendix with a small description of what these are. This would make the paper self-contained an easier to read (I had too look up a few of them).
The authors mentioned that they first train models for scaling factor of x2 and then use them for training settings higher magnification. How is this exactly done? Please provide details.
I am curious of weather using some for of distillation techniques would be useful here.
Did you try scaling factors larger than x4? Scaling factors of x2 does not seem very relevant, as simpler methods can achieve already quite competitive results (such as simple interpolation methods)
The authors seem to be citing Zhang et al (2018) as a reference to attention mechanisms. To my knowledge the paper that proposed these mechanisms is [E].
The citation style is not used properly throughout the manuscript. As an example:
“… proposed in StrassenNets Tschannen et al (2017).” Should be “… proposed in StrassenNets (Tschannen et al, 2017).” Or “… proposed in StrassenNets proposed by Tschannen et al (2017).”
[A] Johnson, J. et al. "Perceptual losses for real-time style transfer and super-resolution." ECCV, 2016.
[B] Bruna, J. et al "Super-resolution with deep convolutional sufficient statistics." ICLR 2016.
[C] Ledig, C. et al. "Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network." CVPR. Vol. 2. No. 3. 2017.
[D] Sønderby, C. K., et al. "Amortised map inference for image super-resolution." arXiv preprint arXiv:1610.04490(2016).
[E] Bahdanau, D. et al "Neural machine translation by jointly learning to align and translate." arXiv (2014).


LARGE SCALE GRAPH LEARNING FROM SMOOTH SIGNALS
Graphs are a prevalent tool in data science, as they model the inherent structure of the data. Typically they are constructed either by connecting nearest samples, or by learning them from data, solving an optimization problem. While graph learning does achieve a better quality, it also comes with a higher computational cost. In particular, the current state-of-the-art model cost is O n 2 for n samples. In this paper, we show how to scale it, obtaining an approximation with leading cost of O (n log(n)), with quality that approaches the exact graph learning model. Our algorithm uses known approximate nearest neighbor techniques to reduce the number of variables, and automatically selects the correct parameters of the model, requiring a single intuitive input: the desired edge density.

Learning graphs from data fine tunes standard similarity graph constructions such as k-nearest neighbor graphs.
There has been a line of research works that focuses on learning graphs and that shows that this results in superior
results in various machine learning tasks.  The current state-of-the-art method is the method proposed by Kalofolias,
which however is slow.   The authors suggest a method to avoid searching for the parameters that achieve a desired
level of sparsity by providing closed a formula. The parameter that determines the sparsity is theta, see proposition 1 on page 4. This was originally shown by Kalofolias. To achieve their goal, the authors first consider the degree of any given node by looking at equation (8), page 4. They prove theorem 1, that is intuitive and  provides the form of the optimal
weights that connect this node to the rest of the nodes in the graph.  The proof is based on applying the KKT conditions on
the objective (8), with the single constraint that there are no negative weights.  Finally, since we care about the
sparsity of the graph as a whole, the authors use the average of the parameter theta over all nodes. The authors perform
experiments on real-world graphs, and show basic properties of their method, as well as the main source of mistakes ,i.e., disconnected nodes, figure 5.
Essentially, this paper starts from the work of Kalofolias  and improves it significantly. This by itself is
a neat contribution, but the authors could improve their paper by showing a more complete view  of graph
learning methods, with respect to the quality of the produced graphs and the scalability. I find this aspect of the paper narrowing its contribution, hence my evaluation. Some remarks follow.
- A different family of graph learning methods is based on the objective ||LX||_F^2 or equivalently tr(X^TLLX).
For this objective, Daitch et al. proved certain neat properties, such as the existence of a sparse optimal graph.
This allows Daitch et al. to solve the primal dual significantly faster than O(n^2) since by their theorem,
O(nd) edges are required where d is the dimension of the data points. When d is large, a random projection can be applied.
The paper should compare with this family of methods that are more scalable both with respect to the accuracy,
and to the runtimes.
- While the proposed method scales significantly better than Kalofolias, the datasets used are small.
- Using LSH for k-nn graphs results in a  scalable, practical way to construct similarity graphs. The authors should cite
the following related work, and compare with such methods.
“Efficient K-Nearest Neighbor Graph Construction for Generic Similarity Measures“ by Dong, Charikar, Li.
- An interesting experiment would be to inject outliers in the dataset, or use some dataset with outliers.
Would this affect the tightness of the interval in equation (17)?


REPRESENTATION DEGENERATION PROBLEM IN TRAINING NATURAL LANGUAGE GENERATION MOD- ELS
We study an interesting problem in training neural network-based models for natural language generation tasks, which we call the representation degeneration problem. We observe that when training a model for natural language generation tasks through likelihood maximization with the weight tying trick, especially with big training datasets, most of the learnt word embeddings tend to degenerate and be distributed into a narrow cone, which largely limits the representation power of word embeddings. We analyze the conditions and causes of this problem and propose a novel regularization method to address it. Experiments on language modeling and machine translation show that our method can largely mitigate the representation degeneration problem and achieve better performance than baseline algorithms.

The paper presents and discusses a new phenomenon that infrequent words tend to learn degenerate embeddings. A cosine regularization term is proposed to address this issue.
Pros
1. The degenerate embedding problem is novel and interesting.
2. Some positive empirical results.
Cons and questions
1. The theory in Section 4 suggests that the degeneration problem originates from underfitting; i.e., there's not enough data to fit the embeddings of the infrequent words, when epsilon is small. However, the solution in Section 5 is based on a regularization term. This seems contradictory to me because adding regularization to an underfit model would not make it better. In other words, if there's not enough data to fit the word embeddings, one should feed more data. It seems that a cosine regularization term could only make the embeddings different from each other, but not better.
2. Since this is an underfitting problem (as described in Section 4), I'm wondering what would happen on larger datasets. The claims in the paper could be better substantiated if there are results on larger datasets like WT103 for LM and en-fr for MT. Intuitively, by increasing the amount of total data, the same word gets more data to fit, and thus epsilon gets large enough so that degeneration might not happen.
3. "Discussion on whether the condition happens in real practice" below Theorem 2 seems not correct to me. Even when layer normalization is employed and bias is not zero, the convex hull can still contain the origin as long as the length of the bias vector is less than 1. In fact, this condition seems fairly strong, and surely it will not hold "almost for sure in practice".
4. The cosine regularization term seems expensive, especially when the vocab size is large. Any results in terms of computational costs? Did you employ tricks to speed it up?
5. What would happen if we only apply the cosine term on infrequent words? An ablation study might make it clear why it improves performance.
UPDATE:
I think the rebuttal addresses some of my concerns. I am especially glad to see improvement on en-fr, too. Thus I raised my score from 5 to 7.


The vast majority of neural models in Natural Language Processing adopt a form of structureless distributed representations. While these models are powerful at making predictions, the representational form is rather crude and does not provide insights into linguistic structures. In this paper we introduce novel language models with representations informed by the framework of Holographic Reduced Representation (HRR). This allows us to inject structures directly into our wordlevel and chunk-level representations. Our analyses show that by using HRR as a structured compositional representation, our models are able to discover crude linguistic roles, which roughly resembles a classic division between syntax and semantics.

Summary:
========
Theis paper proposes a method for learning decomposable representations in the context of a language modeling task. Using holographic reduced representations (HRR), a word embedding is composed of a role and a filler. The embedding is then fed to an LSTM language model. There is also an extension to chunk-level representations. Experimentally, the model achieves perplexity comparable to a (weak) baseline LSTM model. The analysis of the learned representations shows a separation into syntactic and semantic roles.
The paper targets an important problem, that of learning decomposable representations. As far as I know, it introduces a novel perspective using HRR and does so in the context of language modeling, which is a core NLP task. The analysis of the learned representations is quite interesting. I do have some concerns with regards to the quality of the language model, the clarity of some of the model description, and the validity of using HRR in this scenario. Please see detailed comments below.
Comments:
=========
1. Section 2 refers to Plate (1995) for the conditions when the approximate decoding via correlation holds. I think it's important to mention these conditions and discuss whether they apply to the language modeling case. In particular, Plate mentions that the elements of each vector need to be iid with mean zero and variance 1/n (where n is the length of the vector). Is this true for the present case? Typically, word embeddings and LSTM states are do not exhibit this distribution. Are there other conditions that are (not) met?
2. Learning separate bases for different role-filler bindings is said to encourage the model to learn a decomposition of word representation. On the other hand, if I understand correctly, this means that word embeddings are not shared between roles, because s^w_i is also a role-specific vector (not just a word-specific vector). Is that a cause of concern?
3. It's not clear to me where in the overall model the next word is predicted. Figure 1b has an LSTM that predicts filler embeddings. Does this replace predicting the next word in a vanilla LSTM? Equation 5 still computes a word score. Is this used to compute the probability of the next word as in equation 2?
4. Comparison to other methods for composing words. Since much of the paper is concerned with composing words, it seem natural to compare the methods (and maybe some of the results) to methods for composing words. Some examples include [2] and the line of work on recursive neural networks by Socher et al., but there are many others.
5. Perplexity results:
- The baseline results (100.5 ppl on PTB) are very weak for an LSTM. There are multiple papers showing that a simple LSTM can do much better. The heavily tuned LSTM of [1] gets 59.6 but even less tuned LSTMs go under 80 or 80 ppl. See some results in [1]. This raises a concern that the improvements from the HRR model may not be significant. Would they hold in a more competitive model?
- Can you speculate or analyze in more detail why the chunk-level model doesn't perform well, and why adding more fillers doesn't help in this case?
6. Motivation:
- The introduction claims that the dominant encoder-decoder paradigm learns "transformations from many smaller comprising units to one complex emedding, and vice versa". This claim should be qualified by the use of attention, where there is not a single complex embedding, rather a distribution over multiple embeddings.
- Introduction, first paragraph, claims that "such crude way of representing the structure is unsatisfactory, due to a lack of transparency, interpretability or transferability" - what do you mean by these concepts and how exactly is the current approach limited with respect to them? Giving a bit more details about this point here or elsewhere in the paper would help motivate the work.
7. Section 3.3 was not so clear to me:
- In step 1, what are these r_i^{chunk}? Should we assume that all chunks have the same role embeddings, despite them potentially being syntactically different? How do you determine where to split output vectors from the RNN to two parts? What is the motivation for doing this?
- In prediction, how do you predict the next chunk embedding? Is there a different loss function for this?
- Please provide more details on decoding, such as the mentioned annealing and regularization.
- Finally, the reliance on a chunker is quite limiting. These may not be always available or of high quality.
8. The analysis in section 4.3 is very interesting and compelling. Figure 2 makes a good point. I would have liked to see more analysis along these lines. For example, more discussion of the word analogy results, including categories where HRR does not do better than the baseline. Also consider other analogy datasets that capture different aspects.
9. While I agree that automatic evaluation at chunk-level is challenging, I think more can be done. For instance, annotations in PTB can be used to automatically assign roles such as those in table 4, or others (there are plenty of annotations on PTB), and then to evaluate clustering along different annotations at a larger scale.
10. The introduction mentions a subset of the one billion word LM dataset (why a subset?), but then the rest of the papers evaluates only on PTB. Is this additional dataset used or not?
11. Introduction, first paragraph, last sentence: "much previous work" - please cite such relevant work on inducing disentangled representations.
12. Please improve the visibility of Figure 1. Some symbols are hard to see when printed.
13. More details on the regularization on basis embeddings (page 4) would be useful.
14. Section 3.3 says that each unique word token is assigned a vectorial parameter. Should this be word type?
15. Why not initialize the hidden state with the last state from the last batch? I understand that this is done to assure that the chunk-level models only consider intra-sentential information, but why is this desired?
16. Have you considered using more than two roles? I wonder how figure 2 would look in this case.
Writing, grammar, etc.:
======================
- End of section 1: Our papers -> Our paper
- Section 2: such approach -> such an approach; HRR use -> HRR uses; three operations -> three operations*:*
- Section 3.1: "the next token w_t" - should this be w_{t+1)?
- Section 3.2, decoding: remain -> remains
- Section 3.3: work token -> word token
- Section 4.1: word analogy task -> a word analogy task; number basis -> numbers of basis
- Section 4.2: that the increasing -> that increasing
- Section 4.3: no space before comma (first paragraph); on word analogy task -> on a word analogy task; belong -> belongs
- Section 4.4: performed similar -> performed a similar; luster -> cluster
- Section 5: these work -> these works/papers/studies; share common goal -> share a common goal; we makes -> we make; has been -> have been
References
==========
[1] Melis et al., On the State of the Art of Evaluation in Neural Language Models
[2] Mitchell and Lapata, Vector-based Models of Semantic Composition


DEEP ANOMALY DETECTION WITH OUTLIER EXPOSURE
It is important to detect anomalous inputs when deploying machine learning systems. The use of larger and more complex inputs in deep learning magnifies the difficulty of distinguishing between anomalous and in-distribution examples. At the same time, diverse image and text data are available in enormous quantities. We propose leveraging these data to improve deep anomaly detection by training anomaly detectors against an auxiliary dataset of outliers, an approach we call Outlier Exposure (OE). This enables anomaly detectors to generalize and detect unseen anomalies. In extensive experiments on natural language processing and small-and large-scale vision tasks, we find that Outlier Exposure significantly improves detection performance. We also observe that cutting-edge generative models trained on CIFAR-10 may assign higher likelihoods to SVHN images than to CIFAR-10 images; we use OE to mitigate this issue. We also analyze the flexibility and robustness of Outlier Exposure, and identify characteristics of the auxiliary dataset that improve performance.

I have read authors' reply.  In response to authors' comprehensive reply and feedback. I upgrade my score to 6. As authors mentioned, the extension to density estimators is an original novelty of this paper, but I still have some concern that OE loss for classification is basically the same as [2]. I think it is better to clarify this in the draft.
Summary===
This paper proposes a new fine-tuning method for improving the performance of existing anomaly detectors. The main idea is additionally optimizing the “Outlier Exposure (OE)” loss on outlier dataset. Specifically, for softmax classifier, the authors set the OE loss to the KL divergence loss between posterior distribution and uniform distribution. For density estimator, they set the OE loss to a margin ranking loss. The proposed method improves the detection performance of baseline methods on various vision and NLP datasets. While the research topic of this paper is interesting, I recommend rejections because I have concerns about novelty and the experimental results.
Detailed comments ===
1. OE loss for softmax classifier
For softmax classifier, the OE loss forces the posterior distribution to become uniform distribution on outlier dataset. I think this loss function is very similar to a confidence loss (equation 2) proposed in [2]: Lee et al., 2017 [2] also proposed the loss function minimizing the KL divergence between posterior distribution and uniform distribution on out-of-distribution, and evaluated the effects of it on "unseen" out-of-distribution (see Table 1 of [2]). Could the authors clarify the difference with the confidence loss in [2], and compare the performance with it? Without that, I feel that the novelty of this paper is not significant.
2. More comparison with baselines
The authors said that they didn’t compare the performance with simple inference methods like ODIN [3] since ODIN tunes the hyper-parameters using data from (tested) out-of-distribution. However, I think that the authors can compare the performance with ODIN by tuning the hyper-parameters of it on outlier dataset which is used for training OE loss. Could the authors provide more experimental results by comparing the performance with ODIN?
3. Related work
I would appreciate if the authors can survey and compare more baselines such as [4] and [5].
[1] Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and out-of-distribution examples in neural networks. International Conference on Learning Representations, 2017.
[2] Kimin Lee, Honglak Lee, Kibok Lee, and Jinwoo Shin. Training confidence-calibrated classifiers for detecting out-of-distribution samples. International Conference on Learning Representations, 2018.
[3] Shiyu Liang, Yixuan Li, and R. Srikant. Enhancing the reliability of out-of-distribution image detection in neural networks. International Conference on Learning Representations, 2018.
[4] Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attacks. In NIPS, 2018.
[5] Apoorv Vyas, Nataraj Jammalamadaka, Xia Zhu, Dipankar Das, Bharat Kaul, and Theodore L. Willke. Out-of-Distribution Detection Using an Ensemble of Self Supervised Leave-out Classifiers, In ECCV, 2018.


We present a deep generative model, named Monge-Ampère flow, which builds on continuous-time gradient flow arising from the Monge-Ampère equation in optimal transport theory. The generative map from the latent space to the data space follows a dynamical system, where a learnable potential function guides a compressible fluid to flow towards the target density distribution. Training of the model amounts to solving an optimal control problem. The Monge-Ampère flow has tractable likelihoods and supports efficient sampling and inference. One can easily impose symmetry constraints in the generative model by designing suitable scalar potential functions. We apply the approach to unsupervised density estimation of the MNIST dataset and variational calculation of the two-dimensional Ising model at the critical point. This approach brings insights and techniques from Monge-Ampère equation, optimal transport, and fluid dynamics into reversible flow-based generative models.

This paper proposes a novel parameter-efficient generative modeling approach that is based on the Monge-Ampere equation. In the proposal, a feed-forward neural network is trained as an ODE integrator which solves (2) and (3) for a fixed time interval $[0,T]$, so that the distribution $p(x,t)$ at time 0 is a simple base distribution such as a Gaussian, and that at time $T$ mimics the target distribution.
[pros]
- The proposal provides a parameter-efficient approach to generative modeling, via parameter sharing in the depth direction.
- I think that the idea itself is quite interesting and that it is worth pursuing this direction further.
[cons]
- The Applications section is not convincing enough to demonstrate usefulness of the proposal as an approach to generative modeling.
- How the gradient-based learning in the proposal behaves is not discussed in this paper.
[quality]
How the gradient-based learning in the proposal behaves is not discussed. I understand that the non-convex nature of the loss function poses problems already in the conventional back-propagation learning of a multilayer neural network. On the other hand, in the proposal, the loss function (e.g., (4)) is further indirectly parameterized via $\varphi$. It would be nice if the parameterization of the loss in terms of $\varphi$ is regular in some sence.
[clarity]
Description of this paper is basically clear. In the author-date citation style employed in this paper, both the author names and publication year are enclosed in parentheses, with exception being the author names incorporated in the text. This paper does not follow the above standard convention for citation and thus poses strong resistance to the reader. For example, in the first line of the Introduction section, "Goodfellow et al. (2016)" should read "(Goodfellow et al., 2016)".
[originality]
The idea of considering the Monge-Ampere equation in its linearized form to formulate generative modeling seems original.
[significance]
In the experiment described in Section 4.1, it is not clear at all from the description here whether the learned system is capable of successfully generating MNIST-like fake images, which would question the significance of the proposal as a framework for generative modeling. It is well known that the KL divergence $D(P\|Q)$ tends to put more penalty when $P$ is large and $Q$ is small than the opposite. One can then expect in this experiment that it tolerates the model, appearing as $Q$ in $D(P\|Q)$, to put weights on regions where the data are scarce, which might result in generation of low-quality fake images. It would be nice if the authors provide figures showing samples generated via mapping of Gaussian samples with the learned system.
Also, in the experiment described in Section 4.2, I do not see its significance. It is nice to observe in Figure 4 that the loss function approaches the true free energy as well as that the snapshots generated by the model seem more or less realistic. My main concern however is regarding what the potential utilities of the proposal are in elucidating statistical-physical properties of a system. For example, it would be nice if the proposal could estimate the phase-transition point more easily and/or more accurately compared with alternative conventional approaches, but there is no such comparison presented in this paper, making the significance of this paper obscure.
Minor points:
The reference entitled "A proposal on machine learning via dynamical systems" would be better cited not as "E (2017)" but rather as "Weinan (2017)".
Page 6, line 10: the likelihoods of these sample(s)
----Updated after author feedback----
Upon reading the author feedback, I have downgraded my rating from 7 to 6, because the author feedback is not satisfactory to me in some respects. In my initial review, my comment on the experiment on MNIST is not on correlation between the maximum likelihood estimation and visual quality of generated images, on which the author feedback was based, but regarding the well-known property of the KL divergence due to its asymmetry between the two arguments. Also, regarding the experiment on the Ising model, the proposal in this paper provides an approximate sampler, whereas for example the MCMC provides an exact sampler with exponential slowing down in mixing under multimodal distributions. In statistical physics, one is interested in studying physical properties of the system, such as phase transition, with samples obtained from a sampler. In this regard, important questions are how good the samples are and how efficiently they are generated. As for the quality, it would have been nice if results of evaluated free energy as a function of inverse temperature (that is K_ij in the case here) were provided. The author feedback was, on the other hand, mainly explanation of general variational approach, of which I am aware.
I still think that this paper contains interesting contributions, and accordingly have put my rating above the threshold.


STCN: STOCHASTIC TEMPORAL CONVOLUTIONAL NETWORKS
Convolutional architectures have recently been shown to be competitive on many sequence modelling tasks when compared to the de-facto standard of recurrent neural networks (RNNs), while providing computational and modeling advantages due to inherent parallelism. However, currently there remains a performance gap to more expressive stochastic RNN variants, especially those with several layers of dependent random variables. In this work, we propose stochastic temporal convolutional networks (STCNs), a novel architecture that combines the computational advantages of temporal convolutional networks (TCN) with the representational power and robustness of stochastic latent spaces. In particular, we propose a hierarchy of stochastic latent variables that captures temporal dependencies at different time-scales. The architecture is modular and flexible due to decoupling of deterministic and stochastic layers. We show that the proposed architecture achieves state of the art log-likelihoods across several tasks. Finally, the model is capable of predicting high-quality synthetic samples over a long-range temporal horizon in modeling of handwritten text.

This paper presents a generative sequence model based on the dilated CNN
popularized in models such as WaveNet. Inference is done via a hierarchical
variational approach based on the Variational Autoencoder (VAE). While VAE
approach has previously been applied to sequence modeling (I believe the
earliest being the VRNN of Chung et al (2015)), the innovation where is the
integration of a causal, dilated CNN in place of the more typical recurrent
neural network.
The potential advantages of the use of the CNN in place of
RNN is (1) faster training (through exploitation of parallel computing across
time-steps), and (2) potentially (arguably) better model performance. This
second point is argued from the empirical results shown in the
literature. The disadvantage of the CNN approach presented here is that
these models still need to generate one sample at a time and since they are
typically much deeper than the RNNs, sample generation can be quite a bit
slower.
Novelty / Impact: This paper takes an existing model architecture (the
causal, dilated CNN) and applies it in the context of a variational
approach to sequence modeling. It's not clear to me that there are any
significant challenges that the authors overcame in reaching the proposed
method. That said, it certainly useful for the community to know how the
model performs.
Writing: Overall the writing is fairly good though I felt that the model
description could be made more clear by some streamlining -- with a single
pass through the generative model, inference model and learning.
Experiments: The experiments demonstrate some evidence of the superiority
of this model structure over existing causal, RNN-based models. One point
that can be drawn from the results is that a dense architecture that uses multiple levels of the
latent variable hierarchy directly to compute the data likelihood is
quite effective. This observation doesn't really bear on the central message
of the paper regarding the use of causal, dilated CNNs.
The evidence lower-bound of the STCN-dense model on MNIST is so good (low)
that it is rather suspicious. There are many ways to get a deceptively good
result in this task, and I wonder if all due care what taken. In
particular, was the binarization of the MNIST training samples fixed in
advance (as is standard) or were they re-binarized throughout training?
Detailed comments:
- The authors state "In contrast to related architectures (e.g. (Gulrajani et
al, 2016; Sonderby et al. 2016)), the latent variables at the upper layers
capture information at long-range time scales" I believe that this is
incorrect in that the model proposed in at least Gulrajani et al also
- It also seems that there is an error in Figure 1 (left). I don't think
there should be an arrow between z^{2}_{t,q} and z^{1}_{t,p}. The presence
of this link implies that the prior at time t would depend -- through
higher layers -- on the observation at t. This would no longer be a prior
at that point. By extension you would also have a chain of dependencies
from future observations to past observations. It seems like this issue is
isolated to this figure as the equations and the model descriptions are
consistent with an interpretation of the model without this arrow (and
including an arrow between z^{2}_{t,p} and z^{1}_{t,p}.
- The term "kla" appears in table 1, but it seems that it is otherwise not
defined. I think this is the same term and meaning that appears in Goyal et
al. (2017), but it should obviously be defined here.


Many machine learning problems involve iteratively and alternately optimizing different task objectives with respect to different sets of parameters. Appropriately scheduling the optimization of a task objective or a set of parameters is usually crucial to the quality of convergence. In this paper, we present AutoLoss, a meta-learning framework that automatically learns and determines the optimization schedule. AutoLoss provides a generic way to represent and learn the discrete optimization schedule from metadata, allows for a dynamic and data-driven schedule in ML problems that involve alternating updates of different parameters or from different loss objectives. We apply AutoLoss on four ML tasks: d-ary quadratic regression, classification using a multi-layer perceptron (MLP), image generation using GANs, and multi-task neural machine translation (NMT). We show that the AutoLoss controller is able to capture the distribution of better optimization schedules that result in higher quality of convergence on all four tasks. The trained AutoLoss controller is generalizable -it can guide and improve the learning of a new task model with different specifications, or on different datasets.

Summary: This paper proposes a meta-learning solution for problems involving optimizing multiple loss values. They use a simple (small mlp), discrete, stochastic controller to control applications of updates among a finite number of different update procedures. This controller is a function of heuristic features derived from the optimization problem, and is optimized using policy gradient either exactly in toy settings or in a online / truncated manor on larger problems. They present results on 4 settings: quadratic regression, MLP classification, GAN, and multi-task MNT. They show promising performance on a number of tasks as well as show the controllers ability to generalize to novel tasks.
This is an interesting method and tackles a impactful problem. The setup and formulation (using PG to meta-optimize a hyper parameter controller) is not extremely novel (there have been similar work learning hyper parameter controllers), but the structure, the problem domain, and applications are. The experimental results are through, and provide compelling proof that this method works as well as exploration as to why the method works (analyzing output softmax). Additionally the "transfer to different models" experiment is compelling.
Comments vaguely in order of importance:
1. I am a little surprised that this training strategy works. In the online setting for larger scale problems, your gradients are highly correlated and highly biased. As far as I can tell, you are performing something akin to truncated back back prop through time with policy gradients. The biased introduced via this truncation has been studied in great depth in [3] and shown to be harmful. As of now, the greedy nature of the algorithm is hidden across a number of sections (not introduced when presenting the main algorithm). Some comment as to this bias -- or even suggesting that it might exist would be useful. As of now, it is implied that the gradient estimator is unbiased.
2. Second, even ignoring this bias, the resulting gradients are heavily correlated. Algorithm 1 shows no sign of performing batched updates on \phi or anything to remove these corrections. Despite these concerns, your results seem solid. Nevertheless, further understanding as to this would be useful.
3. The structure of the meta-training loop was unclear to me. Algorithm 1 states S=1 for all tasks while the body -- the overhead section -- you suggest multiple trainings are required ( S>1?).
4. If the appendix is correct and learning is done entirely online, I believe the initialization of the meta-parameters would matter greatly -- if the default task performed poorly with a uniform distribution for sampling losses, performance would be horrible. This seems like a limitation of the method if this is the case.
5. Clarity: The first half of this paper was easy to follow and clear. The experimental section had a couple of areas that left me confused. In particular:
5.1/Figure 1: I think there is an overloaded use of lambda? My understanding as written that lambda is both used in the grid search (table 1) to find the best loss l_1 and then used a second location, as a modification of l_2 and completely separate from the grid search?
6. Validation data / test sets: Throughout this work, it is unclear what / how validation is performed. It seems you performing controller optimization (optimizing phi), on the validation set loss, while also reporting scores on this validation set. This should most likely instead be a 3rd dataset. You have 3 datasets worth of data for the regression task (it is still unclear, however, what is being used for evaluation), but it doesn't look like this is addressed in the larger scale experiments at all. Given the low meta-parameter count of the I don't think this represents a huge risk, and baselines also suffer from this issue (hyper parameter search on validation set) so I expect results to be similar.
7. Page 4: "When ever applicable, the final reward $$ is clipped to a given range to avoid exploding or vanishing gradients". It is unclear to me how this will avoid these. In particular, the "exploding" will come from the \nabla log p term, not from the reward (unless you have reason to believe the rewards will grow exponentially). Additionally, it is unclear how you will have vanishing rewards given the structure of the learned controller. This clipping will also introduce bias, this is not discussed, and will probably lower variance. This is a trade off made in a number of RL papers so it seems reasonable, but not for this reason.
8. "Beyond fixed schedules, automatically adjusting the training of G and D remains untacked" -- this is not 100% true. While not a published paper, some early gan work [2] does contains a dynamic schedule but you are correct that this family of methods are not commonplace in modern gan research.
9. Related work: While not exactly the same setting, I think [1] is worth looking at. This is quite similar causing me pause at this comment: "first framework that tries to learn the optimization schedule in a data-driven way". Like this work, they also lean a controller over hyper-parameters (in there case learning rate), with RL, using hand designed features.
10. There seem to be a fair number of heuristic choices throughout. Why is IS squared in the reward for GAN training for example? Why is the scaling term required on all rewards? Having some guiding idea or theory for these choices or rational would be appreciated.
11. Why is PPO introduced? In algorithm 1, it is unclear how PPO would fit into this? More details or an alternative algorithm in the appendix would be useful. Why wasn't PPO used on all larger scale models? Does the training / performance of the meta-optimizer (policy gradient  vs ppo) matter? I would expect it would. This detail is not discussed in this paper, and some details -- such as the learning rate for the meta-optimizer I was unable to find.
12. "It is worth noting that all GAN K:1 baselines perform worse than the rest and are skipped in Figure 2, echoing statements (Arjovsky, Gulrajani, Deng) that more updates of G than D might be preferable in GAN training." I disagree with this statement. The WGAN framework is built upon a loss that can be optimized, and should be optimized, until convergence (the discriminator loss is non-saturating) -- not the reverse (more G steps than D steps) as suggested here. Arjovsky does discuss issues with training D to convergence, but I don't believe there is any exploration into multiple G steps per D step as a solution.
13. Reproducibility seems like it would be hard. There are a few parameters (meta-learning rates, meta-optimizers) that I could not find for example and there is a lot of complexity.
14: Claims in paper seem a little bold / overstating. The inception gain is marginal to previous methods, and trains slower than other baselines. This is also true of MNT section -- there, the best baseline model is not even given equal training time! There are highly positive points here, such as requiring less hyperparameter search / model evaluations to find performant models.
15. Figure 4a. Consider reformatting data (maybe histogram of differences? Or scatter plot). Current representation is difficult to read / parse.
Typos:
page 2, "objective term. on GANs, the AutoLoss: Capital o is needed.
Page 3: Parameter Learning heading the period is not bolded.
[1] Learning step size controllers for robust neural network training. Christian Daniel et. al.
[2]http://torch.ch/blog/2015/11/13/gan.html
[3] Understanding Short-Horizon Bias in Stochastic Meta-Optimization, Wu et.al.
Given the positives, and in-spite of the negatives, I would recommend to accept this paper as it discusses an interesting and novel approach when controlling multiple loss values.


GENERATIVE PREDECESSOR MODELS FOR SAMPLE- EFFICIENT IMITATION LEARNING
We propose Generative Predecessor Models for Imitation Learning (GPRIL), a novel imitation learning algorithm that matches the state-action distribution to the distribution observed in expert demonstrations, using generative models to reason probabilistically about alternative histories of demonstrated states. We show that this approach allows an agent to learn robust policies using only a small number of expert demonstrations and self-supervised interactions with the environment. We derive this approach from first principles and compare it empirically to a stateof-the-art imitation learning method, showing that it outperforms or matches its performance on two simulated robot manipulation tasks and demonstrate significantly higher sample efficiency by applying the algorithm on a real robot.

The submission builds up on recent advances in neural density estimation to develop a new algorithm for imitation learning based on a probabilistic model for predecessor state dynamics. In particular, the method trains masked autoregressive flows as a probabilistic model for state action pairs conditioned on future states. This model is used to estimate the gradient of the stationary distribution of a policies visited states. Finally, the proposed objective uses this estimate and the gradient of the log likelihood of expert actions under the policy to maximise the similarity of the expert’s and agent’s stationary state-action distributions.
The proposed method outperforms existing imitation learning approach (GAIL & BC) on 2 simulation-based manipulation tasks. It performs particularly well in terms of sample efficiency.
The magnitude of difference between the sample efficiencies of GAIL and the proposed approach seems quite surprising and it would be beneficial if the authors could explicitly state if the measured number of samples include the ones used for training of the probabilistic model as well as the policy (apologies if I have missed a section fulfilling this purpose).
While the improvements on the presented experiments are clear, the experimental section represents a small shortcoming of the submitted paper. The 2 experiments (clip and peg insertion) are quite similar in type and to not take into account other common domains e.g. locomotion tasks from the original GAIL paper. Furthermore, an additional comparison to SAIL would be recommended since the approaches are closely related as the authors acknowledge. The provided comparison with different types of available expert data is quite interesting and could possibly be extended to test other state-of-the-art methods (action-free versions of GAIL, AIRL,etc.).
Nonetheless, the paper overall presents a strong submission based on novelty & relevance of the proposed method and is recommended for publication.
Minor issues:
- Related work: improve transitions between the section about trajectory tracking and BC.
- Ablation studies with less flexible probabilistic models would strengthen the experiment section further.
- Add derivation from Eq. 3 to 4 and 5 to appendix to render the paper more self-contained and easier to access.
- A release of the code base would further strengthen the contributions of the submission.
General recommendation:
- The authors are encouraged to further investigate off-policy corrections for improved convergence.


Recent years have seen remarkable progress of text generation in different contexts, including the most common setting of generating text from scratch, the increasingly popular paradigm of retrieval and editing, and others. Text infilling, which fills missing text portions of a sentence or paragraph, is also of numerous use in real life. Previous work has focused on restricted settings, by either assuming single word per missing portion, or limiting to single missing portion to the end of text. This paper studies the general task of text infilling, where the input text can have an arbitrary number of portions to be filled, each of which may require an arbitrary unknown number of tokens. We develop a self-attention model with segment-aware position encoding for precise global context modeling. We further create a variety of supervised data by masking out text in different domains with varying missing ratios and mask strategies. Extensive experiments show the proposed model performs significantly better than other methods, and generates meaningful text patches.

Summary
This paper proposes an approach to fill in gaps in sentences. While usually generation assumes a provided left context, this paper generalizes generation to any missing gap of any number of words.
The method is based on Transformer which does attention on all the available context and generates in a left to right manner.
The authors validate their approach on rather small scale datasets. I am unclear about several details of their empirical validation.
Relevance: this work is on text generation, which is very relevant to this conference.
Clarity: the description of the method is very clear, less so the description of the empirical validation.
Novelty: The method per se is not novel, but the combination of method and application is.
Empirical validation: The empirical validation is not very clear and potentially weak.
First, I did not understand the baselines the authors considered: seq2seq and GAN. There are LOTS of variants of these methods and the references cited by the authors do not immediately apply to filling-in tasks. The authors should explain in detail what these baselines are and how they work.
Second, MaskGAN ICLR 2018 was proposed to solve the filling task and code is publicly available by the authors of that paper. A direct comparison to MaskGAN seems a must.
Third, there are lots of details that are unclear, such as : how are segments defined in practice? the ranking metric is not clearly defined (pair-wise ranking or ranking of N possible completion?). Could the authors be a little bit more formal?
Speaking of metrics, why don't the authors considered precision at K for the evaluation of small gaps?
Fourth, the datasets considered, in particular the Grimm's dataset, is very small. There are only 16K training sentences and the vocabulary size is only 7K. How big is the model? How comes it does not overfit to such a small dataset? Did the authors measure overlap between training and test sets?
More general comments
The beauty of the proposed approach is its simplicity. However, the proposed model feels not satisfactory as generation proceeds left to right, while the rightmost and the leftmost missing word in the gap should be treated as equal citizens.
Minor note: positional embeddings are useful also with convolutional models.


The tremendous memory and computational complexity of Convolutional Neural Networks (CNNs) prevents the inference deployment on resource-constrained systems. As a result, recent research focused on CNN optimization techniques, in particular quantization, which allows weights and activations of layers to be represented with just a few bits while achieving impressive prediction performance. However, aggressive quantization techniques still fail to achieve full-precision prediction performance on state-of-the-art CNN architectures on large-scale classification tasks. In this work we propose a method for weight and activation quantization that is scalable in terms of quantization levels (n-ary representations) and easy to compute while maintaining the performance close to full-precision CNNs. Our weight quantization scheme is based on trainable scaling factors and a nested-means clustering strategy which is robust to weight updates and therefore exhibits good convergence properties. The flexibility of nested-means clustering enables exploration of various n-ary weight representations with the potential of high parameter compression. For activations, we propose a linear quantization strategy that takes the statistical properties of batch normalization into account. We demonstrate the effectiveness of our approach using state-of-the-art models on ImageNet.

Summary: This paper proposes a technique for quantizing the weights and activations of a CNN. The main contribution is in replacing the heuristic to find good quantization intervals of (Zhu et al, 2016) with a different heuristic based on a hierarchical clustering algorithm, and empirically validating its effectiveness.
Strenghts:
- The proposed nested-means heuristic is simple and makes sense intuitively.
- The experiments on two modern architectures seem solid and demonstrate good empirical performance.
Weaknesses:
- The main weakness is the limited novelty of this paper. The proposed setup is almost identical to the one in (Zhu et al, 2016), except for the replacement of the heuristic to find quantization intervals with another one. While the experiments demonstrate the empirical effectiveness of the method as a whole, what is missing is a direct, controlled comparison between the original heuristic and the proposed one. Now it is hard to tell whether the accuracy increases are obtained through the proposed adaptation or because of other factors such as a better implementation or longer training.
- In section 4, it is not made clear whether the activations are quantized according to the same scheme as the weights (apart from the issue of selecting a good clipping interval, which is addressed).
- The paper is a bit short on references, considering the many recent works on quantized neural networks.
Minor comments and questions:
- The wording is sometimes imprecise, making some arguments hard to follow. Two examples:
-- "Lowering the learning rate for re-training can diminish heavy changes in the weight distribution, at the cost of longer time to converge and the risk to get stuck at plateau regions, which is especially critical for trainable scaling factors"
-- "This approach is beneficial because it defines cluster thresholds which are influenced by large weights that were shown to play a more important role than smaller weights (Han et al., 2015b)"
- The title says "for compression and inference acceleration", so it would be nice if the paper reports some compression and timing metrics in the experiments section.
- The notation in section 3.1 overly complicated, could probably be simplified a bit for readability.
- Section 3.3: "However, having an additional hyperparameter t_i for each scaling factor alpha_i renders the mandatory hyperparameter tuning infeasible." -> From section 4.2 in (Zhu et al, 2016), I believe the constant factor t is shared across all layers, making it only a single hyperparameter.
- Last paragraph of section 4: "(Cai et al., 2017) experimentally showed that the pre-activation distribution after batch normalization are all close to a Gaussian with zero mean and unit variance. Therefore, we propose to select a fixed clipping parameter gamma.". -> But what about the activations *before* the batchnorm layer where the assumption of zero mean and unit variance does not hold?


We revisit the bias-variance tradeoff for neural networks in light of modern empirical findings. The traditional bias-variance tradeoff in machine learning suggests that as model complexity grows, variance increases. Classical bounds in statistical learning theory point to the number of parameters in a model as a measure of model complexity, which means the tradeoff would indicate that variance increases with the size of neural networks. However, we empirically find that variance due to training set sampling is roughly constant (with both width and depth) in practice. Variance caused by the non-convexity of the loss landscape is different. We find that it decreases with width and increases with depth, in our setting. We provide theoretical analysis, in a simplified setting inspired by linear models, that is consistent with our empirical findings for width. We view bias-variance as a useful lens to study generalization through and encourage further theoretical explanation from this perspective.

This paper suggests to rethink about the bias-variance tradeoff from statistical machine learning in the context of neural networks. Based on some empirical observations, the main claims in this work are that (1) it is not always the case that the variance will increase when we use bigger neural network models (particularly, by increasing the network width); (2) the variance should be decomposed into two parts: one part accounts for the variance caused by random initialization of network parameters/optimization and the other part is caused by "sampling of the training set".
For the first claim is based the empirical observation that increasing the number of hidden units did not cause the incrase of variance (as in figure 1). However, to my understanding, it only means increasing the number of hidden units is probably not a good way to increase the network capacity. In other words, this cannot be used as an evidence that the bias-variance tradeoff is not valid in neural network learning.
For the second claim, I don't like the way that they decompose the variance into two parts. To be clear, the classical bias-variance tradeoff doesn't consider the optimization error as an issue. For a more generic view of machine learning errors, please refer to "The Tradeoffs of Large Scale Learning" (Bottou and Bousquet, 2008). In addition, if the proposed framework wants to include the optimization error, it should also cover some other errors caused by optimization, for example, early stopping and the choice of a optimization algorithm.
Besides these high-level issues, I also found the technical parts of this paper is really hard to understand. For example,
- what is exactly the definition of $p(\theta|S)$? The closely related case I can think about is in the Baysian setting, where we want to give a prior distribution of model (parameter). But, clearly, this is not the case here.
- similar question to the "frequentist risk", in the definition of frequentist risk, model parameter $\theta$ should be fixed and the only expectation we need to compute is over data $S$
- in Eq. (5), I think I need more technical detail to understand this decomposition.


Understanding and improving Generative Adversarial Networks (GAN) using notions from Optimal Transport (OT) theory has been a successful area of study, originally established by the introduction of the Wasserstein GAN (WGAN). An increasing number of GANs incorporate OT for improving their discriminators, but that is so far the sole way for the two domains to cross-fertilize. In this work we address the converse question: is it possible to recover an optimal map in a GAN fashion? To achieve this, we build a new model relying on the second Wasserstein distance. This choice enables the use of many results from OT community. In particular, we may completely describe the dynamics of the generator during training. In addition, experiments show that practical uses of our model abide by the rule of evolution we describe. As an application, our generator may be considered as a new way of computing an optimal transport map. It is competitive in low-dimension with standard and deterministic ways to approach the same problem. In high dimension, the fact it is a GAN-style method makes it more powerful than other methods.

The paper W2GAN describes a method for training GAN and computing an optimal transport (OT) map
between distributions. As far as I can tell, it is difficult to identify the original contributions
of the paper. Most results are known from the OT community. The differences with the work of Seguy, 2018
is also not obvious. I encourage the authors to establish more clearly the differences of their work
with this last reference. Most of the theoretical considerations of Section 3 is either based on
unrealistic assumptions (case 1) or make vague assumptions 'if we ignore the possibly significant effect ...'
that seem unjustified so far. Experimental results do not show evidences of superiority wrt. existing works.
All in all I would recommend the authors to better focus on the original contribution of their works wrt.
state-of-the-art and explain why the theoretical analysis on convergence following a geodesic path in a
Wasserstein space is valuable from a practical view. Finally, I did not understand the final claim of the
Abstract : 'Perhaps surprisingly, we also provide empirical evidence that other GANs also approximately following
the Optimal Transport.'. What are those empirical evidences ? It seems that this claim is not supported somewhere
else in the paper.
Minor remarks:
 - regarding the penalization in eq. (5), the expectation is not for all x and y \in R^2, but for x drawn from \mu and y from \nu.
   Same for L_2 regularization
 - Proposition 1 is mainly due to Brenier
Brenier, Y. (1991). Polar factorization and monotone rearrangement of vector‐valued functions. Communications on pure and applied mathematics, 44(4), 375-417.
 - from Eq (7), you should give precisely over what the expectations are taken.
 - Eq (10) : how do you inverse sup and inf ?
 - when comparing to Seguy 2018, are you using an entropic or a L_2 regularization ? How do you set the regularization strength ?
 - where is Figure 2.a described in section 4.2 ?
Related works :
 - what is reference (Alexandre, 2018) ?
 - regarding applications of OT to domain adaptation, there are several references on the subject.
   See for instance
Courty, N., Flamary, R., Tuia, D., & Rakotomamonjy, A. (2017). Optimal transport for domain adaptation. IEEE transactions on pattern analysis and machine intelligence, 39(9), 1853-1865.
or
Damodaran, B. B., Kellenberger, B., Flamary, R., Tuia, D., & Courty, N. (2018). DeepJDOT: Deep Joint distribution optimal transport for unsupervised domain adaptation. ECCV
for a deep variant.
 - Reference Seguy 2017 and 2018 are the same and should be fused. The corresponding paper
   was published at ICLR 2018
   Regarding this last reference, the claim 'As far as we know, it is the first demonstration of a GAN achieving reasonable generative modeling results and an approximation of the optimal transport map between two continuous distributions.' should maybe be lowered ?


Despite significant advances in the field of deep Reinforcement Learning (RL), today's algorithms still fail to learn human-level policies consistently over a set of diverse tasks such as Atari 2600 games. We identify three key challenges that any algorithm needs to master in order to perform well on all games: processing diverse reward distributions, reasoning over long time horizons, and exploring efficiently. In this paper, we propose an algorithm that addresses each of these challenges and is able to learn human-level policies on nearly all Atari games. A new transformed Bellman operator allows our algorithm to process rewards of varying densities and scales; an auxiliary temporal consistency loss allows us to train stably using a discount factor of γ = 0.999 (instead of γ = 0.99) extending the effective planning horizon by an order of magnitude; and we ease the exploration problem by using human demonstrations that guide the agent towards rewarding states. When tested on a set of 42 Atari games, our algorithm exceeds the performance of an average human on 40 games using a common set of hyper parameters.

Summary: This paper proposes three new techniques for improving Atari performance over APE (Horgan 2018).  Two of them are closely linked in that they deal with improving stability.  Another involves integrating the use of expert trajectories from DQfD.
I will summarize each:
Transformed Bellman: This applies a rescaling function (it's basically a monotonically increasing version of the sqrt(x) function) to the Q-function and applies the inverse of the function to the max Q-value of the next state (such that the contracting effect h-function is not "applied" multiple times when doing the TD backup).
Temporal Consistency: This encourages the "next state" after where the TD-update is applied to not change too much.  This addresses a problem discussed in (Durugkar 2018).  I think the intuition here is that the state which follows the state with the TD update may be visually similar, but it does not impact the value in the past states, so its value function should not have a highly correlated change with the previous state's change in value function.
DQfD: Storing an expert replay buffer and an actor replay buffer.  The expert replay buffer is fixed and the actor replay buffer stores the most recent "actor processes".  Train with both a supervised imitation loss (only for the highest return episode) and the original TD loss.  Additionally, the pre-training phase is removed and the ratio of expert-learned trajectories is fixed (both seem like steps in the right direction).
Review: This paper proposes a few changes to DQN training, two of which are aimed at reducing instability, and one is aimed at improving exploration (expert trajectories).  Because all of these changes are well justified and the experiments are fairly thorough, I recommend acceptance.  My main reservation is that the ideas presented are not very strongly thematically linked.  The presence of ablation studies compensates for this to some extent.
Strengths:
  -The discussion of related work and comparison to baselines is pretty extensive.  For example I appreciated the ablation study removing "transformed Q-learning" and comparison to the pop-art method.
  -The results, at least for Ape-X DQfD seem impressive to me in that the method works without reward clipping and with a much higher discount factor.  Additionally the results generally outperform DQfD (uses expert trajectories) and Rainbow (no human trajectories).  Additionally evidence was presented that the learned policies often exceed the performance of the human demonstrations (for example in time to achieve rewards).
Weaknesses:
  -Two of the techniques: "transformed bellman" and "temporal consistency" seem well-linked thematically, but the expert demonstration idea seems orthogonal.  I would have preferred splitting that idea out into a separate paper, given that the paper is already 20 pages.
  -The motivation for temporal consistency just references (Durugkar 2018).  The readability of this paper would be improved if it were discussed more here as well.  I also feel like the analysis could be more thorough here, for example a result using the temporal consistency loss on Baird's counter example really should be shown (like figure 2 in Durugkar's paper).
-It would be nice to see a visualization or a toy problem with the "transformed bellman".
Questions:
-Is the "highest return episode" idea (3.4) general or is it exploiting the fact that Atari is deterministic?  It seems like in general we'd want to use many high reward episodes, or the highest reward episodes that go into different parts of state space.  It seems like it could be a very bad idea on certain settings (for example if the reward has a lot of randomness).
-"Proposition 3.1 shows that in the basic cases when either h is linear or the MDP is deterministic, Th
has the unique fixed point h ◦ Q∗".  From 3.1, it looks if h is linear, then it distributes over r(x,a) + maxh^{-1}(Q) and then it also won't effect which is the max, so it would reduce to h*r(x,a) + max(Q) - which means it's just rescaling the original reward.  So then this result is trivial?  Please correct me if I misunderstood something here.
-Could an MDP be constructed which causes the transformed bellman operator to perform badly?  I am imagining something where the MDP is just a single step, and there is a stochastic action which behaves like a lottery.  So perhaps there is a 1-in-1-million chance to win 1-billion dollars by taking an action.  If I understand correctly the transformed bellman operator will destroy the large reward here (because in a single step, there is just r(x,a) which h is applied to).  Which would make the action seem bad even though it's actually appealing.
Notes:
  -I did not read the proofs in the appendix.


In this work we offer an information-theoretic framework for representation learning that connects with a wide class of existing objectives in machine learning. We develop a formal correspondence between this work and thermodynamics and discuss its implications.

This paper attempts to establish a notion of thermodynamics for machine learning. Let me give an attempt at summary. First, an objective function is established based on demanding that the multi-information of two graphical models be small. The first graphical model is supposed to represent the actual dependence of variables and parameters used to learn a latent description of the training data, and the model demands that the latents entirely explain the correlation of the data, with the parameters marginalized out. Then, a variational approximation is made to four subsets of terms in this objective function, defining four "thermodynamic"  functionals. Minimizing the sum of these functionals puts a variational upper bound on the objective. Next, the sum is related to an unconstrained Lagrange multiplier problem making use of the facts (1) that such an objective will likely have many different realizations of the thermodynamic functionals for specific value of the bound and (2) that on the optimal surface the value of one of the functional can be parametrized in terms of the three others. If we pick the entropy functional to be parameterized in terms of the others, we find ourself precisely in the where the solution to the optimization is a Boltzmann distribution; the coefficients of the Lagrange multipliers will then take on thermodynamic interpretations in of temperature, generalized chemical potentials, etc. At this point, the machinery of thermodynamics can be brought to bear, including a first law, Maxwell relations (equality of mixed partial derivatives), etc.
I think the line of thinking in this paper is very much worth pursuing, but I think this paper requires significant improvement and modifications before it can be published. Part of the problem is that the paper is both very formal and not very clear. It's hard to understand why the authors are establishing this analogy, where they are going with it, what's its use will be, etc. Thermodynamics was developed to explain the results of experiments and is often explained by working out examples analytically on model systems. This paper doesn't really have either such a motivation or such examples, and I think as a result I think it suffers.
I also think the "Tale of Two Worlds" laid out in Section 2 requires more explanation. In particular, I think more can be said about why Q is the the "world we want" and why minimizing the difference between these worlds is the right way to create an objective. (I have no real problem with the objective once it is derived.) Since this paper is really about establishing this formal relationship, and the starting point is supposed to be the motivating factor, I think this needs to be made much clearer.
The I(Z_i, X_i, Theta) - I(X_i, Z_i) terms could have been combined into a conditional mutual information. (I see this is discussed in Appendix A.) This leads to a different set of variational bounds and a different thermodynamics. Why do we prefer one way over the other? At the level of the thermodynamics, what would be the relationship between these different ways of thinking? Since it's hard to see why I want to bother with doing this thermodynamics (a problem which could be assuaged with worked examples or more direct and clear experiments), it's hard to know how to think about this sort of freedom in the analogy. (I also don't understand why the world Q graphical model is different in Appendix A when we combined terms this way, since the world Q lead to the objective, which is independent of how we variationally bound it.) I think ultimately the problem can be traced to the individual terms in the objective (7) not being positive definitive, giving us the freedom to make different bounds by arranging the pieces to get different combinations of positive definite terms. How am I supposed to think about this freedom?
In conclusion, I would really like to see analogies like this worked out and be used to better understand machine learning methods. But for this program to be successful, I think a very compelling case needs to be made for it. Therefore, I think that this paper needs to be significantly rewritten before it can be published.


DEEP ONLINE LEARNING VIA META-LEARNING: CONTINUAL ADAPTATION FOR MODEL-BASED RL
Humans and animals can learn complex predictive models that allow them to accurately and reliably reason about real-world phenomena, and they can adapt such models extremely quickly in the face of unexpected changes. Deep neural network models allow us to represent very complex functions, but lack this capacity for rapid online adaptation. The goal in this paper is to develop a method for continual online learning from an incoming stream of data, using deep neural network models. We formulate an online learning procedure that uses stochastic gradient descent to update model parameters, and an expectation maximization algorithm with a Chinese restaurant process prior to develop and maintain a mixture of models to handle non-stationary task distributions. This allows for all models to be adapted as necessary, with new models instantiated for task changes and old models recalled when previously seen tasks are encountered again. Furthermore, we observe that meta-learning can be used to meta-train a model such that this direct online adaptation with SGD is effective, which is otherwise not the case for large function approximators. In this work, we apply our meta-learning for online learning (MOLe) approach to model-based reinforcement learning, where adapting the predictive model is critical for control; we demonstrate that MOLe outperforms alternative prior methods, and enables effective continuous adaptation in non-stationary task distributions such as varying terrains, motor failures, and unexpected disturbances.

The paper presents a nonparametric mixture model of neural networks for learning in an environment with a nonstationary distribution. The problem setup includes having access to only a few "modes" of the distribution. Training of the initial model occurs with MAML, and distributional changes during test/operation are handled by a combination of online adaptation and creations of new mixture components when necessary. The mixture is nonparametric and modeled with a CRP. The application considered in the paper is RL, and the experiments compare proposed model against baselines that do not utilize meta-learning (achieved in the proposed method with MAML), and baselines which utilize only a single model component.
I thought the combination of meta-learning and a CRP was a neat way to tackle the problem of modeling and learning the "modes" of a nonstationary distribution. Applications in other domains would have been nice, but the presented results in RL sufficiently demonstrate the benefits of the proposed method.
* Questions/Comments
Figure 3 left vs right?
Is the test in the middle of Algorithm 1 correct?


Models parameterized by deep neural networks have achieved state-of-the-art results in many domains. These models are usually trained using the maximum likelihood principle with a finite set of observations. However, training deep probabilistic models with maximum likelihood can lead to the issue we refer to as input forgetting. In deep generative latent-variable models, input forgetting corresponds to posterior collapse-a phenomenon in which the latent variables are driven independent from the observations. However input forgetting can happen even in the absence of latent variables. We attribute input forgetting in deep probabilistic models to the finite sample dilemma of maximum likelihood. We formalize this problem and propose a learning criterion-termed reflective likelihood-that explicitly prevents input forgetting. We empirically observe that the proposed criterion significantly outperforms the maximum likelihood objective when used in classification under a skewed class distribution. Furthermore, the reflective likelihood objective prevents posterior collapse when used to train stochastic autoencoders with amortized inference. For example in a neural topic modeling experiment, the reflective likelihood objective leads to better quantitative and qualitative results than the variational auto-encoder and the importance-weighted autoencoder.

Summary:
This paper proposes maximizing the “reflective likelihood,” which the authors define as: E_x E_y [log q(y|x) - \alpha log q(y)] where the expectations are taken over the data, q is the classifier, and \alpha is a weight on the log q(y) term.  The paper derives the reflective likelihood for classification models and unsupervised latent variable models.  Choices for \alpha are also discussed, and connections are made to ranking losses.  Results show superior F1 and perplexity in MNIST classification and 20NewsGroups modeling.
Pros:
I like how the paper frames the reflective likelihood as a ranking loss.  It does seem like subtracting off the marginal probability of y from the conditional likelihood should indeed ‘focus’ the model on the dependent relationship y|x.  Can this be further formalized?  I would be very interested in seeing a derivation of this kind.
I like that the authors test under class imbalance and report F1 metrics in the experiments as it does seem the proposed method operates through better calibration.
Cons:
My biggest issue with the paper is that I find much of the discussion lacks rigor.  I followed the argument through to Equation 3, but then I became confused when the discussion turned to ‘dependence paths’: “we want our learning procedure to follow the dependence path—the subspace in Θ for which inputs and outputs are dependent. However this dependence path is unknown to us; there is nothing in Eq. 1 that guides learning to follow this dependence path instead of following Eq. 3—the independence path” (p 3).  What are these dependence paths?  Can they be defined mathematically in a way that is more direct than switching around the KLD directions in Equations 1-3?  Surely any conditional model x-->y has a ‘dependence path’ flowing from y to x, so it seems the paper is trying to make some stronger statement about the conditional structure?
Moving on to the proposed reflective likelihood in Equation 4, I could see some connections to Equations 1-3, but I’m not sure how exactly that final form was settled upon.  There seems to be a connection to maximum entropy methods?  That is,   E_x E_y [log q(y|x) - \alpha log q(y)] = E_x E_y [log q(y|x)] + \alpha E_y [ -log q(y)] \approx E_x E_y [log q(y|x)] + \alpha H[y], if we assume q(y) approximates the empirical distribution of y well.  Thus, the objective can be thought of as maximizing the traditional log model probability plus an estimate of the entropy.  As there is a long history of maximum entropy methods / classifiers, I’m surprised there were no mentions or references to this literature.  Also, I believe there might be some connections to Bayesian loss calibration / risk by viewing \alpha as a utility function (which is easy to do when it is defined to be data dependent).  I’m less sure about this connection though; see Cobb et al. (2018) (https://arxiv.org/abs/1805.03901) and its citations for references.
The data sets used in the experiments are also somewhat dissatisfying as MNIST and 20NewsGroups are fairly easy to get high-performing models for.  I would have liked to have seen more direct analysis / simulation of what we expect from the reflective likelihood.  As I mentioned above, I suspect its really providing gains through better calibration---which the authors may recognize as F1 scores are reported and class imbalance tested---but the word ‘calibration’ is never mentioned.  More direction comparison against calibration methods such as Platt scaling would be make the experiments have better focus.  It would be great to show that this method provides good calibration directly during optimization and doesn’t need the post-hoc calibration steps that most methods require.
Evaluation:  While the paper has some interesting ideas, they are not well defined, making the paper unready for publication.  Discussion of the connections to calibration and maximum entropy seems like a large piece missing from the paper’s argument.


We propose a new notion of 'non-linearity' of a network layer with respect to an input batch that is based on its proximity to a linear system, which is reflected in the non-negative rank of the activation matrix. We measure this non-linearity by applying non-negative factorization to the activation matrix. Considering batches of similar samples, we find that high non-linearity in deep layers is indicative of memorization. Furthermore, by applying our approach layer-by-layer, we find that the mechanism for memorization consists of distinct phases. We perform experiments on fully-connected and convolutional neural networks trained on several image and audio datasets. Our results demonstrate that as an indicator for memorization, our technique can be used to perform early stopping.

This paper propose a new way of analyzing the robustness of neural network layers by measuring the level of "non-linearity" in the activation patterns on samples belonging to the same class, and correlate that to the level of "memorization" and generalization.
More specifically, the paper argues that a good representation cluster all the samples in a class together, therefore, in higher layers, the activation pattern of samples from the same class will be almost identical. In this case, the activation matrix will have a small non-negative rank. An approximation algorithm (via non-negative matrix factorization) is then used to compute the robustness and evaluate the robustness (by replacing the activation matrix with its low rank non-negative activation) is measured in a number of experiments with different amount of random label corruptions. The experiments show that networks trained on random labels are less robust than networks trained on true labels.
While the concept is interesting, I find the arguments in the paper a bit vague, and the usefulness of the algorithm might be hampered by its computation complexity, which is not discussed in the paper.
First of all, the paper lacks a clear notion of "memorization". While it is generally accepted that learns on random labels can be called "memorization", the paper seem to be defining it as how well is the network clustering points from the same class. Several questions need to be addressed in order for this notion to be justified:
1. Are (well generalized) networks really clustering samples of the same class to a centroid? It would be great if some empirical verifications are shown. Because the networks are using linear classifier in the last layer to classify the samples, it seems only linearly separability would be suffice for the work, which does not necessarily imply clustering.
2. Given two networks (of the same architecture), assume somehow network-1 decides to use the first 9 layers to compute a well clustered representation, while network-2 decides to use the first 5 layers to do the same thing. Do we say network-1 is (more) memorizing in this case?
3. The notion seems to be more about the underlying task than about the networks. Given the measurement, if a task is more complicated, meaning the input samples in the class have higher variance and requiring more efforts to cluster, then it seems the network will be doing more memorization. In other words, while networks will be doing more memorization when comparing a random label task to a true label task, it might also be "doing more memorization" when comparing learning on imagenet to learning on MNIST / CIFAR. One the one hand, this does not seem to fit our "intuition" about memorization; on the other hand, the heavy dependency on the underlying data distribution makes it difficult to compare results learned on different data -- especially since the measurements are based on per-class samples, "random labels" and "true labels" have very different class-conditional distributions.
I also have some questions about Figure 2(c). I will continue numbering the question for easier discussion.
4. Why for all cases, the lower layers all have higher AUC than the higher layers (except the last one)? The argument given in the paper is that the lower layers are the feature extraction phase while the upper layers are memorization phase. I think if clearly verified, this is a very interesting observation. But the paper currently do not have experiments to verify the hypothesis. Also more studies on this with different networks would be good. For example, with deeper networks, does the feature extraction phase include more layers?
5. The p=1 and p<1 curves seem to be very different. If one is to sample more densely between p=0.8 and p=1, would there still be a clear phase transition?
Some other questions:
6. Can you add discussions to the computation requirements for the proposed analysis? This is especially important for the cases where the analysis is used during training as tools to help deciding early stopping.
7. For the early stopping experiment, the main text says "These include the test error (in blue)" while in the figure the label axis is "Test loss". I'm assuming it is the cross entropy loss given the values are greater than 1. In this case, can you show in parallel the same plots in error rate, as the test error is more important than the test loss and the test loss could sometimes be artificially huge due to high confident mistakes on ambiguous test examples.
Some minor issues:
* Please proof read the paper for typos. E.g. on the 3rd paragraph of the 1st page: "that networks that networks that".
* The convention with subplots seem to be putting sub-captions under the figures, not above.


DEEP REINFORCEMENT LEARNING WITH RELATIONAL INDUCTIVE BIASES
We introduce an approach for augmenting model-free deep reinforcement learning agents with a mechanism for relational reasoning over structured representations, which improves performance, learning efficiency, generalization, and interpretability. Our architecture encodes an image as a set of vectors, and applies an iterative message-passing procedure to discover and reason about relevant entities and relations in a scene. In six of seven StarCraft II Learning Environment mini-games, our agent achieved state-of-the-art performance, and surpassed human grandmasterlevel on four. In a novel navigation and planning task, our agent's performance and learning efficiency far exceeded non-relational baselines, it was able to generalize to more complex scenes than it had experienced during training. Moreover, when we examined its learned internal representations, they reflected important structure about the problem and the agent's intentions. The main contribution of this work is to introduce techniques for representing and reasoning about states in model-free deep reinforcement learning agents via relational inductive biases. Our experiments show this approach can offer advantages in efficiency, generalization, and interpretability, and can scale up to meet some of the most challenging test environments in modern artificial intelligence.

The authors present a deep reinforcement learning approach that uses a “self-attention”/“transformer”-style model to incorporate a strong relational inductive bias. Experiments are performed on a synthetic “BoxWorld” environment, which is specifically designed (in a compelling way) to emphasize the need for relational reasoning. The experiments on the BoxWorld environment clearly demonstrate the improvement gained by incorporating a relational inductive bias, including compelling results on generalization. Further experimental results are provided on the StarCraft minigames domain. While the results on StarCraft are more equivocal regarding the importance of the relational module—the authors do set a new state of the art and the results are suggestive of the potential utility of relational inductive biases in more general RL settings.
Overall, this is a well-written and compelling paper. The model is well-described, the BoxWorld results are compelling, and the performance on the StarCraft domain is also quite strong. The paper clearly demonstrates the utility of relational inductive biases in reinforcement learning.
In terms of areas for potential improvement:
1) With regards to framing, a naive reader would probably get the impression that this is the first-ever work to consider a relational inductive bias in deep RL, which is not the case, as the NerveNet paper (Wang et al., 2018) also considers using a graph neural network for deep RL. There are clear differences between this work and NerveNet—most prominently, NerveNet only uses a relational inductive bias for the policy network by assuming that a graph-structured representation is known a priori for the agent. Nonetheless, NerveNet does also incorporate a relational inductive bias for deep RL and shows how this can lead to better generalization. Thus, this paper would be improved by properly positioning itself w.r.t. NerveNet and highlighting how it is different.
2) As with other work using non-local neural networks (or fully-connected GNNs), there is the potential issue of scalability due to the need to consider all input pairs. A discussion of this issue would be very useful, as it is not clear how this approach could scale to domains with very large input spaces.
3) Some details on the StarCraft experiments could be made more rigorous and quantitative. In particular, the following instances could benefit from more experimental details and/or clarifications:
Figure 6: The performance of the control model and relational model seem very close. Any quantitative insight on this performance gap would improve the paper. For instance, is the gap between these two models significantly larger than the average gap between runs over two different random seeds? It would greatly strengthen the paper to clarify that quantitive aspect.
Page 8: ”We observed that—at least for medium sized networks—some interesting generalization capabilities emerge, with the best seeds of the relational agent achieving better generalization scores in the test scenario” — While there is additional info in the appendix, without quantitative framing this statement is hard to appreciate. I would suggest more quantitive detail and rigorous statistical tests, e.g.,  something like “When examining the best 10 out of ??? seeds, the relational model achieved an average performance increase of ???% compared to the control model (p=???, Wilcoxon signed-rank test). However, when examining all seeds ???? was the case.”
Page 8: “while the former adopted a "land sweep strategy", controlling many units as a group to cover the space, the latter managed to independently control several units simultaneously, suggesting a finer grained understanding of the game dynamics.” This is a great insight, and the paper would be greatly strengthened by some quantitive evidence to back it up (if possible). For instance, you could compute the average percentage of agents that are doing the same action at any point in time or within some distance from each other, etc. Adding these kinds of quantitative statistics to back up these qualitative insights would both strengthen the argument, while also making it more explicit how you are coming to these qualitative judgements.
Figure 8 caption: “Colored bars indicate mean score of the ten best seeds” — how bad is the drop to the n-10 non-best seeds? And how many seeds where used in total?
Page 13: “following Table 4 hyperparameter settings and 3 seeds” — if three seeds are used in these experiments, how are 10+?? seeds used for the generalization experiments? The main text implies that the same models for the “Collect Mineral Shards” were re-used, but it appears that many more models with different seeds were trained specifically for the generalization experiment. This should be clarified. Alternatively, it is possible that “seeds” refers to both random seeds and hyperparameter combinations, and it would improve the paper to clarify this. It is possible that I missed something here, but I think it highlights the need for further clarification.


Model distillation aims to distill the knowledge of a complex model into a simpler one. In this paper, we consider an alternative formulation called dataset distillation: we keep the model fixed and instead attempt to distill the knowledge from a large training dataset into a small one. The idea is to synthesize a small number of data points that do not need to come from the correct data distribution, but will, when given to the learning algorithm as training data, approximate the model trained on the original data. For example, we show that it is possible to compress 60, 000 MNIST training images into just 10 synthetic distilled images (one per class) and achieve close to original performance with only a few steps of gradient descent, given a particular fixed network initialization. We evaluate our method in a wide range of initialization settings and with different learning objectives. Experiments on multiple datasets show the advantage of our approach compared to alternative methods in most settings.

=== Post rebuttal update ===
Thanks to the addition of better baselines, I've increased my score for this paper. While I'm still not super convinced of its potential for application, I find the idea original and worth discussing at the conference.
=== Pre-rebuttal review ===
This paper presents an approach to compress a dataset into a much smaller number of synthetic samples that are optimized to yield as good performance as possible when a given model is trained on that smaller dataset. This is done by unrolling the gradient descent procedure of training such a model to allow for gradient-based optimization of synthetic samples themselves as well as the used learning rates.
In summary, my evaluation is as follow:
*Pros*
- Pretty original problem formulation
- Generally well written paper
*Cons*
- Lack of comparison with simple baselines in basic dataset distillation setting
- Use in practical applications (domain adaptation, data poisoning) yet to be convincingly demonstrated
- Possibly a mistake in the theoretical analysis of the linear case
Indeed, I found the paper to be generally quite clear and enjoyed reading it. One minor thing I struggled a bit with is the distinction between "SG steps" and "Epochs" (I believe the former corresponds to when the synthetic samples are different between GD steps, whereas the later corresponds to the number of times the method repeatedly cycles over these samples) so I would perhaps encourage the authors to emphasize that difference.
I also find the problem statement that proposed to be interesting and thought provoking, and the solution that's proposed seems quite appropriate and well thought out.
That said, I'm worried about the following:
- Unless I misunderstood, in the basic dataset distillation setting a comparison is never provided with training on a randomly selected subset of the training set. Presumably the results are worse, but I think these results should be in the paper. I would also argue for having another baseline, which would try to (approximately) optimize the choice of which training examples are put in the subset. A very simple approach would be to take the 200 runs already performed for the random selection and select the subset providing the best accuracy on the full training set and only report the performance of that subset (instead of the mean and std of all 200 runs). In short, this would help determine to what extent there is value to synthesizing entirely new samples. Moreover, I think a simple alternative baseline for creating synthesized samples should be considered. Specifically, I'd personally would like to know the performance of using per-class k-means clustering and training on the cluster centroids as the distilled dataset.
- While I appreciate that the authors identify potential applications and report some results on them, I think they currently fall short of convincing the reader of the potential of dataset distillation for these applications. For domain adaptation, no actual domain adaptation baseline is compared against (a good candidate would be method from Daume III 2007, at the very least). For data poisoning, I find that the assumptions for attacks are pretty strong, i.e. that a) you have access to the parameters of the pre-trained models to attack and b) that the model is doing additional updates *only* on the synthesized data. If there are reasons to think that such assumptions are reasonable, I'd at least expect the paper to motivate why that is.
- In the analysis of the simple linear case, in Equation 7, there appears to be a mistake, specifically some missing parentheses:
d^Td( (I-\eta/M d^Td)\theta_0 + \eta/M \tilde{d}^T\tilde{t}) = d^T t
i.e. there should be parentheses right after "d^Td" and right before "=". This is from replacing \theta^* by the expression for \theta_1 in Equation 6, which is what I think Equation 7 is supposed to be doing. This possibly doesn't affect some of the conclusions taken from this section, but I'd like to see this potential mistake discussed/addressed.
That said, if the authors can sufficiently address the 3 points above, I'd be willing to increase my rating for this paper.
Finally, I have a few other more minor (nice-to-have) points:
- Having in the related work a discussion on the relationship with coreset methods would be nice
- Experiments showing how well the distilled datasets transfer to different network architectures than those used in training would be interesting? Even other ML algorithms would be quite interesting?
- "We often find that the number of distilled images required to achieve good performance is an informative indicator of the dataset diversity" => I'm not sure what in the paper actually justifies / demonstrates this statement.
- Figure 4 is presented as an "Ablation study", but an ablation study is where you remove certains parts of a model or algorithm and see what happens, which isn't the case here. I think it's better described as a hyper-parameter sensitivity study.
- Some typos:
   * the below objective => the objective below
   * w.r.t. to => w.r.t
   * the discrete part rather => the discrete parts rather
   * necesary => necessary