Many machine learning problems can be expressed as the optimization of some cost functional over a parametric family of probability distributions. It is often beneficial to solve such optimization problems using natural gradient methods. These methods are invariant to the parametrization of the family, and thus can yield more effective optimization. Unfortunately, computing the natural gradient is challenging as it requires inverting a high dimensional matrix at each iteration. We propose a general framework to approximate the natural gradient for the Wasserstein metric, by leveraging a dual formulation of the metric restricted to a Reproducing Kernel Hilbert Space. Our approach leads to an estimator for gradient direction that can trade-off accuracy and computational cost, with theoretical guarantees. We verify its accuracy on simple examples, and show the advantage of using such an estimator in classification tasks on Cifar10 and Cifar100 empirically.

Thank you for your revision and for the rebuttal. This a  strong submission with insightful angle on natural gradients and with provable guarantees. The authors improved a lot the manuscript and incorporated reviewers feedback. I am increasing my score to 8.

###
Summary of the paper:
The paper provides a way to estimate the natural Wasserstein gradient using Kernel estimators. The idea is neat and novel. Natural Wasserstein Gradient similar to the so called natural fisher gradients preconditions the gradient using a matrix that uses the local curvature of the manifold of the parametric distribution.
Authors give variational forms of the Fisher information matrix of an explicit model , using  the variational form of the chi squared or the Fisher Rao divergence. Similarly authors give a variational form of the wasserstein natural gradient . Let theta be the parameter of the parametric implicit model, theta in R^q.  For a descent direction $u$, the variational form is obtained via finding an objective S,  $\sup_{f\in C^c_{\infty}} S(f, u ) = u^{\top}G_{W}u$, where $G_{W}$ is a form of  "Wasserstein information matrix".
Authors then propose to learn the function f in an RKHS and propose to find the descent direction by solving
 $\min_{u} <u, \nabla_{\theta} Loss (p_{\theta})>  + \sup_{f\in RKHS}  S(f, u ) + r(u)- \lambda ||f||^2_{rkhs}$
where r(u) is a quadratic regularizer on u.
The sup problem has a closed form solution and can be approximated  using Nystrom approximation and randomization on dimensions. The problem in u has also a closed form solution , and one used u as the proxy to the natural  descent.
Authors under some assumption  show that the  estimated natural W gradient in RKHS  is concentrated around the true one.
Experiments on synthetic data and in classification on CIFAR 10 and CIFAR 100 shows that the preconditioning of the gradients that the method offers allows faster convergence in both well conditioned and ill conditioned initialization of the weights of the neural network.

Review :
The paper is not easy to follow and the high level intuition how the method works is not well explained.
It would be easier for the reader, to motivate the natural wasserstein descent from how one defines natural Fisher descent , where one seeks a first order approximation of $KL(p_{\theta},p_{\theta+ \epsilon u})$ as we perturb in the parameter space and this well known that this epsilon $u^{\top}F u$.
Hence natural Gradient descent is :
$\min _{u} <u, \nabla_{\theta}Loss(p_{\theta})> + KL(p_{\theta},p_{\theta+  u}) \approx min _{u} <u, \nabla_{\theta}Loss(p_{\theta})> + u^{\top}F u$
Now for the wasserstein distance one has also similarly:
$\min _{u} <u, \nabla_{\theta}Loss(p_{\theta})> + W^2_2(p_{\theta},p_{\theta+  u})$
and it is known that as epsilon goes to zero we have:
 $W^2_2(p_{\theta},p_{\theta+ \epsilon u})/\epsilon = ||p_{\theta} - p_{\theta+ \epsilon u}||^2_{H^{-1}(p_{\theta})}+o(\epsilon) = \sup_{f} \int f (p_{\theta} - p_{\theta+\epsilon u}) - \frac{1}{2} \mathbb{E}_{p_{\theta}}||\nabla_x f(x)||^2+o(\epsilon) $
Now replacing with the implicit model as epsilon goes to zero we get the expression given in the paper using a simple taylor expansion:
$=   \sup_{f} \int <\nabla_{\theta} h_{\theta}^{\top}\nabla_x f(h_{\theta}),u > d\nu - \frac{1}{2}\mathbb{E} _{p_{\theta}}||\nabla_x f(x)||^2$
in a sense the paper is proposing to linearize $W^2_2$ around the perturbation in the parameter space of the implicit model and this can be done using  $|| .||_{H^{-1}(q)}$ , as pointed and used in many recent works.  then the paper proposes to approximate $|| .||_{H^{-1}(q)}$ in RKHS which was already proposed in Mroueh et al  in Sobolev Descent.  AISTATS 2019.
We encourage the authors to layout in the beginning the derivations form this point of view which will make the paper easier to digest, the expression in Equation 7 seems mysterious and pulled out of a hat, but it is easier to understand by going to perturbation analysis usually done on KL for Fisher Natural gradient and to do it also here starting from the linearization of $W_2$ with $||.||_{H^{-1}(q)}$  , and how to approximate it in RKHS as it was already proposed in the literature in Mroueh et al Sobolev Descent.
I read carefully the proofs of Proposition 1, 2, 3. I did not ready full the proofs of the concentration of the estimator , but they seem sensible as they follow usual bounding strategies in this context.
Questions:
- There is nothing special about the wasserstein natural gradient flow variational form and implicit model, once can apply the same to the variational form of Fisher, that would be probably more efficient? It would be great to baseline this one ?
-the constraint $\int f(x)p_{\theta}(x)=0$ is not imposed in the kernelized version?
- the method comes disappointing since it seems that the preconditioning that the Wasserstein gradient gives is not enough and $r(u)=u^{\top}D u$ is need where D is diagonal depends on T. Have you tried with $D=Identity$? it might be that the scaling of the gradients is coming only from that $D^{-1}$?
- Can you give timings for computing each gradient update and how it compares to regular SGD or diagonal approximation of Fisher natural gradient?
- Does one need preconditioned gradient if the network was self normalized (like batch norm or spectral norm etc)?
Overall assessment:
That is a good theoretical work with provable guarantees. The computational complexity of each gradient estimate is large which makes the method not quite appealing in practice.


ABSTRACT Auto-regressive model architectures are the workhorse of sequence generation tasks in text, images, audio, and more. These models are predominantly trained with teacher-forcing, wherein the model is encouraged to predict the next ground-truth token conditioned on all previous ground-truth tokens in the sequence. At test time, the model does not have access to ground-truth tokens and must instead condition on its own prior predictions, resulting in a train-test discrepancy known as exposure bias (Ranzato et al., 2015). Scheduled Sampling (Bengio et al., 2015) aims to mitigate exposure bias by randomly replacing some tokens with model predictions during the course of training. Unfortunately, this technique is inherently sequential and unlike teacher-forcing, cannot be trivially parallelized across time. In this paper, we introduce a simple technique to parallelize Scheduled Sampling, significantly increasing training throughput. Experimentally, we find the proposed technique leads to equivalent or better performance on image generation, summarization, dialog generation, and translation compared to teacher-forcing. In dialog response generation, Parallel Scheduled Sampling achieves a 1.6 BLEU score (11.5%) improvement over teacher-forcing with more than 250 times higher throughput than Scheduled Sampling. In image generation, our approach achieves 20% and 13.8% improvement in Fréchet Inception Distance (FID) and Inception Score (IS) respectively. Finally, we discuss the effects of different hyper-parameters associated with Scheduled Sampling on the model performance.

Scheduled Sampling aims to overcome the problems that come with the discrepancy between train and test
when training sequence generation models using teacher-forcing. However one of the drawbacks scheduled sampling is that is it is hard to parallelize this is simply because some of the decode input tokens aren't known beforehand and need to be inferred i.e. loss of some tokens are dependent on some previous generations.
Authors introduce a method to perform parallel scheduled sampling by iteratively train using teacher forcing however after each iteration the gold references are mixed with the model predictions with a probability p, this allows parallelization as all decoder input tokens can be known beforehand. The final loss is calculated as before, conditioning on the final mixture.
In theorem 2.1 Authors proof that parallel scheduled sampling can converge to scheduled sampling when P is set to 1 and the number of iterations is larger than the sentence length.
In experiments over summarization, dialogue response generation and image generation authors show that the parallel scheduled sampling can achieve comparable performance to scheduled sampling with high-performance gains reaching 300 times faster and very comparable to teacher forcing.
pros:
The work presents a simple idea that is presented neatly with sufficient experiments. One of the outcomes of this method, that might not been stressed enough in the paper, is a neat way of doing scheduled sampling for the transformers architecture which wasn't straight forward before.
Current proposals include some architecture modifications such as in Mihaylova et al. 2019 https://arxiv.org/pdf/1906.07651.pdf
cons:
The only drawback I can find in this paper is it is lacking content. while the idea is interesting and supported by experiments, I find the content is slightly below the amount of content in average ICLR papers.
The explanation of the proposed scheduled sampling can be much simplified. The idea is quite simple however for example I find the pseudo code in Algorithm 2 is quite hard to grasp maybe due to some typos
Questions to authors:
- We only see performance comparison in dialogue response generation experiments. There are other factors that can affect performance or make parallelization effective.  I wonder what are the performance gains of parallel scheduled sampling on normal scheduled sampling with respect to avg. number of tokens / sentence or batch size.
- I had a hard time grasping Algorithm 2 although I understood the corresponding text could you please verify there aren't any typos. What it says is in the first iteration (k=1) scheduled sampling will be performed which doesn't make sense.


Transformers have obtained significant success modeling natural language as a sequence of text tokens. However, in many real world scenarios, textual data inherently exhibits structures beyond a linear sequence such as trees and graphs; an important one being multi-hop question answering, where evidence required to answer questions are scattered across multiple related documents. This paper presents Transformer-XH, which uses eXtra Hop attention to enable the intrinsic modeling of structured texts in a fully data-driven way. Its new attention mechanism naturally "hops" across the connected text sequences in addition to attending over tokens within each sequence. Thus, Transformer-XH better answers multi-hop questions by propagating information between multiple documents, constructing global contextualized representations, and jointly reasoning over multiple pieces of evidence. This leads to a simpler multi-hop QA system which outperforms previous state-of-the-art on the HotpotQA FullWiki setting by large margins.

This paper introduces the Transformer XH model architecture, a transformer architecture that scales better to long sequences / multi-paragraph text compared to the standard transformer architecture. As I understand, Transformer XH is different from the standard transformer architecture in two main ways: 1) adding attention across paragraphs (or sub-sequences) via the CLS token and 2) defining the structure of that attention based on the entities in the paragraphs (or sub-sequences). The paper tackles an important problem (learning from long sequences) and achieves good empirical results on HotpotQA.
Modification #1 has already been explored by previous works which should be discussed in the paper. "Generating Wikipedia by Summarizing Long Sequences" by Liu, Saleh, et al. propose the T-DMCA model, with a similar motivation: enabling the transformer to work on/scale to longer sequences. T-DMCA and Transformer XH have some difference (T-DMCA seems to have more capacity while Transformer XH is simpler); I think it is necessary to compare against Transformer XH against T-DMCA on HotpotQA to know whether a new architecture is really necessary for HotpotQA. "Generating Long Sequences with Sparse Transformers" from Child et al. also proposes another general transformer architecture that can handle long sequences, and it would be ideal to compare against this architecture as well. Sparse Transformers reduce the time complexity of attention to reduce O(n√ n), which seems similar to the reduction that Transformer XH gets.
For modification #2 (defining the attention structure beforehand using e.g. entity linking), it does not seem too difficult to learn the attention structure directly instead, as confirmed by the ablation in Table 3 which uses a fully connected graph structure. A model that learned the attention pattern or used a fully connected graph would be more general (but more similar to T-DMCA and sparse transformers).
The empirical results are good. It's nice that a simple/straightforward architecture like Transformer XH works quite well (compared to some previous approaches which were not as elegant). However, I do not feel that prior work has explored the best transformer architectures for HotpotQA (such as Sparse Transformers or T-DMCA), and since this work is specifically proposing a new transformer architecture, I think it is important to compare directly against other transformer architectures. Other (previously) SOTA models like SR-MRS are quite simple (conceptually), so it's likely that such models will also be outperformed by transformer architectures that are better adapted to long sequences. In general, I think that the most relevant baseline already present is SR-MRS rather than CogQA; the fact that such a simple approach like SR-MRS works well is an important fact about HotpotQA to take into account (even if SR-MRS is concurrent). In other words, SR-MRS shows that previous models like CogQA are likely weaker baselines.
Because of the missing baselines and limited novelty compared to prior work, I overall lean towards rejecting the paper (despite the good empirical results).
I do have a few specific questions for the authors:
- Would you mind providing further details about the following sentence? "For better retrieval quality, we use a BERT ranker (Nogueira & Cho, 2019) on the set Dir ∪ Del and keep top K ranked ones."
- Is only answer-level supervision used? Or is supporting-fact level supervision used to train any of the rankers or pipeline?


Supervised deep-embedding methods project inputs of a domain to a representational space in which same-class instances lie near one another and different-class instances lie far apart. We propose a probabilistic method that treats embeddings as random variables. Extending a state-of-the-art deterministic method, Prototypical Networks (Snell et al., 2017), our approach supposes the existence of a class prototype around which class instances are Gaussian distributed. The prototype posterior is a product distribution over labeled instances, and query instances are classified by marginalizing relative prototype proximity over embedding uncertainty. We describe an efficient sampler for approximate inference that allows us to train the model at roughly the same space and time cost as its deterministic sibling. Incorporating uncertainty improves performance on few-shot learning and gracefully handles label noise and out-of-distribution inputs. Compared to the state-of-the-art stochastic method, Hedged Instance Embeddings (Oh et al., 2019), we achieve superior large-and open-set classification accuracy. Our method also aligns class-discriminating features with the axes of the embedding space, yielding an interpretable, disentangled representation.

The paper proposes stochastic prototype embeddings (SPE) for few-shot learning. The method is an extension of Prototypical Networks (PN, [1]) with Gaussian embeddings. The idea is to take representation uncertainty into account when classifying objects which makes the model more robust to input and label noise. The authors propose an efficient sampling algorithm to train SPE which outperforms naive Monte Carlo sampling. They conduct a range of experiments on few-shot learning tasks on a synthetic dataset, Omniglot and N-digit MNIST and compare to Prototypical Networks and previous stochastic embedding state-of-the-art method HIB [3]; SPE was shown to outperform prior work in most settings. The authors also plot interpretable disentangled representations learned in 2D embedding space.
However, in my opinion there are a few weaknesses in the paper in its current state: (1) the clarity of sections 3.1-3.2 could be significantly improved as currently the proposed probabilistic model framework is confusing and not well-defined; (2) the experimental results are provided only for embedding spaces of dimensionality 2-3 which significantly hinders performance of Prototypical Networks compared to having a much higher dimensional embedding space, so it would help to see comparisons of SPE and PN using high-dimensional embeddings; (3) the central idea and proposed model seem to be very close to those of [2] which is mentioned in the related work, so, please, list differences with this prior work in the updated version.
For these reasons, I recommend a weak reject for the paper. I explain these issues in more detail below.
(1) SPE model assumes that each embedding z is Gaussian-distributed p(z|x) = N(z; mu_x, sigma^2_x I) where parameters of the distribution mu_x and sigma_x are outputted by an embedding neural network. In equation (3) the author define each class prototype as rho_y = z_i + eps where z_i are instances of class y and eps is noise ~ N(0, sigma^2_eps I), so p(rho_y | z_i) = N(rho_y | z_i, sigma^2_eps I) for every z_i from class y. This definition is confusing to me since rho_y is redefined for each z_i, so it is not clear what the generative process for rho_y is. In the Introduction section it is mentioned “...each class instance is assumed to be a Gaussian perturbation of the prototype” which suggests z_i = rho_y + eps for every z_i, so graphical model would be rho_y -> z_i instead of z_i -> rho_y implied in section 3.1. Please, clarify in the rebuttal which the graphical model is implied in SPE. Is it x_i -> z_i -> rho_y? How is then the distribution of rho_y defined given all z_i from class y?
Further, in the equation (4) the likelihood of rho_y given all x_i from class y is proportional to product of p(rho_y | x_i) so the expression was factorized over the condition which is the reflection of the authors’ assumption of consistency of the prototype. p(rho_y | x_1, … x_n) could be also written as proportional to the product of p(x_i | rho) times p(rho) using Bayes rule. It would help if authors could explain the differences between the two possible expressions for p(rho_y | x_1 … x_n) and why the former is chosen.
Also, in equation (6) the right-hand-side is p(z | y, S) = p(z | rho_y, S) = N(z | mu_y, \hat{sigma}^2_y I): why is the variance is \hat{sigma}^2_y (so extra sigma^2_eps added to sigma^2_y) instead of just sigma^2_y defined in equation (5)? The noise term with sigma^2_eps seems to be already included in sigma^2_y.
(2) In all presented experiments, the dimensionality of the embedding is 2-3 instead of high dimensional embedding spaces in the original Prototypical Network paper which causes a significant drop in performance: e.g., on Omniglot 1-shot 5-class classification, the accuracy dropped from 98.8% to 75.7%. SPE outperformed PN, but an important aspect of the PN model, embedding space dimensionality, was changed from the original in a way that hindered the performance very much, so SPE’s advantage is unclear. It is mentioned in section 4.2 that “We also compared PN and SPE using a 64D embedding, but with high dimensional embeddings, both methods are near ceiling on this data set, resulting in comparable performance between the two methods. (See Appendix D for additional results, broken down by condition.)” but the results on 64D embeddings are not added and discussed in the appendix. If the performance on high dimensional embeddings in the standard setting is similar for SPE and PC, it would help to see the 64D embedding experiment with corrupted support and query data, because if SPE outperformed PN on the corrupted data setting, it would be a fair advantage. It is interesting to see what the 2D embedding space looks like in the visualizations (Figures 2, 5, 7), and SPE is learning interpretable disentangles representations. However, for better classification performance it makes sense to use higher dimensional embedding space to allow learning more expressive representations.
For Figures 2c, 5 and 7, it would be helpful to see how the embedding space of PN looks like and compare it visually to the one of SPE, e.g. whether PN has disentangled representations. It seems like we could get a similar embedding space to, for example, Figure 2c and the label uncertainty would come from the roughly equal distance of object representation to different class prototypes. Please, add respective embedding visualizations for PN and explain what the advantages of SPE learned representations over PN’s representations are.
(3) The prior work of [2] referenced in the paper seems to have a very similar method (judging from Figure 2 and equations 5-6 which compute parameters of the prototype distribution, [2]). Please, list the differences with this prior work and provide experimental comparison if possible.
Other questions:
1. In Figure 2c, please plot and highlight the learned distribution of the class prototypes p(rho | S).
2. Section 4.1: did each object in the training set have only one label or multiple labels (since classes are overlapping)?
3. Experiments: please add HIB method to synthetic data and Omniglot experiments if possible. It would also be interesting to compare to PN on miniImageNet (section 3.2 of [1]). Another interesting experiment which could show advantage of SPE could be out-of-distribution data detection (e.g., comparing likelihoods p(z | y) for in-distribution representations z and out-of-distribution z).
[1] Snell, Jake, Kevin Swersky, and Richard Zemel. "Prototypical networks for few-shot learning." Advances in Neural Information Processing Systems. 2017.
[2] Fort, Stanislav. "Gaussian prototypical networks for few-shot learning on omniglot." arXiv preprint arXiv:1708.02735 (2017).
[3] Oh, Seong Joon, et al. "Modeling uncertainty with hedged instance embedding." arXiv preprint arXiv:1810.00319 (2018).


Adaptive gradient algorithms perform gradient-based updates using the history of gradients and are ubiquitous in training deep neural networks. While adaptive gradient methods theory is well understood for minimization problems, the underlying factors driving their empirical success in min-max problems such as GANs remain unclear. In this paper, we aim at bridging this gap from both theoretical and empirical perspectives. First, we analyze a variant of Optimistic Stochastic Gradient (OSG) proposed in (Daskalakis et al., 2017)  for solving a class of nonconvex non-concave min-max problem and establish O( −4 ) complexity for finding -first-order stationary point, in which the algorithm only requires invoking one stochastic first-order oracle while enjoying state-of-the-art iteration complexity achieved by stochastic extragradient method by (Iusem et al., 2017). Then we propose an adaptive variant of OSG named Optimistic Adagrad (OAdagrad) and reveal an improved adaptive complexity O − 2 1−α 1 , where α characterizes the growth rate of the cumulative stochastic gradient and 0 ≤ α ≤ 1/2. To the best of our knowledge, this is the first work for establishing adaptive complexity in nonconvex non-concave min-max optimization. Empirically, our experiments show that indeed adaptive gradient algorithms outperform their non-adaptive counterparts in GAN training. Moreover, this observation can be explained by the slow growth rate of the cumulative stochastic gradient, as observed empirically.

--------------------------
After the revisions made by the authors, my main concerns about the paper have been removed. Therefore, I am raising my score to 6 (weak accept).
--------------------------
Summary
The present work is concerned with the development of algorithms for the solution of variational inequalities in the stochastic setting, that is when the gradient computations are corrupted by noise. In this setting, the authors propose a variation of the extragradient method which they call Optimistic Stochastic Gradient (OSG), and show that by using a suitable method of variance reduction, the convergence rate of the algorithm matches the state of the art rate of convergence while relaxing the assumptions on the VI from pseudomonotone to assuming that the associated minty variational inequality has a solution. The authors furthermore introduce an adagrad-version of the same algorithm and show that improved convergence rates can be obtained depending on the growth rate of the cumulative stocastic gradients. An extensive suite of experiments studies the empirical performance of the proposed algorithms and compares them to the commonly used Adam optimizer.
Decision
In its present form, while some of the contributions of the paper seem to be relevant contributions (theoretical analysis of adagrad version, state of the art performance of optimistic mirror descent), other contributions (some of which are advertised strongly in the abstract) are incremental or implicit in earlier work (optimistic stochastic gradient descent, relaxation from pseudomonotone to minty VI). Furthermore, while the experimental part is detailed, the connection to the theory could arguably be strengthened more. I believe that by more concisely focusing the paper on its innovative aspects while relating it more directly to existing prior work, it could make for a much more valuable contribution to the literature, which is why I vote for rejecting the paper in its present form.
Additional Detail on decision
Novelty of OSG
The authors themselves note that the difference between OSG and OMD of Deskalakis et al is the inclusion of a projection step and the variance reduction by averaging multiple gradient evaluation at each iteration (which corresponds to choosing a different batch size for different iterations). I don't think these modifications are major enough to warrant "introduction of a new algorithm". The fact that all experiments seem to be conducted without constraints and constant batch size further strengthens this impression.
Relaxation to Minty VI
Another claimed improvement of the paper is relaxing the assumption of pseudomonotonicity to the mere assumption of existence of a variational inequality. However, if I'm not mistaken (please correct me if this assesment is incorrect), the only part where the pseudomonoonicity assumption enters the proof in Iusem et al is on page 36, to prove the last inequality of Equation (105). Here, however, the assumption of a Minty VI could equally be used. Thus, the weakened monotonicity assumption is not related to the use of SGO as opposed to extragradient, which is not apparent from the paper.
Suggestions for revision
I do think that the material can make for a solid paper, but I think it would strengthen rather than weaken the contribution to point out in more detail the close precursors of some of the results in the paper. Notably, rather than inventing a new algorithm (OSG), the paper proposes a way to combine variance reduction with Optimistic mirror descent of Deskalakis et al such as to achieve state of the art convergence rates which so far were only known for the variance reduced extragradient method. Furthermore, it points out that these convergence results (just as in the case of variance reduced extragradient) hold as soon as the associated Minty variational inequality holds true. This latter point would be much stronger, if examples were given that illustrate why this is a meaningful extension. Finally, the paper derives improved rates of an OAdagrad. In my opinion this point should be made more prominent by deemphasizing the part on OSG.
There is a typo in the definition of pseudomonotonicity.
Questions for authors
(1) Does figure 1 show iterations or epochs on the x axis? In order to support the theoretical claims, wouldn't it need to show the epochs on the x-axis? Otherwise, a larger batch-size simply corresponds to having access to more calls of the stochastic gradient oracle.
(2) Is there a new technical difficulty to overcome when replacing the extragradient method with the OSG compared to the proof of Iusem et al? If yes, can you give a concise description of it?
(3)It is not "evident from the definition" to me, why pseudo-monotonicity implies the existence of a solution of the Minty variational inequality, even though I believe that this is true. Could the authors explain this to me?
(4) The main improvement of OSG over extragradient lies in it needing to compute half the number of gradients per iteration. Is it possible to  explicitly compare the constants in the bounds on the convergence rates of the two algorithms? It seems that this would be required to truly make the case of a reduced complexity of OSG.


Word-embeddings are a vital component of Natural Language Processing (NLP) systems and have been extensively researched. Better representations of words have come at the cost of huge memory footprints, which has made deploying NLP models on edge-devices challenging due to memory limitations. Compressing embedding matrices without sacrificing model performance is essential for successful commercial edge deployment. In this paper, we propose Distilled Embedding, an (input/output) embedding compression method based on low-rank matrix decomposition with an added non-linearity. First, we initialize the weights of our decomposition by learning to reconstruct the full word-embedding and then fine-tune on the downstream task employing knowledge distillation on the factorized embedding. We conduct extensive experimentation with various compression rates on machine translation, using different data-sets with a shared word-embedding matrix for both embedding and vocabulary projection matrices. We show that the proposed technique outperforms conventional low-rank matrix factorization, and other recently proposed word-embedding matrix compression methods.

This paper proposes a method for compressing embedding matrices of both encoder/decoder embeddings.
The basic idea of the proposed method is to reconstruct the embedding matrix by what they called the “funneling decomposition” method, whose parameter shape is identical to the SVD (low-rank matrix) decomposition with additional non-linear function.
Therefore, the idea itself is not so novel and innovative.
Moreover, their method requires the embedding matrix as the teacher signal for calculating the reconstruction loss.
We need to note that the memory requirement of the proposed method during training will increase.

One of the notable advantages of the proposed method is that their proposed method seems to successfully reduce the embedding matrix even if it shares the parameters with the output layer, which is a de-facto standard model architecture for NMT.
As pointed out by the authors, this seems to be the first success of reducing the embedding matrix with a tied embedding setting.
1,
The authors claim that “We demonstrate that at the same compression rate our method outperforms existing state-of-the-art methods.” at the end of the Introduction section.
However, according to Tables 1, 2, and 3, it seems that the performance gain is marginal compared with similar methods.
For example,
37.78 (proposed) <=> 37.78 (Shi & Yu (2018))
26.97 (proposed) <=> 26.75 (Chen et al. (2018)
and
 42.62 (proposed) <=> 42.37 (SVD with rank 64),
which are the at most 0.25 BLEU gain.
I believe that most of MT researchers hardly say that BLEU 0.25 difference is a significant improvement. Besides, the authors should perform a statistically significant test if they say “our method outperforms existing state-of-the-art methods.”

2
I am a bit confused about the following inconsistency;
The authors say that “Compressing embedding matrices without sacrificing model performance is essential for successful commercial edge deployment” in the abstract.
However, according to Table 1, the number of parameters for embeddings is 16.3M, which is only 27% of the total number of parameters in Transformer base.
By this fact, compressing embedding matrices seems not essential for successful commercial edge deployment.

In Table 6, it is explicitly unclear what is the difference between
“Funneling with Emb. Distillation”, “Funneling (with non-linearity),” and “Funneling (with retraining all weights).”
Please give us a more precise explanation.


Machine Learning (ML) solutions are nowadays distributed and are prone to various types of component failures, which can be encompassed in so-called Byzantine behavior. This paper introduces LIUBEI, a Byzantine-resilient ML algorithm that does not trust any individual component in the network (neither workers nor servers), nor does it induce additional communication rounds (on average), compared to standard non-Byzantine resilient algorithms. LIUBEI builds upon gradient aggregation rules (GARs) to tolerate a minority of Byzantine workers. Besides, LIUBEI replicates the parameter server on multiple machines instead of trusting it. We introduce a novel filtering mechanism that enables workers to filter out replies from Byzantine server replicas without requiring communication with all servers. Such a filtering mechanism is based on network synchrony, Lipschitz continuity of the loss function, and the GAR used to aggregate workers' gradients. We also introduce a protocol, scatter/gather, to bound drifts between models on correct servers with a small number of communication messages. We theoretically prove that LIUBEI achieves Byzantine resilience to both servers and workers and guarantees convergence. We build LIUBEI using TensorFlow, and we show that LIUBEI tolerates Byzantine behavior with an accuracy loss of around 5% and around 24% convergence overhead compared to vanilla TensorFlow. We moreover show that the throughput gain of LIUBEI compared to another state-of-the-art Byzantine-resilient ML algorithm (that assumes network asynchrony) is 70%.

The paper considers distributed stochastic gradient descent, where some (unknown) compute nodes may be unreliable. New heuristics for filtering out replies from unreliable servers are introduced alongside a new protocol that helps keeping nodes in sync.
In general, I miss a more clear indication of how the individual contributions are different from other methods. I am also missing more detailed ablation studies showing which of the new ideas contribute the most to efficient learning. As far as I can tell, the experiments do not really show an improvement over existing methods in this domain.
This is not my area of expertise, but I cannot recommend the paper for publication in its current form as
(a) it's not clear to me that the paper improves on existing methods, and
(b) it's not clear to me what the real novelty of the work is.
Post-rebuttal:
I acknowledge the response of the authors. They clarified some aspects for me, and the paper appears to have improved over the rebuttal period.
I did not change my rating, but I want to emphasize that this is only because my knowledge of this field is so limited. My rating is largely based on "gut feeling" rather than actual knowledge, and I won't argue against acceptance.


While deep neural networks have been shown to perform remarkably well in many machine learning tasks, labeling a large amount of supervised data is usually very costly to scale. Therefore, learning robust representations with unlabeled data is critical in relieving human effort and vital for many downstream applications. Recent advances in unsupervised and self-supervised learning approaches for visual data have benefited greatly from domain knowledge. Here we are interested in a more generic unsupervised learning framework that can be easily generalized to other domains. In this paper, we propose to learn data representations with a novel type of denoising autoencoder, where the noisy input data is generated by corrupting clean data in the gradient domain. This can be naturally generalized to span multiple scales with a Laplacian pyramid representation of the input data. In this way, the agent learns more robust representations that exploits the underlying data structures across multiple scales. Experiments on several visual benchmarks demonstrate that better representations can be learned with the proposed approach, compared to its counterpart with single-scale corruption. Furthermore, we also demonstrate that the learned representations perform well when transferring to other vision tasks.

Denoising auto-encoder (DAE)  is a representation learning framework proposes in [1], which uses noisy input to reconstruct the clean “repaired” input. In the conventional DAE, the noise is directly added to the input space. This paper introduces a novel type of noise, which corrupts the Laplacian pyramid representation, and uses it to train a DAE. The effect of the proposed perturbation is to allow larger scale and semantically meaningful corruption, which may help to learn more transferable representation. Several experiments are conducted showing the effectiveness of the method
1) On MNIST, LapDAE provides better reconstruction images
2) On Cifar-10, LapDAE provides better image retrieval results
3) On Imagenet, combining with the transformation technique in [2], LapDAE achieve state-of-the-art result
4) On Pascal VOC,  combining with the transformation technique, LapDAE achieve state-of-the-art result on transfer learning
Overall, I find the idea natural and simple (simplicity of an approach is a good quality to me). At the same time, I also find the contribution a bit limited. The following are further comments and questions.
* From the experiments, standard DAE are harder to train comparing to the proposed LapDAE. In my opinion, this suggests that the local noise are harder to remove comparing to the Laplacian noise. It would be interesting to perform the following experiment to further understand the difference between DAE and LapDAE: train a DAE and LapDAE then
a) Apply LapDAE to reconstruct an image where the random noise is added on the input space (as the standard DAE setting).
b) Apply DAE to reconstruct an image where the noise is added to the Laplacian pyramid representation.
* How does the corrupted set C been selected? Moreover, how does the selected layer in the Laplacian pyramid effects the performance? For example, is there a difference in terms of performance of the model when training on LPS4 versus LPS8?
* The transformation technique seems to be very helpful for the performance, is it possible to combine it with other benchmark methods like RotNet or Counting?
[1]  Vincent et al. 2010, Stacked denoising autoencoders: Learning useful representations in a deep network with a local denoising criterion
[2] Zhang et al. 2019, AET vs. AED: Unsupervised Representation Learning by Auto-Encoding Transformations rather than Data


We present a Function Feature Learning (FFL) method that can measure the similarity of non-convex neural networks. The function feature representation provides crucial insights into the understanding of the relations between different local solutions of identical neural networks. Unlike existing methods that use neuron activation vectors over a given dataset as neural network representation, FFL aligns weights of neural networks and projects them into a common function feature space by introducing a chain alignment rule. We investigate the function feature representation on Multi-Layer Perceptron (MLP), Convolutional Neural Network (CNN), and Recurrent Neural Network (RNN), finding that identical neural networks trained with different random initializations on different learning tasks by the Stochastic Gradient Descent (SGD) algorithm can be projected into different fixed points. This finding demonstrates the strong connection between different local solutions of identical neural networks and the equivalence of projected local solutions. With FFL, we also find that the semantics are often presented in a bottom-up way. Besides, FFL provides more insights into the structure of local solutions. Experiments on CIFAR-100, NameData, and tiny ImageNet datasets validate the effectiveness of the proposed method.

I first wanted to thank the authors for their proposed approach in this paper. The paper discusses an interesting idea for quantizing the similarity of neural networks, based on weight similarity. However, overall this assumption is based on the fact that similar layers within a particular architecture will learn similar semantics across different runs (and surprisingly authors add tasks to this as well, which I don't understand why this is the case).
Unfortunately, the paper is hard to read. It is not easy to understand the research questions, and experimental setup. Partly due to overloaded terms, such as “solution” being used for describing multiple different concepts across the paper (solution class, local solution classification, local solution retrieval, none of them particularly well defined in the paper). I highly recommend the authors to describe their findings in a more concrete manner. I did not see any attempts at giving *insights*, mostly numerical comparison.
My concerns with the methodology of the paper are as follows:
1. What are the findings of the paper? Permutation of the neural networks is certainly an area worth studying. However, in this paper, authors make the assumption that weights across the same layer (say layer #2) are somehow always going to learn similar values (except in permutation) across different runs of the model? Major clarification in this area is required.
2. I am not quite sure what the term solution class means, or what the authors want the reviewer to believe it means. Please elaborate. Also solution label? This terminology seems a bit obsolete and cumbersome, unless properly defined at the beginning of the paper.
3. Can you elaborate more on the goals of the experiments? Right now the outcome of the experiments are a bit vague based on lack of hypotheses.
4. Can you elaborate what the goal of local solution classification is? It is not clear if this is simply the classification accuracy of a trained model, or what is vaguely described in section 3.3
I will make a re-evaluation of the paper after the above questions are answered. Overall, I suggest a rewrite of the paper to make claims, experimental hypotheses and design more clear.


We introduce the Mutual Information Machine (MIM), an autoencoder framework for learning joint distributions over observations and latent states. The model formulation reflects two key design principles: 1) symmetry, to encourage the encoder and decoder to learn different factorizations of the same underlying distribution; and 2) mutual information, to encourage the learning of useful representations for downstream tasks. The objective comprises the Jensen-Shannon divergence between the encoding and decoding joint distributions, plus a mutual information regularizer. We show that this can be bounded by a tractable cross-entropy loss between the true model and a parameterized approximation, and relate this to maximum likelihood estimation and variational autoencoders. Experiments show that MIM is capable of learning a latent representation with high mutual information, and good unsupervised clustering, while providing NLL comparable to VAE (with a sufficiently expressive architecture).

The authors present the Mutual Information Machine (MIM), which is essentially a  latent variable model learned via Adversarially Learned Inference with regularizations that prefer learning representations with high mutual information. While the method is interesting, it does not seem to be well justified in representation learning, and the empirical results are not well-justified due to the use of weak baselines.
Pros:
The idea of formulating the L_MIM objective appears interesting, which uses the mixture of P_\theta and Q_\theta and minimizes an upper bound to H(x) + H(z) - I(x, z), so it is effectively minimizing a lower bound to I(x, z), under the proposed symmetric distribution.
Cons:
Symmetry argument:
	- Prior work on maximizing mutual information for latent variables focus on the MI as evaluated in the "encoder distribution", whereas in this paper the argument is focused on symmetry, i.e. the MI estimated is based on \mathcal{M}_s.
	- However, for practical purposes, it is unclear how this is superior than the "encoder distribution" paradigm. Eventually, we wish to use the z for some downstream tasks, which requires us to obtain z from some distribution (e.g. the encoder q(z|x)) and then run classification on this z.
	- Unless we sample z | x from the conditional distribution of \mathcal{M}_s as proposed by the authors, it is unclear why we need this symmetry in representation learning; it seems slower to sample from the MIM distribution though (see motivation for A-MIM).
	- The authors did not demonstrate results that compare A-MIM and MIM on the same architecture.
Failure to discuss certain related literature, such as Alemi et al. 2017 and Zhao et al. 2018. These are also "strongly related to VAEs" and related to mutual information objectives with latent variables.
	- Alemi et al. 2017 discussed this in the context of "rate discussion trade-off"
	- Zhao et al. 2018 proposed objectives to find specific trade-offs under constraints
In fact, Zhao et al. 2018 considered the MIs in both P_theta and Q_theta, and one should be able to obtain a "symmetric" MI by adding these two Mis uniformly (although again, I am not sure how this is useful for downstream applications on the model).
Using VAE as a baseline comparison.
	- It is well known that the VAE objective does not encourage maximizing mutual information, and various existing works have already been proposed to address this problem. Outperforming VAE in terms of MI is hence not very surprising.
	- Methods that encourage mutual information maximization under the latent variable generative modeling framework would be more suited for comparison. How does MIM compare with the methods introduced in Chen et al. 2016, Alemi et al. 2017 and Zhao et al. 2018?
	- In the context of ALI and BiGAN, one could also apply other approaches like MINE / InfoMAX to maximize MI. Again, there is no comparison with these approaches.
	- Alemi and Zhao both showed that there is a trade-off between consistency and mutual information under the same architecture (using their objectives). It seems that what MIM empirically does is also to sacrifice NLL for better MI, and not clear whether MIM gives comparable or better trade-offs.
The paper would be more convincing if the authors empirically compare with other sensible baselines and justify why we should use MIM as opposed to A-MIM in representation learning.
[relevant papers]
Chen et al. 2016 Lossy VAEs
Alemi et al. 2017 Fixing a broken ELBO
Zhao et al. 2018 A Lagrangian Perspective to Latent Variable Generative Models


Is it optimal to use the standard Gaussian prior in variational autoencoders? With Gaussian distributions, which are not weakly informative priors, variational autoencoders struggle to reconstruct the actual data. We provide numerical evidence that encourages using Student's t-distributions as default priors in variational autoencoders, and we challenge the usual setup for the variational autoencoder structure by comparing Gaussian and Student's t-distribution priors with different forms of the covariance matrix.

Summary:  This paper studies alternative priors for VAEs, comparing Normal distributions (diagonal and full covariance) against Student-t’s (diagonal and full covariance).   In particular, the paper is concerned with posterior collapse---i.e. posterior remains at the prior, limiting the model’s ability to reconstruct the data.  Experiments are performed on a synthetic 2D dataset, ‘Gaussian ovals,’ and on OMNIGLOT.  Results primarily take the form of visualizations of the reconstructed data and MSE / SSIM numbers.
Pros:  Systematic study and investigation of alternative priors for deep generative models is an under-studied area.  Moreover, heavy-tailed priors such as the student-t---while widely successful for robust regression---have not been explored as extensively for latent variable models, to the best of my knowledge.  This paper makes some steps towards solving these open problems.
Cons:  I have two primary critiques of the paper: (i) the experimental hypothesis is unclear, (ii) no engagement with the results of Mathieu et al. [ICML 2019], who also study Student-t priors for reconstruction (and disentanglement).
Regarding (i), the paper seems to be testing two hypotheses simultaneously: the effect of diagonal vs full covariance matrices and exponential-tailed vs heavy-tailed priors.  The latter seems more crucial for purposes of reconstruction according to Figure 3 (since the diagonal St-t has good reconstruction).  Yet a proper study of the effect of tails vs reconstruction would report results as the degree of freedom parameter is gradually increased (as this controls the tails directly).  No careful ablation study of this sort is performed.  For comparison, see Figure 2 of Mathieu et al. [ICML 2019].  Moreover, the text simply calls the student-t a “weakly informative prior,” but it needs to be much more specific about what characteristics of the student-t are crucial.  If all we require is something “weakly informative,” why aren’t alternatives like a diffuse Gaussian or uniform also considered?
Regarding (ii), Mathieu et al. [ICML 2019] also study priors formed by products of student-t marginals, but their work is not cited.  Mathieu et al. [ICML 2019] also show that, due to student-t’s not being rotationally invariant (unlike the diagonal Gaussian), they improve disentanglement with only a minor degradation of reconstruction.  As this work also studies reconstruction in student-t VAEs, it should include some discussion of Mathieu et al. [ICML 2019]’s results---if not direct engagement with their hypotheses.
Final Evaluation:  While I like the general motivation for this work, there are no clear experimental hypotheses being tested in the experiments.
Mathieu, E., Rainforth, T., Narayanaswamy, S. and Teh, Y.W., 2018. Disentangling Disentanglement in Variational Autoencoders. ICML 2019.


In a multi-agent competitive environment, we would expect an agent who can quickly adapt to environmental changes may have a higher probability to survive and beat other agents. In this paper, to discuss whether the adaptation capability can help a learning agent to improve its competitiveness in a multi-agent environment, we construct a simplified baseball game scenario to develop and evaluate the adaptation capability of learning agents. Our baseball game scenario is modeled as a two-player zero-sum stochastic game with only the final reward. We propose a modified Deep CFR algorithm to learn the strategies of agents in a half-inning game. We also propose a strategy adaptation mechanism that continuously updates strategies based on the anticipation of the opponent's strategy in the inference time. We form several teams, with different teams adopting the same adaptation mechanism but different initial strategies, trying to analyze (1) whether an adaptation mechanism can help in increasing the winning percentage and (2) what kind of initial strategies can help a team to get a higher winning percentage. The experimental results show that the winning percentage can be increased for the team with an initial strategy learned from the modified Deep CFR algorithm. Nevertheless, those teams with deterministic initial strategies actually become less competitive.

UPDATE: Thank you for the detailed response. I think the changes you have made to the paper have improved it, but there remains significant work to complete before the paper reaches its full potential. For example, Figure 3 is a useful additional insight but it provides no quantification of the variation between repeated runs nor a comparison to a suitable baseline approach. Similarly, the additional hyperparameter details in Appendix A are helpful to enable reproducability but without explaining how these values were chosen it is impossible to assess the rigor of the empirical evaluation.
--
This paper proposes modification to Deep CFR and introduce a simplified baseball environment to evaluate the modifications. The rigour of and detail in documenting the empirical evaluation is not currently of the standard I would expect for publication at ICLR. I will detail suggestions for improvement below, but my most pressing concern for discussion in the rebuttal is with regard to the 2nd conclusion - If the team has learnt a Nash equilibrium as initial strategy, then how can performance of either team be improved by only one team further adapting their policy as shown in Table 5 lower avg2 by the difference between No Adapt (0.61) and Guest Adapt (0.51)? If I have interpreted the authors results correctly, this demonstrates that the initial policy is not a Nash equilibrium.
Suggested Improvements:
In the related work section, the narrative of the paper would be clearer if the authors introduced why the meta-learning literature is being reviewed. I would also recommend weakening the claim that "instability of the training process" is "the critical issue in MARL" to just noting it is "an issue" and request the authors justify the claim that this "is particularly severe in a competitive environment". Why is this more of an issue in fully competitive environments than it is in general sum games?
In Section 3.1 the authors claim they need to switch to TD learning from tree search due to stochastic state transitions but there are forms of search that can accommodate stochastic transitions so this claim needs to be removed. Perhaps the authors can motivate this change in another way? This section also introduces 2 further simplifications of the domain (1) learning strategies for half-innings and applying them to the whole game; (2) agents knowing their opponents true action chosen and not the noisy observation (e.g. pitchers target location instead of actual pitched location) and (3) "both agents know each other's strategies"  - these should be included in Section 2.1 when the domain is described.
Section 3.1. closes by stating "In principle, their average strategies will gradually converge to the Nash-equilibrium strategy." This is very weakly argued, to make such a claim the authors should provide evidence that their environment and modifications to Deep CFR meet the requirements of the theory where this guarantee was proven.
Section 3.2. notes that the learning rate is "manually determined" but the precise methodology of tuning hyperparameters is not provided and no values of settings used for any hyperparameter are included in the paper. Without details of the methodolofy it is unclear if a rigorous empirical evaluation was performed and without the precise hyperparameter settings used the results are not reproducible.
Section 4.1. notes "It is very interesting to observe that the learned Nash-equilibrium strategy is actually quite similar to real-life baseball strategies." Setting aside the issue regarding whether a Nash equilibrium has been learnt, this is a subjective opinion not an rigorous empirical observation. The same applies to the comment "The simulation results are very similar to real-life baseball games" on page 9  Can you support these claims that the learnt strategy is similar to real-life by comparison to the data collected from the MLB Statcast?
In Section 4.2. there is further imprecision in the discussion of results. For example "as the strategy adaption mechanism is employed, the WP of most teams … actually decreases." As this mechanism is a core contribution of the paper, this evaluation needs to be more rigorous. Is there a statistically significant difference caused by using the proposed mechanism?
All results presented should include quantification of variation as well as average values and the number of repeats these averages are taken from should be clearly documented. The inclusion of a limited subset of teams in empirical evaluations (e.g. Table 4 only including teams 0,1,2,3 and 12 and Table 5 only including teams 5 and 13) should be clearly justified.
All references to papers published in conference proceedings or journals should cite the published version of the paper and not the arxiv version. All references should include the full publication venue and not abbreviations (e.g. Zhang and Lesser, 2010 is currently listed as just AAAI). References to online resources should include a note of the date accessed.
Minor Comments:
1) The word "purpose" is used frequently in place of "propose" (e.g. line 7 of the abstract)
2) Page 2: "plays the-best-of-three game" -> plays the best-of-three games
3) Page 2: "that worth further study" -> that are worth further study
4) Page 3: "To simply the game" -> To simplify the game
5) Page 3: "listed in C" -> listed in Appendix C
6) Page 8 (and recurring): Initial speech marks are the wrong way around, this looks like a Latex error.
7) Page 10: "we would relief" -> we would relax
8) Page 10: "in the real life" -> in real life


Deep neural networks have had unprecedented success in computer vision, natural language processing, and speech largely due to the ability to search for suitable task algorithms via differentiable programming. In this paper, we borrow ideas from Kolmogorov complexity theory and normalizing flows to explore the possibilities of finding arbitrary algorithms that represent data. In particular, algorithms which encode sequences of video image frames. Ultimately, we demonstrate neural video encoded using convolutional neural networks to transform autoregressive noise processes and show that this method has surprising cryptographic analogs for information security.
In algorithmic information theory, the Kolmogorov complexity of an object is the length of the shortest computer program that, without additional input, produces the object as output (Ming & Paul, 2008). For example, a long sequence of ones contains very little information because a short program for i ← 0, n do print 1 can output the data. Similarly, the transcendental number π = 3.1415 . . . is an infinite sequence of seemingly random decimal digits but contains only a small amount of information since there exists a very short program that produces the consecutive digits of π forever (Bailey et al., 1997). While this idea of exchanging data with computation is theoretically intriguing, from a practical point of view, explicit specification of instructions to obtain some target representation is generally an intractable approach for arbitrary data.
Recently, however, artificial intelligence had broad success in computer vision, natural language processing, and speech largely due to the ability to search for suitable task algorithms using differentiable programming together with large datasets to obtain abstract computations in the form of neural networks. In this paper, we turn the process around and leverage this same differentiable programming approach to find abstract computations which represent data. That is, we seek algorithms which encode arbitrary information for storage and retrieval. In particular, we demonstrate methods for the encoding of sequential video data within the weights of convolutional neural networks.
In the simplest formulation, video data, X, is treated as an finite ordered sequence of image frames
and we seek an autoregressive computation of the form
where y 0 ← x 0 and f is some convolutional neural network parametrized by weights θ such that y i ≈ x i . It is possible that f can be trained using a self-supervised learning approach where consecutive image frames form training examples and the sum-of-squares, y i − x i 2 , is minimized. That is, during training, f learns a mapping from x i to x i+1 , or rather, learns to advance the current frame by one step. The idea is somewhat similar to conventional autoencoders (Hinton & Zemel, 1993), but there, f is trained with the identity mapping, from x i on to x i . The catch here is that after training, during recall, f does not receive x i as input but instead y i from the previous evaluation. Since y i only approximates x i , reconstruction error accumulates at each iteration of eq. 2 until sequence reproduction fails 1 .
The natural inclination at this point is to extend the approach to train over the full sequence by evaluating eq. 2 N times to produce {y 1 , . . . , y N } and minimizing the mean squared error over the entire sequence at each training iteration. This way, how the network is trained and how the sequence is recalled post-training are congruent. Unfortunately, for sequences beyond a few elements, this is not tractable as long unrolled gradients readily vanish or explode and there is no opportunity for batching which significantly limits training performance and scalability.
The situation can be salvaged by adopting a curriculum learning strategy (Bengio et al., 2009) whereby the training sequence length is slowly increased after achieving some nominal loss objective over shorter intervals. It is important to note that, because CNNs have no recurrent hidden state, this approach can be successfully batched during training over random subsequences of X. Therefore, modulating the training subsequence length to one yields the aforementioned self-supervised learning approach whereas a training sequence length equal to the number of frame elements yields the full sequence loss. While this batched curriculum learning strategy does mitigate challenges associated with teacher forcing, gradients can still be temperamental for larger training sequence lengths necessitating the use of gradient clipping and learning rate scaling (You et al., 2017;).
An interesting situation occurs when
That is, there exists a nonunique mapping or sequence ambiguity which is not resolvable from a purely supervised learning point of view since a unique input evaluates to multiple output. It is perhaps surprising to some readers that eq. 2 above is capable of handling this situation successfully, despite CNNs having no recurrent hidden state. Due to the lack of teacher forcing and high dimensionality of the data, the network learns to encode (and decode) state information within the pixels thus willfully incurring a local reconstruction penalty in order to facilitate sequence completion. The success of the batched sequence learning strategy depends on the nature of the ambiguity.
Just as the state of a dynamical system is not well defined by a single observation, often encoding is facilitated by sequence dilation of the learned mapping which more uniquely defines the trajectory of sequence evolution. A single frame mapping from x i to x i+1 exhibits a dilation of d = 1 and can be naturally extended to d = 2 by channel concatenation to learn a computation of the form [y i ;
with dimension (2C) × H × W ; and so on. 2 This helps to individuate samples and thereby allay the need for contextual sequence information to achieve recall. For example, the alternating sequence ambiguity in {A, B, A, C, A, D} is easily resolved by dilation d = 2 since all training examples, f (AB) → BA, f (BA) → AC and so on, are unique.
The KTH (Schüldt et al., 2004) and DAVIS17 (Pont-Tuset et al., 2017) datasets are used to investigate the methods described above. KTH data provide a simple initial investigative platform with monochromatic videos featuring simple repetitious actions performed by humans (clapping, hand waving, punching, etc) against a static background while the DAVIS17 data features multi-channel RGB capture of generic action scenes across a variety of dynamic environments (surfing, breakdancing, paragliding, etc). See Figure 1 and Figure 2 for encoding examples using a simple autoencoder style network defined in (1) and applied according to eq. (2). Although compression and fidelity are peripheral considerations (see Wu et al. (2018) for full treatment), the examples presented generally yield faithful reproductions according to a multi-scale similarly measure (Wang et al., 2003) and achieve respectable compression ratios of 10x -15x simply by modulating the number of convolutional layers in the CNN. Compression ratios can be further improved by applying mixed precision training techniques (Micikevicius et al., 2017).
A variety of practical engineering considerations arise during implementation. For example, it is possible to analyze sequence loss during post-production and inject a ground truth frame at a particular deterioration threshold to renew the recall process; analogous to the use of key-frames in standard compression algorithms. It is also possible to train additional networks which upscale the base reproduction. This upscaling approach has advantages in controlling network parameter growth (and hence the compression ratio) as two convolution layers each with 32 filters have only half the parameters compared to a single layer of 64 filters. In this way, eq. 2 is written as the composition of two functions
where f θ advances the sequence and the subsequent application of f θ * bolsters fidelity. From this point of view, f θ * performs the role of denoising autoencoder (Vincent et al., 2008) which strives to  Algorithm 1 Pseudo definition of the autoencoder style network used throughout this work and applied as defined in eq. (2). All layers define nconv convolutional filter channels of size 3 × 3. The first and final layers takes C × d number of input and output channels respectively where C is the number of image channels and d is the sequence dilation. Where necessary, stride and padding are used to maintain dimensional equality of input and output tensors. form a 'repaired' output from 'corrupted' input. This allows computational demands during recall to be modulated on a frame-by-frame basis according to platform limitations, real-time constraints, and so on. This application of upscaling is shown in Figure 3.
From here, we can generalize further and leverage the recent advancements in normalizing flows whereby a simple initial density is transformed into a more complex one by applying a sequence of transformations until a desired level of complexity is attained (Tabak & Turner, 2013;Dinh et al., 2014;Jimenez Rezende & Mohamed, 2015;Kingma et al., 2016). The density q K (z) obtained by successively transforming a random variable z 0 with distribution q 0 through a chain of K transformations f k is:
where eq. (4) is shorthand for the composition f M (f M −1 (. . . f 1 (z 0 ))). In this way, the initial density is said to 'flow' through the sequence of mappings. For our purposes, the original sequence X defined in (1) can be seen as just an arbitrary random noise process in some high-dimensional space. Rather than model the process X directly, instead, it can be approximated, to some acceptable level of fidelity, through a chain of transformations over some known latent process, Z = {z 0 , . . . , z N }. That is, given a random process, Z, defined by g, we have
where z j i ≈ x i represents j consecutive transformations applied to z i , and where each f j is a deep neural network defined by Algorithm 1. These transformations are simply a series of bijective mappings (under the given restricted domain and range) and therefore alleviate the aforementioned sequence training challenges associated with realizing computations of the form defined by eq. (2).

Summary:
This paper introduces convolutional neural networks to encode video sequences. In particular, it exploits an autoencoder style convolutional network to compress videos with certain ratios by modulating the number of convolution layers. Furthermore, it studies an autoregressive model to learn the sequential information over videos.
Strength:
The idea of using convolutional networks to encode the videos is interesting.
The used autoencoder and autoregressive techniques are promising for video encoding.
Weakness:
The paper gives me an impression that there are very few works to apply convolutional networks (including autoencoder with autoregressive processing) to encode videos. I cannot see any existing works sharing the same motivation, and the paper does not evaluate any related works for comparison. But it seems the use of autoencoder and autoregressive techniques look very straightforward to me. Please clarify the concern.
About technical contribution, it is not clear to see if there are any improvement over the conventional autoencoder and autoregressive models. It would be better to make some necessary discussions in the paper. In addition, it seems no descriptions showing how to learn the bijective mappings f_1, ..f_j.
Regarding the evaluation, the paper uses visual results and multi-scale SSIM (mssim) to study the effectiveness of the proposed method on the KTH and DAVIS’17 datasets. While a small number of video frames are used, it would be necessary to evaluate the overall (average) mssim on the whole dataset.
There are quite a few typos or grammatical problems on sentences such as “...algorithms which encode sequences of video image frames.”, “f is some convolutional neural network…”, “analogous to the use of key-frames in standard compression algorithms.


We present DiffSim, a new differentiable programming language tailored for building high-performance differentiable physical simulators. Based on an imperative programming language, DiffSim generates gradients of simulation steps using source code transformations that preserve arithmetic intensity and parallelism. A light-weight tape is used to record the whole simulation program structure and replay the gradient kernels in a reversed order, for end-to-end backpropagation. We demonstrate the performance and productivity of our language in gradientbased learning and optimization tasks on 10 different physical simulators. For example, a differentiable elastic object simulator written in our language is 4.2× shorter than the hand-engineered CUDA version yet runs as fast, and is 188× faster than the TensorFlow implementation. Using our differentiable programs, neural network controllers are typically optimized within only tens of iterations. Figure 1: Left: Our language allows us to seamlessly integrate a neural network (NN) controller and a physical simulation module, and update the weights of the controller or the initial state parameterization (blue). Our simulations typically have 512 ∼ 2048 time steps, and each time step has up to one thousand parallel operations. Right: 10 differentiable simulators built with DiffSim.

This paper introduces DiffSim, a programming language for high-performance differentiable physics simulations. The paper demonstrates 10 different simulations with controller optimization. It shows that the proposed language is easier to use and faster than the other alternatives, such as CUDA and TensorFlow. At the end, the paper provides insightful discussions why the gradient of the simulation could be wrong.
Differentiable physics simulation is an important research area, especially for optimal control and reinforcement learning. While I am impressed by the large variety of examples demonstrated in the paper, I am leaning towards rejecting the paper because of its poor presentation. The paper only gives a simple and high-level example of the language (optimizing the rest length of springs that form a triangle), very brief descriptions of 10 examples and some discussions about the difficulty of computing useful gradients, but without any in-depth discussion how everything is implemented. This is not enough for an ICLR paper. For example, the paper does not answer some of the fundamental problems of differentiable physics. For example, collision and contact are inherently non-differentiable. How does the paper handle it in the examples of locomotion and billiards (Figure 4)? In addition, how does the paper back-propagate the gradient through the incompressibility conditions (Poisson solve) of fluid simulation?
Here is my suggestions how to improve the writing. There are several ways to write the paper, with different emphasis. If this paper is more about introducing a new programming language, Appendix B Compiler Design and Implementation would be important and should be moved to main text. If the paper want to emphasize how to handle the non-differentiable cases of the simulation, then detailed derivations of contact, collision, and linear/nonlinear solving (due to incompressibility conditions or implicit integrators) should be presented. If the paper would like to demonstrate how differentiable physics simulation can help with controller optimization, then two to three examples, such as the locomotion control for soft bodies or rigid bodies, should be analyzed in far more details, and compared with traditional method without differentiable simulation. It is good to focus on one of the above points, based on the venue that this paper is submitted to. Currently, the paper is trying to touch all three. But due to the page limit, it is not thorough, or detailed in any one of them.
-------------------------Update after rebuttal------------------------------
Thank you for the revision of the paper and the additional comparisons with Jax. The revised version reads much better. The response and the revision addressed most of my concerns. Thus, I raised my rating to weak accept.


MixUp is a data augmentation scheme in which pairs of training samples and their corresponding labels are mixed using linear coefficients. Without label mixing, MixUp becomes a more conventional scheme: input samples are moved but their original labels are retained. Because samples are preferentially moved in the direction of other classes we refer to this method as directional adversarial training, or DAT. We show that under two mild conditions, MixUp asymptotically convergences to a subset of DAT. We define untied MixUp (UMixUp), a superset of MixUp wherein training labels are mixed with different linear coefficients to those of their corresponding samples. We show that under the same mild conditions, untied MixUp converges to the entire class of DAT schemes. Motivated by the understanding that UMixUp is both a generalization of MixUp and a form of adversarial training, we experiment with different datasets and loss functions to show that UMixUp provides improved performance over MixUp. In short, we present a novel interpretation of MixUp as belonging to a class highly analogous to adversarial training, and on this basis we introduce a simple generalization which outperforms MixUp.

This paper proposes a novel data augmentation method, untied MixUp (UMixUp), which is a general case of both MixUp and Directional Adversarial Traning (DAT). DAT is referred to in this paper as a scheme that only input feature vectors are mixed, while MixUp also incorporates their corresponding labels. The authors provide a theoretical discussion that both DAT and UMixUp converges to be equivalent to each other when the number of training samples becomes infinity. Experimental results on Cifar 10, Cifar 100, MNIST, and Fashion MNIST show quantitative comparisons among the baseline, MixUp, and UMixUp.
According to the author guideline,
> There will be a strict upper limit of 10 pages for the main text. Reviewers will be instructed to apply a higher standard to papers in excess of 8 pages.
The authors use nine pages. Therefore, the review should be more careful about its quality.
Currently, I have three major concerns that keep me from judging this paper acceptable in ICLR 2020.
First, the authors failed to cite two closely related papers below:
- Tokozume et al., LEARNING FROM BETWEEN-CLASS EXAMPLES FOR DEEP SOUND RECOGNITION. ICLR, 2018.
- Tokozume et al., Between-class Learning for Image Classification. CVPR, 2018.
The first one is published in the previous ICLR and mixing two samples belonging to different classes. The second one is an application to image classification using ImageNet dataset, which is larger than the dataset used in this paper.  What's more important is that both papers propose that the mixing ratio of two samples is not linearly but depending on the strength of their signals. Since UMixUp is also focusing on the mixing ratio between two training samples, Between-Class Learning should have been compared to the proposed method.
Secondly, the theoretical discussion is not so fascinating. Actually, both MixUp and UMixUp are shown to converge to DAT when the number of training samples tends to infinity. Data augmentation is, however, performed to remedy the lack of training samples in general. The discussion that the number of training samples is assumed to be large is the opposite situation.
Thirdly, the experimental results show that the performance gain by UMixUp is relatively small in comparison to that of the original MixUp. There are no ablation studies using different values for alpha and beta, which are parameters for the policy of UMixUp. The authors reported that these values are defined using a heuristic search. Thus, we cannot see if the performance is sensitive to the parameter selection.
I lean to reject this paper because of these concerns. I'm looking forward to seeing the revised version in another conference.


In real-world underwater environment, exploration of seabed resources, underwater archaeology, and underwater fishing rely on a variety of sensors, vision sensor is the most important one due to its high information content, non-intrusive, and passive nature. However, wavelength-dependent light attenuation and back-scattering result in color distortion and haze effect, which degrade the visibility of images. To address this problem, firstly, we proposed an unsupervised generative adversarial network (GAN) for generating realistic underwater images (color distortion and haze effect) from in-air image and depth map pairs based on improved underwater imaging model. Secondly, U-Net, which is trained efficiently using synthetic underwater dataset, is adopted for color restoration and dehazing. Our model directly reconstructs underwater clear images using end-to-end autoencoder networks, while maintaining scene content structural similarity. The results obtained by our method were compared with existing methods qualitatively and quantitatively. Experimental results obtained by the proposed model demonstrate well performance on open real-world underwater datasets, and the processing speed can reach up to 125FPS running on one NVIDIA 1060 GPU.

In this article, the authors propose a generative adversarial network named UWGAN to generate realistic underwater images from the pairs of in-air images and depth images. Then, a U-Net was leveraged to enhance the results.
However, the text suffers from too many language problems. The authors should consult professional proofreading services. As a courtesy towards referees, the quality of writing needs meticulous attention before a scientific paper should be submitted.
	Other comments:
1.	The literature is limited. I found some novel works being done in the field that must be addressed and listed in the background and experiments.
2.	The underwater imaging model presented in this paper derives from the Jaffe-McGlamery model, which is a common sense in this field. The authors use a generator to produce underwater images that only implements the common model by a neural network. Moreover, the statement of section 2.2 is not clear. Please rewrite this section.
3.	The authors used U-Net without any improvement to enhance the results generated from UWGAN, which is the integration of existing models.
4.	The authors claimed that their model is better than others, while there is no evidence to indicates that. For example, 1) in (page 5, line 4 from bottom), “It can be seen that our proposed method has achieved a higher score.”, can we observe this from the Table 1 and 2? 2) “The method we proposed has the fastest processing speed compared to other methods. Moreover, the method proposed in this paper has the fewest parameters compared to other deep-learning-based methods.”, it is suggested that a study about the parameters and FLOPs of the involved methods should be given.
5.	Please carefully check the references. For example, “Hummel R. Image enhancement by histogram transformation[J]. Unknown, 1975.” lacks the journal name.
6.	High-resolution figures should be given in the manuscript.


Storage assignment, the act of choosing what goods are placed in what locations in a warehouse, is a central problem of supply chain logistics. Past literature has shown that the optimal method to assign pallets is to arrange them in increasing duration of stay (DoS) in the warehouse (the DoS method), but the methodology requires perfect prior knowledge of DoS for each pallet, which is unknown and uncertain under realistic conditions. Attempts to predict DoS have largely been unfruitful due to the multi-valuedness nature (every shipment contains identical pallets with different DoS) and data sparsity induced by lack of matching historical conditions. In this paper, we introduce a new framework for storage assignment that provides a solution to the DoS prediction problem through a distributional reformulation and a novel neural network, ParallelNet. Through collaboration with a world-leading cold storage company, we show that the system is able to predict DoS with a MAPE of 29%, a decrease of ∼30% compared to a CNN-LSTM model, and suffers less performance decay into the future. The framework is then integrated into a first-of-its-kind Storage Assignment system, which is being deployed in warehouses across United States, with initial results showing up to 21% in labor savings. We also release the first publicly available set of warehousing records to facilitate research into this central problem.

The paper tackles the problem of assigning pallets to storage positions in a warehouse so as to minimize total travel time. Previous literature has shown that if durations of stay (DoS) is known in advance then the optimal storage position is inversely proportional to this. The paper proposes to predict a DoS distribution for each pallet using a 19 point CDF with a neural network. The authors show that the proposed network is better than a few other neural baselines and that it can predict the DoS reasonably well.
The authors report significant positive real life results in two warehouses and releases their dataset.
The paper is interesting and seems to make significant progress on an important real-life issue. The release of a large realistic dataset is a major contribution to this field.
Despite this I will not recommend this paper for publishing at ICLR, simply because I think it falls outside the scope of the conference.
I would recommend the authors seek to publish in a venue more focused on logistics or operational research (OR), where I think the paper could have a great impact.
I would have appreciated a more in-depth description of the problem. It seems that the general problem would be to minimize the expected cost (distance moved) over the lifetime of the warehouse. Under what simplifying assumptions does that reduce to minimizing the expected cost of each pallet individually as done here?
Similarly, I would have preferred a more formal definition of the loss, e.g. start from the expected distance moved and derive the (approximate) loss from there. Approximations are fine as long as you tell the reader what you're approximating, why you need to do it, and what the biases are.
I would also move the detailed description and figures of the actual neural network to the appendix since it's not central to the understanding of the paper (aside from the CDF formulation).
------------ UPDATE ------------
The area chair didn't get back to me, but I'll assume the paper is in scope and as such have changed my rating to weak reject.
I'd prefer if the paper spend less space arguing that a specific network architecture is the right choice and prioritize introducing the problem more formally. I would move the simulations of different storage strategies (figure 2, blue and black lines) into the introduction, as part of introducing the reader to the problem and evaluation, and explain how figure 2 is related to what you really want to minimize: "total pallet distance moved over time x". I wonder if the pallet percentile distribution (figure 2) is actually interesting, other than as a way to compute the "total pallet distance moved over time x".
I fear the authors feel the need to introduce a "novel" NN in order to publish at ICLR and thus spend a lot of effort describing something that is really not very interesting (even giving it a name). Combining textual and non-textual features is not a new concept, and combining a RNN layer and a CNN layer is not novel. I'm skeptical that this exact network structure is needed. It's clear you should combine the textual and non-textual features, but is this *exact* network structure needed? I would find that very surprising. How about MLP(concat(CNN(text), RNN(text), non-text))?
Another gripe is the heuristic nature of the loss function. It's clear predicting the DoS distribution better is the right thing to do, but is the sum of L2 distances to the emprical log CDF the *right* measure or a heuristic? What is the right thing to do? What is it approximating if anything? Why?
I would expect the right thing to do is to maximize the probability of the observed duration of stay under the model, i.e max_\theta p(DoS|x, \theta), i.e. min_\theta \sum_i -log(p(DoS_i|x_i, \theta). If this is log-normally distributed, then a log-normal distribution is probably a good distribution. Or perhaps a mixture of log-normals. I think you can even use the 19 point CDF output layer and evaluate the data likelihood under this, but I *don't* think the 19 point L2 distance to the empirical probability distributions is the correct thing to do. Is that even defined if there's less than 19 pallets with the same inputs x?


Title
Active learning (AL) integrates data labeling and model training to minimize the labeling cost by prioritizing the selection of high value data that can best improve model performance. Readily-available unlabeled data are used for selection mechanisms, but are not used for model training in most conventional pool-based AL methods. To minimize the labeling cost, we unify unlabeled sample selection and model training based on two principles. First, we exploit both labeled and unlabeled data using semi-supervised learning (SSL) to distill information from unlabeled data that improves representation learning and sample selection. Second, we propose a simple yet effective selection metric that is coherent with the training objective such that the selected samples are effective at improving model performance. Experimental results demonstrate superior performance of our proposed principles for limited labeled data compared to alternative AL and SSL combinations. In addition, we study an important problem -"When can we start AL?". We propose a measure that is empirically correlated with the AL target loss and can be used to assist in determining the proper start point.

This paper proposes a semi-supervised active learning method to reduce the labeling cost. In the proposed method, a selection criterion to better integrate AL selection mechanism in SSL training framework is designed. The simple metric that aims to measure the inconsistency across a certain number of meaningful perturbations. It considers N perturbed samples of the original input data x, which can be obtained by standard augmentation operations (e.g. random crops and horizontal flips for image data). Then the variance is adopted to quantify consistency.  In this way, the proposed method prefers data samples with high values, which may possess varying level of difficulty for the model to classify. To verify the effectiveness of the proposed method, several baseline methods are compared on several benchmark data sets, and the proposed method has achieved better performance. Meanwhile, to deal with the “cold start” problem, a measure that is found to be empirically correlated with the AL target loss is proposed, and this measure can be used to assist in determining the proper start size. However, there are some minor concerns:
[1] The consistency of a sample is measured based on the perturbed samples. How to generate these perturbed samples may have a great influence on the query results. In the paper, it said that these samples are generated by standard augmentation operations (e.g. random crops and horizontal flips for image data). This representation is hard to follow in the experiments. If possible, it is better to show in details.
[2] In the uncertainty of active learning, the samples are selected from different distributions in the unlabeled data, for example, the marginal sampling selects the samples around the classification hyperlanes (Settles, Burr. "Active learning." Synthesis Lectures on Artificial Intelligence and Machine Learning 6.1 (2012): 1-114.). Can you show which samples may be selected in the unlabeled data. In this way, the proposed criterion can be followed more easily.
[3] In the experiments, whether the proposed method can select a batch of samples at each iteration. How about the influence of the batch size.


Meta-learning algorithms use past experience to learn to quickly solve new tasks. In the context of reinforcement learning, meta-learning algorithms acquire reinforcement learning procedures to solve new problems more efficiently by utilizing experience from prior tasks. The performance of meta-learning algorithms depends on the tasks available for meta-training: in the same way that supervised learning generalizes best to test points drawn from the same distribution as the training points, meta-learning methods generalize best to tasks from the same distribution as the meta-training tasks. In effect, meta-reinforcement learning offloads the design burden from algorithm design to task design. If we can automate the process of task design as well, we can devise a meta-learning algorithm that is truly automated. In this work, we take a step in this direction, proposing a family of unsupervised meta-learning algorithms for reinforcement learning. We motivate and describe a general recipe for unsupervised meta-reinforcement learning, and present an instantiation of this approach. Our conceptual and theoretical contributions consist of formulating the unsupervised meta-reinforcement learning problem and describing how task proposals based on mutual information can be used to train optimal metalearners. Our experimental results indicate that unsupervised meta-reinforcement learning effectively acquires accelerated reinforcement learning procedures without the need for manual task design and these procedures exceed the performance of learning from scratch.

Summary: this paper claims to design an unsupervised meta-learning algorithm that does automatically design a task distribution for the target task. The conceptual idea is to propose a task based on mutual information and to train the optimal meta-learner. They also use experiments to show the effectiveness of the proposed approach.
Overall Comments:
I would think this paper requires a major revision. It is written in a very confusing way. Many terms are directly used without a definition. The problem is also not clearly defined. I have tried to understand everything, but I have to give up in Section 3. Overall, I do not think this paper is ready for publication.
Detailed comments:
	• It would benefit a lot if you can clearly define the original meta-learning procedure and then compare that with the one proposed in this paper.
	• Define ”hand-specified” distribution. This word does not make sense if you claim this is the difference between the meta-learning procedure proposed in this paper and the original meta-learning algorithm. In this paper, you used p(z) to specify a task. I would think p(z) is also “hand-specified”.
	• I am not very sure by what you mean for “task-proposal procedure”, “goal-proposal procedure”
	• In the first paragraph of the intro: what do you mean by “specifying a task distribution is tedious”, is specifying p(z) also “tedious”
	• 2nd paragraph of intro: “automate the meta-training process by removing the need for hand-designed meta-training tasks”. Again, why p(z) is not “hand-designed”
	• Why compare with the original meta-RL algorithm on p(z) is not fair?
	• What do you mean by “acquire reinforcement learning procedures”?
	• “Environment”, “task” are not clear when they first appear
	• The word “learn” is used everywhere, and is confusing. E.g. what do you mean by “learn new tasks”, “learn a learning algorithm f”, “learn an optimal policy”, “learn a task distribution” …
	• “Reward functions induced by p(z) and r_z(s,a)”: isn’t r_z(s,a) already a reward function? What is “induced”?
	• What is “meta-training” time?
	• What is “no free lunch theorem”?
	• The “controlled-MDP” setting is actually much easier: perhaps you just need to learn the probability distribution. Then for every r_z, we just solve it. Why not compare with this simple algorithm?
	• “Regret” is not defined when it first appears
	• “The task distribution is defined by a latent variable z and a reward function r_z”: why “distribution” is defined by an r.v.?
	• In (2), “regret” should be the (cost of the algorithm) - (the total cost of an optimal policy) — it is not hitting time
	• (3) is confusing, no derivation is given
	• Based on the usual definition of “regret”, how can a “policy” have low regret? Any fixed “policy” would have linear regret …