My title
The practical usage of reinforcement learning agents is often bottlenecked by the duration of training time. To accelerate training, practitioners often turn to distributed reinforcement learning architectures to parallelize and accelerate the training process. However, modern methods for scalable reinforcement learning (RL) often tradeoff between the throughput of samples that an RL agent can learn from (sample throughput) and the quality of learning from each sample (sample efficiency). In these scalable RL architectures, as one increases sample throughput (i.e. increasing parallelization in IMPALA (Espeholt et al., 2018)), sample efficiency drops significantly. To address this, we propose a new distributed reinforcement learning algorithm, IMPACT. IMPACT extends IMPALA with three changes: a target network for stabilizing the surrogate objective, a circular buffer, and truncated importance sampling. In discrete action-space environments, we show that IMPACT attains higher reward and, simultaneously, achieves up to 30% decrease in training wall-time than that of IMPALA. For continuous control environments, IMPACT trains faster than existing scalable agents while preserving the sample efficiency of synchronous PPO.

Reinforcement learning (RL) training speed is broadly evaluated on two dimensions:  sample efficiency (the number of environment interactions required) and wall-clock time.  Improved wall-clock training time has been achieved through distributed actors and learners, but often at the expense of sample efficiency.  IMPACT repurposes successful concepts from deep RL - the target network, importance sampling and a replay buffer to demonstrate improvements on both axes in on three continuous environments and three games from the Atari Learning Environment.
Positives
This was a well-written paper proposing to address the sample efficiency of distributed RL algorithms.  The diagrams of the algorithm were also well-done.  Improving the sample efficiency of algorithms is an important objective and the approaches followed here are sensible.
Additionally, the ablations and examination of the sensitive hyperparameters of the algorithm are useful analyses.  These indicate relative insensitivity to the target network update frequency, but both the importance sampling equation and the circular buffer hyperparameters are described.
Negatives
IMPACT introduces additional hyperparameters which are tuned for each continuous control task and discrete control task.  However, there is no description of the hyperparameter tuning budget allocated to IMPACT, PPO, IMPALA.
Regarding the discrete environment, the game selection should be elaborated upon and if sufficient compute is available, the algorithm should be tested elsewhere.  Specifically, it is atypical (though not necessarily incorrect) to tune specific hyperparameters for each game in the Atari Learning Environment.  Traditionally, algorithms have been justified as robust and useful by the lack of need to tune per game.  Table 4 demonstrates a high degree of tuning for IMPACT due to game-specific changes for clip param, grad clip, lambda, num sgd iter, train_batch_size, value function loss coeff, kl coeff.  However, Table 5 (IMPALA) and Table 6 (PPO) have fewer noted changes.
Small nits:
- Define advantage in the policy gradient equation
- Figure 4 is ahead of Figure 3 in the compiled LaTeX
Questions
- How were the discrete control games selected?
- What was the hyperparameter tuning budget for IMPACT versus PPO or IMPALA?
- If a fixed hyperparameter budget is allocated in advance and new environments are randomly selected, does IMPACT favorably compare to IMPALA and PPO?
- IMPALA performs remarkably badly in the three continuous control tasks, even on wall-clock time.  What validations have been done here to ensure the algorithm is operating as intended?
I will increase my rating if the robustness and improvements of this algorithm can be validated in randomly chosen games/continuous control environments for a fixed hyperparameter budget for IMPACT and both baselines.  Also, the IMPALA baseline should be validated for the continuous control tasks - it's surprising that this once SOTA-algorithm flounders in even simple tasks like Hopper-v2 or HalfCheetah-v2.
-----------------
Update:
The authors have addressed my initial concerns carefully through extra experiments and details in the Appendix and I have updated my rating accordingly.  Thanks!


Though word embeddings and topics are complementary representations, several past works have only used pretrained word embeddings in (neural) topic modeling to address data sparsity problem in short text or small collection of documents. However, no prior work has employed (pretrained latent) topics in transfer learning paradigm. In this paper, we propose a framework to perform transfer learning in neural topic modeling using (1) pretrained (latent) topics obtained from a large source corpus, and (2) pretrained word and topic embeddings jointly (i.e., multiview) in order to improve topic quality, better deal with polysemy and data sparsity issues in a target corpus. In doing so, we first accumulate topics and word representations from one or many source corpora to build respective pools of pretrained topic (i.e., TopicPool) and word embeddings (i.e., WordPool). Then, we identify one or multiple relevant source domain(s) and take advantage of corresponding topics and word features via the respective pools to guide meaningful learning in the sparse target domain. We quantify the quality of topic and document representations via generalization (perplexity), interpretability (topic coherence) and information retrieval (IR) using short-text, long-text, small and large document collections from news and medical domains. We have demonstrated the state-ofthe-art results on topic modeling with the proposed transfer learning approaches.

On the basis of existing topic modelling approaches, the authors apply a transfer learning approach to incorporate additional knowledge to topic models, using both word embeddings and topic models. The underlying idea is that topic models contain a global view that differs on a thematic level, while word embeddings contain a local, immediate contextual view. The combination of both local and global view transfer to enhance a topic model is the main contribution of this paper, especially when using multiple sources (therefore the title: multi-source multi-view transfer).
Given a document collection, DocNADE is used to generate the topic-word matrix. In the local view transfer step, the pre-trained WordPool is used, from which knowledge is transferred on the target document. The global view transfer is done by transferring knowledge from the pre-trained TopicPool to the target. As described in Algorithm 1 in the paper, both Word- and TopicPool are jointly used in the transfer learning process.
For evaluation, three different measures are taken into account: Perplexity, Topic Coherence and Precision (Information Retrieval). In comparison to a DocNADE only approach, all values are better in the settings that use the transfer learning approach. Compared to DocNADE + word embeddings, the results are competitive as well. In both experiments, the multi-source setting evaluates best overall.
In conclusion, the paper shows that exploiting multiple sources and views in transfer learning leads to an overall improvement in the given tasks. The main contribution is the usage topic models in a transfer learning framework. Additionally the use of multi-source word embeddings is novel too, especially in the joint setting with the topic model transfer. The paper shows how the DocNADE approach is enhanced to make use of both local and global view transfer and how this enhancement leads to improved performance on various related tasks.
Still, the overall contribution is mostly in combining existing methods and can be judged as rather incremental.
Minor note: A small mistake has been found in Table 5. The best perplexity value in the first column is not the bold 638, but the 630 in the local-view transfer setting.
Edit after rebuttal: In my review I did not value the contribution of the transfer learning approach enough. So, when also considering the extensive evaluation I am now leaning towards accept.


Many state of the art models rely on two architectural innovations; skip connections and batch normalization. However batch normalization has a number of limitations. It breaks the independence between training examples within a batch, performs poorly when the batch size is too small, and significantly increases the cost of computing a parameter update in some models. This work identifies two practical benefits of batch normalization. First, it improves the final test accuracy. Second, it enables efficient training with larger batches and larger learning rates. However we demonstrate that the increase in the largest stable learning rate does not explain why the final test accuracy is increased under a finite epoch budget. Furthermore, we show that the gap in test accuracy between residual networks with and without batch normalization can be dramatically reduced by improving the initialization scheme. We introduce "ZeroInit", which trains a 1000 layer deep Wide-ResNet without normalization to 94.3% test accuracy on CIFAR-10 in 200 epochs at batch size 64. This initialization scheme outperforms batch normalization when the batch size is very small, and is competitive with batch normalization for batch sizes that are not too large. We also show that ZeroInit matches the validation accuracy of batch normalization when training ResNet-50-V2 on ImageNet at batch size 1024.

This paper conducts extensive experiments to verify two practical benefits of batch normalization. i) It increases the final test accuracy and the largest stable learning rate; ii) it enables efficient training with larger batches and a larger learning rate. In addition, the authors propose a new initialization scheme, “ZeroInit”, to train a deep ResNet to improve the test accuracy. My detailed comments are as follows.
Positive points:
1. The experiments are sufficient. In this paper, the authors conduct extensive experiments to explore the benefits of batch normalization, and verifies the effectiveness of the proposed “ZeroInit”.
2. The method is effective in some cases. Specifically, the proposed “ZeroInit” outperforms batch normalization when the batch size is small, and it is competitive with batch normalization when the batch size is not too large.
Negative points:
1. The importance and novelty of the empirical study should be emphasized. The practical benefits of batch normalization can be also found in other papers. For the first benefit, most studies (Bjorck et al. 2018) have found that batch normalization is able to improve the test accuracy. For the second benefit, batch normalization requires a large batch size and a large learning rate (Santurkar et al., 2018). Therefore, what is the difference between this paper and others? More critically, it is necessary to explain why batch normalization has these benefits. It would be better to provide empirical or theoretical justifications to support these.
2. The motivation of the proposed “ZeroInit” is not clear. (Balduzzi et al., 2017) states that “the correlations can be preserved by initializing deep networks close to linear functions”. It is not clear how “ZeroInit” preserves the correlations?
3. Why initialize the scalar multiplier and biases to zero? What are the benefits of the zero initialization? Actually, the scalar multiplier and biases can be randomly initialized. When they are randomly initialized, what is the performance of the initialization? It is an important baseline to justify the effectiveness of the proposed initialization method.
4. The technical details of “ZeroInit” are not clear. It would be better to express the proposed initialization “ZeroInit” in the mathematical formulation.
5. The proposed initialization “ZeroInit” is designed for deep ResNets. How to extend it to the other deep neural networks?
6. This paper states that “the empirical success of batch normalization …improves the conditioning of the loss landscape. However, our results conclusively demonstrate that this is not the case”. Does it mean that batch normalization does not improve the conditioning of the loss landscape? However, the empirical results cannot justify this statement and explain the success of batch normalization.
7. Some results of the figures are missing. In Figure 1, the experimental results of w/o batch norm with varying batch sizes (2^0 ~ 2^5) are missing. Similarly, Figure 2 also has missing results. Please provide more discussions about these missing results.


Recently mean field theory has been successfully used to analyze properties of wide, random neural networks. It gave rise to a prescriptive theory for initializing feed-forward neural networks with orthogonal weights, which ensures that both the forward propagated activations and the backpropagated gradients are near 2 isometries and as a consequence training is orders of magnitude faster. Despite strong empirical performance, the mechanisms by which critical initializations confer an advantage in the optimization of deep neural networks are poorly understood. Here we show a novel connection between the maximum curvature of the optimization landscape (gradient smoothness) as measured by the Fisher information matrix (FIM) and the spectral radius of the input-output Jacobian, which partially explains why more isometric networks can train much faster. Furthermore, given that orthogonal weights are necessary to ensure that gradient norms are approximately preserved at initialization, we experimentally investigate the benefits of maintaining orthogonality throughout training, and we conclude that manifold optimization of weights performs well regardless of the smoothness of the gradients. Moreover, we observe a surprising yet robust behavior of highly isometric initializationseven though such networks have a lower FIM condition number at initialization, and therefore by analogy to convex functions should be easier to optimize, experimentally they prove to be much harder to train with stochastic gradient descent. We conjecture that stability of model linearized around its initial parameters has implication beyond the lazy training regime-stability of the linearization impacts the degree to which the initial curvature is predictive of the local curvature during training.

This paper analyzes the connection between the spectrum of the layer-to-layer or input-output Jacobian matrices and the spectrum of the Fisher information matrix / Neural Tangent Kernel. By bounding the maximum eigenvalue of the Fisher in terms of the maximum squared singular value of the input-output Jacobian, this paper provides a partial explanation for the successful initialization procedures obeying "dynamical isometry". By additionally investigating optimization on the orthogonal weight manifold, this paper sheds light on the important of maintaining spectral uniformity throughout training.  These two analyses help fill in important gaps in the understanding of initialization, dynamical isometry, and the training of deep neural networks. For these reasons I recommend this paper for acceptance.
There are two aspects of the paper that could nevertheless use some clarification and improvement. First, unless I missed something, this paper does not provide any bounds on the condition number or the minimum eigenvalue of the Fisher or the NTK. It seems like the main arguments only depend on the maximum eigenvalues. Generally, I think the insights into the maximum eigenvalues are useful and important on their own, but perhaps some additional discussion up front clarifying which results were derived theoretically and which were observed empirically could be useful.
Second, it should be noted that the the networks trained in the experiments are likely in a regime that is well outside the NTK regime, in two important ways: the dataset is large compared to the width and the optimal learning rates may be large as well.
Overall, I think this is a good paper that adds important insights into the study of initialization, local geometry, and their effects on training speed.


Olfaction has been and still is an area which is challenging to the research community. Like other senses of the body, there has been a push to replicate the sense of smell to aid in identifying odorous compounds in the form of an electronic nose. At IBM, our team (Cogniscent) has designed a modular sensor board platform based on the artificial olfaction concept we called EVA (Electronic Volatile Analyzer). EVA is an IoT electronic nose device that aims to reproduce olfaction in living begins by integrating an array of partially specific and uniquely selective smell recognition sensors which are directly exposed to the target chemical analyte or the environment. We are exploring a new technique called temperature controlled oscillation, which gives us virtual array of sensors to represent our signals/ fingerprint. In our study, we run experiments on identifying different types of beers using EVA. In order to successfully carry this classification task, the entire process starting from preparation of samples, having a consistent protocol of data collection in place all the way to providing the data to be analyzed and input to a machine learning model is very important. On this paper we will discuss the process of sniffing volatile organic compounds from liquid beer samples and successfully classifying different brands of beers as a pilot test. We researched on different machine learning models in order to get the best classification accuracy for our Beer samples. The best classification accuracy is achieved by using a multi-level perceptron (MLP) artificial neural network (ANN) model, classification of three different brands of beers after splitting one-week data to a training and testing set yielded an accuracy of 97.334. While using separate weeks of data for training and testing set the model yielded an accuracy of 67.812, this is because of drift playing a role in the overall classification process. Using Random forest, the classification accuracy achieved by the model is 0.923. And Decision Tree achieved 0.911.

Main comments:
Quality: This paper reads like a project report rather than a typical ICLR paper. It describes a direct application of naïve learning tool (random forest and neural network) without sufficient insight.
Clarity: In general, the descriptions on data collection and training are clear.
Originality: The novelty of this work is very limited, if any.
Contributions:
The contribution are limited. If the main contributions lie it the construction of the device, I suggest a resubmission to a device-related conference.
From the machine learning perspective, this work is a simple and direct application of existing machine learning tools..
Other comments:
This paper mentions the authors are from a group (Cogniscent) at IBM, which violates the double-blind policy (at least to certain degree).


Efficient exploration is essential to reinforcement learning in huge state space. Recent approaches to address this issue include the intrinsically motivated goal exploration process (IMGEP) and the maximum state entropy exploration (MSEE). In this paper, we disclose that goal-conditioned exploration behaviors in IMGEP can also maximize the state entropy, which bridges the IMGEP and the MSEE. From this connection, we propose a maximum entropy criterion for goal selection in goal-conditioned exploration, which results in the new exploration method novelty-pursuit. Novelty-pursuit performs the exploration in two stages: first, it selects a goal for the goal-conditioned exploration policy to reach the boundary of the explored region; then, it takes random actions to explore the non-explored region. We demonstrate the effectiveness of the proposed method in environments from simple maze environments, Mujoco tasks, to the long-horizon video game of SuperMarioBros. Experiment results show that the proposed method outperforms the state-of-the-art approaches that use curiosity-driven exploration.

The authors study the problem of exploration in deep reinforcement learning. The authors borrow the ideas developed in the intrinsically motivated goal exploration process, and entropy maximization and propose a method on Noverly-Pursuit. The authors then empirically study the proposed method.
1) The authors investigate an important problem and I would appreciate the authors if they could motivate its importance more in their work.
2) In the second paragraph, the author mentioned that goal-conditioned exploration behaviours can maximize entropy. Later in the same paragraph, they claim that "The exploration policy leads
to maximize the state entropy on the whole state distribution considering tabular MDP". I guess the authors' point was this approach might increase the entropy rather than maximizing it. If the claim is, in fact, maximization, a reference would be helpful. If the authors prove it in this paper, implying it in this paragraph is also helpful.
3) In the background section, the authors did not specify whether they provide background on tabular MDP or beyond that. By calling the transition kernel the state transition probabilities, it seems they introduced a tabular MDP, but a more concrete introduction and preliminaries would help to follow the paper.
3) The first paragraph of section 2, the author mentioned that
"The target of reinforcement learning is to maximize the expected discounted return". I hope the authors mean"one of the targets in the study of reinforcement learning ..."
4) in the same paragraph, why the \gamma = 0 is excluded? is there any specific reason? Also, when the authors include the gamma = 1, do they make sure the maximization in line 9 of the same paragraph is well defined in regular cases?
5) Regarding the experiment in figure 2. It would be useful to the readers if the authors provide more details about this experimental study.
6) In the few paragraphs below Figure 2, it would be nicer if the authors provide a clear definition of each term. In order to follow the paper, I relied on my imperfect inference to infer the definitions. Also, I find it probably useful to distinguish the random variables and their realizations in the notation.
6) Regarding the theorem1. I would recommend making the statement more transparent and more clear. I also recommend to even not calling it a theorem since it, as mentioned, is as clear as the definitions. Also, arent x_t(i)s non-negative by definition?
7) In this sentence:
"However, we don’t know what non-visited states are and
where non-visited states locate in practice since we can’t access ..."
I think the authors' point was that "we might not have access to it in general".
8) It would be helpful to me to evaluate this paper if the authors explain more how the following statements go through:
"To deal with this problem, we assume that state density over the whole state space is continuous, thus visited states and non-visited states are close". I am not sure how "thus visited states and non-visited states are close" follows from continuity of density and what is the notion of closeness.
9-1) Theorem 2: while H seems to be a function of d_\pi1:t(s), I am not sure how to interpret the argmax_{e_t}. A bit of help from the authors would be appreciated.
9-2) Theorem 2: In the proof, I was not able to justify to my self the transition form g(xi; xj) = Hxi [d1:t+1] 􀀀 Hxj [d1:t+1] to the second line. Also, what is the definition of H_x_i?
10) In equation 2, the authors use a notation d, I guess as distance. It would not only be helpful to define it but also would be helpful to use a different notation for distance and the d used on page 3, presumably for "empirical state distribution".
11) At the beginning of the paper, the authors motivated the maximum entropy but the final algorithm is based on other approaches.
12) Despite the fact that I could not find this paper ready enough and well-posed, I also have a concern about the novelty of the approach. I think it is not novel enough for publication at ICLR, but I am open to reading other reviewers', as well as commenters', and more especially the authors' rebuttal response.
13) I also encourage the authors to provide a discussion on the cases where the novelty (whatever that could mean) does not matter, rather the  novelty of state-action pair matters.


Generative Adversarial Networks have been shown to be powerful tools for generating content resulting in them being intensively studied in recent years. Training these networks requires maximizing a generator loss and minimizing a discriminator loss, leading to a difficult saddle point problem that is slow and difficult to converge. Motivated by techniques in the registration of point clouds and the fluid flow formulation of mass transport, we investigate a new formulation that is based on strict minimization, without the need for the maximization. This formulation views the problem as a matching problem rather than an adversarial one, and thus allows us to quickly converge and obtain meaningful metrics in the optimization path.

The paper addresses the task of constructing a generative model for data using a novel optimal transport-based method. The paper proposes an alternative view of obtaining generative models by viewing the generation process as a transport problem (specifically, fluid flow mass transport) between two point clouds living in high-dimensional space. To solve the transport problem, a discretization scheme is proposed, which gives rise to a variant of point cloud registration problem, which is solved using numerical optimization. The results are provided on synthetic data and a real MNIST data.
With generative models, and particularly generative adversarial networks (GANs) being notoriously hard to train, alternative ways of constructing generative models are needed. The paper does a good job displaying the potential optimization issues arising when training GANs, and the basic theoretical foundations used in the paper have been validated in prior work. However, with the introduction of a transport-based formulation, its respective issues may arise, that are not described in the paper. Will the optimization always converge, and if yes, to which kind of optimum? What are the requirements for the point clouds R and T?
While the conceptual contribution is that the , the technical novelty is limited and mostly amounts to the appropriate choice of distributions in R and T and applying the discretization schemes to be able to compute the experiments. Unfortunately, the model is not studied theoretically, i.e. no description is given regarding the class of generative tasks that could be solved using the method, or the class of functions that could be learn in such a way.
The experimental evaluation demonstrates on only two simple examples the results of the work. More experiments are needed to fully understand the possibilities of the framework.
The convergence speed is not indicated, and the efficiency of the optimization is not described.
To summarize, I believe the paper should not be accepted in its present (early) form, as (1) more detailed theoretical insight are needed, and (2) much more computational experiments are needed to fully validate the method.


Gating mechanisms are widely used in neural network models, where they allow gradients to backpropagate more easily through depth or time. However, their saturation property introduces problems of its own. For example, in recurrent models these gates need to have outputs near 1 to propagate information over long time-delays, which requires them to operate in their saturation regime and hinders gradient-based learning of the gate mechanism. We address this problem by deriving two synergistic modifications to the standard gating mechanism that are easy to implement, introduce no additional hyperparameters, and improve learnability of the gates when they are close to saturation. We show how these changes are related to and improve on alternative recently proposed gating mechanisms such as chrono-initialization and Ordered Neurons. Empirically, our simple gating mechanisms robustly improve the performance of recurrent models on a range of applications, including synthetic memorization tasks, sequential image classification, language modeling, and reinforcement learning, particularly when long-term dependencies are involved.

This paper introduces and studies modifications to gating mechanisms in RNNs.
The first modification is uniform gate initialization. Here the biases in the forget and input gate bias are sampled such that after the application of the sigmoid the values are in the range (1/d, 1-1/d) where d is the dimensionality of the hidden space for the bias. The second modification is the introduction of a refine gating mechanism with a view to allow for gradients to flow when the forget gates f_t in the LSTM updates are near saturation. The idea is to create an effective gate g = f_t +/- phi(f_t, x_t). The paper proposes using phi (f_t, x_t) = f_t(1-f_t) * (2*r_t-1) where (r_t is between 0 or 1). The effect of such a change is that g_t can reach values of 0.99 when the value of f_t is 0.9 allowing gradients to flow more freely through the parameters that constitute the forget gate. Overall the change corresponds to improving gradient flow for the forget gate by interpolating between f_t^2 and (1-f_t)^2. i.e. the authors note that the result of these changes is that it corresponds to sampling biases from a heavier tailed distribution while the refine gate (by allowing the forget gate to reach values close to 0 and 1), allows for capturing information on a much longer time scale.
The paper studies various combinations of the two changes proposed to gating architectures. Other baselines include a vanilla LSTM, a Chrono initialized LSTM, and an ordered Neuron LSTM. The models are trained on several synthetic and real world tasks. On the copy and add tasks, the LSTMs that contain the refine gate converge the fastest. A similar story is observed on the task of pixel by pixel image classification. The refine gate was also adapted to memory architectures such as the DNC and RMA where it was found to improve performance on two different tasks.
Overall, the paper is written well, I like the (second) idea of the refine gate and the contributions are explained in an accessible manner. While I'm not entirely convinced about the proposed initialization scheme but across the many different tasks tried, the use of the refine gate does appear to give performance improvements that lead me to conclude that this aspect of the work is a solid contribution to the literature.
Questions and comments:
* This manuscript already quite long and has several formatting issues. Several of the figures are unreadable when printed. For example, every piece of text on Figure 2(d) is unreadable on paper. Figure 3 and 5 are difficult to read; they contain too many alternatives with a colour scheme that makes it difficult to distinguish between them -- consider displaying a subset of the options via a plot and using a table to display (# steps to convergence) as a metric instead. It also appears as if the caption for Table 6 is deleted?
* I think that for this approach to work, two conditions need to be satisfied (a) there must be foreseeable improvements in the use of a forget gate that can reach values close to 0/1 for the task at hand and (b) r_t needs to function well despite not being too close to 0 or 1 (lest its parameters suffer from gradient flow issues).
  * Was there any visualizations done on whether (a) happened? i.e. for the URLSTMs that performed well, were the values of the forget gate closer to 0/1 than the baselines?
  * What were typical values of r_t, did the models need the refine gate to reach values close to 0 or 1 for the overall approach to work?


How can deep learning systems flexibly reuse their knowledge? Toward this goal, we propose a new class of challenges, and a class of architectures that can solve them. The challenges are meta-mappings, which involve systematically transforming task behaviors to adapt to new tasks zero-shot. The key to achieving these challenges is representing the task being performed in such a way that this task representation is itself transformable. We therefore draw inspiration from functional programming and recent work in meta-learning to propose a class of Homoiconic Meta-Mapping (HoMM) approaches that represent data points and tasks in a shared latent space, and learn to infer transformations of that space. HoMM approaches can be applied to any type of machine learning task, including supervised learning and reinforcement learning. We demonstrate the utility of this perspective by exhibiting zero-shot remapping of behavior to adapt to new tasks.

This paper introduces a zero-shot task adaptation methodology by learning a meta-learner that reuses a latent representation of data points and tasks (Homoiconic Meta-Mapping). The proposed approach is applied to learn multivariate polynomial, a card game and the paper claims that such a method can apply in both supervised and RL settings.
+ves:
+ Task-to-task mapping is a relevant topic in the contemporary context, and the paper seeks to address this relevant problem.
+ The overall idea of considering winning-losing strategies in meta-learning is interesting.
+ The experimental settings of card games and polynomials are interesting to study the proposed setting, and the results are promising.
Concerns:
- A key issue is that the paper claims to be the first to propose a method for zero-shot task adaptation (pg 8, para before “Future Directions”). The paper seems to have missed papers such as (1) Zero-Shot Task Transfer, Pal et. al, CVPR 2019, and, (2) Zero-Shot Task Generalization with Multi-Task Deep Reinforcement Learning, Junhyuk Oh et al, ICML 2017. In particular, the CVPR 2019 paper proposes zero-shot task transfer using meta-mappings. Citing these papers and comparing against them is essential, considering this paper addresses the same objective.
- The proposed methodology does not clearly detail how the zero-shot problem setting is used. Section F.2 (Appendix) gives the formulation of a meta-learner and zero-shot task a bit. However, how the meta-learner performs zero-shot task parameter regression is not clear. The notations used in mathematical equations are also not very clear, and a reader needs to swap between different sections to understand the proposed method. (Moving Sec F.2 to the main paper would be very useful. Similarly, moving Figs 5, 7, 8 etc. to the main paper would have helped greatly too.)
- How is the notion of a “task” defined? Are different classes of a dataset tasks (already done in [3]), or are different vision-related objectives as tasks (reference [1])? Are language-related objectives (such as: document classification, sentiment analysis, Named-entity recognition etc.) tasks? In an RL setting, do zero-shot tasks deal with a new environment? It would have been nice to see clearer definitions for the kind of tasks that this method is suited to.
- It was not clear on how the relationship between source tasks and zero-shot tasks are modeled or computed. Clearly, the performance of such a system depends on the correlations between the tasks, and these details are not clear, especially in generalizing the proposed method. The paper seems to correlate tasks and language-related hints; however getting a language hint is not always available, especially for datasets such as Taskonomy [1, see Refs below] - which is a dataset for multiple tasks.
- The zero-shot part is described using a winning-losing scenario, where we have a partial/side knowledge of winning of task_1 and the model can predict losing strategy. Now, what if:
We do not have any knowledge of the winning strategy (or losing strategy) of a zero-shot task?
Can we predict the losing strategy of a game, let’s say card game, from the knowledge of other games like: soccer, tennis, badminton etc. (the initial motivation of the paper). This is quite possible in the continual learning scenario (the paper mentioned some studies on continual learning). If so, then please comment on the negative transfers in that scenario and how it will affect the inference of the meta-learner.
The proposed method also mentioned language descriptions. In case of tasks that are not easy to describe (say, a vision task such as segmentation), how will the proposed method will compute the task-prior of seen-tasks and zero-shot tasks?
The paper mentions “winning-losing” strategy and the metanetwork gives losing strategy parameters. What if the scenario is the otehr way - given a losing strategy, we now have to perform the winning strategy (IMHO this has more value in the real world). Does the paper show the result on this, or am I missing something?
- Experiments are limited to synthetic data and card games, and it would be nice to see the validation of the proposed method on different domains. The notion of zero-shot tasks are also not clear for multivariate polynomial experiment and card-game experiment.
- Figure 1: please clearly mention in the caption what is the meaning and purpose of probes. It is not completely clear whether probes (also referred as held-out set In figure 1 caption) are taking part in training, or only at the time of validation?
- Shared Z vs. Separate task embedding: reuse of M (or, H network) network will only be possible if the input and output dimensions are same - i.e. the dimension of Z_output of M is the same as the dimension of Z_input of the M. What if  input and output dimensions are not the same?
Other minor comments:
* One general comment is: please mention the “Zero-shot” experiments more clearly (may be a separate paragraph or subsection), otherwise it is hard to follow.
* Figure 2(a) please remove the question mark (?) after Held-out. Similarly, Figure 5: Please complete the sentence “ Performing from …”.
* Figure 1: (a)  Basic meta-learning: Please use some other color to highlight “(a)  Basic meta-learning”, the grey gives an illusion that the step ((a)  Basic meta-learning is not important.
* A term such as “Homoiconic” would have benefited from a formal definition, especially considering it’s not a commonly used term.
* The presentation is in general not well-polished, and needs revision.
References:
[1] Taskonomy, CVPR 2018, Zamir et al
[2] Zero-shot Task Transfer, CVPR 2019, Pal et al
[3] Learning To Detect Unseen Object Classes by Between-Class Attribute Transfer, CVPR 2009, Lampert et al.
[4] Zero-Shot Task Generalization with Multi-Task Deep Reinforcement Learning, ICML 2017, Junhyuk Oh et al.
======= POST-REBUTTAL COMMENTS=========
I thank the authors for the response. Some of the queries/concerns in the review comments above were clarified. I am updating my rating to Weak Reject, but unfortunately am not completely convinced about accepting the paper in its current form for the following reasons:
- Thank you for pointing out the difference between this work and other recent efforts (Oh et al, 2017 and Pal et al, 2019), and updating the manuscript with this content. However, considering these methods have been proposed, it is only natural to ask that the proposed method needs to be compared against them, to judge the usefulness of meta-mappings as against other ways to achieve zero-shot task adaptation, at least on some common datasets/settings. The paper indeed does propose a new idea, but without a comprehensive comparison/evaluation, the inferences on its usefulness may not be conclusive without such studies.
- Considering methods solving the same/similar problem using other approaches have tested on real-world datasets, experiments on synthetic datasets alone seems limiting.  I agree and appreciate the amount of material that is already in the paper, but it is also important to have the correct material to make conclusive inferences that benefit the community.
I once again thank the authors for the efforts in the response, but just think the paper may benefit from revisions based on the comments shared here, to be more impactful/useful.


Word embeddings learnt from large corpora have been adopted in various applications in natural language processing and served as the general input representations to learning systems. Recently, a series of post-processing methods have been proposed to boost the performance of word embeddings on similarity comparison and analogy retrieval tasks, and some have been adapted to compose sentence representations. The general hypothesis behind these methods is that by enforcing the embedding space to be more isotropic, the similarity between words can be better expressed. We view these methods as an approach to shrink the covariance/gram matrix, which is estimated by learning word vectors, towards a scaled identity matrix. By optimising an objective in the semi-Riemannian manifold with Centralised Kernel Alignment (CKA), we are able to search for the optimal shrinkage parameter, and provide a post-processing method to smooth the spectrum of learnt word vectors which yields improved performance on downstream tasks.

This paper presents a novel method for unsupervised post-processing of pretrained word embeddings that enforces the distributional word vector space to be more isotropic, which in turn improves the expressiveness and quality of the space in terms of similarity. The method is based on the shrinkage of the covariance/Gram matrix and its effects on the input space are evaluated across a range of intrinsic evaluation tasks. While I like the idea overall and this line of work in general, there are still some concerns with the current version of the paper:
* Overall, although its design seems more principled, the proposed method does not seem significantly better than the previous (very similar) method of Mu et al. I would like to see more evidence in the favour of the proposed method, pointing that we should use that instead of Mu's method. Also, the gains over non-processed spaces often seem insignificant, and offer only small benefits.
* The derivation of the method seems too verbose, especially in light of the fact that it is directly inspired by previous work on the shrinkage estimation of covariance matrices and CKA. I would suggest the authors to spend more time on linking their high-level hypotheses to the low-level mathematical implementations instead of flooding the paper with equations - for the interested reader a lot of the derivation process can be put into an appendix, the paper should focus on conveying the key principles instead. This would also offer additional space for more experiments.
* One very relevant paper is not mentioned at all: https://arxiv.org/pdf/1809.02094.pdf (Artetxe et al., CoNLL 2018). I would suggest the authors to cite that work and ideally even compare to it on their set of intrinsic tasks (e.g., word similarity, word analogy), and then discuss the difference in results and their approach to unsupervised post-processing. This shouldn't be so difficult to do as the code from that paper is available online: https://github.com/artetxem/uncovec
* The paper mixes true similarity datasets (such as SimLex) with broader semantic relatedness datasets (MEN, WordSim-353), even mixing true semantic similarity of the same dataset (WordSim353-SIM) with its relatedness subset (WordSim353-REL). In light of the known conceptual differences between the relations of similarity versus relatedness, I would suggest to report the results separately for the two tasks. For instance, another true similarity dataset, which is not used in the evaluation is SimVerb-3500. Along the same line, it is also not clear what type of similarity is meant when the authors state that through post-processing 'the similarity between words can be better expressed'. What does it mean to better express the similarity between words in the first place? Do we talk about true similarity or relatedness or both? However, the two relationships between words support different classes of downstream applications, so therefore it is even more problematic 1) not to distinguish between the two and 2) not to report any results in any downstream (extrinsic) tasks where the post-processed embeddings are used as features (STS is still a semi-intrinsic task imho; BLI is considered as an intrinsic task in cross-lingual settings).
* One of the key reasons to apply post-processing is to mitigate the frequency artefacts: however, such an evaluation that goes towards that direction is never executed. For instance, I would like to see a focused experiment that measures how post-processing affects high-frequency versus mid-to-lower frequency or rare words. A recently developed CARD-660 dataset might be used to this end.
* It is not clear how exactly the authors run Mu et al.'s method, that is, how many top principal components are removed for the input vectors? How is this selected? Are always the optimal results reported for the baseline Mu et al.'s method? Why not reporting the results with removing 2 and 3 at the same time to further prove the point that their method is non-automatic? This would also give the reader a hint how much the results with Mu's method actually differ/decrease if one just decides to make the method 'automatic' by just fixing the method to always remove the same number of top principal components.
Minor remarks:
* The title of the paper is a bit imprecise: in the word embedding literature, the term post-processing is often referred to the methods that fine-tune word embeddings using some external knowledge after (i.e., post) the initial distributional training (e.g., the so-called retrofitting methods). However, in the context of the paper post-processing actually refers to some unsupervised post-training steps on the input space without injecting any external information. This should be made clearer in the paper, and perhaps adding a paragraph which outlines the core difference to other work on retrofitting would be helpful as well.
* I might be missing something while reading Section 3, but it is currently not clear to me how the oracle Gram matrix K is obtained in the first place. Perhaps it makes sense to briefly summarize this in a quite direct way to avoid the reader's confusion?
* It is great to see a summary of the key post-processing steps at the very end of Section 3; this is really helpful for everyone who would like to try out the proposed method off-the-shelf. However, the summary is not self-contained as it is not clear what \mathcal{L}'' refers to (and the reader must search through the derivations again to find its meaning).
* I like the evaluation on word translation, and I believe that the proposed post-processing methods could actually improve word translation through some pre-alignment perturbations. It is a pity that the method is not evaluated on more distant language pairs, as I believe that the method might have much more effect there than on the already-saturated EN-to-ES/FR/IT bilingual lexicon induction tasks.
* For de-en word translation Mu's method actually beats the proposed method (incorrect number in bold)
* It is not clear why MUSE is used for word retrieval experiments, given the fact that it is known to be unstable (Sogaard et al., ACL 2019), and there are more robust and more effective methods available such as VecMap (Artetxe et al., ACL 2018) or RCSLS (Joulin et al., EMNLP 2019)


The Expectation-Maximization (EM) algorithm is a fundamental tool in unsupervised machine learning. It is often used as an efficient way to solve Maximum Likelihood (ML) and Maximum A Posteriori estimation problems, especially for models with latent variables. It is also the algorithm of choice to fit mixture models: generative models that represent unlabelled points originating from k different processes, as samples from k multivariate distributions. In this work we define and use a quantum version of EM to fit a Gaussian Mixture Model. Given quantum access to a dataset of n vectors of dimension d, our algorithm has convergence and precision guarantees similar to the classical algorithm, but the runtime is only polylogarithmic in the number of elements in the training set, and is polynomial in other parameters -as the dimension of the feature space, and the number of components in the mixture. We generalize further the algorithm by fitting any mixture model of base distributions in the exponential family. We discuss the performance of the algorithm on datasets that are expected to be classified successfully by those algorithms, arguing that on those cases we can give strong guarantees on the runtime.

Quantum machine learning is a hot topic, recently. There are several "schools of thought":
-- quantum kernel embeddings, which work per-observation, and allow for application on noisy intermediate-scale devices,
-- work based on quantum optimization solvers, which try to claim one could apply quantum eigensolvers to various training problems, but which does not seem to be applicable on intermediate-scale devices, given the scale of the requisite input,
--  novel quantum algorithms.
In principle, the novel quantum algorithms hold the promise of an exponential speed-up.
The authors propose a novel variant of expectation minimisation for parameter estimation of a gaussian mixture model with uniform mixing coefficients, targeting an ill-defined model of quantum computing of their own coinage.  Unfortunately, the paper is very sloppy.
The sloppiness starts with the model of quantum computing, which seems to assume anything the authors needed:
-- Definition 1: one can perform |k|d arithmetic operations in time polylog(d)
-- sentence above Lemma 3.4: post-selection (which makes it possible to solve at least all of PP, cf. https://arxiv.org/abs/quant-ph/0412187)
-- sentence below Lemma 3.6: "quantum linear algebra subroutines and tomography" in no time at all (?), wherein there are information-theoretic limits \Omega(exp(kd)) on the state tomography.
In terms of statistics and learning theory, the authors:
-- do not consider the separation of the Gaussians as a parameter. It is well-known that with o(1) separation, exponentially many samples are required, and with sqrt(log(k)) separation, polynomially many samples suggest (http://ieee-focs.org/FOCS-2017-Papers/3464a085.pdf). Unless Definition 1 of the authors "subsumes" processing of exponentially large numbers of samples in polylog(d) time, the authors may need to add an assumption on the separation.
-- the authors explain that their approach works only for the gaussian mixture model with uniform mixing coefficients only at the top of page 4, in a completely obscure notation of their own, and do not reference it as an assumption in the theorems later.
The sloppiness continues with the description of the Experiments in Section 4. Authors present a table of some results, but do not mention whether these have been obtained on quantum-computing hardware, a simulator thereof (what simulator? with noise?), or whether this is some fully-classical variant of the algorithm being tested. This clearly violates any "reproducibility checklist".
Within the "minor comments" category:
-- the introduction of the EM algorithms for GMM is sloppy. It is well known that EM for GMM is super-sensitive to noise and balance of the mixing coefficients (https://ieeexplore.ieee.org/document/8635825) and can get stuck in arbitrarily bad local optima, which is not mentioned once.
-- the references to other algorithms for GMM mention only Dasgupta 1999, rather than the subsequent 20 years of research, e.g., of Ankur Moitra at MIT (https://math.mit.edu/directory/profile.php?pid=1502).
-- on the other hand, there are plentiful references to arxiv pre-prints of Kerenidis et al, at least some of which have been shown to be vacuous, e.g., https://arxiv.org/abs/1808.09266 with an infinite upper bound on the run-time and no relation to the present paper?
-- there are some formulae missing or ending half-way through, e.g. Page 3 before "alas", Page 3 after "GMM would be".
-- there are a number of language issues: "we can thresholding", "some other real-world dataset".
-- the discussion starting with "Let's have a first high-level comparison" is completely wrong. Especially the condition number estimates of 5 seem to have no justification what so ever.
-- authors say that "polynomial dependence on the rank, the error, and the condition number, make these algorithms impractical on interesting datasets" -- but it is not clear whether they mean that their algorithm is also impractical?
Overall, while I like the idea of parameter estimation on a quantum computer, I could not recommend accepting the paper in its current form.


Many advances in Natural Language Processing have been based upon more expressive models for how inputs interact with the context in which they occur. Recurrent networks, which have enjoyed a modicum of success, still lack the generalization and systematicity ultimately required for modelling language. In this work, we propose an extension to the venerable Long Short-Term Memory in the form of mutual gating of the current input and the previous output. This mechanism affords the modelling of a richer space of interactions between inputs and their context. Equivalently, our model can be viewed as making the transition function given by the LSTM context-dependent. Experiments demonstrate markedly improved generalization on language modelling in the range of 3-4 perplexity points on Penn Treebank and Wikitext-2, and 0.01-0.05 bpc on four character-based datasets. We establish a new state of the art on all datasets with the exception of Enwik8, where we close a large gap between the LSTM and Transformer models.

I have read the authors' response. Their points regarding baseline comparisons are sensible in that there isn't a reason to expect the observations to *not* generalization to other datasets. It is odd that mLSTM is outperformed by LSTM in Table 3, but as the authors note in section 4.2 this may be due to instability of mLSTM during training. The results in the paper demonstrate significant improvement over LSTM, and while there are not as many baseline comparison to similar models as I would have liked to see, the quality of this work is sufficiently high that this is not a fatal flaw. In light of the author response and other reviews, I am revising my rating to 6: Weak Accept.
=====
This paper proposes a modification of LSTM networks in the context of language modeling called Mogrifier LSTM. Ordinary LSTMs are defined as recurrent operations on the current input, previous hidden state, and previous cell state. The proposed Mogrifier LSTM utilizes the same recurrent unit as the LSTM, but the input and previous hidden state are updated with several rounds of mutual gating. In each round, the input  is multiplied elementwise by a gate computed as a function of the hidden state (or vice versa). The authors experiment on word-level and character-level modeling and compare their Mogrifier LSTMs to several state-of-the-art approaches. They also conduct an ablation study to show the effect of various design choices and hyperparameters and experiments on a reverse copy task.
Specific contributions include:
* Proposal of a novel approach for modulating inputs to a recurrent unit by mutual gating.
* Experiments demonstrating strong performance on a number of language modeling tasks.

The paper in its current state is borderline, leaning towards weak reject. Points in favor of acceptance include the high clarity of writing, good experiments of the proposed model, and a discussion of possible reasons for why the mogrification operation works well. The main shortcoming of the paper is experimental comparison to baselines.
The authors were able to train baseline LSTMs to high levels of performance (presumably due to tuning of hyperparameters) and then demonstrate that Mogrifier LSTMs improve upon LSTMs significantly. This is perhaps not entirely surprising, because the hyperparameter range of the Mogrifier LSTM includes zero rounds of updates, which would render it identical to the baseline LSTM. Therefore, if the hyperparameters are tuned sufficiently well, the performance of the Mogrifier LSTM should be at least as good as the LSTM. What the experiments do not show is that the proposed mogrification outperforms other forms of multiplicative interaction and/or gating. The closest that the authors come to this is the single validation perplexity of the Multiplicative LSTM in Table 3. If thorough hyperparameter tuning is applied to the Multiplicative LSTM or the approaches of Wu et al. (2016) and/or Sutskever et al. (2011), does the Mogrifier LSTM still outperform them?
Other than this critical issue of baseline comparison, the experiments are quite informative. The ablation study showing the effect of different design decisions and the hyperparameter visualiztion in Appendix B are particularly useful. The mogrification operation is described precisely enough for other researchers to implement and the arguments made in 4.4 are compelling.
Question for the authors:
* Some qualitative analysis of the learned mogrification operation would be helpful for understanding the nature of the modulation. For example, how do the predictions change depending on the modulation? If x is modulated by different hidden states, is there a noticeable effect on the output?
* Did you experiment with other forms of modulation before arriving upon the mogrification formulation? There are some naive approaches such as concatenating the hidden state to the input and applying a nonlinear layer, or predicting affine parameters for the input as a function of the hidden state in the style of FiLM [1]. Are there obvious shortcomings in these naive approaches that mogrification handles gracefully?
[1] Perez, E., Strub, F., De Vries, H., Dumoulin, V. and Courville, A., 2018, April. Film: Visual reasoning with a general conditioning layer. In Thirty-Second AAAI Conference on Artificial Intelligence.


Meta-learning methods, most notably Model-Agnostic Meta-Learning (Finn et al., 2017) or MAML, have achieved great success in adapting to new tasks quickly, after having been trained on similar tasks. The mechanism behind their success, however, is poorly understood. We begin this work with an experimental analysis of MAML, finding that deep models are crucial for its success, even given sets of simple tasks where a linear model would suffice on any individual task. Furthermore, on image-recognition tasks, we find that the early layers of MAML-trained models learn task-invariant features, while later layers are used for adaptation, providing further evidence that these models require greater capacity than is strictly necessary for their individual tasks. Following our findings, we propose a method which enables better use of model capacity at inference time by separating the adaptation aspect of meta-learning into parameters that are only used for adaptation but are not part of the forward model. We find that our approach enables more effective meta-learning in smaller models, which are suitably sized for the individual tasks.

This paper analyzes the popular MAML (Model-Agnostic Meta-Learner) method, and thereafter proposes a new approach to meta-learning based on observations from empirical studies. The key idea of the work is to separate the base model and task-specific adaptation components of MAML. This decoupling of adaptation and modeling reduces the burden on the model, thus enabling smaller memory efficient deep learning models to adapt and give high performance on meta learning tasks. The paper proposes a learnable meta-optimizer consisting of a parametrized function U such that the knowledge of adaptation is embedded into its parameters (A,b), instead of forward model parameters. The computational challenges posed by the proposed method are addressed by expressing the parameter matrix A as a Knonecker product of small matrices which is more efficient from memory and time complexity view point. The results on Omniglot and CIFAR-FS are promising, and the paper shows that the proposed meta-optimize is "more expressive", as well as can adapt a shallower model to the same level of performance as MAML.
+ves:
+ The discussion on the deficiency of MAML combined with shallow models is well-supported experimentally.
+ The idea to leverage the parameters of a meta-optimizer for adaptation instead of using model parameters is novel and interesting.
+ The paper is well-written and easy to follow. It motivates its choices well, both in the proposed method and the experiments.
+ The paper presents fair comparison in all experiments with appropriately chosen baseline models, and the proposed approach is validated for both linear as well as non linear models using benchmark datasets.
Concerns:
- While MAML was a seminal work and is widely followed, there have been many follow-ups of MAML, including another widely used method Reptile (Nichol et al, On First-Order Meta-Learning Algorithms). How is the proposed method relevant more broadly to this genre of methods? Some discussion of this would have been useful to understand the generalizability of the idea.
- The choice of the Kronecker product to handle the dimensionality of the meta-optimizer is supported by the paper, but is not very convincing. How important is this choice? What if other decompositions were used?
- The paper seems to state that shallow models are convex (Sec 3.2); however, weight symmetry induces non-convexity even in shallow models. This perspective of the problem may not be very well-justified.
- In Sec 3.2, the paper compares the 1-step adaptation accuracy of a shallow network and a deeper 4 layered linear network and claim that shallow networks underperform. However this underperformance might be due to the difference in required number of steps to reach optimal performance by the two models, and may not be a fair comparison. Why is this conclusive inference? Considering these inferences motivate the full paper, this is important.
- All the presented results are on small CNNs. The paper motivates this as “easing the computational burden”. The original MAML work shows results on state-of-the-art convolutional and recurrent models. It may be important to show results on deeper models to be more confident about its applicability.
- Although one can obtain smaller meta-learned models using the proposed method, training via this method will incur a higher computational burden than MAML-trained deep models. The paper does not talk about this additional complexity at all. Comparisons of wall-clock times or asymptotic analysis of the proposed method w.r.t. MAML would have greatly helped understand the pros and cons of the method.
I am on the borderline on this work - it is a well-written paper with a clear objective and support. But lack of rigorous analysis of the proposed method in terms of the method (how important is the Kronecker factorization?), experiments (with deeper architectures) and a more generalizable understanding of the proposed idea seems to be limiting the work's impact.
========POST-REBUTTAL COMMENTS===============
I thank the authors for their response, and all the efforts in the updated manuscript. Some of the clarifications sought were answered clearly. However, unfortunately, I continue to remain on the borderline on this work for the reasons below. (I would be willing to increase my rating to 4 or 5, which however are not available on the drop down, but perhaps not beyond).
- The response to AnonReviewer1 says that "there have been multiple empirical and theoretical works dedicated solely to the study of MAML [7-11]", hence supporting this work dedicating its focus to MAML alone. However, on close observation, most of these efforts are not published on peer-reviewed avenues and are only on arXiv at this time. Ref [7] (Finn and Levine, 2017) is published but has significantly stronger contributions. Considering the largely empirical nature of this work, showing its generalizability would be required, in my opinion, to make the conclusions of this work useful to the audience. Expecting that it would naturally hold for other methods like REPTILE may not be sufficient. In my opinion, this is a significant limitation.
- I personally remained unconvinced about the response to the question on number of adaptation steps, as well as on the lack of deeper models in the empirical studies.
I once again appreciate the authors for all the additional efforts, it may just be good for the work to be more comprehensive to be relevant and useful.


This paper shows how to train binary networks to within a few percent points (∼ 3 − 5%) of the full precision counterpart with a negligible increase in the computational cost. In particular, we first show how to build a strong baseline, which already achieves state-of-the-art accuracy, by combining recently proposed advances, and carefully tuning the optimization procedure. Secondly, we show that by attempting to minimize the discrepancy between the output of the binary and the corresponding real-valued convolution additional significant accuracy gains can be obtained. We materialize this idea in two complementary ways: (1) with a loss function, during training, by matching the spatial attention maps computed at the output of the binary and real-valued convolutions, and (2) in data-driven manner, by using the real-valued activations being available during inference prior to the binarization process for re-scaling the activations right after the binary convolution. Finally, we show that, when putting all of our improvements together, the resulting model reduces the gap to its real-valued counterpart to less than 3% and 5% top-1 error on CIFAR-100 and ImageNet, respectively, when using a ResNet-18 architecture.

This paper greatly reduces the gap between binarized and real valued imagenet, using a variety of techniques. The most significant contributions of this paper are engineering based, and the careful combination and integration of approaches from previous papers. I believe that this is of significant practical importance to the field. I particularly appreciate the effort put into developing a very strong baseline that combined ideas from many previous papers!
My biggest concern is that ResNet is itself a very wasteful architecture in terms of compute and parameter count. If the goal is to develop a compute- and memory-efficient architecture, it would be good to also consider real-valued-network baselines that were proposed with computational and/or memory efficiency as a design goal.
Additionally, the specific choices for the new student-teacher loss, and new scaling network architecture, seem fairly ad-hoc.
Detailed comments:
"this implies a reduction of 32× in memory usage" , assuming the parameter count is held constant.
Fig 1 right: This is motivated in terms of preserving scaling factors that are lost by the binarization, but the functional form for this makes it look a lot like a learned gating operation. If the sigmoid is dropped from the architecture, does performance worsen? It would be nice to see some discussion of the degree to which this is helpful because it reverses information loss due to binarization, vs. introduces a new architectural feature which is itself helpful.
Add a sentence describing what "double skip connections" are. I wasn't familiar with this phrase.
eq. 2:
This functional form is pretty weird.
Why is Q a square norm rather than a norm? Square error on an already-squared property is an unusual choice.
Why is the denominator itself a norm? Taking the norm of a square norm is similarly an unusual choice. (eg, why not just take an average or sum over Q)
Say what Q_s and Q_t are (student and teacher network from context)
"thus, at test time, these scaling factors will not be fixed but rather inferred from data" nit: Would not generally call this an inference process. "Inference" typically refers to values that are computed indirectly (eg by Bayesian reasoning), while in this case the values are computed directly. Would rather say that scaling factors are a function of data, or are determined by data, or similar.
"By doing so, more than 1/3 of the remaining gap with the real-valued network is bridged." text is shifting back and forth between using % and fractional gap to describe benefits. Would just use one measure consistently.
Computational cost analysis:
This is very useful.
Note though that ResNet is a very wasteful architecture in terms of compute! It would be good to include a comparison to imagenet architectures that have computational and memory efficiency as a design goal. (eg, MobileNet comes to mind)
Very nice on the ablation studies.


In this work, we present a novel upper bound of target error to address a major problem faced by many unsupervised domain adaptation methods. Recent studies reveal that a deep neural network can learn transferable features which generalize well to novel tasks. Furthermore, a theory proposed by Ben-David et al. (2010) provides an upper bound for target error when transferring the knowledge, which can be summarized as minimizing the source error and distance between marginal distributions simultaneously. However, common methods based on the theory usually ignore the joint error such that samples from different classes might be mixed together when matching marginal distribution. And in such case, no matter how we minimize the marginal discrepancy, the target error is not bounded due to an increasing joint error. To tackles this problem, we propose a general upper bound taking joint error into account, such that the undesirable case can be properly penalized. In addition, we utilize constrained hypothesis space to further formalize a tighter bound as well as a novel cross margin discrepancy to measure the dissimilarity between hypotheses which alleviates instability during adversarial learning. Extensive empirical evidence shows that our proposal outperforms related approaches in image classification error rates on standard domain adaptation benchmarks.

Summary
-------
This paper presents a novel theoretical analysis for unsupervised domain adaptation by revisiting the \lambda joint error term charactering adaptability in the seminal analysis of Ben-David et al. They propose to replace it by considering discrepancy between information on constrained class hypothesis and a possible discrepancy term that related the learned model with the class A cross-margin discrepancy is proposed in the multiclass context. They extend their approach by proposing to reweight differently some errors with an expected performance of target models on source and an additional one where they distinguish the performance with respect to accuracy on pseudo-labeled target data. Their approach leads to an adversarial-based loss function to optimize which is evaluated on two visual domain adaptation tasks.
Evaluation
--------
The idea is novel and I find the discussion on the considered restriction on the hypothesis class interesting. The experimental evaluation is interesting with good results reported on the two problems considered. I think that some parts could be improved in terms of presentation, in particular The paper contains many typos that make sometimes the reading difficult.
Other comments
------------
-The comparison with other existing bounds is interesting. I think the authors should expand this by also taking into consideration the bound of Mansour et al., COLT'09 which has some links with the proposed approach (check their comparison to Ben-David's bound).
The links with this bound and the one of Ben-David could also be summarized in an appendix, I would like to see a better characterization of the cases where the proposed bound is better and worse.
-About original proposal. I am wondering if the authors could discuss the relationship between the value of \gamma and the expressiveness of the considered model. If the model is powerful enough, \gamma should certainly be large. In a context of a training with "Learning without forgetting" strategy, the performance on source could maintained in a high standard.
Note that in this context, the classic joint error of Ben-David et al. can be considered as rather small.
-About the alternative proposal: I am a bit skeptical on the ration \eta and (1-\eta) for accuracy on source and pseudo-labeled data respectively. Indeed, since the pseudo-labels are obtained from a classifier that make use a lot of source information, one may think that their performance is rather related. So using to correlated ratios would probably be more relevant here, but maybe the authors can bring some arguments against.
-In the experimental evaluation, the tuning of the different parameters, in particular \gamma and \mu, is not particularly discussed and there is probably an issue. I tend to think that these values are rather difficult to assess. A discussion on this point would be welcomed.
-I am not sure to understand the optimization problem (8), (9) and (18), in particular the term after the "s.t.": I would expect an inequality somewhere, otherwise everything can be added to the general objective function. If there is an alternate optimization scheme, this should be mentioned explicitly.
-Table 1 and Table 2: use a third identifier different from the second for the last version of your method.
-Please check the typos.


Despite the widespread adoption of Transformer models for NLP tasks, the expressive power of these models is not well-understood. In this paper, we establish that Transformer models are universal approximators of continuous permutation equivariant sequence-to-sequence functions with compact support, which is quite surprising given the amount of shared parameters in these models. Furthermore, using positional encodings, we circumvent the restriction of permutation equivariance, and show that Transformer models can universally approximate arbitrary continuous sequence-to-sequence functions on a compact domain. Interestingly, our proof techniques clearly highlight the different roles of the self-attention and the feed-forward layers in Transformers. In particular, we prove that fixed width self-attention layers can compute contextual mappings of the input sequences, playing a key role in the universal approximation property of Transformers. Based on this insight from our analysis, we consider other simpler alternatives to selfattention layers and empirically evaluate them.

CONTRIBUTIONS:
C1. Transformers (without positional encodings and without layer normalization), with 2 attention heads of dimension 1 and feed-forward layers (FFN) with 4 hidden nodes, are universal approximators of continuous permutation-equivariant functions f of compact support, relative to any Lp metric (1 <= p < \infty). (Thm. 2, p. 3). (Without positional encodings, a function f computed by a Transformer is permutation equivariant: f(P(X)) = P(f(X)) for any permutation P of the columns of X, which are the vector encodings of the input tokens.)
C2. For transformers with trainable positional encodings, the same result holds without the restriction to permutation equivariance (Thm. 3, p. 4)
C3. The notion of ‘contextual mapping’ is introduced (Def. 5.1, p. 6); this notion is central to the proofs of the theorems in C1-C2. Such a mapping q takes any two vectors L, L’ in a finite subset of a real vector space and maps them to a vector space such that all elements (in R) of q(L) and q(L’) are distinct. (In the permutation-equivariant context, L’ must not be a permutation of L.)
RATING: Weak accept
REASONS FOR RATING (SUMMARY). ‘Accept’ because the results seem important. ‘Weak’ because the explanation for the crux of the key result is inaccessible. It relies crucially on the notion of C3, the utility of which for understanding how real Transformers work is questionable.
REVIEW
Sec. 4 does a good job of sketching the key proof. But Sec. 5 falls down at a crucial point, Lemma 6.
What I can glean from the paper (Sec 5.0) is that the idea of the proof is this: (1) use the continuity of f to approximate it with a piecewise constant function c, which takes a fixed value within each hypercube of a discretization of the compact support of f; (2) use a FFN g to map f’s input space to a discrete subset, a regular finite grid G defining the corners of the hypercube discretization; (3) use modified Transformer attention layers to create a contextual mapping k of G; (4) use a FFN h on R to map each (unique) real number in the vector outputs of k to the correct output value so that h([k(g(X))]_i) = [c(X)]_i; (5) approximate the modified Transformer attention layers of k with standard Transformer attention layers. (The modified attention layers replace softmax with argmax and replace ReLU with a (varying) piecewise-linear activation functions “with at most 3 pieces, at least one of which is constant” [Step 2, p. 4.])
The paper should focus on steps (3) and (5), the only parts of the proof that pertain to what is special about the Transformer: attention. These should be explained fully and clearly in the main text. To make room for this, the rest, concerning FFN approximation and discretization, can be reduced to a paragraph each in the main text, since they are standard, not particular to Transformers.
My attempts to understand the proofs of (3) and (5) from the Appendix were not successful. To illustrate the kind of difficulty I had in several places: in the proof of Lemma 9 on p. 12, the conclusion is “Thus, given any \bar{g} \in \bar{\cal{T}}^{2,1,1}, there exists a function g \in \cal{T}^{2,1,4} arbitrarily close to \bar{g}, by appropriately choosing the parameters to be large enough.” (“2,1,4” means a Transformer attention layer with 2 heads of dimension 1 followed by a FFN with 4 hidden units) But how do the numbers {2,1,1} and {2,1,4} relate to the equations internal to this proof? I can see some possible connections, but the authors should spell this out very clearly rather than making the reader work to fill in the gap. It should be clear how the ATTENTION MECHANISM is crucial for the proof by understanding just how that mechanism does the work needed in steps (3) and (5) above.
Given the time-consuming but ultimately unsuccessful attempt to understand the first few pages of the 11-page Appendix, I did not attempt to absorb the remaining pages.
It is clear that the notion of contextual mapping plays an important role in their proof of the universal approximation theorem, but does it play any role in the operation of Transformers in practice? Spraying the points of a grid G in the Transformer’s input space into a collection of vectors in which the same real number never appears twice is a nice step in an approximation proof, because it reduces the problem to mapping a set of unique real numbers to another set of real numbers (the elements of the output vectors). But does an actual trained Transformer in practice do anything resembling this? It seems implausible on the face of it, but perhaps there is a theoretical argument, or empirical evidence, that makes it plausible. Thus, for understanding how real Transformers actually do their work, the value of the notion of contextual mapping, and therefore the discussion at the end of the paper of alternatives to attention for achieving it, appears questionable to me. The subtitle of Sec. 5, “Demystifying Transformers”, is not clearly justified.


Deep reinforcement learning (RL) agents often fail to generalize to unseen environments (yet semantically similar to trained agents), particularly when they are trained on high-dimensional state spaces, such as images. In this paper, we propose a simple technique to improve a generalization ability of deep RL agents by introducing a randomized (convolutional) neural network that randomly perturbs input observations. It enables trained agents to adapt to new domains by learning robust features invariant across varied and randomized environments. Furthermore, we consider an inference method based on the Monte Carlo approximation to reduce the variance induced by this randomization. We demonstrate the superiority of our method across 2D CoinRun, 3D DeepMind Lab exploration and 3D robotics control tasks: it significantly outperforms various regularization and data augmentation methods for the same purpose.

This paper proposes methods to improve generalization in deep reinforcement learning with an emphasis on unseen environments. The main contribution is essentially a data augmentation technique that perturbs the input observations using a noise generated from the range space of a random convolutional network. The empirical results look impressive and demonstrate the effectiveness of the method. The experiments are thorough (includes even adversarial attack) and the core method is novel as far as I am aware.
That said, I have a couple of concerns regarding this paper and I would be willing to change my score if authors can address these.
1) Feature matching loss (Eq 2) is presented as a novel contribution without referring to related work in semisupervised learning literature. This is essentially consistency training. See:
a) Miyato, Takeru, et al. "Virtual adversarial training: a regularization method for supervised and semi-supervised learning." IEEE transactions on pattern analysis and machine intelligence 41.8 (2018): 1979-1993.
b) Xie, Qizhe, et al. "Unsupervised data augmentation." arXiv preprint arXiv:1904.12848 (2019).
2) The main contribution appears to be a data augmentation technique where we add a random neural net based perturbation to the state. My question is:
*Why don't you first evaluate this on computer vision tasks given that the core idea is data augmentation for images?*
If this technique is so powerful, shouldn't this do a great job in CIFAR10, Imagenet etc? Instead authors only provide a niche example (bright vs dark cat/dogs).
If this can compete with top augmentation techniques on Imagenet (e.g. autoagument), then it can explain the RL performance. Otherwise, please provide some intuition on why this works so well on RL but not as well on computer vision tasks. Is it the unseen environment diversity of RL challenges?
3) While proposed method performs well on the benchmarks, it is not clear whether authors compare to the state-of-the-art algorithms. For each task (CoinRun, DeepMind Lab, etc), please explicitly state the best prior result (e.g. Espeholt et al, Tobin et al, Cobbe et al etc) so that proposed method's performance can be better assessed.
-------------------------
After rebuttal: Authors addressed most of my comments. I also found the new experimental results (Fig 5 and 7) very insightful. I increase my score to Weak Accept.
For future improvement: More realistic experiments on computer vision tasks (besides cats and dogs) would be welcome. Otherwise, please justify why proposed strategy is particularly good for RL (rather than traditional computer vision benchmarks) in boosting robustness to new domains.


Evaluating the relative difficulty of widely-used benchmark datasets across time and across data modalities is important for accurately measuring progress in machine learning. To help tackle this problem, we propose DIME, an informationtheoretic DIfficulty MEasure for datasets, based on conditional entropy estimation of the sample-label distribution. Theoretically, we prove a model-agnostic and modality-agnostic lower bound on the 0-1 error by extending Fano's inequality to the common supervised learning scenario where labels are discrete and features are continuous. Empirically, we estimate this lower bound using a neural network to compute DIME. DIME can be decomposed into components attributable to the data distribution and the number of samples. DIME can also compute per-class difficulty scores. Through extensive experiments on both vision and language datasets, we show that DIME is well-aligned with empirically observed performance of state-of-the-art machine learning models. We hope that DIME can aid future dataset design and model-training strategies.

This paper suggests a measure for the inherent difficulty of datasets: DIME.  Fano's inequality gives
a lower bound on the best possible predictor in terms of the conditional entropy of the labels
given the input.  The paper uses a MINE style estimator for this conditional entropy to give dataset
complexity measures.
The idea of using Fano's inequality and tight estimates of conditional entropy to give us a sense of how good we are doing on different datasets is a good one.
Howver, I believe this paper should be rejected. While the idea of using Fano's inequality to give a complexity score for datasets is interesting, this paper does not provide what appears to be a close measure of this conditional entropy.
Any fit model's log loss also provides a variational upper bound on the conditional entropy due to the positivity of KL:
$$ \int dx\, p(x) \int dy\, p(y|x) \log \frac{p(y|x)}{q(y|x)}  \geq 0   \implies  \int dx\, p(x) \int dy\, p(y) p(y|x) \log p(y|x) \geq  \int dx\, p(x) \int dy\, p(y|x) \log q(y|x)  \implies H(Y|X) \leq \mathbb{E}_{p(y,x)}[-\log q(y|x)]$$
The log loss of any model provides a variational upper bound on the conditional entropy H(Y|X).  In table 1, for all but EMNIST there exist models that give tighter upper bounds on the conditional entropy than DIME does.  While in principle an independent estimate of the conditional entropy might provide a useful signal that there is room to go in terms of the classification task (as it seems to do here in the case of EMNIST), in general the estimates provided by their MINE style estimator seem to not perform well.
Additionally, MINE does not provide a valid bound on the conditional entropy, (see e.g. "On variational bounds of mutual information" arXiv:1905.06922) as it when used as suggested it requires taking a monte carlo expectation inside a negative log.  A monte carlo estimate of a log expectation provides a stochastic lower bound of the log expectation (by Jensen's) so provides a stochastic upper bound of the proposed lower bound on KL, breaking the bound.  Couple this with the observation that MINE provides very high variance estimates of mutual information, in particular when the values are large, along with the general failure of the estimates in Table I and the specific combination of using MINE to give useful upper bounds on conditional entropy seems not to work well in practice, even on datasets with rather small expected conditional entropy (e.g. MNIST).  I suggest leveraging the known marginal in this instance to use an IWHVI style mutual information lower bound, as in
http://artem.sobolev.name/posts/2019-08-10-thoughts-on-mutual-information-more-estimators.html
The paper extends Fano's inequality to the mixed discrete-continuous case.  I always thought Fano's inequality would just carry over to the mixed discrete-continuous case because it doesn't rely at all on the cardinality of the input (X in this case).  Shouldn't a simple sup over all finite partitions of X give a natural extension (as it does to mutual information and conditional entropy in Cover & Thomas).  While there is utility in a careful extension or even clarifying note on the role of Fano's inequality in the mixed case, it's not clear it requires an entire paper, perhaps a note on the arxiv would help others who were interested in the extension.
In general, it doesn't seem as though the suggested method provides utility in giving better estimates of the difficulty of datasets than can already be gotten from our observed model log losses.  The MINE estimator is flawed and using it to estimate DIME is likely (and demonstrated in the paper) to give worse estimates than just training a conditional discriminative model and comparable cost, so I have to vote to reject the paper in its current form.
I do still think it is an interesting idea to try to assess whether or not we are within spitting distance of the optimal performance we could expect on our datasets.  To provide a proper lower bound would require a lower bound on the conditional entropy, for which I'm not sure of any general purpose results that could be leveraged, but in these settings it's not too dangerous to assume we have access to at least a known marginal over the labels and there is some hope that a decent lower bound on the conditional entropy might be able to be formed in this case.   For upper bounds, I really believe just the log loss of the best model the community has garnered is going to provide the tightest upper bound in almost all cases.   Could we still use this to figure out whether or not we have room to go?  While I have observed that in general for most models there isn't the best correlation between log loss and accuracy, the result of Fano's inequality suggests that when we really nail a dataset we might expect that the log loss provides a very tight upper bound on the true conditional entropy and the observed error rates provides a tight upper bound to the optimal Fano rate.  This suggest that it might be useful to collect paired results of the log loss and error from a whole slew of models and scatter the log loss versus ( H(e) + P(e) log (|Y|-1) ).   Then as just a sort of visual test as to whether the community is within spitting distance of the true discriminative density would be whether the best models start to show a linear relationship between their log loss and the transformed error rate.


This paper studies the lower bound complexity for the optimization problem whose objective function is the average of n individual smooth convex functions. We consider the algorithm which gets access to gradient and proximal oracle for each individual component. For the strongly-convex case, we prove such an algorithm can not reach an ε-suboptimal point in fewer than Ω((n + √ κn) log(1/ε)) iterations, where κ is the condition number of the objective function. This lower bound is tighter than previous results and perfectly matches the upper bound of the existing proximal incremental first-order oracle algorithm Point-SAGA. We develop a novel construction to show the above result, which partitions the tridiagonal matrix of classical examples into n groups to make the problem difficult enough to stochastic algorithms. This construction is friendly to the analysis of proximal oracle and also could be used in general convex and average smooth cases naturally.

** Summary
The paper derives a novel lower bound on the complexity of optimizing finite-sum convex functions (under different assumptions) using algorithms that have access to point-wise evaluation of the function, its gradient, and proximal information.
** Overall evaluation
Finite-sum convex functions are very common in machine learning problems and how the optimization complexity scales with their properties (e.g., condition number) and the number of components (e.g., number of samples in typical ML problems) is a very important question. This paper addresses the question from a lower bound point of view, showing that there is no proximal incremental first-order algorithm that can optimize such functions at an accuracy level of epsilon in less than a term which depends linearly with number of components n and sqrt(k) (k being the condition number). The paper fills an existing gap in the literature and it achieves two very interesting results:
1- The lower bound now matches an existing upper bound for Point-SAGA, showing that no better algorithm can exist (at least in a worst-case sense).
2- This result also illustrate that proximal algorithms are not necessarily more powerful than first-order methods that only access the gradient of the function. This is also very interesting, as it was still an open question whether proximal information could possibly give an advantage.
The paper is also well written, although some elements could be improved:
1- Def 2.4: the authors consider algorithms where the sampling distribution cannot adapt through iterations. Although this is standard, I am wondering whether adaptivity may buy anything in the performance or whether the lower bound applies to adaptive algorithms as well.
2- Although similar constructions to create worst-case functions were used before in deriving complexity lower bounds, it would be useful to have an intuition about the specific choice made in eg Eq.5/6 and how this enables the refined analysis presented in the paper.
3- More in general, I encourage the authors to illustrate how their techniques compare and differ from previous lower bound proofs.
4- In all theorems, the analysis is done by linking the dimension d to all other parameters of the problem. As pointed out by the authors, the requirements on the dimensionality in the theorems of this paper are milder than previous results. It would be helpful to illustrate how the lower bound would behave when the dimensionality changes and provide an intuition about the specific choice in the theorems