We introduce a method to create Universal Adversarial Perturbations (UAP) for a given CNN in a data-free manner. Data-free approaches suite scenarios where the original training data is unavailable for crafting adversaries. We show that the adversary generation with full training data can be approximated to a formulation without data. This is realized through a sequential optimization of the adversarial perturbation with the proposed dilate loss. Dilate loss basically maximizes the Euclidean norm of the output before nonlinearity at any layer. By doing so, the perturbation constrains the ReLU activation function at every layer to act roughly linear for data points and thus eliminate the dependency on data for crafting UAPs. Extensive experiments demonstrate that our method not only has theoretical support, but achieves higher fooling rate than the existing data-free work. Furthermore, we evidence improvement in limited data cases as well.

This paper proposed a white-box (known network architecture, known network weight) data free (without need to access the data) adversarial attacking method. The main idea is to find a perturbation that maximizes the activations at different layers jointly. But the optimization is done sequentially, treating each layer’s activation (before ReLU) as a linear transformation output.
The method is compared with existing methods (only one existing approach for the problem, GDUAP by Mopuri et al. 2018) in terms of the fool rate. It shows significant improvement. Ablation study is carried out to compare with baselines like perturbation maximizing only first layer activation, only last layer activation, etc. Also on some other settings (black-box testing, less data) the proposed method outperforms GDUAP.
The problem of data-free white-box attack is very interesting and does make sense. The proposed method achieve significant improvement over the previous one (GDUAP). I do have the following concerns though.
1), the novelty of the proposed idea seems relatively limited. The proposed idea seeks perturbation maximizing activations over all layers. It incur perturbation before ReLU. But overall, the flavor of the idea is not significantly different from GDUAP, despite the significant performance boost.
2), it was mentioned that compare with GDUAP, this paper has more theoretical analysis. But this is not very convincing to me. There are many steps of approximation/relaxation from the original problem (Equation (1)) to the final formula (Equation (10)). Many assumptions are made over the steps. It is OK to use these steps to derive a heuristic. But these steps can hardly be called "theoretical analysis".
I am particularly uncomfortable with Equation (5), which is the basis of the main idea. It assumes that all data in $W_1X$ are in the same orthant as $W_1p$. But this is unrealistic as different data in X will for sure incur different activation patterns. Did I misunderstand anything?
3) I do like the experimental results. It looks impressive. But the baselines are really limited (granted, there are not many existing approaches). There is only one task (image classification). How about other tasks like segmentation etc shown in Mopuri et al. 2018? Also it would be nice to also show the results of other UAP methods, as it gives us a better sense of the gap between with and without data.
4) I wonder how will the attack affect some model which has been trained with some defense mechanism, e.g., adversarial training.
Typo:
Equation (5), RHS missing a max


Imitation learning algorithms provide a simple and straightforward approach for training control policies via standard supervised learning methods. By maximizing the likelihood of good actions provided by an expert demonstrator, supervised imitation learning can produce effective policies without the algorithmic complexities and optimization challenges of reinforcement learning, at the cost of requiring an expert demonstrator -typically a person -to provide the demonstrations. In this paper, we ask: can we use imitation learning to train effective policies without any expert demonstrations? The key observation that makes this possible is that, in the multi-task setting, trajectories that are generated by a suboptimal policy can still serve as optimal examples for other tasks. In particular, in the setting where the tasks correspond to different goals, every trajectory is a successful demonstration for the state that it actually reaches. Informed by this observation, we propose a very simple algorithm for learning behaviors without any demonstrations, user-provided reward functions, or complex reinforcement learning methods. Our method simply maximizes the likelihood of actions the agent actually took in its own previous rollouts, conditioned on the goal being the state that it actually reached. Although related variants of this approach have been proposed previously in imitation learning settings with example demonstrations, we present the first instance of this approach as a method for learning goal-reaching policies entirely from scratch. We present a theoretical result linking self-supervised imitation learning and reinforcement learning, and empirical results showing that it performs competitively with more complex reinforcement learning methods on a range of challenging goal reaching problems.

This work presents the goal-conditioned supervised learning algorithm (GCSL), which learns goal conditioned policies using only behavioral-cloning of the agent's own actions.  The intuition behind the algorithm is the goal of an observed trajectory can be identified after the fact, by simply looking at the states reached during that trajectory.  GCSL treats each executed action as a sample from the expert policy conditioned on each of the states reached after that action is taken.  Given a distribution over goal states, GCSL alternates between executing its current goal-conditioned policy on randomly selected goals, and learning to imitate the generated actions conditioned on the states they actually reached.  Experimental results demonstrate superior performance against a base (non-goal conditioned) RL algorithm (TRPO), and against another approach to learning goal-conditioned polices (TD3-HER), on a relatively diverse set of control problems.
A major issue is that the proof of the main theoretical result appears to be wrong.  As there don't appear to be any constraints placed on the policy pi_old, it would seem that the surrogate loss would collapse to 0 for any policy pi if pi_old is such that the target goal is never reached (the probability of any trajectory t reaching g is 0 under pi_old(t|g)).  It seems to be the case that the quality of the GCSL loss depends on the relationship between pi_old and the goal distribution p(g).  The fact that the theoretical results are incorrect does not mean that the algorithm, or the general approach do not have value, but it does highlight the fact that this approach may only be effective for a specific class of problems similar to the experimental domains.
While not a flaw in the work itself, it should be made clear in the text that the notion of optimality for the learning tasks considered in this work (i.e. achieving the goal by the end of episode), avoids one of the apparent limitations of the algorithm.  A randomly generated trajectory is itself optimal for any state that it reaches, if we define optimality as simply reaching a state.  Such a trajectory may not be the most efficient way of reaching that state however, so the relabelling process would seem to be prone to learning policies that achieve the conditioned goals, but not doing so in an efficient manner.  It isn't clear how well this approach would work for tasks where the efficiency, in terms of the time required to reach the objective, is a key part of the evaluation.  Again, this is not a flaw in the work itself, and it is possible that the algorithm will be effective in such tasks, perhaps because the likelihood of an action resulting in a given state is higher if that action brings us closer to this state.  It might be useful to conduct some additional experiments where evaluation is based on the time required to solve a task, rather than just the accuracy of the final state.


Model-based Reinforcement Learning(RL) has shown great advantage in sampleefficiency, but suffers from poor asymptotic performance and high inference cost. A promising direction is to combine model-based reinforcement learning with model-free reinforcement learning, such as model-based value expansion(MVE). However, the previous methods do not take into account the stochastic character of the environment, thus still suffers from higher function approximation errors. As a result, they tend to fall behind the best model-free algorithms in some challenging scenarios. We propose a novel Hybrid-RL method, which is developed from MVE, namely the Risk Averse Value Expansion(RAVE). In the proposed method, we use an ensemble of probabilistic models for environment modeling to generate imaginative rollouts, based on which we further introduce the aversion of risks by seeking the lower confidence bound of the estimation. Experiments on different environments including MuJoCo and robo-school show that RAVE yields state-of-the-art performance. Also we found that it greatly prevented some catastrophic consequences such as falling down and thus reduced the variance of the rewards.

Summary
This paper expands on previous work on hybrid model-based and model-free reinforcement learning. Specifically, it expands on the ideas in Model-based Value Expansion (MVE) and Stochastic Ensemble Value Expansion (STEVE) with a dynamically-scaled variance bias term to increase risk aversion over the course of learning, which the authors call Risk Averse Value Expansion (RAVE). Experimental results indicate notable improvements over their selected model-free and hybrid RL baselines on continuous control tasks in terms of initial learning efficiency (how many environment steps are needed to achieve a particular level of performance), asymptotic performance (how high the performance is given the same large number of environment steps), and avoidance of negative outcomes (how infrequently major negative outcomes are encountered over the course of training).
Review
The core contribution of the paper is an extension to STEVE that uses an idea from risk-averse RL of biasing the underlying estimators away from high-variance predictions, and adds a dynamic weight to that bias.
Strengths
- The addition of a risk-aversion term to STEVE is a good contribution to the literature on safe RL. While it may be possible to criticize this contribution as somewhat trivial, I am disinclined to do so, as finding simple ideas that are effective is the hallmark of an important contribution, in my opinion.
- Under the assumption that the baselines are fair, the empirical results show substantial improvements for a good selection of challenging tasks along three metrics: initial efficiency, asymptotic performance, and avoidance of negative outcomes.
- The paper provides a careful and detailed section on the relevant preliminaries, giving precise notation that is expanded step-by-step from basic actor-critic approaches all the way through the description of RAVE. (Although see my comment below about possibly making the notation more concise.)
- Overall, the paper’s presentation is clear and natural. (Although see my comment below about a strong need for proofreading.)
Weaknesses
- The paper opens with a claim that MBRL has worse performance than MFRL in environments with “noisy environments and long trajectories”. However, recent work [1] has shown that MBRL can outperform MFRL on many of the tasks considered in this paper using far fewer environment steps, even when training directly from pixels.
- The experiments are all based on observations of the true state vectors rather than pixels, which dramatically simplifies working with imagination rollouts, as the dimensionality is so much lower. This hides the high computational cost and modeling challenges involved in using imagination-based MBRL or hybrid RL approaches in real-world environments, and (in my opinion) is a major shortcoming of this line of research (not only of this paper). Since the DDPG family of algorithms doesn’t have to do rollouts, they have a strong advantage over this type of approach on more realistic settings where the true state is unknown and must be inferred from high-dimensional observations. If experiments are not going to be done using pixels, the discussion should at least mention the trade-offs involved between the proposed algorithm and the baselines in that setting.
- The core baseline of the paper is DDPG, which is unnecessarily weak. D4PG [2] came out a year and a half ago and has the same high-level properties as DDPG, but outperforms it on all of the Mujoco tasks considered in this paper. Additionally, it includes some of the ideas presented in this work, but in a model-free setting, including a distributional treatment and multi-step rollouts, so it is easy to imagine that some of the experimental gains presented here would be erased when using it as a baseline.
- The use of ensembling is another confounder in the experiments. Ensembling models almost always yields an improvement, so any technique that relies on some form of ensembling needs to additionally demonstrate that the gains presented are not solely due to ensembling. In this case, the experiments with STEVE and RAVE should minimally be performed with different numbers of models in their ensembles.
- (Minor) Similarly, the prediction horizon can have a large impact, and only STEVE and RAVE consider prediction horizons other than 1. Showing how the horizon affects performance would help make the comparisons more fair.
- Figure 1 shows a bias problem in some current approaches in a clear toy problem, but does not show whether RAVE addresses that problem. Even though this section precedes the presentation of RAVE, it is necessary to show that the proposed solution actually helps on the toy problem.
- There are a number of problems with the proposal for adaptively computing the alpha parameter. In general, such adaptive hyperparameters add a great deal of complexity to hyperparameter tuning, so such suggestions should be either strongly motivated by theory or by empirical results. Neither seems to be the case here.
  - Only one proposal for adaptive alpha (equation 13) is considered. Its justification is plausible, but it would be more convincing if other dynamic approaches were considered in the experiments. For example:
    - alpha(env_step) = alpha * env_step / max_env_step
    - alpha(s_t,a_t) = alpha * min{1.0, 1 / (Z * ||E[fhat(s_t,a_t)] - s_t+1||^2)}
    - The opposite of the proposed approach, where alpha starts high and gets lower over the course of training.
  - Indeed, a potentially useful quantity during evaluation, when s_t+1 is unknown, would be based on the variance of the predicted next state, rather than the difference of the expectation of the predicted next state and the true next state. E.g.:
    - alpha(s_t,a_t) = alpha / Var[fhat(s_t,a_t)]
    This formulation says that the precision of the prediction determines the confidence of the model, which is also intuitively reasonable (to me, anyway) and doesn’t rely on knowing the future.
  - (Minor) The adaptive variant appears to be only slightly better than alpha=0.5, but requires two hyperparameters that are unspecified in the main body of the paper — alpha and Z. The appendix lists those parameters, but no discussion is made on how much tuning was done to determine that pair of parameters.
  - (Minor) A plot of how the dynamic value of alpha changes during training would be useful, at least in the appendix.
Recommendation
In light of my comments above, I cannot currently recommend the acceptance of this paper at ICLR. However, I think that the core idea is likely to hold up under more careful experimental comparisons. If the authors submit a revised draft that addresses the substance of my concerns, I would be very likely to increase my rating. In particular, I would like to see much more careful experimental treatment of the idea, so that readers can have high confidence about the circumstances where RAVE is likely to be a good choice.
[1] Hafner et al., “Learning Latent Dynics for Planning from Pixels”. ICML 2019. https://arxiv.org/abs/1811.04551
[2] Barth-Maron et al., “Distributed Distributional Deterministic Policy Gradients”. ICLR 2018. https://arxiv.org/abs/1804.08617
Other Comments and Questions
- This paper needs a lot of proofreading. A few examples of errors that should be fixed before publication:
  - “equation. 2”: This should read “Equation 2”, “equation 2”, “Eq. 2” or “eq. 2”. The period in the last two options indicates that letters have been elided. The mistake of using a period where no elision has happened occurs throughout the paper.
  - “prophesy”: This is not the correct word. Just say “prediction”.
  - “image”: This is often used when the correct word would be either “imagine” or “imagination”.
  There are many other errors that could be fixed easily with the help of a native English speaker.
- The mathematical notation in sections 3 through 5 is precise, but it is also a bit heavy. Consider whether there would be any ambiguity added if, for example, {\hat Q}^DVE_{\zeta_s,\zeta_r,\zeta_d,\theta’,\phi’} were instead notated {\hat Q}^DVE_{\zeta,\theta’,\phi’}.
- Figure 2 adds nothing of value to the paper and should be removed.
- Consider comparing on DMControl, which is the same set of tasks as Mujoco, but the scores are standardized such that each task has a maximum reward of 1000 per episode.
- Why does DDPG get worse over time on the RoboSchool tasks? Without a clear explanation, it looks like a bug, and a bug like that calls into question the rest of the DDPG results as well.


Combining clustering and representation learning is one of the most promising approaches for unsupervised learning of deep neural networks. However, doing so naively leads to ill posed learning problems with degenerate solutions. In this paper, we propose a novel and principled learning formulation that addresses these issues. The method is obtained by maximizing the information between labels and input data indices. We show that this criterion extends standard crossentropy minimization to an optimal transport problem, which we solve efficiently for millions of input images and thousands of labels using a fast variant of the Sinkhorn-Knopp algorithm. The resulting method is able to self-label visual data so as to train highly competitive image representations without manual labels. Our method achieves state of the art representation learning performance for AlexNet and ResNet-50 on SVHN, CIFAR-10, CIFAR-100 and ImageNet.

Summary:
The paper proposes a self-supervised learning procedure to train deep neural networks within an unsupervised learning setting. The authors build their work on a pretext task that consists in maximizing the information between the input data samples and labels that are basically obtained by a self-labeling procedure that clusters them in K distinct classes. This is similar to what was done in DeepCluster, a previous self-supervised algorithm that self-labels samples through clustering. However, differently from that approach, the current method does not introduce any additional clustering cost functions. Instead it implicitly achieves self-labeling by simply adding the constraint that label assignments equally partition the dataset. This constraint acts as a "regularizer" that allows the authors to minimize the cross-entropy loss between inputs and pseudo-labels while avoiding the degenerate trivial solution where all samples are assigned to the same pseudo-label. As a result, the authors are able to derive a self-labeling method that optimizes the same cross-entropy loss as the classification task. That is done by remarking that minimizing the loss function over the pseudo-label assignments (under the equal partition constraint) can be formulated as an optimal transport problem that can be solved efficiently with a fast version of the Sinkhorn-Knopp algorithm. In practice, what the authors do at training time is to alternate between 1) minimizing the average cross-entropy loss by training the neural network (feature extractor + linear classification head) given a fixed pseudo-labels assignment, and 2) optimizing the pseudo-label assignments implemented as a matrix Q of posterior distribution of labels given the sample index, which, as said, can be done efficiently with a KL-regularized version of Sinkhorn. Moreover, this last step can be carried out simultaneously for multiple distinct classification heads (with possibly different number of labels), each sharing the same feature extractor but inducing a different matrix Q. At this point, the number of classification heads can be treated as a hyperparameter of the algorithm.
The authors then go on to show that this new algorithm is competitive with current state of the art method with several architectures in terms of providing a good feature extractor for downstream image classification, detection and segmentation tasks. They for instance consistently beat DeepCluster, the main direct competitor, on classification and detection tasks.
They also conduct ablation studies that provide interesting insights on the functioning of their algorithms and the effects of the multiple classification heads, the number of clusters, and the quality of the learned assignment. Intriguingly, on ImageNet they find for example that AlexNet obtains better performance at validation when it's trained from scratch on labels obtained with their self-labeling procedure, as opposed to the original labels.
Decision:
In my opinion this paper should be a clear accept. The paper is well written, presents an elegant idea in a clear and straight-forward manner, and is solidly built on top of the current literature on self-supervised learning for image processing, which is also very well summarized.
The feature extractors obtained with the proposed algorithm are convincingly tested and validated on several downstream tasks (like classification on ImageNet, PascalVOC classification, detection and segmentation), and that is done for several base architectures, obtaining performances that are competitive with state-of-the-art. In addition, a series of careful ablation studies help in gleaning some scientific understanding on the method.
Minor comments:
- The authors refer to traditional clustering like k-means as being "generative", which a little confusing. Clustering algorithms can be derived within a probabilistic framework by positing a generative model of the data, however, strictly speaking, k-means is not by itself is not a generative approach. It's a minor point, but it could be helpful to be more precise about this, in order to avoid possible confusion. The main point that the authors want to make in this regard is that their framework eschews having to posit an additional clustering cost function.


Recent breakthroughs of pretrained language models have shown the effectiveness of self-supervised learning for a wide range of natural language processing (NLP) tasks. In addition to standard syntactic and semantic NLP tasks, pretrained models achieve strong improvements on tasks that involve real-world knowledge, suggesting that large-scale language modeling could be an implicit method to capture knowledge. In this work, we further investigate the extent to which pretrained models such as BERT capture knowledge using a zero-shot fact completion task. Moreover, we propose a simple yet effective weakly supervised pretraining objective, which explicitly forces the model to incorporate knowledge about real-world entities. Models trained with our new objective yield significant improvements on the fact completion task. When applied to downstream tasks, our model consistently outperforms BERT on four entity-related question answering datasets (i.e., WebQuestions, TriviaQA, SearchQA and Quasar-T) with an average 2.7 F1 improvements and a standard fine-grained entity typing dataset (i.e., FIGER) with 5.7 accuracy gains.

This paper aims to incorporate world knowledge for the pretraining approach so that (1) pretrained models contain useful information about the world, and (2) benefit downstream NLP tasks. The paper does so by introducing the objective which distinguishes the groundtruth entity and the false entity in the Wikipedia text. This was carefully done by detecting entities in the text, find the corresponding entity in Wikidata, randomly choose another entity which has the same type as the original entity, and make sure this doesn’t happen for neighboring entities or too much in order to avoid context change. Adding this objective to the original masked LM objective, this pretrained model is shown to be effective and outperform baselines significantly in many tasks such as zero-shot fact completion, question answering, and fine-grained entity typing.

Particularly, I appreciate (1) the fact that they compare with other knowledge-aware pretrained models such as ERNIE, and (2) their ablation in Table 6 which compares only knowledge learning objective, more masking or finetuning with knowledge learning objective starting from MLM instead of multi-task learning (actually, is BERT+1M MLM updates a correct term? Since MLM is masked LM, shouldn’t it be BERT+1M knowledge learning?): it clearly shows that it is important to do both MLM and knowledge learning, appropriate ratio for masking, and multi-task learning between MLM and knowledge learning instead of some kind of pretrain-finetune approach.
Some marginal concerns I have is that some settings for downstream tasks are not clear. For example, for WebQuestions the number of validation examples is not specified, but I believe they should have split the train set into train/valid for development (such as early stopping or hyperparameter tuning), and conventionally people have split the train set into 90/10 for train/valid. In addition, as far as I know, TriviaQA has a bunch of settings such as Wiki setting, Web setting, unfiltered setting and open setting. What exactly is this setting? The paper mentions they follow Lin et al. 2018, but the statistics shown in Table 2 are different from their statistics. In addition, the authors compare results in different settings in Table 4 (for example, ORQA is in open setting, while I think the setting in this paper is not).
Some clarification questions:
1. I understand Wikipedia anchor links means hyperlinks. How do you know that the mention with a hyperlink is always an entity? Also, there are many cases that the mention with the hyperlink does not match in meaning with the linked article. For example, in “https://en.wikipedia.org/wiki/Barack_Obama”, there is a sentence “He was elected over Republican John McCain and was inaugurated on January 20, 2009” where “elected” is linked to “https://en.wikipedia.org/wiki/2008_United_States_presidential_election” which is probably not what we want.
2. If there are same mentions in the text chunk, are all of them replaced? Probably with the same negative entity?
3. How often were entities to replace chosen? E.g. is one entity in the text chunk replaced at a time? Or multiple ones?
4. How do you look up the type of the entity? For instance, in the example of “Marvel Comics” illustrated in Figure 1, I see there are two triples, <Marvel Comics, instance of, business> and <Marvel Comics, instance of, book publishing company>. Is "instance of" used? How did you choose “book publishing company” over “business”? Or is it randomly chosen? I think ideally more fine-grained type should be chosen, but wonder if there is a way to find which one is more fine-grained.
5. Perhaps not necessary, but I wonder what happens if an entity is detected, and there is another masked LM objective which only predicts this entity, without the process of looking up its type and randomly choose the negative entity.
- I’m curious because I’m not sure how much effect do negative entities have. Although they are the same type of entities, it’s hard that the chosen entity is a strong negative entity. For instance, Barack Obama (Q76) is an instance of ‘human’, and I don’t think another entity of human that is randomly chosen is helpful. A similar observation was found in WikiHop dataset, which is a multi-choice QA dataset that all negative candidates are guaranteed to be the same type, but still question-candidates-only baseline (without context paragraphs) outperforms state-of-the-art models (https://openreview.net/forum?id=B1lf43A5Y7). It indicates that entities with the same type are not strong negative candidates.
- For this reason, I think it is possible that negative entities do not help, but the fact that the training objective is more entity-centric helps the performance on various different downstream tasks. That’s why I think the ablation with entry-centric masked LM can be good to see.
- Of course, this is just one of my hypotheses, and I believe the results in this paper are significant regardless, but I just wonder what authors think about this.


Environments in Reinforcement Learning (RL) are usually only partially observable. To address this problem, a possible solution is to provide the agent with information about past observations. While common methods represent this history using a Recurrent Neural Network (RNN), in this paper we propose an alternative representation which is based on the record of the past events observed in a given episode. Inspired by human memory, these events describe only important changes in the environment and, in our approach, are automatically discovered using self-supervision. We evaluate our history representation method using two challenging RL benchmarks: some games of the Atari-57 suite and the 3D environment Obstacle Tower. Using these benchmarks we show the advantage of our solution with respect to common approaches.

The paper focuses on the problem learning state representations that can effectively capture historical information
in POMDPs. Specifically, the paper proposes an alternative approach to using RNN's for capturing such history - the authors adopt a variant of recently proposed Invariant Information Clustering approach to discover important events in past observation and then use these events to learn a probability distribution over observations to represent the state information. The goal here is to address the unstable and inefficient training issues associated with RNN based architectures. The authors validate their approach with experiments on seven tasks on Atari 57 benchmark and Obstacle Tower, both with discrete action space and compare their performance against both original and RNN versions of PPO.
The paper addresses an interesting problem to help learning better state representation in absence of  complete information about the environment. The approach of using IIC clustering (or any clustering approach) for learning state representations is novel. The overall goal of replacing RNN based methods in order to achieve stable, efficient learning in presence of budget constraints is very useful and hence this approach is potentially a good step in that direction. Although not adequate, the results in figure 3 provides good insight into effectiveness of the method in making training easier.
However, I am inclining to reject this paper for the following reasons:
(1) The motivation for using proposed clustering approach for history representation is neither clear  and nor well exposed. MI based methods are inherently difficult to learn and hence this approach needs  rigorous analysis on why it works when it does and how it fails.
(2) The paper fails to position the new approach in comparison to related works both in discussions and experiments.
(3) The experiments are very limited in nature and fails to demonstrate the efficacy of the proposed approach effectively. Further, the key contribution focusing on learning effective representations under constrained budget is not adequately tested.
Major concerns:
Motivation
-------------
- It is not clear how the proposed clustering mechanism to discover events allows to successfully capture information that an RNN based approach does. For instance, RNN helps to capture long-term dependencies but it is very hard to interpret the proposed model from that aspect. Also, why is this particular clustering approach (Invariant Information Clustering) chosen? The motivation for using this approach is not clear and overall combination appears adhoc.
- Further, RNN based methods can retain order information in sequential history of observations. However,  this method appears to not consider that information. Is this is true or am I mistaken here? If this is true, why would this not create issues in learning good representations for task where order is indeed important?
- The paper discusses several articles that provide background on the methods used however it fails to position the exposition in comparison to existing RNN based approaches which is a big miss as the goal of the paper is to replace RNN based approaches [1,2,3,4].
Method
------
- In the event discovery stage, it is not clear why using consecutive observations is not useful. Also, if one does use L=1 (consecutive observations), can one reduce this method to RNN as you end of capturing all the previous history? Also, why L=3 is good across all different tasks? does it have any relation with the use of 4 frames in I(t)?
- The authors mention tat H(t) matrix is highly sparse but also low-dimensional in all their experiments.  Would this is be the case for any other task? If not, is this a limitation of the method that you need S and C  to be small?
- The authors attempt to use clustering based approach with the hope of recovering important information, however
mention that H(t) stores all the past events. Isn't this contradictory? Also, why can RNN with an attention based mechanism not achieve similar effect?
- It is also useful to analyse how will this method work in presence of long vs short history? Will the clustering itself and hence the learned representations get affected by length of available history?
Experiments
----------
- Focusing only on PPO and designing an RNN version of PPO constrains the effectiveness of experiments in validating the approach. Authors mention difficulty of training with RNN as one reason for PPO with RNN's under performance but this is not convincing Could the performance of PPO with RNN be limited only due to specific RNN architecture used?
- Authors must compare with other methods that use RNN approaches (e.g. [1]). Also, if methods such [2],[3],[4] are not  directly applicable, they must atleast, use their RNN based architectures to modify PPO and compare several baselines.
- Why do the authors not report PPO with RNN for Obstacle Tower?
- For the experiments, authors use a specific 10000 steps budget but this seems to be highly curated. The experiments
would be stronger if the authors show experiment over a range of budget. This will also give insights on when does RNN becomes better and is there a budget after which both methods perform equally well or RNN based approaches
surpass the current approaches.
- Figure 3: Training dynamics seem to favor RNN approach for BreakOut, Gravitar. Do the authors have insight on why
this is the case?

Minor points to improve submission not affecting the score:
- The paper needs to be proofread for various typos and sentence construction issues
- Table 1: For Qbert, original PPO (Sch.) is best performing, not EDHR
- [5] talks about difficulties in MI based methods and I encourage the authors to look at the analysis in
this paper. As it is only an arXiv version and not published yet, I have not based my assessment on the
existence of this paper but still connection to such analysis will make this paper stronger.
[1] Deep Variational Reinforcement Learning for POMDPs, Igl et. al.
[2] On improving deep reinforcement learning for POMDPs, Zhu et. al.
[3] Policy Learning with continuous memory states in partially observed robotic control, Zhang et. al.
[4] Memory-based control with recurrent neural networks, Heess et. al.
[5] On Mutual Information Maximization and Representation Learning, Tschannen et. al.


Mutual Information (MI) plays an important role in representation learning. However, MI is unfortunately intractable in continuous and high-dimensional settings. Recent advances establish tractable and scalable MI estimators to discover useful representation. However, most of the existing methods are not capable of providing an accurate estimation of MI with low-variance when the MI is large. We argue that directly estimating the gradients of MI is more appealing for representation learning than estimating MI in itself. To this end, we propose the Mutual Information Gradient Estimator (MIGE) for representation learning based on the score estimation of implicit distributions. MIGE exhibits a tight and smooth gradient estimation of MI in the high-dimensional and large-MI settings. We expand the applications of MIGE in both unsupervised learning of deep representations based on InfoMax and the Information Bottleneck method. Experimental results have indicated the significant performance improvement in learning useful representation.

This paper proposes MIGE---a novel estimator of the mutual information (MI) gradient, based on estimating the score function of an implicit distribution. To this end, the authors employ the spectral Stein gradient estimator (SSGE) and propose its scalable version based on random projections of the original input. The theoretical advantages of the method are presented using a toy experiment with correlated Gaussian random variables, where both the mutual information and its gradient can be computed analytically. In this setting, MIGE provides gradient estimates that are less biased and smoother than baselines. The method is also evaluated on two more complicated tasks: unsupervised representation learning on Cifar-10 and CIfar-100 via DeepInfoMax (DIM) and classification on MNIST with Information Bottleneck (IB), where MIGE outperforms all baselines by a significant margin.
I recommend ACCEPTing this paper. This work discusses a vital problem and proposes a novel, well-motivated, principled and very performant solution; additionally, it demonstrates the broad applicability of the introduced method. While the proposed technique consists of previously known building blocks (spectral Stein gradient estimator and random projections), it is cleverly applied in a novel context of estimating MI gradients.
While the paper is solid, I believe that it could be improved in the following ways. Firstly, I would like to see 1) more extensive and 2) larger-scale evaluations. In the DIM experiment, 1) would correspond to trying the DIM(L) approach, which maximises patch-wise MI. In fact, I strongly recommend including this experiment as it corresponds to and could improve current state-of-the-art. If it turns out that MIGE does not work well on DIM(L), then this would correspond to a serious issue with the method. In this experiment, 1) would also include providing other metrics for learned representations. It would be much more convincing to include estimates of true mutual information (e.g. InfoNCE bound evaluated with a large number of samples [1]) and showing that MIGE can attain higher values than baselines. 2) would correspond to evaluation on bigger datasets: (tiny) ImageNet and STL-10 dataset. Also, the toy experiment would benefit from a higher-dimensional setting (e.g. d=256 to d=1024), since these are often used in practice.
Secondly, the paper is sloppily-written, which quite a few grammar and stylistic mistakes (e.g. sentence in sec 3, paragraph 2: “we assume obtain to…”, which starts with a lower-case letter and doesn’t make sense). Finally, the paper would benefit from the following clarifications: 1) explain what is the Nystr\:om method, 2) provide either a proof or a citation for eq (19); also the error bound for SSGE should be provided for the paper to be self-contained, 3) explain the difference between q_\psi and p_\psi, which seem to be used interchangeably.
Additional remarks:
Sec 2.2, 2), “streamlining” is unclear
Circumstances 2 and 3 can be quite easily derived from circumstance 1; also they are not evaluated empirically; it would be nice to have experiments for them, and they can be moved to the appendix in case of lack of space
Eq (19) while nice, seem to bear no significance for the proposed method and the rest of the paper; consider removing it
Section 4.2 paragraph 2: “shrinking” for different layers wasn’t mentioned before, and is not immediately clear what it means; the reader needs to be intimately familiar with the DIM paper to understand.
Section 4.3 mentions “threshold” for stein gradient estimator, which was not mentioned before. Please explain what it is.
Equations (8-10) are just simple derivations and are not necessary; it would be enough to provide Eq (10).
The authors talk about MINE, which optimizes the InfoNCE bound [1], which is also used in DIM and CPC [2]. I strongly encourage the authors to cite [1] and [2] and mention them in the related works. Additionally, it would be clear if Figure 1 and related references and description used “InfoNCE” instead of “MINE” as the name of the method since InfoNCE is an estimator and MINE is just a particular implementation of the method.

[1] Poole et. al., “On variational bounds of mutual information”, ICML 2019.
[2] van den Oord et. al, “Representation Learning with Contrastive Predictive Coding“, arXiv 2018.


We compare the model-free reinforcement learning with the model-based approaches through the lens of the expressive power of neural networks for policies, Q-functions, and dynamics. We show, theoretically and empirically, that even for one-dimensional continuous state space, there are many MDPs whose optimal Q-functions and policies are much more complex than the dynamics. We hypothesize many real-world MDPs also have a similar property. For these MDPs, model-based planning is a favorable algorithm, because the resulting policies can approximate the optimal policy significantly better than a neural network parameterization can, and model-free or model-based policy optimization rely on policy parameterization. Motivated by the theory, we apply a simple multi-step model-based bootstrapping planner (BOOTS) to bootstrap a weak Q-function into a stronger policy. Empirical results show that applying BOOTS on top of modelbased or model-free policy optimization algorithms at the test time improves the performance on MuJoCo benchmark tasks.

This paper presents a mainly theoretical argument comparing the expressivity of model-free and model-based RL methods contrary to analysis in the past which usually relies on sample complexity. They construct a family of MDPs, where the true dynamics belong to a simple function class (in terms of the number of linear pieces needed to define the function), but the corresponding optimal Q-function belongs to a function class not necessarily expressible by a simple function. The paper then builds a similar case for randomly/semi-randomly generated MDPs. Finally, they propose to bootstrap the Q-function with n-step returns to boost the expressivity exponentially.
I would lean towards accepting this paper, as this paper looks at an interesting problem and their analysis seems to be rigorous and valuable for the community to build upon. My questions/comments are as follows:
1. Previous work, for example [1], also talks about bootstrapping a neural net Q-function using an n-step approximation so as to improve the efficiency of a pure model-free algorithm. I think this paper should be cited.
2. In terms of the experiments, it is hard to understand the significance and connection to the theory. The theory talks about the expressive power of Q-functions, which suggests that we should look at only the asymptotic performance on these tasks, but most of the results are similar to MBPO or SAC in terms of asymptotic performance, although with a different learning speed, which could have to do with different factors.
3. This paper shows the existence and provides a constructive proof for a family of MDPs where expressing optimal Q-functions is exponentially harder than expressing dynamics. But how likely is such a setting to arise in MDPs in practice? For example, on the gym benchmarks, we would expect fairly not so complicated Q-functions -- although they might take longer to learn.
4. There is also a divide between learning Q*, and learning a policy that optimizes Q* reasonably well. Further, the requirement for the Q-function is only to get relative ordering between Q-values for different actions at states visited by the optimal policy right, which might not be the same as the Q-function landscape across state-action pairs. So, I am not sure if the expressivity of the optimal Q-function, in general, is the right metric to answer this question, but I also completely agree that this is a good starting point.
5. I think when applying RL to the current benchmarks or problems we have, the main problem could be linked to optimization of a neural-net Q-function via bootstrapping (in the sense of approximate dynamic programming) as compared to the expressive power of optimal Q-functions. But I agree that the problem being looked at in the paper would also exist.
References:
[1] Plan Online, Learn Offline: Efficient Learning and Exploration via Model-Based Control, Lowrey et.al.


When performing multi-task predictions with time-series data, knowledge learned for one task at a specific time step may be useful in learning for another task at a later time step (e.g. prediction of sepsis may be useful for prediction of mortality for risk prediction at intensive care units). To capture such dynamically changing asymmetric relationships between tasks and long-range temporal dependencies in time-series data, we propose a novel temporal asymmetric multi-task learning model, which learns to combine features from other tasks at diverse timesteps for the prediction of each task. One crucial challenge here is deciding on the direction and the amount of knowledge transfer, since loss-based knowledge transfer (Lee et al., 2016; does not apply in our case where we do not have loss at each timestep. We propose to tackle this challenge by proposing a novel uncertaintybased probabilistic knowledge transfer mechanism, such that we perform knowledge transfer from more certain tasks with lower variance to uncertain ones with higher variance. We validate our Temporal Probabilistic Asymmetric Multi-task Learning (TP-AMTL) model on two clinical risk prediction tasks against recent deep learning models for time-series analysis, which our model significantly outperforms by successfully preventing negative transfer. Further qualitative analysis of our model by clinicians suggests that the learned knowledge transfer graphs are helpful in analyzing the model's predictions.

The submission proposes a probabilistic method for multitask learning for sequence data focusing on the use of uncertainty to control the amount of transfer between tasks. In comparison to prior work, the approach directly models uncertainty by parametrising Gaussian distributions and dropout variational inference. The approach additionally models transfer between features from different time-steps. The evaluation shows that the method outperforms recent state-of-the-art models on two medical benchmarks.
The paper is overall well written but includes some vagueness regarding the proposed method as well as the experimental setup. While describing techniques to model two kinds of uncertainty the aspect of the model which adapts transfer to the amount of uncertainty in source and target features is not described in detail (F_theta). It is described as if it only received samples for both features without any additional information regarding uncertainty. This makes it improbable that the model is able to take the feature’s uncertainty into account when modelling transfer.
The paper follows in section 3.3 with an argument for the lower computational complexity of only transferring from previous time-steps. I might misunderstand the description but having every time-step’s features be influenced only by previous timesteps should still result in complexity O(T^2) (even though the amount of computation is reduced by a constant factor).
My background is not in machine learning with clinical data (so I do not know common datasets) but it is unclear to me why the datasets evaluated on only consist of a very small section of the overall dataset they’re taken from (in case 1). Looking at table 4 the main factor seems to be probabilistic modelling so additional ablations with either version of uncertainty would be interesting.
By introducing probabilistic modelling to control the amount of transfer between tasks the paper provides an interesting perspective and is able to show strong performance. However it is also quite vague on important aspects such as the exact modelling of transfer and limited regarding evaluation.
Minor aspects:
- V from the final equation on page 5 is never described.
- After describing modelling both kinds of uncertainty in the method section, the reader is uninformed which uncertainty is shown in plots ⅘
- Table 4 does not have the two highest results in bold (see AMTL samestep)
- Very limited multitask and transfer learning references pre 2016.
- Some descriptions are unclear including the mention of ‘attention allocation’


The growth in the complexity of Convolutional Neural Networks (CNNs) is increasing interest in partitioning a network across multiple accelerators during training and pipelining the backpropagation computations over the accelerators. Existing approaches avoid or limit the use of stale weights through techniques such as micro-batching or weight stashing. These techniques either underutilize of accelerators or increase memory footprint. We explore the impact of stale weights on the statistical efficiency and performance in a pipelined backpropagation scheme that maximizes accelerator utilization and keeps memory overhead modest. We use 4 CNNs (LeNet-5, AlexNet, VGG and ResNet) and show that when pipelining is limited to early layers in a network, training with stale weights converges and results in models with comparable inference accuracies to those resulting from non-pipelined training on MNIST and CIFAR-10 datasets; a drop in accuracy of 0.4%, 4%, 0.83% and 1.45% for the 4 networks, respectively. However, when pipelining is deeper in the network, inference accuracies drop significantly. We propose combining pipelined and non-pipelined training in a hybrid scheme to address this drop. We demonstrate the implementation and performance of our pipelined backpropagation in PyTorch on 2 GPUs using ResNet, achieving speedups of up to 1.8X over a 1-GPU baseline, with a small drop in inference accuracy.

This paper investigates the impact of stale weights on the statistical efficiency and performance in a pipelined backpropagation scheme that maximizes accelerator utilization while keeping the memory overhead modest. The paper proposes to combine pipelined and non-pipelined training in a hybrid scheme to address the issue of significant drop in accuracy when pipelining is deeper in the network. The performance of the proposed pipelined backpropagation is demonstrated on 2 GPUs using ResNet with speedups of up to 1.8X over a 1-GPU baseline and a small drop in inference accuracy.
The paper is well written and easy to follow. The proposed idea is interesting and its effectiveness is well demonstrated with a promising speed and a small drop in accuracy. The proposed approach is compared to two existing works:  PipeDream [1] and GPipe [2]. Though promising results have been demonstrated, a drawback of the proposed method is that it introduces more memory overhead compared to GPipe. Although a detailed discussion is provided related to the memory consumption between the proposed method and PipeDream, no detailed discussion is provided with respect to GPipe. Further, no proper convergence analysis of the proposed approach is provided and is desired due to the likely divergence in the optimization. Minor comment: An interesting line of work is that of [3] which could be included in the discussion.
Overall, the proposed approach is interesting and is shown to achieve promising results. However, memory overhead is still an issue compared to existing method.
[1] Aaron Harlap, Deepak Narayanan, Amar Phanishayee, Vivek Seshadri, Nikhil Devanur, Greg Ganger, and Phil Gibbons. Pipedream: Fast and efficient pipeline parallel DNN training, 2018. URL http://arXiv:1806.03377.
[2] Yanping Huang, Yonglong Cheng, Dehao Chen, HyoukJoong Lee, Jiquan Ngiam, Quoc V. Le, and Zhifeng Chen. Gpipe: Efficient training of giant neural networks using pipeline parallelism, 2018. URL http://arXiv:1811.06965.
[3] Guanhua Wang, Shivaram Venkataraman, Amar Phanishayee, Jorgen Thelin, Nikhil Devanur, Ion Stoica: Blink: Fast and Generic Collectives for Distributed ML. arXiv:1910.04940, 2019.


Combining multiple function approximators in machine learning models typically leads to better performance and robustness compared with a single function. In reinforcement learning, ensemble algorithms such as an averaging method and a majority voting method are not always optimal, because each function can learn fundamentally different optimal trajectories from exploration. In this paper, we propose a Temporal Difference Weighted (TDW) algorithm, an ensemble method that adjusts weights of each contribution based on accumulated temporal difference errors. The advantage of this algorithm is that it improves ensemble performance by reducing weights of Q-functions unfamiliar with current trajectories. We provide experimental results for Gridworld tasks and Atari tasks that show significant performance improvements compared with baseline algorithms.

This paper addresses the problem of combining separately learned action-value functions into a single, no-longer-learning, action selection algorithm. The result is not necessarily a combined policy, strictly speaking, because it need not take the same action (or produce the same action selection probabilities) each time it is in the same state. Such “ensemble methods” have been a big topic in supervised learning, where ensemble methods are often effective, but less so in reinforcement learning, where the evidence in support of the idea is, perhaps, weak. This paper just assumes it is a good idea, while citing prior work that i don’t think really shows that it is. A new method is proposed, called TDW, based on weighting the learner’s action values according to their recent squared temporal-difference errors. This is compared with simple ensemble method that have no memory other than their action values, on two gridworld problems and six Atari games. Comparison is also made with single (non-ensemble) methods that are given as much training data as one of the members of the ensemble. The results are consistent with TDW being better than the simpler ensemble methods on both gridworlds and Atari games, and with both ensemble methods being better than the single methods on Atari games but worse than the single methods on the gridworld problems.
Does this work make a contribution to the field of reinforcement learning? A lot depends on whether you think the problem of combining separate learners is an important problem. I am skeptical. First, this problem requires one to be able to identify separate learners that are operating in independent copies of the identical environment, and whose states and actions can be mapped one-to-one onto each another. (This is also a requirement of asynchronous methods, so this criticism applies to them as well.) Of course it is easy for us to arrange these things with our simulated test environments, and there may even be _a few_ real problems like this, but most of the time this is unrealistic. If a method requires this, then it is not a general method. Second, and I suppose relatedly, this new problem bears little relation to the original problem: Does our field benefit from the introduction of new problems and solution methods that apply only in special cases? Or are these special cases primarily distractions from the general case and thus ways to avoid coming to grips with the real problem?
Different researchers might answer these rhetorical questions in different ways. It is a judgement call, but they reflect a real concern. A greater problem is that this work does not do a good job addressing the problem of ensembles. First, the introduction glosses over the question of whether ensembles are a good idea; it kinda suggests that their effectiveness has already been established, but does not commit itself to that statement; it does not make a clear claim that might be true or false about that prior work and that would validly support interest in this area. Second, this paper contains the wrong sort of comparisons of ensemble methods with single (non-ensemble) methods. It compares an ensemble of 10 learners with a single learner who only has as much data as one of the 10 learners in the ensemble. Thus, each single learner has only one-tenth as much data as has gone into the ensemble. Surely this is not the right comparison. It is a real problem that an early proponent of combining learners makes an unfair comparison with non-combined learners. It sets a poor standard and example for those who come later. The third way in which this work on ensemble RL agents is not very good it that its ensemble methods are not well suited to the task. That is, if one was going to combine learners, then there are many interesting natural ways to do this, and these are not done. It would be natural for each learner to keep track of how much experience or confidence it has in each part of the state space. The TDW algorithm can be seen as one way of doing this, but it is just one rather idiosyncratic way; there are many more natural ways. If we are going to explore this new problem, and new algorithms for solving it, then one really ought to do better than propose only methods that retain only their action-value function and nothing else that might facilitate the subsequent combination. In these three ways I feel this paper is not a good example of exploring the new problem of ensemble learning, even if you think that problem is a worthy one.


The objective in deep extreme multi-label learning is to jointly learn feature representations and classifiers to automatically tag data points with the most relevant subset of labels from an extremely large label set. Unfortunately, state-of-theart deep extreme classifiers are either not scalable or inaccurate for short text documents. This paper develops the DeepXML algorithm which addresses both limitations by introducing a novel architecture that splits training of head and tail labels. DeepXML increases accuracy by (a) learning word embeddings on head labels and transferring them through a novel residual connection to data impoverished tail labels; (b) increasing the amount of negative training data available by extending state-of-the-art negative sub-sampling techniques; and (c) re-ranking the set of predicted labels to eliminate the hardest negatives for the original classifier. All of these contributions are implemented efficiently by extending the highly scalable Slice algorithm for pretrained embeddings to learn the proposed DeepXML architecture. As a result, DeepXML could efficiently scale to problems involving millions of labels that were beyond the pale of state-of-the-art deep extreme classifiers as it could be more than 10x faster at training than XML-CNN and AttentionXML. At the same time, DeepXML was also empirically determined to be up to 19% more accurate than leading techniques for matching search engine queries to advertiser bid phrases. Source code for DeepXML can be downloaded from (Anonymous, 2019).

This paper considers extremely multi-label learning (XML) where the label size is very large . It aims to improve the accuracy as well as the scalability of XML algorithms, especially for short text inputs. The accuracy for XML with short text inputs can be significantly improved using deep learning representation than using TFIDF features. This paper proposes several tricks to handle the issue for efficiently learning both neural network parameters and classification weights of extremely large number of labels. The proposed method borrowed ideas from several previous literature, and is mainly based on SLICE, where a pre-trained fixed deep learning representation for the inputs are used with ANNS graph for labels to improve the scalability. The main difference is that instead of using a fixed input embedding, the proposed method learns the word embedding parameters via a set of head labels. The remaining labels are then trained using SLICE with fixed word embeddings from the learned word embedding model.
Overall the paper tackles the problem well. And the empirical results show improved results. However, I don't think this paper is ready for publication due to the following concerns.
1. My main concern is that the proposed method seems to be a combination of a number of tricks. This makes the overall algorithm/model very complicated and introduces a lot of hyper-parameters, for example, head label portion, L-h', c, beta, s neural network hyper-parameters and so on. Hence, it will be hard to be used in real applications.
2. Another concern is about the experiments.
    a. The most significant improvement of the proposed method over existing method happens in the private dataset, Q2B-3M, which can't be reproduced.
     b. On the public datasets, DeepXML seems to show good results on small datasets, WikiSeeAlsoTItles-350K and WkipediaTitle-500K, while on large datasets, DeepXML performance is close to the existing methods.
     c. The largest label size in the experiments is 3M. SLICE can be scaled up to 100M labels.
3. The writing and the organization of this paper needs to be improved.
    a. Some notations are not clearly defined. For example, L_h in Line 6 and X' in Line 9 on Page 5.
    b. Several method names are not defined. For example, DeepXML-fr, AttentionXML-l, Sliced-CDSSM, DeepXML-SW, DeepXML-f. I have to guess what they are.
    c. The last two paragraphs on Page 4 seems to be related work, while there is a section called "Related work".
Other minor comments:
1. It seems it is not stated how Beta is set.
2. I am wondering if it's true that the shorter the input text is, the better improvement over non-deep-learning methods DeepXML can achieve.
3. In the first paragraph of Sec 3.1, it is mentioned "clustering just the top 3 labels into 300 clustering". Why choose 3 and 300? Are these numbers used for all datasets?


This paper studies learning the representations of whole graphs in both unsupervised and semi-supervised scenarios. Graph-level representations are critical in a variety of real-world applications such as predicting the properties of molecules and community analysis in social networks. Traditional graph kernel based methods are simple, yet effective for obtaining fixed-length representations for graphs but they suffer from poor generalization due to hand-crafted designs. There are also some recent methods based on language models (e.g. graph2vec) but they tend to only consider certain substructures (e.g. subtrees) as graph representatives. Inspired by recent progress of unsupervised representation learning, in this paper we proposed a novel method called InfoGraph for learning graph-level representations. We maximize the mutual information between the graph-level representation and the representations of substructures of different scales (e.g., nodes, edges, triangles). By doing so, the graph-level representations encode aspects of the data that are shared across different scales of substructures. Furthermore, we further propose InfoGraph*, an extension of InfoGraph for semi-supervised scenarios. InfoGraph* maximizes the mutual information between unsupervised graph representations learned by InfoGraph and the representations learned by existing supervised methods. As a result, the supervised encoder learns from unlabeled data while preserving the latent semantic space favored by the current supervised task. Experimental results on the tasks of graph classification and molecular property prediction show that InfoGraph is superior to state-of-the-art baselines and InfoGraph* can achieve performance competitive with state-of-the-art semi-supervised models.

In this paper, the authors propose a graph-level representation, which extends the existing node-level representation learning mechanism. Besides, both unsupervised and semi-supervised learning are leveraged for InfoGraph and InfoGraph*, receptively. The authors naturally apply Deep Graph Infomax, a contrastive representation learning method, for the whole graph level instead of the previous node embedding learning. The experiments on graph classification and molecular property prediction indicate the effectiveness, even compared to the supervised methods.
The learned graph-level representation looks good to me. Instead of some heuristics based graph-level pooling, the proposed method automatically figure out the best way to produce the fixed-length feature vector in a data-driven approach. Such motivation is pretty reasonable and natural for me. Also, the usage of both unsupervised and semi-supervised learning procedure is well-motivated. Overall speaking, the paper is well written. The definitions of problems, the details of methods, and the settings of experiments are clear to me.
I have some questions and suggestions for the authors:
1. The overall writing looks good to me. Besides, It could be much better if the authors could provide more and better illustrations for the method. Both figures 1 and 2 are not that informative to me, to be honest. I know it could hard to visualize the graph-level representation, but it worth it. There are many steps and equations in the paper, and the illustration could play an essential role in putting all the steps together to demonstrate the big picture.
2. The authors carefully discuss the difference between this submission and concurrent work (Information Maximizing Graph Network) or existing work (Deep Graph Infomax), which helps a lot for the reader to understand the literature better. It could be much better if this submission could be more self-contained. For example, for the semi-supervised learning setting, the authors let the reader read some external papers. I am suggesting that a small section in the appendix could make life much more comfortable.


With the increase of data volume and data dimension, sparse representation learning attracts more and more attention. For high-dimensional data, randomized block coordinate descent methods perform well because they do not need to calculate the gradient along the whole dimension. Existing hard thresholding algorithms evaluate gradients followed by a hard thresholding operation to update the model parameter, which leads to slow convergence. To address this issue, we propose a novel hard thresholding algorithm, called Semi-stochastic Block Coordinate Descent Hard Thresholding Pursuit (SBCD-HTP). Moreover, we present its sparse and asynchronous parallel variants. We theoretically analyze the convergence properties of our algorithms, which show that they have a significantly lower hard thresholding complexity than existing algorithms. Our empirical evaluations on real-world datasets and face recognition tasks demonstrate the superior performance of our algorithms for sparsity-constrained optimization problems.

===== Update after author response
Thanks for the clarifications and edits in the paper.
I recommend acceptance of the paper.
Other comments:
Definition 1 in the updated version is still too vague ("difference of what?" -- function values? distance in norm between iterates?) -- this should be clarified.
========
This paper considers the problem of sparsity-constrained ERM and asks whether one can design a variant of the stochastic hard thresholding approaches where the hard-thresholding complexity does not depend on a (sparsity dependent) condition number, unlike all previous approaches (Table 1). It proposes a method which combines SVRG-type variance reduction, with block-coordinate updates, leaving the hard thresholding operation outside the inner loop, to accomplish this goal. It provides a convergence analysis which significantly improves the previous best rates (by having both the sparsity level shat which is significantly lower (kappa_shat vs. kappa_stilde^2) as well as a condition number independent hard thresholding complexity (Table 1). An asynchronous and sparse (in the features) variant is also proposed, with even better complexity. Some standard experiments on sparse linear regression and sparse logistic regression is presented showing an improvement in both number of iterations as well as CPU time.
I think the clarity of the paper should be quite improved (see detailed comment), hence why I think the paper is borderline, but I am leaning towards an accept given the significant theoretical improvements over the past literature (and positive empirical results), even though the algorithmic suggestion is somewhat incremental.
The proposed Algorithm 1 seems very close to the one of Chen & Gu (2016), the paper should be more clear about this. There seems to be mainly two changes: a) extending the support projection of the gradient to the union of the sampled block with the one of the support of the reference parameter wtilde (vs. just the sampled block in Chen & Gu (2016) and b) moving the hard-thresholding iteration outside of the SVRG-inner loop. These small tweaks to the algorithm yield a significant theoretical improvement, though.
== Detailed comments ==
Clarity: the number of long abbreviations with only one letter change make it hard to follow the different algorithms; perhaps a better more differentiating naming scheme could be used. Moreover, I think more background on the sparse optimization setup should be provided in the introduction or at least in the preliminaries, as I do not think the wider ICLR community is very familiar with it (in particular, no cited paper was at ICLR). For example, define early the separation in optimization error and statistical error; and point out that F(w_t) might even be lower than F(w*) as the sparsity threshold s might be much higher than s*. This will make Table 1 more concrete and less abstract for people who not are not yet experts on this particular analysis framework.
- Table 1: I would suggest to put the rate for S2BCD-HTP instead on the last row and mention instead that the rate for ASBCD is similar under conditions on the delay; as it is interesting to already have a better gradient complexity for S2BCD vs. SBCD.
** Questions:  **
1) In Corollary 1, how is the gradient oracle complexity defined or computed? And more specifically, how does one compare fairly the cost of doing a gradient update in Algorithm 1 on the *bigger set* S = Gtilde U G_jt vs. just G_jt for the Chen & Gu ASBCD algorithm? Is this accounted in the computation?
2) In Figure 1, which "minimum" is referred to and how is it found? I suspect it is not F(w*) (as it could be higher than F(w_t)), i.e. it is *not* the minimum of (1) with s*. One natural guess is that it might be min_w F(w) s.t. ||w||_0 <= s, though I do not see any guarantee in the main paper that running the algorithm would make F(w_t) converge to such a value (i.e. all we know from Thm 1 is that F(w_t) might be within O(||nabla_Itilde F(w*)||^2) of F(w*) ultimately. Please explain and clarify!
== Potential improvement ==
The current result in Theorem 1, which is building on a similar proof technique as the original SVRG paper, has the annoying property of requiring the knowledge of the condition number in setting the size of the inner loop iteration. I suspect that this is an artifact of using an outdated version of the SVRG algorithm. This has been solved since then by considering a "loopless" version of SVRG which implicitly defines the size of the inner loop in a random manner using a quantity *which does not depend on the condition number*. This was proposed first by Hofmann et al. [2015], and then re-used by Lei & Jordan [2016] and more recently by Kovalev et al. [2019] e.g. Note that Leblond et al. (2017) that you cited profusely also used this variant of SVRG. I suspect that this technique could be re-used in your case to obtain a similar result with a loopless variant (which also gives cleaner complexity results). (Though I only skimmed through your proof.)
Caveat: the sensibility of the theory in the main paper seems reasonable, but I did not check the proofs in the appendix.
= References:
- Hofmann et al. [2015]: Variance Reduced Stochastic Gradient Descent with Neighbors, Thomas Hofmann, Aurelien Lucchi, Simon Lacoste-Julien and Brian McWilliams, NeurIPS 2015
- Lei & Jordan [2016]: Less than a Single Pass: Stochastically Controlled Stochastic Gradient Method, Lihua Lei and Michael I. Jordan, AISTATS 2016
- Kovalev et al. [2019]: Don't Jump Through Hoops and Remove Those Loops: SVRG and Katyusha are Better Without the Outer Loop, Dmitry Kovalev, Samuel Horvath and Peter Richtarik, arXiv 2019


We propose a word attribute transfer framework based on reflection to obtain a word vector with an inverted target attribute for a given word in a word embedding space. Word embeddings based on Pointwise Mutual Information (PMI) represent such analogic relations as −−→ king − −−→ man + − −−−− → woman ≈ −−−→ queen. These relations can be used for changing a word's attribute from king to queen by changing its gender. This attribute transfer can be performed by subtracting a difference vector −−→ man − − −−−− → woman from −−→ king when we have explicit knowledge of the gender of given word king. However, this knowledge cannot be developed for various words and attributes in practice. For transferring queen into king in this analogy-based manner, we need to know that queen denotes a female and add the difference vector to it. In this work, we transfer such binary attributes based on an assumption that such transfer mapping will become identity mapping when we apply it twice. We introduce a framework based on reflection mapping that satisfies this property; queen should be transferred back to king with the same mapping as the transfer from king to queen. Experimental results show that the proposed method can transfer the word attributes of the given words, and does not change the words that do not have the target attributes.

Summary: This paper proposes a method for binary word attribute transfer based on reflection. It applies a single reflection-based mapping that relates the locations of two vectors in a Euclidean space by a hyperplane, and results in an identity mapping when it is applied twice. It's sort of like roundtrip-translation, but for word vector attributes. The paper also proposes a pretty smart idea: the mirror functions are parameterized to take advantage of the fact that inversions differ, even for the "same" word attributes.
I am giving the paper a weak accept, because I think idea is really fun, and definitely has legs. I like the idea for this paper quite a bit, it really got me thinking, but I think it isn't quite structured how I would like. Unfortunately, as it is, it also feels a bit spare in terms of contributions. I would like much more discussion/analysis of the parameterizations (and their parameters), or treatment of more binary-attribute linguistic analogy types (antonymy, scalar terms), or both! Another thing that could be added is comparisons across types of word vectors (word2vec alone is a bit narrow). Any of those things would have justified a strong accept.
Suggestions:
- As I was reading, I was not at all convinced by the "idealness" of your transfer function in 4.1 (maybe change it from "ideal" to "idealized"). "Ideal" assumes that there will be pairs sharing a stable attribute. But, often "gendered" words don't come in pairs, and "gender" is far from a stable attribute. To be more specific, despite your Fig 3a: "actress" may be feminine, but "actor" is clearly neutral (anyone can be described as an "actor" but only women can be described as "actress" in most cases); thus "actress" is the gendered one. We can go ahead and pair "actress" with "actor"---they do share a morphological relationship after all---but "actor" isn't as obvious a masculine counterpart as "king" is for "queen". How would you square this with phi(queen)=king and phi(king)=queen, where the identity of counterparts is clearer? Do we ideally want the same invertible method here for both actor-actress and queen-king (I think I don't, hence the function is not "ideal"). I wouldn't be surprised if the same situation arises in other analogies. This is, I think, why you brought up parameterization. But you miss the opportunity to clearly motivate why we need to parameterize! You should include more motivation and lay the linguistic facts out in a clearer way that incorporates examples like the one I gave, and preferably before S4.1.
-The total number of words you chose to use is pretty small, which you acknowledge. I would've loved to see a few more phenomena, perhaps antonymy, part-whole, morphological reinflection (e.g., a generalization of your sing-pl set, datasets exist for past-tense at least), etc.
-it would have been neat to compare word2vec to other types of vector embeddings, maybe contextualized? They are what everyone's using right now (this suggestion didn't factor heavily into my rating though). Also the framing in your introduction made me think this is where you were going (you started with discussing vectors, as opposed to just jumping in on binary word attribute transfer. I think the paper could just jump right in with binary word attribute transfer, and skip the basic vectors paragraph).
-I would've liked more discussion of your parameterized mirrors, since that was the neatest/prettiest part, in my opinion. (maybe I missed something, but how did you decide on how many z to use? or did every pair get its own? Do your zs for each attribute correlate at all? you could use something straightforward like CCA to check, would be neat.)
Small Notes/Musings:
-on p.1 "(PMI (citation..." with nested parentheses is not easy to read, I prefer "(PMI; citation)".
- It is fine to operate on the assumption that gender is binary for now, but you should acknowledge that binary gender is an assumption (and a socially problematic one), that isn't real.
-in 5.5, I'm not impressed by "When non-attribute word the was given, Ref(v the) became 'the' without knowledge that the has no gender attribute", because "the" is the most frequent word in English, perhaps show that it works with rare, non-gendered words instead. Incidentally, does it know syntactic category? Because that might be a neat thing to check.
-what do you think about trinary attributes (cold, warm, hot)? It would be a cool extension.


Ensembles, where multiple neural networks are trained individually and their predictions are averaged, have been shown to be widely successful for improving both the accuracy and predictive uncertainty of single neural networks. However, an ensemble's cost for both training and testing increases linearly with the number of networks.
In this paper, we propose BatchEnsemble, an ensemble method whose computational and memory costs are significantly lower than typical ensembles. BatchEnsemble achieves this by defining each weight matrix to be the Hadamard product of a shared weight among all ensemble members and a rank-one matrix per member. Unlike ensembles, BatchEnsemble is not only parallelizable across devices, where one device trains one member, but also parallelizable within a device, where multiple ensemble members are updated simultaneously for a given mini-batch. Across CIFAR-10, CIFAR-100, WMT14 EN-DE/EN-FR translation, and contextual bandits tasks, BatchEnsemble yields competitive accuracy and uncertainties as typical ensembles; the speedup at test time is 3X and memory reduction is 3X at an ensemble of size 4. We also apply BatchEnsemble to lifelong learning, where on Split-CIFAR-100, BatchEnsemble yields comparable performance to progressive neural networks while having a much lower computational and memory costs. We further show that BatchEnsemble can easily scale up to lifelong learning on Split-ImageNet which involves 100 sequential learning tasks.

This paper aims to improve the efficiency of ensembles of neural nets in traditional supervised learning and life-long learning (learning on a series of tasks). The main idea is to let all the neural nets in an ensemble share the same weights W for each layer, and the weights for each neural net is generated by the Hadamard product of W and a specific rank-one matrix of the same size as W that is different across members in the ensemble. In experiments, they evaluate the method with some baselines on life-long learning, traditional classification, NMT tasks, and uncertainty modeling.
The paper relates the proposed method to several different learning problems and applications and lists many potential advantages in these applications: it covers a lot of things. However, it lacks in-depth discussion to several key problems, rigorous analysis or complete experimental study to support the main claims, for example:
Why can the simple method achieve a more compelling trade-off between accuracy and efficiency/memory costs comparing to a large single model or a naive ensemble of small models? Any mathematical or information-theoretical explanation behind that?
It is easy to understand that the ensemble defined here can improve efficiency and reduce memory cost. But as an alternative to the naive ensemble, we also expect the performance to not suffer from severe drawbacks. How to control efficiency-performance trade-off in the proposed method?
How were the baselines for each experiment selected? How to determine the specific setting in each experiment (any reason behind choosing the parameters in the settings)?
In the life-long learning settings, the shared weights W is only trained on the first task and then keeps fixed: this can leads to both large variance and bias. Why does it simply work well without causing any serious problems?
The rank-one extension of a shared model W enforces a very strong regularization to the model for each task. Will the method work promisingly when the tasks are more different from each other or harder to solve? For example, what if we increase the classes in each task? Is the rank-one extension still flexible and expressive enough to handle this situation?
These are some of the most important questions needed to be answered in the first place before showing higher evaluation metrics and listing the potential advantages of the proposed method. But it is not clear to me at all how they can be answered according to the contents in the current paper. I notice that the authors mentioned the last two questions at the end of Section 3, but no explanations/discussions were given.
Other major concerns:
1) Mathematically, comparing to single model Wx, the proposed ensemble method equals to applying a dimension-wise scaling to the input x and a dimension-wise scaling to the output Wx, and the scaling factors vary across different tasks. Hence, the proposed structure is exactly the same as fine-tuning two groups of batch normalization scaling factors before and after applying transformation W. It does not make much sense in the experiments that the performance of BN-Tuned in Figure 3a is much worse than the proposed method since they share exactly the same structure and math (note the memory and computational costs are also the same). The paper does not give an explanation about this. Moreover, the baseline BN-Tuned is only compared on only one of those datasets in the paper. It should be one of the most important baselines and needs to be compared in all experiments.
2) On each benchmark dataset (except the last one), only 1-2 baselines are compared and most baselines are not state-of-the-art methods or not methods specifically designed for the problem (e.g., many are dropout and its variants). This makes the comparisons not convincing, especially considering that the experimental settings are determined by the authors and might be chosen for the best performance of the proposed method.
3) At least two baselines should be included in all experiments: 1) single model with the equal number of model parameters, and 2) naive ensemble not sharing parameters across member models. However, each experiment only includes one or even none of these two baselines.
4) Memory and training/test computational costs need to be reported for each experiment. However, the currently reported results are incomplete here and there.
5) Comparing to the currently limited number of baselines on the incomplete evaluation metrics, the proposed method does not show significant improvements, for example, the results in Figure 4, Table 1 and Table 2.
6) The proposed method requires the models for different tasks should have exactly the same architecture. This could be a strong limitation in many scenarios. For example, when different tasks have significantly different numbers of classes.


Over-parameterized deep neural networks trained by simple first-order methods are known to be able to fit any labeling of data. Such over-fitting ability hinders generalization when mislabeled training examples are present. On the other hand, simple regularization methods like early-stopping can often achieve highly nontrivial performance on clean test data in these scenarios, a phenomenon not theoretically understood. This paper proposes and analyzes two simple and intuitive regularization methods: (i) regularization by the distance between the network parameters to initialization, and (ii) adding a trainable auxiliary variable to the network output for each training example. Theoretically, we prove that gradient descent training with either of these two methods leads to a generalization guarantee on the clean data distribution despite being trained using noisy labels. Our generalization analysis relies on the connection between wide neural network and neural tangent kernel (NTK). The generalization bound is independent of the network size, and is comparable to the bound one can get when there is no label noise. Experimental results verify the effectiveness of these methods on noisily labeled datasets.

This paper proposes two regularization methods for learning on noisily labeled data: the first penalizes the distance w.r.t. Euclidean norm from an initial point and the second uses an additional auxiliary variable for each example to learn a noise. In the theoretical part, the paper shows that an original clean dataset can be learned from a noisily labeled dataset based on NTK-theory. Finally, the effectiveness of proposed regularization methods is well verified empirically for 2-layer NN, CNN, and ResNet on image classification tasks (MNIST, CIFAR-10).
Contributions:
- Propose two simple regularization techniques for learning from a noisily labeled dataset.
- Give generalization guarantees for these methods
Clarity:
The paper is well organized and easy to read.
Quality:
The work is of good quality and is technically sound.
Significance:
Since proposed methods are in some sense related to the early-stopping for the (stochastic) gradient descent, the developed theory is useful in understanding the generalization ability of over-parameterized neural networks falling into NTK-regime. Although an additive noise setting for the regression problem is rather common in statistical learning theory, an artificially flipped label setting for the classification problem may be new except for a few recent studies [Li+(2019)]. A result (Theorem 5.1) for the regression problem with the squared loss is not so surprising because the generalization error of gradient descent in a high-dimensional space (e.g., over-parameterized NNs) or an RKHS (i.e., infinite-dimensional model) has been well studied and the generalization error is composed of the (constant) variance and the distance between the model output and the true label. Thus, existing results of generalization error analyses for the regression are directly applied to bound the prediction error for clean labels. However, I basically like this paper and I think this paper makes a certain contribution to understanding the effect of over-parameterization.
A few questions:
- Usually, the regularization parameter goes to zero as the number of examples increases. Conversely, the regularization parameter in the proposed methods also increases. Could you explain why this difference happens?
- In classification tasks, the original problem setting is recovered by setting $p=lamba=0$. However, the generalization bound by Theorem 5.3 is still affected by $\lambda$. Is this theorem tight?
-----
Update:
I thank the authors for the response. I have read the revised version of the paper and have confirmed an improvement of Theorem 5.2. I will keep my score. This paper is of good quality.


Tasks involving localization, memorization and planning in partially observable 3D environments are an ongoing challenge in Deep Reinforcement Learning. We present EgoMap, a spatially structured neural memory architecture. EgoMap augments a deep reinforcement learning agent's performance in 3D environments on challenging tasks with multi-step objectives. The EgoMap architecture incorporates several inductive biases including a differentiable inverse projection of CNN feature vectors onto a top-down spatially structured map. The map is updated with ego-motion measurements through a differentiable affine transform. We show this architecture outperforms both standard recurrent agents and state of the art agents with structured memory. We demonstrate that incorporating these inductive biases into an agent's architecture allows for stable training with reward alone, circumventing the expense of acquiring and labelling expert trajectories. A detailed ablation study demonstrates the impact of key aspects of the architecture and through extensive qualitative analysis, we show how the agent exploits its structured internal memory to achieve higher performance.

This very well written and executed paper synthesizes several ideas recently published in the field of deep reinforcement learning-based goal-driven navigation. It elegantly combines these ideas together by presenting a neural agent architecture that consists of:
* a perception module (e.g. a convnet) that extracts coarse visual feature maps s_t from an RGBD image
* a differentiable map canvas M_t that is rotated at each step based on affine egocentric velocity (dx_t, dy_y, d \phi_t)
* differentiable inverse projection mapping, which uses known camera parameters, projective geometry and the depth channel of the image to project the visual feature vectors s_t onto a 2D map and add it to the existing canvas M_t
* a recurrent module (GRU) for update a state h_t that is used for computing the policy distribution and value function
* additional inputs to the policy and value function, that include a global map read r_t, as well as a query q_t (produced by the policy head) based retrieval of features from the map
* position indexing of features retrieved from the map
The algorithm is trained end-to-end, without extra supervision, using Advantage Actor-Critic (A2C) RL. Based on the strong inductive biases regarding the map, namely affine transforms of the map given information about relative movement, and projective geometry transformations of visual features in the map frame, it seems that the question of where to write is solved, and that the network only needs to learn what to write in the differentiable map. Evaluation is done on 3 games in VizDoom: finding the exit of the Labyrinth, object retrieval and find and return / Minotaur.
Criticism:
The authors could justify better the choice of using the projective geometry inductive prior. They use sentences like "We argue that projective geometry is a strong law imposed on any vision system working from egocentric observations" (not quite related to grid and place cells, despite being in that section) and "this inverse mapping operation is second nature to many organisms" without giving any reference.
Several papers have been published in the last two years, focusing on differential memory architectures with a 2D map structure, projective geometry. This paper goes further by building and iteratively updating a 2D occupancy map using visual features and image geometry, just like RGBD-SLAM (which would merit a citation, e.g., [1] and [2]). This paper essentially combines existing ideas (see table 1): projective geometry, reward-based learning of M_t, RL, multitask navigation, semantic features. While this is not novel, seeing all this combined in a single technique does have merit.
What is disappointing, given that this is a combination paper, is that the environment is so simply, and that photorealistic environments were not tested. For example, the VizDoom environment uses 2D sprites for objects, making the visual feature extraction from objects much simpler. Would the method work equally well with the objects in DeepMind Lab, which are seen from multiple view points? And what in an environment like AdobeIndoorNav?
[1] Henry et al (2010) "RGB-D mapping: Using depth cameras for dense 3D modeling of indoor environments"
[2] Izadi et al (2011) "KinectFusion: real-time 3D reconstruction and interaction using a moving depth camera"


Recent work has exhibited the surprising cross-lingual abilities of multilingual BERT (M-BERT) -surprising since it is trained without any cross-lingual objective and with no aligned data. In this work, we provide a comprehensive study of the contribution of different components in M-BERT to its cross-lingual ability. We study the impact of linguistic properties of the languages, the architecture of the model, and the learning objectives. The experimental study is done in the context of three typologically different languages -Spanish, Hindi, and Russian -and using two conceptually different NLP tasks, textual entailment and named entity recognition. Among our key conclusions is the fact that the lexical overlap between languages plays a negligible role in the cross-lingual success, while the depth of the network is an integral part of it.

What is the task?
Comprehensive study of the contribution of different components in Multilingual BERT to its cross-lingual ability.
What has been done before?
(Wu & Dredze, 2019) and (Pires et al., 2019) identified the cross-lingual success of the model and tried to understand it. However, both works treated M-BERT as a black box and compared M-BERT’s performance on different languages. This work, on the other hand, examines how B-BERT performs cross-lingually by probing its components, along multiple aspects. Some of the architectural conclusions have been observed earlier, if not investigated, in other contexts.
Authors claim that “Pires et al. (2019) hypothesizes that the cross-lingual ability of M-BERT arises because of the shared word-pieces between source and target languages.” is not entirely correct. Pires et al. (2019) did show M-BERT’s ability to transfer between languages that are written in different scripts, and thus have effectively zero lexical overlap. There were results (e.g. Figure 1) showing that while performance using EN-BERT depends directly on word piece overlap, M-BERT’s performance is largely independent of overlap, indicating that it learns multilingual representations deeper than simple vocabulary memorization.
Pires et al. (2019) also showed that structural similarity is crucial for cross-lingual transfer
What are the main contributions of the paper?
First comprehensive study of the contribution of different components in Multilingual BERT to its cross-lingual ability. Novel findings about the effect of network architecture, input representation and learning objective on cross lingual ability of M-BERT
Methodology that facilitates the analysis of similarities between languages and their impact on cross-lingual models by mapping English to a Fake-English language, that is identical in all aspects to English but shares no word-pieces with any target language.
What are the key dimensions studied/analyzed?
Different components/aspects of Multilingual BERT investigated:
(i) Linguistics properties and similarities of target and source languages  (has been studied in prior work)
(ii) Network Architecture (novel)
(iii) Input and Learning Objective (moderately novel)
What are the main results? Are they significant?
Lexical overlap between languages plays a negligible role in the cross-lingual success, while the depth of the network is an important part of it.
Strengths
Novel findings about the effect of network architecture, input representation and learning objective on cross lingual ability of M-BERT
Weaknesses
Pires et al. (2019) work has been misrepresented. (see above for more details)
Pires et al. (2019) did  study linguistics properties and similarities of target and source languages for Multilingual BERT and had similar findings as this work.
Questions
What kind of difference in the numbers is considered significant by the authors ? For example, according to them, an increase of more than 2 points in accuracy in Table 3 (e.g. 1 head vs. 6 heads) is considered insignificant. But a decrease of a point in accuracy in Table 5  (e.g. enfake-ru NSP vs. No-NSP) is significant.