Visually-grounded embodied language learning models have recently shown to be effective at learning multiple multimodal tasks such as following navigational instructions and answering questions. In this paper, we address two key limitations of these models, (a) the inability to transfer the grounded knowledge across different tasks and (b) the inability to transfer to new words and concepts not seen during training using only a few examples. We propose a multitask model which facilitates knowledge transfer across tasks by disentangling the knowledge of words and visual attributes in the intermediate representations. We create scenarios and datasets to quantify cross-task knowledge transfer and show that the proposed model outperforms a range of baselines in simulated 3D environments. We also show that this disentanglement of representations makes our model modular and interpretable which allows for transfer to instructions containing new concepts. †

I thank the authors for their detailed response and appreciate their hard work in bringing us this paper.
I think that my main point is that this work relies too much on the extra information/constraints in the synthetic env. E.g., 1. since the vocab size is small, thus the feature map could be designed 'equal to the vocabulary size' 2. The bag-of-words representation is effective but it is not the case for natural language. Although the authors kindly point me to some recent works on sim2real, I am still not convinced whether this proposed method could be transferred to real setups based on the referenced papers.
However, it is a personal research taste that I always take real setup into considerations, because I have worked on both synthetic and real setup (on both lang and visn sides) for years and observed a large gap. My opinion is that methods of synthetic setups are not naturally convertible to the real ones. If AC/meta-reviewer considers the ability of vision-and-language interactions could be effectively studied through this setup with synthetic language and simulated-unrealistic images, I am OK with acceptance. I have downgraded my confidence scores (but kept my overall score) for this purpose.
-----------------------------------------------------------------------------------
Pros:
(1) The proposed model makes sense to me, which tries to have two attention layers to extract the information related to the questions. It seems to have the ability to deal with "and"/"or" logical relationships as well.
(2) Fig. 4 is impressive. It is clear and well-designed.
(3) The results in Table 2 are convincing. They show that both the proposed dual-attention method and multi-task learning would contribute to the performance.
Cons:
(1) It seems that the two main contributions are related to the language. Thus the synthetic language might not be proper to study. For example, in Eqn. 2, the first GA multiplies the BOW vector with the vision feature map, which could filter out unrelated instruction. This method could not be directly transferred to a real setup where natural language and natural images are involved.
(2) The designed attention modules is lack of generalizability. It implements a two-step attention module, while the first step selects the related visual regions w.r.t the words and the second step gathers the information regarding these attended regions. However, it might not be aware of the spatial relationships and thus be limited to simple questions. For example, if the question is "What is the object on top of the apple?". To my understanding, the current module would not explicitly handle this one-hop spatial relationship.
Comments:
(1) According to Sec. 3, 70 instructions and 29 questions are involved in this task. Using GRU to encoder these questions seems to be redundant. A simple one-hot embedding for these instructions might already be enough to encode the information.
(2) I am not sure why the visual attention map x_S could be used as the state of the module.
(3) After Eqn. 3, the paper says that "ReLU activations ... make all elements positive, ensuring ...". I am confused about the intuition behind this argument because of the softmax activation. Softmax will projects 0 to 1. So the sum of the all-zero vector would still be non-zero after softmax.
Typo:
- In Sec. 4, X_{BoW} \in \{0, 1\}^V.
- In Sec. 4.1, "this matrix is multiplied ..." --> this tensor.


We study the problem of regret minimization for distributed bandits learning, in which M agents work collaboratively to minimize their total regret under the coordination of a central server. Our goal is to design communication protocols with near-optimal regret and little communication cost, which is measured by the total amount of transmitted data. For distributed multi-armed bandits, we propose a protocol with near-optimal regret and only O(M log(M K)) communication cost, where K is the number of arms. The communication cost is independent of the time horizon T , has only logarithmic dependence on the number of arms, and matches the lower bound except for a logarithmic factor. For distributed d-dimensional linear bandits, we propose a protocol that achieves near-optimal regret and has communication cost of order O ((M d + d log log d) log T ), which has only logarithmic dependence on T .

The paper considers the problem of distributed multi-arm bandit, where M players are playing in the same stochastic environment. The goal of the paper is to have small over-all regret for all the players without a significant amount of communication between the players.
The main contribution of this paper is obtaining regret ~root(M KT) with ~M bits of communication in MAB, and regret ~d*root(MT) with ~Md bits of communication in linear bandit setting.
The main intuition of the algorithms in this paper is to do "best arm identification" with epoching: At every epoch t, the central server sends the set of possible best arms to each player and each player pulls it for 2^t /M times, followed by a communication round. Thus, the cumulative regret is comparable to having one player doing this epoch strategy for MT iterations, where the regret follows.
The problem considered in this paper is interesting and the result is new, the technique looks simple on paper but it requires a masterful combination of known tricks in (linear) MAB to obtain the best bound.
It seems that in the MAB setting, the lower bound could be further strengthened with a log(K) factor, since removing this factor would ultimately require "dynamic epoching" which is not possible with limited communication. This would mostly complete the picture in the distributed MAB regime.
Missing citation:
The authors are missing citations relevant to distributed MAB with collisions, see for example
"Non-Stochastic Multi-Player Multi-Armed Bandits: Optimal Rate With Collision Information, Sublinear Without"
After Rebuttal: I have read the authors' responses and acknowledge the sensibility of the statement.


Transferability of learned features between tasks can massively reduce the cost of training a neural network on a novel task. We investigate the effect of network width on learned features using activation atlases -a visualization technique that captures features the entire hidden state responds to, as opposed to individual neurons alone. We find that, while individual neurons do not learn interpretable features in wide networks, groups of neurons do. In addition, the hidden state of a wide network contains more information about the inputs than that of a narrow network trained to the same test accuracy. Inspired by this observation, we show that when fine-tuning the last layer of a network on a new task, performance improves significantly as the width of the network is increased, even though test accuracy on the original task is independent of width.

This paper considers the effect of network width of the neural network and its ability to capture various intricate features of the data. In particular, the central claim of this paper is what the title claims "Wider networks learn features that are better". They make this claim using the visualization technique called "activation atlasses". They find that wider networks learn features in the hidden neurons that are more "interpretable" in this visualization framework. Additionally, they also notice that fine-tuning a _linear model_ using the learned features for the wider networks provide better accuracy for new (but related) tasks over the shallower counterparts. For most experiments of this paper, "shallow network" refers to a width of 64 and "wide network" refers to a width of 2048. The main datasets used for the experiments are MNIST, CIFAR 10/100 and a "translated" version of MNIST images.
Overall the paper is written well and the ideas and results are communicated crisply. I have a few comments. First, regarding the related work, I think that the reader would be served better if the authors also list the recent works related to effect of network width on convergence and generalization (e.g., [1] and references that cite this). The reason I say this is so that the reader should not (wrongly) interpret that this is the first work that finds "favorable" properties of wider networks (the paper does not make this claim, but it is easy for a reader to interpret it). Second, I find it slightly concerning that a lot of findings have been extrapolated from just one architecture. In particular, I find the experiments in section 5 to be the most informative (and also objective), since it is a single number which is easy to think about. To be clear, I like the visualization experiments and it gives credibility to the claim about interpretability. Given that there are many levers in a neural net (batch norm, architectural choices, hyper-params etc.) one could fiddle with, to make the claim made in the introduction one needs a more extensive set of experiments. I acknowledge that the authors say they haven't explored the possibility of fine-tuning the hyper-params for instance, but I think considering some of these choices is really helpful. This will help _isolate_ the effect of width independent of the architecture choice.
Given the above observations, my current decision of this paper is that it doesn't meet the bar. I find the results promising but the paper is not yet ready.
[1] - https://papers.nips.cc/paper/8076-neural-tangent-kernel-convergence-and-generalization-in-neural-networks.pdf


Reinforcement learning algorithms use correlations between policies and rewards to improve agent performance. But in dynamic or sparsely rewarding environments these correlations are often too small, or rewarding events are too infrequent to make learning feasible. Human education instead relies on curricula-the breakdown of tasks into simpler, static challenges with dense rewards-to build up to complex behaviors. While curricula are also useful for artificial agents, handcrafting them is time consuming. This has lead researchers to explore automatic curriculum generation. Here we explore automatic curriculum generation in rich, dynamic environments. Using a setter-solver paradigm we show the importance of considering goal validity, goal feasibility, and goal coverage to construct useful curricula. We demonstrate the success of our approach in rich but sparsely rewarding 2D and 3D environments, where an agent is tasked to achieve a single goal selected from a set of possible goals that varies between episodes, and identify challenges for future work. Finally, we demonstrate the value of a novel technique that guides agents towards a desired goal distribution. Altogether, these results represent a substantial step towards applying automatic task curricula to learn complex, otherwise unlearnable goals, and to our knowledge are the first to demonstrate automated curriculum generation for goal-conditioned agents in environments where the possible goals vary between episodes.

This paper proposes an autocurricula scheme to train a goal-conditional agent in a dynamic and sparse-rewarding environment. The main idea is to train a setter model to sample goals for next-step training, where the setter can make the decision either based on the training history or the environmental observation (conditional case). The paper proposes three criteria which leads to three types of loss to train the setter model, i.e., goal validity (the goal should be achievable by some existing policy), goal feasibility (how probable the current policy can achieve the goal), and goal coverage (the sampled goals by the setter need to cover all possible goals). A judge model is needed to output the feasibility of a given goal. So the autocurricula scheme contains the solver (agent), the setter, and the judge, each having its own combination of loss and they are trained together. Given a desired goal distribution, the paper proposes to additionally train a discriminator whose optimization objective is Wasserstein loss. In experiments, they evaluate the proposed method on three types of tasks in two environments, i.e., 3D color finding and grid world alchemy. The goals in the two environments are similar in that they all aim to achieve some color or color pairs. The difference lies in that the first one finds colors while the second pick up colors. Each environment can be changed between episodes by changing the colors of objects in the scenes. Experimental results show that different combinations of the three types of losses can bring improvements in some scenarios. Making setter and judge conditioned on environment observation can further improve the success rate. Given a desired distribution of goals, the learning becomes more efficient. The paper compares this method with Goal GAN as a baseline and outperforms it on the three tasks.
This paper is an early exploration of the effectiveness of autocurricula on goal-conditional tasks. The experiments to some extent verify the effectiveness of the proposed method. The high-level idea of setter-solver learning can be possibly generalized to other tasks. However, it is not convincing that the detailed techniques proposed in this paper can be easily generalized to more complicated environments and tasks. The writing of this paper is not clear and enough and the whole paper is difficult to understand. The experimental comparison is insufficient since only one baseline of curriculum (goal GAN) is compared (some of the methods mentioned in Section 2 can be easily applied though some of them were not specifically designed for goal-conditional tasks).
Detailed comments:
1) There are too many details and hyperparameters involved when applied to specific tasks as shown in this paper. Considering the two environments in this paper is easier than most game environments studied in other papers (many of them also uses some types of the curriculum), the experimental result is not very strong. The proposed scheme is possible to be too simple for a slightly more complicated environment or task.
2) The success of training relies on interactions between the three to four types of losses, but they all have the same weights in the combined loss. Is there any special reason for not setting different weights for different losses?
3) It is not clear how the inverse transform of the setter model (i.e., S^{-1} in L_val) is achieved.
4) There are three random quantities involved in the loss terms, i.e., the noise \xi applied to the sampled goals, the latent input z, the feasibility f. It is not clear whether the randomness on them will dominate the curriculum or not.
5) The setter model takes z and f as inputs, where z is a vector but f is a scalar. Will it result in a setter model whose output does not change too much when changing f?
6) How does the proposed method compare to simple goal selection by uncertainty, hardness, or curiosity? How does it compare to hindsight experience reply methods?
7) It is helpful to visualize the generated curriculum, i.e., the trajectory of selected goals during training in the 2D/3D grid.
8) Computing L_judge requires to test whether the agent can finally achieve the sampled goals. Is it too computational expensive during training?
---------------------------
Update after rebuttal:
Thanks for the detailed reply from the authors! They answer most of my questions. I think this paper has a moderate contribution and thus will keep my "weakly accept" rating.


Recent work suggests improving the performance of Bloom filter by incorporating a machine learning model as a binary classifier. However, such learned Bloom filter does not take full advantage of the predicted probability scores. We proposed new algorithms that generalize the learned Bloom filter by using the complete spectrum of the scores regions. We proved our algorithms have lower False Positive Rate (FPR) and memory usage compared with the existing approaches to learned Bloom filter. We also demonstrated the improved performance of our algorithms on real-world datasets.

The paper proposed an adaptive learned bloom filter. Rather than setting a threshold of prediction score, the paper partitions the score into several intervals; for query insider each interval, the paper either uses a group of independent hash functions to hash the query in one unified bloom filter or introduce an independent bloom filter. The paper proposes efficient ways to tune the hyper-parameters, and provides the analysis of the error. Experiments on two applications show the effectiveness of the proposed methods.
The idea seems useful. However, I have a few concerns/questions. My decision depends on the responses of the author(s).
(1) Although the paper claims the score information can be fully exploited, the paper seems to do hashing for all possible queries. Why not set up a tau such that only when the score is below tau, we conduct the adaptive hashing? When the score is bigger than tau, we still claim an identification of the key? In this way, a bunch of keys can still be saved and without extra hashing.
(2) The proposed method seems to be a hierarchical hashing strategy.  The first level is to hash the queries into different intervals through a score function learned from data. Why not compose another group of random hash to do the first level hashing? What is the major benefit of collecting training examples to run a machine learning model? Accordingly, why not compare with such a baseline using a group of random hashing to do the first level?


When presented with Out-of-Distribution (OOD) examples, deep neural networks yield confident, incorrect predictions. Detecting OOD examples is challenging, and the potential risks are high. In this paper, we propose to detect OOD examples by identifying inconsistencies between activity patterns and class predicted. We find that characterizing activity patterns by feature correlations and identifying anomalies in pairwise feature correlation values can yield high OOD detection rates for far-from-distribution examples. We identify anomalies in the pairwise feature correlations by simply comparing each pairwise correlation value with its respective range observed over the training data. Unlike many approaches, this can be used with any pre-trained softmax classifier and does not require access to OOD data for fine-tuning hyperparameters, nor does it require OOD access for inferring parameters. The method is applicable across a variety of architectures and vision datasets and, for the important and surprisingly hard task of detecting far-from-distribution outliers, it generally performs better than or equal to state-ofthe-art OOD detection methods (including those that do assume access to OOD examples).

This paper uses Gram matrices for OOD detection. This enables the reliable detection of far-from-distribution examples, which is a long-standing and surprisingly difficult problem in OOD detection.
This is the best paper in my batch due to the strength of the results. However, this paper should more accurately reflect the contribution: this helps with far-from-distribution examples, not near-distribution yet OOD examples.
For instance, I used their code and found that their technique leads to an AUROC of 79.01% when using CIFAR-10 as the in-distribution and CIFAR-100 as the OOD set. Likewise, with their code I found having CIFAR-100 as in and CIFAR-10 as out gives an AUROC of 67.95%. This is much worse than currently existing techniques. Hence the paper should re-frame or qualify their results as helping with far-from-distribution detection or the detection of obvious anomalies. This paper makes a solid stride in improving the detection of garbage inputs, but the paper should modify its message so as not to suggest this helps with all currently considered OOD detection tasks.
Small comments:
> Lee et al. (2018b)—to the best of our knowledge, the current SOTA technique by a significant margin
This should be again qualified and expanded. If you assume access to knowledge of the test distribution, then Mahalanobis is easily the best. If not, the Outlier Exposure is best. If you assume access to no extra data during training, then the maximum softmax probability + rotation prediction is best [1].
> However, while the OE method is able to generalize across different non-training distributions, it does not achieve the SOTA rates of Lee et al. (2018b) on most cases.
There are different senses of state-of-the-art and these should be qualified.
Does this technique do better if you do 5th and 9th percentile instead of min and max? Is it important to do the min and max with training examples instead of validation examples? (Not a pressing question.)
> Can work without access to OOD validation examples?
Table 1 is deceptive. OE does not need the "validation" examples.
I suggest two columns instead of one. "Can this work without knowledge of OOD test examples? Does this use OOD training examples?"
Show AUROC and AUPR. Detection accuracy is an unusual metric relative to AUPR (OOD as positive).
Show CIFAR-10 vs CIFAR-100 in the tables or I'll downgrade my rating, since otherwise the paper is not leaving an accurate impression.
Since OE is complementary, perhaps this technique can be combined to tackle these near-distribution cases?
The title is confusing. "Zero-Shot" could be applied to various techniques in this space. Perhaps emphasize Gram matrices?
In their code:
        validation_indices = random.sample(range(len(all_test_deviations)),int(0.1*len(all_test_deviations)))
        test_indices = sorted(list(set(range(len(all_test_deviations)))-set(validation_indices)))
These indices change with every power, which is unrealistic. Please fix the sets beforehand.
Since neural style transfer, which uses Gram matrices, works much better with VGG architectures than ResNets, does this technique work better with VGG architectures?
[1] Using Self-Supervised Learning Can Improve Model Robustness and Uncertainty. Hendrycks et al. NeurIPS 2019.


We investigate statistical methods for automatically scheduling the learning rate (step size) in stochastic optimization. First, we consider a broad family of stochastic optimization methods with constant hyperparameters (including the learning rate and various forms of momentum) and derive a general necessary condition for the resulting dynamics to be stationary. Based on this condition, we develop a simple online statistical test to detect (non-)stationarity and use it to automatically drop the learning rate by a constant factor whenever stationarity is detected. Unlike in prior work, our stationarity condition and statistical test apply to different algorithms without modification. Finally, we propose a smoothed stochastic line-search method that can be used to warm up the optimization process before the statistical test can be applied effectively. This removes the expensive trial and error for setting a good initial learning rate. The combined method is highly autonomous and it attains state-of-the-art training and testing performance in our experiments on several deep learning tasks.

This paper proposes a new way of automatically scheduling the learning rate in stochastic optimization algorithms: Stochastic Approximation with Line-search and Statistical Adaptation (SALSA).
By first introducing a necessary condition for stationarity, the authors use this condition to make a simple statistical test for non-stationarity. Using this test, the authors propose the following strategy for the learning rate schedule in stochastic optimization problems:
(1) They first apply a new Line-Search algorithm (Smoothed Stochastic Line-Search - SSLS) to increase a small initial learning rate until the process becomes stationary according to their statistical test. At this stage, the learning rate is assumed to be optimally initialized for the objective function considered.
(2) The second step is to decrease the learning rate gradually every time the process is stationary again. For this, the authors derived their own version of a Statistical Adaptive Stochastic Approximation algorithm called SASA+ based on their statistical test.
The resulting strategy benefits from being simpler than previous statistical tests while being equally effective empirically.
The authors empirically demonstrated that their new learning rate scheduling mechanism achieves comparable, if not better, accuracy on two image classification tasks with ResNet-18 neural networks. It is important to note that the compared baselines got their parameters slightly fine-tuned, while (according to the authors) the proposed approach was not fine-tuned, and only the default parameters were used. This shows the robustness of the proposed approach against its various parameter settings.
I would accept this submission because the authors propose a new learning rate scheduling mechanism that seems to perform well empirically while being robust against its different initial parameter settings (including the choice of the initial learning rate).
This paper has several novelties: first, it proposes a simpler, yet as effective as previous approaches, Statistical Adaptive Stochastic Approximation (SASA) algorithm based on a new statistical test for non-stationarity. Second, it manages to relax the dependence of SASA algorithms on their optimal initial learning rate by introducing a Smoothed Stochastic Line Search (SSLS) algorithm that is responsible for finding such an optimal initial learning rate. Combined together, these two sub-routine provide a robust mechanism to schedule the learning rate in stochastic optimization problems.
One improvement I could suggest to better motivate the proposed approach is to experiment it not only on Convolutional-based networks with image classification tasks but also on Recurrent-based networks with text datasets. For instance, keeping the ImageNet experiments, the CIFAR-10 experiments could be replaced by an NLP task. This would show that the proposed approach is robust to different types of deep learning problems.
Overall I found the paper well written, and relatively easy to follow, even for non-theoretical practitioners. A few details listed below could improve even more the quality of this paper:
- At the top of page 2, the last sentence of the top paragraph ("However, these learning rate schedules are insufficient ...") requires a citation.
- In Figure 5: it is a little confusing to have the SASA+(NAG) algorithm in both rows: once in the top row, twice in the bottom row. The difference between the two SASA+(NAG) in the bottom row is well explained, but is there any difference between SASA+(NAG) in the top row and the ones in the bottom row?
- In Figure 5 - bottom row, left graph: if all lines are SASA+ algorithm, the legend should be consistent: either add "sasa+" to all lines or remove it from all lines.
- On page 9, ImageNet paragraph: a small typo: "On the other hand, both both SASA+ and SALSA...".


We introduce a novel synthetic oversampling method for variable length, multifeature sequence datasets based on autoencoders and generative adversarial networks. We show that this method improves classification accuracy for highly imbalanced sequence classification tasks. We show that this method outperforms standard oversampling techniques that use techniques such as SMOTE and autoencoders. We also use generative adversarial networks on the majority class as an outlier detection method for novelty detection, with limited classification improvement. We show that the use of generative adversarial network based synthetic data improves classification model performance on a variety of sequence data sets.
Dealing with imbalanced datasets is the crux of many real world classification problems. These problems deal with complex multivariate data such as variable length, multi-feature sequence data. Canonical examples can be found in the finance world, for example, questions related to stock market data of several securities or credit card fraud detection often deal with sequence data with many features. Other imbalanced data problems include questions in the medical field such as tumor detection and post surgery prognosis (Zięba et al., 2014). In each of these problems, false positives are more desirable than false negatives, they require sequential data, and the classes are imbalanced.
Class imbalances in datasets oftentimes lead to increased difficulty in classification problems as many machine learning algorithms assume that the dataset is balanced. There are two general approaches to improve classification accuracy for unbalanced datasets. One method is algorithmic, for example, a modified loss function can be used so that misclassifications of minority labeled data are penalized more heavily than misclassifications of majority labeled
This motivates the development of methods to improve classification accuracy on variable length, multi-feature sequence data. Given a sequence of T feature vectors, we want to predict labels of the sequence. Oftentimes it is not obvious how to apply methods for unbalanced data to sequence data in a way that takes advantage of the fact that sequential events have the potential to be highly correlated. SMOTE (Chawla et al., 2002) is widely used for oversampling, but does not capture the sequential dimension. Enhanced Structure Preserved Oversampling (ESPO) (Cao et al., 2013) allows one to generate synthetic data that preserves the sequence structure, however it requires that the feature vector has only a single feature at each of the T time points and that the output label is a scalar. As there is no obvious extension to the case where there are multiple features at each time point and the output is also a sequence of labels, the situations where ESPO can be applied are limited.
We develop a method based on deep learning models for sequences in order to decrease data imbalances of sequence data with an arbitrary number of features. We call each feature vector, x i ∈ R n , an event in the sequence. We consider the use of generative adversarial networks (GANs) to generate synthetic data. Here, we build a generative model that generates both the feature vectors in a sequence as well as the corresponding labels. We benchmark this synthetic data generation technique against a number of models. We demonstrate that the model trained on the GAN based synthetic data outperforms the baseline model, other standard synthetic data generation techniques, and a GAN based novelty detection method. For each of the synthetic data generation methods, we train a sequence-to-sequence model (Sutskever et al., 2014) on the dataset that outputs a sequence with the same length as the label sequence. In addition to benchmarking against existing synthetic data generation techniques, we also train a model on the unaugmented dataset. All of the models are embedded within the standard ensemble approach. On all of our datasets, we observe that the GAN based synthetic data generation model significantly improves over the baseline models by 15% to 127% depending on the dataset, while the novelty detection based GAN performs similarly to the baseline model.
The main contributions are as follows:
1. a novel synthetic data generation technique that uses a GAN with an Autoencoder component to generate synthetic data for variable length, multi-feature sequential data in a way that preserves the structure of sequences for both feature vectors and labels;
2. a new novelty detection method for sequential data that uses a GAN as an outlier detection function;
3. a computational study of existing imbalanced classification techniques on highly imbalanced sequential datasets.
In the next section, we discuss relevant literature. Section 3 discusses all of the models, while the computational results are presented in Section 4.

The paper consider important and interesting problem: how to generate a sequence from minority class if we want to do oversampling with synthetic data in a way similar to SMOTE.
Now from the paper, the exact used approach is not clear, as the details are too scarce (see some examples below). Experiments are not convincing, as the authors don't compare to the state of the art approaches. As the exact problem statement is hard to grasp, it is hard to identify the exact contribution of authors.
Note, that the proposed approaches, in my opinion, are not that different from approaches from modern data generation for imbalanced classification [1, 2], as we propose some kind of GAN to generate new data and so have only two variables: how we select loss for this GAN and how we select the architecture.
I assume, that to be accepted at a major venue a deeper investigation is required at the moment.
See also the following comments:
1. Figure 4: title is not required, as we have a caption with the same information. Better to use confidence bars too.
2. The figures will benefit from usage of vector format.
3. Figure 3: tSNE can vary from one run to another. It is better to provide at least 3 random figures or even better train e.g. a simple classification model for t-SNE embedded model and present ROC AUC scores for identification synthetic/non-synthetic.
4. Table 1&2 formatting is different from that usually used in Academy (see e.g. https://dl.sciencesocieties.org/files/publications/style/chapter-05.pdf)
5. F1 score is often not the best metric for imbalanced problems. The paper will benefit from providing also PR AUC (average precision) scores.
6. Figure 2: avoid confusion matrices presented in this form, as they take much space providing almost no information. Classic tables are better.
7. From the problem statement at the very beginning of section 2 it is not clear what kind of labels do we expect (I suppose that for each sequence we have a specific label i.e. all y_i are 0 but one, that is 1)
8. Sometimes bigger weights for minority objects or dropping significant part of majority sequences are enough, so results for these approaches also should be included
9. How the hyperparameters mu and lambda were selected?
[1.] Guo, Ting, et al. "Discriminative Sample Generation for Deep Imbalanced Learning." Proceedings of the 28th International Joint Conference on Artificial Intelligence. AAAI Press, 2019. IJCAI 2019, https://www.ijcai.org/proceedings/2019/0334.pdf
[2.] Douzas, Georgios, and Fernando Bacao. "Effective data generation for imbalanced learning using conditional generative adversarial networks." Expert Systems with applications 91 (2018): 464-471.


Learning rich representations from predictive learning without labels has been a longstanding challenge in the field of machine learning. Generative pre-training has so far not been as successful as contrastive methods in modeling representations of raw images. In this paper, we propose a neural architecture for selfsupervised representation learning on raw images called the PatchFormer which learns to model spatial dependencies across patches in a raw image. Our method learns to model the conditional probability distribution of missing patches given the context of surrounding patches. We evaluate the utility of the learned representations by fine-tuning the pre-trained model on low data-regime classification tasks. Specifically, we benchmark our model on semi-supervised ImageNet classification which has become a popular benchmark recently for semi-supervised and self-supervised learning methods. Our model is able to achieve 30.3% and 65.5% top-1 accuracies when trained only using 1% and 10% of the labels on ImageNet showing the promise for generative pre-training methods.

Contributions:
The paper aims to develop generative pre-training method for learning representations of images. Although representation learning for images has been widely investigated, the present work distinguishes itself by a combination of the following:
a) building on the use of transformers as a series of layers after initial convolutional layers;
b) using self attention for aggregating context;
c) learning spatial dependencies across patches;
d) training on the task of predicting two bit gray scale version of randomly masked patches in an image
Results:
Limited experiments aim to compare against the CPC (Hefnaf et al 2019) and Selfie (Trinh et al 2019) algorithms both of which are contrastive unlike the generative approach adopted in the paper. After pre-training on unlabeled imageNet datasets the proposed approach is competitive with these algorithms with roughly similar results.
Evaluation/Suggestions:
Overall the paper combines ideas from several previous works in ways that are not sufficiently novel in the opinion of this reviewer and the experiments are very limited to the imageNet dataset with 1%, 10% and 20% of labels provided to downstream classification modeling, and evaluated on top-1 and top-5 accuracies. The paper could improve on its experimental evaluation bycomparing on multiple datasets, showing error bars when averaging across multiple samplings (eg for getting the 1% label set from the entire imageNet dataset) and also comparing with other approaches even when they dont directly aim to learn representation from unlabeled data (eg Image Transformers by Parmar et al). In addition the description is very high level and does not provide enough details for experimental reproducibility. For example the reviewer had to actually guess at some of specifics of the overall end-to-end architecture since it was not fully described precisely eg in a diagram. It would be relatively easy (but important) to provide suffcient detail for reproducibility


The critical locus of the loss function of a neural network is determined by the geometry of the functional space and by the parameterization of this space by the network's weights. We introduce a natural distinction between pure critical points, which only depend on the functional space, and spurious critical points, which arise from the parameterization. We apply this perspective to revisit and extend the literature on the loss function of linear neural networks. For this type of network, the functional space is either the set of all linear maps from input to output space, or a determinantal variety, i.e., a set of linear maps with bounded rank. We use geometric properties of determinantal varieties to derive new results on the landscape of linear networks with different loss functions and different parameterizations.

The authors use tools from algebraic geometry to study the critical points that arise in linear neural networks under a variety of loss functions.  In particular, they study linear maps that are parameterized (or overparameterized) as the product of multiple matrices.  The authors introduce a distinction between two types of critical points (for the linear function composed with a provided loss): there are those that are due to the actual manifold of functions that are provided to the loss (pure critical points), and there are those that are due to parameterizations of this manifold (spurious critical points).  The authors show that in the case that the linear function has maximal rank (is "filling"), then there are no non-global minima in the loss landscape.  In the case of lower rank ("nonfilling") linear functions, this property does not hold generally, but does hold for quadratic losses.   The authors show that for this framework, there can be exponentially many disconnected components in parameter space corresponding to global minima.
This paper provides important insight into the loss landscape of neural networks, and it exhibits nuanced observations even in the linear case.  Naturally, there are many important follow-up studies that can be done in the case of non-linear networks.  It is vital to have a thorough understanding before that work can be satisfactorily undertaken.
It would benefit the authors to provide more summary/commentary in the main results section.  Currently, that section comes across as a list of lemmas, theorems, propositions, and corollaries, with relatively little narration in between.  It was challenging for me to identify which of these items were the main results, as summarized on in the first paragraph of the conclusions section.  I recommend that any theorem/corollary/proposition that is the primary basis for a claim made in the conclusion should receive significantly more commentary in Section 3.
Minor comment:
In the first paragraph of the conclusion, it says that "... the absence of ... is a rather general property ...".  The word "rather" should be clarified.


We present Group-size Series (GroSS) decomposition, a mathematical formulation of tensor factorisation into a series of approximations of increasing rank terms. GroSS allows for dynamic and differentiable selection of factorisation rank, which is analogous to a grouped convolution. Therefore, to the best of our knowledge, GroSS is the first method to simultaneously train differing numbers of groups within a single layer, as well as all possible combinations between layers. In doing so, GroSS trains an entire grouped convolution architecture search-space concurrently. We demonstrate this with a proof-of-concept exhaustive architecure search with a performance objective. GroSS represents a significant step towards liberating network architecture search from the burden of training and finetuning.

Summary
---
This paper proposes to learn simultaneously all the parameters of grouped convolutions by factorizing the weights of the convolutions as a sum of lower rank tensors. This enables architecture search for the convolution parameters in a differentiable and efficient way.
Comments
---
I think was paper is well written, and was clear at least until 3.2. I believe some clarifications could be useful here, it is not written clearly that t' and u', the 2 first dimensions of the core are R times smaller than t and u. There is some explanation in the bracket (3) but 1) it should be stated clearly in the text, 2) I believe there are several typos on the 4th line of bracket (3) making it hard to understand.
I did not know about the expansion function, and while I trust the authors that it is correctly used, I would have like either more explanations on how it works or some reference.
Can you justify the softmax and the very high temperature? For N = 8, s_1 will be sampled 98.2% of the time s_2 1.8% and the other sampling probabilities are close to neglibigle. While I understand it seems to work better in practice, it looks extremely aggressive.
In 4.4 you say you perform finetuning for 150 epochs, which is huge, while on the abstract you said "GroSS represents a significant step towards
liberating network architecture search from the burden of training and finetuning". Can you comment?
As you say GroSS is an alternative to NAS (for the convolutions parameters that is), is the GroSS method proposed really faster and more accurate than a NAS baseline for finding these architectures?
I don't find the column titles in Table 3 to be always informative. "After train" means after the finetuning? I took me some time to realize the delta was the delta in accuracies, it is not very informative and it was not clear for me for some time what it meant. Either the titles should be chosen more carefully or the caption should be more precise I believe.
In figure 1, the legend should be more informative, at least incorporate a "alpha" or "temperature" title in the legend.
Conclusion
---
While the method is interesting I am wondering whether GroSS enables more efficient architecture search that tradional methods as there is still a long finetuning step, furthermore it can only be applied to grouped convolutions parameters. As the authors present it in the abstract and introduction as an alternative to NAS, I believe a comparison to a NAS would be needed.


Despite the growing interest in continual learning, most of its contemporary works have been studied in a rather restricted setting where tasks are clearly distinguishable and task boundaries are known during training. However, if our goal is to develop an algorithm that learns as humans do, this setting is far from realistic and it is essential to develop a methodology that works in a task-free manner. Meanwhile, among several branches of continual learning, expansion-based methods have the advantage of eliminating catastrophic forgetting by allocating new resource to learn new data. In this work, we propose an expansion-based approach for task-free continual learning for the first time. Our model, named Continual Neural Dirichlet Process Mixture (CN-DPM), consists of a set of neural network experts that are in charge of a subset of the data. CN-DPM expands the number of experts in a principled way under the Bayesian nonparametric framework.
With extensive experiments, we show that our model successfully performs taskfree continual learning for both discriminative and generative tasks such as image classification and image generation.

This paper proposes Continual Neural Dirichlet Process Mixture Model (CN-DPM) to solve task-free continual learning. The core idea is to employ Dirichlet process mixture model to create novel experts in online fashion when task distributions change.  The proposed method is validated on various tasks and demonstrated to perform well compared to the other baselines.
Overall I find this paper to be well-written and the experiments are conducted thoroughly. The method is compared to proper baselines in various settings, and the paper describes detailed experimental settings and architectural choices to help readers willing to reproduce. I’ve gone through the appendix and they provide enough additional experiments to support the author’s claim.
The main algorithm itself cannot be considered to be novel. DPM or other Bayesian nonparametric models have been extensively used for the problems requiring to adapt the model size according to the change of data. Nevertheless, the application of DPM in task-free continual learning context seems to be considered as a contribution.
I have much experience in implementing Bayesian nonparametric models with parametric distributions and compared various methods to conduct the posterior inference of them. In my experience, even for the low-dimensional parametric models, the posterior inference algorithms for DPM usually suffer from local optima, and the sequential methods such as SVA depends heavily on the data processing order. According to the experimental setting presented in the paper, the algorithm goes through a single pass over the data stream, yet still able to reasonably train (deep) neural networks and identify mixture components jointly. Do you have any intuition about how this becomes feasible?
I don’t fully understand why generative modeling is required. In page 4 the authors stated that the generative model prevents catastrophic forgetting. But in my understanding, using expert-specific parameters is the part that prevents catastrophic forgetting, not the generative model itself. Learning generative model in online fashion may work well in simple structured data such as MNIST,  but I highly doubt that the generative model could be trained properly for CIFAR10 or CIFAR100, especially in online setting. My concern is that learning generative model part may even impede the discriminative learning. Could you elaborate more on this?
Another minor concern is the way the concentration parameter alpha is selected. The authors stated that they chose proper value of alpha according to the number of tasks known in advance. I think this does not make sense. Alpha should also be inferred along with other parameters, or fixed to non-informative value if the performance of the algorithm is not very sensitive to the choice of alpha.
I think it would be more helpful to show how the task-assignment p(z=k|x) is learned. For instance, the clustering accuracy according to p(z=k|x) against the ground-truth task label can be measured, or at least qualitatively show what examples were assigned to each task.


Some conventional transforms such as Discrete Walsh-Hadamard Transform (DWHT) and Discrete Cosine Transform (DCT) have been widely used as feature extractors in image processing but rarely applied in neural networks. However, we found that these conventional transforms have the ability to capture the cross-channel correlations without any learnable parameters in DNNs. This paper firstly proposes to apply conventional transforms on pointwise convolution, showing that such transforms significantly reduce the computational complexity of neural networks without accuracy degradation. Especially for DWHT, it requires no floating point multiplications but only additions and subtractions, which can considerably reduce computation overheads. In addition, its fast algorithm further reduces complexity of floating point addition from O(n 2 ) to O(n log n). These non-parametric and low computational properties construct extremely efficient networks in terms of the number of parameters and operations, enjoying accuracy gain. Our proposed DWHT-based model gained 1.49% accuracy increase with 79.4% reduced parameters and 49.4% reduced FLOPs compared with its baseline model (MoblieNet-V1) on the CIFAR 100 dataset.

This paper presents a new pointwise convolution (PC) method which applies conventional transforms such as DWHT and DCT. The proposed method aims to reduce the computational complexity of CNNs without degrading the performance. Compared with the original PC layer, the DWHT/DCT-based methods do not require any learnable parameters and reduce the floating-point operations. The paper also empirically optimizes the networks by removing ReLU after the proposed PC layers and using conventional transforms for high-level features extraction. Experiments on CIFAR100 show that the DWHT-based model improves the accuracy and reduces parameters and FLOPs compared with MobileNet-V1.
Although this paper is well organized and easy to follow, the novelty of the proposal seems limited and the performance improvement claimed by the author(s) is not very convincing due to the insufficiency of experiments. Firstly, the proposed method is just a manually designed and fixed 1*1 convolutional kernel, and its superiority over random initialization seems very limited as shown in Table 1. Also, the proposed method makes accuracy degrade when applied to low- and middle-level features. I wonder whether there is a more theoretical explanation for that. Moreover, the experiments are performed only on a small dataset CIFAR100. According to my own experience, the artificial convolutional kernels with some prior knowledge may work well on small datasets but tend to fail on larger ones. More experiments on larger-scale datasets like ImageNet are recommended to make results more convincing.
Therefore, my decision leans to a rejection.
Some questions:
1. How is the performance when applying RCPC only to low-/middle-/high-level features? I suppose it should be proved that the proposed method is definitely better than random initialization.
2. Why is only applying the proposed block to high-level layers working? How is the trade-off between parameters and accuracy different for each level of features?
Spelling mistake:
Page 6: in the second last paragraph, 'non-parameteric' should be 'non-parametric'.


Applying reinforcement learning (RL) to real-world problems will require reasoning about action-reward correlation over long time horizons. Hierarchical reinforcement learning (HRL) methods handle this by dividing the task into hierarchies, often with hand-tuned network structure or pre-defined subgoals. We propose a novel HRL framework TAIC, which learns the temporal abstraction from past experience or expert demonstrations without task-specific knowledge. We formulate the temporal abstraction problem as learning latent representations of action sequences and present a novel approach of regularizing the latent space by adding information-theoretic constraints. Specifically, we maximize the mutual information between the latent variables and the state changes. A visualization of the latent space demonstrates that our algorithm learns an effective abstraction of the long action sequences. The learned abstraction allows us to learn new tasks on higher level more efficiently. We convey a significant speedup in convergence over benchmark learning problems. These results demonstrate that learning temporal abstractions is an effective technique in increasing the convergence rate and sample efficiency of RL algorithms.

Summary:
This paper develops a method for learning a latent action representation based on prior experiences (and specifically, prior action sequences). Additionally, the paper proposes to regularize the learning of this representation using an information-theoretic constraint, yielding Temporal Abstraction with Information-theoretic Constraints (TIAC). Indeed, one promise of HRL is to allow for learning and decision making algorithms to take the long-term consequences of a decision into account when planning, exploring, assigning credit, or simply acting. The options framework (Sutton, Precup, and Singh; 1999) is a promising and well-studied toolkit for investigating these capacities of HRL. For this reason, the topic of the paper is well chosen: continuing to understand how options can benefit and accelerate RL in rich environments is an important direction for research. The idea at the core of the paper is new to my knowledge: learning an encoding of action sequences with a continuous latent representation. It could be a promising technique for HRL. Experiments are conducted to evaluate the effectiveness of the method in several environments, including a continuous gridworld, control tasks, and problems involving transfer learning.
Verdict: Due to lack of clarity in describing the main methods, and missing comparison to any HRL/option baselines, I recommend rejection.
More Detail:
The paper is lacking clarity in its current form. I view the main contribution as the development of the architecture and loss function that together learn an appropriate latent action representation. There are two key issues with clarity at present: 1) The presentation of the core technical contributions could be improved (see comments below in "Q1"), and 2) Motivation for this style of option learning is missing, with evidence that the proposed method is in fact learning an appropriate thing.
Toward (1): I provide suggestions where clarity could be improved below in "Q1'.
Toward (2): There are a few aspects of the motivation that could be improved. First, the paper mentions that the learned options/representation will help in planning, but planning is not studied in the paper. For example: "Further, the interpolations between two sequences smoothly transfer from one to the other, which is a desired property to have during planning, because the smooth option space provides the RL algorithm with a better search space." By my reading of the paper, this claim is not studied. Similarly, in the intro: "...allow us to do planning at a higher level, and easily transfer the knowledge between different tasks". Including experiments that explicitly evaluate the capacity of the learned representation to carry out planning would help support these claims. Or, alternatively, the contribution could be focused to model-free and policy-based learning, which is where the empirical evidence currently offers the most support. Second, no HRL baselines are compared to in the experiments. One natural comparison to include would be to the Option-Critic, which was the first technique for combining option learning with deep RL. To determine whether TIAC is a sensible approach to learning and using options, a comparison to at least one other option learning method is needed. The paper currently highlights the fact that the option-critic requires a pre-specified number of options: this is true, but it is not discussed why is this problematic, or how the current proposal remedies this difficulty. Others that may be relevant include FuN (Vezhnevets et al. 2017), the recent methods of Nachum et al. (2018), among others (Tiwari and Thomas 2019, Harb et al. 2018, Harutyunyan et al. 2019, Levy et al. 2019).
In short: the results here are promising, so I encourage the authors continue in this direction. The paper will be improved if the presentation of Section 3 is sharpened (see questions regarding clarity below) and a comparison with relevant baselines is included.
Main Questions:
Q1: The exposition of the main method (Section 3) was unclear to me. Here are a few questions I was left with:
	(a) Why is the posterior (on $o$) conditioned only on the action history, and not state?
	(b) Additionally $o$ is being treated as a random variable through 3.2. So, what is $o$? Where is the randomness coming from?
	(c) Section 3.3 states "it encodes the action sequences with respect to the L2 distance in action space". Does this mean the action space is always a subset of $\mathbb{R}$? But, it looks like $\mathcal{A}$ is just defined as some set: in Section 3.1, "$\mathcal{A}$ is the action set". So, I am confused as to what the $L_2$ is distance defined with respect to. If the actions are always assumed to be real numbers that is entirely okay, but it would be helpful to have that stated early on. From the additional text in Section 3.3, it sounds like the transition function of each action is involved in computing this distance ("...only have small difference in each step of action. Due to the error compounding, the two sequences...").
	(d) How is the estimate of the posterior actually used to act? The output of "D" in Figure 2 is $\hat{a_{0...k}}$. What is the type of this entity? Is it guaranteed to be an element of $\mathcal{A}$? If so, then the "option" here is a policy that maps $o$ and the action history $a_{0...k}$ to a new action, correct? Ah, so in Figure 3, it looks like D will have different output depending on how the termination condition is handled. Are the actions output by $D$ then executed by the RL agent, or is there some additional decision making that goes on downstream?
	(e) Early on the section states "In contrast to precisely reconstructing the action sequences, our goal is to extract the latent variable capturing  the information which could benefit RL training." It might be helpful to include some intuition about what this information would look like. It's unclear why action history would be all that meaningful on its own (without say, the state history). It would help the section to provide some intuition for such a latent variable existing; is there an idealized, simple case that would help convey the idea? Note that this proposal comes across as different from the original proposal of the options framework: As an example, Sutton, Precup, and Singh (1999) say: "options enable temporally abstract knowledge and action to be included in the reinforcement learning framework in a natural and general way". This temporally abstract knowledge need not be a function of the entire action history. I like this aspect of the method as it makes the proposed algorithm quite novel, but the motivation for why this should work didn't come through for me.
	(h) Should the mutual information in Eq. 4 be the conditional mutual information given $a_{0...k}$? (Same question for the remaining uses of $I$ and $H$).
	(i) It is unclear how the option learning coordinates with the RL algorithm used. That is, suppose we train the HRL component to learn the mapping from $s, a_{0...k}$ to the constituents identified in Figure 2/3. Where does the actual RL take place? Does the algorithm just execute the actions output by $D$ at each time step?
Q2: In the first experiment, it is stated: "because the smooth option space provides the RL algorithm with a better search space." Any thoughts as to why this is true? Including some discussion here might help motivate the approach.
Minor Comments:
	C1: I do not understand Figure 6. The color is said to denote "the distribution of options", but I couldn't quite make out what this was, precisely. It would be helpful to know the range of values it can take on, and how those values map to the displayed colors. Moreover, what is the take away from the figure? The text states "with information-theoretic constraints the options and state changes become more correlated" but I am having trouble connecting that claim with the visuals themselves. Some additional discussion here would be really helpful.

	C2: In Figure 5, what does "dimension disturbance in option space" mean?
Minor Typos/Writing Suggestions [did not affect evaluation]:
	Abstract:
	- "Applying reinforcement learning (RL) to"::"Applying reinforcement learning (RL) algorithms to"
	- I am having trouble parsing this phrase: "to learn new tasks on higher level more efficiently". Perhaps: "to learn new tasks at a higher level of abstraction more efficiently"
	- "over benchmark learning problems"::"over baseline learning algorithms on benchmark problems"
	Sec. 1 (Intro):
	- Plural acronyms tend to have an 's' at the end. So: Recurrent Variational AutoEncoders (RVAEs).
	- "conveys meaningful information and benefit the RL training"::"conveys meaningful information and can benefit learning"
	Sec. 2 (Related Work):
	- "the policy sketches"::"policy sketches"
	- Personal preference, by I always prefer "use" to "utilize".
	Sec. 3 (Approach):
	- Your $\mapsto$ operators should be replaced by $\rightarrow$. The $\mapsto$ operator indicates what is applied to elements on the left, while $\rightarrow$ specifies the domain and codomain of the function. Thus, the $\mapsto$ variation would be $P : (s,a) \mapsto s'$. The story is the same for $\beta$: it should read "$\beta : \mathcal{S} \rightarrow [0,1]$". Note that this (using $\rightarrow$) is how Sutton, Precup, and Singh (1999) define $\beta$ as well.
	- "Sub-policy is defined as a function over the random variable."::"Now, the sub-policy is defined as a function over the random variable."
	- Not a sentence: "So that the options with similar consequences become closer in the option space." Consider combining with the previous sentence.
	- This sentence runs on: "Given a set of past experiences...". Consider defining $\Lambda$ first as its own sentence, then definines the problem. Something like: "We let $\Lambda = ...$ Then, our problem is to learn...".
	- "it is empirically shown"::"it has been demonstrated empirically"
	- Latex quote issue: "”go reach the door".
	- In Equations 4-9: in general, mutual information is a function of random variables. Is $o$ a random variable? For instance I have trouble expanding $H(o)$. What is $p(o)$?
	- "the encode $E$ is regularized"::"the encoder $E$ is regularized"
	Sec. 4 (Experiments):
	- "task for proof of the concept"::"task as a proof of concept"
	- "that allows us easily visualize the option we learned from the experience"::"that allows us to easily visualize the options learned from experience"
	- "that the RVAE nicely capturing the direction":::"that the RVAE captures the direction"
	- Misuse of $\mapsto$: "learn a control policy $\pi : \mathcal{S} \mapsto \mathcal{A}$" should be "learn a control policy $\pi : \mathcal{S} \rightarrow \mathcal{A}$" or "learn a control policy $\pi : s \mapsto a$".
	- "HarfCheetah"::"HalfCheetah"
References:
Vezhnevets, Alexander Sasha, et al. "Feudal networks for hierarchical reinforcement learning." Proceedings of the 34th International Conference on Machine Learning-Volume 70. JMLR. org, 2017.
Nachum, Ofir, et al. "Data-efficient hierarchical reinforcement learning." Advances in Neural Information Processing Systems. 2018.
Tiwari, Saket, and Philip S. Thomas. "Natural option critic." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 33. 2019.
Harb, Jean, et al. "When waiting is not an option: Learning options with a deliberation cost." Thirty-Second AAAI Conference on Artificial Intelligence. 2018.
Harutyunyan, Anna, et al. "The Termination Critic." AISTATS 2019
Levy, Andrew, et al. "Learning multi-level hierarchies with hindsight." ICLR 2019.


Off-Policy Actor-Critic (Off-PAC) methods have proven successful in a variety of continuous control tasks. Normally, the critic's action-value function is updated using temporal-difference, and the critic in turn provides a loss for the actor that trains it to take actions with higher expected return. In this paper, we introduce a novel and flexible meta-critic that observes the learning process and meta-learns an additional loss for the actor that accelerates and improves actor-critic learning. Compared to the vanilla critic, the meta-critic network is explicitly trained to accelerate the learning process; and compared to existing meta-learning algorithms, meta-critic is rapidly learned online for a single task, rather than slowly over a family of tasks. Crucially, our meta-critic framework is designed for off-policy based learners, which currently provide state-of-the-art reinforcement learning sample efficiency. We demonstrate that online meta-critic learning leads to improvements in a variety of continuous control environments when combined with contemporary Off-PAC methods DDPG, TD3 and the state-of-the-art SAC.

This paper proposes a meta reinforcement learning approach where a meta-critic is trained in addition to a conventional actor and critic, so that the meta-critic boosts the training of the actor over a single task. The approach is combined to DDPG, TD3 and SAC and is claimed to convey superior performance (or learning speed) over these state-of-the-art actor-critic algorithms.
At first glance, the paper looks convincing, but closer inspection reveals a potential issue that I would like the authors to discuss.
The first cue is that in Fig. 6, the meta-loss does not seem to converge to anything, as the authors say it just fluctuates. Shouldn't it converge once the actor converges to close to optimal performance?
The second cue is that appart from the rllab tasks (ant and half-cheetah), it is not clear that the meta-critic approach brings some gain in the end of training. Particularly in Reacher (Fig 7), the performance seems to collapse faster with DDPG-MC than with DDPG, and on the rest of curves of Figs 7 and 8 it is had to determine whether the MC approach brings something significant or not. And in Fig. 3, in Walker2D, TD3-MC looks rather unstable.
The third cue is that in Section 3.2, the paragraphs "Updating MC parameters" and "Designing MC" are rather unclear (I'll come back to that) and lack a theoretical justification.
So I'm wondering what exactly MC is doing and I would like to see a more detailled analysis. Couldn't a similar performance improvement be obtained by just increasing the actor learning rate and addiing some noise to the actor learning gradient? Isn't this more or less what the meta-loss does, when looking at Fig. 6?
To me, unless the authors give a clear answer to the points above, the paper should be rejected as it does not provide clear enough evidence and justification in favor of the proposed meta-learning approach.
As stated above, I found the paragraph "Updating MC parameters" lacking a principled justification.
About "Designing MC", it could be much clearer. You first express two requirements (i) and (ii). Fine.
Then you explain how to meet requirement (i), but you say this is not what you are doing (!), and then you try to explain what you are doing but without going back to the requirements. In particular, it is not clear at all why your design is "permutation invariant". You should be much more direct. Besides, the way to extract features and the relationship to batch-wise set-embedding should be made more explicit. Well, it is complicated and should be explained more clearly...
"TD3 borrows the Double Q learning idea...": no, TD3 does more than this, as it uses the min between both critics, which double Q-learning and DDQN do not.
You use HalfCheetah-v2 from gym-mujoco and HalfCheetah from rllab. Why? One might suspect this is because the performance of the MC appraoch is only better with rllab...
You say you are using 10 million steps, but this is not always the case.
It is interesting to see that the performance of TD3 on Half-Cheetah collapses at around 4 million steps. Any idea why? It seems that in many papers experiments are stopped before performance collapses, and a strong study about this phenomenon is missing as the authors don't want to show this. This is an open call to readers: a paper on that would be great! :)
"SAC-MC" gives a clear-boost for several tasks": well, looking at the figures it is not so clear. Do you mean faster learning, higher final performance or both? I would like to see this claim backed-up by some proper statistical significance test and a clear specification of the number of seeds, etc. Fig.5 conveys the adequate data for such test if your claim is that this is the final performance that matters (but beware that curves are crossing eachother, so the final performance depends a lot on where you stop...).
typos:
p1 "For example, ... (Zheng et al., 2018)." is not a sentence (no main verb).
p2 "the on-policy (approach?) needs to interact with (the) environment..."
p2 "is less effective than off-policy transitions." => unclear statement
p3 to assist(s)
Equations (2), (6) and (8) should finish with a dot as they close a sentence.
Eq X => Eq (X) : use \eqref{label} instead of just \ref{label}
p5 It's input => Its
p5 two key technologies: I would not call this a technology. Ingredients?
p5 "computational cost is ? by using": missong word
p6: asmyptotic
p9: we removing


The selection of initial parameter values for gradient-based optimization of deep neural networks is one of the most impactful hyperparameter choices in deep learning systems, affecting both convergence times and model performance. Yet despite significant empirical and theoretical analysis, relatively little has been proved about the concrete effects of different initialization schemes. In this work, we analyze the effect of initialization in deep linear networks, and provide for the first time a rigorous proof that drawing the initial weights from the orthogonal group speeds up convergence relative to the standard Gaussian initialization with iid weights. We show that for deep networks, the width needed for efficient convergence to a global minimum with orthogonal initializations is independent of the depth, whereas the width needed for efficient convergence with Gaussian initializations scales linearly in the depth. Our results demonstrate how the benefits of a good initialization can persist throughout learning, suggesting an explanation for the recent empirical successes found by initializing very deep non-linear networks according to the principle of dynamical isometry.

This paper studies the convergence of deep linear networks under orthogonal initialization. Most of the problem setup, analysis techniques and proof roadmap are adapted from Du & Hu (2019). The difference is the orthogonal initialization compared with the Gaussian initialization in Du & Hu (2019). Since the product of orthogonal weight matrices is identity, the new initialization can remove the dependency of the number of nodes $m$ on the depth $L$ of the network. The authors also proved a lower bound of the loss function trained by Gaussian initialization within certain iterations. This justifies the disadvantage of Gaussian initialization.
My biggest concern is that this paper seems to be very similar to Du & Hu (2019) in many places. The whole Sections 3 & 4 are almost the same as Sections 3,4,5 & 7 in their paper. The contribution in this paper is too incremental given previous work.
What is the advantage of using orthogonal matrices in (4) compared with just using identity matrices as initialization? Can we prove the same result with just identity initialization? In this case, what would we lose by restricting us to this special case?
In the proof of Lemma 4.2, it seems that only the randomness of the last layer $W_L(0)$ is used. Why do we need all the layers to be uniformly sampled?
It should be explained in more details that $W_L(0)$ is drawn from a uniform distribution over orthogonal matrices in $d_y\times d_y$ space. Then $1/\sqrt{m} W_L(0)\cdot z /\|z\|^2$ is not distributed on the whole space of $d_y$-sphere. The argument in the proof of Lemma 4.2 thus needs more justification.
It would be interesting to see an empirical comparison of the proposed initialization and the Gaussian initialization. Due to the lower bound proved in this paper, the experiments are expected to show distinct difference between these two.


Understanding causes and effects in mechanical systems is an essential component of reasoning in the physical world. This work poses a new problem of counterfactual learning of object mechanics from visual input. We develop the COPHY benchmark to assess the capacity of the state-of-the-art models for causal physical reasoning in a synthetic 3D environment and propose a model for learning the physical dynamics in a counterfactual setting. Having observed a mechanical experiment that involves, for example, a falling tower of blocks, a set of bouncing balls or colliding objects, we learn to predict how its outcome is affected by an arbitrary intervention on its initial conditions, such as displacing one of the objects in the scene. The alternative future is predicted given the altered past and a latent representation of the confounders learned by the model in an end-to-end fashion with no supervision of confounders. We compare against feedforward video prediction baselines and show how observing alternative experiences allows the network to capture latent physical properties of the environment, which results in significantly more accurate predictions at the level of super human performance.

Summary of what the paper claims and contributes
---
This paper presents a new object forecasting task in the setting of "intuitive physics" that requires counterfactual reasoning and also presents method to perform this task. The task is to predict the alternative future trajectory of objects in 3D simulation given a video (A_frame,B_frames) of how the objects move in one situation and an alternative single frame (C_frame) that corresponds to modifying object position(s) at the first timestep of the input video. Due to unobserved confounding factors such as mass and friction that are not directly observable in either the video or the intervened frame, successful forecasting requires either implicitly or explicitly estimating these confounders. The motivation for this task is the hypothesis that counterfactual reasoning is a necessary component to forecast in general, unobserved situations. Prior work on explicit causal reasoning mainly concerns lower-dimensional problems, and prior work on high-dimensional "intuitive physics" has not evaluated the capability of models to perform counterfactual reasoning.
The paper proposes a neural network that learns to implicitly predict sufficient statistics of confounders of the situation. Training occurs in two stages. First, it is trained to estimate 3d positions of objects from images given pairs of (image, ground truth 3d positions). Then, it is trained to forecast future unobserved object positions with a graph convolutional network. Inference is performed by estimating the current object positions in all of the frames of the input (A_frame, B_frames), and then using the final representation of the input situation as the representation of confounders that is fed into the alternative-future prediction.
The experiments find:
1) Humans are not very good at these specific tasks, as evidenced by a simple position-copying baseline that outperforms humans. Given that the simple copying baseline is so performant, the claim in the abstract of "super human performance"  is also true for the copying baseline, and therefore that claim isn't very meaningful and should be removed.
2) The proposed approach outperforms position-copying and two non-counterfactual approaches that do not reason (explicitly or implicitly) about confounders,.
3) The approach attempts to show that confounder representations are learned as the main evidence that the model can perform counterfactual reasoning. However, several aspects of this experiment are unclear, making the claim difficult to evaluate.
Finally, although the abstract and the main text say that the task is to perform counterfactual forecasting in high-dimensional scenarios in a unsupervised way, the paper ends up using direct supervision of the object positions. With this knowledge, the problem is not that different than counterfactual forecasting in low-dimensional settings, as the positions of the objects at all frames corresponding to \mathbf{A}, \mathbf{B}, and \mathbf{C} could simply be extracted with this detector. Thus, through this supervision the approach seems to circumvent most of the challenges imposed by high-dimensional forecasting.
Evaluation
---
>Originality:
Are the tasks or methods new?
The task of counterfactual forecasting from high-dimensional observations is new. The proposed method extends prior work to the new task.
Is the work a novel combination of well-known techniques?
Yes.
Is it clear how this work differs from previous contributions?
Yes.
Is related work adequately cited?
To my knowledge, yes.
>Quality:
Is the submission technically sound?
Yes.
Are claims well supported by theoretical analysis or experimental results?
Some of the claims are supported, but the evidence for the counterfactual representation recovery claim is currently unclear. Additionally, the high-level motivation of counterfactual forecasting in high-dimensional settings is significantly undermined by the use of low-level supervision information that could feasibly be predicted at test time.
Is this a complete piece of work or work in progress?
It seems relatively complete.
Are the authors careful and honest about evaluating both the strengths and weaknesses of their work?
Weaknesses are not explicitly explored. It would be good to include discussion that illuminates or hypothesizes when and why the method fails, and how it could be made to perform better.
>Clarity:
Is the submission clearly written?
The submission is mostly clear, however there are some significant ambiguities in the experiments.
Is it well organized?
Yes.
Does it adequately inform the reader?
Mostly -- see ambiguities.
>Significance:
Are the results important?
It's currently unclear whether the results are significant, but it's clear the task is important (only when not supervised by object positions). However, justification with respect to other recent intuitive physics simulators is needed.
Are others (researchers or practitioners) likely to use the ideas or build on them?
Other researchers are likely to build on the task and data released.
Does the submission address a difficult task in a better way than previous work?
N/A (new task)
Does it advance the state of the art in a demonstrable way?
There is no SOTA on this new task.
Does it provide unique data, unique conclusions about existing data, or a unique theoretical or experimental approach?
Yes, one of the main strengths of the paper is the task and dataset that will be released. This experimental approach could be very useful for future research.
Additional feedback
---
The experiments to validate the cofounder estimation is unclear. Ambiguities:
0) In the text, the classification of mass and friction is presented as follows: "The obtained results are shown in Table 6 (middle)", yet the results are in Table 5. The impact of confounder supervision is described as follows "we do, however, explore what effect supervision could have on performance, as shown in Table 5", yet the results are in Table 6. These typos, when combined with the layout of Tables 5 and 6, makes it very difficult to interpret these results.
1) What is feedforward method? I can't find any description. Is this the IN or the NPE baseline, or something different? Why aren't both used? Without more details, its unclear that the proposed method actually learns confounder representations better than other methods.
2) What is the "random baseline level"? Is this just chance results from a uniform distribution as a baseline classifier?
3) What are the details of the classifier that predicts the mass and friction coefficients?
It is a bit odd to include comparisons to humans on a task on which the humans are outperformed by a very simple baseline. Either downplay the role of these comparisons, or remove these comparisons, or change the task such that humans cannot be outperformed by a simple baseline.
More details about the NPE and IN baselines are missing. Specifically, do they also leverage access to the ground-truth positions at training time? If so, how do they do this? This discussion is important for the paper to be self-contained.
Page 9 "The CophyNet model also consistently outperforms the augmented variants..." the outperformance isn't present in all cases (i.e. not in 4->2 or 4->3); change to "usually".
It's not clear if there is any stochasticity in the dynamics of the created data (which would be relevant to modeling and evaluation).
It wasn't clear until page 7 that the training leveraged GT positions in 3d space. The training objective (equation after Eq (5)) has an undefined o^*, which the reader must infer corresponds to GT, or read appendix A.1, to understand. The description of the method was introduced in terms of A, B, C, and D which are images or videos. To me, the fact that GT 3D positions are used seems like it was "buried" later in the paper, rather than described up-front alongside the introduction of A, B, C, and D.
The meaning of the colors, if any, in Fig 2 is unclear. If they are meant to illustrate observed information, perhaps grayscale shading could be used instead, as is standard in PGMs.
The latent representation in Fig 4 appear to emerge just from the red object's RNN, rather than all of the objects RNNs.
Tables 1 and 2 appear very far from their description in the text.
More discussion is needed that relates the tasks in to the tasks in Riochet2018 and Bakhtin2019. It's not clear why a new benchmark is needed without this discussion. Specifically, could the proposed method and evaluation be applied to these benchmark simulators? If not, why not? The answers to these questions seem to be the main motivation for the proposed tasks.
The simplest baseline would be to simply stack (A,B,C) as contextual input to a learned regressor of the future positions in D. Another simple baseline is to use the learned position estimator to predict all of the positions in (A,B,C) and use these positions as input to a learned regressor.
Use two backtick characters instead of the double-quote character for starting forward quotations.
Recommendation
---
Taken together, the novelty of the task and dataset, the discrepancy between problem motivation (high-dimensional counterfactual reasoning vs. uses low-dimensional ground-truth), and the partial clarity of the experimental results, leads me to conclude that although the proposed task and data are quite interesting and promising, the paper needs significant work to clarify its motivation and justification of its claims. I would rate the paper borderline, however it appears I'm forced to discretize to WA or WR. I choose WA because I am optimistic that the authors could address my doubts.


Knowledge graph embedding research has overlooked the problem of probability calibration. We show popular embedding models are indeed uncalibrated. That means probability estimates associated to predicted triples are unreliable. We present a novel method to calibrate a model when ground truth negatives are not available, which is the usual case in knowledge graphs. We propose to use Platt scaling and isotonic regression alongside our method. Experiments on three datasets with ground truth negatives show our contribution leads to well calibrated models when compared to the gold standard of using negatives. We get significantly better results than the uncalibrated models from all calibration methods. We show isotonic regression offers the best the performance overall, not without trade-offs. We also show that calibrated models reach state-of-the-art accuracy without the need to define relation-specific decision thresholds.

1. Summary
The paper studies probability calibration for three different knowledge graph embedding methods, with a focus on TransE evaluated on the task of knowledge graph triple classification. It studies Brier and log loss performance of Platt scaling and isotonic regression probability calibration on WN11 for TransE and claims that better calibration yields better performance as measured by mean reciprocal rank. Calibration plots for other datasets are also included as evidence. Furthermore, evidence is presented that probability calibration can lead to better performance for the task of triple classification. The main contributions of the paper also include the adaption of sampling techniques introduced by Bordes et al. (2013) adapted for estimating negatives for probability calibrations.
2. Decision (See the updated decision in the comment below)
Probability calibration is a very relevant issue, particularly in industry and when combining knowledge graph embedding models as external data in other models. Thus I see this work as a valuable contribution to the literature. In particular, I like the analysis from multiple views: Calibration plots, calibration metrics, and model performance. However, there is currently not enough evidence in the paper to make recommendations or judgments about when researchers and practitioners may want to use probability calibration. I also believe the datasets and models are not well tied into the literature, for example, in 2018/2019 I can find 3 papers for triple classification and 9 papers for link prediction as triple/entity ranking and from the data, it is not clear how probability calibration affects the latter. In the current state of the work, I recommend rejecting this work.
3. Further supporting arguments
As a researcher and practitioner in this area, I know very well that predictions of most knowledge graph embedding models usually live near the decision boundary so that there is little difference between probability or score of a true positive and false positive. Also talking with people in industry, I heard that word embedding models are currently not that useful practically because they make too many useless predictions. This shows me that probability calibration is an important topic and I see this study as an important contribution to the field which is often mindlessly following evaluation metrics.
However, from experience I also know that evaluation of knowledge graph embedding methods is not very reliable, that is people often get widely varying results and replication is difficult. Thus it is difficult to trust results if they are not well tied into the literature and compared against multiple datasets and models. This work focuses on three models TranseE, DistMult and ComplEx. DistMult and ComplEx have become models that are viewed as quite reliable to compare against. However, their performance is mostly studied on a learning-to-rank objective on datasets such as FB15k-237 and WN18RR. The authors report Brier score for their synthetic calibration method these datasets, but do not report any modeling results. Inclusions of results on these datasets would greatly improve this work.
The authors also currently focus on establishing that probability calibration improves the performance of the models. They claim that low Brier score or log loss are tied to good performance, but Pairwise and Multiclass-NLL loses achieve similar Brier/log loss performance while the MRR is double for Multiclass-NLL compared to the Pairwise loss. NLL and Multiclass NLL losses have similar MRR but very different Brier/log loss performance. As such I do not think this claim is sufficiently substantiated. I do not believe it is necessary to establish that better probability calibration is correlated with better model performance. I view the careful study of probability calibration and its effects per se as more useful.
As mentioned above, I also believe the results on WN11, FB13, and YAGO39k to not be sufficient to evaluate the effect of probability calibration.
4. Additional feedback
I really like this work. I think adding more results would make this paper great and I would be happy to change my acceptance decision.
As mentioned above I believe including results on FB15k-237 and WN18RR would make the results easier to interpret. Please also add more results to the table (no need to rerun those experiments, take them from other papers). I really like the analysis of Brier Score/Log loss and MRR. I think if you would extend this it would give very valuable insights into how probability calibration relates to performance.
One additional experiment which I do not deem critical, but which would improve your work further would be to tie probability calibration into a more practical setting. A setting that is also very interesting to researchers is if probability calibration would affect the results in tasks where you use word embedding models as an external "knowledge source". I really like  Kumar et al., 2019[1] since their word embedding model integrated into an entity linking model beats a strong BERT baseline. But I think a study of any task/model of your choice that integrates a knowledge embedding model would be a valuable addition to your work.
Again as I mentioned above, I do not believe it is critical to show improve performance on these tasks, a study of the effects of probability calibration is valuable in its own right. You might want to slightly pivot into this direction if you have sufficient evidence to make judgments about the effects of probability calibration.
Further small details: In the introduction, you make specific claims and justify them by citing a survey paper (Nickel et al., 2016). It would be easier for the reader to look up these claims in the source rather than in the survey paper. I believe there is a typo in your derivation in equation (6): the denominator of the second term should be just w-N + N or N(w- + 1).
[1] Zero-shot Word Sense Disambiguation using Sense Definition Embeddings: https://www.aclweb.org/anthology/P19-1568/


ABSTRACT A machine learning model that generalizes well should obtain low errors on unseen test examples. Thus, if we know how to optimally perturb training examples to account for test examples, we may achieve better generalization performance. However, obtaining such perturbation is not possible in standard machine learning frameworks as the distribution of the test data is unknown. To tackle this challenge, we propose a novel regularization method, meta-dropout, which learns to perturb the latent features of training examples for generalization in a metalearning framework. Specifically, we meta-learn a noise generator which outputs a multiplicative noise distribution for latent features, to obtain low errors on the test instances in an input-dependent manner. Then, the learned noise generator can perturb the training examples of unseen tasks at the meta-test time for improved generalization. We validate our method on few-shot classification datasets, whose results show that it significantly improves the generalization performance of the base model, and largely outperforms existing regularization methods such as information bottleneck, manifold mixup, and information dropout.

The paper proposes learning to add input-dependent noise to improve the generalization of MAML-style meta-learning algorithm. The proposed method is evaluated on OmniGlot and miniImageNet. The paper reports improvements upon MAML, MAML with meta-learned parameter-wise learning rates, as well as a few regularization methods that are based on input/hidden state perturbations (Mixup, Variational Information Bottleneck). An ablation study also compares the proposed meta-dropout algorithm with a number of modifications, such as a fixed noise, input-independent noise, etc. It is furthermore shown that meta-dropout somewhat improves the model’s robustness against an adversarial attack.
The paper is somewhat incremental considering that Li et al, (2017) and Balaji et al, (2018) have already proposed meta-learning parameter-wise learning rates and parameter-wise regularization coefficient respectively. One difference from the methods above is that in the proposed method noise is controlled by the input. The ablation however shows that in 5-shot classification case simply adding non-trainable noise works quite well.
It seems like the choice of the particular method for adding the noise was performed using the test set. If it’s true, this is methodologically wrong: model selection should be performed on a development set (or meta-development) set. Futhermore, Table 2 contains some results named “Add.”, which I guess stands for additive noise. I did not find an explanation of what is the specific method for adding noise used in this case. Such additive noise is also missing from ablation experiments.
Overall, it seems that paper falls short of clearly proving that back-propagating through MAML to the noise parameters is helpful. The “Deterministic Meta-Dropout” performs better than baseline MAML, and arguably, meaning that some part of the improvement upon MAML can be due to the architectural differences and not due to noise. “Independent Gaussian” and “Weight Gaussian” baselines perform worse than non-trainable noise (“Fixed Gaussian”). Learning the variance for the noise is shown to be detrimental. There is just too much confusion in the results, the improvements are not very robust.
The paper writing is okay, but there are serious issues. I am not sure I understand the argument in Section 3.2 that meta-dropout performs variational inference. It seems like Equation 7 is wrong because  y_i is missing from the second argument of the KL divergence term. The transition to Equation 8 is therefore also wrong, and as far as I can understand, the whole argument breaks down. Line 7 in Algorithm 1 in Appendix A (which by the way should really be in the main text) does not make sense.
Other issues:
- the second sentence of the abstract is not implied by the first, the usage of “thus” does not seem appropriate
- the intro should probably mention L1 and L2 regularization as well
- in Section 3.1 there is a forward reference to Equation 5, makes understanding the text quite hard
- “meta-droput”, “robustenss”: typos in many places
- Figure 4 visualization is not clear.
- the architectural change required to add noise is not explained in the paper (i.e. what is \phi and how it’s used)
- no comparison to meta-learned L1 regularization
- a baseline is missing in which \phi is treated as a part of \theta and trained with vanilla MAML