Recent research shows that the following two models are equivalent: (a) infinitely wide neural networks (NNs) trained under 2 loss by gradient descent with infinitesimally small learning rate (b) kernel regression with respect to so-called Neural Tangent Kernels (NTKs) (Jacot et al., 2018). An efficient algorithm to compute the NTK, as well as its convolutional counterparts, appears in Arora et al. (2019a), which allowed studying performance of infinitely wide nets on datasets like CIFAR-10. However, super-quadratic running time of kernel methods makes them best suited for small-data tasks. We report results suggesting neural tangent kernels perform strongly on low-data tasks.

This paper conducts very interesting and meaningful study of kernels induced by infinitely wide neural networks on small data tasks. They show that on a variety of tasks performance of these kernels are superior to both finite neural networks and Random Forest methods.
While neural tangent kernel (NTK) [1] is motivated for studying training dynamics of neural networks, it is also important to ask to find utility of these new powerful kernels that captures functional priors of neural networks. This paper conducted important study on small dataset regime and on a wide range of tasks (90 UCI datasets, small subset of CIFAR-10, few shot image classification task on VOC07.
Authors introduce a family of generalized NTK kernels interpolating between NNGP kernels [2] to original NTK[1] by fixing first L’ layers and allowing to train remaining layers. Treating L’ as a hyperparameter, the authors try both NNGP/NTK and kernels in between as well.
Another contribution I observe is applying kernel SVM where one utilizes NTK and shows that it can work well. This paper shows that kernels induced by infinitely wide networks could become useful for real world applications where data size is not so large.
There are few small concerns regarding experiments which are discussed in detailed comments. Overall I think the message of the paper is clear and well supported therefore I recommend accepting the paper.

Detailed comments

1) From reading the paper it was not easy to grasp where point 4 of the abstract was based on.
2) In the first footnote, small nit is that, in practice one should not invert matrix but just do a linear solve for better numerical stability and efficiency (still O(N^3) but with better constant)
3) In section 3, there seems to be no bias. Are NTK and NNs considered in this work contain no bias? Or is bias ignored for ease of presentation?
4) Nit p4 first paragraph in section 4 : multiplayer -> multilayer
5) Regards to NTK initialization performing better than standard He initialization: It was observed in [3] that for multilayer perceptron both parameterization is on-par but for CNN or WideResNet case standard parameterization performed significantly better.
6) Note that similar to analysis in section 5,  for CIFAR-10 with fully connected model [1] shows that for all dataset size(100-45k) NNGP performs better than trained neural networks.
7) One may worry that ResNet-34 is not properly tuned as most hyperparameters were fixed for large dataset.
8) Regards to hyperparameters for NTK, is there a consistent trend one could find regards to L’? What percentage of tasks that NTK performed well actually have a high L’?
9) To help the readers, I would suggest adding a little more description on statistics used for comparison as well as what VOC07 task entails.
[1] Jacot et al., Neural Tangent Kernel: Convergence and Generalization in Neural Networks, NeurIPS 2018
[2] Lee et al., Deep Neural Networks as Gaussian Processes, ICLR 2018
[3] Park et al., The Effect of Network Width on Stochastic Gradient Descent and Generalization: an Empirical Study, ICML 2019
EDIT AFTER AUTHOR RESPONSE:
I have read the response from authors. I appreciate all the efforts to improve the paper.


Pre-trained models have demonstrated their effectiveness in many downstream natural language processing (NLP) tasks. The availability of multilingual pretrained models enables zero-shot transfer of NLP tasks from high resource languages to low resource ones. However, recent research in improving pre-trained models focuses heavily on English. While it is possible to train the latest neural architectures for other languages from scratch, it is undesirable due to the required amount of compute. In this work, we tackle the problem of transferring an existing pre-trained model from English to other languages under a limited computational budget. With a single GPU, our approach can obtain a foreign BERT BASE model within a day and a foreign BERT LARGE within two days. Furthermore, evaluating our models on six languages, we demonstrate that our models are better than multilingual BERT on two zero-shot tasks: natural language inference and dependency parsing.

In this work, the authors propose a way to transfer a pre-trained English BERT model to a new language within a short amount of time. The key insight is to map English embeddings to the foreign language and have separate embeddings for both English and the foreign language. The resulting bilingual LM is evaluated for zero-shot transfer learning on two tasks: XNLI and dependency parsing.
Pros:
- The authors provide good details into their hyperparameter settings and about how the obtain the foreign language word embeddings.
- By leveraging existing pre-trained models, they’re able to do pre-training for their bilingual LM within 2 days.
Cons:
I find that a key comparison point in this paper is missing, which is Bilingual BERT trained on just the two languages that are being considered for their RAMEN system. This is not a fair comparison while mBERT which is trained on 100+ languages is not.
All comparisons are not fair since a simple baseline of just training mBERT on two languages with monolingual data and with a shared WPM is not evaluated here.
The proposed system has an unfair advantage over mBERT since it’s initialized from BERT/RoBERTA and fine-tuned only on two languages. Hence most of the parameters are used for just the two languages while mBERT uses the parameters for 104 languages.
Given this unfair comparison, I’m not sure if we can draw a meaningful conclusion from all the experiments.
Rating justification:
Given the lack a fair comparison between the bilingual and multilingual BERT models, I don't think the conclusions are insightful.


Distillation is a method to transfer knowledge from one model to another and often achieves higher accuracy with the same capacity. In this paper, we aim to provide a theoretical understanding on what mainly helps with the distillation. Our answer is "early stopping". Assuming that the teacher network is overparameterized, we argue that the teacher network is essentially harvesting dark knowledge from the data via early stopping. This can be justified by a new concept, Anisotropic Information Retrieval (AIR), which means that the neural network tends to fit the informative information first and the non-informative information (including noise) later. Motivated by the recent development on theoretically analyzing overparameterized neural networks, we can characterize AIR by the eigenspace of the Neural Tangent Kernel(NTK). AIR facilities a new understanding of distillation. With that, we further utilize distillation to refine noisy labels. We propose a self-distillation algorithm to sequentially distill knowledge from the network in the previous training epoch to avoid memorizing the wrong labels. We also demonstrate, both theoretically and empirically, that self-distillation can benefit from more than just early stopping. Theoretically, we prove convergence of the proposed algorithm to the ground truth labels for randomly initialized overparameterized neural networks in terms of 2 distance, while the previous result was on convergence in 0-1 loss. The theoretical result ensures the learned neural network enjoy a margin on the training data which leads to better generalization. Empirically, we achieve better testing accuracy and entirely avoid early stopping which makes the algorithm more user-friendly.

This paper proposes a new perspective on understanding knowledge distillation as a transfer of information defined with respect to the neural tangent kernel. Additionally, a new framework for learning the classifier on a noisy labeled dataset is proposed based on the knowledge transfer framework.
Overall, I think the paper lacks justification (and explanation) for its main statement on how knowledge distillation is related to the early stopping of the teacher network. Especially, it is confusing since Section 2 and 3 make different statements. Specifically, Section 2 shows that early stopping "helps" knowledge distillation while Section 3 shows that knowledge distillation can "replace" early stopping. The former observation implies that early stopping is complementary to knowledge distillation, while the latter implies otherwise.
Furthermore, Section 2 mainly explains why the eigenspaces associated with the largest eigenvalue of the neural tangent kernel is "informative information". However, there is no elaboration on how the knowledge distillation process leads to the transfer of such information, i.e., there is no connection between the neural tangent kernel and the knowledge distillation process. Although Figure 1. suggests that early stopping indeed improves the knowledge distillation process, they are not enough to support the statement convincingly enough.
Without proper support on the main statement of this paper, the paper looses much of its claimed contributions. The label refinery algorithm for the noisy labeled dataset is interesting, but it is not evaluated thoroughly enough to demonstrate its superiority over existing algorithms. It also does not have much originality when compared to similar algorithms [1, 2]. Bagherinezhad  et al., [1] also tried to remove "noisy supervisions" that were generated by harsh augmentation on images. Han et al., [2] and Li et al., [3] also use distillation-like processes to learning noisy datasets.
[1] Label Refinery: Improving ImageNet Classification through Label Progression, Bagherinezhad  et al., 2018
[2] Co-teaching: Robust Training of Deep NeuralNetworks with Extremely Noisy Labels, Han et al., 2018
[3] Learning from Noisy Labels with Distillation, Li et al., 2017


Generative adversarial networks (GANs) learn to map samples from a noise distribution to a chosen data distribution. Recent work has demonstrated that GANs are consequently sensitive to, and limited by, the shape of the noise distribution. For example, a single generator struggles to map continuous noise (e.g. a uniform distribution) to discontinuous output (e.g. separate Gaussians) or complex output (e.g. intersecting parabolas). We address this problem by learning to generate from multiple models such that the generator's output is actually the combination of several distinct networks. We contribute a novel formulation of multi-generator models where we learn a prior over the generators conditioned on the noise, parameterized by a neural network. Thus, this network not only learns the optimal rate to sample from each generator but also optimally shapes the noise received by each generator. The resulting Noise Prior GAN (NPGAN) achieves expressivity and flexibility that surpasses both single generator models and previous multigenerator models.

Overall:
IIUC, the main contribution of this paper is to take previous work on training GANs with
multiple generators and add a learned assignment of z -> G_i, so that we have
G(z) = \sum_i NP(G_i | z) * G_i(z)
where NP is this learned assignment.
I think that technically this idea is new,
but there is important related work [1] that (IMO)
  - does essentially the same thing
  - better experimentally validates the thing they do
  - was published in last ICLR.
I also have a number of issues with the experimental design: see my detailed comments below.
Finally, the technique seems like it can't easily scale up to larger GANs or data sets
because it requires instantiating many copies of the generator in memory?
For these reasons, I lean somewhat strongly toward rejection.
Detailed comments on draft:
It's worth noting that conditional GANs also sample from multiple disconnected `manifolds'.
I guess the value of adding your technique is that it can work without labels?
> Our approach differs significantly from those previously
strange sentence.
>  By dividing up the space into four slices...
I don't get this part.
If you fed all the slices to a single generator you could
generate four `disconnected manifolds' with just 1 generator, no?
> , multiple-discriminator mixture in (Arora et al., 2017)
Nit: you don't need the parens there, IMO.
> our model learns to model the data over the full collection of generators in each minibatch
Given that modern GAN techniques (e.g. bigGAN), this is going to have some pretty unpleasant
performance characteristics, right?
Each generator has to be instantiated in memory all the time.
> LGAN
Nit: surely, given the state of the GAN literature, this name has been used before.
> We compare our NPGAN’s ability to model this distribution to
The following baseline seems like it would be much simpler and get the job done:
Train a GAN on a prior that's a mixture of like 100 gaussians or whatever.
Then it seems like it could learn to assign 70, 50, and 30 of those modes
to each one of the gaussians in your underlying data?
I guess I haven't tried this myself, but it seems like a more fair comparison.
The way the current experiment is set up, you know a priori that your model
is the only one that can work.
Overall, I don't really understand why it's necessary to have multiple generators.
If I want the prior my generator `sees' to be disconnected, can't I just
pass samples from the prior through a standard relu network?
Surely a relu network can learn to separate one mode into two, and so forth.
But once you do that, you've essentially got [1].
> We next demonstrate
Would be nice to have a new subesection here.
I think your single-generator baseline in Fig 2 and 3 is unrealistically bad.
See fig 3 of [2], in which it looks bad but not nearly as bad as you've shown it.
I think this points to another issue with the experimental design.
If the generator is a big enough relu network, it ought to be able to automatically
generate `disconnected bits', but it has to be big enough, and you've compared a
generator with N parameters to 2 and 3 generators with N parameters each.
Moreover, your learned noise prior has some extra number of parameters (I don't see
where you described these but I may be missing it).
> A single generator is unable to model this non-disconnected data.
Again, I have a feeling that this is because of the small generator you use.
I'm pretty sure that a motivated person could get a normal GAN to model the
distribution in fig 5 reasonably well.
> The other models can only work if the precise optimal number of generators is known a
priori.
Isn't this not true for the method that throws out redundant generators?
The experiments described in 4.3 and 4.4 are pretty contrived.
It's ok to have some contrived experiments, but it seems like all experiments
on which you were able to provide evidence that your technique was helpful are contrived.
It also seems like the baselines you used are not obviously the right choice for these
experiments?
Re: your CIFAR experiments:
I think this supports my earlier claim that the extra parameters and bells and whistles
in modern GAN techniques *already implicitly do what your method is proposing to do*.
I think a lot of people miss this point when writing papers about GANs.
When people request comparisons against more modern techniques,
they're not (merely) being difficult:
they want to know if the technique you propose "stacks" with other techniques,
in the sense that your method is doing something that wasn't already being done implicitly
by the old methods.
References:
[1] On Self Modulation for Generative Adversarial Networks (https://arxiv.org/abs/1810.01365)
[2] Discriminator Rejection Sampling (https://arxiv.org/pdf/1810.06758.pdf)


Deep neural networks suffer from the inability to preserve the learned data representation (i.e., catastrophic forgetting) in domains where the input data distribution is non-stationary, and it changes during training. Various selective synaptic plasticity approaches have been recently proposed to preserve network parameters, which are crucial for previously learned tasks while learning new tasks. We explore such selective synaptic plasticity approaches through a unifying lens of memory replay and show the close relationship between methods like Elastic Weight Consolidation (EWC) and Memory-Aware-Synapses (MAS). We then propose a fundamentally different class of preservation methods that aim at preserving the distribution of the network's output at an arbitrary layer for previous tasks while learning a new one. We propose the sliced Cramér distance as a suitable choice for such preservation and evaluate our Sliced Cramér Preservation (SCP) algorithm through extensive empirical investigations on various network architectures in both supervised and unsupervised learning settings. We show that SCP consistently utilizes the learning capacity of the network better than online-EWC and MAS methods on various incremental learning tasks.

[Summary]
This paper proposes a new method for overcoming catastrophic forgetting in continual learning, based on distribution-based regularization using the sliced Cramer distance, i.e. Sliced Cramer Preservation (SCP). Unlike previous work on catastrophic forgetting, this paper tackles unsupervised learning scenarios as well as supervised learning. They evaluate the proposed SCP on permutated MNIST, sequential learning in autoencoder task, and sequential learning for segmentation.
[Pros]
- This paper tackles unsupervised learning scenarios beyond the classification on benchmark datasets.
- This paper employes sliced Cramer distance with theoretical justification.
- The analysis on EwC and MAS in terms of geometric view
- Experimental results look promising.
[Cons]
- Even if MAS[1] describes the details on synaptic concept and Hebbian rule, many readers might be not familiar with synaptic or neuro-science terms. So, in prelimiary session, more explanation can help readers to understand.
- In addition to permute-MNIST, it is required to be evaluated on more conventional tasks such as MNIST->SVHN or CIFAR-10, 100 datasets. Also, more recent work should be compared such as IMM [2] and PGMA [3] for supervised learning scenarios.
- All graph result figures can be improved for enhancing legibility. In Figure 6, In particular, the subtitle of Summer case confuses me.

[1] Aljundi et al. Memory Aware Synapses: Learning what (not) to forget, ECCV 2018.
[2] Lee et al. Overcoming Catastrophic Forgetting by Incremental Moment Matching, NIPS 2017.
[3] Hu et al. Overcoming Catastrophic Forgetting for Continual Learning via Model Adaptation, ICLR 2019.


Graph neural networks have been adopted in numerous applications ranging from learning relational representations to modeling data on irregular domains such as point clouds, social graphs, and molecular structures. Though diverse in nature, graph neural network architectures remain limited by the graph convolution operator whose input and output graphs must have the same structure. Due to this restriction, representational hierarchy can only be built by graph convolution operations followed by non-parameterized pooling or expansion layers. This is very much like early convolutional network architectures, which later have been replaced by more effective parameterized strided and transpose convolution operations in combination with skip connections. In order to bring a similar change to graph convolutional networks, here we introduce the bipartite graph convolution operation, a parameterized transformation between different input and output graphs. Our framework is general enough to subsume conventional graph convolution and pooling as its special cases and supports multi-graph aggregation leading to a class of flexible and adaptable network architectures, termed BiGraphNet. By replacing the sequence of graph convolution and pooling in hierarchical architectures with a single parametric bipartite graph convolution, we (i) answer the question of whether graph pooling is necessary, and (ii) accelerate computations and lower memory requirements in hierarchical networks by eliminating pooling layers. Further, with concrete examples, we demonstrate that the general BiGraphNet formalism (iii) provides the modeling flexibility to build efficient architectures such as graph skip connections, and graph autoencoders.

This paper proposes a new graph neural network model named BiGraphNet, which introduces a parameterized bipartite graph convolution operation to perform transformation between input and output graphs. The proposed method is claimed to have advantages over existing deep hierarchical GCN architectures mainly in terms of being able to construct analogous building blocks employed by modern lattice CNN architectures and the reduced computational and memory cost. The main weaknesses of this paper are listed as follows:
1) The motivation is relatively weak, which is to bring in the analogous building blocks in CNN architectures. Although GNN is closely related to CNN and RNN, the graph learning tasks may not have the same property as in computer vision or natural language processing. It would be better to convince the readers from the GNN itself and carefully argue the necessity of the proposed method.
2) The experiments in this paper are rather weak and not convincing. First there is no performance comparison to state-of-the-art GNN models, such as DGCNN, DIFFPOOL and GIN, etc. At least on the D&D dataset, many existing models report graph classification accuracy over 78.0, but the baseline method used in this paper only achieves 72.5. Thus it is not fair to claim the proposed method can retain or improve the performance of existing GNN models.
3) The related work comparison is not sufficient. For example, some existing works have already explored to apply skip connections to the graph neural networks, such as [1], which is not mentioned and compared in this paper.
Based on the above arguments, I would like to recommend a reject for this paper.
[1] Xu, Keyulu, et al. "Representation learning on graphs with jumping knowledge networks." arXiv preprint arXiv:1806.03536 (2018).


Recently, Generative Adversarial Network (GAN) and numbers of its variants have been widely used to solve the image-to-image translation problem and achieved extraordinary results in both a supervised and unsupervised manner. However, most GAN-based methods suffer from the imbalance problem between the generator and discriminator in practice. Namely, the relative model capacities of the generator and discriminator do not match, leading to mode collapse and/or diminished gradients. To tackle this problem, we propose a GuideGAN based on attention mechanism. More specifically, we arm the discriminator with an attention mechanism so not only it estimates the probability that its input is real, but also does it create an attention map that highlights the critical features for such prediction. This attention map then assists the generator to produce more plausible and realistic images. We extensively evaluate the proposed GuideGAN framework on a number of image transfer tasks. Both qualitative results and quantitative comparison demonstrate the superiority of our proposed approach.

This paper proposes an extension of the conditional GAN objective, where the generator conditions on an attention map produced by the discriminator in addition to the input image. The motivation is that the discriminator is usually too powerful, and so the gradient the generator receives is often too small in magnitude. By conditioning on the attention map, the generator could leverage information about the regions in the image that the discriminator attends to and use it to generate a new image that better fools the discriminator.
My main concern is about whether the proposed extension achieves the desired goal. The intuitive motivation provided in the paper aims to add a cooperative component to the two-player game, but the min-max objective corresponds to a zero-sum adversarial game. As a result, when training the discriminator, the discriminator is encouraged to reveal as little information as possible via the attention map, so that the loss maximized. This appears to be the opposite of the desired behavior, so the objective needs to be reformulated.
Also, it is unclear how inference is performed: at test time, the attention map is unknown and so some placeholder must be used in its place. The paper should clarify what is done at test time, and clearly state the shortcomings as a result of this, i.e. different procedures are used for training and testing, which is not principled. I imagine the generator could rely too much on the attention map as a result - how this is alleviated/prevented should be explained.
Figure 2: Only the qualitative results for unsupervised image-to-image translation are available; qualitative results for supervised image-to-image translation should also be provided.
While the quantitative improvement over existing methods is somewhat insignificant, I appreciate the authors discussing their hypotheses why this might be the case. It would be more useful to empirically validate these hypotheses as well. For example, for the claim that "maybe the attention map only focuses on a few domain specific classes so the generator works too hard on those classes and ignores others", it might be good to compute the average per-class attention map intensity to show that some classes appear rarely in the attention map.
The evaluation protocol should be explained in greater detail (perhaps in the appendix); the segmentation model (which I assume is FCN) should be described and each of the evaluation metrics (per-pixel acc., per-class acc. and IoU) should be described for the benefit of researchers outside the area.
pg. 4: "in their implementation contains several Resblock (He et al., 2016), which makes it infeasible in our framework". Why is it infeasible?
pg. 6: What are the architectures used by the baselines? Are they comparable to the architecture the proposed method used?
Minor Issues:
pg. 3: "differences between P_x and G_Y \cdot P_y, P_y and G_X \cdot P_x are minimized" - confusing; should rephrase as "the difference between P_x and G_Y \cdot P_y and the difference between P_y and G_X \cdot P_x are minimized". Also should replace \cdot with \circ.
pg. 4: "like random noisy" -> "like random noise"
pg. 4, last paragraph: "Our trainable attention module follows the same structure of the attention block in RAM (Wang et al., 2017). They built a very deep network with several such blocks, each containing two branches: mask branch and trunk branch. Mask branch cascades the input features through a bottom-up top- down architecture that mimics human attention. Trunk branch is applied as feature processing." - this is very confusing; it would be easier to refer readers to the appendix.
pg. 5 - "Attention mask can potentially break good property of the raw input." - what does this mean?
pg. 6 - "as showed in Table 4.1" -> "as shown in Table 4.1"


Multi-stage training and knowledge transfer from a large-scale pretraining task to various finetuning tasks have revolutionized natural language processing (NLP) and computer vision (CV), with state-of-the-art performances constantly being improved. In this paper, we develop a multi-stage influence function score to track predictions from a finetuned model all the way back to the pretraining data. With this score, we can identify the pretraining examples in the pretraining task that contribute most to a prediction in the finetuning task. The proposed multistage influence function generalizes the original influence function for a single model in , thereby enabling influence computation through both pretrained and finetuned models. We study two different scenarios with the pretrained embeddings fixed or updated in the finetuning tasks. We test our proposed method in various experiments to show its effectiveness and potential applications.

The authors derive the influence function of models that are first pre-trained and then fine-tuned. This extends influence functions beyond the standard supervised setting that they have been primarily considered in. To do so, the authors make two methodological contributions: 1) working through the calculus for the pre-training setting and deriving a corresponding efficient algorithm, and 2) adding $L_2$ regularization to approximate the effect of fine-tuning for a limited number of gradient steps.
I believe that these are useful technical contributions that will help to broaden the applicability of influence functions beyond the standard supervised setting. For that reason, I recommend a weak accept. I have some questions and reservations about the current paper:
1) Does pretraining actually help in the MNIST/CIFAR settings considered? These seem to be non-standard pretraining settings. More generally, can we relate influence to some objective measure that we care about (say test accuracy), for example by showing that removing the top X% of influential pretraining data hurts test accuracy as much as predicted? Minor: section 4.2 also seems non-standard. Are the exact same bird vs. frog examples being used for both pretraining and finetuning?
2) In what situations might we want to examine the influence of pretraining data, and can we design experiments that show those situations? For example, perhaps we're wondering if different types of sentences in the one-billion-word dataset might be more or less useful. Can we verify those claims using these multi-stage influence functions? It is otherwise difficult to assess the utility of the qualitative results (e.g., Figure 3 and Appendix C).
3) It'd be helpful to get a better understanding of the technical contributions of this paper. Specifically,
a. What is the impact of $\alpha$ in equation 12 and how does it interact with the number of fine-tuning steps taken?
b. If the Hessian has negative eigenvalues, we can still take $H^{-1}b$ by solving CG with $H^2$, but what does this correspond to? Is the influence equation well defined (or the Taylor approximation justified) if $H$ is not positive definite?


Current work on neural code synthesis consists of increasingly sophisticated architectures being trained on highly simplified domain-specific languages, using uniform sampling across program space of those languages for training. By comparison, program space for a C-like language is vast, and extremely sparsely populated in terms of 'useful' functionalities; this requires a far more intelligent approach to corpus generation for effective training. We use a genetic programming approach using an iteratively retrained discriminator to produce a population suitable as labelled training data for a neural code synthesis architecture. We demonstrate that use of a discriminator-based training corpus generator, trained using only unlabelled problem specifications in classic Programming-by-Example format, greatly improves network performance compared to current uniform sampling techniques.

# Summary
This paper describes an approach for generating synthetic training corpora for neural program synthesis systems. It
assumes a test set of tasks to be solved, and designates a special "discriminator" network to force a distribution of
I/O examples derived from randomly sampled programs to be as close as possible to the I/O examples supplied for the
desired programs. The system first randomly samples a set of programs, then retains only those that are "sufficiently
close" to the desired I/O examples, and mutates them to further refine this training corpus. The whole approach is
evaluated on a DSL of array-manipulating programs, assuming 40 reasonable test programs and randomly sampled I/O pairs.
# Strengths
I like the core idea of the paper: assuming some desired distribution of I/O examples, construct the training corpus of
random programs in such a way that its associated I/O examples approximate the desired distribution. I can see how it
could force the sampling process to discover useful program snippets for training. If evaluated properly, this work
might get accepted at a future conference.
# Weaknesses
I have some issues with the (a) evaluation, and (b) presentation of the work in its current state.
First, the evaluation process fixes a set of 40 "useful" programs along with their associated I/O pairs, uses these I/O
pairs to build the training corpus as described, and then tests the finally trained synthesizer on the **same** set.
While the test programs themselves are not are not used in corpus building, only their I/O pairs are, this is still
sufficient to significantly bias the training set toward the test set. To properly test the core idea above, I'd expect
_at least_ something like:
- Write down or collect a set of "useful" test programs. Ideally make sure they all use different compositions of the
  language operators but still cover the space of combinatorial operator combinations well.
- Split this set into "dev" and "test" randomly. Use the I/O pairs from the "dev" set to build the training corpus.
  Evaluate the final synthesizer on the test set.
- Ideally also evaluate the sensitivity of this process to the chosen dev/test split.
Second, the evaluation baselines for corpus generation are straightforward: random sampling and genetic programming. The
former is self-evidently strawman. The latter is interesting, but is effectively a different implementation of the same
core idea of the paper, with two changes: (a) hardcoded fitness function instead of a trained discriminator, (b)
tournament selection for population evolution instead of roulette.
In any case, neither of the baselines are truly independent strong alternatives. The authors remarked that they were
"unable re-use baselines from ... literature, as for example ... DeepCoder framework caused by the relative generality
of our programming language for synthesis", which is a confusing statement for two reasons:
1. The programming language considered is a simple DSL of array manipulations, not that different in spirit from the
   DeepCoder DSL, AlgoLisp [Polosukhin & Skidanov, 2018], or the list manipulations DSL of Ellis et al. [2018].
2. Regardless of the DSL, what is being compared are **corpus generation techniques, not synthesis algorithms**.
   DeepCoder is not a baseline for that, its corpus generation is not a contribution.
The most proper baseline would be the work of Shin et al. [2019], which the authors mention in Section 2. It was
evaluated on Karel, which leaves the authors the choice of either (a) additionally evaluate their technique on the
Karel DSL, or (b) adapt the method of Shin et al. by designing some salient variables for the DSL of this paper.
I would prefer (a), which would significantly strengthen the paper, but (b) is also acceptable assuming honest effort
in designing the salient variables to make the baseline strong.
Finally, the presentation of the technical sections is informal and often confusing. Section 3 required several passes
to understand the corpus generation process. The authors should introduce proper formalism for all the involved concepts
such as corpora, train/test programs, their associated I/O pair, both neural networks with their input-output signatures
and architecture, and so on. Most of these concepts have established mathematical notation in the literature which would
make Section 3 much easier to follow. In addition, Sections 3.3-3.5 need to be formalized in algorithmic pseudocode.
# Minor remarks
* Please use \citet and \citep appropriately: citations should be enclosed in parentheticals if they don't participate
  in a sentence.
* In your synthesizer network architecture, do you know the number of program output lines ahead of time?
* In this DSL, how many possibilities are there in total for an output line (including all its arguments)?
  Given the beam size of 1M (which is a lot!), I want to compare it with the total space (#possibilities × #lines).
# References
Ellis, K., Morales, L., Sablé-Meyer, M., Solar-Lezama, A., & Tenenbaum, J. (2018). Learning libraries of subroutines for
neurally–guided bayesian program induction. In Advances in Neural Information Processing Systems (pp. 7805-7815).
Polosukhin, I., & Skidanov, A. (2018). Neural program search: Solving programming tasks from description and examples.
arXiv preprint arXiv:1802.04335.
Shin, R., Kant, N., Gupta, K., Bender, C., Trabucco, B., Singh, R., & Song, D. (2019). Synthetic Datasets for Neural
Program Synthesis. In ICLR.


Generative adversarial networks (GANs) nowadays are capable of producing images of incredible realism. One concern raised is whether the state-of-the-art GAN's learned distribution still suffers from mode collapse. Existing evaluation metrics for image synthesis focus on low-level perceptual quality. Diversity tests of samples from GANs are usually conducted qualitatively on a small scale. In this work, we devise a set of statistical tools, that are broadly applicable to quantitatively measuring the mode collapse of GANs. Strikingly, we consistently observe strong mode collapse on several state-of-the-art GANs using our toolset. We analyze possible causes, and for the first time present two simple yet effective "black-box" methods to calibrate the GAN learned distribution, without accessing either model parameters or the original training data.

This work addresses the important problem of generation bias and a lack of diversity in generative models, which is often called model collapse. It proposed a new metric to measure the diversity of the generative model's "worst" outputs based on the sample clustering patterns. Furthermore, it proposed two blackbox approaches to increasing the model diversity through resampling the latent z. Unlike most existing works that address the model collapse problem, a blackbox approach does not make assumptions about having access to model weights or the artifacts produced during model training, making it more widely applicable than the white-box approaches.
In terms of experiment setup, the authors chooses face generation as the area to investigate and measures the diversity by detecting the generated face identity. With the proposed methods, the authors showed that most STOA methods have a wide gap between the top p faces of the most popular face identities and randomly sampled faces. It further showed that the proposed blackbox approaches increases the proposed diversity metric without sacrificing image quality.
The proposed diversity measuring metric is lacking both in terms of experimental proofs and intuitive motivations. While the black-box calibration of a GAN model may be attractive under specific settings, the authors did not consider the restrictions under those situations and their design may be hard to implement as a result. For those reasons, I propose to REJECT this paper.
Missing key experiments that will provide more motivation that 1. the new metric reflects human perception of diversity 2. the new metric works better than existing ones:
1. Please provide experiments and/or citation for using the face identity as a proxy for face image diversity. this is important since all your experiments rely on that assumption.
2. Were there experiments that applies your metric to the training datasets like CelebA and FFHQ? In theory your metric should show no gap between N_R_obs and N_R_ref measured on the training dataset since that's the sampled ground truth.
Missing assumptions about blackbox calibration approaches:
1. If we do not have access to the model parameter, the training data, or the artifacts during training like the discriminator, what are some of the real world situations that fit this description? In those cases, is it too much to assume that we can control the random seed input to G?
2. Is it reasonable to assume some constraints on how much data we can get from the blackbox generator? A website that just exposes the image generation API may not allow you to ping their service 100k times to improve the generation diversity. If you are allowed to do that, it may be reasonable to assume that you can contact the API provider to get access to the rest of the model.
Minor improvements that did not have a huge impact on the score
1. I found the argument about FID in section 2.1 unconvincing. Are there proofs or citations for the claim that real images don't follow multivariate gaussian distribution after applying FID? Copying is indeed an issue that FID cannot detect, but it may be tangential to model collapse for real world concerns like privacy.
2. The statement "IS, FID and MODE score takes both visual fidelity and diversity into account." under "Evaluation of Mode Collapse" is contradictory to the description in sec 2.1 that IS in fact does not measure diversity.
3. You may want to consider stating the work as "a pilot study" (sec 6.) earlier in the abstract or in the introduction, so that the reader knows what to expect.


This work provides an additional step in the theoretical understanding of neural networks. We consider neural networks with one hidden layer and show that when learning symmetric functions, one can choose initial conditions so that standard SGD training efficiently produces generalization guarantees. We empirically verify this and show that this does not hold when the initial conditions are chosen at random. The proof of convergence investigates the interaction between the two layers of the network. Our results highlight the importance of using symmetry in the design of neural networks.

PAPER SUMMARY: This paper studies the problem of training a single hidden layer neural network to represent an arbitrary symmetric function. These are functions $f : \{0,1\}^n \to \{-1, 1\}$ which are invariant to permutations in the input coordinates. The authors' main result (Theorem 1) shows that if you take a single hidden layer network with $O(n)$ hidden units and initialize the weights in a particular way, then for any symmetric $f$, SGD training will converge to an empirical risk minimizer with guaranteed small generalization error. On the other hand, the authors' experiments suggest that arbitrary symmetric functions are not learnable from random initialization. Taken together, these results point to the importance of designing network architectures/ initializations that respect the structure in the function class you're trying to represent.
REVIEW SUMMARY: I lean towards rejecting this paper however, because I am not convinced of the results' significance. We already know how to learn symmetric functions (see Exercise 3.26 in Mohri et al., 2018). The authors' results show that we can inject this knowledge into a neural network at initialization, and then run SGD without making things too much worse. I do not see how these ideas might apply to more substantial learning problems where our prior knowledge is less precise. Moreover, while the proofs are clearly presented overall, I have one concern with a key step in Lemma 4.
MAJOR COMMENTS:
1) The key property of symmetric functions is that their output depends only on $|x|$. Thus, one can first extract "cardinality features" $x \mapsto |x|$, after which learnability follows by standard generalization theory results (as the authors note in the proof of Theorem 1).
The basic idea of Theorem 1 then seems to be to realize this feature map as the hidden layer of a single hidden layer ReLU network (this is essentially what the initialization does) and then show that running SGD will not move the weights too far from the initialization (Lemma 4).
(a) First, I think it would be helpful to the reader if the authors could make this intuition more explicit. In the submission the authors do not give much explanation for the choice of initialization.
(b) Second, because this is a learning problem we already know how to solve, the results seems a little contrived. I do not see how these ideas could extend to more challenging cases where our prior knowledge of symmetry (e.g. translation invariance) does not by itself lead to an algorithm with efficient learnability guarantees.
2) I could not follow one step in the proof of Lemma 4 (used to show that SGD does not move the weights too far from the initialization). Why does Theorem 2 imply that the number of updates is at most $20 R^2 / \gamma^2$? In Theorem 2, $R$ is fixed whereas in Lemma 4 it varies with $t$. To me this seems important, since without a bound on the number of steps it is unclear how you can control how far the embeddings move.
MINOR COMMENTS
3) In the statement of Lemma 4, linear separability of $V$ should be with respect to some fixed partition $Y$?.
4) In Figure 5, why is empirical error not decreasing over epochs?
5) I think the figures referenced in the text should be in the paper, not the appendix.
Mohri, M., Rostamizadeh, A., & Talwalkar, A. (2018). Foundations of machine learning. MIT press.


The learning of hierarchical representations for image classification has experienced an impressive series of successes due in part to the availability of largescale labeled data for training. On the other hand, the trained classifiers have traditionally been evaluated on a handful of test images, which are deemed to be extremely sparsely distributed in the space of all natural images. It is thus questionable whether recent performance improvements on the excessively re-used test sets generalize to real-world natural images with much richer content variations. In addition, studies on adversarial learning show that it is effortless to construct adversarial examples that fool nearly all image classifiers, adding more complications to relative performance comparison of existing models. This work presents an efficient framework for comparing image classifiers, which we name the MAximum Discrepancy (MAD) competition. Rather than comparing image classifiers on fixed test sets, we adaptively sample a test set from an arbitrarily large corpus of unlabeled images so as to maximize the discrepancies between the classifiers, measured by the distance over WordNet hierarchy. Human labeling on the resulting small and model-dependent image sets reveals the relative performance of the competing classifiers and provides useful insights on potential ways to improve them. We report the MAD competition results of eleven ImageNet classifiers while noting that the framework is readily extensible and cost-effective to add future classifiers into the competition.

The paper proposed a novel image classifier comparison approach that went beyond one fixed testing set for all. Instead, for a pair of classifiers to be compared, it advocated to sample their "most disagreed" test set from a large corpus of unlabeled images. The level of disagreement was measured by a semantic-aware distance derived from WordNet ontology. Because of the efficacy of such "worst-case" comparison, the needed set size is very small and thus minimizes the human annotation workload.
The proposed MAD competition distinguishes classifiers by finding their respective counterexamples. It is therefore an "error spotting" mechanism, rather than a drop-in replacement of standard test accuracy. I feel the approach to implicitly assume that the classifiers to be compared are already "reasonably accurate"; since if not, both classifiers might be easily falsified by certain trivial examples, making the "disagreed examples" not as meaningful. If that is true, I would suggest the authors to make this hidden assumption clearer in the paper
The idea shows clear liaison to the "differential testing" concept in software engineering besides the cited work of perceptual quality assessment. The idea has a cross-disciplinary nature and is fairly interesting to me. I can see the paper to be of interest to a quite broad audience and can motivate many subsequent works.
One minor comment: for images in "Case III", the authors considered them "contribute little to performance comparison between the two classifiers" and therefore did not source labels for them. However, since the authors adopted an affinity-aware distance, two incorrect predictions can still be compared based on their semantic tree distances to the true class.


Deep representation learning has become one of the most widely adopted approaches for visual search, recommendation, and identification. Retrieval of such representations from a large database is however computationally challenging. Approximate methods based on learning compact representations, have been widely explored for this problem, such as locality sensitive hashing, product quantization, and PCA. In this work, in contrast to learning compact representations, we propose to learn high dimensional and sparse representations that have similar representational capacity as dense embeddings while being more efficient due to sparse matrix multiplication operations which can be much faster than dense multiplication. Following the key insight that the number of operations decreases quadratically with the sparsity of embeddings provided the non-zero entries are distributed uniformly across dimensions, we propose a novel approach to learn such distributed sparse embeddings via the use of a carefully constructed regularization function that directly minimizes a continuous relaxation of the number of floating-point operations (FLOPs) incurred during retrieval. Our experiments show that our approach is competitive to the other baselines and yields a similar or better speed-vs-accuracy tradeoff on practical datasets.

Summary

This paper focuses on learning a representation that facilitates efficient content-based retrieval. Although the representations that are learned from deep neural networks can contain rich information, it is computationally expensive to use those representations to perform a search for the best match. In particular, computing the Euclidean distance between a query and an instance scales linearly with the size of the representation. Prior approaches to this problem have focused either on: (1) compactifying the learned representations into another form, such as a Hamming code, in a way that preserves the identifiability of an instance; (2) resorting to approximate methods that sacrifice accurate search for efficiency.

To address these inconvenient trade-offs, this paper proposes an algorithm to learn a high-dimensional, sparse representation that is directly used in retrieval, instead of learning a separate, more compact representation. The novelty of this algorithm is its focus on minimizing the number of FLOPs in computing queries of instances, taking note as well of the role of the distribution of non-zero values in determining the number of FLOPs. A continuous relaxation of the equations thus derived provides the proposed algorithm. The paper notes differences between the algorithm and SDH, a recent candidate that learns a sparse, high-high dimensional hash. Based on experiments, the paper claims that the proposed algorithm yields a similar or better speed vs. recall tradeoff compared to baselines. The paper also provides additional experiments demonstrating the sparsity of the representations and the even distribution of non-zero values of the proposed regularizer.

Decision: accept
The algorithm is clearly and succinctly motivated from the standpoint of reducing the number of FLOPs. The presentation of the distribution that minimizes FLOPs is convincing, and there is easy-to-follow buildup into the continuous relaxation of the FLOPs minimization problem.
The proposed algorithm is itself relatively simple (just an additional regularizer term that can be optimized with any SGD-based optimizer), compared to other methods that learn a separate representation or that use approximate nearest-neighbour search, and directly tries to address aforementioned trade-offs between efficiency of retrieval and richness of the representation.
I found helpful the comparison both to the nearest competitor method (SDH) and the unrelaxed regularizer. The additional experiments comparing the continuous relaxation and the unrelaxed regularizer were interesting, but I found Figure 2b a little hard to understand. I also appreciated the intuition developed at the end of section 4 for how the regularizer promotos orthogonality.
The recall/time trade-off curves in figure 3 support the main empirical claim of the paper. The sparsity plots in figure 3 contributed to a broader understanding of the algorithm besides based on metrics other than accuracy.
Nevertheless, there were some experimental presentation issues. Are errors bars possible for figs 3ac? Some evaluation metrics that are present in other papers are also missing, which might provide a more complete understanding of algorithm's advantages and disadvantages. Specifically, precision@k (Jeong and Song, 2018) and true-positive/false-positive curves (Kemelmacher-Shlizerman et. al., 2016) could help readers in making an informed algorithmic choice. It also seems that the paper could have included more comparisons to other ways of learning sparse representations (e.g., L2, top-k autoencoders, dropout), and included a broader literature review of learning sparse representations (see here: https://arxiv.org/pdf/1505.05561.pdf).
Questions
1. Did confidence thresholds in ranking (Appendix B, re-ranking) affect the results in any way? Did it depend on the algorithm used?
Minor comments that did not affect the rating
1. Typo in figure 3; the second sentence should say "curves are produced by"
2. Typos in Appendix A
a. Lemmas 2, 5 should have X_+ instead of X in the statements
b. Second last line of proof of lemma 1 should have X_+
3. Would be helpful to have whole procedure from learning to retrieval in an algorithm box
4. Include a summary of the problem setting before section 3. It is at first a bit unclear what the problem is for newcomers
5. Add algorithm header to the retrieval algorithm presented in fig 1


Effectively and accurately synthesizing point clouds is yet a challenge task due to the high-dimensionality of the observation space. In this article, we focus on the generative model to capture the informative intrinsic structure in the latent low-dimensional space and the diversity in the ambient space. We propose a new dual adversarial model which combines Generative Adversarial Network (GAN) and AutoEncoder (AE) with well-designed loss. Extensive experiments demonstrate the effectiveness of the framework on 3D point clouds synthesis. The dual adversarial model achieves the state-of-the-art performance comparing with several well-known point clouds generation models. We also perform experiments on MNIST dataset that exhibit the competitive performance on 2D data.

Summary:
The paper focuses on designing a generative framework for 3-D point data clouds. These point clouds correspond to objects shapes in 3-dimensions. According to the paper, previous approaches for generating such 3-D point clouds involved autoencoder and GANs used separately. The authors propose a framework combining both autoencoder and GAN in a single network. The authors claim that part of the network learns effective latent space embedding for 3-d point clouds corresponding to different objects and thus the entire network is more effective in generating 3-d point cloud. Experimental results are presented to support the claims for efficient embeddings, and better generation of 3-d point clouds.
Decision
The paper has some lacking in explanation. I am particularly interested in the answers to the following questions:
1.	The encoder module (denoted as E in the paper) uses a loss function that apparently does not involve the encoder output (z in the paper). How the module weights can be updated this way is unclear.
2.	G1 is updated twice in each iteration, according to algorithm 1 in paper (once during update of 𝜽G1 and 𝜽G). How the generator G1’s output manages to converge to output z from encoder E is unclear.
3.	The authors claim that the proposed model has both better performance and efficiency. Although experimental results are provided to support claim for better performance, none have been offered for efficiency claim. (calculation of EMD distance seems to be very expensive computationally)
Feedback
For section 5.5, a quantitative analysis might make the claim for portability of the framework stronger. In paragraph 1 of introduction, disadvantage of 3d point cloud data can be better explained.


Federated learning has a variety of applications in multiple domains by utilizing private training data stored on different devices. However, the aggregation process in federated learning is highly vulnerable to adversarial attacks so that the global model may behave abnormally under attacks. To tackle this challenge, we present a novel aggregation algorithm with residual-based reweighting to defend federated learning. Our aggregation algorithm combines repeated median regression with the reweighting scheme in iteratively reweighted least squares. Our experiments show that our aggregation algorithm outperforms other alternative algorithms in the presence of label-flipping, backdoor, and Gaussian noise attacks. We also provide theoretical guarantees for our aggregation algorithm.

The paper proposes an algorithm for mitigating poisoning attacks in federated learning settings and compares it , on four different datasets, against state-of-the-art baselines.
Except for some minor issues (see the list below), the paper is well-written and -organized. The description of the proposed algorithm (in pseudo code and using the illustration in Figure 2) is very clear. Overall, the experiments are carefully described.
My main concern is that many choices in the design of the proposed algorithm lack context/discussion and thus appear rather ad-hoc. For instance,
- Why is the repeated median estimator used for estimating the linear regression? Could other robust estimators have been used?
- Similarly, could alternative weighting schemes be used in equations (3)-(5)?
I think it's important to provide more context and discuss possible alternatives. An important element is the exact threat model that the authors are considering. E.g., in the last paragraph on page 4, the authors mention specific attack strategies like altering only 10% of the model parameters. It appears that the design of the model weighting scheme aims at defending against these specific types of attacks. It will be good to either discuss or evaluate empirically how this scheme performs against other strategies.
The theoretical guarantee in Section 3.2 is a bit sketchy in my opinion. In what sense is $\mu$ the "expected value of the global model"? I.e. what is the expectation over? Consequently, I could not follow the statement in equation (14). Some explanation in plain text is needed here, too: in what sense does this equation provide a guarantee?
In the experiments, several aspects deserve further discussion: (1) the poor performance of FoolsGold almost across the entire board (except for the Gaussian noise attacks), which may indicate that this method was applied outside the threat model it was designed for; (2) the failure of all the baselines on CIFAR-10 for the naive attacking approach, while they perform fairly well on MNIST; (3) why does the attack success rate starts increasing in Figure 4 for the baseline methods only after ~25 iterations? (4) why do the baseline methods perform so poorly against label-flipping against on MNIST (Figure 3) while performing fairly well on CIFAR-10 and Amazon reviews (Table 1/2)? - I think that answering those questions may shed insights into the type of attacks that the different defences can / cannot withstand. I'd also like to challenge the authors to address whether they expect their defence to match or outperform the baselines on *any* attack strategy, or whether they can come up with scenarios where some of the baselines perform better? I would expect that the latter should be possible; it would not diminish the value of the proposed defence but shed more clarity on its possible limitations.
List of minor issues:
- in the abstract: "aggression" -> "aggregation"
- p.1: I would omit the statement in brackets "less than 100 lines".
- p.2: some of the related work discussion repeats content from the introduction
- p.3: "summaries" -> "summarizes"
- p.3: what does that mean: "has a high breakdown point of 50%"? Please explain/clarify.
- p.4: "is the k-the diagonal of matrix in Hn" -> "is the k-th diagonal element of the matrix Hn"
- p.4: my pdf reader couldn't render the binary operator on the right hand side of equation (8)
- p.5: "the details of the proof is presented" -> "are"
- p.8: "that of which" -> "whose"
- p.8: upper case "We" after comma
- p.8: first column "Acc" in Table 3: FedAvg has the highest accuracy. Generally, bold numbers in tables do not always mark the best-performing method. Sometimes, bold numbers are entirely missing. In cases where the difference is insignificant (which often appears to be the case) I would mark multiple numbers in bold, as appropriate.


Machine learning algorithms are vulnerable to poisoning attacks: An adversary can inject malicious points in the training dataset to influence the learning process and degrade the algorithm's performance. Optimal poisoning attacks have already been proposed to evaluate worst-case scenarios, modelling attacks as a bi-level optimization problem. Solving these problems is computationally demanding and has limited applicability for some models such as deep networks. In this paper we introduce a novel generative model to craft systematic poisoning attacks against machine learning classifiers generating adversarial training examples, i.e. samples that look like genuine data points but that degrade the classifier's accuracy when used for training. We propose a Generative Adversarial Net with three components: generator, discriminator, and the target classifier. This approach allows us to model naturally the detectability constrains that can be expected in realistic attacks and to identify the regions of the underlying data distribution that can be more vulnerable to data poisoning. Our experimental evaluation shows the effectiveness of our attack to compromise deep networks, bypassing different type of existing defences against data poisoning.

This paper proposed a method pGAN based on Generative Adversarial Networks to generate poisoning examples in order to degrade the performance of classifiers when trained on the poisoned training data. The authors evaluated pGAN on both synthetic datasets and commonly used MNIST and Fashion MNIST datasets in machine learning.
The paper is self-contained and easy to read. My main concern is on the experiment results. The detailed questions are as follows:
Q1: Has the authors tried more complicated datasets such as CIFAR-10 to evaluate the pGAN method? It would make the paper more convincing to add results on more complex datasets.
Q2: Can the authors structure the experimental results with different sections? Currently it is just a single section which is difficult to read.
Q3: The authors noticed that “But, as we decrease the value of α, the distribution of red points shifts towards the region where both green and blue distributions overlap”. This observation is interesting as it finds that the poisoned input tends to lie on the overlap of two classes. But this can easily lead to a defense method: remove those training examples that are close to the other class. This defense mechanism can be used together with other sanitization approaches. So I would like to see how would pGAN perform in this case?
Q4: The authors mentioned “Comparison with existing poisoning attacks in the research literature is challenging: Optimal poisoning attacks as in Munoz-Gonzalez et al. (2017) are computationally very expensive for the size of the networks and datasets used in our experiments in Fig. 2.”.
However, I can not agree because you can simply generate poisoned data and train the neural networks on the poisoned data regardless of the underlying approach that is targeted in generating the poisoned data. This would be an effective baseline to compare. (Correct me if I am wrong here.)
I will change my score if the authors can address my concerns here.
================================================================
Thanks for the rebuttal. I am more convinced now.


Recent studies on catastrophic forgetting during sequential learning typically focus on fixing the accuracy of the predictions for a previously learned task. In this paper we argue that the outputs of neural networks are subject to rapid changes when learning a new data distribution, and networks that appear to "forget" everything still contain useful representation towards previous tasks. Instead of enforcing the output accuracy to stay the same, we propose to reduce the effect of catastrophic forgetting on the representation level, as the output layer can be quickly recovered later with a small number of examples. Towards this goal, we propose an experimental setup that measures the amount of representational forgetting, and develop a novel meta-learning algorithm to overcome this issue. The proposed meta-learner produces weight updates of a sequential learning network, mimicking a multi-task teacher network's representation. We show that our meta-learner can improve its learned representations on new tasks, while maintaining a good representation for old tasks.

Summary: This paper introduces a variation on measuring catastrophic forgetting in sequential learning at the representation level and attempts to resolve forgetting issue with the help of a meta-learner that predicts weight updates for previous tasks while it receives supervision from a multi-task learner teacher. The new method is evaluated on sequences of two tasks while task 1 data remains available at all times to the teacher.
Pros:
(+): This paper is very well-written and very well-motivated.
(+): Tackling continual learning from a meta-learning approach is novel and not yet well-explored.
(+): Literature review is done precisely well.
Cons that significantly affected my score and resulted in rejecting the paper are two-fold.
First, based on my understanding from the paper, it appears that this work has a significant contradictory assumption with a regular continual learning setup and that is to provide access to the entire dataset from an old task while we learn a new task. This changes the problem from continual/sequential/lifelong learning to multi-task learning. All the prior work that were beautifully reviewed in section 1 and 2 obey this assumption where access to previous tasks’ data is either impossible (ex. [1,3,4,5,6,7,8] in the below list ) or is very limited (ex. [2]).
Second, is the experimental setting. The experiments are accurately described and performed but authors have only considered sequence of 2 tasks which is far from being considered as a continual learning setting. I would like to ask the authors to explain how this method can be extended to multiple tasks and how much of the past data they should provide while training? Another drawback in the experiments is about the baselines. Despite addressing the most recent papers in section 2, authors have only made comparison against two relatively old approaches (EWC by Kirkpatrickthat et al from 2016 as well as LwF by Li & Hoiem presented at ECCV 2016, I believe the authors have cited the journal version of the work published in 2018 but the work is actually from ECCV 2016). Although these methods are still included as baselines in the literature, more recent approaches which have outperformed these need to be provided as well. I have provided a list of papers which achieved superior performance to the current baselines below which is arranged chronologically and is indeed not limited to this list as it is not realistic to list all prior work since 2016 in here.
I would be happy to change my score if authors can address the above concerns about considering distinguishing multi-task learning from continual learning and providing a realistic evaluation setup with more than 2 tasks and comparison with current state of the art methods.
[1] Zenke, Friedemann, Ben Poole, and Surya Ganguli. "Continual learning through synaptic intelligence." Proceedings of the 34th International Conference on Machine Learning-Volume 70. JMLR. org, 2017.
[2] Lopez-Paz, David, and Marc'Aurelio Ranzato. "Gradient episodic memory for continual learning." Advances in Neural Information Processing Systems. 2017.
[3] Shin, Hanul, et al. "Continual learning with deep generative replay." Advances in Neural Information Processing Systems. 2017.
[4] Nguyen, Cuong V., et al. "Variational continual learning." arXiv preprint arXiv:1710.10628 (2017).
[5] Serrà, J., Surís, D., Miron, M. & Karatzoglou, A.. (2018). Overcoming Catastrophic Forgetting with Hard Attention to the Task. Proceedings of the 35th International Conference on Machine Learning, in PMLR 80:4548-4557
[6] Schwarz, Jonathan, et al. "Progress & compress: A scalable framework for continual learning." arXiv preprint arXiv:1805.06370 (2018).
[7] Mallya, Arun, and Svetlana Lazebnik. "Packnet: Adding multiple tasks to a single network by iterative pruning." Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2018.
[8] Ebrahimi, Sayna, et al. "Uncertainty-guided Continual Learning with Bayesian Neural Networks." arXiv preprint arXiv:1906.02425 (2019).
[9] Aljundi, Rahaf, et al. "Online continual learning with no task boundaries." arXiv preprint arXiv:1903.08671 (2019).
------------------------------------------------------------------------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------------------------------------------------------------
POST-REBUTTAL review:
I disagree with the authors claiming that this work is continual learning (sequential learning + avoiding forgetting).
Despite introducing 9 recent continual learning work to authors in my initial review, they added 2 meta-learning baselines (MAML,REP), keeping 2 naive and old CL baselines is not acceptable. I reply to authors comment below regarding the baselines:
[Authors' reply:] Lopez-Paz, David, and Marc'Aurelio Ranzato. "Gradient episodic memory for continual learning." and Shin, Hanul, et al. "Continual learning with deep generative replay." and Nguyen, Cuong V., et al. "Variational continual learning” and Aljundi, Rahaf, et al. "Online continual learning with no task boundaries." We argue that these paper have a different setting compared to ours since they require a buffer whereas our method has no storage of past data/gradient. Having past data storage can usually improve the performance and our method can potentially also get a boost. Having a data buffer can also cost a lot of memory storage depending on input/weight dimension. Therefore, we argue it won’t be a fair setting to compare with methods with data buffers.
[Reviewer's reply:] GEM (Lopez-Paz et al., 2017) and its faster version (A-GEM) (Chaudhry, et al. 2018) and other memory based methods  such as MER (Riemer et al. 2018), ER-RES (Chaudhry et al. 2019), they use memory sizes of at most 6MB to store samples but they only do a **single epoch** through the data. So if it is not fair, it would be for those methods given the computational expenses of this paper. VCL (Nguyen et al. 2018) in its vanilla version does not use coreset memory if that is still your concern.
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
[Authors' reply:] Zenke, Friedemann, Ben Poole, and Surya Ganguli. "Continual learning through synaptic intelligence." has very similar performance to EWC in their paper. We have cited this work already.
[Reviewer's reply:] This method is an online version of EWC which is faster despite the on-par performance. So it has its own advantage.
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
[Authors' reply:] Serrà, J., Surís, D., Miron, M. & Karatzoglou, A.. (2018). Overcoming Catastrophic Forgetting with Hard Attention to the Task and other pruning based papers. Thanks for pointing out. We have cited and will compare to them in the future. One thing to note is that, in these works, the model needs to know which task ID it is currently dealing with, and thus can turn on the pruning procedure for the next session. This can potentially be a limitation for dynamic incoming tasks.
[Reviewer's reply:] Comparing with HAT paper (Serrà et al. 2018) is really easy using their provided code and is one of the strongest baselines in continual leaning literature. They do NOT do any pruning. Their approach simply learns an attention mask which regularizes weights and prevent changes on them without using any memory. Regarding the task number, this is indeed not an issue for your approach with 2 tasks.
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
[Authors' reply:] Ebrahimi, Sayna, et al. "Uncertainty-guided Continual Learning with Bayesian Neural Networks." has not been published in a conference venue, and it may be too early to compare with it.
[Reviewer's reply:] I agree that this work is not published and hence can't be asked for comparison but I encourage authors to read it.
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Authors are neglecting one important difference between meta-learning and continual learning: in MAML and REP, it is assumed that we have access to ALL tasks distributions from which we sample from in the beginning (look at page 3, algorithm 1, line 3). This is in contrast to continual learning where one cannot even assume how many tasks will be given. Moreover, the computational expense of this work which causes performing more than 2 tasks to be a future work is also not acceptable when there are significantly cheaper and are able to do a lot more than 2. (In all the references I mentioned, the length of the sequence in experiments is at least 5.)
While this work might be interesting to meta-learning community, I think it is far from being introduced as a method that prevents catastrophic forgetting and hence be included in the CL literature. Therefore, I intend to keep my score as reject.


Various methods of measuring unit selectivity have been developed with the aim of better understanding how neural networks work. But the different measures provide divergent estimates of selectivity, and this has led to different conclusions regarding the conditions in which selective object representations are learned and the functional relevance of these representations. In an attempt to better characterize object selectivity, we undertake a comparison of various selectivity measures on a large set of units in AlexNet, including localist selectivity (Bowers et al., 2014), precision (Zhou et al., 2015, class-conditional mean activity selectivity (CCMAS) (Morcos et al., 2018), network dissection (Zhou et al., 2018a), the human interpretation of activation maximization (AM) images, and standard signaldetection measures. We find that the different measures provide different estimates of object selectivity, with precision and CCMAS measures providing misleadingly high estimates. Indeed, the most selective units had a poor hit-rate or a high falsealarm rate (or both) in object classification, making them poor object detectors. We fail to find any units that are even remotely as selective as the 'grandmother cell' units reported in recurrent neural networks. In order to generalize these results, we compared selectivity measures on a few units in VGG-16 and GoogLeNet trained on the ImageNet or Places-365 datasets that have been described as 'object detectors'. Again, we find poor hit-rates and high false-alarm rates for object classification.

The paper empirically studies the category selectivity of individual cells in hidden units of CNNs. It is a sort of "meta-study" and comparison of different metrics proposed to identify cells with a preference for a specific target category. The claimed finding is that there are no cells that are "sufficiently" selective to be called object detectors.
The paper is seemingly motivated by the authors' perceiving a contradiction: it is assumed that the power of neural networks is (among others) due to the distributed representation; whereas the presence of object detectors would, in the extreme case, mean that the representation is disentangled into a separate unit per category. It may be a matter of terminology, but this is where my disagreement with the authors start. I do not see a simplistic dichotomy, where one could or should determine which of the two interpretations is "right" or "wrong". In my view, which I believe is the mainstream interpretation, a distributed representation does not contradict the presence of specialised units. Some categories probably are easily identified by few distinctive features, so there will be more detector-like units; others are complex and hence more diffusely spread through the network; and of course there is no guarantee that the learned "object detectors" are tuned to exactly the target categories, after all it is the purpose of the network to gradually translate the data distribution to the label distribution - if the categories were directly apparent in the data, nearest-neighbour would be enough. So it is not only possible, but rather likely that the learned "object detectors" are to some degree driven by the statistics of the data, not the labels - e.g., there could be a highly selective "bird" unit which nevertheless has high false positive rate for any of the more specific bird species categories in the imageNet nomenclature. And vice versa, there could be a highly specific "Ferrari" detector that is so specialised that it has low recall for the "sports car" class (this case includes, among others, the case of viewpoint-specific detectors for certain categories). In the words of the paper, the "selective units are sensitive to some feature that is frequently, but not exclusively associated with the class" - I thought this is the standard majority view, not a surprising finding. In this context terminology matters: the study effectively tries to disprove that the network learns "near-perfect single output-category detectors", but who claims that it would do that?
I agree with the authors that there is by now a zoo of selectivity metrics that are not always highly correlated. But is that a problem? We have a zoo of quality metrics for many machine learning problems - that is not necessarily a weakness, but simply reflects the obvious fact that a single number is not enough to characterise performance in a complex cognitive task. It is the job of the researcher/user to chose the metric that s most suitable for their specific question, and to correctly interpret its numerical value.
Regarding the methodology, the paper did a lot of work to systematically crunch the numbers and analyse network units. It is a laudable effort that someone took on that job. A few technical decisions are unclear to me. Why analyse only some of the units? If one collects statistics over >2000 units of a fully connected layer, one might as well do the complete job and use all 4096 units. Similarly, why analyse only the correctly classified images? While it is clear that one must separately look at them, also the activations on incorrectly classified ones could provide valuable insights. E.g., do false positives of class X on average activate a certain "class X detector"? Why chose only the class with highest mean activation for CCMAS? That might be unrepresentative, e.g., a neuron might, for that particular class, always have high activation due to some very common background context, and still be not selective at all.
Regarding the results, I find them much less clear-cut than the paper claims. For example, I find it quite remarkable that some unit has 8% recall at perfect precision. After all, only approximately 0.1% of the images are in the correct category, so a unit that flags 8% of them without making a mistake is a pretty good detector for (part of) the target class, cf Fig. 3. Also regarding Fig. 2 / maximum informedness, the statistics actually do not look bad. Of course false alarm proportion remains high - but the chance level here is 99.9%, so even a 99% false alarm rate means that your unit can, on its own, reject 90% of the true negatives. I find the proposed "minimum condition" for an object detector (>50% recall at >50% precision) unrealistic: the top-1 accuracy of AlexNet is, to my knowledge, <63%. Even the complete network probably never reaches 50% recall for most classes.
Especially the user study - which is again a commendable effort - in my view does not confirm the claims. According to that study, almost 60% 0f all fc8 units are "object detectors", with very high conherence between humans and selectivity metrics.
Overall, while it is an interesting study, it remains unclear to me what I should learn from it. I don't see why different measures provide "misleading conclusions" that need to be rectified. Conclusions are the responsibility of the researcher interpreting the numbers, not of the formula to calculate some statistical performance metric. I am in a difficult situation here: the study is one of those things (like determining human performance on ImageNet, or re-coding some baseline where the original code is not available) where I find it valuable that someone did them in the community, but still I don't think they need a reviewed paper.  A note on the blog, or on arXiv, is enough.


We consider composition problems of the form
Composition optimization arises in many important machine learning applications: reinforcement learning, variance-aware learning, nonlinear embedding, and many others. Both gradient descent and stochastic gradient descent are straightforward solution, but both require to compute 1 n n j=1 G j (x) in each single iteration, which is inefficient-especially when n is large. Therefore, with the aim of significantly reducing the query complexity of such problems, we designed a stochastically controlled compositional gradient algorithm that incorporates two kinds of variance reduction techniques, and works in both strongly convex and non-convex settings. The strategy is also accompanied by a mini-batch version of the proposed method that improves query complexity with respect to the size of the mini-batch. Comprehensive experiments demonstrate the superiority of the proposed method over existing methods.

This paper proposes a new method for empirical composition problems to which the vanilla SGD is not applicable because it has a finite-sum structure inside non-linear loss functions. A proposed method (named SCCG) is a combination of stochastic compositional gradient descent (SCGD) and stochastically controlled stochastic gradient (SCSG). In a theoretical analysis part, a linear convergence rate and a sub-linear convergence rate are derived under the strong convex and non-convex settings, respectively. In experiments, the superior performance of the method to competitors is verified on both strongly convex and non-convex problems.
Clarity:
The paper is clear and well written.
Quality:
The work is of good quality and is technically sound.
Significance:
The problem treated in this paper is important and contains several applications as mentioned in the paper. Hence, developing an efficient method for this problem is important and interesting. Although derived convergence rates are better than existing primal methods, this paper lacks a comparison with the recently proposed primal-dual method by [A.Devraj & J.Chen (2019)].
[A.Devraj & J.Chen (2019)] Stochastic Variance Reduced Primal Dual Algorithms for Empirical Composition Optimization. NeurIPS, 2019.
A convergence rate obtained in [A.Devraj & J.Chen (2019)] seems faster than that of SCCG for ill-conditioned strongly convex problems. However, there exists a certain setting (large-scale setting) where SCCG outperforms their method. Thus, the contribution of the paper is not lost, but it is better to compare SCCG with the method in [A.Devraj & J.Chen (2019)], empirically and theoretically.
If the authors can show an empirical advantage over their method, it will make the paper stronger.
-----
Update:
I thank the authors for the response and hard work. I am convinced of the advantage of the proposed method. I would like to keep my score.


Exploration while learning representations is one of the main challenges Deep Reinforcement Learning (DRL) faces today. As the learned representation is dependant in the observed data, the exploration strategy has a crucial role. The popular DQN algorithm has improved significantly the capabilities of Reinforcement Learning (RL) algorithms to learn state representations from raw data, yet, it uses a naive exploration strategy which is statistically inefficient. The Randomized Least Squares Value Iteration (RLSVI) algorithm (Osband et al., 2016), on the other hand, explores and generalizes efficiently via linearly parameterized value functions. However, it is based on hand-designed state representation that requires prior engineering work for every environment. In this paper, we propose a Deep Learning adaptation for RLSVI. Rather than using hand-design state representation, we use a state representation that is being learned directly from the data by a DQN agent. As the representation is being optimized during the learning process, a key component for the suggested method is a likelihood matching mechanism, which adapts to the changing representations. We demonstrate the importance of the various properties of our algorithm on a toy problem and show that our method outperforms DQN in five Atari benchmarks, reaching competitive results with the Rainbow algorithm.

The paper proposes to extend the popular linear-control algorithm, RLSVI, to utilize learned representations. This is done by adapting a work from bandit literature that utilizes BLR with representations that are learned via a DNN. The proposed solution is then compared to DQN with fixed epsilon as the exploration strategy in a chain MDP, and to the Rainbow agent and DQN in 5 selected Atari games to show sample-efficiency improvements.
The idea presented by the work is interesting, and utilizing a complete Bayesian Linear Regression framework (variable variance, as opposed to fixed variance in the prior) does sound appealing in terms of adaptability - as it is what the authors argue for, as being key to their proposal. But the work in the paper is not in an acceptable form due to the following key reasons: (1) the presentation of the ideas and the algorithm, "despite the lack of theoretical guarantees" is hard to understand, (2) the DQN baseline compared to (definitely in the Augmented Chain Environment, and possibly in the Atari games), are based on a fixed epsilon exploration strategy whereas DQN as proposed uses a epsilon-annealing strategy for exploration, (3) the baselines compared to are not comprehensive, (4) generally the paper is rather unpolished.
Following are main points of feedback, regarding which I would like to know the authors opinions/responses in the rebuttal if possible. After that, are some concrete questions to clarify the contents of the paper.
Feedback:
(1) the linear contextual bandit work the paper is built on is currently under review at ICLR 2020. While this should not be the reason for rejection, the idea of likelihood matching as motivated from that bandit work does not simply transfer to the RL case. For instance, based on the pseudocode in the paper (and the provided explanation), after likelihood matching is done, the new BLR updates still seem to use the old representations and targets (as the target net update is done after the loop). This is possibly a typo, or something deeper is left unexplained here.
(2) While it is true that the BDQN work does not utilize the last layer of their weights, and is built on the DDQN algorithm, I think they still are a reasonable baseline as the key idea distinguishing the two is variable variance prior vs. a fixed variance prior. While I understand having all the baseline experiments from the exploration in deep RL literature is hard as it is quite vast - Bootstrap DQN, UBE, Bootstrap Prior - I think the closest baseline has to be BDQN to your proposal and therefore is a natural competitor.
(3) DDQN has been shown to be reducing the overestimation bias prevalent in DQN and therefore was the framework BDQN was built on. Why do you choose to use DQN instead of DDQN? A discussion regarding this seems a natural part of the paper.
(4) Are the benefits of adaptive sigma present if the base algorithm is changed to DDQN? I think this is an important point of discussion/analysis. It does not have to outperform DDQN, but even a comparative study empirically would be an insightful and comprehensive contribution.
(5) I assume that the likelihood utilizes the inverse of the covariance as opposed to the covariance -- something that seems amiss in the current write-up.
(6) Isn't S_i in Section 4.3 already a scalar? Why is there a trace of a scalar?
(7) Section 5.1.3 -- the version that does not match likelihood is not BDQN as it is built on DDQN -- presumable significant regret differences.
Questions:
(1) DQN as proposed is with annealed e-greedy. Are the experiments in Augmented Chain Environment utilizing this or a fixed e-greedy strategy?
(2) What exactly is the connection between catastrophic forgetting and likelihood matching in the policy improvement context? Why should an improved policy match the likelihood of features as learned by an older policy?
(3) Why choose to sample a set of value functions instead of sampling every timestep? Only because sampling is expensive or does it provide any stability?
(4) Why did you pick these 5 Atari games?
Given the algorithm is a particular design choice, and the argument for its utility is empirical, I definitely think the design choice needs to be discussed more thoroughly, and the manuscript currently does not do that. While empirical experiments in the Atari suite can be hard ask depending on the availability of computational resources, I think the work algorithmic choices made here are left undiscussed and the empirical results aren't really convincing. Further, the presentation is rather imprecise and error-ridden. Therefore, I do not think the work can be accepted.
There are many imprecise statements and typos in the paper, which are listed here to aid the future versions:
(1) Please review your psuedocode based on comment (1) in Feedback.
(2) The content in the Introduction does not make a note of many algorithms proposed for exploration in Deep RL -- UBE [1], Bootstrap DQN [2], Randomized prior for Bootstrap DQN [3], Parameter noise[4].
(3) Section 2, para 2 -- Recent work (Osband --> Recent works (Osband
(4) Section 2, para 3 -- acts greedy --> acts greedily
(5) Section 3, para 1 -- gamma in (0,1) --> [0,1]
(6) Section 3, para 1 -- survey --> review
(7) Section 3.1, para 1 -- acts the greedy action according to the --> acts greedily wrt
(8) Section 4.1, buffer consists of gamma/termination flag
(9) Section 4.2, para 1 -- First, rather than solving .. -- so did RLSVI. I guess you mean the regression problem is not solved every tilmestep.
(10) Section 4.2 last sentence -- please expand why that assumption is good?
(11) opening inverted commas everywhere.
(12) Section 5.1.4, last sentence -- grammar.
(13) Footnote 1 and content in Section 5.2 "we used publicly available learning curves" are contradictory.
[1] O'Donoghue, Brendan, et al. "The uncertainty bellman equation and exploration." arXiv preprint arXiv:1709.05380 (2017).
[2] Osband, Ian, et al. "Deep exploration via bootstrapped DQN." Advances in neural information processing systems. 2016.
[3] Osband, Ian, John Aslanides, and Albin Cassirer. "Randomized prior functions for deep reinforcement learning." Advances in Neural Information Processing Systems. 2018.
[4] Plappert, Matthias, et al. "Parameter space noise for exploration." arXiv preprint arXiv:1706.01905 (2017).


Autonomous agents that must exhibit flexible and broad capabilities will need to be equipped with large repertoires of skills. Defining each skill with a manuallydesigned reward function limits this repertoire and imposes a manual engineering burden. Self-supervised agents that set their own goals can automate this process, but designing appropriate goal setting objectives can be difficult, and often involves heuristic design decisions. In this paper, we propose a formal exploration objective for goal-reaching policies that maximizes state coverage. We show that this objective is equivalent to maximizing the entropy of the goal distribution together with goal reaching performance, where goals correspond to full state observations. To instantiate this principle, we present an algorithm called Skew-Fit for learning a maximum-entropy goal distributions. Skew-Fit enables self-supervised agents to autonomously choose and practice reaching diverse goals. We show that, under certain regularity conditions, our method converges to a uniform distribution over the set of valid states, even when we do not know this set beforehand. Our experiments show that it can learn a variety of manipulation tasks from images, including opening a door with a real robot, entirely from scratch and without any manually-designed reward function.

Summary :
The paper proposes an exploratory objective that can maximize state coverage in RL. They show that a formal objective for maximizing state coverage is equivalent to maximizing the entropy of a goal distribution. The core idea is to propose a method to maximize entropy of a goal distribution, or a state distribution since goals are full states. They show that the proposed method to maximize the state or goal distribution can lead to diverse exploration behaviour sufficient for solving complex image based manipulation tasks.
Comments and Questions :
	- The core idea is to maximize the entropy of the state visitation frequency H(s). It is not clear from the paper whether the authors talk about the normalized discounted weighting of states (a distribution) or the stationary distribution? The entropy of the state visitation distribution only deals with valid states - but I am not sure what it means to maximize the entropy of this term exactly in terms of exploration, since it is neither the discounted weighting of states or the stationary distribution for an infinite horizon task?
	- The authors do mention that maximizing the entropy of H(s) is not sufficient - so instead suggests for maxmizing entropy of H(s|g). But why is this even sufficient for exploration - if I do not consider new tasks at test time but only the training task? How is this a sufficient exploration objective? Furthermore, since it is the conditional entropy given goal states, the fundamental idea of this is not clear from the paper.
	- Overall, I am not convinced that an objective based on H(s|g) is equivalent to an maximizing H(s), and why is this even a good objective for exploration? The meaning of H(s) to me is a bit vague from the text (due to reasons above) and therefore H(s|g) does not convince to be a good exploration objective either?
	- The paper then talks about the MI(S;G) to be maximized for exploration - what does this MI formally mean? I understand the breakdown from equation 1, but why is this a sufficient exploration objective? There are multiple ideas introduced at the same time - the MI(s;g) and talking about test time and training time exploration - but the idea itself is not convincing for a sufficient exploration objective. In light of this, I am not sure whether the core idea of the paper is convincing enough to me.
	- I think the paper needs more theoretical insights and details to show why this form of objective based on the MI(s;g) is good enough for exploration. Theoretically, there are a lot of details missing from the paper, and the paper simply proposes the idea of MI(s;g) and talks about formal or computationally tractable ways of computing this term. While the proposed solutuon to compute MI(s;g) seems reasonable, I don't think there is enough contribution or details as to why is maximizing H(s) good for exploration in the first place.
	- Experimentally, few tasks are proposed comparing skew-fit with other baselines like HER and AutoGoal GAN - but the differences in all the results seem negligible (example : Figure 5).
	- I am not sure why the discussion of goal conditioned policies is introduced rightaway. To me, a more convincing approach would have been to first discuss why H(s) and the entropy of this is good for exploration (discounted weighting or stationary state distribution and considering episodic and  infinite horizon tasks). If H(s) is indeed a difficult or not sufficient term to maximize the entropy for, then it might make sense to introduce goal conditioned policies? Following then, it might be convincing to discuss why goal conditioned policies are indeed required, and then tractable ways of computing MI(s;g).
	- Experimentally, I think the paper needs significantly more work - especially considering hard exploration tasks (it might be simple setups too like mazes to begin with), and then to propose a set of new experimental results, without jumping directly to image based tasks as discussed here and then comparing to all the goal conditioned policy baselines.
Overall, I would recommend to reject this paper, as I am not convinced by the proposed solution, and there are lot of theoretical details missing from the paper. It skips a lot of theoretical insights required to propose a new exploration based objective, and the paper proposes a very specific solution for a set a very specific set of experimental setups.