While many sentiment classification solutions report high accuracy scores in product or movie review datasets, the performance of the methods in niche domains such as finance still largely falls behind. The reason of this gap is the domainspecific language, which decreases the applicability of existing models, and lack of quality labeled data to learn the new context of positive and negative in the specific domain. Transfer learning has been shown to be successful in adapting to new domains without large training data sets. In this paper, we explore the effectiveness of NLP transfer learning in financial sentiment classification. We introduce FinBERT, a language model based on BERT, which improved the state-of-the-art performance by 14 percentage points for a financial sentiment classification task in FinancialPhrasebank dataset.

This paper described the application of BERT in the field of financial sentiment analysis. Authors find that when fine-tuned with in-domain data, BERT outperforms the state-of-the-art, demonstrating that language model pre-training can transfer knowledge learned from unsupervised large corpus to new domain with minimum effort. Experiments are conducted to explore 1) the utility of different in-domain dataset for further pre-training; 2) strategies to avoid catastrophic forgetting, and 3) effectiveness of fine-tuning a subset of the full model.
I am in favor of rejecting this paper and my reasons are as follows:
First, this paper may lack deeper innovation, although it demonstrates a good application of the BERT models in financial domain. For example, the framework of general-domain LM pretraining, to in-domain LM pretraining and finally in-domain classifier fine-tuning, as well as techniques of catastrophic forgetting were already proposed in Howard & Ruder 2018. Therefore, I think this paper may be more suitable for other (finance) application-oriented venues.
Second, the dataset used in evaluation is of small size (for example, Financial PhraseBank test set has one 1K). Thus, even though the paper is about transfer learning to domains without large data, I find it might be more convincing to draw a solid conclusion with a larger test set.
This paper is well organized and easy to follow. It may be beneficial to clarify in a few places (if space permits):
1) Some description or statistics of the data may be helpful (e.g., average sentence length or some examples);
2) Citations to Elmo and ULMFit can be made more explicit. Authors did cite Peters 2018 and Howard 2018 at the beginning of the paper, but may want to explicitly associate them with ‘Elmo’ and ‘ULMFit’ when these two terms first occur respectively;
3) For table 2, does the ‘all data’ or ‘data with 100% agreement’ include training data (80%) or just the test data (20%)?
The difference between FinBERT(-domain) and ULMFit can be explicitly contrasted in the paper. Is the former initialized with BERT while latter with ULMFit?


Deep learning training accesses vast amounts of data at high velocity, posing challenges for datasets retrieved over commodity networks and storage devices. We introduce a way to dynamically reduce the overhead of fetching and transporting training data with a method we term Progressive Compressed Records (PCRs). PCRs deviate from previous formats by using progressive compression to convert a single dataset into multiple datasets of increasing fidelity-all without adding to the total dataset size. Empirically, we implement PCRs and evaluate them on a wide range of datasets: ImageNet, HAM10000, Stanford Cars, and CelebA-HQ. Our results show that different tasks can tolerate different levels of compression. PCRs use an on-disk layout that enables applications to efficiently and dynamically access appropriate levels of compression at runtime. In turn, we demonstrate that PCRs can seamlessly enable a 2× speedup in training time on average over baseline formats.

Summary: This paper introduces a new storage format for image datasets for machine learning training. The core idea is to use progressive JPEG to create sequential scans of the input image, from lower resolution to higher resolution. The authors found that on some datasets, using half of the scans is already enough to reach similar accuracy but speeded up the convergence by a factor of 2.
Detailed feedbacks:
-	The paper presents a simple idea that directly uses the nature of JPEG compression. The paper shows that it can work well and can be potentially integrated into real machine learning dataset storage applications.
-	Related work section is thorough.
-	The experiments are limited to image classifications, and some of the datasets are subsampled (e.g. ImageNet and CelebA). This may not well represent real machine learning tasks, and practitioners may be unsure about the reliability of the compression. The “Cars” dataset contains fine-grained classification, in which the proposed method is
-	Figure 1 is not very clear what is the key advantage of the proposed method, and what are the different mechanisms.
-	Alternatively, one can subsample the pixels and store incremental subsets of those pixels. It would be good if the paper can discuss about this baseline.
-	The data storage format is only loosely related to the main goal of the paper, which is to show that network can still train very well and even faster when receiving partial input data. Once they figured out the number of scans needed for this application, they don’t necessarily need to keep a full lossless version and can just go for a lossy version. In other words, the experiment section can be replaced by any other lossy compression by varying the compression ratio.
-	In my opinion, there could be two reasons for faster convergence. 1) lowered image quality makes the data easier to learn and 2) the smaller data size allows faster reading of data from disk. The paper only shows wall-clock speed-up, but it is unclear which factor is bigger. 2) can be potentially addressed by faster disk reading such as SSD or in-memory datasets. One of the motivations is to help parallel training of dataset and it is also mentioned how non-random sampling of data can hurt training performance. It would be good to showcase how the proposed method can help in those parallel training settings.
Conclusion: This paper presents a simple and effective idea and can be potentially beneficial. However, my main concern is whether the experiments can be representative enough for large scale experiments (e.g. using non-subsampled ImageNet dataset with parallel training using SSD storage). Therefore, my overall rating is weak accept.


In recent years we have witnessed tremendous progress in unpaired image-toimage translation methods, propelled by the emergence of DNNs and adversarial training strategies. However, most existing methods focus on transfer of style and appearance, rather than on shape translation. The latter task is challenging, due to its intricate non-local nature, which calls for additional supervision. We mitigate this by descending the deep layers of a pre-trained network, where the deep features contain more semantics, and applying the translation between these deep features. Specifically, we leverage VGG, which is a classification network, pre-trained on ImageNet with large-scale semantic supervision. Our translation is performed in a cascaded, deep-to-shallow, fashion, along the deep feature hierarchy: we first translate between the deepest layers that encode the higher-level semantic content of the image, proceeding to translate the shallower layers, conditioned on the deeper ones. We show that our method is able to translate between different domains, which exhibit significantly different shapes. We evaluate our method both qualitatively and quantitatively and compare it to state-of-the-art image-to-image translation methods. Our code and trained models will be made available.

* Summarize what the paper claims to do/contribute.
This paper claims to extend existing image translation works, like CycleGAN, to domain pairs that are not similar in shape. It is proposed to do so by using a VGG network trained on classification (I assume on Imagenet), extracting features from the two domains and learn 5 CycleGANs to translate for each level of the feature hierarchy. At each level of the hierarchy the translation from the previous level is used to condition the translation for the current level. During inference, the final image translation is done by "feature inversion" (a technique proposed in Dosovitsikiy and Brox, 2016) from the final feature layer. The technique is show on example from a number of pairs of domains like Zebra-to-Elephant (and back), Giraffe-to-Zebra (and back), Dog-to-Cat (and back) and is compared with a number of baselines qualitatively and quantitatively with the FID score.
* Clearly state your decision (accept or reject) with one or two key reasons for this choice.
Weak Reject.
Major reasons:
- The problem itself, as stated in the introduction, seems ill-posed to me. One of the struggles I had while looking through the results was to understand what the images should be looking like. ie What should a zebra translated to a giraffe look like? The motivation for such a problem is also not immediately clear either.
- Most of the resulting images do not seem "translated" to me. As stated in the paper (end of p.2) "one aims to transform a specific type of object without changing the background." As one can see in eg Fig. 1 the resulting translations are completely different images with the foreground object of the new domain in roughly similar poses. The background in most cases does not persist. What I suspect is actually happening here is that the high-level semantics from the first image are used as some sort of noise to generate new images from the new domain. One question I had, for example: could we be getting similar results if we used the VGG bottleneck as the noise vector in an InfoGAN? Since the VGG network is pretrained and used in the same way in both domains, I imagine we would be seeing something very similar. (and it would be def. preferrable to tuning 10 GANs!)
* Provide supporting arguments for the reasons for the decision.
Some of the decisions made in the paper were unclear and not supported adequately. The questions (in rough order of importance) that made some of the contributions unclear to me:
- Why wasn't a final translator used for the final image, conditioned on the final \tilde{b}_1?
- Is the VGG network pretrained on ImageNet? Why wasn't another task used that could be retaining more of the relevant features? eg on semantic segmentation
- Could this be used for networks pretrained on other datasets? Presumably ImageNet has information about the animals translated in this paper. Even better, could we somehow learn these features for the domain pairs automatically somehow?
- How meaningful is the FID score really in this case?
- How were the 10 GANs tuned?
* Provide additional feedback with the aim to improve the paper. Make it clear that these points are here to help, and not necessarily part of your decision assessment.
- It is mentioned on p.4 that "clamping is potentially a harmful irreversible operation" but that harmful results were not observed. As I was reading that I was wondering how these results would actually look like.
- On p. 6 it is mentioned that the number of images for 2 categories are reported in another paper. I think it'd take less space to actually report the number of images here.
- On p.7 it is mentioned that the number of instances is preserved, however it should be made clear that it's is perserved in some (or most if that is what was observed) of the examples.


This work views neural networks as data generating systems and applies anomalous pattern detection techniques on that data in order to detect when a network is processing a group of anomalous inputs. Detecting anomalies is a critical component for multiple machine learning problems including detecting the presence of adversarial noise added to inputs. More broadly, this work is a step towards giving neural networks the ability to detect groups of out-of-distribution samples. This work introduces "Subset Scanning" methods from the anomalous pattern detection domain to the task of detecting anomalous inputs to neural networks. Subset Scanning allows us to answer the question: "Which subset of inputs have largerthan-expected activations at which subset of nodes?" Framing the adversarial detection problem this way allows us to identify systematic patterns in the activation space that span multiple adversarially noised images. Such images are "weird together". Leveraging this common anomalous pattern we show increased detection power as the proportion of noised images increases in a test set. Detection power and accuracy results are provided for targeted adversarial noise added to CIFAR-10 images on a 20-layer ResNet using the Basic Iterative Method attack.

The paper is the first paper, in my knowledge, that introduces the problem of identifying anomalous (or corrupted) subset of data input to a neural network. The corrupted inputs are identified vis-a-vis a set of “clean” background set (e.g., the training/ validation data set). Experimental evaluation is performed on the problem of identifying the subset of noisy CIFAR-10 images created by adding targeted adversarial perturbations.
To achieve this, the problem is posed as that of subset detection and subset scanning approaches are adapted for the same. The activation of a node under consideration is converted to a p-value (how extreme the activation is, for that input data, w.r.t. the background set).
A goodness of fit is then defined for any subset using the Berk-Jones test statistic which is used to create an NPSS scoring function to be maximized over the set of all subsets of the product set – (set of nodes x set of input images). The authors suggest that this combinatorial problem can be addressed efficiently as NPSS has been shown before to have the linear-time subset scanning (LTSS) property (Neill 2012).
However, since the search is now over a product subspace and the LTSS property holds for individual subspaces (selection of activation nodes or input images while the other is held fixed), the LTSS property doesn’t trivially extend to the product space. The authors suggest an algorithm which alternatively iterates between the two LTSS steps. Since the above algorithm is not guaranteed to obtain the global optimum, multiple random restarts are suggested. This is the weakest part of the paper – neither is a formal proof of convergence provided, nor is the efficiency of the propped algorithm demonstrated empirically. Also, the performance of the algorithm (accuracy) can only be weakly used to judge the goodness of the local optimum obtained as it confounds the power of NPSS with the gap between the local and global optimum.
Contributions
In summary, the main contribution of the paper is the introduction of the problem of the identification of anomalous subsets of adversarially corrupted examples on deep neural networks and the first approach to address the problem via a NPSS scoring function that (partially) satisfies the LTSS property suggesting the existence of efficient (and exact?) algorithms for solving the problem. The work has incremental novelty as it seems to be largely based on prior art (not on DNNs). Initial results look promising.
Negatives
However, I have the following concerns:
- (Clarity) Mathematical formulation for the NPSS score function and the optimization problem to be solved should be clearly stated (and not inside paragraphs of text).
- The proposed algorithm is ad hoc - an obvious extension of the standard algorithm that exploits the LTSS property to the product of two set spaces. The algorithm may not find a good optimum. No proof, analysis or study of its properties – optimality gap, convergence, and, efficiency is presented.
- At deployment, the attacks may be non-targeted or the target may be different for different corrupted inputs. This will dilute the signature as the set of nodes where the ‘extreme values’ appear may not be the same across the corrupted examples. It is not clear if it is the case if the target is the same. In this case, subset scanning may not work. This assumption should’ve been directly tested.
- ‘Berk-Jones can be interpreted as the log-likelihood ratio for testing whether p-values are uniformly distributed on [0,1] vs. … alternate distribution’. Why should this be the test? P-values on clean, background data can be directly obtained and whatever that distribution can be empirically tested or uniformity can be tested against after a whitening transformation?
- Comparison with reasonable ‘baselines’ – for example, with Feinman et al. 2017? Comparison with current state of the art defenses on CIFAR10?
- Why is the BIM attack (Kurakin et al. 2016b) used? There are more powerful attacks available now?
- The NPSS scoring function is supposed to maximize over significance values \alpha. What range was used and how was it decided? In the discussion on Detection Power reported in Table 1, it is suggested that different \alpha_max will trade off precision vs recall. This has not been experimented with.


Undirected neural sequence models such as BERT (Devlin et al., 2019) have received renewed interest due to their success on discriminative natural language understanding tasks such as question-answering and natural language inference. The problem of generating sequences directly from these models has received relatively little attention, in part because generating from such models departs significantly from the conventional approach of monotonic generation in directed sequence models. We investigate this problem by first proposing a generalized model of sequence generation that unifies decoding in directed and undirected models. The proposed framework models the process of generation rather than a resulting sequence, and under this framework, we derive various neural sequence models as special cases, such as autoregressive, semi-autoregressive, and refinement-based non-autoregressive models. This unification enables us to adapt decoding algorithms originally developed for directed sequence models to undirected models. We demonstrate this by evaluating various decoding strategies for a cross-lingual masked translation model (Lample and Conneau, 2019). Our experiments show that generation from undirected sequence models, under our framework, is competitive with the state of the art on WMT'14 English-German translation. We also demonstrate that the proposed approach enables constant-time translation with similar performance to linear-time translation from the same model by rescoring hypotheses with an autoregressive model.

This paper proposes a generalized framework for sequence generation that can be applied to both directed and undirected sequence models. The framework generates the final label sequence through generating a sequence of steps, where each step generates a coordinate sequence and an intermediate label sequence. This procedure is probabilistically modeled by length prediction, coordinate selection and symbol replacement. For inference, instead of the intractable naive approach based on Gibbs sampling to marginalize out all generation paths, the paper proposes a heuristic approach using length-conditioned beam search to generate the most likely final sequence. With the proposed framework, the paper shows that masked language models like BERT, even though they are undirected sequence models, can be used for sequence generation, which obtains close performance to the traditional left-to-right autoregressive models on the task of machine translation.
Overall the paper has significant contributions in the following aspects:
1. It enables undirected sequence models, like BERT, to perform decoding or sequence generation directly, instead of just serving as model pre-training.
2. The proposed framework unifies directed and undirected sequence models decoding, and it can represent a few existing sequence model decoding as special cases.
3. The coordinate sequence selection function in the framework can be dependent on the intermediate label sequence. A few simple selection approaches proposed in the paper are shown to be effective. It could be further extended.
4. The analysis of the coordinate selection order is interesting and helpful for understanding the algorithm.
5. The experiment results for decoding masked language models on machine translation are promising. It also provides the comparison to recent related work on non-autoregressive approaches.
The presentation of the paper is also clear. I am leaning towards accepting the paper.
However, there are some weaknesses:
1. It should be analyzed more why different coordinate selection approaches perform differently in linear-time decoding vs. constant-time decoding. Even in constant-time decoding, the conclusion varies in different decoding setting, easy-first is the worst for the L->1 case, but the best for the L/T case, why is that?
2. What is the motivation for "hard-first"?
3. The setting of "least2most" with L->1 is similar to Ghazvininejad et al. 2019. But Table 4 in the appendix shows the result in this paper is still worse (21.98 vs. 24.61, when both systems use 10 iterations without AR). Also, the gap from the AR baseline is larger than that in Ghazvininejad et al. 2019. Given the two systems are considered similar, it should be explained in the paper the possible reasons for these discrepancies in results.
Additional minor comments for improving the paper:
1. In the introduction, it mentions the baseline AR is (Vaswani et al. 2017), while in the experimental settings, it mentions the baseline AR is (Bahdanau et al. 2015). Please clarify which one is used.
2. In Table 1, how does T = 2L work for the "Uniform" case while the target sequence length is only T, since it is mentioned the positions are sampled without replacement. Similarly, how does T = 2L work for the "Left2Right" case? Is it just always choosing the last position when L < t <= 2L? In these two cases, it seems T > L is not needed.
3. In Table 3, the header for the 2nd column should be o_t, as defined in Section 4 - "Decoding scenarios". What is the actual value of K and K'' for the constant-time machine translation experiments in the paper?
4. "Rescoring adds minimal overhead as it is run in parallel" - it still needs to run left-to-right in sequence since it is auto-regressive. Please clarify what it means by "in parallel" here.
5. What is the range and average for the target sentence length? How is T = 20 for constant-time decoding compared to linear-time decoding in terms of speed?


The knowledge that humans hold about a problem often extends far beyond a set of training data and output labels. While the success of deep learning mostly relies on supervised training, important properties cannot be inferred efficiently from end-to-end annotations alone, for example causal relations or domain-specific invariances. We present a general technique to supplement supervised training with prior knowledge expressed as relations between training instances. We illustrate the method on the task of visual question answering (VQA) to exploit various auxiliary annotations, including relations of equivalence and of logical entailment between questions. Existing methods to use these annotations, including auxiliary losses and data augmentation, cannot guarantee the strict inclusion of these relations into the model since they require a careful balancing against the end-to-end objective. Our method uses these relations to shape the embedding space of the model, and treats them as strict constraints on its learned representations. In the context of VQA, this approach brings significant improvements in accuracy and robustness, in particular over the common practice of incorporating the constraints as a soft regularizer. We also show that incorporating this type of prior knowledge with our method brings consistent improvements, independently from the amount of supervised data used. It demonstrates the value of an additional training signal that is otherwise difficult to extract from end-to-end annotations alone.

The paper argues for encoding external knowledge in the (linguistic) embedding layer of a multimodal neural network, as a set of hard constraints. The domain that the method is applied to is VQA, with various relations on the questions translated into hard constraints on the embedding space. A technique which involves distillation is used to satisfy those constraints during learning.
The question of how to encode external knowledge in neural networks is a crucial one, and the limitations of end-to-end learning with supervised data is well-made. Overall I feel that this is a potentially interesting paper, addressing an important question in a novel way, but I found the current version a highly-frustrating read (and I read the paper carefully a number of times); in fact, so frustrating that it is hard for me to recommend acceptance in its current form. More detailed comments below.
Major comments
--
The main problem I have with the paper lies with the first part of section 3, which is a key section describing the main method by which the constraints are satisfied during learning. This is very confusing. The need for the two-step procedure, in particular, and the importance of distillation needs much more explanation, and not relegated to the Appendix (which reviewers are not required to read - see call for papers). I'm not suggesting that the whole of the appendix needs moving to the body of the paper, but I would suggest perhaps 1/2 a page.
A related comment is the use of the distillation technique. This looks crucial, but I don't believe distillation is mentioned at all until the end of the related work section, and even there it comes as a bit of a surprise since there's no mention anywhere of this technique in the introduction.
I would say a little more about the distinction between the embedding space and parameter space, since you say that the external knowledge is encoded in the former and not the latter, and this is important to the overall method. Since embeddings are typically learned (or at least fine-tuned) it's not clear where the boundary is here. Another comment is that embedding space in this paper means the linguistic embedding space. Since this is ICLR and not, eg, ACL, I would make clear what you mean by embedding space.
I don't understand the diagram in Fig. 3 of the architecture, nor the explanation. What's an operation here? Is it *, or *6? I don't get why 3 is embedded by itself in the diagram, and then combined with the remainder using the MLP. Why not just run the RNN over the sequence?
Why are the training instances {3,+1...} and {4,*2,...} equivalent. I stared at this a while, and still have no idea. Also, how are these "known to be equivalent" - what's the procedure?
Minor comments including typos etc.
--
The paper has the potential to be really nicely written and well-presented. Currently it reads like it was thrown together just before the deadline (which only adds to the overall frustration as a reader).
In fig. 1 the second equivalent question example is interesting, since strictly speaking "box" and "rectangular container" are not synonyms (e.g. boxes can be round). Since strict synonymy is hard to find, does that matter? (I realise the dataset already exists and was presented elsewhere, but this might be worth a footnote).
missing (additional) right bracket after Herbert (2016)
Not sure footnote 1 needs to be a footnote. It's already been said, I think, but if it does need repeating it probably deserves to be in the body of the text.
between pairs questions
see Fig.3 -> figure 2?
see Fig.1 -> Tab. 1? (on p.5)
footnote 1 missing a right bracket
usually involve -> involves
+9]) - extraneous bracket
Fig. 4.1 -> Fig. 3? (p.6)
p.7 wastes a lot of space. In order to bring some of the appendix into the main body, I would do away with the very large bulleted list. (I don't mean lose the content - just present it more efficiently)
Remember than
Finally in Fig. 4.2 - some other figure
due of the long chains


For distributions p and q with different supports, the divergence D(p||q) may not exist. We define a spread divergenceD(p||q) on modified p and q and describe sufficient conditions for the existence of such a divergence. We demonstrate how to maximize the discriminatory power of a given divergence by parameterizing and learning the spread. We also give examples of using a spread divergence to train and improve implicit generative models, including linear models (Independent Components Analysis) and non-linear models (Deep Generative Networks).

The paper introduced a way to modify densities such that their support agrees and that the Kullback-Leibler divergence can be computed without diverging. Proof of concept of using the spread KL divergence to ICA and Deep Generative Models ($\delta$-VAE) are reported based on the study of spread MLE.
Comments:
In Sec 1, mention that f should be strictly convex at 1. Also mention
Jensen-Shannon divergence, a KL symmetrization, which is always finite
and used in GAN analysis.
In Sec 2, you can also choose to dilute the densities with a mixture:
(1-\epsilon)p+\epsilon noise.
Explain why spread is better than that? Does spreading introduce
spurious modes?, does it change distribution sufficiency?
(Fisher-Neymann thm)
In Formula 4, there is an error: missing denominator of \sigma^2. See
Appendix D too.
In footnote 4, page 8, missing a 1/2 factor in from of TV (that is upper
bounded by 1 and not 2)
KL is relative entropy= cross-entropy minus entropy. What about spread KL?
In general, what statistical properties are kept by using the spread?
(or its convolution subcase?)
Is spreading a trick that introduces a hyperparameter that can then be
optimized for retaining discriminatory power, or is there
some deeper statistical theory to motivate it. I think spread MLE should
be further explored and detailed to other scenarii.
Spreading can be done with convolution and in general by Eq.3:
Then what is the theoretical interpretation of doing non-convolutional
spreading?
A drawback is that optimization on the spread noise hyperparameter is
necessary (Fig 3b is indeed much better than Fig 3a).
Is there any first principles that can guide this optimization rather
than black-box optimization?
Overall, it is a nice work but further statistical guiding principles
or/and new ML applications of spread divergences/MLE will strengthen the
work.
The connection, if any, with Jensen-Shannon divergence shall be stated
and explored.
Minor comments:
In the abstract, state KL divergence instead of divergence because
Jensen-Shannon divergence exists always.
Typos:
p. 6 boumd->bound
Bibliography : Cramir->Cramer, and various upper cases missing (eg.
wasserstein ->Wasserstein)


We propose a model that is able to perform unsupervised physical parameter estimation of systems from video, where the differential equations governing the scene dynamics are known, but labeled states or objects are not available. Existing physical scene understanding methods require either object state supervision, or do not integrate with differentiable physics to learn interpretable system parameters and states. We address this problem through a physics-as-inverse-graphics approach that brings together vision-as-inverse-graphics and differentiable physics engines, enabling objects and explicit state and velocity representations to be discovered. This framework allows us to perform long term extrapolative video prediction, as well as vision-based model-predictive control. Our approach significantly outperforms related unsupervised methods in long-term future frame prediction of systems with interacting objects (such as ball-spring or 3-body gravitational systems), due to its ability to build dynamics into the model as an inductive bias. We further show the value of this tight vision-physics integration by demonstrating data-efficient learning of vision-actuated model-based control for a pendulum system. We also show that the controller's interpretability provides unique capabilities in goal-driven control and physical reasoning for zero-data adaptation.

This paper presents an approach for unsupervised estimation of physical parameters from video, using the physics as inverse graphics approach. The approach uses a feedforward encoder for localising object positions from which a velocity is estimated. These are fed as inputs to an (action-conditioned) physics simulator which generates future predictions of object positions. This simulator has knowledge of the system dynamics apriori, needing estimation of a few physical parameters such as gravity and spring constants. The outputs of this simulator is fed to a co-ordinate consistent decoder, a neural network that uses a Spatial Transformer to render the corresponding output image. The whole system is trained end-to-end on videos of dynamical systems, in an unsupervised manner. Results on two and three body interaction settings and an MNIST digit motion dataset show promising performance. The system is able to recover the underlying physical parameters accurately while also making consistent long-term ex predictions. Additionally, the model is used for visual MPC on a simulated cartpole task where it outperforms state of the art model-based and model-free RL baselines.
This paper is well written and clearly motivated, albeit a bit incremental in its approach. Many of the building blocks have been explored in prior work, with the major component being the co-ordinate consistent decoder. The experiments are visually simplistic and it is not obvious if the system will scale to more complex settings. A few comments:
1. A major limitation of the approach is the assumption that the equations governing the system are known. This makes it harder to generalise the system to novel tasks and more complicated settings with contact and object interactions. A potential way to overcome this could be to use ideas from prior work such as Interaction Networks where the dynamics are modelled as unary and binary interactions. While such dynamics models are black-box and not as interpretable compared to the current approach, they can easily generalise to novel tasks. Additionally, using positions and velocities as the latent state representation together with IN style transition models can be a sensible middle ground.
2. It would be great if there is an additional ablation experiment where the known equations of motion are replaced with a black-box neural network (while still retaining the position and velocity representation). This can quantify the effect of known dynamics and make the contributions of the paper (with regards to the decoder) more clear.
3. An alternative way of generating consistent object positions from the encoder is to compute a mask-weighted average of the image co-ordinates. This can be a nice way of adding additional structure to the networks that can regularise training.
4. The approach uses a 3-layer MLP for generating velocity estimates — could this not be done via finite differencing? (e.g. higher-order backward differencing)
5. Both the content and mask vectors are learnable but fixed for the entire task — i.e. it is not a function of the input image. This makes the approach not applicable to novel objects or even objects with minor color changes.
6. It is not clear how the translations, rotations and scale parameters for the Spatial Transformer are estimated. I presume that the positions and orientations predicted by either the encoder or physics simulator are directly used. This needs to be clarified in the main paper.
7. The paper mentions that the background masks are known when localising the objects via the encoder. If this is the case, the localisation problem becomes somewhat trivial. This should be clarified.
8. There needs to be a clear discussion on the limitations of the current approach — it does not scale to novel objects, needs to know the number of objects apriori and has not been shown to estimate object properties such as mass, friction etc.
Overall, the approach presents promising initial results towards an unsupervised method for modelling dynamical systems from video. There are several limitations that need to be made explicit and some additional experiments on more complicated systems and a few ablation studies can significantly improve the strength of the paper. I would suggest a borderline accept.
Typos:
1. Section 4.1, Setup: 5 values of (K, t_pred, t_ext) are given, need only 4
2. “K” is not introduced till the results section
Additional citation:
The following paper on learning physically consistent position and velocity representations for dynamical systems (and for use in visual MPC settings) should be cited:
Jonschkowski, Rico, et al. "Pves: Position-velocity encoders for unsupervised learning of structured state representations." arXiv preprint arXiv:1705.09805 (2017).


We propose and address a novel few-shot RL problem, where a task is characterized by a subtask graph which describes a set of subtasks and their dependencies that are unknown to the agent. The agent needs to quickly adapt to the task over few episodes during adaptation phase to maximize the return in the test phase. Instead of directly learning a meta-policy, we develop a Meta-learner with Subtask Graph Inference (MSGI), which infers the latent parameter of the task by interacting with the environment and maximizes the return given the latent parameter. To facilitate learning, we adopt an intrinsic reward inspired by upper confidence bound (UCB) that encourages efficient exploration. Our experiment results on two grid-world domains and StarCraft II environments show that the proposed method is able to accurately infer the latent task parameter, and to adapt more efficiently than existing meta RL and hierarchical RL methods.

This paper proposes a new meta-reinforcement learning algorithm, MSGI, which focuses on the problem of adapting to unseen hierarchical tasks through interaction with the environment where the external reward is sparse. The authors make use of subtask graph inference to infer the latent subtask representation of a task through interacting with the environment using an adaptation policy and then optimize the adaptation policy based on the inferred latent subtask structure. Each task in the paper is represented as a tuple of subtask precondition and subtask reward, which are inferred via logic induction and MLE of Gaussians respectively. At meta-test time, MSGI rollouts a subtask graph execution (SGE) policy based on the graph inferred from the interactions between the environment and the adaptation policy. The authors also propose a UCB-inspired intrinsic reward to encourage exploration when optimizing the adaptation policy. Experiments are conducted on two grid-world domains as well as StarCraft II.
Overall, this paper is mainly an extension of the prior work [1], which uses a subtask graph for tackling hierarchical RL problems. This work builds upon [1] by extending to meta-learning domains and studying generalization to new hierarchical tasks. While the contribution seems a bit incremental and the experimental setting is a bit unclear and limited to low-dimensional state space, the inference of task-specific subtask graphs based on past experiences and the proposal of a UCB-inspired reward shed some interesting insights on how to approach meta-hierarchical RL where long-horizon tasks and sparse rewards have been major challenges. Given some clarification on the experimental setup and additional results on more challenging domains in the author's response, I would be willing to improve my score.
Regarding the experimental setup, the set of subtasks is a Cartesian product of the set of primitive actions and a set of all types of interactive objects in the domain, while the state is represented as a binary 3-dimensional tensor indicating the position of each type of objects. Such a setup seems a bit contrived and is limited to low-dimensional state space and discrete action space, which makes me doubt its scalability to high-dimensional continuous control tasks. It would be interesting to see how/if MSGI can perform in widely used meta-RL benchmarks in Mujoco. I also wonder how MSGI can be compared to newly proposed context-based meta-RL methods such as PEARL.
As for the results, the authors don't provide an ablation study on the UCB exploration bonus though they claim they would show it in the paper. Moreover, the result of GRProp+Oracle is also missing in the comparison. I also don't understand why MSGI-Meta and RL2 would overfit in the SC2LE case and are unable to adapt to new tasks. Is that a limitation of the method? The authors also introduce MSGI-GRProp in this setting, which is never discussed before, and claim that MSGI-GRProp can successfully generalize to new tasks. It seems that the authors don't use a meta-RL agent in order to get this domain to work. I believe more discussion on this part is needed.
[1] Sungryull Sohn, Junhyuk Oh, and Honglak Lee. Hierarchical reinforcement learning for zero-shot
generalization with subtask dependencies. In NeurIPS, pp. 7156–7166, 2018.


This paper investigates the intriguing question of whether we can create learning algorithms that automatically generate training data, learning environments, and curricula in order to help AI agents rapidly learn. We show that such algorithms are possible via Generative Teaching Networks (GTNs), a general approach that is, in theory, applicable to supervised, unsupervised, and reinforcement learning, although our experiments only focus on the supervised case. GTNs are deep neural networks that generate data and/or training environments that a learner (e.g. a freshly initialized neural network) trains on for a few SGD steps before being tested on a target task. We then differentiate through the entire learning process via meta-gradients to update the GTN parameters to improve performance on the target task. GTNs have the beneficial property that they can theoretically generate any type of data or training environment, making their potential impact large. This paper introduces GTNs, discusses their potential, and showcases that they can substantially accelerate learning. We also demonstrate a practical and exciting application of GTNs: accelerating the evaluation of candidate architectures for neural architecture search (NAS), which is rate-limited by such evaluations, enabling massive speed-ups in NAS. GTN-NAS improves the NAS state of the art, finding higher performing architectures when controlling for the search proposal mechanism. GTN-NAS also is competitive with the overall state of the art approaches, which achieve top performance while using orders of magnitude less computation than typical NAS methods. Speculating forward, GTNs may represent a first step toward the ambitious goal of algorithms that generate their own training data and, in doing so, open a variety of interesting new research questions and directions.

This paper proposes a meta-learning algorithm Generative Teaching Networks (GTN) to generate fake training data for models to learn more accurate models. In the inner loop, a generator produces training data and the learner takes gradient steps on this data. In the outer loop, the parameters of the generator are updated by evaluating the learner on real data and differentiating through the gradient steps of the inner loop. The main claim is this method is shown to give improvements in performance on supervised learning for MNIST and CIFAR10. They also suggest weight normalization patches up instability issues with meta-learning and evaluate this in the supervised learning setting, and curriculum learning for GTNs.
To me, the main claim is very surprising and counter-intuitive - it is not clear where the extra juice is coming from, as the algorithm does not assume any extra information. The actual results I believe do not bear out this claim because the actual results on MNIST and CIFAR10 are significantly below state of the art. On MNIST, GTN achieves about 98% accuracy and the baseline “Real Data” achieves  <97% accuracy, while the state of the art is about 99.7% and well-tuned convnets without any pre-processing or fancy extras achieve about 99% according to Yann LeCunn’s website. The disparity on CIFAR seems to be less egregious but the state of the art stands at 99% while the best GTN model (without cutout) achieves about 96.2% which matches good convnets and is slightly worse than neural architecture search according to https://paperswithcode.com/sota/image-classification-on-cifar-10.
This does not negate the potential of GTNs which I feel are an interesting approach, but I believe the paper should be more straightforward with the presentation of these results. The current results basically  show that GTNs improve the performance of learners with bad hyper-parameters. On problems that are not as well-studied as MNIST or CIFAR10 this could still be very valuable (as we do not know what performance is good or bad in advance). Based on the results, GTN does seem to be a significant step forward in synthetic data generation for learning compared to prior work (Zhang 2018, Luo 2018).
The paper proposes two other contributions: using weight normalization for meta-learning and curriculum learning for GTNs. Weight normalization is shown to stabilize GTNs on MNIST. I think the paper oversteps in the relevant method section, hypothesizing it may stabilize meta-learning more broadly. The paper should present a wider set of experiments to make this claim convincing. But the point for GTNs on MNIST nevertheless stands. For curriculum learning: the description of the method is done across section 2 and section 3.2 and does not really describe it completely. How exactly are the samples chosen in GTN - All Shuffled? How does GTN - Full Curriculum and Shuffled Batch parametrize the order of the samples so that it can be learned? I suggest that this information is all included as a subsection in the method (section 2). The results seem to show the learned curriculum is superior to no curriculum.
At a high level it would be very surprising to me if the way forward for better discriminative models was to learn good generative models and use them again for training discriminative models, simply because discriminative models have proved thus far significantly easier to train. If this work does eventually show this result, it would be a very interesting result. At the moment, I believe it does not, but I would be happy to change my mind if the authors provide convincing evidence. Alternatively, I feel that the paper could be a valuable contribution to the community if the writing is toned down to focus on the contributions, presents the results comparing to well-tuned hyperparameters and not over-claim.
More comments:
What is the outer loop loss function? Is it assumed to be the same as the inner one (but using real data instead of training data)? I think this should be made explicit in the method section.
There are some additional experiments in other settings such as RL and unsupervised learning. Both seem like quite interesting directions but seem like preliminary experiments that don’t work convincingly yet. The RL experiment shows that using GTN does not change performance much. There is a claim about optimizing randomly initialized networks at each step, but the baseline which uses randomly initialized networks at each step with A2C is missing. The GAN experiments shows the GAN loss makes GTN realistic (as expected) but there are no quantitative results on mode collapse. (Another interesting experiment would be to show how adding a GAN loss for generating data affects the test performance of the method.) Perhaps it would benefit the paper to narrow in on supervised learning? Given that these final experiments are not polished, the claim in the abstract that the method is “a general approach that is applicable to supervised, unsupervised, and reinforcement learning” seems to be over-claiming. I understand it can be applicable but the paper has not really done the work to show this outside the supervised learning setting.
Minor comments:
Pg. 4: comperable -> comparable


Modelling statistical relationships beyond the conditional mean is crucial in many settings. Conditional density estimation (CDE) aims to learn the full conditional probability density from data. Though highly expressive, neural network based CDE models can suffer from severe over-fitting when trained with the maximum likelihood objective. Due to the inherent structure of such models, classical regularization approaches in the parameter space are rendered ineffective. To address this issue, we develop a model-agnostic noise regularization method for CDE that adds random perturbations to the data during training. We demonstrate that the proposed approach corresponds to a smoothness regularization and prove its asymptotic consistency. In our experiments, noise regularization significantly and consistently outperforms other regularization methods across seven data sets and three CDE models. The effectiveness of noise regularization makes neural network based CDE the preferable method over previous non-and semi-parametric approaches, even when training data is scarce.

The paper considers the problem of parametric conditional density estimation, i.e. given a set of points {(x_n, y_n)} drawn from a distribution $p(x,y)$, the task is to estimate the conditional distribution p(x|y). The paper considers parametric estimation where in given a parametrized family of distributions f_{theta} we wish to minimize the likelihood of seeing the given data over theta. The parametric family in a lot of applications consists of highly expressive families like neural networks, which leads to the issue of overfitting in small data regimes. This has been tackled via regularization over the parameter space which might be hard to interpret as the associated inductive bias is not well understood and depends on the parametric family under consideration. On the other hand the paper proposes to add explicit noise in the examples used during training, i.e. irrespective of the optimization procedure (which could be mini-bath sgd) the paper proposes to draw examples from the data set, explicitly add noise onto the examples and create a proxy objective over the augmented data set.
The paper establishes two theoretical results. First is a simple taylor approximation based analysis to highlight the effect of the variance of noise. The conclusion is that higher variance penalizes the high curvature areas of the resultant density and hence this kind of noise addition could be seen as making resulting density smoother. The second contribution is to show that this procedure is asymptotically consistent, i.e. as n goes to infinity and the number of augmented data points go to infinity, the resulting density converges to the target density. This of course requires the noise variance to follow a decreasing schedule to 0.
The main merit of the idea is in its agnostic nature, as it can be applied to any parametric family and the experiements show that it seems to uniform improvement across models,  The basic idea proposed by the error, has existed in the space of deep learning based methods forever. This is the same idea behind image augmentation which forms a crucial part of training supervised models in vision. The authors claim that this idea is novel in the space of parametric density estimation, however I do not know enough about the area to verify the claim. It would surprise that this very natural idea has not been tried before.
I have gone through the theoretical derivations in the paper and they look sound to me. However the results are all asymptotic in nature without establishing explicit rates which is a little bit of disappointment. Since I am not completely familiar in nature, but I guess such asymptotic consistency might be achievable using other forms of regularization under suitable assumptions. In that light, the theoretical contributions while being sound did not lend much intuition about why such a method might outperform others. Intuition does arise from the derivation for the effect of noise on the objective which helps understand the nature of the noise, but one wonders if similar intuitions could be derived for other forms of regularization as well. It would be great to see this derivation being extended to some concrete scenarios under well understood parametric families and seeing the effect explicitly.
Regarding the experiments - The experiments definitely look promising as the improvments seem uniformly good across the cases considered. I am not an expert however in this setting so it is hard for me to judge the quality and significance of the benchmarks. The experiment methodology nevertheless looks sound.


We design a new algorithm for batch active learning with deep neural network models. Our algorithm, Batch Active learning by Diverse Gradient Embeddings (BADGE), samples groups of points that are disparate and high-magnitude when represented in a hallucinated gradient space, a strategy designed to incorporate both predictive uncertainty and sample diversity into every selected batch. Crucially, BADGE trades off between diversity and uncertainty without requiring any hand-tuned hyperparameters. While other approaches sometimes succeed for particular batch sizes or architectures, BADGE consistently performs as well or better, making it a useful option for real world active learning problems.

This paper examines the problem of warm-starting the training of neural networks. In particular, a generalization gap arises when the network is trained on the full training set from the start versus being warm-started, where the network is initially (partially) trained on a subset of the training set, then switched to the full training set. This problem is practical, as it is often preferable to train online while data is collected to make up-to-date predictions for tasks (such as in online advertising or recommendation systems), but it has been found that retraining is necessary in order to obtain optimal performance. The paper also mentions active learning, domain shift, and transfer learning as two other relevant and important problems.
The paper attempts to investigate this phenomena from a few different avenues: (1) simple experiments segmenting the training set into two different subsets, then training to completion or partial training on the first subset before switching to training on the full set; (2) looking at the gradients of warm-started models; (3) adding regularization; (4) warm-starting all layers, then training only last layer; (5) perturbing the warm-started parameters.
Strengths:
I believe very strongly in the practical impact of the problems presented in this paper. These indeed are challenging problems that are relevant to industry that have not been given sufficient attention in the academic literature. I appreciate the initial experimentation on this subject, and the clear demonstration of the problem through simple experiments. The paper is also well-written.
Weaknesses:
Some questions I had include:
- Why is the Pearson correlation between parameters of the neural network a good way of measuring the correlation to their initialization?
- Why is it surprising that the magnitude of the gradients of the "new" data is higher than at a random initialization?
- Why does this phenomena occur even though the data is sampled from the same distribution?
- Does this work have any relationship with work on generalization such as:
[1] Recht, Benjamin, et al. "Do CIFAR-10 classifiers generalize to CIFAR-10?." arXiv preprint arXiv:1806.00451 (2018).
[2] Recht, Benjamin, et al. "Do ImageNet Classifiers Generalize to ImageNet?." arXiv preprint arXiv:1902.10811 (2019).
etc.
Although I like the topic of this paper, the investigation seems too preliminary at this point. There is no clear hypothesis towards answering the problems proposed in the paper. There is also no analysis, which places the burden on the numerical experiments to demonstrate something interesting, and the experiments seem sparse and small-scale. For these reasons, I am inclined to reject this paper at this time, but I strongly encourage further exploration into the topic.
Some potential questions or directions could include:
1. What if only a single epoch of training is used on 50% of the data? Does the gap appear in that setting? I ask because one would expect that a single epoch of training on 50% of the data, then training on new data would be equivalent to training on the full dataset from the start.
2. How does this gap change with respect to the (relative) amount of new data introduced into the problem? For example, if one were to only add a single datapoint to the training set, would one still observe this behavior? Could one potentially add data more incrementally (rather than half of the training set) and potentially mitigate this drastic change in the problem?
3. There are optimization algorithms specifically designed for stochastic optimization (with a fixed distribution) versus for online optimization (online gradient, Adagrad). Is the online optimization framework perhaps more "realistic" than the stochastic optimization framework in these streaming/warm-starting settings?


This paper introduces the Behaviour Suite for Reinforcement Learning, or bsuite for short. bsuite is a collection of carefully-designed experiments that investigate core capabilities of reinforcement learning (RL) agents with two objectives. First, to collect clear, informative and scalable problems that capture key issues in the design of general and efficient learning algorithms. Second, to study agent behaviour through their performance on these shared benchmarks. To complement this effort, we open source github.com/anon/bsuite, which automates evaluation and analysis of any agent on bsuite. This library facilitates reproducible and accessible research on the core issues in RL, and ultimately the design of superior learning algorithms. Our code is Python, and easy to use within existing projects. We include examples with OpenAI Baselines, Dopamine as well as new reference implementations. Going forward, we hope to incorporate more excellent experiments from the research community, and commit to a periodic review of bsuite from a committee of prominent researchers.

This paper presents the « Behavior Suite for Reinforcement Learning » (bsuite), which is a set of RL tasks (called « experiments ») meant to evaluate an algorithm’s ability to solve various key challenges in RL. Importantly, these experiments are designed to run fast enough that one can benchmark a new algorithm within a reasonable amount of time (and money). They can thus be seen as a « test suite » for RL, limited to small toy problems but very useful to efficiently debug RL algorithms and get an overview of some of their key properties. The paper describes the motivation behind bsuite, shows detailed results from some classical RL algorithms on a couple of experiments, and gives a high-level overview of how the code is structured.
I really believe such a suite of RL tasks can indeed be extremely useful to RL researchers developing new algorithms, and as a result I would like to encourage this initiative and see it published at ICLR to help it gain additional traction within the RL community.
The paper is easy to read, motivates well the reasons behind bsuite, and shows some convincing examples. However, in my opinion there remain a few important issues with this submission:
1.	There is no « related work » section to position bsuite within the landscape of RL benchmarks (ex: DMLab, ALE / MinAtar, MuJoCo tasks, etc.). I believe it is important to add one.
2.	The current collection of experiments appears to be quite limited. The authors acknowledge the lack of hierarchical RL, but what about other aspects like continuous control, parameterized actions, multi-agent, state representation learning, continual learning, transfer learning, imitation learning / inverse RL, self-play, etc? It is unclear to me whether the goal is to grow bsuite in all these directions (and more) over time, or if there is some kind of « boundary » the authors have in mind regarding the scope of bsuite. Regardless, the fact is that in its current form, bsuite appears to be suited only to a limited subset of current RL research.
3.	I wish an anonymized version of the code had been provided, so that reviewers could test it. In particular I wonder (a) if it is easy to setup and run under Windows, and (b) if it is straighforward to plug a bsuite experiment within an algorithm based on the popular OpenAI gym API (I think the latter is true from what is said at the end of Section 4, but I would have appreciated being able to try it out myself).
Additional minor remarks:
•	I noticed two anoymity-related issues with the provided links: (1) the Google Colab notebook revealed to me the name of its author when clicking the « Open in Playground » link to be able to run it, and (2) the bsuite-tutorial link asks for permission, which might let the authors access reviewer info. I would not hold it against the authors though as I believe these are genuine mistakes and they did their best to preserve anonymity.
•	t > 2 in Section 2.1 should probably be t >= 2
•	In FIg. 2b the label for the y axis seems incorrect since good results are near 0
•	Please explain what is the dashed grey line in Fig. 4b
•	I was unable to understand the last 2 sentences of Section 4
•	Sections C.2, D.2 and E.2 all have the same plots
•	A few typos: incomplete sentence near bottom of p.3 (« the internal workings… »), « These assessment », « expeirments », « recurrant », « length ? 1 », « together with an analysis parses this data », « anonimize », « bsuite environments by implementing », « even if require »
Review update: the authors have addressed my concerns, and I look forward to using bsuite in my research => review score increased to "Accept"


Deep Neural Networks (DNNs) often rely on very large datasets for training. Given the large size of such datasets, it is conceivable that they contain certain samples that either do not contribute or negatively impact the DNN's optimization. Modifying the training distribution in a way that excludes such samples could provide an effective solution to both improve performance and reduce training time. In this paper, we propose to scale up ensemble Active Learning methods to perform acquisition at a large scale (10k to 500k samples at a time). We do this with ensembles of hundreds of models, obtained at a minimal computational cost by reusing intermediate training checkpoints. This allows us to automatically and efficiently perform a training data subset search for large labeled datasets. We observe that our approach obtains favorable subsets of training data, which can be used to train more accurate DNNs than training with the entire dataset. We perform an extensive experimental study of this phenomenon on three image classification benchmarks (CIFAR-10, CIFAR-100 and ImageNet), analyzing the impact of initialization schemes, acquisition functions and ensemble configurations. We demonstrate that data subsets identified with a lightweight ResNet-18 ensemble remain effective when used to train deep models like ResNet-101 and DenseNet-121. Our results provide strong empirical evidence that optimizing the training data distribution can provide significant benefits on large scale vision tasks.

Review Summary
--------------
Overall, I'm not quite convinced this method would be worth the trouble to implement. On the more realistic benchmarks, they need to keep ~80% of the total dataset size and the claimed "improvement" is rather small (less than 0.6% absolute gain in accuracy, e.g. from 81.86% to 82.37% on CIFAR100 and from 72.33% to 72.78% on ImageNet). There is no runtime comparison, there are missing baselines, and most of the method development seems guided by trying out many options instead of taking a principled approach. Without these, the paper is just not ready for a top conference like ICLR.
Paper Summary
-------------
The paper considers a new take on active learning for image classification: given a large fully labeled dataset, identify a subset of the data that, when training on that subset alone, yields similar performance as training on the (much larger) full dataset. The paper focuses on "ensembles" of deep neural network classifiers as the prediction model, following Lakshminarayanan et al. (2017).
The presented method is summarized in Algorithm 1. Given a suitably initialized "acquisition model", the model makes predictions on each example in the full dataset, then ranks examples using an acquisition function to find the subset of size N_s (top N_s examples by rank) where there is most "disagreement" among the model ensemble. This subset is then used to train a "subset" model (again, an ensemble of DNNs).
Experiments consider several possible initializations, acquisition functions, and ensemble sizes. Evaluation is done using the validation sets of three prominent image classification benchmarks: CIFAR10, CIFAR100, and ImageNet (1000 classes).
Significance
------------
I don't think a successful case has been made that the proposed solution would generate significant widespread interest, because the gains demonstrated here are too minimal. Looking at the primary results in Table 2, it's really only when using 80% of the total images of imagenet or cifar100 (the most realistic benchmarks) that there is a small (<1%) absolute gain in accuracy over the simpler approach of just using the full dataset. Thus, the approach is not going to significantly reduce computational burden but adds a lot of complexity.
Novelty
-----------
The method seems new to me.
Experimental Concerns
---------------------
## E1: Need to consider runtime in evaluation
None of the figures/tables that I can see report elapsed runtimes for the different methods. To me this is the fundamental tradeoff: not how many fewer examples can I learn from, but how much faster is the method than the "standard" of using the full dataset? Showing curves of validation progress over wallclock time would be a better way to present results.
The important thing here is that even *full dataset* makes progress after each minibatch. You'd need to show progress at checkpoints for each epoch in  0.2, 0.4, 0.6, 0.8, 1, 1.2, 1.4, ....
## E2: Potential missing baseline: Random subset with balanced classes
When you select a random subset, are you ensuring class balance? If not, that seems like a more refined baseline. Perhaps won't make too much difference for cifar10, but could be important for ensuring rarer classes in ImageNet are represented well.
Presentation Concerns
---------------------
## P1: Initialization description confusing
I didn't understand the "build up" method as described in Sec. 3. How large is the subset used at each phase? How do you know when to stop? This could use a rewrite to improve clarity.
## P2: Missing some details for reproducibility
How was convergence assessed for all models? How were learning rates set? Many of these are crucial to understanding the runtime required for different models. (Sorry if these are in the appendix, but some short summary is needed in the main paper)
## P3: Title Change Recommended
I don't think the presented method is really doing a "Distribution Search"... I would suggest "Training Data Subset Selection with Ensemble Active Learning"
Minor Method Concerns
---------------------
## M1: What about regression?
Acquisition functions seem specialized to classification. What to do for regression or structure learning? Any general principles to recommend?


We introduce a new black-box attack achieving state of the art performances. Our approach is based on a new objective function, borrowing ideas from ∞ -white box attacks, and particularly designed to fit derivative-free optimization requirements. It only requires to have access to the logits of the classifier without any other information which is a more realistic scenario. Not only we introduce a new objective function, we extend previous works on black box adversarial attacks to a larger spectrum of evolution strategies and other derivative-free optimization methods. We also highlight a new intriguing property that deep neural networks are not robust to single shot tiled attacks. Our models achieve, with a budget limited to 10, 000 queries, results up to 99.2% of success rate against InceptionV3 classifier with 630 queries to the network on average in the untargeted attacks setting, which is an improvement by 90 queries of the current state of the art. In the targeted setting, we are able to reach, with a limited budget of 100, 000, 100% of success rate with a budget of 6, 662 queries on average, i.e. we need 800 queries less than the current state of the art.

This paper proposed a new query efficient black-box attack algorithm using better evolution strategies. The authors also add tiling trick to make the attack even more efficient. The experimental results show that the proposed method achieves state-of-the-art attack efficiency in black-box setting.
The paper indeed presented slightly better results than the current state-of-the-art black-box attacks. It is clearly written and easy to follow, however, the paper itself does not bring much insightful information.
The major components of the proposed method are two things: using better evolution strategies and using tiling trick. The tiling trick is not something new, it is introduced in (Ilyas et al., 2018) and also discussed in (Moon et al., 2019). The authors further empirically studied the best choice of tiling size. I appreciated that, but will not count it as a major contribution. In terms of better evolution strategies, the authors show that (1+1) and CMA-EA can achieve better attack result but it lacks intuition/explanations why these helps, what is the difference. It would be best if the authors could provide some theories to show the advantages of the proposed method, if not, at least the authors should give more intuition/explanation/demonstrative experiments to show the advantages.
Detailed comments:
- In section 3.2, is the form of the discretized problem a standard way to transform from continuous to discrete one? What is the intuition of using a and b? Have you considered using only one variable to do it?
- In section 3.3.2 what do you mean by “with or without softmax, the optimum is at infinity”? I hope the authors could further explain it.
- In eq (2), do you mean  max_{\tau} L(f(x + \epsilon tanh(\tau)), y) ?
- In section 3.3.1, the authors said (1+1)-ES and CMA-ES can be seen as an instantiation of NES. Can the authors further elaborate on this?
- Can the authors provide algorithm for DiagonalCMA?
- It is better to put the evolution strategy algorithms in the main paper and discuss it.
- Can the authors also comment/compare the results with the following relevant paper?
Li, Yandong, et al. "NATTACK: Learning the Distributions of Adversarial Examples for an Improved Black-Box Attack on Deep Neural Networks." ICML 2019.
Chen, Jinghui, Jinfeng Yi, and Quanquan Gu. "A Frank-Wolfe Framework for Efficient and Effective Adversarial Attacks." arXiv preprint arXiv:1811.10828 (2018).
-  In Table 1, why for Parsimonious and Bandit methods, # of tiles parts are missing? I think both of the baselines use tilting trick? And they should also run using the optimal tiling size? The result seems directly copied from the Parsimonious paper? It makes more sense to rerun it in your setting and environment cause the sampled data points may not be the same. Since CMA costs significantly more time, it makes a fair comparison to also report the attack time needed for each method.
- In Table 3, why did not compare with Bandit and Parsimonious attacks?
======================
after the rebuttal
I thank the authors for their response but I still feel that there is a lot more to improve for this paper in terms of intuition and experiments. Therefore I decided to keep my score unchanged.


In reinforcement learning, robotic control tasks are often useful for understanding how agents perform in environments with deceptive rewards where the agent can easily become trapped into suboptimal solutions. One way to avoid these local optima is to use a population of agents to ensure coverage of the policy space (a form of exploration), yet learning a population with the "best" coverage is still an open problem. In this work, we present a novel approach to population-based RL in continuous control that leverages properties of normalizing flows to perform attractive and repulsive operations between current members of the population and previously observed policies. Empirical results on the MuJoCo suite demonstrate a high performance gain for our algorithm compared to prior work, including Soft-Actor Critic (SAC).

RL in environments with deceptive rewards can produce sub-optimal policies. To remedy this, the paper proposes a method for population-based exploration. Multiple actors, each parameterized with policies based on Normalizing Flows (radial contractions), are optimized over iterations using the off-policy SAC algorithm. To encourage diverse-exploration as well as high-performance, the SAC-policy-gradient is supplemented with gradient of an “attraction” or “repulsion” term, as defined using the KL-divergence of current policy to another policy from an online archive. When applying the KL-gradient, the authors find it crucial to only update the flow layers, and not the base Gaussian policy.
I have 2 issues with this paper:
1.	Lack of novelty – Most of the components of the proposed algorithm have been researched in other works. SAC with NF was contributed by Mazoure et. al., and this paper seems to use the exact same architecture.  The diversity-enforcing objective (Equation 1.) was proposed by Hong et. al., albeit not with a separate archive of policies, but using the experience replay (for off-policy) and recent policies (for on-policy). The objective is Section 3.2 is also the same as that in Hong et. al.
2.	I have major concerns about how the results have been reported:
     a.	The authors have omitted SAC and SAC-NF from Figure 4, and therefore the figure gives the impression that ARAC is significantly better than the presented baselines in some of the environments. For example, take Humanoid-v2. The original SAC paper reports reaching close to 5k score in about 0.5 million timesteps. It therefore seems that most of the benefit of ARAC (over the presented baselines) comes just from using SAC. Another example is Humanoid-sparse, where SAC-NF achieves ~550 (from Table 1), but is not shown in the figure, making ARAC (score ~600) look awesome. The authors also incorrectly mention in text that “ARAC is the only method that can achieve non-zero performance” on this.
     b.	Table 1 reports “maximum” average return. This is very misleading and non-standard in RL. Take Humanoid-sparse for instance. The number reported for ARAC is 816 which is the peak performance during training. As we see in Figure 4, ARAC is unstable and the perf. drops to ~600 at end of training. The peak performance during training is an irrelevant metric.
     c.	Humanoid-rllab range on y-axis looks incorrect
Minor points:
For creating a diverse archive, I’m not sure if k-means on the episodic-returns is the most effective mechanism. As explored in Conti et al., and also mentioned in the introduction of this paper, behavioral-diversity is a more useful characterization, and episodic-returns may not be aligned to that (especially in sparse reward environments).
Some of the missing related work (on population-based diversity): DIAYN, Learning Self-Imitating Diverse Policies, Divide and Conquer RL.


Most of existing advantage function estimation methods in reinforcement learning suffer from the problem of high variance, which scales unfavorably with the time horizon. To address this challenge, we propose to identify the independence property between current action and future states in environments, which can be further leveraged to effectively reduce the variance of the advantage estimation. In particular, the recognized independence property can be naturally utilized to construct a novel importance sampling advantage estimator with close-to-zero variance even when the Monte-Carlo return signal yields a large variance. To further remove the risk of the high variance introduced by the new estimator, we combine it with existing Monte-Carlo estimator via a reward decomposition model learned by minimizing the estimation variance. Experiments demonstrate that our method achieves higher sample efficiency compared with existing advantage estimation methods in complex environments.

This paper proposes a novel advantage estimate for reinforcement learning based on estimating the extent to which past actions impact the current state. More precisely, the authors train a classifier to predict the action taken k-steps ago from the state at time t, the state at t-k and the time gap k. The idea is that when it is not possible to accurately predict the action, the action choice had no impact on the current state, and thus should not be assigned credit for the current reward, they refer to this as the "independence property" between current action and future states. Based on this idea, the authors introduce a "dependency factor", using the ratio P(s_{t+k},a_{t+k}|s_t,a_t)/P(s_{t+k},a_{t+k}|s_t). They later show that this can be reworked using Bayes theorem into a ratio of the form P(a_t|s_t,s_{t+k})/\pi(a_t|s_t) which is more convenient to estimate. The authors show mathematically that, when the dependency factor is computed with the true probabilities and use to weight each reward in a trajectory, the result is an unbiased advantage estimator. Importantly the expectation, in this case, is taken over trajectories sampled according to the policy pi conditioned only on S_t. This is distinct from the Monte-Carlo estimator which is based only on samples in which A_t, the action whose advantage is being estimated, was selected.
They go on to say that this estimator will tend to have lower variance than the conventional Monte-Carlo estimator when future rewards are independent of current actions. However, the variance can actually be higher, due to the importance sampling ratio used, when future rewards are highly dependent on the current action. They propose a method to combine the two estimators on a per reward while maintaining unbiasedness using a control variate style decomposition. This introduces a tunable reward decomposition parameter which determines how to allocate each reward between the two estimators. The authors propose a method to tune this parameter by approximately optimizing an upper bound on the variance of the combined estimator.
As a final contribution, the authors introduce a temporal-difference method of estimating the action probability P(a_t|s_t,s_{t+k}) required by their method.
In the experiments, the authors provide empirical evidence that various aspects of their proposed method can work as suggested on simple problems. They also provide a simple demonstration where their advantage estimator is shown to improve sample efficiency in a control problem.
This paper suffers from moderate clarity issues, but I lean toward acceptance primarily because I feel that the central idea is a solid contribution. The idea of improving credit assignment by explicitly estimating how much actions impact future states seems reasonable and interesting. The temporal difference method introduced for estimating P(a_t|s_t,s_{t+k}) is also interesting and clever. I'm less confident in the introduced method for trading off between the Monte Carlo and importance sampling estimators. The experiments seem reasonably well executed and do a fair job of highlighting different aspects of the proposed method.
The derivation of the combined estimator was very confusing to me. It's strange that the derivation of the variance lower bound includes terms which are drawn from both a state conditional and state-action conditional trajectory. You're effectively summing variances computed with respect to two different measures, but the quantity being bounded is referred to as just the "variance of the advantage estimator". What measure is this variance supposed to be computed with respect to? Especially given that as written the two estimators rely on samples drawn from two different measures. It doesn't help that the advantage estimator whose variance is being constructed is never explicitly defined but just referred to as "advantage estimator derived from equation 3". Nevertheless, if we ignore the details of what exactly it is a lower bound of, the sum of the three variances in equation 5 seems a reasonable surrogate to minimize.
Related to the above point I don't fully understand what the variances shown in table 1 mean in the experiments section. For the IAE estimator for example, is the variance computed based on each sample using three independent trajectories (one for each term) or is it computed from single trajectories? If it's from single trajectories I can't understand how the expression would be computed.
Questions for the authors:
-Could you please explicitly define the "advantage estimator derived from equation 3"?
-Could you please explain precisely how the variance is computed in table 1?
Update:
Having read the other reviews and authors response I will maintain my score of a weak accept, though given more granularity I would raise my score to a 7 after the clarifications. I appreciate the authors' clarification of the advantage estimator and feel the related changes to the paper are very helpful. I still feel the central idea of the work is quite strong.
However, I also feel the control variate part of the work is very loosely specified. In particular, given the use of function approximation in practice instead of actually sampling 3 trajectories the validity of the control variate method applied is questionable. As the authors say "if the random variable in either term has high variance, function approximators will tend to have large error", This may be true initially but the function approximator can already reduce variance over time by learning, so it's not clear how the function approximators and control variate complement each other. This is something I feel would be worthwhile to explore more in future work.
Also, I feel it's worth pointing out that a concurrent paper presenting a very similar idea is scheduled to be presented at NeurIPS 2020, which can be found here: https://papers.nips.cc/paper/9413-hindsight-credit-assignment. I don't feel this in any way undermines the contribution of the work presented here, but merely wanted to make the meta reviewer aware in case it was relevant to their decision. In fact, I feel this work complements that one in a number of ways, including the presentation of the temporal difference method for learning the action probabilities.


Deep image prior (DIP) (Ulyanov et al., 2018), which utilizes a deep convolutional network (ConvNet) structure itself as an image prior, has attracted huge attentions in computer vision community. It empirically shows the effectiveness of ConvNet structure for various image restoration applications. However, why the DIP works so well is still unknown, and why convolution operation is useful for image reconstruction or enhancement is not very clear. In this study, we tackle these questions. The proposed approach is dividing the convolution into "delay-embedding" and "transformation (i.e., encoder-decoder)", and proposing a simple, but essential, image/tensor modeling method which is closely related to dynamical systems and self-similarity. The proposed method named as manifold modeling in embedded space (MMES) is implemented by using a novel denoising-auto-encoder in combination with multi-way delay-embedding transform. In spite of its simplicity, the image/tensor completion, super-resolution, and deconvolution results of MMES are quite similar even competitive to DIP in our extensive experiments, and these results would help us for reinterpreting/characterizing the DIP from a perspective of "low-dimensional patch-manifold prior".

In this paper, the authors present a natural image model based on the manifold of image patches.  It is similar to the Deep Image Prior in that it is untrained and has a convolutional-like structure.  It leads to an optimization problem with a reconstruction loss term and an auto encoding term.  The authors show empirical results in time series recovery, non-semantic inpainting, and super resolution.  In the image processing tasks, the performance of the proposed algorithm is on par (sometimes slightly worse, sometimes slightly better) than that of DIP.
I think the perspective of image patch analysis is a useful addition to the knowledge base for unlearned image priors.  That said, the paper says it tackles the questions for why the DIP "works so well" and why convolution operations are "essential for image reconstruction or enhancement".  After reading the paper, it is unclear to me how this work addresses these questions.  In particular, demonstrating a similar convolutional system does not rule out the possibility that there are non-convolutional systems that also explain the effect.  For example, the Deep Geometric Prior paper is nonconvolutional (it is entirely a MLP), and it also has the effect of fitting a smooth signal without training (subject to early stopping).  The DGP could be applied to images as well, resulting in a nonconvolutional deep image prior.  I think the authors should address more clearly and thoroughly the logical connection between their results and the explanation of the DIP, especially in light of the DGP.
The paper claims that the proposed method is more interpretable, and it would be nice if they could demonstrate this interpretability and the benefits it brings in solving image reconstruction problems.
As a result, I am inclined to recommend a weak reject, but if the concerns above are addressed, I envision my rating could improve upon rebuttal.


Attribution methods provide insights into the decision-making of machine learning models like artificial neural networks. For a given input sample, they assign a relevance score to each individual input variable, such as the pixels of an image. In this work we adapt the information bottleneck concept for attribution. By adding noise to intermediate feature maps we restrict the flow of information and can quantify (in bits) how much information image regions provide. We compare our method against ten baselines using three different metrics on VGG-16 and ResNet-50, and find that our methods outperform all baselines in five out of six settings. The method's information-theoretic foundation provides an absolute frame of reference for attribution values (bits) and a guarantee that regions scored close to zero are not necessary for the network's decision. Deep neural networks have become state of the art in many realworld applications. However, their increasing complexity makes it difficult to explain the model's output. For some applications such as in medical decision making or autonomous driving, model interpretability is an important requirement with legal implications. Attribution methods (Selvaraju et al., 2017; Zeiler & Fergus, 2014;Smilkov et al., 2017) aim to explain the model behaviour by assigning a relevance score to each individual input variable. When applied to images, the relevance scores can be visualised as heatmaps ovä er the input pixel space, thus highlighting salient areas relevant for the network's decision.

Summary
---
(motivation)
Lots of methods produce attribution maps (heat maps, saliency maps, visual explantions) that aim to highlight input regions with respect to a given CNN.
These methods produce scores that highlight regions that are in a vague sense "important."
While that's useful (relative importance is interesting), the scores don't mean anything by themselves.
This paper introduces another new attribution method that measures the amount of information (in bits!) each input region contains, calibrating this score by providing a reference point at 0 bits.
Non-highlighted regions contribute 0 bits of information to the task, so they are clearly irrelevant in the common sense that they have 0 mutual information with the correct output.
(approach - attribution methods)
An information bottleneck is introduced by replacing a layer's (e.g., conv2) output X with a noisy version Z of that output.
In particular, Z is a convex combination of the feature map (e.g., conv2) with Gaussian noise with the same mean and variance as that feature map.
The weights of the combination are found so they minimize the information shared between the input and Z and maxmimize information shared between Z and the task output Y.
These weights are either optimized on
1) a per-image basis (Per-Sample) or
2) predicted by a model trained on the entire dataset (Readout).
(approach - evaluation)
The paper uses 3 metrics with differing degrees of novelty:
1) The bbox metric rewards attribution methods that put a lot of mass in ground truth bounding boxes.
2) The original Sensitivity-n metric from (Ancona et al. 2017) is reported with a version that uses 8x8 occlusions.
3) Least relevant image degredation is compared to most relevant image degredation (e.g., from (Ancona et al. 2017)) to form a new occlusion style metric.
(experiments)
Experiments consider many of the most popular baselines, including Occlusion, Gradients, SmoothGrad, Integrated Gradients, GuidedBP, LRP, Grad-CAM, and Pattern Attribution. They show:
1) Qualitatively, the visualizations highlight only regions that seem relevant.
2) Both Per-Sample and Readout approaches put higher confidence into ground truth bounding boxes than all other baselines.
3) Both Per-Sample and Readout approaches outperform all baselines almost all the time according to the new image degredation metric.
Strengths
---
The idea makes a lot of sense. I think heat maps are often thought of in terms of the colloquial sense of information, so it makes sense to formalize that intuition.
The related work section is very well done. The first paragraph is particularly good because it gives not just a fairly comprehensive view of attribution methods, but also because it efficiently describes how they all work.
The results show that proposed approaches clearly outperform many strong baselines across different metrics most of the time.
Weaknesses
---
* I'm not sure why the new degredation metric is a useful addition. What does it add that MoRF and LeRF don't capture on their own independently?
* I think [1] would be a nice addition to the evaluation section as it tests for something qualitatively different than the various metrics from section 4. It would also be a good addition to the related work.
Missing Details / Points of Confusion
---
* I think there's an extra p(x) in eq. 11 in appendix D.
* I think the variable X is overloaded. In eq. 1 it refers to the input (e.g., the pixels of an image) while in eq. 2 it refers to an intermediate feature map (e.g., conv2) even though it later seems to refer to the input again (e.g., eq. 3). Different notation should be used for intermediate feature maps and inputs.
Presentation Weaknesses
---
* In section 3.1 is lambda meant to be constrained in the range [0, 1]? This is only mentioned later (section 3.2) and should probably be mentioned when lambda is introduced.
* "indicating that all negative evidence was removed." I think this should read "indicating that only negative evidence was removed."
Suggestions
---
"The bottleneck is inserted into an early layer to ensure that the information in the network is still local"
I'd like this to be explored a bit more. Though deeper feature maps are certainly more spatially coarse they still might be somewhat "local". To what degree to they loose localization information? My equally vague alternative intuition goes a bit differently: The amount of relevant information flowing through any spatial location seems like it shouldn't change that much, only the way its represented should change. If the proposed visualizations were the same for every choice of layer then it would confirm this intuition. That would also be an interesting result because most if not all of the cited baseline approaches (where applicable) produce qualitatively different attributions at different layers (e.g., see Grad-CAM).
[1]: Adebayo, Julius et al. “Sanity Checks for Saliency Maps.” NeurIPS (2018).
Preliminary Evaluation
---
Clarity: The paper is clearly written.
Originality: The idea of using the formal notion of information in attribution maps is novel, as is the bbox metric.
Significance: This method could be quite significant. I can see it becoming an important method to compare to.
Quality: The idea is sound and the evaluation is strong.
This is a very nice paper in all the ways listed above and it should be accepted!
Post-rebuttal comments
---
The author responses and other reviews have only increased my confidence that this paper should be accepted.