Graph Neural Networks (GNNs) have recently received tremendous attention due to their power in handling graph data for different downstream tasks across different application domains. Many GNN models have been proposed, which mainly differ in their graph filter design. However, most of these models believe there is a best filter for all the graph data. Instead, we attempt to provide in-depth analysis on (1) Whether there exists an optimal filter that performs the best on all graph data; (2) Which graph properties should be considered for finding the optimal graph filter; and (3) How to design appropriate filters that adapt to a given graph. In this paper, we focus on addressing the above three questions for the semi-supervised node classification task. We propose a novel assessment tool, called Graph Filter Discriminant Score (GFD), for evaluating the effectiveness of graph convolutional filters for a given graph in terms of node classification. Using the assessment tool, we find out that there is no single filter as a "silver bullet" that performs the best on all possible graphs, and graphs with different properties are in favor of different graph convolutional filters. Based on these findings, we develop Adaptive Filter Graph Neural Network (AFGNN), a simple but powerful model that can adaptively learn data-specific filters. For a given graph, AFGNN leverages graph filter assessment as an extra loss term and learns to combine a set of base filters. Experiments on both synthetic and real-world benchmark datasets have demonstrated that our proposed model has the flexibility in learning an appropriate filter and consistently provides state-of-the-art performance across all the datasets.

In this paper, the authors raise and address three questions about graph neural network: (1) Whether there is a best filter that works for all graphs. (2) Which properties of the graph will influence the performance of the graph filter. (3) How to design a method to adaptively find the optimal filter for a given graph.
The paper proposes an assessment method called the Graph Filter Discriminant Score based on the Fisher Score. It measures how well the graph convolutional filter discriminate node representations of different classes in the graph by comparing the Fisher score before and after the filter.
Based on the GFD scores of different normalization strategy and different order of the graph filter in the experiments on synthetic data, the authors answer the first two questions: (1) There is no optimal normalization for all graphs. (2) row normalization performs better with lower power-law coefficient, but works worse with imbalanced label classes and large density gap.
For the third question, the authors propose a learnable linear combination of a limited family of graph convolutional filters as the layer of model AFGNN, which can learn the optimal arguments of the combination based on the FGD score.
The paper focuses on a significant topic and proposes an assessment tool for the graph filters. Based on that, it also introduces a model to choose filters from a family of filters for any specific graph.
The description of preliminaries is clear.
The observations of the impact of the graph properties on the filter choice are interesting and explanations are provided.
The results of the test accuracy on both bench mark and synthetic datasets demonstrate the good performance of the proposed model.
It is good that the paper provides proof for the claim that the graph convolutional can help the non-linear separable data to be linearly separable, so it is reasonable to use Fisher score. However, does this claim support the second term in equation (3), where the Fisher score is used to evaluate before the filters applied?
The presentation of the last paragraph of “graph filter discriminant score” in page 4 can be improved. The Figure references seem incorrect and confusing.
The analysis of the influence of label ratio seems not accurate enough.
For the GFD score comparison in Figure 4, why choose order 1,3,7 for density and different order 2,3,6 for density gap?
What is the meaning of the symbol psi(l)?
It would be better if the explanation of the raining loss section is more detailed and clear.
What is “AFGNN_P” in the experiment analysis?
It could be interesting to see the comparison of time between the proposed method and the GAT.
For the graph filter discriminate analysis, is it fair to compare the learned layer with the other base filter using the GFD score? Since the learned layer is picked with highest GFD score. Maybe one or two sentences on this will be helpful.
The writing of the paper must be improved. Too many typos and grammar problems will impair the presentation and the reader can be distracted.
Minor comments:
The layout of the sub caption of Figure 1 can be improved.
The usage of capital letter in the phrase “density gap” is inconsistent.
“As shown in figure” instead of “As is shown in figure”.
Many sentences miss article.
There are many typos in the writing.
For example, “Note that for given (feature)”, “…make the representation of nodes in different (class) more separable.”, “Noted that there are some other (variant) of GNN filters that (does) not fall into…” in page 4.
“Here we give (a) empirical explanation to this phenomenon”, “this normalization strategy (take) into account…”, “Thus even in the case that the other two (doesn’t) perform well…” in page 5.
“…a very important factor that (influence) the choice of normalization strategy”, “when power-law coefficient (decrease)”, “when the (sizes) of each class (become) more imbalanced”,  “This is because column normalization better (leverage) …” , “in a similar manner (with) label ratio”, “when the density or density gap (increase)”, “high-order filters can help gather… and thus (makes) the representations…”, “when the density gap (increase)” in page 6.
These can be continued but it is obvious that this paper needs proofreading.


This paper proposes METAGROSS (Meta Gated Recursive Controller), a new neural sequence modeling unit. Our proposed unit is characterized by recursive parameterization of its gating functions, i.e., gating mechanisms of METAGROSS are controlled by instances of itself, which are repeatedly called in a recursive fashion. This can be interpreted as a form of meta-gating and recursively parameterizing a recurrent model. We postulate that our proposed inductive bias provides modeling benefits pertaining to learning with inherently hierarchically-structured sequence data (e.g., language, logical or music tasks). To this end, we conduct extensive experiments on recursive logic tasks (sorting, tree traversal, logical inference), sequential pixel-by-pixel classification, semantic parsing, code generation, machine translation and polyphonic music modeling, demonstrating the widespread utility of the proposed approach, i.e., achieving state-of-the-art (or close) performance on all tasks.

This paper proposes a neural sequence modelling unit called METAGROSS.  In principle, the aim of this unit is to introduce recursive parametrization of  gating functions, building on the gated RNN paradigm.  The authors motivate this work by arguing that while gated-RNNs tackle vanishing gradient problems and facilitate learning long-range dependencies in sequences, improvements can be made with respect to learning on hierarchically-structured data.  The authors propose a method to do so by also learning the depth of the parametrization, and claim that the inductive bias that emerges from this configuration is beneficial to learning such tasks.
I think the idea behind this work is sensible: introducing a meta-controller with recursive parametrization of gating functions for hierarchical tasks is sensible.  Another also strong point of this paper is that several experiments under different settings are presented, along with ablation studies and some model exploration.  The authors also show that integrating a non-autoregressive variant of the proposed meta-controller into the Transformer architecture can also be beneficial.  Results in general do show improvement over compared architectures.
On the negative side and besides empirical/experimental evidence, the paper would be much more convinving with some more insight into the model itself and some more qualitative evidence.  Figure 1 shows the architecture of the proposed method with a max depth of 3 indicating soft recursion with grayscale levels.  However, this figure is not referenced in the text and not explained further.  The non-autoregressive version (sec 2.3) simply does away with the dependence on hidden states and applies the proposed architecture directly on the input.  I think it would be very beneficial to see a simple toy example where the benefits of utilizing this meta-architecture can be qualitatively explained.  Finally, one can argue that this work is incremental, in the sense that it is a (relatively straightfwd) combination of meta-controllers with recursive architectures.  Differences and variations with respect to other methods in literature should be more clearly explained.
some more questions
- although most experiments show some resilience with respect to the varying max depth parameter, have the authors noticed any problems and limitations arising from setting the max-depth to be very high?  for example, in Table 1 it seems that accuracy may drop when increasing max length.  I think that more discussions and ellaboration on this could be useful, as the authors also propose a way of learning the depth parameters and also state that this is task dependent.
- Figures 7-8-9 show variations of dynamic recursion on three databases.  Although these figures do show variation in learning (ranging from high activation fluctuations to static), it would be interesting to examine why this fluctuations occur in CIFAR
- which brings me to the second question:  although the results do not show this, could this task-dependent nature of the controller lead to more chances of overfitting on a given training set that may be noisy?
- Although the authors perform ablation tests with multiple units and evaluate for max depth, we don't see many experiments with depth of more than 2 or 3 (besides table 1 - in many experiments the max depth is not mentioned).  Are the conclusions the same with all experiments wrt depth?
- It would be useful to have a comment on model complexity


How can we transfer knowledge from a source domain to a target domain when each side cannot observe the data in the other side? The recent state-of-theart deep architectures show significant performance in classification tasks which highly depend on a large number of training data. In order to resolve the dearth of abundant target labeled data, transfer learning and unsupervised learning leverage data from different sources and unlabeled data as training data, respectively. However, in some practical settings, transferring source data to target domain is restricted due to a privacy policy. In this paper, we define the problem of unsupervised domain adaptation under double blind constraint, where either the source or the target domain cannot observe the data in the other domain, but data from both domains are used for training. We propose TAN (Transfer Alignment Network for Double Blind Domain Adaptation), an effective method for the problem by aligning source and target domain features. TAN maps the target feature into source feature space so that the classifier learned from the labeled data in the source domain is readily used in the target domain. Extensive experiments show that TAN 1) provides the state-ofthe-art accuracy for double blind domain adaptation, and 2) outperforms baselines regardless of the proportion of target domain data in the training data.

###Summary###
This paper tackles the transfer learning problem with the double-blind unsupervised domain adaptation, where either the source or the target domain cannot observe the data in the other domain, but data from both domains are used for training. The high-level intuition of this paper is based on the observation that in some practical settings, the transferring source data to the target domain is restricted due to the privacy policy. The goal is to learn a classifier which performs well in target classification task under double-blind constraint.
The setting of this paper is slightly different from the conventional domain adaptation. In this paper, the source domain has abundant unlabeled data and a small number of labeled data. The target domain only contains a limited number of unlabeled data.
The paper proposes a transfer alignment network (TAN) which comprises two autoencoders, one trained on the source domain and one trained on the target domain. In the domain adaptation phase, the model leverages an aligner to transfer the output of the target encoder to an aligned latent variable.  The aligner is trained to map the target code to source code on the target unlabeled data. The objective function is L2 distance between the source code and the mapped target code.
The whole pipeline is trained with four steps:
1) The source encoder and source decoder are trained with L2 reconstruction loss.
2) The source encoder and source classifier are trained with cross-entropy classification loss.
3) The target encoder and target decoder are trained with L2 reconstruction loss.
4) Train the aligner to map target code to source code on target unlabeled data with L2 distance loss.
The paper proposes to compare the TAN with three baselines: S(UL):  a stack of encoder and a neural network classifier trained using source data and tested on target data without finetuning. S(UL)-T(U): a model retrains the S(UL) with the target unlabeled data. S(UL)-T(U)-Large: a model which is similar to S(UL)-T(U) but contains more layers and parameters in MLP.
The experiments are performed on five multivariate datasets: HIGGS, HEPMASS, SUSY, Sensorless, and Gas.
### Novelty ###
The experimental setting proposed in this paper is interesting. However, the proposed model is trivial. The TAN model is composed of autoencoders and aligner. The training losses in the framework are L2 reconstruction loss and L2 distance loss. Thus, the novelty of this paper is incremental.
The experimental results in this paper are weak. First of all, the datasets used in this paper are not standard benchmarks. Secondly, the baselines in this paper are too trivial.
###Clarity###
Overall, the paper is well organized and logically clear. The images are well-presented and well-explained by the captions and the text.
###Pros###
1) The paper proposes an interesting transfer learning framework where either the source or the target domain cannot observe the data in the other domain.
2) The paper is applicable to many practical scenarios since the data privacy in the real-world application is critical.
3) The paper is overall well-organized. The claims of the paper are verified by the experimental results.
###Cons###
1) The paper proposes double-blind unsupervised domain adaptation as accessing the source and target domains is restricted in some practical settings. However, the source and target domain share the models trained on themselves, as well as the features extracted from the source domain and target domain data. The information about the original data can be recovered with the shared features and weights, which violates the settings proposed in this paper.
2) The main issue of this paper is the novelty is incremental. The proposed model is trivial as it only contains the auto-encoders and L2 loss.
3) The experimental part of this paper is weak. The datasets used in this paper are not the standard domain adaptation benchmark. It would be nice to see how does the proposed model work on standard domain adaptation benchmarks such as Office31, VisDA, Office-Home, DomainNet, etc.
VisDA: The Visual Domain Adaptation Challenge
https://arxiv.org/pdf/1710.06924.pdf
Office-Home : Deep Hashing Network for Unsupervised Domain Adaptation
http://hemanthdv.org/OfficeHome-Dataset/
The baselines used in this paper is also trivial. It is desirable to compare the proposed method with state-of-the-art domain adaptation methods.
Based on the summary, cons, and pros, the current rating I am giving now is "reject". I would like to discuss the final rating with other reviewers, ACs.


When automatically generating a sentence description for an image or video, it often remains unclear how well the generated caption is grounded, or if the model hallucinates based on priors in the dataset and/or the language model. The most common way of relating image regions with words in caption models is through an attention mechanism over the regions that are used as input to predict the next word. The model must therefore learn to predict the attentional weights without knowing the word it should localize. This is difficult to train without grounding supervision since recurrent models can propagate past information and there is no explicit signal to force the captioning model to properly ground the individual decoded words. In this work, we help the model to achieve this via a novel cyclical training regimen that forces the model to localize each word in the image after the sentence decoder generates it, and then reconstruct the sentence from the localized image region(s) to match the ground-truth. Our proposed framework only requires learning one extra fully-connected layer (the localizer), a layer that can be removed at test time. We show that our model significantly improves grounding accuracy without relying on grounding supervision or introducing extra computation during inference for both image and video captioning tasks.

# 1. Summary
The paper deals with the problem of learning grounded captions from images without joint text-location information, but instead texts (words in captions) and locations are provided independently and the model needs to figure out their link. The model is built upon GVD (Zhou et al., 2019): each word generated by the encoder is grounded to the locations provided by the region proposal module (Up-Down model (Anderson et al., 2018)), used to reconstruct the ground-truth caption.
My weak reject decision was guided by the following strengths and weaknesses of the paper.
Strengths:
* The reconstruction formulation of the problem is interesting and relevant for the task
* Proposed new metric to measure grounding performance

Weaknesses:
* Questionable motivations: it is not clear what application is grounding text to the image useful for?
* Marginal (not statistically significant?) improvement on image captioning metrics by using grounded text (proposed method) compared to not grounding (GVD)
* Limited novelty: extension of GVD, where attention is removed and object locations are used instead

# 2. Clarity and Motivation
The paper is generally well written, however there are some concerns on motivations.
One concern is related to the motivations of the paper. The authors use grounding as a proxy to improve image captioning results, which improvement is marginal wrt GVD (see Table 1). Why do we need to localize text if this has very marginal impact on the captioning metrics? It is missing the link between the potential applications where the localization of words is relevant.
The authors claim that they do not uses any grounding annotation, however the pre-trained Faster-RCNN has been trained using annotations which consist of bounding boxes + categories. Therefore, the model do (in an implicit way) rely on grounding annotations, especially because there might be an overlap between the words (classes) used to pretrain the detector and the words in the captions. The authors should assess if this ovelap/bias in the pre-trained Faster-RCNN exists or not.
Some other questions are still to be answered:
* How are the regions R parametrized? Is it the visual representation or bounding box locations?
* What happens to words that are not grounded to the image (e.g., verbs or articles)? Do you have a special way to deal with those?
* What is the intuition of multiplying word embedding and region embeddings to generate z in Eq. 6?
# 3. Novelty
The proposed method is an extension of the existing model GVD, where the attentional module is removed and its functionality is replaced by the cyclical training mode with reconstruction of the object locations. From the technical point of view this is limited novelty, but still an interesting improvement of the model; however the experiments and results do not support the claim that using such model improves image captioning result in a significant way. One way to answer to this question would have been by showing an application where the outputted locations are used for downstream tasks.
# 4. Experimentation
The experiments are carried out in a scrupulous way, by showing the comparing with GVD (with and without attention; with and without grounding supervision). The non-convincing part of them (as mentioned above already) is the fact that the improvements on these datasets might be non significant for image captioning. For example, let's consider the image captioning results in Table 1 (Flickr30k Entities): cyclical have a max improvement of 0.7 (CIDER) and min of 0 (B@4) when compared with GVD without grounding supervision. There is an obvious huge improvement on the grounding evaluation, which is obvious since GVD does not do it explicitly. The same trend is in Table 2.
These results are not convincing, combined by the fact that it is not clear in which applications one would want a very accurate grounded text.
# Minor Points
* Sec. 3.1: it is not clear that the Language LSTM is the decoder. Please explicitly say it before Eq. 1
* Caption of Table 3: which dataset is this?


Most generative models of audio directly generate samples in one of two domains: time or frequency. While sufficient to express any signal, these representations are inefficient, as they do not utilize existing knowledge of how sound is generated and perceived. A third approach (vocoders/synthesizers) successfully incorporates strong domain knowledge of signal processing and perception, but has been less actively researched due to limited expressivity and difficulty integrating with modern auto-differentiation-based machine learning methods. In this paper, we introduce the Differentiable Digital Signal Processing (DDSP) library, which enables direct integration of classic signal processing elements with deep learning methods. Focusing on audio synthesis, we achieve high-fidelity generation without the need for large autoregressive models or adversarial losses, demonstrating that DDSP enables utilizing strong inductive biases without losing the expressive power of neural networks. Further, we show that combining interpretable modules permits manipulation of each separate model component, with applications such as independent control of pitch and loudness, realistic extrapolation to pitches not seen during training, blind dereverberation of room acoustics, transfer of extracted room acoustics to new environments, and transformation of timbre between disparate sources. In short, DDSP enables an interpretable and modular approach to generative modeling, without sacrificing the benefits of deep learning. The library will be made available upon paper acceptance and we encourage further contributions from the community and domain experts.

This paper presents a model for audio generation/synthesis  where the model is trained to output time-varying parameters of a vocoder/synthesiser rather than directly outputting audio samples/spectrogram frames. The model is trained by mining an L1 loss between the synthesised audio and the real training audio. Overall, I found the paper to be well written, I found the online supplementary material helpful in getting an intuition for the quality of audio samples generated and to understand the various parts of the proposed architecture. I also found the figures to be extremely  informative, especially figure 2. Although, I think the title of the paper could be updated to something more specific like Differentiable Vocoders or something similar, since the description and experiments very specifically deal with audio synthesis with vocoders, even though the components might be general DSP components. I think the paper presents a reasonable alternative to current autoregressive audio generation methods and should be accepted for publication.
Minor Comments
1. The reference provided for RNNs, Sutskever et. al. 2014, should be supplemented with older references from the 80s when RNNs were first trained with backdrop through time.
2. “The bias of the natural world is to vibrate.” I am not sure what exactly is meant here. Is it really a “bias” if every object in the universe vibrates? I think the term bias has been overloaded in the paper and is confused with structural/inductive priors. Bias as used in the paper, seems to have several means. It would help the description if the authors cleared up this confusing use of the term.
3. In Section 1.3, the authors claim that one of the strengths of the proposed method is that the models are “relatively small”, however all discussion of the sizes is relegated to the appendix. It would be helpful to the reader if some model complexity comparisons are presented in the results section and a high level summary is presented at the end of Section 1.
4. In Section 3, some additional high-level description for the Harmonic plus Noise model (Serra & Smith, 1990) should be provided to motivate the discussion and experiments in the rest of the paper.
5. Section 4.2 should provide some description of the parameters counts for each of the components considered and how this compares with existing auto-regressive generation algorithms.


Deep learning interpretation is essential to explain the reasoning behind model predictions. Understanding the robustness of interpretation methods is important especially in sensitive domains such as medical applications since interpretation results are often used in downstream tasks. Although gradient-based saliency maps are popular methods for deep learning interpretation, recent works show that they can be vulnerable to adversarial attacks. In this paper, we address this problem and provide a certifiable defense method for deep learning interpretation. We show that a sparsified version of the popular SmoothGrad method, which computes the average saliency maps over random perturbations of the input, is certifiably robust against adversarial perturbations. We obtain this result by extending recent bounds for certifiably robust smooth classifiers to the interpretation setting. Experiments on ImageNet samples validate our theory.

This paper introduces an extension of Cohen et al. (2019)’s result that allows one to derive robustness certificates for interpretation methods, as well as a bound on the top-K overlap of saliency methods. These results motivate the introduction of Sparsified SmoothGrad and a relaxation of this method that has differentiable elements. These introduced approaches adapt previous methods so the derived bounds are applicable. The proposed methods are shown to perform as well as Quadratic SmoothGrad (Smilkov et al. 2017) in CIFAR-10 experiments.
I’m not familiar with the field so it is hard for me to judge how novel the presented results are or whether the used baselines are the proper ones. That being said, the paper presents an interesting idea and it is relatively easy to read (I really appreciate the fact that for every theorem there is an interpretation, in words, for it). The only thing that sometimes makes the paper hard to read is when it starts to refer to too many constants without remind the reader what they are about. I have two complaints/questions about the relevance of the introduced bounds though. Right now, to me, it seems that the derived theoretical guarantees are not that relevant, hopefully the questions below will help clarify that.
In page 6, before introducing the “Sparsified SmoothGrad and its Relaxations”, it is said that q is set to 2^13 because otherwise the gap would be too large in images from ImageNet, for example, when comparing to traditional values of q. However, ImageNet is never revisited in the paper. I was expecting to see ImageNet results in the experimental section but they are not there (or maybe some correlation between the gap and performance -- robustness). More than that, the Quadratic SmoothGrad, which doesn’t have any theoretical guarantee, seems to perform as well as the proposed methods. So where is the gap/theoretical result relevant? What are the settings in which having a method with the derived theoretical guarantees shine? What are the limitations of Quadratic SmoothGrad? Right now, it seems to me that the “Sparsified SmoothGrad and its Relaxations” and its empirical analysis weaken the paper, because they take a big chunk of it when there is not enough evidence to claim them as an important contribution. Am I missing something? I gave this paper a relatively low score because I’m not certain about the relevance of its results, but if my questions are satisfactory answered, I’ll be happy to update my score.
------
>>> Update after rebuttal: I stand by my score after the rebuttal.
Unfortunately I'm not an expert in this area and I don't feel confident in having a very strong opinion about this paper. That being said, enough presentation issues were raised that make me uneasy about raising my score. I do agree with some of the concerns raised by other reviewers.


A key element of understanding the efficacy of overparameterized neural networks is characterizing how they represent functions as the number of weights in the network approaches infinity. In this paper, we characterize the norm required to realize a function f : R d → R as a single hidden-layer ReLU network with an unbounded number of units (infinite width), but where the Euclidean norm of the weights is bounded, including precisely characterizing which functions can be realized with finite norm. This was settled for univariate functions f : R → R in Savarese et al. (2019), where it was shown that the required norm is determined by the L 1 -norm of the second derivative of the function. We extend the characterization to multi-variate functions (d ≥ 2, i.e., multiple input units), relating the required norm to the L 1 -norm of the Radon transform of a (d + 1)/2-power Laplacian of the function. This characterization allows us to show that all functions in Sobolev spaces W s,1 (R d ), s ≥ d + 1, can be represented with bounded norm, to calculate the required norm for several specific functions, and to obtain a depth separation result. These results have important implications for understanding generalization performance and the distinction between neural networks and more traditional kernel learning.

The paper studies the function space regularization behavior of learning with an infinite-width ReLU network with a bound on the l2 norm of weights, in arbitrary dimension, extending the univariate study of Savarese et al. (2019).
The authors show that the corresponding regularization function is more or less an L1 norm of the (weak) (d+1)st derivatives of the function, and provide a rigorous formal characterization in terms of the "R-norm", which is expressed via duality through the Radon transform and powers of the Laplacian.
In addition, the paper provides a number of implications of this study, such as approximation results through Sobolev spaces, an analysis of the norm of radial bump functions, and a new type of depth separation result in terms of norm as opposed to width.
Overall, this is a strong paper making several interesting and important contributions for our understanding of the inductive bias of ReLU networks. I thus recommend acceptance.
A few comments:
* is it possible to obtain precise characterizations of interpolating solutions in this setting (other than a mere representer theorem with ReLUs), as done in Savarese et al (2019, Theorem 3.3) for the univariate case?
* perhaps the results of Section 5.1 should be contrasted with those of Bach (2017, e.g. Prop. 5), which only require ~ d/2 derivatives instead of ~ d here, albeit with stronger requirements, for essentially the same functional space (though the approximation result is obtained from an associated RKHS, which is smaller).
* are the results on radial bump functions intended to provide insight on approximation or depth separation? what was the motivation behind this section?
Other minor comments/typos:
- after Prop. 1: "intertwining" appears twice
- eq. (22): missing f in l.h.s.
- eq. (23): is the first minus sign needed?
- before Thm. 1: point to which Appendix
- Section 4.1, "In particular, this is what would happen ... d+1": this should be further explained
- Section 4.1, final paragraph, "in order R-norm to be": rephrase
- Section 5.4, "required norm with three layers is finite": which norm? maybe point to a reference? Also, Example 5 could be explained in further detail
- Section 5.5: what is an RKHS semi-norm? you'd always have ||f|| = 0 => f = 0 in an RKHS, by the reproducing property


A major challenge in reinforcement learning is exploration, especially when reward landscapes are sparse. Several recent methods provide an intrinsic motivation to explore by directly encouraging agents to seek novel states. A potential disadvantage of pure state novelty-seeking behavior is that unknown states are treated equally regardless of their potential for future reward. In this paper, we propose an exploration objective using the temporal difference error experienced on extrinsic rewards as a secondary reward signal for exploration in deep reinforcement learning. Our objective yields novelty-seeking in the absence of extrinsic reward, while accelerating exploration of reward-relevant states in sparse (but nonzero) reward landscapes. We implement the objective with a two-policy Q-learning method in which Q and Q x are the action-value functions for extrinsic and secondary rewards, respectively. Secondary reward is given by the absolute value of the TD-error of Q. Training is off-policy, based on a replay buffer containing a mix of trajectories sampled using Q and Q x . We characterize performance on a set of continuous control benchmark tasks, and demonstrate comparable or faster convergence on all tasks when compared with other state-of-the-art exploration methods.

This paper proposes an exploration algorithm for reinforcement learning agents based on learning a separate Q-value function which treats TD-errors as rewards. Experiments on continuous control tasks with a difficult exploration component are used to highlight the effectiveness of the approach.
The proposed algorithm is interesting and has an intuitive appeal, the magnitude of the TD-error as an "auxiliary reward" seems like a natural choice to guide exploration. The experiments are comprehensive, including some ablation studies and robustness tests, with one caveat. Due to some experimental details (particularly concerning the environments), the empirical results may be invalid and it is difficult to assess them. There are also some certain parts that are unclear in the experimental results. Overall, because of these issues, I cannot recommend acceptance although I would be willing to increase my score if my concerns are addressed.
Concerning the SparseHalfCheetah task, from Appendix F: "...baseline reward level is -1 while a successful state provides 0 reward, but report reward values on a 0 to 500 scale for direct comparison with previous work". This reward function is not sparse since it gives -1 at every step except at the goal which would incentivize the agent to explore naturally if it were acting greedily (or approximately greedily) with respect to the value function. For a sparse reward task, I would expect the reward to be 0 everywhere except at the goal state(s), where it would have some positive value. This is the case for the original reward function for SparseHalfCheetah described in [1]. This change to the reward function makes any comparisons to previous work questionable (Table 1) and could also affect the qualitative results in section 4.5.
Similarly, the other 3 tasks also do not have a sparse reward structure as the agent receives -1 at every timestep. This again is a confounder for assessing the exploration capabilities of the algorithm. It would be preferable to use other environments such as those in [1], which indeed have sparse rewards.
For these sparse reward problems where most rewards are zero, then before any nonzero reward is observed, it seems like Qx would mostly behave like previous algorithms based on a state novelty term (as discussed in sec 3.4). This could happen since the agent would only be observing rewards of 0 at this point during training. It is unclear to me then which benefits QXplore could have compared to previous algorithms such as DORA.
Other points:
1) Some discussion of possible pitfalls for the algorithm could be added. For example, while the noisy TV problem may be avoided, there could be a noisy reward problem now. The absolute TD error could be high simply due to stochasticity in rewards. The agent could also be vulnerable to scenarios in which a stochastic transition brings the agent to either a state eventually leading to very high reward or to a state leading to no reward as this would cause the TD error be high (even if the value function converges).
2) The sentence before sec 3.3, "Further TD-error-based exploration with a dedicated exploration policy removes the exploitation-versus-exploration
tradeoff that ... to an optimal Q-function." Could the authors clarify how TD-error-based exploration avoids the exploitation-versus-exploration
tradeoff? I do not see the connection here.
3) In the experiments, how are actions chosen for the epsilon-greedy baseline since the action space is continuous (and not discrete)?
4) I am bit confused as to what the difference between the lines Q and Qx are in Fig. 3.
5) Why was DORA not included as a baseline algorithm? It would seem to be the most closely related to QXplore when rewards are mostly constant (or zero).
6) Are two separate buffers necessary instead of a single shared buffer (with uniform sampling)? It seems to needlessly introduce complexity and additional hyperparameters (the ratios of samples from each buffer). If this is a key choice, then it should be mentioned in the paper.
The text could also be clarified to indicate there are two buffers. In some places the writing suggests there is only one, e.g. sec3.3 "...policy with a replay buffer shared between Q_\theta andthe Q-function maximizing rx, which we term Qx."
7) In section 4.4, for the "Single-Policy QXplore", why was Q(s,a) replaced by V(s)? If I understand correctly, this experiment tries to test a variant where the TD error is used as a reward bonus. If this is the case, it seems like the best comparison would be to leave the original Q(s,a) which is learned by Q-learning instead of changing it to a state-value function.
Suggestions (did not impact score)
- I wonder if there is a connection between Qx and the variance of the returns since the latter can be learned by using the squared TD-errror as a reward (see the Direct Variance algorithm in [2]) and the absolute TD-error is a similar quantity. In this way, it could be possible to frame QXplore as following a type of risk-seeking stategy (see [3] for an algorithm that makes use of the variance).
- I am not sure 'adversarial' is the right term to describe Q and Qx since the two policies are not in direct competition with each other.
[1] "VIME: Variational Information Maximizing Exploration" by Houthooft et al.
[2] "Directly Estimating the Variance of the λ-Return Using Temporal-Difference Methods" by Sherstan et al.
[3] "Deep Reinforcement Learning with Risk-Seeking Exploration" by Dilokthanakul and Shanahan


Large-scale benchmark datasets have been among the major driving forces in AI, supporting training of models and measuring their progress. The key assumption is that these benchmarks are realistic approximations of the target tasks in the real world. However, while machine performance on these benchmarks advances rapidly -often surpassing human performance -it still struggles on the target tasks in the wild. This raises an important question: whether the surreal high performance on existing benchmarks are inflated due to spurious biases in them, and if so, how we can effectively revise these benchmarks to better simulate more realistic problem distributions in the real world. In this paper, we posit that while the real world problems consist of a great deal of long-tail problems, existing benchmarks are overly populated with a great deal of similar (thus non-tail) problems, which in turn, leads to a major overestimation of true AI performance. To address this challenge, we present a novel framework of Adversarial Filters to investigate model-based reduction of dataset biases. We discuss that the optimum bias reduction via AFOPTIMUM is intractable, thus propose AFLITE, an iterative greedy algorithm that adversarially filters out data points to identify a reduced dataset with more realistic problem distributions and considerably less spurious biases. AFLITE is lightweight and can in principle be applied to any task and dataset. We apply it to popular benchmarks that are practically solved -ImageNet and Natural Language Inference (SNLI, MNLI, QNLI) -and present filtered counterparts as new challenge datasets where the model performance drops considerably (e.g., from 84% to 24% for ImageNet and from 92% to 62% for SNLI), while human performance remains high. An extensive suite of analysis demonstrates that AFLITE effectively reduces measurable dataset biases in both the synthetic and real datasets. Finally, we introduce new measures of dataset biases based on K-nearest-neighbors to help guide future research on dataset developments and bias reduction.

Summary: This paper hypothesizes that even though we are able to achieve very impressive performance on benchmark datasets as of now (e.g. image net), it might be due to the fact that benchmarks themselves have biases. They introduce an algorithm that selects more representative data points from the dataset that allow to get a better estimate of the performance in the wild. The algorithm ends up selecting more difficult/confusing instances.
This paper is easy to read and follow (apart from some hickup with a copy of three paragraphs), but in my opinion of limited use/impact.
Comments:
1) There is a repetition of the " while this expression formalizes.." paragraph and the next paragraph and the paragraph "As opposed to .." is out of place. Please fix
2) I am not sure
- What applications the authors suggest. They seem to say that benchmark authors should run their algorithm and make benchmarks harder. To me it seems that benchmarks become harder because you remove most important instances from the training data (so Table 4 is not surprising - you remove the most representative instances so the model can't learn)
- how practically feasible it is.  Even if in previous point I am wrong, the algo requires retraining the models on subsets (m iterations). How large is this m?
3) Other potential considerations:
-  When you change the training size, the model potentially needs to be re-tuned (regularization etc) (although it might be not that severe since the size of the training data is preserved at t)
- How do u chose the values of hyperparams (t, m,k eta), how is performance of your algorithm depends on it
4) I don't see any good baselines to compare with - what if i just chose instances that get the highest prediction score on a model and remove these. How would that do? For NLP (SNLI) task i think this would be a more reasonable baseline than just randomly dropping the instances,
5) I wonder if you actually retrain the features after creating filtered dataset, new representation would be able to recover the performance.
I read authors rebuttal and new experiments that show that the models trained on filtered data generalize better are proving the point, thanks. Changing to weak accept


Mixed precision training (MPT) is becoming a practical technique to improve the speed and energy efficiency of training deep neural networks by leveraging the fast hardware support for IEEE half-precision floating point that is available in existing GPUs. MPT is typically used in combination with a technique called loss scaling, that works by scaling up the loss value up before the start of backpropagation in order to minimize the impact of numerical underflow on training. Unfortunately, existing methods make this loss scale value a hyperparameter that needs to be tuned per-model, and a single scale cannot be adapted to different layers at different training stages. We introduce a loss scaling-based training method called adaptive loss scaling that makes MPT easier and more practical to use, by removing the need to tune a model-specific loss scale hyperparameter. We achieve this by introducing layer-wise loss scale values which are automatically computed during training to deal with underflow more effectively than existing methods. We present experimental results on a variety of networks and tasks that show our approach can shorten the time to convergence and improve accuracy compared to the existing state-of-the-art MPT and single-precision floating point.

The authors propose an adaptive loss scaling method during the backpropagation stage for the mix precision training to reduce the underflow. Compared with the previous work, which scales the loss by human design, and needs to be consistent in all layers. The authors state that they can decide the scale rate layer by layer automatically to reduce the underflow in a low precision situation.
They calculate the scale rate using the statistic information of the layer weight and gradient. By adaptively scale each layer’s loss and gradient, this method can reduce the underflow rate better than the previous work. Additionally, the authors claim that the computation overhead is not significant, so it is efficient to use rather than searching from a set of fix scale rates.
The experiments present on image classification and objective detection benchmarks. From the result, we can see that the adaptive loss scale reaches a relatively high point on all the tasks.

Pros:

- The method is straight forward and easy to understand. The motivation is good. They get some impressive results on ResNet110 and SSD512 comparing with the fixed scaling method. Besides, they give some analysis of their advantages and disadvantages in different networks, which looks promising to me.

Cons:
- One question in Section 3.2.1, the assumption that w, g, p can be treated as the random variable with Gaussian distribution seems not natural in the training process. Especially p is a zero-mean distribution. The cited paper uses this assumption in a more convincing case, such as the weight initialization task. Notice that He et al., 2015 claim that the product of weight and gradient can be a zero-mean normal distribution is based on the weight is a symmetric distribution around zero, which is not true in neither this paper’s assumption nor the real training situation.
- In the objective detection part, I can not find which dataset the authors use. Though the author state that they follow Liu et al., 2016 ’s work, there are also several tasks in Liu’s paper, and I can not directly match the resulting point with any of those tasks, which makes me hard to confirm the experiment result.
- The experiment setting is unclear. Here are two questions. 1, What is the initial scale at the last layer? It should be manually designed in the experiment, and I think this value may affect the other layer’s scale as well. If the algorithm is robust for this scale, it is better to show some study on that. 2, What update frequency is used in the experiment? The authors say that the overhead can be reduced by reducing the frequency, but they do not clearly show which frequency they use in their experiment, if the frequency does not affect the performance, it is also better to claim or show some study on that.
Minor comments:
- Figures can use a larger font.  Figures 4a and 4b can be aligned better.


Training generative adversarial networks requires balancing of delicate adversarial dynamics. Even with careful tuning, training may diverge or end up in a bad equilibrium with dropped modes. In this work, we introduce a new form of latent optimisation inspired by the CS-GAN and show that it improves adversarial dynamics by enhancing interactions between the discriminator and the generator. We develop supporting theoretical analysis from the perspectives of differentiable games and stochastic approximation. Our experiments demonstrate that latent optimisation can significantly improve GAN training, obtaining state-of-the-art performance for the ImageNet (128 × 128) dataset. Our model achieves an Inception Score (IS) of 148 and an Fréchet Inception Distance (FID) of 3.4, an improvement of 17% and 32% in IS and FID respectively, compared with the baseline BigGAN-deep model with the same architecture and number of parameters.

The paper proposes a novel training scheme for GANs, which leads to improved scores w.r.t. state-of-the-art. The idea is to update the sampled latent code in a direction improving the inner maximization in the min-max problem. The work considers both, the gradient direction and a direction motivated by the natural gradient which is shown to yield excellent performance. The overall scheme is motivated as an approximation to the (prohibitively expensive) symplectic gradient adjustment method.
While the work contains some typos, it was easy to read and overall very well written. The experimental results are impressive and clearly validate the usefulness of the approach. Therefore, I recommend acceptance of the paper.
On the theoretical side, I feel some parts can be improved. I'm willing to further increase my rating if some of the following points are addressed:
1. The connection to SGA is a bit hand-wavy, as terms are dropped or approximated with the only reasoning that they are difficult to compute. In some approximations (e.g. Gauss-Newton approximation of the Hessian) one can argue quite well that certain second-order terms can be dropped under some assumptions (e.g. mild nonlinearity, or vanishing near the optimum).
2. I had some troubles Sec 4 related to the natural gradient.
(a) What is p(t | z)? Is it the same p as in (37), Appendix C? I find the argument that the hinge loss pushing D(G(z)) < 0 during training hand-wavy, as for natural gradient p(t | z) should always be a valid distribution. There are also some typos which made it hard to follow the arguments there. It probably should read "an ideal generator can perfectly fool the discriminator" (not perfectly fool the generator).
(b) Maybe it would be easier to directly argue that one wishes to approximate SGA as well as possible, rather than taking a detour through the natural gradient.
3. Why is the increment \Delta z clipped in Algorithm 1? Is there a theoretical justification? If the goal of the clipping is to stay inside the support of the uniform distribution, shouldn't it rather be z' = [z + \Delta z]? A soft clipping (e.g. performing a mirror descent step) might give better gradients.
Typos, minor comments (no influence on my rating):
- In Eq. (5), it should be \partial^2 in the last "Hessian-block" multiplication.
- presribed -> prescribed
- Eqs. (7) and (8) might be easier to parse if one uses a different notation to distinguish between total derivative and partial derivative, i.e., write on the left side df(z') / d\theta. Also, I think it is clearer to write in the last terms in (7) and (8) \partial f(z') / \partial \delta z instead of \partial f(z') / \partial z'.
- In Appendix B.1, shouldn't it be \gamma=(1+\eta)/2 instead of  \gamma=\eta (1+\eta)/2?
- I've found ELU activations to work well in GAN models which involve the Jacobian w.r.t. z. Maybe it can stabilize things here as well.


We consider the task of few shot link prediction, where the goal is to predict missing edges across multiple graphs using only a small sample of known edges. We show that current link prediction methods are generally ill-equipped to handle this task-as they cannot effectively transfer knowledge between graphs in a multigraph setting and are unable to effectively learn from very sparse data. To address this challenge, we introduce a new gradient-based meta learning framework, Meta-Graph, that leverages higher-order gradients along with a learned graph signature function that conditionally generates a graph neural network initialization. Using a novel set of few shot link prediction benchmarks, we show that MetaGraph enables not only fast adaptation but also better final convergence and can effectively learn using only a small sample of true edges.

Overview: In this paper, a meta-learning approach is proposed to perform link prediction across multi-graphs with scarce data. To do so, each graph is treated as a link prediction "task". Different from the tasks in conventional meta-learning, the graphs here are generally non i.i.d. Based on the variational graph auto-coders, the method learns two important components: the global parameters used for GNNs and a graph signature function used to modulate the parameters of inference model based on the history of observed training graphs.
The idea of formulating link prediction as a few-shot learning problem and solving it via multi-graph meta-learning is novel. The approach basically takes advantage of meta-learning and is expected to generalize well across graphs. The numerical study is extensive in discussing the properties of meta-graph and showing more attractive empirical performance than the existing approaches.
Comments:
1. The explanation of Figure 1 needs to be improved. In its current form, the figure does not illustrate the idea of this paper very clearly. It would be better to provide more explanations on the graphical model for meta-graph and the meta-graph architecture in the context.
2. The description of Algorithm 1 is confusing in some places. For example, the operation in Line 6 is commented as *compute graph signature*. However, what's actually going on is to set the signature function to stop gradient computation which seems not matching the comment.
3. In section 4.2, the 5-gradient update AUC is reported to show the fast adaptation of meta-graph. However, in my opinion, it could be much more informative to show the accuracies of different gradient steps, like what has done in the experimental study of MAML. I don’t think the current experimental results can well explain the property of fast adaptation.
=== update after author response ===
Thank you for your response.   I find my main concerns properly addressed in the feedback.


Reinforcement learning (RL) has seen great advancements in the past few years. Nevertheless, the consensus among the RL community is that currently used model-free methods, despite all their benefits, suffer from extreme data inefficiency. To circumvent this problem, novel model-based approaches were introduced that often claim to be much more efficient than their model-free counterparts. In this paper, however, we demonstrate that the state-of-the-art model-free Rainbow DQN algorithm can be trained using a much smaller number of samples than it is commonly reported. By simply allowing the algorithm to execute network updates more frequently we manage to reach similar or better results than existing model-based techniques, at a fraction of complexity and computational costs. Furthermore, based on the outcomes of the study, we argue that the agent similar to the modified Rainbow DQN that is presented in this paper should be used as a baseline for any future work aimed at improving sample efficiency of deep reinforcement learning.

This paper presents an emprical study of how a properly tuned implementation of a model-free RL method can achieve data-efficiency similar to a state-of-the-art model-based method for the Atari domain.
The paper defines r as ratio of network updates to environment interactions to describe model-free and model-based methods, and hypothesizes that model-based methods are more data-efficient because of a higher ratio r. To test this hypothesis, the authors take Rainbow DQN (model-free) and modify it to increase its ratio r to be closer to that SiMPLe (model-based). Using the modified verison of Rainbow (OTRainbow), the authors replicate an experimental comparison with SiMPLe (Kaiser et al, 2019), showing that Rainbow DQN can be a harder baseline to beat than previously reported (Figure 1). This paper raises an important point about empirical claims without properly tuned baselines, when comparing model-based to model-free methods, identifying the amount of computation as a hyperparameter to tune for fairer comparisons.
I recommend this paper to be accepted only if the following issues are addressed. The first is the presentation of the empirical results. In Figure 1, OTRainbow is compared against the reported results in (Kaiser et al, 2019), along with other baselines, when limiting the experience to 100k interactions. Then, in Figure 2, human normalized scores are reported for varying amounts of experience for the variants of Rainbow, and compared against SiMPLe with 100k interactions, with the claim that the authors couldn't run the method for longer experiences. Unless a comparison can be made with the same amounts of experience, I don't see how Figure 2 can be interpreted objectively. In any case, the results in Figure 1 and the appendix are useful for showing that the baselines used in prior works were not as strong as they could be.
The second has to do with the interpretation of the results. The paper chooses a single method class of model-based methods to do this comparison, namely dyna-style algorithms that use the model to generate new data. But models can also be used for value function estimation (Model Based Value Expansion) and reducing gradient variance(using pathwise derivatives). The paper is written as if the conclusions could be extended to model-based methods in general. Can we get the same conclusions on a different domain where other model-based methods have been successful; e.g. continuous control tasks? A way to improve the paper would be to make it clear from the beginning that these results are about Dyna-style algorithms in the Atari domain.


We design a new algorithm for batch active learning with deep neural network models. Our algorithm, Batch Active learning by Diverse Gradient Embeddings (BADGE), samples groups of points that are disparate and high-magnitude when represented in a hallucinated gradient space, a strategy designed to incorporate both predictive uncertainty and sample diversity into every selected batch. Crucially, BADGE trades off between diversity and uncertainty without requiring any hand-tuned hyperparameters. While other approaches sometimes succeed for particular batch sizes or architectures, BADGE consistently performs as well or better, making it a useful option for real world active learning problems.

The paper proposes a new method for active learning, which picks the samples to be labeled by sampling the elements of the dataset with highest gradient norm, under some constraint of diversity. The aforementioned gradient is computed w.r.t. the predicted label (rather than the true label, that is unknown) and diversity is achieved by sampling via the k-MEANS++ algorithm.
The paper is well written and while the experiments look thorough, the motivation to support the proposed method seem too weak and unconvincing as does the discussion of the results, which is why I am leaning toward rejection.
I am willing to amend my vote if the authors provide stronger (not empirical) motivations on why using the gradient norm w.r.t. the predicted label is a better metric than those in the literature, and  More comments below.

Detailed feedback:
1) The paper lacks a proper motivation as to why using the norm of the gradient is a better metric than the many others already present in the literature. In particular, I cannot think of any case where it would be best to use that than the entropy of the network’s output distribution, even though the empirical results seem to suggest otherwise. Specifically, while I believe that in many cases it will be similarly good, if we consider the case when the network is able to rule out most of the classes but is unsure on a small fraction of them, the entropy will better reflect this uncertainty than the norm of the gradient of the predicted class.
Generally speaking, I believe that the use of the norm of the gradient of the predicted class should be much better motivated, being the core idea of the paper. Stating that it is cheap to compute and empirically performs as well as k-DPP in two experiments is not convincing enough in my opinion.
2) I wonder how much of the performance of BADGE is due to k-MEANS++ and how much to the choice of using the gradient norm. Please perform an ablation study where you can e.g., replace the gradient norm with the entropy, or replace k-MEANS++ with random sampling, and discuss the results.
3) How is the embedding “ground set” space determined for k-MEANS++? How are the centroids determined? In which space? It is unclear to me how k-MEANS++ is used in the context of the norm of the gradients. Please improve the explanation in the main text.
4) Please add a curve for k-DPP to the plots in the main text, rather than having separate plots for it in the appendix. Also, it would be interesting to compare against Derezinski, 2018 as well, if that’s the current state of the art (which is what I infer from your text, but I might be wrong).
5) The paper builds on the claim that the gradient norm w.r.t. the prediction is a lower bound for the gradient norm induced by any other label, yet Proposition 1 that proves it is in Appendix B. This prove is central to the proposed idea and should be in the main text.
6) The authors claim that to capture diversity they collect a batch of examples where the gradients span a diverse set of directions, but it’s unclear to me that k-means++ actually accomplishes that. Where is the *direction* of the gradient taken into account in the algorithm?
7) The “discussion” section is really a “conclusion” one, and indeed a proper in-depth discussion of the experiments is missing. Please expand the comments on the experimental results.
8) The metric to compute the “pairwise comparison” looks quite convoluted. Is it common in the literature? If so, please add a reference. If not, can you motivate the use of this specific formula?
9) The random baseline seems to be very competitive. Why is that? Please provide your intuition. Could this be indicative that the baselines have not been tuned properly?
10) Introduction: the sentence “[deep neural networks] successes have been limited to domains where large amounts of labeled data are available” is incorrect. Indeed, neural networks have been used successfully in many domains where labelled data is scarce, such as the medical images domain for example. Please remove the sentence.
11) Introduction: please add a sentence to explain what a version-space-based approach is.
12) Is Figure 2 the average over multiple runs or a single run?
13) Notation: please do not use g for the gradient (g^y_x) and for the intermediate activations (g(x; V)).
14) The lower margin seem too wide. Please make sure you respect the formatting style of the conference.
Minor:
- Notation: if you must shorten g^{\hat{y}}_{x} please do so with \hat{g}_{x} and equivalently shorten g^{y}_{x} as g_{x}
- Notation: in the pairwise comparison, please don’t reuse i to denote an algorithm (it is used a few lines before to compute the labeling budget)
- Please add reference to Appendix A when k-MEANS++ is first referred to in page 2.
- Page 3,  when Proposition 1 is mentioned add reference to the location where it’s defined.
Typos:
- Page 2: expenive -> expensive
- Page 5: Learning curves. “Here we show ..” -> Remove “here”
- Figure 3: pariwise -> pairwise
- Page 7: Apppendx E
----------------------
Updated review:
I thank the authors for for taking the time to address all my comments, and clarifying some of the misunderstandings I had. I am happy to revise my score accordingly.


Model-based reinforcement learning (MBRL) aims to learn a dynamic model to reduce the number of interactions with real-world environments. However, due to estimation error, rollouts in the learned model, especially those of long horizons, fail to match the ones in real-world environments. This mismatching has seriously impacted the sample complexity of MBRL. The phenomenon can be attributed to the fact that previous works employ supervised learning to learn the one-step transition models, which has inherent difficulty ensuring the matching of distributions from multi-step rollouts. Based on the claim, we propose to learn the synthesized model by matching the distributions of multi-step rollouts sampled from the synthesized model and the real ones via WGAN. We theoretically show that matching the two can minimize the difference of cumulative rewards between the real transition and the learned one. Our experiments also show that the proposed model imitation method outperforms the state-of-the-art in terms of sample complexity and average return.

This paper considers the model-based reinforcement learning (MBRL)
problem in which one of the main parts is to learn the transition
probability matrix of the underlying MDP (called Transition Learning -
TL). The motivation is that if the transition model can be learnt from
some real-world trajectories, the agent can improve by interacting with
simulated environment built from the learnt transition model; hence,
the overall number of real-world interactions is reduced.
Given a policy $\pi(a|s)$, and a transition $T(s’|s,a)$, the occupancy
measure $\rho(s,a)$, defined in Eq (2) in the paper, can be
interpreted as the discounted distribution of state-action pairs in
the rollouts. The main idea is to use Wasserstein-GAN (WGAN) to match
the distribution of $p(s,a,s’) = \rho(s,a) T(s’|s,a)$. In particular,
given a policy, the idea is to get some real-world trajectories and
feed into the WGAN framework in which the generator tries to generate
synthesized trajectories and the discriminator tries to discriminate
between them. These trajectories are then used in policy optimization
to obtain a new policy.
The authors presented two theoretical results: 1) Consistency for
WGAN: if WGAN is trained optimally, then the synthesized transition is
the same as the real transition; and 2) Error bound for WGAN: the
difference between cumulative reward under synthesized transition and
that under real transition is upper bounded by a linear function of
WGAN training error. On the experimental side, the authors used the
model-based benchmark library (MBBL, Wang et al. 2019) to compare the
proposed algorithm with several existing algorithms on four MuJoCo
tasks: Hopper, HalfCheetah, Ant, and Reacher.
Overall, I think this work is positive. The theoretical part is
technically sound (there might be an error in the proof of Proposition
1 but it is fixable). The experimental part is also sensible, although
it would be good to include the performance comparisons for other
tasks in MBBL. Details in the comments below.
Main   comments/suggestions:
- MBBL (Wang et al. 2019) has many other tasks (18 in total, the
  authors only include 4 in this work). It would be good if the
  authors can also include the performance of the proposed algorithm
  with respect to those tasks.

- The claim at the end of Proposition 1’s proof (in side the proof), in my assessment, is not
  established. It is clear from the proof that $p(s,a,s’) = p’(s,a,s’)$ is
  a sufficient condition for $\rho_{T}(s,a) = \rho_{T’}(s,a)$, but I’m
  not convinced that it is also a necessary condition. Note that
  $\rho_{T}(s,a)$ and $\rho_{T’}(s,a)$ are two unique solutions of two
  *different* Bellman equations. It would be good if the authors can
  provide the detailed proof for the necessary condition if it is
  true. Nevertheless, the statement of Proposition 1 only claims the
  sufficient condition so this is fixable.

Other minor comments:
- End of Section 1, page 2: the notations $T$, $T’$, $R(T)$, $R(T’)$ are used but not introduced until Section 3.
- Typo, page 2, last paragraph: “IfD method” ==> “LfD” method?
- Algorithm 1, page 6: what are the $\phi$ and $w$ parameters? It looks to me that they are taken from WGAN paper but with no explanation.
- Two references “Syed et al. (2008a)” and “Syed et al. (2008b)” are actually the same.
- Reference “Langlois et al. (2019)” should be corrected as “Wang et al. (2019)” -- see https://arxiv.org/abs/1907.02057


Recent work has demonstrated how predictive modeling can endow agents with rich knowledge of their surroundings, improving their ability to act in complex environments. We propose question-answering as a general paradigm to decode and understand the representations that such agents develop, applying our method to two recent approaches to predictive modeling -action-conditional CPC (Guo et al., 2018) and SimCore (Gregor et al., 2019). After training agents with these predictive objectives in a visually-rich, 3D environment with an assortment of objects, colors, shapes, and spatial configurations, we probe their internal state representations with a host of synthetic (English) questions, without backpropagating gradients from the question-answering decoder into the agent. The performance of different agents when probed in this way reveals that they learn to encode detailed, and seemingly compositional, information about objects, properties and spatial relations from their physical environment. Our approach is intuitive, i.e. humans can easily interpret the responses of the model as opposed to inspecting continuous vectors, and model-agnostic, i.e. applicable to any modeling approach. By revealing the implicit knowledge of objects, quantities, properties and relations acquired by agents as they learn, question-conditional agent probing can stimulate the design and development of stronger predictive learning objectives.

########## Post-Rebuttal Summary ###########
The authors engaged actively in the rebuttal discussion and in the process we were able to concretize the motivation of the submission (as a result in increased my score). However, I think that the submission in its current form lacks experimental analysis of the proposed conditional probes, especially the trade-offs on the reliability of the representation analysis when performed with a conditional probe as well as a clear motivation for the need of a language interface. I can therefore not recommend the submission in its current form for acceptance.
(for detailed discussions + suggested experiments please see rebuttal discussions)
#######################################
## Paper Summary
The paper tackles the problem of analyzing the information captured in the learned representation of a deep neural network. It proposes to replace the commonly used "probing networks" that try to directly infer the information from the learned representation, with a language interface in the form of a QA model which is trained post-hoc without propagating gradients into the learned representation. The authors argue that such a language interface provides a more natural interface for probing the information in a learned representation. The paper shows representation analysis results for the internal representations of agents trained on an exploratory task in a simple, simulated household environment similar to DM Lab. The authors conduct additional analysis on representations learned with auxiliary generative and contrastive objectives.
## Strengths
- the language of the paper is clear and easy to follow
- the paper covers the related work well
- the provided explanations help understand the content of the paper
- the analysis of how the information captured in the agent's representation changes over the course of a trajectory is interesting (more such visualizations in the appendix would be nice!)
## Weaknesses
(1) the motivation for the proposed problem does not convince: why do I need to train a Q/A system to infer which components of the true state are captured by the learned representation? For each of the properties of the environment I could train a probing network (as is done e.g. in [1,2]) and would get much more precise answers; ground truth labels need to be available whether I train QA or probing networks. The authors provide two motivations for the proposed approach which both do not convince me:
	(a) QA "provides an intuitive investigation tool for humans": I cannot imagine a workflow in which researchers would prefer to ask questions to their model over a plot showing explicit regression accuracies aggregated across many data samples (which the conventional probing networks provide).
	(b) "the space of questions is open-ended, [...] enabling comprehensive analysis of [an agent's] internal state": for each new question type we need to provide a sufficiently large number of question-answer-pairs to train the QA system for this question type. we could therefore also train a probing network using the same labels and would get a better overview of whether the state information is captured in the representation. I fail to see how substituting probing networks with a language interface helps the investigation of the representation's properties.

(2) the paper only provides minor novelty: the main proposal is to replace the probing networks, which were extensively used in prior work, with a natural language interface; a proposal that does not seem to provide a clear advantage (see above). The paper does not provide any further technical novelty.

(3) it is possible that the analysis of the contrastive model could improve substantially with a different sampling scheme of the negative examples: maybe sampling from different sequences makes the task of discriminating too easy so that the model is not encouraged to learn a rich representation. It would be interesting to see whether the representation captures more information if a more standard contrastive objective is chosen that discriminates between future frames with different offset in the same sequence (see for example the objective in [3]).
(4) it seems that both environment and chosen task will have high influence on the representation the agent can learn from the collected data. Therefore the fact that the authors evaluate their approach only on a single environment / task combination, both of which they introduce themselves, weakens any conclusions the authors draw from their experiments. It would help strengthen the message of the paper to apply their methodology to previously proposed environment / task combinations, for example in the AI2-THOR environment [4]
[Novelty]: minor
[technical novelty]: minor
[Experimental Design]: not comprehensive
[potential impact]: low
#########################
[overall recommendation]: Reject - In its current form the paper does not provide a convincing argument for why learning a language interface for probing a representation is better than learning the usual probing networks. Further there are some doubts on the setup of the contrastive objective and the paper lacks comprehensive evaluation on standard environments. Therefore I cannot recommend acceptance in its current form.
[Confidence]: High
[1] Neural Predictive Belief Representations, Guo et al., 2018
[2] Shaping Belief States with Generative Environment Models for Reinforcement Learning, Gregor et al., 2019
[3] Representation Learning with Contrastive Predictive Coding, Van den Oord et al., 2018
[4] AI2-THOR: An Interactive 3D Environment for Visual AI, Kolve et al., 2017
#### Final Rebuttal comment (to make it visible to the authors) ####
I understand the authors point that every method for probing representations can potentially be flawed in that the probing mechanism might not be expressive enough to extract the information that is indeed present in the representation. The point I was trying to make in my previous response is, that in the case of conditional probes that try to solve multiple such "probing tasks" in parallel, such uncertainties accumulate because the probing mechanism might trade off performance on one task for performance on another (if solving all of them at once is too challenging). If the submission's key contribution is to make probes conditional this seems like a trade-off that should be experimentally investigated, as it is vital to practitioners how much they can trust their probing method to extract the relevant information if it is actually in the representation.
One possible experiment would be to train an unconditional probe per attribute (maybe on a subset of all possible attribute-object combinations) and then a conditional probe across all of them to show that the conditional probe's assessments of the representation agree with the ensemble of unconditional probes. In my previous response I additionally raised the point that even if it can be shown that conditional probes are reliable, the authors still need to provide arguments why that conditioning should work via a language interface, not for example a symbolic/one-hot interface. If the claim is that this allows for generalization, it should again be shown that this generalization does not increase the error of the probe substantially, such that meaningful conclusions about the representation are still possible.
Regarding testing on more diverse environments: the questions these experiments are supposed to answer (i.e. how reliable are conditional probes) are inherently empirical, so verifying across diverse environments will make the analysis more conclusive.
I agree with the author's point that training 7k unconditional probes would not be practical, probably the current approach would be to train an image decoder that then reconstructs a top-down view of the whole scene. This, however, would have the same problems as a conditional probe, i.e. the probing decoder could trade-off performance between reconstructing different parts of the image. Therefore, I agree with the authors that investigating conditional probes is an interesting direction, but the submission does not provide a comprehensive analysis of this question.
On a final note, I appreciate the continued, factual discussion with the authors and think that the refinement of the focus towards conditional vs unconditional probes is a step in the right direction. To acknowledge that I am raising my score from "reject" to "weak reject". Yet, I think that the work in its current form lacks experimental analysis of the proposed conditional probes. During the rebuttal discussion I highlighted concerns and proposed possible experiments. I cannot recommend acceptance of the submission in its current form, but I encourage the authors to incorporate the refined motivation and add more comprehensive experimental evaluation for a possible resubmission.


A fundamental challenge in artificially intelligence is to build an agent that generalizes and adapts to unseen environments. A common strategy is to build a decoder that takes a context of the unseen new environment and generates a policy. The current paper studies how to build a decoder for the fundamental continuous control environment, linear quadratic regulator (LQR), which can model a wide range of real world physical environments. We present a simple algorithm for this problem, which uses upper confidence bound (UCB) to refine the estimate of the decoder and balance the exploration-exploitation trade-off. Theoretically, our algorithm enjoys a O √ T regret bound in the online setting where T is the number of environments the agent played. This also implies after playing O 1/ 2 environments, the agent is able to transfer the learned knowledge to obtain an -suboptimal policy for an unseen environment. To our knowledge, this is first provably efficient algorithm to build a decoder in the continuous control setting. While our main focus is theoretical, we also present experiments that demonstrate the effectiveness of our algorithm.

# Summary
- The paper proposes a UCB-inspired algorithm for a contextual LQR problem. The problem itself is introduced in this paper and is similar in spirit to CMDPs, with the difference that instead of learning a mapping from context to transition matrix, a mapping from context to matrices [A, B] figuring in the system dynamics of LQR is learned.
- The proposed algorithm is an online-learning algorithm shown to have sublinear regret in the number of experienced environments. A toy experiment with a 2D moving mass is presented to illustrate the theory.
# Decision
Although the problem setting is interesting and it is encouraging to have a guarantee, several important unclear points in the paper and a missing comparison to a straightforward baseline stop me from recommending it for publication in its present form. I detail my concerns below.
# Concerns
1) First, a conceptual question. I can see a straightforward algorithm that can learn the linear mapping \theta from context to [A, B] as follows.
        - In episode k, obtain trajectory (x_{1:H}, u_{1:H-1})
        - By least squares, find [A, B] from the obtained trajectory
        - Since context [C, D] is observed, find \theta : [C, D] -> [A, B] again by least squares
One can do this using data from K episodes if needed, one can sequentially update the controller for collecting data, etc.
=>  A comparison to such a basic approach should be definitely included in the paper, in my opinion.
2) The authors might argue that the algorithm suggested above has no guarantee. I would be curious to hear in this regard a comment on the practical implementation suggested in the paper. Namely, after deriving the bounds etc., the authors make further approximations and modifications in the practical algorithm. From my point of view, these modifications defeat the purpose of the bounds, because then only empirical evaluation can confirm that these approximations have not destroyed the analysis. Alternatively, one needs to incorporate the introduced approximation errors in the analysis. In more detail,
        - Eq. (9) is not solved exactly but by random sampling. In the 2D toy task, it may be OK, but in higher-dimensional spaces, a significant error can be introduced which is not accounted for.
        - More importantly, the UCB bound \beta in Eq. (11) is not used at all in the experiments.
            => To my understanding, it is the crucial point of UCB to use the UCB-bound. If it is not used, how should one judge the resulting algorithm?
3) This is a concern regarding clarity. I didn't get (i) if matrices [Q, R] are context-dependent or not and (ii) if the agent observes them or not. This is not clearly communicated in the text.
=> Clarify whether [Q, R] are context-dependent and observed.
# AFTER REBUTTAL
After authors' clarifications and improvements on the paper, I update my score to weak reject. The reason I am still against acceptance is the lack of stronger empirical evaluations. As R4 pointed out, some clarifications on the side of the algorithm are also required.


Transformers have replaced long-short term memory and other recurrent neural networks variants in sequence modeling. It achieves state-of-the-art performance on a wide range of tasks related to natural language processing, including language modeling, machine translation, and sentence representation. Lossless compression is another problem that can benefit from better sequence models. It is closely related to the problem of online learning of language models. But, despite this ressemblance, it is an area where purely neural network based methods have not yet reached the compression ratio of state-of-the-art algorithms. In this paper, we propose a Transformer based lossless compression method that match the best compression ratio for text. Our approach is purely based on neural networks and does not rely on hand-crafted features as other lossless compression algorithms. We also provide a thorough study of the impact of the different components of the Transformer and its training on the compression ratio.

Summary:
The paper investigates using the transformer architecture for neural network-based lossless compression of text. The resulting model, obtained through a thorough investigation of the architecture hyper-parameters are on par with standard SOTA compression. The paper is well-written. In particular the authors have done a great job reviewing existing compression literature and positioning their method within the space of prior work.
Recommendation: Weak Reject
While the paper considers an interesting application of the Transformer architecture, and is well-written, it is of limited novelty. Specifically, the bulk of the paper is concerned with describing experimental results of a thorough (but standard) hyper-parameter search - considering things like Transformer context size, learning rate (schedule), number of layers and key, value, query dimensionality; and does not offer any new architectural modifications / insights.
Furthermore, only a single dataset - enwik8 - is considered in the experimental validation and little attention is given to the description of the dataset split and any distribution differences between splits. Taken together, the existing experimental setup potentially creates an unfair advantage for the neural network-based methods - while the standard methods can be expected to perform similarly across a wide range of datasets / texts, the neural-network based methods have been trained and tested on very similar data and could be expected to perform well on these data, but not in case of a distributional shift (e.g. compressing legal texts instead of Wikipedia). The paper does not answer the question of whether or not this is true.
Furthermore, similar to autoregressive models, transformers are known to be slow at inference time. I expect this to lead to very slow decoding. Therefore, methods in table 1 should be compared in compression/decompression time to give a better overview of the practical impact of this work.
Taken together, in its current form the paper may be better suited for a workshop publication rather than a full conference paper.
Major comments:
1. For reasons mentioned above, the paper should include additional experimental evaluation. In particular, it should consider the effect of training the model on one dataset, but evaluating it on another dataset; and discuss how differences in performance (if any) compare to standard methods.
2. Compression/decompression times of the proposed method should be compared against the other compression methods in table 1. I expect the proposed transformer to be slow at decompressing.
3. The paper does not contain the loss that the transformer model was used to optimize. I assume that it is the softmax cross entropy, but this is worth mentioning explicitly. It would also be worthwhile to explain the training procedure - for how many epochs was the model trained (see also next question), what was the dataset size?
4. Description of the “training with revisits” is not very clear. My understanding is that it resembles a pass through the data, where some of it is considered again at specific intervals. My first assessment is that this should not be necessary - the data should already be considered multiple times during the training process.
a) The authors should provide a more detailed description of the training-with-revisits procedure, contrasting it specifically with a procedure where revisits are not done (i.e. normal training).
b) If the goal of the revisits training is to observe some training examples more than once, then it would be very interesting if simply training for a longer time (several epochs == passes through the data) has a similar effect.
c) Is there any motivation for the choice of the revisits hyper-parameters F and M? Was a different batch size used during the revisits training? Is the learning rate evolved during the revisits training phase or is it still decayed?
Minor comments:
1. There is some prior work on using Neural Networks for lossless image compression (e.g. [1], [2]. [3] that achieves SOTA compression ratios compared to standard methods. It may be interesting for the readers to mention these results. In particular the authors’ statement that “[...] purely neural network based models are still far from state of the art [...]” may give the wrong impression to the readers.
2. The authors mention that they “[...] propose several improvements to its (the Transformer) architecture and training to accelerate and stabilize [...] training”. In my view, the experiments described in the paper resemble a hyper-parameter search more than architectural improvements. The authors may want to clarify in the text which specific improvements they refer to.
3. Page 1, last paragraph: “[...] of all the important component [...]” -> “[...] of all the important components [...]”
4. Page 3: “[...] attention span size across all layers as it suggested [...]” -> “[...] attention span size across all layers as was suggested [...]”
5. Page 3: Missing references.
6. Page 3: Use of small n and capital N when talking about n-grams. Should be made consistent.
7. Page 8 (Conclusion): “wihtout” -> “without”
[1] F. H. Kingma, P. Abbeel, and J. Ho. Bit-Swap: recursive bits-back coding for lossless compression with hierarchical latent variables. In International Conference on Machine Learning (ICML), 2019.
[2] Emiel Hoogeboom, Jorn W. T. Peters, Rianne van den Berg, and Max Welling. Integer Discrete Flows and Lossless Compression. arXiv e-prints, 2019.
[3] Jonathan Ho, Evan Lohn, and Pieter Abbeel. Compression with Flows via Local Bits-Back Coding. arXiv e-prints, 2019.


The dependency of the generalization error of neural networks on model and dataset size is of critical importance both in practice and for understanding the theory of neural networks. Nevertheless, the functional form of this dependency remains elusive. In this work, we present a functional form which approximates well the generalization error in practice. Capitalizing on the successful concept of model scaling (e.g., width, depth), we are able to simultaneously construct such a form and specify the exact models which can attain it across model/data scales. Our construction follows insights obtained from observations conducted over a range of model/data scales, in various model types and datasets, in vision and language tasks. We show that the form both fits the observations well across scales, and provides accurate predictions from small-to large-scale models and data.

This work proposes a functional form for the relationship between <dataset size, model size> and generalization error, and performs an empirical study to validate it. First, it states 5 criteria that such a functional form must take, and proposes one such functional form containing 6 free coefficients that satisfy all these criteria. It then performs a rigorous empirical study consisting of 6 image datasets and 3 text datasets, each with 2 distinct architectures defined at several model scales, and trained with different dataset sizes. This process produces 42-49 data points for each <dataset, architecture> pair, and the 6 coefficients of the proposed functional form are fit to those data points, with < 2% mean deviation in accuracy. It then studies how this functional form performs at extrapolation, and finds that it still performs pretty well, with ~4.5% mean deviation in accuracy, but with additional caveats.
Decision: Accept. This paper states 5 necessary criteria for any functional form for generalization error predictor that jointly considers dataset size and model size, then empirically verifies it with multiple datasets and architectures. These criteria are well justified, and can be used by others to narrow down the search for functions that approximate the generalization error of NNs without access to the true data distribution, which is a significant contribution. The empirical study is carefully done (e.g., taking care to subsample the dataset in a way that preserves the class distribution). I also liked that the paper is candid about its own limitations. A weakness that one might perceive is that the coefficients of the proposed functional form still needs to be fit to 40-ish trained NNs for every dataset and training hyperparameters, but I do not think this should be held against this work, because a generalization error predictor (let alone its functional form) that works for multiple datasets and architecture without training is difficult, and the paper does include several proposals for how this can still be used in practice.
(Caveat: the use of the envelope function described in equation 5 (page 6) is not something I am familiar with, but seems reasonable.)
Issues to address:
- Fitting 6 parameters to 42-49 data points raises concerns about overfitting. Consider doing cross validation over those 42-49 data points, and report the mean of deviations computed on the test folds. The extrapolation section did provide evidence that there probably isn't /that/ much overfitting, but cross validation would directly address this concern.
- In addition, the paper provides the standard deviation for the mean deviations over 100 fits of the function as the measure of its uncertainty, but I suspect that the optimizer converging to different coefficients at different runs isn't the main source of uncertainty. A bigger source of uncertainty is likely due to there being a limited amount of data to fit the coefficients to. Taking the standard deviation over the deviations measured on different folds of the data would be better measure of uncertainty.
Minor issues:
- Page 8: "differntiable methods for NAS." differentiable is misspelled.


We extract a controllable model from a video of a person performing a certain activity. The model generates novel image sequences of that person, according to user-defined control signals, typically marking the displacement of the moving body. The generated video can have an arbitrary background, and effectively capture both the dynamics and appearance of the person. The method is based on two networks. The first maps a current pose, and a singleinstance control signal to the next pose. The second maps the current pose, the new pose, and a given background, to an output frame. Both networks include multiple novelties that enable high-quality performance. This is demonstrated on multiple characters extracted from various videos of dancers and athletes.

This paper proposes a method to address the interesting task, i.e. controllable human activity synthesis, by conditioning on the previous frames and the input control signal. To synthesis the next frame, a Pose2Pose network is proposed to first transfer the input information into the next frame body structure and object. Then, a Pose2Frame network is applied to generate the final result. The results on several video sequences look nice with more natural boundaries, object, and backgrounds compared to previous methods.
Pros:
1. The proposed Pose2Pose successfully transfer the pose conditioned on the past pose and the input control signal. The proposed conditioned residual block, occlusion augmentation and stopping criteria seem to help the Pose2Pose network work well. Besides, the object is also considered in this network, which makes the method generalized well to the videos where human holds some rigid object.
2. The Pose2Frame network is similar to previous works but learns to predict the soft mask to incorporate the complex background and to produce shallow. The mask term in Eq. (7) seems to work well for the foreground (body+object) and the shallow regions.
3. The paper is easy to follow.
Cons:
1. Since the method is only evaluated on several video sequences, I am not sure how the method will perform on other different scenes. Results on more scenes will make the performance more convincing. I also wonder if the video data will be released, which could be important for the following comparisons.
2. As to the results of the Pose2Pose network, I wonder if there are some artifacts that will affect the performance of the Pose2Frame network. Then, there will be another question: how the two networks are trained? Are they trained separately or jointly? I assume the authors first train the Pose2Pose network, then use the output to train the Pose2Frame network. Otherwise, the artifacts from Pose2Pose will affect the testing performance of the Pose2Frame network.
3. The mask term seems to work well for the shallow part. I wonder how the straightforward regression term plus the smooth term will perform for the mask. Here, the straightforward regression term means directly regress the output mask to the target densepose mask. Will the proposed mask term perform better?