The lottery ticket hypothesis argues that neural networks contain sparse subnetworks, which, if appropriately initialized (the winning tickets), are capable of matching the accuracy of the full network when trained in isolation. Empirically made in different contexts, such an observation opens interesting questions about the dynamics of neural network optimization and the importance of their initializations. However, the properties of winning tickets are not well understood, especially the importance of supervision in the generating process. In this paper, we aim to answer the following open questions: can we find winning tickets with few data samples or few labels? can we even obtain "good" tickets without supervision? We find that winning tickets found in these scenarios are, perhaps surprisingly, competitive with winning tickets generated on the full ImageNet dataset when evaluated on ImageNet classification task.

This paper empirically studies the lottery ticket hypothesis with limited or no supervision. First, the authors use self-supervised learning to generate winning tickets, showing that "good" (reasonable) winning tickets can be found without labels. Second, the authors show that finding "good" (reasonable) winning tickets can be accelerated by a factor 5 on ImageNet by using only a subset of the data. The authors also argue that using large datasets is important to study lottery tickets, since deep networks trained on CIFAR-10 are natually sparse, making conclusions potentially misleading.
The experimental results are rich and provide more understanding of winning ticket generation with limited or no supervision. The results on self-supervised learning task (including the layer-wise pruning results) and a subset of training dataset are reasonable and kind of expected, but it is still good that this paper provide solid experimental results to verify this. As the paper observed, "none of the tickets found with limited access to labels and or data matches the accuracy of tickets found with all the labeled data when considering moderate pruning rates (more than 10% of unpruned weights)
on ImageNet. Indeed, we consistently observe a decrease in performance compared to the full overparametrized network as soon as we prune the network." In this sense, winning tickets are certainly label and data dependant. This undermines the *bold* claim in the abstract that "we provide a positive answer to both questions, by generating winning tickets with limited access to data, or with self-supervision". From my perspective, the ability to exactly perserve the accuracy while pruning the weights (see the flat regions of "Lables" curves in Figure 1,2,3,4,5) is the interesting part of the lottery ticket hypothesis. We have several different ways to achieve a descreased accuracy with a smaller network, the dynamics there may be a mixture of the lottery ticket hypothesis and standard model pruning, which needs more careful experiment design to separate different dynamics.
"using large datasets is important to study lottery tickets, since deep networks trained on CIFAR-10 are natually sparse, making conclusions potentially misleading." "The definition of “early in training” is somehow ill-defined: network
weights change much more for the first epochs than for the last ones." These two messages are important to future study of the lottery ticket hypothesis. This paper raises the issue of ill-definedness of “early in training”, but did not provide a solution.
Overall, I found that the experimental results in this paper are solid and provide more understandings of the lottery ticket hypothesis. However, I feel that the novelty of this paper is limited, and do not provide much new insights. Therefore, it does not reach the bar of being published at ICLR, from my perspective. Therefore, I say "Weak Reject".


Off-policy temporal difference (TD) methods are a powerful class of reinforcement learning (RL) algorithms. Intriguingly, deep off-policy TD algorithms are not commonly used in combination with feature normalization techniques, despite positive effects of normalization in other domains. We show that naive application of existing normalization techniques is indeed not effective, but that well-designed normalization improves optimization stability and removes the necessity of target networks. In particular, we introduce a normalization based on a mixture of on-and off-policy transitions, which we call cross-normalization. It can be regarded as an extension of batch normalization that re-centers data for two different distributions, as present in off-policy learning. Applied to DDPG and TD3, cross-normalization improves over the state of the art across a range of MuJoCo benchmark tasks.

This paper studies the problem of feature normalization in off-policy RL, more specifically, learning a Q function with continuous action from off-policy data. It shows standard feature normalization methods in supervised learning is indeed not effective for RL settings, due to the fact that a and a’ are from very different distributions with different dynamics. Since the batch of a and a’ come to model iteratively, standard normalization method suffers from this 2-periodic distribution shift. This paper proposes a normalization method, by merging a and a’ into a * single* update step to the batch normalization layer.
This paper does catch a problem and uses a straightforward solution but empirically effective, but I still have two main concerns. 1) This paper only shows benefit 4 tasks in the MoJoCo domain. 2) As the solution is relatively straightforward, with very restrictive applicable settings (particular normalization trick with particular function approximator). It’s less clear to me whether this provides enough contribution and inspiration to other work as a conference paper. I tend to vote for reject at this time.
I would like to first point out the pros of this paper from my perspective then explain my main concerns point by point. This paper does a great job of capturing the dilemma of batch normalization in RL settings. My understanding is that the problem is caused by a periodic distribution shift between a and a’. Because we have to pass a and a’ in separate batch and BatchNorm does an online updating after each batch, we are in a dilemma, as the paper pointed out. If we don’t update BatchNorm in one of them (e.g. target value) that will be biased and make BatchNorm ineffective, and if we do so we will have a systematic difference between Q(s, a) and Q(s’, a’).
Main concerns:
1) This paper only shows benefit 4 tasks in the MoJoCo domain. Given that the empirical result is pretty much the only support of the claim in this paper, the lack of more diverse experiments would weaken the contribution.
2) The dilemma is totally caused by that BatchNorm will immediately perform update according to a batch after it is input, then a lazy update will cancel this: do a single update to the BatchNorm layers, for two batches of data (a, a’). This is equivalent to the proposed solution when alpha=0.5. It needs not to be a weakness for the algorithm itself as we appreciate simple but effective algorithm. However this makes the problem itself more like a design weakness of BatchNorm and a simple patch to fix it. I doubt how much algorithmic insight this paper could contribute, to inspire related research.
Minor point: It also makes me doubt whether we really need an alpha or not.
3) As the paper pointed out “using the training mode in the target calculation would result in different mean subtractions of the Q function and its target.” This means it will have a systematic difference in Q function used to compute Q(s, a) and Q(s’, a’), but isn’t this also true for the target network since we are using a different network to compute Q(s’, a’). Eventually, if the policy converges, those differences will disappear. So why target network will have no problem but this will. In general, I’d like to see a more clear analysis about the dilemma of BatchNorm in off-policy data, and why the two simple ways won’t work.


Owing to their ability to both effectively integrate information over long time horizons and scale to massive amounts of data, self-attention architectures have recently shown breakthrough success in natural language processing (NLP), achieving state-of-the-art results in domains such as language modeling and machine translation. Harnessing the transformer's ability to process long time horizons of information could provide a similar performance boost in partially observable reinforcement learning (RL) domains, but the large-scale transformers used in NLP have yet to be successfully applied to the RL setting. In this work we demonstrate that the standard transformer architecture is difficult to optimize, which was previously observed in the supervised learning setting but becomes especially pronounced with RL objectives. We propose architectural modifications that substantially improve the stability and learning speed of the original Transformer and XL variant. The proposed architecture, the Gated Transformer-XL (GTrXL), surpasses LSTMs on challenging memory environments and achieves state-of-the-art results on the multi-task DMLab-30 benchmark suite, exceeding the performance of an external memory architecture. We show that the GTrXL, trained using the same losses, has stability and performance that consistently matches or exceeds a competitive LSTM baseline, including on more reactive tasks where memory is less critical. GTrXL offers an easy-to-train, simple-to-implement but substantially more expressive architectural alternative to the standard multi-layer LSTM ubiquitously used for RL agents in partially observable environments.

* Summary
This paper introduces architecture modifications for self-attention to stabilize transformers in reinforcement learning.
The new architecture, Gated Transformer-XL, replaces the order of the layer norm blocks to preserve an identity mapping.
Multiple existing gating layers are proposed to replace the residual connections of transformer.
The new architecture is compared against MERLIN and LSTM on DMlab-30, and further ablation studies are done on Numpad and Memory Maze.
It is noted that they use the recent V-MPO objective to train LSTM and transformer.
Their results show that transformers are able to learn in memory-intensive environments, with some gating combinations surpassing LSTM.
* Decision
This paper presents promising empirical results, however the experiments are limited, making it difficult to place in the broader work.
In addition, the contribution is incremental and not well-motivated.
I would recommend a weak rejection.
Still, I think the paper is well written and could be improved upon.
* Reasons
While the empirical results are impressive, they are not put into context.
DMlab-30 is still a relatively new environment suite and it is difficult to place the result of this paper in the context of broader work.
In addition, the comparison is against LSTM on a new objective.
While Transformer is able to beat LSTM on the same objective, it is unclear whether that is a success of the objective or the architecture.
In Numpad, the transformer architecture shows an improvement over LSTM, but no comparisons are made to any other memory-based agents.
The hyperparameter studies on Memory Maze also show improvements in memory-related tasks, but do not help in understanding of the proposed work.
The choice of architecture modification is also not well motivated.
As the paper mentions, initializing near identity has been shown to be important in the supervised learning literature.
For the topic of this paper however, I do not think this adequetly explains the instability of transformers in reinforcement learning.
In the related work section for example, the paper notes that gating mechanisms have been used to handle the vanishing gradients problem.
The paper also notes that vanishing gradients is not an issue in transformers.
Hence, it is unclear why gating would stabilize transformers for reinforcement learning.
The paper is overall well written and the ideas developed are clear.
Unfortunately, the impressive results on DMlab are not sufficient for both the lack of deeper empirical study and better theoretical motivation for the architecture modifications.


Deep neural network (DNN) has rapidly found many applications in different scenarios. However, its large computational cost and memory consumption are barriers to computing restrained applications. DNN model quantization is a widely used method to reduce the DNN storage and computation burden by decreasing the bit width. In this paper, we propose a novel cursor based adaptive quantization method using differentiable architecture search (DAS). The multiple bits' quantization mechanism is formulated as a DAS process with a continuous cursor that represents the possible quantization bit. The cursor-based DAS adaptively searches for the desired quantization bit for each layer. The DAS process can be solved via an alternative approximate optimization process, which is designed for mixed quantization scheme of a DNN model. We further devise a new loss function in the search process to simultaneously optimize accuracy and parameter size of the model. In the quantization step, based on a new strategy, the closest two integers to the cursor are adopted as the bits to quantize the DNN together to reduce the quantization noise and avoid the local convergence problem. Comprehensive experiments on benchmark datasets show that our cursor based adaptive quantization approach can efficiently obtain lower size model with comparable or even better classification accuracy.

The paper proposes a novel way to determine automatically the quantization level at each layer of
deep neural networks while training. The quantization can result in a more compact network requiring
less memory for storing the parameters (weights), with minimal drop in accuracy.
If I understand correctly, the main idea in the paper is utilizing a continuous representation of the quantization using a continuous cursor, and a new differential approach from Neural Architecture Search (NAS), to create a differential learning algorithm for the problem of finding the optimal quantization.
On the positive side, the authors present very good experimental results on popular networks like ResNet,
and improve on other method's compression accuracy, with little to no reduction in accuracy.
Howerver, the paper is not clearly and carefully enough written to convey the author's results which makes it a borderline paper in my opinion.
In my understanding, compressed (quantized) networks can have different advantages besides a more compact representation of the network:
For example, this can also reduce training time because all operations are performed with words with fewer bits - but the authors focus only on the compression ratio and do not discuss this issue. In fact, it seems that their algorithm actually requires more training time than standard training of neural networks, because you start with training w with full 32-bit representation, and then train with different quantizations simultaneously.
It would be good if the authors describe clearly how do different methods (theirs and the others mentioned) compare to each other in terms of savings of different resources (training time, compression, inference/prediction time for the trained networks etc.)  and which of these is more important (e.g. the latter in mobile devices).
Another practical issue is that using a different number of bits for each layer may complicate the design and software/hardware implementation of a network, compared to say allocating 4-bits or 8-bits to each weight. It is correct that you can reduce the overall number of bits, but you may reduce it even further if you allow a different number of bits for each individual weight. I assume that there is a price in power/memory/speed etc. to this non-homogenous property, and it may be better to get a slightly lower compression ratio provided that there is more uniformity in the network weights.  This is not a critique of this particular paper, but of the entire framework of flexible quantization.
The English of the manuscript could be greatly improved. For example:
Page 2: "..multiple bits for different layers.." - I assume the authors mean "..a different number of bits used for different layers .."
Page 2, bottom: "The authors (Wang et al. 2018) presented ..." - the sentence is long and unclear.
Many sentences starting with 'And' - for example: "And a Gumbel softmax function ..." (page 3)
There are a few vague and non-informative statements which do not contribute to the understanding of the field and the authors' contribution. For example:
- Page 3: "A typical search method is random search, however, its efficiency is not ideal." (does efficiency refer to computational efficiency here?)
- Page 4: "The reason why we add regularization item to the loss function is because the regularization can prevent overfitting to some extent" (I thought that here the main purpose of regularization is to get a trade-off between accuracy and compression).
- Page 6: Regarding parameters choice: 'a rather optimal set of them is chosen ..' - this seems quite arbitrary and it is not clear how to choose parameters for the authors' method for future architectures and datasets.
There are also terms that are known to expert but could use a short explanation/reference - for example:
- Adam optimizer
- Cosine annealing
- Gumble Softmax
The mathematical equations are definitions are not clear enough and contain errors:
- Eq. (1): there is (x',y') twice in the equation, but in the sentence afterward there is also (x,y).
There are D_T, D_V with and without tilde.
The space of maximization of w for a given quantization is not defined (I'm assuming all weights vectors w
which can be represented using the number of bits in the quantization).
- Eq. (3) - the loss is loosely defined - what is 'parameter size'? the number of bits for all the weights in each layer?
- Eq. (4) - is w or w_k the full precision weights? why are x and y in the interval [0,1]?
- Eq. (6) - c_i represnts the i-th layer, but d_1,d_2, a_1,a_2 are fixed. Are they also different for different layers?
Also, a_1 and a_2 are the boundaries of the cursor - are they set to [0,1]? or [0,32]?
- The authors defined Conv - the convolution operation - but what are W_1 and X? vectors? how is the convolution
defined precisely? it is also not common in math papers to use '*' for  product
It is not entirely clear to me how exactly the authors define a differentiable loss and a gradient for the quantization part Loss_q.  When a cursor parameter changes (e.g. c_i from 2.5 to 2.6) then the loss is defined by quantization to the two consecutive integers (in that case 2,3 ) and there is a continuous mixture parameter between them which can be changed smoothly, which is nice. But at some point, the parameter will reach 3, and then the quantization will use the two integers 3 and 4 - the fact that 3 has weight 1 in the loss (and 2,4 have weight zero) makes the loss continuous, but is the loss differentiable at this point?  some explanation is needed here on if this is a problem and how is it avoided/solved.
Algorithm 1: The description in Figure 1 can be made more precise. The algorithm seems to alternate between
optimizing the cursor c, and optimizing the weights w for a given value of c.
There are also missing details and parameters like gradient step size.  What does Grad-C * L_C mean? is the gradient multiplied by the loss?? or only applied to it?
In the sentence "Quantize the network ... to update the loss" - which loss is updated and how? L_V, L_T? something else?
A detailed description of the algorithm and parameters or the actual code used by the authors would improve the understanding of their method.
Section 4.1: Figure 2 compares the loss of one-integer vs. two-integers quantization scheme. The authors argue that their two integers scheme is better because it is smoother. But the loss for one integer is actually lower - so why wouldn't it be better to use this one?
will the two-integer method eventually reach a lower loss?


We improve the recently-proposed "MixMatch" semi-supervised learning algorithm by introducing two new techniques: distribution alignment and augmentation anchoring. Distribution alignment encourages the marginal distribution of predictions on unlabeled data to be close to the marginal distribution of groundtruth labels. Augmentation anchoring feeds multiple strongly augmented versions of an input into the model and encourages each output to be close to the prediction for a weakly-augmented version of the same input. To produce strong augmentations, we propose a variant of AutoAugment which learns the augmentation policy while the model is being trained. Our new algorithm, dubbed ReMixMatch, is significantly more data-efficient than prior work, requiring between 5× and 16× less data to reach the same accuracy. For example, on CIFAR-10 with 250 labeled examples we reach 93.73% accuracy (compared to MixMatch's accuracy of 93.58% with 4,000 examples) and a median accuracy of 84.92% with just four labels per class. We make our code and data open-source at URL blinded for peer review.

This paper proposes two modifications for the MixMatch method [1] and achieves improved accuracy on a range of semi-supervised benchmarks. The first modification enforces the distribution of predicted labels to match the distribution of labeled data. The second modification is adding a learned data augmentation strategy, and adapting the method to work with strong data augmentation. The final method is titled ReMixMatch, and improves significantly over MixMatch, especially in low-data regime.
The main contribution of the paper is really strong empirical results. The method achieves state of the art results or close to that on multiple benchmarks, with especially large gains in settings with very scarce labeled data, like 40 labels on CIFAR-10.
Another important contribution is the learned data augmentation strategy, which as far as I understand is novel and overcomes some of the limitations of  existing learned data augmentation techniques. However, the explanation of the strategy wasn’t very clear for me, and the authors didn’t frame it as a major contribution.
The main drawback of the paper is that it seems to be more engineering-focused, and doesn’t provide much insight into semi-supervised learning. The paper can be summarized as adding two modifications to mix-match, and getting better results. The final method becomes fairly involved. Mix-Match is already an elaborate method, and ReMixMatch additionally introduces learned data augmentation, an additional loss term for matching label distributions between labeled and unlabeled data, consistency-loss, and a self-supervised loss (section 3.3).
For the reasons above, I think the paper is borderline, but I am currently voting for acceptance based on the strong empirical performance. At the same time, I think the paper can be made stronger and more interesting to read, if the authors added some experiments aimed at understanding the proposed modifications.
One set of experiments that I think would be interesting is aimed at understanding the distribution-matching part. For example, it would be great if the author could demonstrate that without this loss term the distribution of the predicted classes is wrong in the experiments from Section 4. It would also be interesting to see an experiment where the labeled data has a skewed distribution of classes, but we provide the method with information about the true class distribution, and demonstrating that this information helps predictive performance.
For the learnable data augmentation it would be great if the authors could provide more insight into the method, how it works, and why is it better than the alternatives. Just analyzing the learned data augmentation in different settings and adding more intuition for what happens would make the paper more insightful and interesting to read.
On a more minor note, the paper [1] seems to report 4.95 accuracy for MixMatch on CIFAR-10 with 4k labels, while in this paper it’s being reported as 6.24. What is the reason for the difference? Another paper, [2], reports very competitive results on CIFAR-10 for 4k labels. I would recommend discussing these results briefly in the paper. At the same time the empirical performance of ReMixMatch is really impressive, and I don’t think the results in [1] and [2] affect their significance.
[1] MixMatch: A Holistic Approach to Semi-Supervised Learning
David Berthelot, Nicholas Carlini, Ian Goodfellow, Nicolas Papernot, Avital Oliver, Colin Raffel
[2] There Are Many Consistent Explanations of Unlabeled Data: Why You Should Average
Ben Athiwaratkun, Marc Finzi, Pavel Izmailov, Andrew Gordon Wilson


Policy gradient methods have achieved remarkable successes in solving challenging reinforcement learning problems. However, it still often suffers from sparse reward tasks, which leads to poor sample efficiency during training. In this work, we propose a guided adaptive credit assignment method to do effectively credit assignment for policy gradient methods. Motivated by entropy regularized policy optimization, our method extends the previous credit assignment methods by introducing a more general credit assignment named guided adaptive credit assignment(GACA). The benefit of GACA is a principled way of utilizing off-policy samples. The effectiveness of proposed algorithm is demonstrated on the challenging WikiTableQuestions and WikiSQL benchmarks and an instruction following environment. The task is generating action sequences or program sequences from natural language questions or instructions, where only final binary success-failure execution feedback is available. Empirical studies show that our method significantly improves the sample efficiency of the state-of-the-art policy optimization approaches.

This paper proposes guided adaptive credit assignment (GACA) for policy gradient methods with sparse reward.
GACA attacks the credit assignment problem by
1) using entropy regularized RL objective (KL divergence), iteratively update prior \bar{\pi} and \pi_\theta;
2) generalizing KL to f-divergence to avoid mode seeking behaviour of KL;
3) using 2 tricks to estimate the gradient of f-divergence to update \pi_\theta, a) modified MAPO (Liang et al., 2018) estimator (using two buffers), b) replacing rho_f by the inverse of tail probability (Wang et al., 2018).
Experiments of program synthesis and instruction following are conducted, to show the proposed GACA outperform competitive baselines.
Although the experimental results look promising, I have many concerns with respect to this paper as follows.
1. The organization is bad. The main algorithm has been put into the appendix. It should appear in the paper.
2. There are too many typos and errors in the paper and derivations, which quite affected reading and understanding.
For example:
in Eq. (6), what is z \sim Z? Should be z \in Z? It also appears in many other places.
in Eq. (7), there should not be \sum_{z \in Z} here.
Proof for Prop. 1, I cannot really understand the notations here. Please rewrite and explain this proof. (I can see it follows Grau-Moya et al., 2019, but the notations here are not clear.)
in Eq. (11), \bar{\pi} / \pi_\theta is used, but in Eq. (12), \pi_\theta / \bar{\pi} appeared, which one is correct? While in the proof for Lemma 2, it is \bar{\pi} / \pi_\theta. And in Alg. 1 it is \pi_\theta / \bar{\pi}. Please make this consistent.
Typos, like "Combining Theorem 1 and Theorem 2 together, we summarize the main algorithm in Algorithm 1." in the last paragraph of p6. However, they appeared as Prop. 1 and Lemma 2. Please improve the writing.
3. The mutual information argument Eq. (9) seems irrelevant here. (It follows Grau-Moya et al., 2019, but the notations in the proof are bad and I cannot understand it). Whether the solution is mutual information or not seems not helpful for getting better credit assignment. I suggest remove/reduce related arguments around Eq. (9) and (10), and make space for the main algorithm.
4. The entropy regularized objective and the KL is kind of well known. Maybe reduce the description here. And the key point is Eq. (8), which lays the foundation of iteratively update \bar{\pi} and \pi_\theta. However, Eq. (8) is the optimal solution of KL Eq. (7). Is it also the optimal solution of f-divergence used in the algorithm? If it is, clearly show that. If not, then update \bar{\pi} in Alg. 1 is problematic. Please clarify this point.
5. The 2 tricks used here for estimating the gradient of f-divergence with respect to \pi_\theta, i.e., modified MAPO estimator in Prop. 2, and inverse tail probability in Wang et al., 2018, seems quite important for the empirical performance.
However, motivation is not clear enough. First, why using two replay buffers "leads to a better approximation"? Any theory/intuition or experiment to support this claim? Second, why using inverse tail probability "achieve a trade-off between exploration and exploitation". It seems not obvious to see that. And also, explain why using this trick makes "\pi_\theta adaptively coverage and approximate prior distribution \bar{\pi}".
6. The claim that GACA recovers all the mentioned methods as special cases are questionable. For example, as in E.1, "by simply choosing \rho_f as constant 1", comparing Eq. (12) with the gradient of REINFORCE, there is a difference that REINFORCE has a reward term, but GACA does not have. Then why GACA reduces to REINFORCE? Also in E.5, the RAML objective seems wrong. There is no reward term here. Please check them.
Overall, the proposed GACA method achieves promising results in program synthesis tasks. However, there are many concerns with respect to motivation and techniques that should be resolved.
=====Update=====
Thanks for the rebuttal. I keep my rating since some of my concerns are still not resolved. In particular, "Eq. (8) is the optimal solution of KL Eq. (7). Is it also the optimal solution of f-divergence used in the algorithm?" Eq. (8) looks not the same as the paragraph above Lemma 2 "\bar{\pi} = \pi_\theta" to me. If Eq. (8) is not the optimal solution of Eq. (11), the update in Alg. 1 is somewhat problematic and other better choices exist. Since Algorithm 1 explicitly uses f-divergence, I think at least this point should be clarified by the authors rather than my guess.


Graph convolutional networks (GCNs) and their variants have generalized deep learning methods into non-Euclidean graph data, bringing a substantial improvement on many graph mining tasks. In this paper, we revisit the mathematical foundation of GCNs and study how to extend their representation capacity. We discover that their performance can be improved with an adaptive neighborhood aggregation step. The core idea is to adaptively scale the output signal for each node and automatically train a suitable nonlinear encoder for the input signal. In this work, we present a new method named Diagonal Graph Convolutional Networks (DiagGCN) based on this idea. Importantly, one of the adaptive aggregation techniques-the permutations of diagonal matrices-used in DiagGCN offers a flexible framework to design GCNs and in fact, some of the most expressive GCNs, e.g., the graph attention network, can be reformulated as a particular instance of our model. Standard experiments on open graph benchmarks show that our proposed framework can consistently improve the graph classification accuracy when compared to state-of-the-art baselines.

This paper proposed  a so-called diagonal GCN method with adaptive neighborhood aggregation rules., namely, each node will be associated with an individual importance factor to re-scale the output signal  of the graph convolutional operator, and also an adaptive encoder function is adopted. This is achieved by adding a left diagonal matrix P and a right diagonal matrix Q to the standard message passing of GCN H’ = sigma(AHW), as H’(PAQHW); P and Q are parameterized based on the HW part (node features). Multihead attention and multi-hop versions are also derived. Connection of this formulation to the existing GAT method is also discussed. Empirical results on transductive node classification and inductive node classification are reported against state-of-the-arts.
Overall I feel that the novelty of the proposed method is quite limited. Adding a rescaling on each node seems a minor structural change based on existing GCN framework. Although the authors claim that the GAT method can be deemed as a special case of the proposed methods, the main contribution of inductive attention in GAT should not become a credit of the proposed method. Actually the empirical performance of the proposed method has only a small margin compared with the GAT, on the three node classification benchmark datasets, where the accuracies are around 80%, the improvement of the best version of the proposed methods is only around 0.3%, 0.7%, and 0.1%, respectively. This is somehow predictable since the proposed method does not exploit any extra useful information (such as graph structure) but simply base the new parameters on the existing components of the GCN message passing procedure.
The motivation of the paper is that the diffusion process on the graph should depend on the nodes’ status and influence (for example, some opinion leader nodes should have an amplified spread than other nodes), while existing GCN dies bit examine such ``difference’’ between the nodes. However, the roles or status of the nodes in a network are usually unavailable in practice; and if they can be ever be inferred, one has to use structural clues, Unfortunately, the extra parameters added here, namely the matrix P, Q, are dependent only on the features of the nodes (namely HW part), but not the graph structures,  therefore I do not think such parametrization would lead to significant benefit in improving the diffusion process. Indeed,  the original Gat method (graph attention network) has already achieved adaptive aggregation of the neighborhood information, by learning an inductive attention function  based on the node features; therefore the authors claim that existing GCN “ignores the difference between nodes” sounds very vague, since their new parameters are not capable enough to reflect “node differences” either (as a function of the node features).
Some minor comments
(1)	Figure~1 is not easy to read while it is supposed to visualize the main idea clearly in one picture
(2)	Is there proof that if multi-hop version is used, the results can be improved?
(3)	What is the extra number of parameters compared with the standard GCN message passing？


Stochastic gradient descent (SGD) has been the dominant optimization method for training deep neural networks due to its many desirable properties. One of the more remarkable and least understood quality of SGD is that it generalizes relatively well on unseen data even when the neural network has millions of parameters. We hypothesize that in certain cases it is desirable to relax its intrinsic generalization properties and introduce an extension of SGD called deep gradient boosting (DGB). The key idea of DGB is that back-propagated gradients inferred using the chain rule can be viewed as pseudo-residual targets of a gradient boosting problem. Thus at each layer of a neural network the weight update is calculated by solving the corresponding boosting problem using a linear base learner. The resulting weight update formula can also be viewed as a normalization procedure of the data that arrives at each layer during the forward pass. When implemented as a separate input normalization layer (INN) the new architecture shows improved performance on image recognition tasks when compared to the same architecture without normalization layers. As opposed to batch normalization (BN), INN has no learnable parameters however it matches its performance on CIFAR10 and ImageNet classification tasks.

The paper presents a new gradient update rule for neural net weights, which is a result of relaxing the usual SGD to a regularized linear regression between the descent direction and the original gradient. The method is interpreted as some kind of input normalization similar to Batch Norm later in the paper.
Although the method seems to work and the idea is interesting, I’m not entirely convinced by the experimental evidence. There are some technical concerns:
- In section 2.1, I’m only familiar with the MNIST dataset. Although the proposed method achieves slightly better performance than vanilla SGD, it’s pretty weird the accuracy is only 98.xx% as a very simple CNN could easily achieve >99% accuracy on MNIST. This might be due to the simple network architecture used. A more modern architecture would be preferable in those experiments. Also the improvement on CIFAR seems to close to tell.
- The writing seems to need some improvement. It’s sometimes hard to follow the text, and some result tables (Table A1,A2,A3) are in the appendix not the main text. It might be better to reorganize the text and present everything about the method (fast approximation, CNN, interpretation as normalization) together.
- The authors name the proposed method “deep gradient boosting”, but I’m not entirely sure how it is related to gradient boosting. Eq. 3 concerns sum of weak predictors, but how is this related to Eq. 4? The neural net F(x) doesn’t seem to be exactly a sum of weak learners. I don’t quite follow Eq. 5 – There seems to be missing something.
Typos:
- Eq. 5, d L_t instead of d L_T, and should there be a d \phi / d w_ij term?
- Eq. 7, v_t => v on the right-hand side.
- Eq. 12, should be \sum_i=1^B


We review the limitations of BLEU and ROUGE -the most popular metrics used to assess reference summaries against hypothesis summaries, and introduce JAUNE: a set of criteria for what a good metric should behave like and propose concrete ways to use recent Transformers-based Language Models to assess reference summaries/translations against hypothesis summaries/translations.

=== Summary ===
The authors motivate the development of new (automatic) metrics to evaluate language generation by using similarity with a given reference: standard metrics like BLEU, ROUGE or METEOR have been shown to have poor correlation with human judgment on a number of tasks and are vulnerable to changes in word re-ordering, semantics-changing word replacement, and syntactic transformations.

They then propose a multi-dimensional evaluation criteria to evaluate sentence similarity based on semantic similarity (something that correlates with human judgments of the same), logical equivalence and fluency.

The paper then goes on to describe possible directions to tackle several key problems in evaluation: evaluating semantic similarity by using models trained on the GLUE benchmark, evaluating logical equivalence using models trained on the MNLI corpus and fluency based on the CoLA corpus.
=== Decision ===
The problem this paper seeks to tackle is clearly one of great
importance in the field, but I find it hard to argue that this paper
significantly contributes to the existing body of work (more on this
below) and as a result I vote to reject this paper.
There are two possible contributions for this paper: a set of criteria for what makes a good evaluation metric and the concrete proposals to implement these criteria.

For the first, I find the proposed criteria to be overly generic and not helpful at providing additional clarity on what makes for a good evaluation: for example, how is semantic similarity different from logical consistency? Does it make sense to compare the semantic similarity of two sentences if one of them isn't even near grammatical? A lot of prior work already argue the shortcomings of the existing metrics this paper is making, e.g. Conroy and Dang (2008), Liu et al. (2016), Novikova et al. (2017). I think it would be valuable to present new axes to decompose the evaluation problem, but more work is needed to clarify and develop the axes presented in this paper.

For the second possible contribution, the idea of evaluating language generation along dimensions is not novel and in fact quite standard in the NLP community. The challenge has been showing that there are subset of tasks that can be used a reliable metrics across different domains and systems. Unfortunately, this paper does not actually evaluate its own proposals, making it hard to evaluate how effective its proposals are.


Learning modular structures which reflect the dynamics of the environment can lead to better generalization and robustness to changes which only affect a few of the underlying causes. We propose Recurrent Independent Mechanisms (RIMs), a new recurrent architecture in which multiple groups of recurrent cells operate with nearly independent transition dynamics, communicate only sparingly through the bottleneck of attention, and are only updated at time steps where they are most relevant. We show that this leads to specialization amongst the RIMs, which in turn allows for dramatically improved generalization on tasks where some factors of variation differ systematically between training and evaluation.

This paper proposes a neural network architecture consisting of multiple independent recurrent modules that interact sparingly. These independent modules are not all used simultaneously, a subset of them is active at each time step. This subset of active modules is chosen through an attention mechanism. The idea behind this architecture is that it would allow the different modules to specialize in different mechanisms and that would allow compositionality. The empirical results suggest that the proposed approach is able to generalize better than traditional architectures (which all have the implicit assumption that all processes interact).
This paper is well-written and it provides a very thorough empirical analysis of the proposed idea. Because it is not in my area of expertise I’m not confident that I can assess its novelty or its relationship to other existing approaches.
In terms of presentation, I recommend the authors to enlarge some of the figures in the paper (e.g., I can’t read the small box in Figure 1) and to not use citations as nouns (e.g., “The mechanisms of this attention mechanism follow (Vaswani et al., 2017; Santoro et al., 2018), with the …”). I would also like to point out that although fairly different in how they tackle the problem, the work of Arjovsky et al. (2019) seems to be related to this one.
Three questions I believe were not answered in the paper are:
1) How is the performance related to the total number of subsystems (and the number of *active* ones). I can only see results related to that in Table 1, but the variation in the number of modules is pretty small (4-6). The results also don’t give any indication whether we want to have more modules active at each time step, if there’s a sweet spot, etc. It is said that the method seems to be robust to this choice but this claim is made because it performs similarly for the values 5 and 6 if I recall correctly.
2) Is there any incentive in this architecture for a module to not simply “give up”? I mean, the modules are not necessarily incentivized to be used as often as possible, so could it be the case that a module learns to set its weights to zero?
3) Would it make sense to present baseline results for an architecture that uses attention? It seems to me that LSTM was often the baseline of choice but RIMs have two important components: multiple LSTMs and an attention mechanism. Could the attention mechanism be explaining some of the results we are seeing?
Finally, despite the very long appendix, I feel there are important details missing with respect to the empirical setup, at least in the Atari experiments which I’m more familiar with. Was stochasticity used, that is, sticky actions (Machado et al., 2018)? Moreover, for how long was PPO (and RIMs-PPO) trained in terms of number of frames? Finally, I’d recommend the authors to include a table with the actual average (and standard deviation) performance in each Atari games. It is really hard to know how well a method is doing by just squinting at learning curves. It is hard to know if the results are significant without a notion of variance.
References:
Martín Arjovsky, Léon Bottou, Ishaan Gulrajani, David Lopez-Paz: Invariant Risk Minimization. CoRR abs/1907.02893 (2019)
Marlos C. Machado, Marc G. Bellemare, Erik Talvitie, Joel Veness, Matthew J. Hausknecht, Michael Bowling: Revisiting the Arcade Learning Environment: Evaluation Protocols and Open Problems for General Agents. J. Artif. Intell. Res. 61: 523-562 (2018)
------
>>> Update after rebuttal: I stand by my score after the rebuttal.
Unfortunately I'm not an expert in this area and I don't feel confident in having a very strong opinion about this paper, willing to fight for its acceptance. I also agree with concerns raised by other reviewers. As I stated in the discussion with the authors, the clarifications and additional experiment does improve the paper a bit.


We study the generalization properties of deep convolutional neural networks for image denoising in the presence of varying noise levels. We provide extensive empirical evidence that current state-of-the-art architectures systematically overfit to the noise levels in the training set, performing very poorly at new noise levels. We show that strong generalization can be achieved through a simple architectural modification: removing all additive constants. The resulting "bias-free" networks attain state-of-the-art performance over a broad range of noise levels, even when trained over a narrow range. They are also locally linear, which enables direct analysis with linear-algebraic tools. We show that the denoising map can be visualized locally as a filter that adapts to both image structure and noise level. In addition, our analysis reveals that deep networks implicitly perform a projection onto an adaptively-selected low-dimensional subspace, with dimensionality inversely proportional to noise level, that captures features of natural images.

This paper looks at how deep convolutional neural networks for image denoising can generalize across various noise levels. First, they argue that state-of-the-art denoising networks perform poorly outside of the training noise range. The authors empirically show that as denoising performance degrades on unseen noise levels, the network residual for a specific input is being increasingly dominated by the network bias (as opposed to the purely linear Jacobian term). Therefore, they propose using bias-free convolutional neural networks for better generalization performance in image denoising. Their experimental results show that bias-free denoisers significantly outperform their original counter-parts on unseen noise levels across various popular architectures. Then, they perform a local analysis of the bias-free network around an input image that is now a strictly linear function of the input. They empirically demonstrate that the Jacobian is approximately low-rank and symmetric, therefore the effect of the denoiser can be interpreted as a nonlinear adaptive filter that projects the noisy image onto a low-dimensional signal subspace. The authors show that most of the energy of the clean image falls into the signal subspace and the effective dimensionality of this subspace is inversely proportional to the noise level.
Even though it is theoretically not too well-motivated in the paper why the bias term degrades generalization performance, the experimental results seem to clearly demonstrate the merit of bias-free denoisers. Moreover, the analysis of the network Jacobian and its interpretation as a nonlinear adaptive filter provides some interesting insight in the local properties of bias-free denoisers. Therefore, I would recommend accepting this paper, if the authors provide a theoretical discussion on why the bias term might degrade generalization performance.
Some smaller comments:
-It is not clear if d should be multiplied by \sigma^2 in its definition on page 7. The definition mentions dependence on noise variance, but the formula does not have it.
-In Section 3 in the expression of the mean squared error it is not defined what g(y) means.
-Axis labels are missing on Fig. 3.


Recent evidence shows that convolutional neural networks (CNNs) are biased towards textures so that CNNs are non-robust to adversarial perturbations over textures, while traditional robust visual features like SIFT (scale-invariant feature transforms) are designed to be robust across a substantial range of affine distortion, addition of noise, etc with the mimic of human perception nature. This paper aims to leverage good properties of SIFT to renovate CNN architectures towards better accuracy and robustness. We borrow the scale-space extreme value idea from SIFT, and propose EVPNet (extreme value preserving network) which contains three novel components to model the extreme values: (1) parametric differences of Gaussian (DoG) to extract extrema, (2) truncated ReLU to suppress non-stable extrema and (3) projected normalization layer (PNL) to mimic PCA-SIFT like feature normalization. Experiments demonstrate that EVPNets can achieve similar or better accuracy than conventional CNNs, while achieving much better robustness on a set of adversarial attacks (FGSM,PGD,etc) even without adversarial training.

This paper presents a SIFT-feature inspired modification to the standard convolutional neural network (CNN). Specifically the authors propose three innovations: (1) a differences of Gaussians (DoG) convolutional filter; (2) a symmetric ReLU activation function (referred to as a truncated ReLU; and (3) a projected normalization layer. The paper makes the claim that the proposed CNN variant (referred to as the EVPNet) demonstrates superior performance as well as improved robustness to adversarial attacks.
Clarity: Overall, the paper is not particularly well written. There are multiple missing articles and other grammatical errors that make it a bit arduous to read, though I do not believe they have obstructed my ability to understand the contributions. The section describing the projected normalization layer (second half of page 5) is a bit confusing. Figure 2(c) is not helpful in shedding light on the details, though I think a more detailed figure could be quite helpful. Beyond these issues, the paper is relatively clear in the presentation of the material.
Novelty: Over the last few years there have been many, many proposals for how to vary the basic CNN architecture to improve performance. Some of these have lead to genuine performance gains and have become part of the standard CNN specification. ReLUs, ResNets and Batch Normalization are particularly prominent examples of contributions that have been shown to lead to improvements in performance. Yet the vast majority of these sorts of proposals ultimately make little or no impact on the field. In light of this, I would rate the novelty of the basic goal of this paper as relatively low, though the specific proposal is novel to me and seems reasonable.
Impact: The impact potential for this paper lies with the performance offered by the proposed innovations. With respect to overall performance improvement the proposed method has not been shown to perform quite at a state-of-the-art level, as given by these resource:
SVHN: https://paperswithcode.com/sota/image-classification-on-svhn
CIFAR-10: https://paperswithcode.com/sota/image-classification-on-cifar-10
The authors compare the performance of their proposed EVPNet against a fair baseline - a squeeze-and-excite ResNet model. These sorts of controlled experiments are useful, but the actual reported performance for both models are somewhat off of the state-of-the-art and it's not clear that the relatively small benefit the authors show over their baselines are maintained for higher performing architectural configurations. Can this architecture be competitive with the state-of-the-art? The current paper in it's current
Most of the results relate to the claim that the proposed model is robust to adversarial examples. Unfortunately, this is not a particular area of expertise for me, so it's difficult for me to provide a confident assessment of the contribution here, though I will say two things: (i) the method seems to provide a significant increase in adversarial robustness across the
baseline architectures investigated. (ii) the authors demonstrate that the benefit provided by the proposed architecture seems to persist even when training for adversarial defence is introduced.
I would have liked to see more datasets explored in Experiments section. I  especially would have liked to see results on ImageNet.
My current rating is weak reject based on the weakness of the writing and the lack of strong empirical evidence in support of the effectiveness of the proposed contributions.


Answering complex logical queries on large-scale incomplete knowledge graphs (KGs) is a fundamental yet challenging task. Recently, a promising approach to this problem has been to embed KG entities as well as the query into a vector space such that entities that answer the query are embedded close to the query. However, prior work models queries as single points in the vector space, which is problematic because a complex query represents a potentially large set of its answer entities, but it is unclear how such a set can be represented as a single point. Furthermore, prior work can only handle queries that use conjunctions (∧) and existential quantifiers (∃). Handling queries with logical disjunctions (∨) remains an open problem. Here we propose QUERY2BOX, an embedding-based framework for reasoning over arbitrary queries with ∧, ∨, and ∃ operators in massive and incomplete KGs. Our main insight is that queries can be embedded as boxes (i.e., hyper-rectangles), where a set of points inside the box corresponds to a set of answer entities of the query. We show that conjunctions can be naturally represented as intersections of boxes and also prove a negative result that handling disjunctions would require embedding with dimension proportional to the number of KG entities. However, we show that by transforming queries into a Disjunctive Normal Form, QUERY2BOX is capable of handling arbitrary logical queries with ∧, ∨, ∃ in a scalable manner. We demonstrate the effectiveness of QUERY2BOX on two large KGs and show that QUERY2BOX achieves up to 25% relative improvement over the state of the art.

This paper proposes a method to answer complex logical queries in large incomplete knowledge bases (KB). Specifically it considers the class of existential first-order logical queries (EPFO) which includes the logical and, or and existential operator.  The key contribution of this paper is to represent sets of entities via regions, more specifically as boxes or hyper-rectangles. This is well motivated because such logical queries often involves working over sets of entities at once and involves applying set based operators. Previous work which represented queries as a point in vector space are not well suited for these queries.
Instead, this paper models sets of entities as boxes or axis-aligned hyper-rectangles which is parameterized by two vectors \in \mathbb{R}^{n] denoting the center and the offset respectively. Boxes can also be understood to represent all the points in it (measure by element-wise comparison with the min and max coordinate). Handling the queries require projection and intersection operation. They are defined by simple addition operation (which guarantees the boxes grow in size, due to positive offset values) and a shrinkage function to denote intersection that guarantees the output area is smaller and is inside the set of boxes.
For handling disjunctive queries, they make the clever trick of converting queries to DNF form so that the union operation is at the end of the computation graph which effectively reduces to taking the union of sets at the end.
Experiments are run on standard datasets (FB15k and FB15k-237) — however, they generate their own query patterns. Specifically, they train on 5 patterns involving projection and intersection operation and test on 4 unseen ones. For baselines, they only compare to previous work of Hamilton et al., 2018 that maps queries to vectors.
Strengths:
1. Most knowledge base comprehension benchmark tests on link prediction problems which are queries of kind (e1, r, ?). However, semantic parsers of natural language produce queries that are much richer in shape. This paper (and Hamiltion et al., 2018 before) considers answering complex logical queries (although the shape of query is pre-defined and not arbitrarily complex).
2. Modeling logical queries into regions in vector space is an interesting idea, and it would be nice to see followup work in this direction.
3. The paper is nicely written and ablation experiments were helpful.
4. Compared to the baseline they used, the paper does a better job of modeling complex logical queries.
Weaknesses / Questions:
1. I understand that the papers have considered various pre-determined shapes of queries, but the simple 1p query is similar to the usual benchmarks, I don’t understand why results for 1p were not compared with existing benchmark results. Without that comparison, I don’t have a good sense if region-based method actually are effective for “1p” kind of queries.
2. Even though the model was tested on two variants of freebase datasets, FB15k is well-known to have a lot of issues (Toutanova and Chen, 2015). Why weren't other standard datasets such as Nell-995, WN18RR and so many other biomedical KBs not considered for experiments, especially because the query generation process is very simple and it's easy to run experiments
3. It was not clear to me how the intersection operator would give zero offset for a set of non-overlapping boxes as input. Is the zero value coming from the deep-set model?, If so, how do you ensure that? Minor: Please include the deep-set network here instead of in Sec 4.3. I was confused about what the deepest model is until this point.
4. Regarding the results, is there any particular reason the MRR metric was pushed to the appendix and only results of Hits@3 was shown in the main section of the paper. I believe MRR is a better metric for your case because you are modeling sets of entities as answers and hence a ranking metric that ranks all entities is better, Hits@k is 1 if any of the answers in the set is present in top-k and hence quite a loose metric.
5. Why is the result of 3i is better than 2i. I am not sure why the model would do a better job in handling 3 intersections better than it does 2 intersections.
6. How many answers are there on an average for each question. This will better help me understand how hard the dataset actually is.
7. It is nice to see, that the model prefers boxes of different width. Do you have a sense of which type of entities (or relations) have higher offsets. This analysis would be nice to have for readers in the appendix section


Reinforcement learning agents need to explore their unknown environments to solve the tasks given to them. The Bayes optimal solution to exploration is intractable for complex environments, and while several exploration methods have been proposed as approximations, it remains unclear what underlying objective is being optimized by existing exploration methods, or how they can be altered to incorporate prior knowledge about the task. Moreover, it is unclear how to acquire a single exploration strategy that will be useful for solving multiple downstream tasks. We address these shortcomings by learning a single exploration policy that can quickly solve a suite of downstream tasks in a multi-task setting, amortizing the cost of learning to explore. We recast exploration as a problem of State Marginal Matching (SMM), where we aim to learn a policy for which the state marginal distribution matches a given target state distribution, which can incorporate prior knowledge about the task. We optimize the objective by reducing it to a two-player, zero-sum game between a state density model and a parametric policy. Our theoretical analysis of this approach suggests that prior exploration methods do not learn a policy that does distribution matching, but acquire a replay buffer that performs distribution matching, an observation that potentially explains prior methods' success in single-task settings. On both simulated and real-world tasks, we demonstrate that our algorithm explores faster and adapts more quickly than prior methods.

Summary:
The paper proposes to frame exploration in reinforcement learning as a distribution matching problem. More specifically, the proposed method (SMM) aims to minimize the reverse KL between the state distribution induced by the policy and a desired state distribution. The desired state distribution can be used to guide exploration (e.g. by penalizing bad states) or can be chosen uniformly (resulting in a policy that maximizes state entropy).
SMM iteratively optimizes the policy and a model for its induced state-distribution. The latter is used for approximating the policies state entropy. The algorithm is framed as a two-player zero-sum game between a "policy player" and a "density player". In order to justify that one of the players is assumed fixed while optimizing the other player, SMM optimizes the players against a historical average of the opponent. Such approach is known as fictitious play in game theory and ensures convergence to a Nash equilibrium.
The density player maximizes the likelihood of the states encountered during all previous roll-outs using a VAE. The policy update is framed as a standard reinforcement learning problem where the log-pdf of the target distribution serves as reward and the log-pdf of the model acts as cost. For practical reasons, the policy update does not consider an historical average but only uses the most recent density model. As the learned exploratory policy should correspond to an historical average exploration for the downstream task is achieved by sampling one of the learned policy at the beginning of each roll-out. The historical averaging also benefits prior exploration methods.
Notably, the appendix also describes a modification that learns a mixture of policies in each iteration where a discriminator approximates the responsibilities for each component which is used as additional reward to divide the search space among the components. SMM is compared to prior exploration methods, ICM and Pseudo-Counts and standard SAC, on a simulated robotic box-pushing task and against SAC on simulated point-mass problem and a real-robot valve turning task.
Significance:
Efficient exploration is arguably the main challenge of reinforcement learning as providing shaped reward functions is difficult even for experts and may lead to undesired behavior. I think that maximizing the state-entropy (or matching target state-distributions if available) is a sound objective for learning exploratory policies and the paper could, thus, be of relatively broad interest.
Novelty:
Maximizing the state entropy for exploration has already been used by (Hazan et al., 2018, reference from manuscript). However, in contrast to this prior work, SSM does not need to learn oracles that predict optimal state-distributions/policies for any given policies/reward functions. While distribution matching is a common approach to imitation learning, it has been little employed for manually specified distributions. Still, a similar objective has been used to replace reward functions by desired distributions in a RL-like setting [1] (not for exploration). However, their approach is quite restricted by assuming Gaussian target distributions.
Soundness:
If I understand correctly, fictitious play assumes optimal responses to understand that the state distribution would often be more important than provably converge to a Nash equilibrium. The paper fails to provide stopping criteria for the optimization steps of the individual players; however, I assume that only few gradient steps are used for practical reasons. Hence, I am not sure whether the actual algorithm can be justified by the theory.
The paper mentions that VAEs are used to model the state distribution. Given that VAEs are not likelihood-based, I do not understand how the reward term log(q(s)) can be computed.
Clarity:
The paper is well-written and the structure seems fine.
I think that the density estimation should be better discussed. Models of the state distribution of the policy are often highly desirable, not only for optimizing its entropy, but also, for example, for importance sampling. However, modeling the state distribution is also inherently difficult--especially for large state spaces.
Experiments:
I like that the paper uses a real robot experiment and the ablations with respect to the historical averaging are interesting. Unfortunately, the paper only compares to standard SAC on the hallway task and on the real robot task. I would not consider the entropy regularization of SAC a proper baseline for a proposed exploration technique.
Questions:
Is the same extrinsic reward, i.e. log(p*), also provided to the other exploration methods?
Some imitation learning methods such as GAIL are also able to match desired state-distributions. I think that these methods could be in principle also applied to the setting considered in the manuscript by using samples from the desired distribution as demonstrations. The paper briefly mentions that IRL methods are not applicable because they require access to trajectories, however, the discriminator of GAIL is only trained on individual state samples. I also do not see a problem of providing unachievable demonstrations to such imitation learning methods because, such like SSḾM, they would try to minimize the divergence. I think that GAIL would actually be an important baseline for the proposed method.
How does the method scale with the dimensions of the state? SSM has only be evaluated on relatively low-dimensional problems (compared to some rllab/mujoco tasks with >100 states). I would assume that obtaining meaningful density estimates in such settings might be problematic. May imitation learning methods based on discriminators actually be more promising?
SMM only considers matching state distributions. If I understand correctly, the approach could be easily generalized to state-action distributions, correct? Wouldn't it make sense for some tasks to also explore in the action space?
Decision:
I like the paper in general. The optimization problem seems well-motivated and the algorithm seems reasonable fine. I also like that the paper includes a real robot experiment. However, I am not convinced by the derivation of SMM based on fictitous play and I think that it should be better evaluated with respect to existing exploration methods and also with respect to imitation learning methods. I am therefore slightly leaning towards rejection, currently.
Typo:
"Note[sic] only does SMM explore a wider range of angles than SAC, but its ability to explore increasing[sic] throughout training, suggesting that the SMM objective is correlated with real-world metrics of exploration."
[1] Arenz, Oleg, Hany Abdulsamad, and Gerhard Neumann. "Optimal control and inverse optimal control by distribution matching." 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2016.


With the recent success and popularity of pre-trained language models (LMs) in natural language processing, there has been a rise in efforts to understand their inner workings. In line with such interest, we propose a novel method that assists us in investigating the extent to which pre-trained LMs capture the syntactic notion of constituency. Our method provides an effective way of extracting constituency trees from the pre-trained LMs without training. In addition, we report intriguing findings in the induced trees, including the fact that pre-trained LMs outperform other approaches in correctly demarcating adverb phrases in sentences.

I am satisfied with the author's response and I am increasing the score to 6 after the rebuttal.
===
This paper introduces a new and simple method of probing whether syntax information under the form of constituency trees is present in recent pre-trained language models (e.g. BERT, RoBERTa, XLNet and GPT2) without any additional task-specific training. They use the previously proposed concept of "syntactic distance" [1] as a sufficient statistics for the constituency tree of a given sentence. They compute the distance between neighboring words in sentence using a distance function f(g(w_i), g(w_i+1)) where f is a divergence between distributions and g(w_i) is the self-attention distribution of w_i at a given layer: the intuition is that words that have similar self-attention distributions belong to the same constituent and thus their syntactic distance is small. In order to conform with right-skewness of English syntax, they propose an affine transformation of the distances that encourage right-skewed trees, and tune the hyper-parameter directly on the target F1 score wrt ground-truth trees. Results suggest that large, pretrained models capture constituency trees to some extent.
This is an empirical and analysis paper which can be considered as the “twin” of Hewitt et. al (2019, analyzes whether dependency structure can be probed from large pre-trained models). The paper is well-written and easy to understand. The experiments are in general well-organized even if they lack clarity at some point. The related work is comprehensive and covers most of the work in the domain of grammar induction and unsupervised parsing. In its current form however, I feel like the paper misses a "second half" therefore it isn’t quite above the acceptance bar. I'd be happy if the authors could kindly elaborate on the following major weaknesses: (i) unclear scientific motivation of using the right-skewness bias; (ii) lack of clarity in the experimental results; (iii) lack of depth and perspective.

1) About right-skewness bias:
I appreciate that the authors clearly guard against using explicit right-skewness biases. However, I am not sure why the authors use of Eq. (2) apart from boosting performance. The authors claim that "the main purpose of explicitly injecting such a bias is examining what changes are made to the resulting tree structures" but the purposed changes are not discussed at length in the text. Therefore:
1.1) What's the scientific motivation (apart from boosting performance) of skewing the trees using Eq. (2) ?
1.2) What can you infer from trees without bias and tree with bias ?
2) About clarity in the experimental results:
2.1) (Shen, 2018) use an "implicit" right-skewness bias. From the text, it seems that your results w/o bias use the same bias as (Shen, 2018) ? Therefore they actually have a bias ? This is a bit confusing. What about using the unbiased parsing algorithm as presented in "Straight to the tree..." ?
2.2) This sentence is unclear: "we select one derived from the best choice of f and g in terms of sentence-level F1 (S-F1)
w.r.t. gold-standard trees as a representative for the LM". Are you using S-F1 on *training* for tuning f and g ?
2.3) This sentence is unclear: "As LMs are not fine-tuned with training sets in our framework, we only use the test
set of the PTB", what do you mean by "using the test set" ? Do you mean evaluate or use it for tuning (cf 2.2) ?
3) About perspective and depth:
3.1) The fact that ADVP , ADJP performance is higher for the tested models is interesting. Do you have any hypothesis for why this is happening?
3.2) Right-skewness seems to mainly help with VP. Can you formulate an hypothesis for why current models underperform on this label or do not show right bias?
3.3) The results on Table 2 are interesting, but imho, they are basically showing that these models cannot do constituency parsing with just a linear probe on top of the representations and using the syntactic distance algorithm?
3.4) Did you consider using other parsing algorithms e.g. chart parser, instead of just the syntactic distance ? That would make a stronger paper.


The field of deep learning is commonly concerned with optimizing predictive models using large pre-acquired datasets of densely sampled datapoints or signals. In this work, we demonstrate that the deep learning paradigm can be extended to incorporate a subsampling scheme that is jointly optimized under a desired minimum sample rate. We present Deep Probabilistic Subsampling (DPS), a widely applicable framework for task-adaptive compressed sensing that enables end-toend optimization of an optimal subset of signal samples with a subsequent model that performs a required task. We demonstrate strong performance on reconstruction and classification tasks of a toy dataset, MNIST, and CIFAR10 under stringent subsampling rates in both the pixel and the spatial frequency domain. Due to the task-agnostic nature of the framework, DPS is directly applicable to all real-world domains that benefit from sample rate reduction.

This paper introduces  a novel DPS(Deep Probabilistic Subsampling) framework for the task-adaptive  subsampling case, which attempts to resolve the issue of end-to-end optimization of an optimal subset of signal with jointly learning a sub-Nyquist sampling scheme and a predictive model for downstream tasks. The parameterization is used to simplify the subsampling distribution and ensure an expressive yet tractable distribution.  The new approach contribution is applied to  both reconstruction and classification tasks and demonstrated with a suite of experiments in a toy dataset, MINIST, and COFAR10.
Overall, the paper requires significant improvement.
1. The approach is not well justified either by theory or practice. There is no experiment clearly shows convincing evidence of the correctness of the proposed approach or its utility compared to existing approaches (Xie & Ermon (2019); Kool et al. (2019); PlÂšotz & Roth (2018) ).
2. The paper never clearly demonstrates the problem they are trying to solve (nor well differentiates it from the compressed sensing problem  or sample selection problem)
   The method is difficult to understand, missing many details and essential explanation, and generally does not support a significant contribution.
3. The paper is not nicely written or rather easy to follow. The model is not well motivated and the optimization algorithm is also not well described.
4. A theoretical analysis of the convergence of the optimization algorithm could be needed.
5. The paper is imprecise and unpolished and the presentation needs improvement.
**There are so many missing details or questions to answer**
1. What is the Gumbel-max trick?
2. How to tune the parameters discussed in training details in the experiments?
3. Why to use experience replay for the linear experiments?
4. Are there evaluations on the utility of proposed compared to existing approaches?
5. Does the proposed approach work in real-world problems?
6. Was there any concrete theoretical guarantee to ensure the convergence of the algorithm.
[Post Review after discussion]: The uploaded version has significantly improved over the first submission. It is now acceptable.


Learning Invariant Representations to adapt deep classifiers of a source domain to a new target domain has recently attracted much attention. In this paper, we show that the search for invariance favors the compression of representations. We point out this may have a bad impact on adaptability of representations expressed as a minimal combined domain error. By considering the risk of compression, we show that weighting representations can align representation distributions without impacting their adaptability. This supports the claim that representation invariance is too strict a constraint. First, we introduce a new bound on the target risk that reveals a trade-off between compression and invariance of learned representations. More precisely, our results show that the adaptability of a representation can be better controlled when the compression risk is taken into account. In contrast, preserving adaptability may overestimate the risk of compression that makes the bound impracticable. We support these statements with a theoretical analysis illustrated on a standard domain adaptation benchmark. Second, we show that learning weighted representations plays a key role in relaxing the constraint of invariance and then preserving the risk of compression. Taking advantage of this trade-off may open up promising directions for the design of new adaptation methods.

Summary
-------
This paper presents a revisit of existing theoretical frameworks in unsupervised domain adaptation in the context of learning invariant representation. They propose a novel bound that involves trainable terms taking into account some compression information and a novel interpretation of adaptability. The authors mention also contribution showing that weighting representations can be a way to improve the analysis.
Evaluation
-----
The ideas are novel and the result brings novel and interesting light on the difficult problem of unsupervised domain adaptation.
However, the practical interest in terms of applicability of the proposed framework is not fully demonstrated, the properties of the proposed analysis have to be studied more in details and some parts better justified. The experimental evaluation brings some interesting behavior but is somewhat limited. The weighting aspect of the contribution is not supported by any experiment.
Other comments
------------
-I am a but puzzled by the use of the term "compression". This is maybe subjective, but in the context of learning representation, I would have interpreted it as a way to sparsify the representation, and thus compression could then be measured with respect to a given norm (L2?) or another criterion (Kolmogoroff, ...).
In the paper, the notion of compression is related to a reduction of the hypothesis space after application of a transformation \phi, so I am wondering if using "hypothesis space reduction" would not be more appropriate.
In this case, however, there are maybe links with structural risk minimization that could be investigated here.
A side remark: there is no particular restriction on the space of transformations, we wonder if it would be useful to indicate if all the possible transformations are included as subspaces of a given latent space. Since, to be very general, one can imagine the existence of an unbounded number of transformations that correspond to an increase of the input dimension. For transformations leading to different representations of different dimensions, the way the deduced hypothesis can be compared should also be indicated (for defining properly the inclusion H(\phi_1)\subset H(\phi_2).
On the other hand, the authors seem to need the use of norms over transformations as illustrated in the definition of H_0^\eta in the experimental section. So I suggest that the analysis could be revisited by directly incorporating (representation) norms in the theoretical framework and in particular for defining more properly H_0.
-One weakness of the theoretical framework is for me the lack of definition of H_0 in Section 3. We just know that it is included between two classes of hypothesis of interest, but there is no clear characterisation of H_0 which makes the analysis fuzzy: we have a bound that involves an object without any clear definition and it is for me difficult to really interpret the bound. Trying to define H_0 with some restrictions related to the norm of the transformations, as evoked before, could be a way to address this point (and actually the way the experiments are done tend to confirm this point).
-Another weak point is the lack of qualitative analyse of the bound in Inequality 3 (the same applies for Inequality 5). I would have appreciated if the authors could provide an analysis similar to the one of (Mansour et al., COLT 2009) - it is cited in the paper - when they compared their result to the one of (Ben-David et al., 2007). For example, what happens when source is equal to the target, when is the bound significantly loose, significantly tight, different from other existing results, ...
In particular, if we compare the bound with the one of Ben-David et al. (we can also consider the one of Mansour et al.), there is two additional term, one is weighted by a factor 2, another one involved a supremum and one can think that this bound is rather loose and does not provide any insightful information and said differently it could not give a strong framework for practical considerations.
I may understand that when the bound is tight we could deduce that the compression term is low, but finding cases leading to a tight interesting bound does not seem obvious.
-The experimental evaluation presents some expected behavior in the context of the bound, but I miss a real study trying to make use of the proposed framework to do adaptation in practice with comparisons to other strategies.
Additionally, having additional studies with other models and tasks will probably reinforce the analysis.
-At the beginning of Section 3.2, the authors mention that they restrict their analysis to the square loss, however I think the analysis is true for larger class of losses with more general properties. In the experimental evaluation, the cross entropy is used, so I think that the experimental evaluation should also be consistent with the theoretical analysis by considering the square loss.
-Paragraph below Definition 5 is unclear: the notion of L2 norm has not been introduced in this context, so the message of the authors is a bit unclear.
-I do not find the notation \gamma(\phi,H) appropriate, I woud rather suggest to use \gamma(H\cdot \phi)
-The biblioggrgaphy can be improved by adding the right conferences/journals where the papers have been published in addition to the ArXiv reference.


Stochastic methods with coordinate-wise adaptive stepsize (such as RMSprop and Adam) have been widely used in training deep neural networks. Despite their fast convergence, they can generalize worse than stochastic gradient descent. In this paper, by revisiting the design of Adagrad, we propose to split the network parameters into blocks, and use a blockwise adaptive stepsize. Intuitively, blockwise adaptivity is less aggressive than adaptivity to individual coordinates, and can have a better balance between adaptivity and generalization. We show theoretically that the proposed blockwise adaptive gradient descent has comparable regret in online convex learning and convergence rate for optimizing nonconvex objective as its counterpart with coordinate-wise adaptive stepsize, but is better up to some constant. We also study its uniform stability and show that blockwise adaptivity can lead to lower generalization error than coordinate-wise adaptivity. Experimental results show that blockwise adaptive gradient descent converges faster and improves generalization performance over Nesterov's accelerated gradient and Adam.

The paper proposes adaptive gradient approaches where the step-size is not determined on the per-coordinate basis but rather for blocks of coordinates. Theoretical results are presented in terms of regret in online convex optimization, regarding convergence in non-convex optimization,  and with respect to uniform stability and generalization. These indicate that under certain conditions adaptivity at the block level could outperform coordinate-wise adaptivity. The approach is evaluated against alternatives on simulated and real-world problems.
The paper considers an important topic, which has been the object of many related studies. Though the proposed approach is interesting, there are several issues with the present manuscript that would warrant significant revision.
- Though the discussion in section 3 aims at motivating the use of block-wise adaptivity, it is quite confusing. Indeed the problem considered in that section is different from the setup eventually considered.  Moreover the paper claims that it is more general than the many previous work on layer-wise adaptation. However, the example considered here does consider layers.
- The paper organization could be improved significantly. BAG is presented followed by a regret analysis in convex optimization. Then BAGM is presented followed by the theory on non-convex optimization. It might be more effective to first present the algorithms BAG and BAGM and then have a theory section.  It would also be good to emphasize that the regret analysis is solely for convex optimization, which makes it much less relevant in the context of deep learning.
- The claimed superiority of block-wise adaptivity in the theory relies heavily on an assumption relying on tightness and closeness of  upper bounds on gradient magnitude /  gradient second moment in each block.  As this assumption is crucial to the results, its validity in practice should be more thoroughly investigated, beyond the small study relegated in the appendix  B. It would also be important to assess what happens when the assumption breaks down.
- The aforementioned issue is somewhat also related to the choice of design for the blocks. Some choices might help, some might hurt based on the unknown structure of the data. This is barely touched upon in section 5.1. as it is noted that the more sever the mismatch the worse the results.
- The empirical evaluation needs to be more comprehensive in terms of comparison methods. Among others it would be important to assess performance against previously proposed layer wise stepwise approaches mentioned in the introduction.
- These related approaches should also be discussed more deeply and contrasted against qualitatively.
- The empirical results are not substantially superior. It is also quite disappointing to see that the training, testing and generalization curves on Figure 2 are quite similar with NAG and Adam and do not exhibit less instability etc.


We introduce dynamic instance hardness (DIH) to facilitate the training of machine learning models. DIH is a property of each training sample and is computed as the running mean of the sample's instantaneous hardness as measured over the training history. We use DIH to evaluate how well a model retains knowledge about each training sample over time. We find that for deep neural nets (DNNs), the DIH of a sample in relatively early training stages reflects its DIH in later stages and as a result, DIH can be effectively used to reduce the set of training samples in future epochs. Specifically, during each epoch, only samples with high DIH are trained (since they are historically hard) while samples with low DIH can be safely ignored. DIH is updated each epoch only for the selected samples, so it does not require additional computation. Hence, using DIH during training leads to an appreciable speedup. Also, since the model is focused on the historically more challenging samples, resultant models are more accurate. The above, when formulated as an algorithm, can be seen as a form of curriculum learning, so we call our framework DIH curriculum learning (or DIHCL). The advantages of DIHCL, compared to other curriculum learning approaches, are: (1) DIHCL does not require additional inference steps over the data not selected by DIHCL in each epoch, (2) the dynamic instance hardness, compared to static instance hardness (e.g., instantaneous loss), is more stable as it integrates information over the entire training history up to the present time. Making certain mathematical assumptions, we formulate the problem of DIHCL as finding a curriculum that maximizes an unknown multi-set function f (·) from its partial observations, and derive an approximation bound for a DIH-produced curriculum relative to the optimal curriculum. Empirically, DIHCL-trained DNNs significantly outperform random mini-batch SGD and other recently developed curriculum learning methods in terms of efficiency, early-stage convergence, and final performance, and this is shown in training several state-of-the-art DNNs on 11 modern datasets.

*Revision after author response*
I thank the authors for the comments on my questions.
Unfortunately, I do not feel that these comments addressed my main concerns. For all my experimental analysis questions, the authors promised some analyses for future versions, but I was hoping to see at least a minor preliminary analysis at this point, to see if indeed my concerns are valid or not.
Moreover, for my question number 1 about the optimization problem, the authors referred me to Corollary 1 from the paper, but that didn't really help me because, as the other reviewers also point out, the writing is quite hard to follow.
Because of all these, I have decided to revise my score to a weak reject. While I believe the paper has merit, it requires revisions at many points in order for a reader to truly understand the method and trust the experimental results.
--------------------------------------------------------------------------------------------------------------
The paper proposes a curriculum learning approach that relies on a new metric, the dynamic instance hardness (DIH). DIH is used to measure the difficulty of each sample while training (in an online fashion), and to decide which samples to train on next. The authors provide extensive experiments on 11 datasets as well as some theoretical motivation for the use of this approach.
---- Overall opinion ----
Overall I believe this paper is an interesting take on curriculum learning that is able to achieve good results. I believe this approach is a combination of core ideas from multiple sources, such as boosting, self-paced learning, continual learning and other curriculum learning approaches, but overall it seems different enough from each one of them individually. Because of the resemblance with these many different methods, the method itself does not surprise through the novelty of a new idea, but the authors seemed to have found something that was missing from these methods and that leads to very good results. The experimental results look great, but I believe the paper is missing some ablation studies to assess the importance of certain components (see details below). I also had some trouble understanding certain arguments, which I hope the authors can clarify.
---- Major issues ----
1. I find the arguments section 2.1 quite difficult to follow. In particular, under the assumption stated in the paper that r_t(i) = f(i|S_{1:t−1}) =  f(e_i + S_{1:t−1}) − f(S_{1:t−1}) , why does it follow that r_t(i) can be used instead of f in the minimization problem (2).
2. Based on the method itself, it seems to me that the parameter k_t could would have a lot of influence on how well the method doing.  The authors mention in the experimental section what values they use, but there is no indication on how one would choose this value. Moreover, it would be good to see an analysis of how sensitive the results are to this choice.
3. In Figure 1, it is not clear whether the figure on the right shows the actual loss, or the smooth loss using Equation (1) with instantaneous instance (A). If it is the former, then if the loss is so smooth, why do we need DIH? If it is the latter, then what does the instantaneous loss look like? This actually raises the question of how important the smoothing component is -- could we achieve the same results with an instantaneous loss (i.e. set gamma to 1 in Eq. 1)?
---- Minor issues ----
1. How do you choose T0, gamma and gamma_k?
2. In the conclusions, the authors state that “ The reason [why  MCL and SPL are less stable] is that, compared to the methods that use DIH, both MCL and SPL deploy instantaneous instance hardness (i.e., current loss) as the score to select sample”. Since there are so many other differences in the way training progresses, I think we don’t have enough evidence to attribute this to merely the “instantaneousness” of the loss. In fact, it would be interesting to see how SPL does if you use DIH as a metric (just smoothing the loss over time), but their approach of scheduling samples (easy to hard, and not the opposite and in DIHCL).
3. Appendix C shows some interesting results regarding wall time comparison. I was surprised to see that, despite the extra computations, DHCL is comparable to random mini-batches. This makes me wonder what the stop criteria was, because when you stop matters a lot for run time comparisons. It would also be interesting to see a more ample discussion on this in the main text.
4. In Figure 1, the axes are barely readable.
5. The authors oftentimes reverse the use of \citet and \citep, for example “has been called the “instance hardness” Smith et al. (2014) corresponding to” should have a bracket, whereas “Our paper is also related to (Zhang et al., 2017)” should not have brackets.
6. This is not an issue, but I just wanted to say I appreciated Appendix B.
---- Suggestions ----
1. It would be interesting to make a connection between the DIH and what other papers have discovered about example forgetting (e.g. Toneva et. al, that was mentioned in the paper).
2. Major issues 3 -> a study on the effect of k and how to choose it.
3. While I understand that the models chosen in the experiments are expensive to train, it would be good to report standard deviations in Table 1.
4. Based on Table 1 and Figure 3, there is no concrete winner among the DIHCL methods. It would be good to include some recommendations in your conclusion on which one to choose and when.
---- Questions ----
1. “On average, the dynamics on the hard samples is more consistent with the learning rate schedule, which implies that doing well on these samples can only be achieved at a shared sharp local minima.” -> can you please explain why this is so?
2. See Major issues 3.
3. In Table 1, on some datasets, the authors apply lazier-than-lazy-greedy, and on some not.Why, and how does one decide this for a new dataset?
4. How did you choose T0, gamma and gamma_k, as well as the schedules in Appendix C (page 17)?


We propose a new type of adversarial attack to Deep Neural Networks (DNNs) for image classification. Different from most existing attacks that directly perturb input pixels. Our attack focuses on perturbing abstract features, more specifically, features that denote styles, including interpretable styles such as vivid colors and sharp outlines, and uninterpretable ones. It induces model misclassfication by injecting style changes insensitive for humans, through an optimization procedure. We show that state-of-the-art pixel space adversarial attack detection and defense techniques are ineffective in guarding against feature space attacks.

This paper presents an adversarial attack method, which conducts perturbations in the feature spaces, instead of the raw image space. Specifically, the proposed method firstly learns an encoder that encodes features into the latent space, where style features are learned. At the same time, a decoder is learned to reconstruct the images with the encoded features. To conduct attacks, perturbations are added into the encoded features and attack images are generated with the decoder given the perturbated features. The experiment results look promising, showing that the proposed method achieves better attack performance with realistic adversarial images.
The general idea of perturbating the feature (latent) space is not a novel one, which has been studied in [1]. However, the proposed one is with an autoencoder framework instead of GAN used in [1]. Therefore, the proposed approach is able to construct adversarial examples for specific images. In addition, the training of the encoder is adapted from a style transfer method, which seems to learn good features that capture style features.
It is a bit unclear on the intuition of the constructions of Eq. (5) and (6). The details may be in Huang & Belongie, 2017. But it is better to provide more intuitive explanation and discussion on why these constructions capture style variation.
The results shown in the paper look promising. But it would be more comprehensive to compare with other pixel attacks in addition to PGD. Moreover, it is unclear whether it is a fair comparison between the proposed approach and pixel attacks, even under the same amount of perturbations. It would be good if the code will be released.
Minor:
Last sentence in the first paragraph of page 3: a missing reference.
[1] Song, Yang, Rui Shu, Nate Kushman, and Stefano Ermon. "Constructing unrestricted adversarial examples with generative models." In Advances in Neural Information Processing Systems, pp. 8312-8323. 2018.