Humans excel in continuously learning with small data without forgetting how to solve old problems. However, neural networks require large datasets to compute latent representations across different tasks while minimizing a loss function. For example, a natural language understanding (NLU) system will often deal with emerging entities during its deployment as interactions with users in realistic scenarios will generate new and infrequent names, events, and locations. Here, we address this scenario by introducing a RL trainable controller that disentangles the representation learning of a neural encoder from its memory management role.
Our proposed solution is straightforward and simple: we train a controller to

The authors build on work regarding few-shot learning with memory augmented networks, specifically [Kaiser, et al., ICLR17] where the goal is to learn a memory address mapping such that generalization is achieved by finding the nearest neighbor memory address when predicting the label. For correct predictions, the memory key is updated to include the associated (predicted) address while new memory locations are written for mistakes. Whereas [Kaiser, et al., ICLR17] follows a LRU-like procedure for replacing memory, the current work proposes performing policy-gradient RL where the action space is the memory locations and the reward is reduction of entropy over the memory address assignment distribution over the memory locations. This approach is empirically studied for an RNN approach to NER, specifically considering few-shot learning for NER in the Stanford Task-Oriented Dialogue (STOD) dataset — showing non-negligible improvements over Memory Augmented Networks [Santoro, et al., ICML16] and Matching Networks [Vinyals, et al., NeurIPS16].
From a high-level perspective, memory networks have played an important role in building dialogue managers (and in several applications) and their proposed expansion over [Kaiser, et al., ICLR17] in the context of few-shot, sparse memory architectures seems sensible. Additionally, the empirical results o NER seem promising. However, in my opinion, this paper is making contributions along two dimensions, but neither of them convincingly as detailed below.
First, one potential point of emphasis is the ‘learning to control’ mechanism through PG RL being better than [Kaiser, et al., ICLR17] using with a LRU-like policy. To do this, it would make sense to both compare to the datasets in the previous work (Omniglot, WMT — even if it does require additional architectures that are pretty straightforward expansions) and to include this architecture in the submitted paper as there is a publicly available dataset (https://github.com/himani-arora/learning_to_remember_rare_events). While the synthetic dataset probably isn’t necessary, one could easily think of synthetic datasets that would clearly contrast with the existing method and conduct ablation/configuration studies regarding different number of neighbors, size of memory, exact vs. approximate nearest neighbor for lookup, etc.
A second point of interest is the NER task. From this perspective, there is some existing work (e.g., [Fritzler, Logacheva & Kretov, SAC19; Hou, et al., https://arxiv.org/pdf/1906.08711.pdf (Not required as not accepted yet); Hofer, et al., https://arxiv.org/pdf/1811.05468.pdf (Not clear if accepted)]. While these are largely preliminary works, they do provide datasets (and there are many others) for NER that would potentially make a convincing case for this being a state-of-the-art approach for few-shot learning in NER. However, I am not sure that being the state-of-the-art for few-shot NER would be a sufficient contribution for ICLR, even if done convincingly. If there was something in the algorithm specific to sequence prediction, then I think few-shot sequence prediction would be a sufficient contribution, but given the current architecture, I would think that the bar is a methodological contribution to adding sparse memory units (via RL in this case) is the acceptable level of contribution.
Finally, my belief that the contribution is too narrow and not sufficiently developed is supported by the writing seeming rushed. The paper makes much more sense after reading [Kaiser, et al., ICLR17] as Sections 2, 3 of the submission are missing the aspects of a clear constrast to [Kaiser, et al., ICLR17], a working example to provide readers a more intuitive understanding, a clear example showing dimensionality and notation (beyond what is in Figure 1). There are also many typos (e.g., “mammals exel”, “trainale controller”, “Rathern then”, amongst others) and general need for proofreading. However, this is not the determining factor in the paper. Basically, I think the contribution is insufficient in terms of scope and convincingness of the contribution. Thus, I recommend rejecting in its current form — even if the underlying idea is promising and worth developing further.


Self-attention has recently been adopted for a wide range of sequence modeling problems. Despite its effectiveness, self-attention suffers quadratic compute and memory requirements with respect to sequence length. Successful approaches to reduce this complexity focused on attention to local sliding windows or a small set of locations independent of content. Our work proposes to learn dynamic sparse attention patterns that avoid allocating computation and memory to attend to content unrelated to the query of interest. This work builds upon two lines of research: it combines the modeling flexibility of prior work on content-based sparse attention with the efficiency gains from approaches based on local, temporal sparse attention. Our model, the Routing Transformer, endows self-attention with a sparse routing module based on online k-means while reducing the overall complexity of attention to O(n 1.5 d) from O(n 2 d) for sequence length n and hidden dimension d. We show that our model outperforms comparable sparse attention models on language modeling on Wikitext-103 (15.8 vs 18.3 perplexity) as well as on image generation on ImageNet-64 (3.43 vs 3.44 bits/dim) while using fewer self-attention layers.

Summary: This paper proposes a new mechanism for computing the attention scores efficiently in Transformers to avoid the quadratic computational cost in conventional Transformers (i.e., when computing the QK^T for self-attention). With a content-based "routing" technique ("group/cluster attention" may be a more accurate term to differentiate with the dynamics of Capsule Networks), the time steps of each layer form different clusters, and the self-attention mechanism is only performed within each such cluster. The authors verify their approach on very large-scale and challenging sequence benchmarks such as WikiText-103 (WT103) and enwik-8.
Pros:
+ The math notations are generally clear (e.g., dimensionality) and the paper is well-organized.
+ The paper addresses a very important (efficiency) question in Transformers, and gave a proper review to prior related works (such as Child et al.).
+ The experiments used to evaluate the model are all large-scale, high-dimensional and challenging tasks. The routing Transformer seems to be able to do very well on the WikiText-103 word-level language modeling task.
+ I especially like the analysis of the local vs. routing attention via the JSD score. It's a very interesting and useful way to compare the attention maps without visualizations (which many other works use).
--------------------------------
Cons/Questions:
1. Please number your formulas and equations.
2. Based on your definition of X' (at the end of page 3, $X_i' = \sum_{j < i}^n A_{ij} V_j$) and the subsequent Eq. (2), it seems that you did not add layer normalization and residual connection to the self-attention block. However, almost all prior works that use the Transformer architecture, such as Vaswani et al. [1], Child et al. [2] and Dai et al. [3] has two residual blocks: 1) X' = layernorm(SelfAttention(X) + X); and 2) X'' = layernorm(mlp(X') + X'). Is there any reason that you abandoned this design and used Eq. (2)? Could that have impacted the model performance? In addition, it seems you didn't scale QK^T by 1/\sqrt{d}, as did in prior works?
3. How are the k-means centroid vectors (\mu_1, ..., \mu_k) initialized? Picking the initial centers for k-means clustering has been long known as an interesting/challenging problem, especially in a high-dimensional space (which tasks like language modeling deal with). This challenge spawned works such as K-means++ [4]. I suggest the authors discussing their initialization scheme as well as the initialization's impact on the effectiveness of their approach.
4. In Section 4.2, you said "$FG^\top$ where $F$ and $G$ are binary matrices denoting cluster memberships of queries and keys respectively". However, your previous description seems to suggest that the clusters are formed based on the R matrix (with shape n x d), which contains both Q and K. Does this mean F = G^T? Or could position i in query and position i in key be in different clusters? (I think $ltr(FG^\top * A)$ would only make sense if F=G^T.) This part is confusing in the paper; please clarify.
5. The inequality at the bottom of page 5 (**feel free to clarify if I missed some assumptions from the paper**): given that $\|a-b\|^2 = (a-b)^\top(a-b) = a^\top a + b^\top b - 2a^\top b$, it's unclear in this case ($a = Q_i-K_j$ and $b = Q_j-K_i$) whether $2a^\top b$ is positive or negative (even if $Q_i, K_i$ are unit vectors). Therefore, I'm not sure why it is a >= relation here.
If this is incorrect indeed, then the subsequent analysis the authors made on \epsilon would be a bit problematic, too. (Additionally, the authors seem to suggest that $Q_i^\top K_j = Q_j^\top K_i$ in the subsequent analysis, which should not be the case, unless W_Q = W_K.) Also, just curious, how did you regularize $W_R$ to be orthonormal while training?
6. The authors acknowledge at the end of Section 4 that their method of assigning clusters "does not guarantee that each point belongs to a single cluster". This means that even with balanced cluster sizes, each cluster will have more than n/k routing vectors (i.e., time steps). This makes the $O(nkd + n^2d/k)$ bound the authors mentioned potentially inaccurate. On the other hand, without forcing such cluster balance, in the worst case, the complexity is still O(n^2d), correct?
7. The paper proposes to use $\sqrt{n}$ clustering centers, which makes sense from the perspective of minimizing total computations. However, since the cluster centroids are learned parameters, does this mean the trained routing Transformer cannot easily generalize to different sequence lengths easily? (e.g., train on sequence length 256 but test on sequence length 2000, or simply 15?) Moreover, on longer sequences, won't the sorting/clustering be increasingly inefficient?
8. What is the motivation of using half of the heads for local attention and the other half for routing attention (cf. the intro paragraph of Section 5)? Why not just use all of the heads with routing attention? Without ablations on this, it's hard to tell which part is contributing how much exactly.
9. Could you also show the # of parameters used by your model in Table 1-3 (as did in prior works)?
10. Do you observe better performance with deeper layers (which seems to be the case for the sparse Transformer [2])?
11. Except the computational concern, I still don't quite exactly see the motivation behind clustering attention. For language modeling, for instance, the time steps should be mutually dependent, and it's more like a connected, chain-like structure (where each word depends strongly on its neighbors). However, the cluster attention seems to group the words together, and there is no cross-cluster attention (i.e., the graph is broken into smaller components). In addition, the authors said the centroid parameters (\mu_1, ..., \mu_k) are shared across sequences. So in a certain sense, are they like the "anchors" in the (transformed) word embedding space? Are we then, in language modeling tasks, performing attentions among words only around these "anchors"?
12. While the theoretical complexity may be lower, how does the wall-clock time of an L-layer routing Transformer compare to an L-layer conventional Transformer (on the same sequence length, batch size)? In particular, although the routing mechanism avoids computing A, having to deal with each of the $\sqrt{n}$ clusters means that one will need to process the cluster-based attentions sequentially (and, as mentioned, you have 8 heads for local attention and 8 for routing attention, which I think are also processed in two steps?). Would this make the model actually slower on GPUs?
=====================
Minor issues that didn't impact the score:
13. In the equation at the bottom of page 3 (the definition of X_i'), shouldn't it be (j <= i) instead of (j < i), in the case of autoregressive sequence modeling? E.g., in language modeling, you do use the current word when predicting the next word.
14. At the end of Section 5.1, you said: "while having fewer [...] attention heads". I believe the released Transformer-XL model also used 16 heads? (https://github.com/kimiyoung/transformer-xl/blob/master/pytorch/run_wt103_large.sh)
15. How does the number of cluster centers influence the performance of the model? I think this would be an interesting ablation study to make.
=====================
While I'm impressed with the result the routing Transformer achieved on WikiText-103, I am not fully convinced the effectiveness of the approach (e.g., on the other tasks, the routing Transformer seems to be slightly worse than the SOTA transformer; this is less surprising, as the authors didn't change the underlying design of the Transformers). I think the paper can be improved by including more elements, such as ablative experiments and runtime benchmarking. Also, there seem to be some problems with the derivations that the authors made in the current version of the paper.
[1] https://arxiv.org/abs/1706.03762
[2] https://arxiv.org/abs/1904.10509
[3] https://arxiv.org/abs/1901.02860
[4] http://ilpubs.stanford.edu:8090/778/1/2006-13.pdf


Cold-start and efficiency issues of the Top-k recommendation are critical to largescale recommender systems. Previous hybrid recommendation methods are effective to deal with the cold-start issues by extracting real latent factors of cold-start items(users) from side information, but they still suffer low efficiency in online recommendation caused by the expensive similarity search in real latent space. This paper presents a collaborative generated hashing (CGH) to improve the efficiency by denoting users and items as binary codes, which applies to various settings: cold-start users, cold-start items and warm-start ones. Specifically, CGH is designed to learn hash functions of users and items through the Minimum Description Length (MDL) principle; thus, it can deal with various recommendation settings. In addition, CGH initiates a new marketing strategy through mining potential users by a generative step. To reconstruct effective users, the MDL principle is used to learn compact and informative binary codes from the content data. Extensive experiments on two public datasets show the advantages for recommendations in various settings over competing baselines and analyze the feasibility of the application in marketing.

The work considers the problem of efficient user and item recommendations in the warm- and cold-start settings. It aims at improving computational efficiency of the best candidate selection in these settings by utilizing binary codes representation. The transformation from an actual to a binary code representation is learned in a hybrid manner using both collaborative and content information. In order to keep such representations compact yet expressive enough, the authors impose a set of constraints that ensure balanced and uncorrelated transformations. Once binary codes are learned, the inference can be made by virtue of efficient Hamming distance computations. Moreover, the search for candidate entities can be performed via the generative step that projects binary codes onto actual feature space, where kNN-based techniques can be further utilized.
Major drawback of the work is that it does not provide any quantitative evidence to support the main claim – that the proposed approach is at least more computationally efficient, since it underperforms competing methods in terms of accuracy. Essentially, the work answers the question whether it is possible to utilize hashing techniques based on binary codes; however, the question on the practicality and efficiency of this approach remains open. I would therefore suggest rejecting the work.
One of the weak points of other methods noted by the authors is “the expensive similarity search in real latent space”. The authors aim to resolve that problem by learning hashing functions based on “compact and informative” binary codes representation. However, while an overall problem formulation is clearly described and the learning objective is well explained, no further evidence supporting initial claims is provided. Moreover, an overall logic seems contradictory:
1.	Binary codes allow efficient preference estimation via XOR operation.
2.	Learning binary codes is a difficult discrete optimization task.
3.	Hence, we employ special MDL principle for solving a constraint optimization problem and employ relaxation of hash codes to move away from discrete optimization.
After relaxation, the hash codes are no longer binary. Do you still enforce binary representation by some thresholding or other method? If yes, more words explaining this should be added to the text. If no, then how is it different from classical learning of latent variables? Essentially, relaxed non-binary hash codes are similar to latent vectors.
The lack of description raises concerns in an overall efficiency and the authors, unfortunately, provide no evidence of improved computational performance. Given that the proposed approach underperforms competing methods in terms of accuracy of recommendations, more efforts should be made to demonstrate its competitive advantages in terms of time required for training and generating predictions.
Another contradictory part is the generative step for candidate selection. The idea of using inference to generate the most pertinent user vector for a selected item hash code is novel and interesting. However, it requires searching neighbors in the real feature space, which can be very inefficient, depending on the structure of features. I’m not convinced that it is better than searching neighbors directly in the latent space, which can be done in the majority of hybrid models. Moreover, there exist various approximate nearest-neighbors search methods, e.g. Annoy, NMSLib, Faiss, etc., which allow trading-off accuracy and efficiency. Considering that hash codes also lose some information (which is observed in the results of experiments), it seems necessary to have a comparison with these approximate methods as well.
It should be also noted that in some cases you don’t even have to run the similarity search. Many hybrid models learn latent representation of features directly and cold-start entities are straightforwardly described via combination of the corresponding latent vectors of their features (e.g, Factorization Machines [Rendle 2009]). Hence, affinity between a cold item and some user can be quickly estimated via inner product of their latent vectors.
Suggestions on improving the work:
Worth mentioning, that recent studies raise certain concerns about the superiority of modern neural network-based approaches over simpler (and properly tuned) linear baselines, see the work by [Dacrema, Cremonesi, Jannach 2019] on “A Worrying Analysis of Recent Neural Recommendation Approaches”. The DropoutNet method in your experiments is very similar to CDL in terms of accuracy. The latter, however, underperforms even simple knn models, as shown by the work mentioned above. The haven’t tested it in the cold-start regime, though. Still, I’d strongly recommend adding to your experiments comparison with simpler hybrid models, e.g., Factorization Machines. Also note that there are even stronger baselines published recently, e.g. HybridSVD by [Frolov, Oseledets 2019].
Additional remarks:
1)	Figure 3 and the related text seem to focus on too obvious things. Indeed, by increasing the number of entities to compare against, you increase chances to have a hit. This part of the text, basically, states that the method works, which can already be seen from other results.
2)	A lot of attention is given to the “marketing application”. It’s ok to have it in introduction and make a connection to the real-world problem; however, further mentions of it in the text feel excessive. In the experiment section you describe a standard evaluation procedure for the cold start, there is no need to refer to marketing application again as you do not provide any new metric. It would feel much more organic if you would have the results of A/B testing on real users. Otherwise, I’d suggest to focus more on the problem that you’re solving, not on possible application.
3)	The text is in a very unpolished state. It reads more like a draft version. There are many typos and error both in text and in derivations.
References:
Rendle, Steffen. "Factorization machines." In 2010 IEEE International Conference on Data Mining, pp. 995-1000. IEEE, 2010.
Frolov, Evgeny, and Ivan Oseledets. "HybridSVD: when collaborative information is not enough." In Proceedings of the 13th ACM Conference on Recommender Systems, pp. 331-339. ACM, 2019.
Dacrema, Maurizio Ferrari, Paolo Cremonesi, and Dietmar Jannach. "Are we really making much progress? A worrying analysis of recent neural recommendation approaches." In Proceedings of the 13th ACM Conference on Recommender Systems, pp. 101-109. ACM, 2019.


We theoretically prove that learning with permutation invariant deep neural networks largely improves their generalization performance. Learning problems with data that are invariant to permutations are frequently observed in various applications, for example, point cloud data and graph data. Numerous methodologies have been developed and they achieve great performances, however, understanding a mechanism of the performance is still a developing problem. In this paper, we derive a theoretical generalization bound for invariant deep neural networks with a ReLU activation to clarify their mechanism. Consequently, our bound shows that the main term of their generalization gap is improved by √ n! where n is a number of permuting coordinates of data. Moreover, we prove that an approximation power of invariant deep neural networks can achieve an optimal rate, though the networks are restricted to be invariant. To achieve the results, we develop several new proof techniques such as correspondence with a fundamental domain and a scale-sensitive metric entropy.

This paper presents a derivation of a generalization bound for neural networks designed specifically to deal with permutation invariant data (such as point clouds). The heart of the contribution is that the bound includes a  1/n! (i.e. 1 / (n-factorial)) factor to the major term, where n is the number of permutable elements there are in a data example (think: number of points in a point cloud). This term goes some way towards making the bound tight.
The 1/n! factor in the bound may be an interesting development  but the novelty does appear to be limited.  Also, the authors fail to discuss that --  as part of that same term -- there is a factor: (1 / (epsilon^p)), where p is the dimension of the input and epsilon is a small error term. As p is proportional to n, and epsilon is quite small, this term could well dominate the factorial in many practical settings. A discussion of the relation between these terms is appropriate and seems to be missing.
Clarity:
In general the paper is fairly well written, but there are multiple instances of missing articles and strange idiom violations (eg. p. 4, remark 1: "such the bound" versus "such a bound")
More seriously, the proof of Lemma 1 was quite hard to follow (esp. the second paragraph). I would suggest putting less emphasis on the relatively straightforward construction of the sorting mechanism in Propositions 2 and 3, and use the space to more clearly detail the proof of Lemma 1, which is, after all, the heart of the contribution.
I also found the proof of proposition 4 too confusing to easily follow. What is the interpretation of the indices (1, ..., K) on the functions?
Finally, I would have liked to see some interpretation of the findings in a discussion section (or in an extended conclusion).
Minor issues:
- First sentence of the abstract is difficult to parse and does not seem like an accurate assessment of the contribution of the paper.
- Paragraph 2 of the introduction presents a sequence of argument whose logic seems inconsistent to me. There is a drift from a discussion of generalization of neural networks to a mention of work on the very distinct topic of the representational capacity of neural networks (i.e. universal approximation property of neural networks). The linking text "To tackle the quesiton, ..." is not appropriate.
- Unlike Example 1, Example 2 (p.3) is not helpful in motivating the permutation invariant neural networks. The definition makes direct reference to Proposition 2 that will not be introduced for another 3 pages.
- In Sec. 4.1, it seems like a phi symbol is used when I believe a null symbol was intended
- Proposition 3: "max( z_1, z_1 )"  should be "max( z_1, z_2 )" with the adjustment carrying through to the other side of the equals.


In continual learning, the learner faces a stream of data whose distribution changes over time. Modern neural networks are known to suffer under this setting, as they quickly forget previously acquired knowledge. To address such catastrophic forgetting, many continual learning methods implement different types of experience replay, re-learning on past data stored in a small buffer known as episodic memory. In this work, we complement experience replay with a meta-learning technique that we call "anchoring": the learner updates its knowledge on the current task, while keeping predictions on some anchor points of past tasks intact. These anchor points are learned using gradient-based optimization as to maximize forgetting of the current task, in hindsight, when the learner is fine-tuned on the episodic memory of past tasks. Experiments on several supervised learning benchmarks for continual learning demonstrate that our approach improves the standard experience replay in terms of both accuracy and forgetting metrics and for various sizes of episodic memories.

This paper proposes an approach for continual learning that improves existing memory / replay -based methods, by learning anchors for each previous task. This is achieved by computing a temporary parameter update with a conventional loss (on both the current task and samples from an episodic memory buffer); and trading off this update with an additional loss measuring the reduction in performance on a set of "anchor samples" from previous tasks. Gradient descent is used to estimate the task-specific anchors that are most crucial across all (past and future) tasks.
The paper is well-written, it is a novel and well-explained idea, and the results are compelling. I lean towards acceptance, but I think there are a number of things that could be improved before final publication.
1) On how the anchors are estimated:
a) It is stated on page 3 that "... choose each e_t as a tool to minimize the Forgetting metric (Eq. 3). That is, we are interested in letting e_t be the example from task t that would maximize the amount of forgetting about task t throughout the entire learning experience...."
This appears to be contradictory - is the aim to find the example that would maximise the amount of forgetting for a given task after training on all tasks, in order to keep samples that are most "crucial"?
This section could be clearer, and further discussion and intuition would be beneficial for the reader.
b) Does it always make sense to choose anchors that maximise forgetting on the current task? Is this akin to finding examples that would optimally alter the decision boundary, in a similar fashion to support vectors?
c) I really like the intuition behind optimizing for the anchors via gradient descent (since finding examples that would be most forgotten in future violates the continual learning assumption). It would be interesting to visualize what anchors are found. For example, do they tend to be close to the centroid of a class (such that the mean embedding regulariser is low)? Are they examples that tend to be visually close to other classes, or are they outliers?
d) Given that anchors are already estimated and used to alleviate forgetting, what's the benefit of the additional episodic memory? An ablation is performed with different memory sizes, but what if *only* anchors are used (ie. a size of zero)? This would be useful in delineating the benefits of the anchors versus standard replay.
2) On the two-step optimisation (Eqn. 5):
a) It's not clear to me why this constitutes a meta-learning process: there is no adaptation at test time, nor does this perform task inference or learn how to learn. In this case, the method just uses a single gradient step to compute the change in predictions at the anchor points.
b) I wonder if there is a relationship to the loss used in Riemer et al, ICLR 2019. In particular, it seems that the gradient dot product term (used in that work to maximise transfer) may be related to a first-order Taylor expansion of the L2 term in this work (the change in predictions at anchor points, Eqn 5). It would be good to clarify whether this is the case, and add further discussion and intuition as needed.
3) On the relationship to existing work:
a) The approach seems conceptually most similar to iCARL (cited in the paper), which maintains "exemplars" from each previous task in order to avoid forgetting. I think the writing could be much more explicit about this, and both (i) compare against the performance of iCARL, and (ii) clearly discuss the differences between the iCARL exemplars and the anchors obtained in this work.
The equivalent result for splitCIFAR in the iCARL paper (for 20 tasks x 5 classes) appears to be around 45-50%, so I think it is necessary to compare and evaluate the source of this large improvement in performance.
b) Another paper worth discussing and comparing further is Variational Continual Learning - which is effectively a regularization-based method augmented with a coreset / episodic memory.
c) The paper seems quite centred on memory/replay- based approaches to continual learning; the abstract in particular says "SOTA continual learning methods implement different types of experience replay", but this fails to focus on non- memory-based methods. I would temper the claim to "many approaches to continual learning...", and spread the focus beyond replay-based approaches.


We propose a novel supervised learning method to optimize the kernel in maximum mean discrepancy generative adversarial networks (MMD GANs). Specifically, we characterize a distributionally robust optimization problem to compute a good distribution for the random feature model of Rahimi and Recht to approximate a good kernel function. Due to the fact that the distributional optimization is infinite dimensional, we consider a Monte-Carlo sample average approximation (SAA) to obtain a more tractable finite dimensional optimization problem. We subsequently leverage a particle stochastic gradient descent (SGD) method to solve the derived finite dimensional optimization problem. Based on a mean-field analysis, we then prove that the empirical distribution of the interactive particles system at each iteration of the SGD follows the path of the gradient descent flow on the Wasserstein manifold. We also establish the non-asymptotic consistency of the finite sample estimator. Our empirical evaluation on synthetic data-set as well as MNIST and CIFAR-10 benchmark data-sets indicates that our proposed MMD GAN model with kernel learning indeed attains higher inception scores well as Frèchet inception distances and generates better images compared to the generative moment matching network (GMMN) and MMD GAN with untrained kernels.

This paper proposes to learn a kernel for training MMD-GAN by optimizing over the probability distribution that defines the kernel by means of random features. This is unlike the usual setting of MMD-GAN where the kernel is parametrized by composing a fixed top-kernel with a discriminator network that is optimized during training. The main motivation for this approach is to avoid having to 'manually' fix some parameters of the top-level kernel like the bandwidth of the RBF kernel. They provide an algorithm to achieve such optimization in probability space along with some consistency results and perform experiments on MNIST and Cifar10 to demonstrate empirically the advantage of such an approach over those that fix the top-level kernel.
Theory:
	Theorem 4.1 provides a convergence result of an oracle finite-sample estimator: that is the one obtained by exactly solving the optimization problem in 19b. In that case, they show the consistency of the proposed estimator. The result is somehow expected but the proof relies on nice duality results for measures and is very technical.
The clarity of the proof could be improved:
- Currently, the structure of the main proof mixes direct lemma (lemma B.7) with less obvious ones (lemma B.6). Also, some concepts are introduced in the main proof but not necessary for its understanding: The notion of Orlicz norm is in introduced in Definition B.3 on the fly to state lemma B.6, but only equations 50 and 51 are used in this lemma which does not make use of the notion of Orlicz norm at all.
	Theorem 4.1 doesn't say anything about the consistency of the algorithm itself. To partly address this, the authors show in theorem 4.2 that as the number of particles grows the empirical process converges to a McKean Vlasov PDE (equation 22). This means that the proposed algorithm is approximating some gradient flow in metric space (25).
- However, this gradient flow is a non-convex optimization problem and there is no guarantee that a global solution is reached.  Recent work provides cases when global convergence occurs [Chizat2018] but it is not the case in general. Some further clarification about the connection between Thm 4.1 and Thm 4.2 would be therefore useful.

Thm 4.2 is also curious in the sense that the process defined by equation 16, which is noisy since it relies on one sample from the data, would converge towards (22) which is a Mc-valsov equation with a drift only (no diffusion or other noise).  What happened to the noise coming from sampling from P_v  and P_w in equation 16? Wouldn't there be some sort of diffusion term as in [Hu2018]?

More generally it would be nice to have a discussion of the assumptions and results in the paper as they seem to rely on methods that people in the machine learning community are not totally familiar with.

Experiments:  The experiments are not convincing for several reasons:
 - The comparison with the other methods is somehow unfair since the bandwidth is manually tuned for the competing methods. It is easy to adaptively learn the bandwidth as well:  in this case, it will be just an additional parameter of a discriminator network. This was done in [Arbel2018] where a single gaussian kernel is used and a regularization of the critic allows to learn the bandwidth without manual tuning. Does the proposed method offer an additional advantage compared to those?
- In practice and for a scaling parameter alpha=1, isn't the algorithm strictly equivalent to considering an MMD-GAN with a dot product kernel and a discriminator given by the feature \phi(x,\zeta)?
- Mnist and Cifar10 are somehow very simple, what would happen on more complicated datasets (CelebA or imagenet)?
- I also think there is an ablation that is missing: If the auto-encoder also needs to be optimized then does it also help to optimize over the particles as well or is optimizing the auto-encoder discriminator enough to achieve a similar performance? In other words, does optimizing the auto-encoder compensate for the need to learn the distribution mu? Of course, this would depend on how the auto-encoder is parametrized but I don't see why it wouldn't in many cases.
Overall, I'm not convinced that the proposed approach would lead to any substantial improvement for MMD-GAN in practice, and the experiments are not really convincing as they are now. However, I find the theoretical results interesting and might be used to better analyse the dynamics of GAN's. But as the paper is currently framed, it is hard to put these theoretical results in perspective.


While many option discovery methods have been proposed to accelerate exploration in reinforcement learning, they are often heuristic. Recently, covering options was proposed to discover a set of options that provably reduce the upper bound of the environment's cover time, a measure of the difficulty of exploration. However, they are constrained to tabular tasks and are not applicable to tasks with large or continuous state-spaces. We introduce deep covering options, an online method that extends covering options to large state spaces, automatically discovering taskagnostic options that encourage exploration. We evaluate our method in several challenging sparse-reward domains and we show that our approach identifies less explored regions of the state-space and successfully generates options to visit these regions, substantially improving both the exploration and the total accumulated reward.

Summary
The authors introduce deep covering options, an online mechanism to extend the covering options to large state spaces. They claim their method discovers options that are task agnostic. The method is evaluated in sparse reward domains and claims to gain improvement in exploration and performance as well.  The authors extend the recent developments in eigenfunction estimation of the Laplacian to a principled approach for option discovery to non-linear function approximation.
Covering options compute the second smallest eigenvalue and the corresponding eigenvector f of the Laplacian exactly by solving a constrained optimization problem (Eq2). However, this requires the adjacency matrix A as input, and a constrained optimization problem is hard to solve using gradient-based methods. To overcome this, the authors propose an approximation of the computation of the Laplacian with Eq3. This allows them to have a constraint-free objective to compute the eigenfunction which is now dependent on the trajectories and avoids requiring the state-space graph. I believe the paper presents interesting ideas and is definitely a very useful contribution.  However, the paper needs work on thorough empirical analysis: could use more rigorous baselines especially to fairly evaluate the gains in exploration.
Detailed comments:
(I) Major Concern:  Figure 1 of this work is exactly the same as Figure 2 of the Jinnai et al., 2019b. It is not clear to me whether a) this is being referred for the purpose of giving intuitions. If yes, then it should be cited as Figure from Jinnai et al., 2019b or b) this is a new figure generated differently, it is not clear what the difference is. Overall, there is overlapping content and should be clarified what is original work and what is being referred to.
Reusing content from another paper without proper attribution is normally considered plagiarism. More precisely; Jinnai et al., 2019b. mention that the “second smallest eigenvalue of L is known as the algebraic connectivity of the graph and its corresponding eigenvector is called Fiedler vector” and caption this figure:
“Figure 2: The distance between the red state and all other states, measured via the Fiedler vector (left) and Euclidean distance (right). The Fiedler vector captures the connectivity of the graph, so distances measured using it reflect path lengths in the graph; the pair of nodes with the maximum and the minimum value are the farthest apart”
This  work (currently in review) captions this figure as:
“Figure 1: The distance between the red state and all other states, measured via the second eigenvector (left) and Euclidean distance (right). The second eigenvector captures the connectivity of the graph, so distances reflect path lengths in the graph; the pair of nodes with the maximum and minimum values are the farthest apart.” The only words changed are Fiedler vector to the second eigenvector. Please explain!
(II) An important step in the algorithm is line 3” identify an under-explored region in the state-space using the eigenfunctions”. Where does the parameter k come from? Is this a hand-designed parameter such as in PinBall it is somehow set to 30, for Mujoco tasks this is somehow set to 10, and for Atari, this is set to 4.  Would it be possible to comment on the hyperparameter threshold percentile k? Why is this a hard-coded choice? Would it be possible to learn this *simultaneously* as the options?
(III) Can you comment on to what extent do you see theoretical guarantees of covering options apply to the function approximation case as they no longer hold when going from tabular setting to the nonlinear function approximation? Would it be possible to also comment on what can be said about any guarantees at all if the state space is very very large? Forex: imagine a lifelong learning scenario where the environment is really big, and it is just not almost impossible to visit all states, how does this objective function of minimizing the upper bound on the expected cover time constitute the right choice?
(IV) Intuitively the pseudo-reward seems a lot related to goal-based rewards, where the skill is learned to reach different goals such as in DIAYN. Can you comment on how is this different and why is the proposed approach better in principle?
 Empirically, It is not clear why DIAYN was not compared as it is and with all the stated modifications.  In DIAYN, setting the initiation set to be the whole state space seems counterintuitive.
(V) Regarding the connectivity of the states to generate a diverse set of options: there seem to be connections to the work on constructing options using stronger guarantees Castro & Precup, 2011. It might be useful to comment on this and discuss this in the paper.
(VI) “We sampled 200 episodes of length 2000 with a uniform random policy to generate each option.” Would there be smarter ways to generate each option? Is there a reason to generate each option in this fashion?
(VII) Since the idea here is to connect states that are closer in terms of time, but further apart, how does the proposed approach comparison to successor options. I would imagine a comparison to successor options would be quite valuable to this. Please comment on this.
(VIII) “Termination set generated by deep covering options tends to be larger than..” Intuitively speaking, it is not clear why this is a good idea. Wouldn't we want the options to be peaked in places they terminate and initiate in, and therefore have smaller termination/initiation set?
(IX) In the Mujoco tasks: I would expect to see the baseline performance reported in the main paper i.e. DIAYN for continuous control tasks too. It is not clear by “did not outperform the baseline” as to how the proposed approach fairs in comparison to DIAYN. Looking into the appendix, It is still not clear why the plot of DIAYN in mujoco tasks is not included.
Since the key ideas in this work propose overcoming exploration as the main challenge, one would expect a comparison to the state-of-art in exploration, for instance [1]. Also since the work builds on eigenoptions, it would help compare with [2] as well. Without these comparisons, I find the empirical analysis rather weak.
(X) Pinball exploration visuals show concrete gains in state-space explored 4a-d. This is also evident in continuous control tasks 4e-f-g. However, in both cases, I find the comparison weak as the authors do not compare against state-of-the-art exploration baselines.
(XI)  The termination sets in ALE are interesting in that they do convey that options terminate in different regions of the state space: but one can also see options terminating in different regions of the state space in Harb et al, 2018 in Amidar here for example. It is also counter-intuitive as to why the options terminate in regions that do not overlap with visible goals for example in Montezuma Revenge in the key or skull. Perhaps one benefit here is that there is no reward information, but we do not see either the performance curve or the nature of options in ALE, so it is really hard to make a strong claim either way.
Although the options in pre-training were generated without a reward- I would recommend using these options in different tasks to make the claim “task agnostic options are discovered” much stronger.
[1] Count-based exploration with the successor representation.
[2] A laplacian framework for option discovery in reinforcement learning
Overall
Scales a principled approach to function approximation for deep covering options. The method seems to be computationally tractable. The approach can be applied to both settings where an unsupervised pre-training phase is available and also in a fully online setting.
However, DIAYN is the only baseline in Mujoco and is not shown. It’s unclear why other baselines were not used such as Eigenoptions, which seems like a very valid baseline. In addition, baselines of well-known exploration algorithms also have not been fully explored. The paper needs a more thorough evaluation of the proposed method to make the claims stronger.
Please note that although I have marked a weak reject, I am open to adjusting my score if the rebuttal addresses enough issues.


Planning at roundabout is crucial for autonomous driving in urban and rural environments. Reinforcement learning is promising not only in dealing with complicated environment but also taking safety constraints into account as a as a constrained Markov Decision Process. However, the safety constraints should be explicitly mathematically formulated while this is challenging for planning at roundabout due to unpredicted dynamic behaviour of the obstacles. Therefore, to discriminate the obstacles' states as either safe or unsafe is desired which is known as situation awareness modeling. In this paper, we combine variational learning and constrained reinforcement learning to simultaneously learn a Conditional Representation Model (CRM) to encode the states into safe and unsafe distributions respectively as well as to learn the corresponding safe policy. Our approach is evaluated in using Simulation of Urban Mobility (SUMO) traffic simulator and it can generalize to various traffic flows. (Anonymous code is available to reproduce the experimental results and additional videos are also available 1 .)

This paper tackles the challenge of control with safety constraints using a learned soft constraint formulation. It models data as coming from a hierarchical generative model p(s) p(z|s) p(x|z,s) where s is a safety indicator variable, z is a latent variable, and x is an observation. Then it learns a variational approximation q(z|x) to the true posterior p(z|x). By then measuring a divergence between q(z|x) and p(z|s), the authors aim to evaluate whether a state is safe or unsafe. They then combine this learned safety constraint with a constrained RL method, SSAC, to learn safe policies.
While this setting is interesting and worthy of further exploration, there are sufficient issues with this work that it should not be accepted at this time.
For one thing, the motivation behind the constrained MDP formulation of reinforcement learning, such as the original SSAC paper, is to provide safety throughout training. However, this work learns a soft constraint over the course of training by using examples where the policy leads to catastrophic failures. This means that, unlike SSAC, this work does not encourage safety at all early in training. Since it collects a large number of catastrophic experiences, this method is not categorically different from e.g. training a policy with large negative rewards for crashing.
Furthermore, since there is no constraint preventing crashing, a poorly-performing learned constraint might lead to better performance as measured by reward alone. Since safety during training comes with a tradeoff against rewards, as shown by SSAC, a poorly-functioning learned constraint might lead to improved reward. This work lacks an evaluation of the number of crashes to complement the rewards depicted in Figure 6.
The particulars of the method used for computing whether the safety constraint is violated are somewhat surprising. The authors use a Wasserstein distance to compute a cost as a function of the q(z|x) and p(z|s=1). They motivate this choice by the fact that the KL divergence would go to infinity for non-overlapping distributions; however, in equation (12) that would not significantly affect the computed cost. Since this divergence is calculated in the low-dimensional latent space and one of the distributions is fixed, it is also unclear that this would ever arise. It also seems that a likelihood ratio test between p(x|s=1) and p(x|s=0) would provide a more meaningful signal than simply classifying whether a point is near the "dangerous" prior p(z|s=1).
Overall I think there are some interesting ideas inside this work, but it needs some improvements:
1. Reframing to make the setting make more sense. If you want to compare against SSAC, you need to be minimizing the total number of catastrophic events during training. It might make sense to assume a pre-existing set of "dangerous" examples, e.g. labeled ahead of time by a human.
2. Textual editing and work to make the notation more consistent, e.g. the top of page 3 uses s for states as well as the "safe" indicator.
3. Significantly improved evaluation. The results here lack crucial information about the number of catastrophic events during training. I would also like to see ablations or probe experiments showing what is learned by the encoder. Furthermore, this one environment is extremely simple (one degree of freedom) and to have confidence that the method works more generally, I would like to see the method applied to richer tasks.


High intra-class diversity and inter-class similarity is a characteristic of remote sensing scene image data sets currently posing significant difficulty for deep learning algorithms on classification tasks. To improve accuracy, post-classification methods have been proposed for smoothing results of model predictions. However, those approaches require an additional neural network to perform the smoothing operation, which adds overhead to the task. We propose an approach that involves learning deep features directly over neighboring scene images without requiring use of a cleanup model. Our approach utilizes a siamese network to improve the discriminative power of convolutional neural networks on a pair of neighboring scene images. It then exploits semantic coherence between this pair to enrich the feature vector of the image for which we want to predict a label. Empirical results show that this approach provides a viable alternative to existing methods. For example, our model improved prediction accuracy by 1 percentage point and dropped the mean squared error value by 0.02 over the baseline, on a disease density estimation task. These performance gains are comparable with results from existing post-classification methods, moreover without implementation overheads.

The authors propose a method to extract features utilizing the adjacency between patches, for better classification/regression of satellite image patches. The proposed method achieves better results compared to a straightforward baseline method.
I have several significant concerns:
- In the abstract, the authors claim that existing approaches such as post-classification add computational overhead to the task, whereas the proposed method does not add significant overhead. However, to me, post-classification can be very simple and straightforward, whereas the proposed method adds a series of computations: the proposed method not only extracts features from the input image, but also for another neighboring image; then features are combined (if two images are similar), before feeding into the network. The authors need to validate the claim that their method is more efficient.
- The baseline the authors compare to is weak. There are existing works on satellite image classification/regression. Many of them also use semantic/contextual information, or aim to improve the robustness of features. For example:
[1] Derksen et al. Spatially Precise Contextual Features Based on Superpixel Neighborhoods for Land Cover Mapping with High Resolution Satellite Image Time Series. IGARSS 2018.
[2] Ghassemi et al. Learning and Adapting Robust Features for Satellite Image Segmentation on Heterogeneous Data Sets. Geoscience and Remote Sensing 2019.
I understand that the authors cannot compare to everything. But the authors should compare to representative baseline methods. Methods mentioned in the related work section (Section 2.1) can also be compared to.
- The proposed method is very application specific. The author only discussed the remote sensing application. Given the ICLR community's interest in general methods that can be applied to (or already been tested on) multiple applications, the paper would have been stronger if the methods applicabilityto other domains was discussed (and even better demonstrated).


This paper focuses on the theoretical investigation of unsupervised generalization theory of generative adversarial networks (GANs). We first formulate a more reasonable definition of general error and generalization bounds for GANs. On top of that, we establish a bound for generalization error with a fixed generator in a general weight normalization context. Then, we obtain a width-independent bound by applying p,q and spectral norm weight normalization. To better understand the unsupervised model, GANs, we establish the generalization bound, which uniformly holds with respect to the choice of generators. Hence, we can explain how the complexity of discriminators and generators contribute to generalization error. For p,q and spectral weight normalization, we provide explicit guidance on how to design parameters to train robust generators. Our numerical simulations also verify that our generalization bound is reasonable.

This paper proves generalization bounds for GANs. I think the paper can be improved significantly in several ways:
1- Writing: The first two sections are relatively well-written. The problem starts at section 3 and continues after that. Some of the things that can be improved:
a) The discussion on the different definitions of generalizations is not really helpful in the current format. You might want to explain how these different definitions relate to each other. For example, if generalization in one of them implies generalization in the other one, etc.
b)Theorem 2.3 is a general statement but it is followed by Corollary 3.3 which is a very specific generalization bound. There is no explanation how one can show the corollary.  Even worse is mixing these two in the proof of the theorem in the appendix.   Please consider improving the use of Theorems, Lemmas and Corollaries.
c) Section 3 and 4 have bunch of theorems and collieries without much explanation. It is not clear that all of these are actually helpful for the main purpose of the paper.
d) I don't completely understand the notation in Corollary 3.3. Eg. what is d_{f,\ell}?
2) Related Work: I think authors need to do a more comprehensive literature review on generalization bounds. Since the generalization bounds presented here are built on the supervised learning bounds, authors discuss the generalization bounds in supervised learning. For example, authors heavily rely on Chen et al. (2019) for their generalization bounds while very similar results where shown before by [1] and [2].
[1] Neyshabur, Behnam, Ryota Tomioka, and Nathan Srebro. "Norm-based capacity control in neural networks." Conference on Learning Theory. 2015.
[2] Golowich, Noah, Alexander Rakhlin, and Ohad Shamir. "Size-independent sample complexity of neural networks." Conference on Learning Theory. 2018.
3) Definition of generalization: I don't think the definition of generalization suggested in this work is much different than Arora et. al. since f really doesn't depend on samples from D_g and hence the empirical and true distributions are not very different. In fact, I think the definition provided by Arora et. al. 2017 is preferred because at the end of the day, we have to estimate the distribution D_g by generating some samples.
4) Generalization bound for fixed g: Unfortunately, the novelty of these generalization bounds are very limited as they are a direct application of known generalization bounds in the supervised settings. Therefore, the authors contributions are very limited here.
5) Generalization bounds for all generators: Again, here the novelty and final result is very limited since the bounds achieved by a union bound arguments and does not really go beyond that.
6) Experiments: Experiments can also be improved significantly. Currently, the correlation is reported for 5 trained networks and it is not clear to me that this result is statistically significant. Moreover, only one hyper-parameter is changed in the experiments which could be problematic. I suggest authors to change multiple hyper-parameters and train more networks to improve the evaluation.
******************************
After author rebuttals:
I have read the authors response and looked at the revision. Unfortunately, many of my concerns are not addressed adequately so my score remains the same.


Attention operators have been widely applied on data of various orders and dimensions such as texts, images, and videos. One challenge of applying attention operators is the excessive usage of computational resources. This is due to the usage of dot product and softmax operator when computing similarity scores. In this work, we propose the Siamese similarity function that uses a feed-forward network to compute similarity scores. This results in the Siamese attention operator (SAO). In particular, SAO leads to a dramatic reduction in the requirement of computational resources. Experimental results show that our SAO can save 94% memory usage and speed up the computation by a factor of 58 compared to the regular attention operator. The computational advantage of SAO is even larger on higher-order and higher-dimensional data. Results on image classification and restoration tasks demonstrate that networks with SAOs are as effective as models with regular attention operator, while significantly outperform those without attention operators.

The authors introduce a novel self-attention operator for neural networks. Their self-attention operator computes similarity between elements a and b as (a+b)^Tw where w is a learned parameter and does not use the softmax operator. This leads to improvements in space and time complexity compared to regular self-attention which uses the dot product (a^Tb).
They show that concatenating their operation with convolution brings improvements over the MobileNetv2 baseline on ImageNet classification and over U-Net on restoration tasks.
Attention has been empirically shown to bring improvements in many visual tasks but certain methods (such as self-attention) can be quite expensive in computations and memory. Identifying cheaper attention mechanisms that obtain similar accuracy performance as expensive attention mechanisms is therefore an important direction for work.
However, I take several crucial issues with this work and especially the evaluation/presentation of the methods:
- Although this is the focus of the work, the authors do not report actual memory consumption and latency times for MobileNetv2, SANet and the other attention mechanisms. (There is no need for the simulated scenarios of Table 2 since we already know the theoretical complexities of the different methods).
- The authors only compare their methods against regular self-attention (without or with pooling/softmax), and ignore a longstanding literature of other (potentially cheaper in terms of memory and computations) attention mechanisms in vision (see below). Without comparison to at least Squeeze-and-Excite, it is hard to evaluate the significance of the method presented in the draft.
- The motivation for naming the method "siamese" is quite poor. Siamese networks typically are more complex than a single layer feed forward (which is just a dot-product). Furthermore, the siamese similarity (as introduced by the authors) does not respect the usual properties of similarity functions. For example siasim(a, 0) = a^tw = 1/2 siasim(a,a) can take arbitrary values including negative values ("a can be dissimilar with itself")
- X vs Q, K, V? Self-attention is incorrectly described as "a special case of the attention operator with Q = K = V", instead of Q = XW^Q, K=XW^K, V = XW^V.
- In Table 7, shouldn't SANet w/o params have less params than SANet?
In summary, the paper addresses an important challenge and proposes a technically sound method.  However, the current  draft has fundamental experimental flaws in its evaluation/presentation and lacks comparison against relevant cheap channelwise attention mechanisms (such as Squeeze-and-Excitation). I argue for rejection.
Relevant literature:
- channelwise attention: Squeeze-and-Excitation, Gather-Excite
- Channelwise and spatial attention: Bottleneck Attention Module, Convolutional Block Attention Module
- Relative Self-attention for vision: Attention Augmented Convolutional Networks, An Empirical Study of Spatial Attention Mechanisms in Deep Networks.


We focus on temporal self-supervision for GAN-based video generation tasks. While adversarial training successfully yields generative models for a variety of areas, temporal relationship in the generated data is much less explored. This is crucial for sequential generation tasks , e.g. video super-resolution and unpaired video translation. For the former, state-of-the-art methods often favor simpler norm losses such as L 2 over adversarial training. However, their averaging nature easily leads to temporally smooth results with an undesirable lack of spatial detail. For unpaired video translation, existing approaches modify the generator networks to form spatio-temporal cycle consistencies. In contrast, we focus on improving the learning objectives, and propose a temporally self-supervised algorithm. For both tasks, we show that temporal adversarial learning is key to achieving temporally coherent solutions without sacrificing spatial detail. We also propose a novel Ping-Pong loss to improve the long-term temporal consistency. It effectively prevents recurrent networks from accumulating artifacts temporally without depressing detailed features. We also propose a first set of metrics to quantitatively evaluate the accuracy as well as the perceptual quality of the temporal evolution. A series of user studies confirms the rankings computed with these metrics.

Summary:
This paper proposes a training objective for higher quality video generation for the tasks of Video Super Resolution (VSR) and Unpaired Video Translation (UVT) and also two evaluation metrics tOF and tLP. They provide a comprehensive ablative study of the proposed method and also show comparisons against baselines for the tasks of VSR and UVT.
Pros:
+ Novel video generation method for VSR and UVT.
+ Novel metrics
+ Well written paper
Weaknesses / comments:
- Confused about inputs to discriminator (I_{w,g} and I_{w,b})
I am not sure I understand the inputs I_{w,g} and I_{w,b} to the discriminator network. Based on the description, I_{w,g} = {W(g_{t-1}, v_t), g_t, W(g_{t+1}, v_t^’)} = {g_t, g_t, g_t}, assuming v_t^” means reverse flow. A similar process seems to be happening with I_{w,b}. If this is the case, will the discriminator optimally get the same frame concatenated together (i.e. {g_t, g_t, g_t})? Now, if that’s the case, how is the discriminator modeling temporal consistency other than making the generator learn to put pixels forward and back in place since the “original triplets” (real data) are {g_{t-1}, g_t, g_{t+1}} and {b_{t-1}, g_t, g_{t+1}}, respectively. This is very confusing to me. It would be good if the authors can clarify this in the rebuttal.
- How do you prevent the zero output in PP loss?
The PP-loss compares forward and backward prediction by || g_t - g_t^’ ||. If not careful, the neural network can just learn to output zeros or the same frame for the full video. Did the authors see any behavior like this? Or did the proposed formulations prevent this? Or was there a very small weight applied to this loss?
- TecoGAN vs baselines generator parameters (rather than TecoGAN^{-}).
Based on the experimental section, it looks like TecoGAN is the main network being compared to the baseline methods. TecoGAN has more parameters in the generator compared to TecoGAN^{-}. Did the authors make sure that the generator had the same number of parameters as the other methods? The authors mention a performance difference between TecoGAN and DUF due to DUF having more parameters, but what about the other baselines?
- Evaluation metric contributions in the supplementary material?
The description of the two proposed metrics tOF and tLP have been placed in Appendix B. These are mentioned as contributions of the paper so their description should be in the main text. Please make sure that Appendix B is in the main text in future versions of this paper.
- UVT task evaluation in the supplementary material?
The UVT task is mentioned in the abstract and as a target task in this work, however, the evaluations for this are in the Appendix. Please move them to the main text in future versions of this paper. Secondary experiments should be in the Appendix but I feel this is primary.
- UVT task only evaluated with the proposed evaluations?
The UVT evaluation in the Appendix only has the proposed metrics as evaluation. I understand that, since it’s an unpaired video translation task, there is no ground truth to compare against. However, a human based study could be done where human raters would judge for the more realistic of N videos.
- The same qualitative comparisons in the paper are not in the provided website.
The provided website does not have the same qualitative evaluations as the paper does. For example: There are 7 methods being compared for the same video in Figure 8 in the supplementary material, but I don’t see anything like that in the website.
- For the UVT task, we omit the DsDtPP model because …….
In the paper, the authors mention that they don’t provide the DsDtPP baselines because it requires a lot of computation. However, I feel these missing baselines make the experiments incomplete since it’s a different task.
- Figure 5 has no point of reference with which to compare the frames.
Figure 5 points out that the video b) is better than video a). While it is true that one seems more noisy than the other, there should be a point of reference for readers to make sure of it.
Conclusion:
In conclusion, the paper seems to present a novel method and evaluation metrics but has many issues as stated above. It would make the submission better if the authors can address them in the rebuttal.


We present a novel black-box adversarial attack algorithm with state-of-the-art model evasion rates for query efficiency under ∞ and 2 metrics. It exploits a sign-based, rather than magnitude-based, gradient estimation approach that shifts the gradient estimation from continuous to binary black-box optimization. It adaptively constructs queries to estimate the gradient, one query relying upon the previous, rather than re-estimating the gradient each step with random query construction. Its reliance on sign bits yields a smaller memory footprint and it requires neither hyperparameter tuning or dimensionality reduction. Further, its theoretical performance is guaranteed and it can characterize adversarial subspaces better than white-box gradient-aligned subspaces. On two public black-box attack challenges and a model robustly trained against transfer attacks, the algorithm's evasion rates surpass all submitted attacks. For a suite of published models, the algorithm is 3.8× less failure-prone while spending 2.5× fewer queries versus the best combination of state of art algorithms. For example, it evades a standard MNIST model using just 12 queries on average. Similar performance is observed on a standard IMAGENET model with an average of 579 queries.

Summary:
This paper proposes a black-box adversarial sample generation algorithm, which proceeds by learning the signs of the gradient using an adaptive query scheme, which the authors call _SignHunter_. The authors begin by deriving an upper bound on the query complexity for _SignHunter_. Next, the authors empirically compare the performance of _SignHunter_
with existing sign-based optimization schemes in the literature, and demonstrate the advantages of _SignHunter_.
Main Comments:
- Theoretical analysis of the Adaptive Scheme: The authors emphasize the adaptive nature of the proposed scheme as one of the contributions. However, this claim is not justified theoretically:  it would be interesting if the benefits of adaptivity could be quantified in terms of improved worst-case query complexity etc. as compared to any non-adaptive scheme. The query complexity of 2^{\log(n)+1} given in Theorem 1 is not very informative, since a lower complexity (n) is achieved by a simple open-loop  scheme which serially queries each coordinate (i.e., the first query is coordinate 1, the second query is coordinate 2, and so on).
-  Comparison with Hazan el. al (2018): A closely related work in optimizing real-valued functions with domain \{-1,1\}^n is Hazan et. al published in ICLR 2018, which assumes that the black-box function is sparse or compressible in the Fourier domain, and employs compressed sensing techniques to get fast convergence rates in the optimization error. Since that paper considers a very similar optimization problem and proposes a scheme with provable convergence guarantees,  I think it is important that the authors compare,  either theoretically or empirically,  the performance of their proposed scheme with Harmonica (the algorithm of Hazan et al (2018)) to justify the benefits their proposed scheme.
- Parameter Free Algorithm: The statement of Theorem 1 contains the following hypothesis: "the directional derivative is well approximated by the finite difference (Eq. 1)". This condition must be clarified, because it seems to contradict the claim made by the authors later in the last line of Page 5 that _SignHunter_ is "parameter-free" as they set \delta= \epsilon. The condition in the statement of Theorem 1 will not necessarily be satisfied for any \delta, and admissible values of \delta must depend on the properties of the function. Informally, I think that the condition will only be satisfied for \delta "small enough", and one way to make precise the meaning of "small enough" is in terms of the Lipschitz constant of the neural network. However, in that case, I am not sure that the algorithm would remain parameter-free.  I think the authors should make the assumption in Theorem 1 precise and derive suitable sufficient conditions on \delta under which the assumption is satisfied.
References:
1. Hazan, Elad, Adam Klivans, and Yang Yuan. "Hyperparameter optimization: A spectral approach." ICLR (2018).


Reinforcement learning (RL) typically defines a discount factor (γ) as part of the Markov Decision Process. The discount factor values future rewards by an exponential scheme that leads to theoretical convergence guarantees of the Bellman equation. However, evidence from psychology, economics and neuroscience suggests that humans and animals instead have hyperbolic time-preferences ( 1 1+kt for k > 0). Here we extend earlier work of Kurth-Nelson and Redish and propose an efficient deep reinforcement learning agent that acts via hyperbolic discounting and other non-exponential discount mechanisms. We demonstrate that a simple approach approximates hyperbolic discount functions while still using familiar temporal-difference learning techniques in RL. Additionally, and independent of hyperbolic discounting, we make a surprising discovery that simultaneously learning value functions over multiple time-horizons is an effective auxiliary task which often improves over state-of-the-art methods.

The paper investigates hyperbolic discounting as a more biologically plausible alternative to exponential discounting in reinforcement learning. First, it formulates a notion of hazard in MDPs as constant exponential discounting and shows that hyperbolic discounting is consistent with uncertainty over the hazard rate. The paper then shows how value functions learned with exponential discounting can be used to approximate value functions with other forms of discounting. Specifically, the paper shows in section 4 how exponentially-discounted value functions can be used to approximate hyperbolically discounted value functions. The paper then presents experiments on a small MDP and Atari 2600 games, showing that learning discounted action values with many different discount rates as an auxiliary task improves performance on most Atari games.
Overall, I very slightly tend towards accepting this paper for publication. While the idea of hyperbolic discounting didn't seem to pay off in terms of performance on the Atari 2600 games, the idea still has some merit as being more biologically plausible than exponential discounting, and may perform better on a more suitable environment. In addition, the discovery that learning many action-value functions with different exponential discounting rates as auxiliary tasks improves performance is quite interesting.
Notes:
- The footnote on the first page seems to just trail off. It would be better to explicitly state what the takeaway is. I'm guessing humans and animals in general would prefer \$1M now in the first scenario, but \$1.1M in the second scenario?
- The idea of 1/(1-gamma) being an "effective horizon" seems questionable. For gamma=0.9 the "effective horizon" (aka expected number of timesteps before termination) would be 1/(1-.9)=10 timesteps. However, 0.9^(10)=~0.3486 means ~34% of the probability density is still to the right of that number, so calling 1/(1-gamma) an "effective horizon" seems to violate what we normally mean by "horizon".
- A single hazard rate for the entire environment seems counterintuitive. Some states in life are definitely more hazardous than others, which suggests state-dependent discounting might be an interesting idea to explore.
- Many people in the field think we shouldn't even be doing discounting at all (section 10.4 of Reinforcement Learning by Sutton and Barto), only total reward for episodic problems and average reward for continuing problems.
- Pathworld experiments seem like they don't show anything useful; the hyperbolic discounting matches the true hazard behaviour of the environment (i.e., the hazard rate is drawn from a distribution) so it performs better than exponential discounting. However, if the environment had a fixed hazard rate then exponential discounting would perform better.
- Using only 3 random seeds does not seem like enough for the experiments. Also there is no measure of standard error or statistical significance on the graphs.
- Introducing a hyperparameter that increases computation and memory does not really seem like it's solving the problem, and certainly not "efficiently" as claimed in the abstract.
- How was the value of the new hyperparameter set for the experiments? Were several values tried? More importantly, how can a good value for the new hyperparameter be chosen without prior knowledge of the environment?
- Atari 2600 games don't really seem like a great environment to showcase the benefits of hyperbolic discounting.
- It would be interesting to see a comparison with existing auxiliary task approaches, but due to space constraints should probably be left for future work.


In many complex dynamical systems, artificial or natural, one can observe selforganization of patterns emerging from local rules. Cellular automata, like the Game of Life (GOL), have been widely used as abstract models enabling the study of various aspects of self-organization and morphogenesis, such as the emergence of spatially localized patterns. However, findings of self-organized patterns in such models have so far relied on manual tuning of parameters and initial states, and on the human eye to identify "interesting" patterns. In this paper, we formulate the problem of automated discovery of diverse self-organized patterns in such high-dimensional complex dynamical systems, as well as a framework for experimentation and evaluation. Using a continuous GOL as a testbed, we show that recent intrinsically-motivated machine learning algorithms (POP-IMGEPs), initially developed for learning of inverse models in robotics, can be transposed and used in this novel application area. These algorithms combine intrinsicallymotivated goal exploration and unsupervised learning of goal space representations. Goal space representations describe the "interesting" features of patterns for which diverse variations should be discovered. In particular, we compare various approaches to define and learn goal space representations from the perspective of discovering diverse spatially localized patterns. Moreover, we introduce an extension of a state-of-the-art POP-IMGEP algorithm which incrementally learns a goal representation using a deep auto-encoder, and the use of CPPN primitives for generating initialization parameters. We show that it is more efficient than several baselines and equally efficient as a system pre-trained on a hand-made database of patterns identified by human experts.

The focus of the presented paper is on formulating the automated discovery of self-organized patterns in high-dimensional dynamic systems. The introduced framework uses cellular automata (game of life) as a testbed for experimentation and evaluation and existing machine learning algorithms (POP-IMGEPs). The goal of the paper is to show that these algorithms can be used to discover and represent features of patterns. Moreover, an extension of SOTA algorithms is introduced and several approaches to define goal space representations are compared.
Overall, I have the impression this is an interesting paper that could be accepted to ICLR. The idea of applying IMGEPs to explore parameters of a dynamic system is novel and interesting, which could also simulate further research in this field. Furthermore, the paper well-written, technically sound, and the results are interesting. The overall contribution of the paper is in applying IMGEP algorithms to exploring parameters of dynamic systems and in comparing different algorithms along with an extensive set of experiments. As a point of criticism, a lot of (interesting) material was pushed to the Appendix. Resolving the references makes reading the paper harder. Moreover, given that this paper has more than 35 pages appendix material, it seems this work would better be suited for a journal as for a conference. There is a reason for papers to have a page limit and this work circumvents this limit by presenting a lot of additional material. Therefore, I am not willing to strongly support this work.
Specific Comments:
- Section 3.1: It is not clear how the initial system state is established. In Section 3.1. the text states that 'parameters are randomly sampled and explored' before the process starts, but it is not clear why a random sampling is used and what this means for the subsequent sampling. Later in the text (3.3) it becomes more clear, but here this appears too unclear.
-  Section 3.1: "distribution over a hypercube in \mathcal{T} chosen to be large enough to bias exploration towards the frontiers of known goals to incentivize diversity." This sentence is not clear and needs more details. How is the distribution chosen exactly?
- Section 3.2 appears a bit repetitive and could be more concise. I don't think it is necessary here to contrast manual vs learned features of the goal space.
- Section 3.2 (P3): the last sentence of this paragraph reads as if there exists no approaches for VEAs in online settings. This should be toned down or backed up by a reference.
- Section 3.2: (last sentence): it is not clear how the history is used exactly to train the network. Which strategy is used to sample from the history of observations?
- Section 3.3: What is meant by "The CPPNs are used of the parameters \{theta}"? The details provided after this sentence are not clear and need more details.
- Section 4.2: Please provide more details what "very large" dataset means.
- Section 4.2: 'HGS algorithm' is not defined.
- Section 5: It seems unnecessary to explain what t-SNE does as a method.


In this paper, we study a new graph learning problem: learning to count subgraph isomorphisms. Although the learning based approach is inexact, we are able to generalize to count large patterns and data graphs in polynomial time compared to the exponential time of the original NP-complete problem. Different from other traditional graph learning problems such as node classification and link prediction, subgraph isomorphism counting requires more global inference to oversee the whole graph. To tackle this problem, we propose a dynamic intermedium attention memory network (DIAMNet) which augments different representation learning architectures and iteratively attends pattern and target data graphs to memorize different subgraph isomorphisms for the global counting. We develop both small graphs (≤ 1,024 subgraph isomorphisms in each) and large graphs (≤ 4,096 subgraph isomorphisms in each) sets to evaluate different models. Experimental results show that learning based subgraph isomorphism counting can help reduce the time complexity with acceptable accuracy. Our DIAMNet can further improve existing representation learning models for this more global problem.

This paper proposes a method called Dynamic Intermedium Attention Memory Network (DIAMNet) to learn the subgraph isomorphism counting for a given pattern graph P and target graph G. This requires global information unlike usual GNN cases such as node classification, link prediction, community detection. First, input graphs P and G are converted embedding vectors through sequence models (CNN, RNN, Transformer-XL) or graph models (RGCN), and fed into their DIAMNet that uses an external memory as an intermedium to attend both the pattern and the graph. The external memory is updated based on multi-head attention as in Transformer. The output of DIAMNet is passed to FC that outputs 'count' directly. The training is based on minimizing MSE loss as a regression problem. Extensive experimental evaluations report that DIAMNet showed superior performance over competing methods and baselines.
This paper targets subgraph isomorphism counting as a learning problem for the first time I guess, and the proposed method combined with both graph- and sequence-based encoding is technically interesting. However, there are still two major issues of 1) why counting? 2) the RMSE loss for regression on counts 3) baseline of 'Zero'.
1) the most unclear point is 'why counting?'. If I understand it, this method can be applied to subgraph isomorphism (NP-hard) or graph isomorphism (unknown complexity) as binary classification, and experimental evaluations can use the datasets used in evaluating VF2 or Naughty. It would be better to start this fundamental problem that would have many clear applications. Compared to subgraph isomorphism or graph isomorphism, the need for knowing accurate 'counts' of subgraph isomorphisms is unconvincing (given that we cannot explicitly obtains all subgraph matchings). Note that there is some existing research on GNNs targeted 'graph matching' and 'graph similarity'.
Also, the used datasets intentionally restrict the possible values for the number of subgraph isomorphisms, but the counts would be exponentially large if we consider practical (dense) graphs.
2) the method fits the model using (R)MSE loss, but minimizing log errors ((R)MSLE) would be better considering distributions of response values (counts) of the used datasets in Figure 6. Fitting the MSE loss is not good for such highly skewed cases, and for example, might focus only on the few instances having very large count values. Or, if such instances are very small, training ignores all such extreme instances. Either way would be questionable when we consider learning 'subgraph isomorphism counting' in general.
Also, the error of counts by MSE or MAE would be less informative and it would be unclear how much errors are tolerant in practical use cases of this method.
3) To interpret the RMSE and MAE values, Table 2 has the value for 'Zero'. This is for a constant predictor always returning zeros for any inputs. However, given that the loss is MSE, constant prediction values should be the average counts in the training data, not zero.


Contract bridge is a multi-player imperfect-information game where one partnership collaborate with each other to compete against the other partnership. The game consists of two phases: bidding and playing. While playing is relatively easy for modern software, bidding is challenging and requires agents to learn a communication protocol to reach the optimal contract jointly, with their own private information. The agents need to exchange information to their partners, and interfere opponents, through a sequence of actions. In this work, we train a strong agent to bid competitive bridge purely through selfplay, outperforming WBridge5, a championship-winning software. Furthermore, we show that explicitly modeling belief is not necessary in boosting the performance. To our knowledge, this is the first competitive bridge agent that is trained with no domain knowledge. It outperforms previous state-of-the-art that use human replays with 70x fewer number of parameters.

The authors propose a deep learning agent for automatic bidding in the bridge game. The agent is trained with a standard A3C reinforcement learning model with self-play, and the internal neural network only takes a rather succinct representation of the bidding history as the input. Experiment results demonstrate state-of-the-art performance with a simpler model. The authors discuss some findings with the proposed agent, such as the lack of need to explicitly model the belief and the possibility to self-train with different variants of opponents. Some visualization is also provided to understand how the trained agent behaves.
It is recommended to weak-reject the paper. There is an obvious contribution of using a simple model to reach state-of-the-art performance. The only concern is whether such a contribution is sufficient to warrant acceptance for a top conference. While setting a new milestone for the community is important (and having open-source code and data in the future will lead to a huge impact for the community), the current paper does not appear to have introduced sufficient "other contributions." In particular,
(1) The ablation study is very shallow. Section 4.2 shows four things: [a] training with an auxiliary belief task does not help the proposed agent; [b] simple representation is sufficiently good; [c] using a pool of opponents won't do better than using the most recent model as opponent; [d] frequency of updating the opponents does not matter much.
For [a], it is certainly interesting. But one could get to a deeper understanding by, for instance, zooming in to a ratio of 10^-3 or less, to see if there are sweet points of the choice of r. One could also get to a deeper understanding by checking how well BCELoss is doing, and how well the value loss is doing, to understand the trade-off. It is not even clear what the ranges of BCEloss and the value loss are, making it uncertain on whether r is properly chosen.
For [b], deeper study could be taken for analyzing "why", especially given that the more complicated encoding is significantly worse. The hand-waving explanation of "The potential reason why our encoding performs better is that the intrinsic order of bridge bidding is already kept by the action itself ..." at best suggests that the information between the baseline19 encoding and the author's are of the same information amount, but does not say why baseline19 is significantly worse. The authors leave a big question mark here that does not seem to match the grand title of "simple is better" of this paper.
For [c], deeper study could be taken for analyzing "what if some worse opponents are included" or "what if some targeted opponents are included." For instance, what if the authors include early-stage models to increase diversity? What is the trade-off between opponent capability/diversity and performance? What if using baseline16 as opponents---would the agent then beat baseline16 more easily? Note that baseline16 and the proposed agent basically learned a semi-natural bidding system. What if a precision system is set as a potential opponent---would the learned policy be very different?
For [d], getting a result that the update frequency leads to similar performance can hardly match the claim "We show that selfplay schedule and details are critical in learning imperfect information games." Also, the big error bar and the inconclusive trend in Table 1(B) does not answer the question on what frequency should best be used. Deeper discussions shall be included.
(2) Besides the ablation study, there is very little information about the design choices. For instance, why does the authors choose the specific RFC block? Would a full FC block do better or worse? Would a shallower network do better or worse? Actually, the title "simple is better" may be misleading, as the authors did not study even simpler choices. For RL algorithm, why A3C but not other algorithms?
(3) The authors' finding that the agent is conservative and does not bid high appears interesting. Nevertheless, it perhaps suggests that the authors have not compared with strong bidding systems (like precision) that aimed for bidding high to win more. That is, the claimed advantages are over semi-natural bidding systems. More study on whether the agent should bid high could greatly enrich this paper.
===
I've read the rebuttal. I thank the authors for the discussion on some possibilities in deepening the paper. I suggest the authors to avoid some hand-waving claims such as "We argue it is easier to learn the connections between them." In general, some of the claims (such as "simple is better" in the title) lacks sufficient scientific support, and I believe that this paper will have more impacts if it focuses more on supporting the claims with science evidence, not with human descriptions.