Despite their success, generative adversarial networks (GANs) cannot easily generate structured objects like molecules or game maps. The issue is that such objects must satisfy structural requirements (e.g., molecules must be chemically valid, game maps must guarantee reachability of the end goal) that are difficult to capture with examples alone. As a remedy, we propose constrained adversarial networks (CANs), which embed the constraints into the model during training by penalizing the generator whenever it outputs invalid structures. As in unconstrained GANs, new objects can be sampled straightforwardly from the generator, but in addition they satisfy the constraints with high probability. Our approach handles arbitrary logical constraints and leverages knowledge compilation techniques to efficiently evaluate the expected disagreement between the model and the constraints. This setup is further extended to hybrid logical-neural constraints for capturing complex requirements like graph reachability. An extensive empirical analysis on constrained images, molecules, and video game levels shows that CANs efficiently generate valid structures that are both high-quality and novel.

In this paper the authors present a Generative Adversarial Neural Networks with Xu et al.’s semantic loss applied to the generator. They call this GAN a Constrained Adversarial Network or (CAN) and identify it as a new class of GAN. The authors present three different problem domains for their experiments focused on the generation of constrained images, chunks of Super Mario Bros.-style levels, and molecules. For each domain they include particular constraints for the semantic loss, which biases the generator towards creating valid content according to these constraints.
The paper at present has a number of issues holding it back. First, I am not convinced by the author’s claims that the application of an existing loss function to the generator is sufficient to identify a new class of GAN. Second, there is a lack of technical detail in the experiments necessary to replicate them. Third, there is a lack of discussion of the experimental results to place them in context for readers. Finally following from the earlier points, there seems to be a lack of technical contributions in the paper.
I certainly agree with the authors about the inability of GANs to learn structural constraints with insufficient training data, as this has been demonstrated in many examples of prior work. I also agree that particular problem domains, as identified by the authors, have stronger structural requirements. However, it is unclear to me why in these instances one would use GANs and not some alternative approach such as constraint-based solvers. Or even if one wanted to employ GANs, what the benefit of adapting the constraints into a loss function is compared to say constraining generated output in a post-hoc process.
The descriptions of the two of the three experiments do not include any discussion of the GAN architectures or hyperparameters. While this is not strictly necessary in the paper text some discussion in an appendix or a citation to a prior application of the architecture(s) would be appropriate. Without this, it is impossible for future researchers to replicate these results. Further, it is difficult for readers to place the results in context. For individual experiments, such as the Super Mario Bros. experiment, it is unclear why certain choices were made. For example, why train a GAN on just level 1-3 or 3-3, and not train a single model on multiple levels as is common in the field of procedural content generation via machine learning.
There is a lack of discussion in the paper on the results of each experiment. For example, the output of the GANs for all the experiments seems quite low, and the differences in terms of the results between the GAN and the CAN across the experiments do not seem to be substantial. Some discussion to put this into context for readers would be helpful.
As far as I can understand the primary technical contributions of the paper are: (1) the application of Xu et al.’s semantic loss to GANS, (2) the constraints developed for the three experiments, and (3) the results of the three experiments. I am unconvinced of the utility of these contributions to a general machine learning audience.
---
Updated my review as the authors included extra detail regarding the experiments in a new draft, which helped with the reproducibility issue. However, I am still unconvinced in the contributions of the paper outside of what I previously listed. While I am also unfamiliar with any prior example demonstrating that GANs produce invalid structure, this is not a surprising result. Especially as validity can be defined in an arbitrary, domain-specific manner.


Many applications require quickly and repeatedly solving a certain type of optimization problem, each time with new (but similar) data. However, state-of-theart general-purpose optimization methods may converge too slowly for real-time use. This shortcoming is addressed by learning to optimize (L2O) schemes, which construct neural networks from parameterized forms of the update operations of general-purpose methods. Inferences by each network form solution estimates, and networks are trained to optimize these estimates for a particular distribution of data. This results in task-specific algorithms (e.g., LISTA, ALISTA, and D-LADMM) that can converge order(s) of magnitude faster than general-purpose counterparts. We provide the first general L2O convergence theory by wrapping all L2O schemes for convex optimization within a single framework. Existing L2O schemes form special cases, and we give a practical guide for applying our L2O framework to other problems. Using safeguarding, our theory proves, as the number of network layers increases, the distance between inferences and the solution set goes to zero, i.e., each cluster point is a solution. Our numerical examples demonstrate the efficacy of our approach for both existing and new L2O methods.

This paper presents a unified framework for parametrizing provably convergent algorithms and learning the parameters for a training dataset of problem instances of interest. The learned algorithm can then be used on unseen problems. One key idea to this algorithm is that it is safeguarded, meaning it will perform some standard, non-learned iterations, if the predicted iterate is not good enough under some condition.
There are three main features of the proposed approach:
1- It unifies various previous approaches such as LISTA, ADMM, non-negative least squares, etc. By defining some operators and safeguarding rules, the same learning approach can be leveraged for these different optimization problems.
2- It is shown that the learned algorithms are provably convergent under some mild assumptions.
3- Empirically, it is shown that the learned algorithms converge faster than the non-learned counterpart on sparse coding, ADMM and non-negative least squares; they use safeguarding sparingly, particularly when used to solve test instances from the same distribution as the training instances.
Additionally, the paper is very well-written. I did not verify the proofs in detail but they seemed OK at a high-level; however, I am not an expert in convex optimization so I hope other reviewers will be able to comment on this aspect.
I do have some deep concerns about the evaluation metrics used to report the results that I will discuss next; these are the main reason for my current score, but I am willing to adjust it if the authors address them convincingly. I also have some comments about related work.
Experimental evaluation:
- The error metric (15) is not suitable for evaluating the performance of an optimization algorithm. You should compute the expectation of the relative error, i.e. E_{d~D} [(f_d(x)-f_d^*) / f_d^*]. This is similar to the average approximation ratio used in the learning to optimize papers for discrete problems (see refs. below). (15) is just the ratio of the expected absolute error to the expected optimal value; I don't think that is equivalent to what I suggested.
- The relative error values are massive in some cases, e.g. Fig. 3. What's going on there? Are all methods performing that horribly? Am I misinterpreting the metric?
- Why do the plots for the seen distribution extend over thousands of iterations but only for tens of iterations for the unseen distribution?
- Please use the same scale for the y-axes in Figs. 1-3.
Methodology:
- Your method requires learning per-iteration parameters. The other L2O methods for gradient descent (see refs. below) use shared parameters instead. This allows them to run for many iterations, possibly beyond what they were trained for. Your method does not allow for that. On the other hand, such models are recurrent and thus possibly more difficult to train than your unrolled feedforward model. Is the fixed number of iterations a limitation of your method? Please discuss this.
Related work:
- Learning for gradient descent: I am surprised these papers are not mentioned although they are quite relevant. They are rather recurrent networks with shared parameters across iterations, but you should also compare against them both conceptually and experimentally:
"Learning to optimize." arXiv preprint arXiv:1606.01885 (2016).
"Learning to learn by gradient descent by gradient descent." Advances in neural information processing systems. 2016.
- Learning to optimize in the discrete setting: there is lots of recent work on this that you should at least point to in passing, e.g.:
"Learning combinatorial optimization algorithms over graphs." Advances in Neural Information Processing Systems. 2017.
"Combinatorial optimization with graph convolutional networks and guided tree search." Advances in Neural Information Processing Systems. 2018.
(Survey) "Machine Learning for Combinatorial Optimization: a Methodological Tour d'Horizon." arXiv preprint arXiv:1811.06128 (2018).
- Theory for learning to optimize: Since you have a theoretical basis for your framework, you should discuss connections to other recent frameworks such as the one below by Balcan et al. It is geared towards the discrete setting and sample complexity rather than convergence, but you should nevertheless discuss it.
Balcan, Maria-Florina, et al. "How much data is sufficient to learn high-performing algorithms?." arXiv preprint arXiv:1908.02894 (2019).
Clarification questions:
- "The choice of parameter ζ k in Line 3 may be any value that results in a well-deﬁned operator T L2O": what is "well-defined" here? that T_{L20} is averaged?
Minor:
- Page 3: "A classic theorem states sequences" -> "A classic theorem states that sequences"
- Appendix proofs: please organize into sections and restate the statements before the proofs.


Recommendation is a prevalent application of machine learning that affects many users; therefore, it is crucial for recommender models to be accurate and interpretable. In this work, we propose a method to both interpret and augment the predictions of black-box recommender systems. In particular, we propose to extract feature interaction interpretations from a source recommender model and explicitly encode these interactions in a target recommender model, where both source and target models are black-boxes. By not assuming the structure of the recommender system, our approach can be used in general settings. In our experiments, we focus on a prominent use of machine learning recommendation: ad-click prediction. We found that our interaction interpretations are both informative and predictive, i.e., significantly outperforming existing recommender models. What's more, the same approach to interpret interactions can provide new insights into domains even beyond recommendation.

This paper is focused on identifying/discovering feature interactions in blackbox models (with a focus on recommendations). Specifically the technique works by first corrupting datapoints and then using this "local dataset" to find interacting features via lasso-regularized multi-level perceptron approach (NID from Tsang et al). Using an expanded feature space and repeated calls to the above steps this is then used to find the most commonly occurring patterns.
The empirical section is quite interesting, with some nice mix of qualitative and quantitative results. The quantitative results in particular are quite intriguing -- across recommendation and non-recommendation tasks.
Overall the paper makes for an interesting read. On the whole I lean slightly positive -- largely due to the empirical section and the quantitative gains observed by adding the feature interactions as features to the model.
That said I had quite a few concerns about the work:
- In general the exposition was quite lacking in the methodological section. I had to re-read the paper a few times to be able to make out to fully understand the underlying methodology. I would make the overall pipeline very clear and explain out the MADEX pipeline very clearly. The feature expansion section (4.3) in particular was not very clear -- as to how it fit in with the rest of the pipeline and something that could do with more work.  Nomenclature like source and target model are used somewhat arbitrarily and again can be clarified.
- Another concern I had with the approach is how this would work if the underlying features itself were not known. Say for example you had a text understanding model -- the tokenization and vocabulary/OOV etc .. may all be components of the blackbox. Likewise for vision models, what features are being used may vary.
By assuming (effectively perfect) information about the features used, this is no longer really a "blackbox" model. I would have wanted to see some deeper discussion on this topic.
I also had some concerns about the scalability of dense features and their bucketization in this approach. I'm not entirely convinced the method would work as well and efficiently in such feature spaces. I would have wanted to see some more empirical evidence and understanding on this topic.
If possible it would have also been great to understand higher-order interactions and the ability of the model to find them empirically.
- One thing I wasn't entirely certain of was the amount of novelty in the work since it leverages existing works like NID (Tsang et al) and LIME (Ribeiro et al) for some of the key aspects of the method. It would be nice to see some discussion on this front as well.
- I also would have liked to see significance testing being performed in general across the experiments.
- Another smaller concern stemmed from a lack of mention of the stopping criterion of the model. In particular I'm wondering how models were stopped further training / best checkpoint was picked. This would be good to elaborate on to make sure that the models were all compared fairly from a computational perspective.
- I also had some concerns with the scalability of the approach in large features spaces but am willing to give the authors the benefit of doubt on this one based on the 3 hour number they quoted from their experiments (which indicates something that is not prohibitively slow)


We present a systematic framework for the Nesterov's accelerated gradient flows in the spaces of probabilities embedded with information metrics. Here two metrics are considered, including both the Fisher-Rao metric and the Wasserstein-2 metric. For the Wasserstein-2 metric case, we prove the convergence properties of the accelerated gradient flows, and introduce their formulations in Gaussian families. Furthermore, we propose a practical discrete-time algorithm in particle implementations with an adaptive restart technique. We formulate a novel bandwidth selection method, which learns the Wasserstein-2 gradient direction from Brownian-motion samples. Experimental results including Bayesian inference show the strength of the current method compared with the state-of-the-art.

I acknowledge reading the rebuttal of the authors. Thank you for your clarifications and explanation. My point was this paper would make a good submission to ICLR if it was better motivated presented and explained to a wider audience. Unfortunately in its current form it can only reach a limited audience.
####
Summary of the paper:
The paper proposes accelerated information flows under Wasserstein or Fisher rao metric . i.e a method for solving minimization of probability functional , where the probability space is either endowed with the Wasserstein or the Fisher distance.
Gradient descent methods in euclidian spaces can be accelerated using a type of momentum , this paper extends this to gradient flows, using similar formalism of Hamiltonian that appeals to the dynamic of a the particles and the velocity (or momentum, $H(x,p)=\frac{1}{2}||p||^2+ \mathcal{E}(x)$).  Writing down the lagrangian one obtains two co-evolving PDE one of the dynamic of the density and one for the potential . The PDEs are specified for both Fisher Rao, and Wasserstein distance. The hamiltonian for example for the Wasserstein distance $H(\rho_t,\Phi_t)=\frac{1}{2}\int||\nabla_x \Phi_t||^2+ \mathcal{E}(\rho_t)$.  and PDEs amounts a continuity equation for the density  evolving with drift $\nabla \Phi_t$, the evolution of the momentum $\Phi_t$ is also given by a PDE.
Proposition 2 of the paper gives the particles differential equation corresponding to the system of PDEs. For the energy being the KL divergence an explicit expression is given , this expression remains difficult in practice since it needs the knowledge of the density $\rho_t$. Authors propose in the application section to use gaussian approximation , or using a kernel density estimators. The Bandwidth of the kernel is then choosen using a heuristic proposed in the paper.
The paper then focuses on deriving expression for flows when the densities are centered gaussians, and this amounts to an ODE on the covariance , the ODE is discretized in Appendix D.2 to lead to  computational method. Then convergence of the flow is analyzed for the wasserstein accelerated flows, under "\beta- convextiy" in the wasserstein sense of the functional.
Some experiments of the particle based method are shown on synthetic experiments and in bayesian logistic regression.
Review:
Contribution/ Clarity:
The main contribution of the paper is in deriving the accelerated gradient flow for the wasserstein distance this was also addressed in a recent paper [Taghevia and Mettha 2019].
The technical contribution is interesting but given that this field of flows in probability space is still not very well spread in the ML community, I wonder if ICLR is the best fit for this type of work.  I support good theoretical work, but I think the authors could have done a better job in exposing the ideas how they extend form euclidean space, to manifolds, to probability spaces gradually. Simple derivations of euclidean space Hamiltonian will help the reader that is not exposed to such literature. I think the paper will benefit from a less technical writing in introducing the ideas coming from euclidean space and in conveying the intuitions.
Comments:
- In the proof of Proposition 2 you give the expression of evolution of $dV_t$ by conservation of the momentum. Could you please elaborate more how you obtain this expression, and where you proved the conservation of momentum?
- In term of damping if ones uses the Wasserstein Fisher Rao flows  , one obtain also accelerartion , maybe you can comment on that ? since you analyze both flows , would be interesting to discuss the relation to Global convergence of neuron birth-death dynamics, that shows that an acceleration is obtained via WFR flows, since it will introduce a damping as well.
- since MCMC and BM method lead to similar result what is the advantage of the wasserstein accelerated flow? one could also implement also an accelerated langevin dynamic


Obtaining high-quality uncertainty estimates is essential for many applications of deep neural networks. In this paper, we theoretically justify a scheme for estimating uncertainties, based on sampling from a prior distribution. Crucially, the uncertainty estimates are shown to be conservative in the sense that they never underestimate a posterior uncertainty obtained by a hypothetical Bayesian algorithm. We also show concentration, implying that the uncertainty estimates converge to zero as we get more data. Uncertainty estimates obtained from random priors can be adapted to any deep network architecture and trained using standard supervised learning pipelines. We provide experimental evaluation of random priors on calibration and out-of-distribution detection on typical computer vision tasks, demonstrating that they outperform deep ensembles in practice.

This work introduces a simple technique to obtain uncertainty estimates for deep neural networks. This is achieved by having a set of random networks (i.e. neural networks where their parameters are randomly initialized) and then computing an uncertainty value based on the difference in the predictions between those random networks and networks that are trained to mimic them on a finite collection of points. The authors further show that this method results into uncertainties that are conservative, meaning that they are higher than the uncertainty of a hypothetical posterior, and concentrate, i.e. they converge towards zero when we get more and more data. The authors further draw connections to ensemble methods and discuss how such a method can be effectively realized in practice. They then evaluate their approach on an out-of-distribution detection task, they measure the calibration of their uncertainty estimates and finally perform a small ablation study for their concentration result.
This work is interesting as it seems to provide a simple way to obtain reasonable uncertainty estimates. For this reason it can potentially serve as a strong baseline for this field. The theoretical considerations also help in providing some guarantees about such an approach. Having said that, in my opinion the writing could use some more work in order to make things more clear as some critical experimental details and baselines are missing and thus do not make the method as convincing. Furthermore, I also believe that some clarifications on the theoretical aspects of this work, will help in boosting its quality. More specifically:
- How exactly do you apply your method on the classification scenario? Do you select an arbitrary hidden layer of the classification model for the prior and predictor network architectures or the output logits / softmax probabilities?  In appendix A you mention the architecture but not precisely how it is employed. I believe this can be an important piece of information in order to decipher the importance of e.g. the output dimensionality on the uncertainty quality, as higher dimensional outputs might be harder to approximate thus could induce a larger squared error and hence uncertainty.
- What is the average training error of the predictor networks for the out-of-distribution task and subsampling ablation task, i.e. how far away from concentration were the priors?
- An effect that I found weird is the following: what happens for the out-of-distribution examples when the predictor networks can perfectly predict the prior network outputs? Wouldn’t that then imply that the uncertainty would be zero for any input (even an out-of-distribution one), as the prior network and predictor network always agree? One could imagine that for e.g. simple priors and with sufficiently dense sampling of the domain of the function this can happen in practice.
- For the conservatism you show that your uncertainty estimate is higher, on average, than the posterior variance when you sample points from the model itself. In a sense this guarantee translates to the actual data when the prior is “correct”. How do those conservatism guarantees translate to the case when there is model misspecification, i.e. when the prior is not correct? Perhaps a small toy example would be informative.
- For the predictor networks as described in figure 2; do you train both the green and red parts of the network or only the red parts and keep the green part fixed to the values you used for the prior f? (This helps in understanding how easy / difficult is the task of the predictor network).
- What is the accuracy on the actual in-distribution prediction task for the RP and baselines? What did “B” correspond to for the dropout networks? Was it the number of dropout samples you averaged over to get the final predictive?
- How sensitive are the results on the actual initialization strategy of the prior network? It would be good to see e.g. some form of performance / init variance curve in order to decipher the sensitivity.
Other comments
- It is worth pointing out that [1] showed that Monte-Carlo dropout performs approximate MAP inference, which seems more plausible than the approximate Bayesian inference perspective of [2].
- In the introduction you argue that Bayesian neural networks rely on procedures different from standard supervised learning and thus most ML pipelines are not optimized for them in practice. Could you elaborate a bit about this statement? Variationally trained BNNs with e.g. the reparametrization trick [3, 4] are straightforward since you can just use backpropagation to update their (variational) parameters.
- What is the x-axis for Figure 3 for the baselines? (I take it that for RP it is the \hat{sigma}^2(x)).
- I believe that a comparison against a simple variationally trained BNN would make the results more convincing.
Misc
- Second page, “Figure 1, top two plota” -> “Figure 1, top two plots”
- Third page, “[…] introduced in equation 2 denotes the posterior covariance [….]” -> “[…] introduced in equation 2 denotes the posterior variance […]“
- Fifth page, “[…] this makes it is reasonable for W large enough […]” -> “[…] this makes it reasonable for W large enough […]”
- Sixth page, “Corollary 1 and proposition 2”; where is corollary 1? Do you mean Proposition 1?
- Seventh page, “[…] inspired by, an builds on, […]” -> “inspired by, and builds on, […]”
- Ninth page “montonicity” -> “monotonicity”
Overall, I tend to accept this work, although, depending on the author rebuttal and other discussions, I am willing to change my rating accordingly.
[1] Eric Nalisnick, José Miguel Hernández-Lobato, Padhraic Smyth, Dropout as a Structured Shrinkage Prior, 2019
[2] Yarin Gal, Zoubin Ghahramani, Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning, 2016
[3] Diederik P. Kingma, Max Welling, Auto-Encoding Variational Bayes, 2014
[4] Danilo Jimenez Rezende, Shakir Mohamed, Daan Wierstra, Stochastic Backpropagation and Approximate Inference in Deep Generative Models, 2014


Continual learning is a critical ability of continually acquiring and transferring knowledge without catastrophically forgetting previously learned knowledge. However, enabling continual learning for AI remains a long-standing challenge. In this work, we propose a novel method, Prototype Recalls, that efficiently embeds and recalls previously learnt knowledge to tackle catastrophic forgetting issue. In particular, we consider continual learning in classification tasks. For each classification task, our method learns a metric space containing a set of prototypes where embedding of the samples from the same class cluster around prototypes and class-representative prototypes are separated apart. To alleviate catastrophic forgetting, our method preserves the embedding function from the samples to the previous metric space, through our proposed prototype recalls from previous tasks. Specifically, the recalling process is implemented by replaying a small number of samples from previous tasks and correspondingly matching their embedding to their nearest class-representative prototypes. Compared with recent continual learning methods, our contributions are fourfold: first, our method achieves the best memory retention capability while adapting quickly to new tasks. Second, our method uses metric learning for classification, and does not require adding in new neurons given new object classes. Third, our method is more memory efficient since only class-representative prototypes need to be recalled. Fourth, our method suggests a promising solution for few-shot continual learning. Without tampering with the performance on initial tasks, our method learns novel concepts given a few training examples of each class in new tasks.

The proposed method addresses continual learning, by learning a mapping from the input space to an embedding space, and employing a loss that encourages clustering the embeddings by class (and task?) around some centroids called prototypes. Catastrophic forgetting is mitigated by adding a penalty term that is proportional to the distance of the embeddings under the current network of some samples from the past tasks, and the centroids previously associated to each of them.
The idea might have some merit, but the paper in its current state fails to convey it clearly and is too vague on the experimental settings for the results to be considered in any way. The notation is at times unnecessary heavy (see specific comments below), and many details on the method and experimental setup are missing to the point that it’s difficult to judge it and it would be impossible to reproduce it. In addition, it’s unclear whether the baselines have been properly tuned since the reported performance is incompatible with that of the original paper.
The submitted paper is clearly not ready for publication. I can consider modifying my vote if the paper undergoes a major review, fixing the notation, explaining the method and the experimental settings extensively and clearly, with proper comparison in terms of capacity/performance tradeoff w.r.t. the state of the art.
Detailed feedback:
1) Notation
- I believe the exposition would be more fluent and easier to follow if you described the method under the assumption that all tasks have the same number of classes and a shared dataset, and mentioning that the method can be easily extended to work in the more general setting. Under this assumption, e.g., c_{kT} would simplify to c_{k}.
- When you refer to a generic task, please use a lower case letter (e.g., t) instead of T (e.g., first paragraph of page 3, Eqn. 1, Eqn. 2, ..). Use capital letters for constants only (e.g., use T for the total number of tasks). When you need to refer to two tasks that are in a particular relative ordering. For instance, Eqn. 1 should be the argmin of “\sum_{s=1}^{t} [...], with s < t”. Similarly, Sec 2.2 should refer to tasks s and t, rather than t and T (unless T is specifically referring to the last task alone, which doesn’t seem the case here as from what I gather the same should hold for all tasks, i.e., for a specific task t and any previous task s, with s < t.).
- If I am not mistaken, L_T is defined just before Eqn. 1 and only used there. It is not very informative and it’s based on some undefined loss L. I would suggest to drop it entirely, and simplify Eqn. 1 accordingly.
- Please avoid subscript with commas as much as possible, as they are difficult to parse. E.g., L_{classi, Dt} can become L_{classi_t} (or even L_{c_t} or L_{c}^{(t)}), remaining equally informative. All other “D_t” subscripts can be similarly replaced by “t” throughout the text. Generally speaking, I would recommend the ^{(t)} notation to denote task t (which you already adopted when defining \tilde{D}_t
- I believe Eqn. 1 can be simplified by merging the two summations into a single expression. Also, I think it would be clearer to define the argmin to be over “f \in \mathcal{F}”, since it is a search over the space of all possible functions. Finally, I also find a bit confusing the use of f and f_t to denote an evolving (currently evaluated) function and a fixed (best function from previous search) one respectively, but I am not sure I have a suggestion to improve this notation.
- Please avoid unnecessary repetitions. In a number of cases a big part of a formula is defined inline in the text just before the equation. This doesn’t add to the conversation and makes it unnecessarily hard to parse the text (e.g., inline formulas just before Eqns. 4 and 6. Also prefer a textual description instead of the variable when introducing a definition (e.g., before Eqn. 7, “we define the distance as” or “we define the distance between the functions f and f_t embeddings as”)
- Please fix all citations according to the ICLR style guidelines: When the authors or the publication are included in the sentence, the citation should not be in parenthesis (as in ``See \citet{Hinton06} for more information.''). Otherwise, the citation should be in parenthesis (as in ``Deep learning shows promise to make progress towards AI~\citep{Bengio+chapter2007}.''). For instance, all citations in Sec 2.2 should use \citep.
2) To my best understanding classification in the embedding space is performed with some variation of k-means, with k being the number of “prototypes” for the task at hand (i.e., the number of classes for the specific task). If this is correct, referring to k-means in the paper and highlighting the similarities and differences to it, would have been a more effective way to describe your method.
3) The authors claim not to use a final classification, as opposed to competing methods, hence not being subject to memory usage increases when the number of classes increase (sec 2.1). Surprisingly though, they then employ a softmax over the prototypes (Eqn. 3) which to the best of my knowledge would require an incoming layer whose dimensionality would need to grow with the increase of the number of prototypes, (i.e., classes). Please update the text to either correct the claim or explain exactly how your method can drive the softmax in the case of an increasing number of classes without increasing the dimensionality of the layer before the softmax.
4) Similarly, the authors claim that “Unlike parameter regularization methods or iCARL or FSR, our approach further reduces the memory storage by replacing logits of each data or network parameters with one prototype of each class in the episodic memory”. This is not entirely true, as they also need to store a (unspecified) number of samples from all tasks to prevent forgetting. It is unclear that replacing a snapshot of the network parameters is better (in terms of memory consumption) than storing some data from each task, and a proper discussion on this crucial point of the paper is missing.
5) Apart from the losses, all the details on the architecture (type and number of layers, activation functions, ..) and on the hyperparameters are missing. The paper only mentions that the same feedforward architecture as EWC is used, with similar memory usage, but also mentions that unlike all other methods (including EWC I assume), theirs doesn’t need to vary the architecture if the number of tasks is increased. It is then completely unclear to me how this is possible if the architecture is the same. If it’s not, which parts of the network are shared and which are task specific. I assume most of the network to be shared among tasks and only the last layer (that drives the softmax that determines the probabilities of each class (prototype) for the current task) to be task-specific. This details are crucial to the understanding of the method and to be able to reproduce the results.
6) Crucially, the reported Permuted MNIST performance on EWC is much worse than that on the original paper. This suggests that the baselines have not been properly tuned or there are issues in the implementation.
7) Unless I am mistaken, in Sec 3.3 there seems to be a flaw in the calculation of the number of examples that can be stored by the proposed method in order to use approximately the same memory as the baselines. Indeed, the memory allocation for weight regularization is independent of the number of classes and so should be the equivalent capacity allocated for samples retention, i.e., in order to be comparable with EWC (and similar methods), the proposed model should be allowed to store 530 examples in total, rather than per class.
8) I suggest to drop the definitions of the obvious functions, such as the softmax and the NLL (that is even defined twice!), and use that space to improve the description of the model and algorithm instead.
9) Can you clarify what you mean by “The pairwise distance of one embedding and one prototype within the same class should be smaller than the intra-class ones.”? Should it be inter-class?
10) In my opinion the narrative would be clearer backwards: rather than first defining the loss as a sum of three sub-losses, I’d suggest to define the sub-losses first and finally aggregate them. In its current state, Eqn. 1 is not informative until \delta_{D_t} is defined, a full page later.
11) Eqns. 3 and 5 are not “distance distributions” but rather “class probabilities distributions”, if I understood correctly. Can you confirm and amend the text?
12) Eqn. 5 is more confusing than informative, I recommend to remove it and replace it with a better textual description. There is no need to redefine c_{kt}, as it is already defined in Eqn. 2 exactly in the same way. Secondly, there is no need to define the softmax over the distances a second time, as it’s already defined in Eqn. 3 and only the arguments change.
13) Re Sec 2.3, to my best understanding there is no “emphasis on reviewing earlier tasks”, as the number of rehearsal samples is the same for all tasks. In other words, \tilde{D}_t is sampled uniformly across tasks I believe. Is this right?
14) Sec 3.1, “typical continual learning schemes assume that a large amount of training data over all tasks is always available for fine-tuning”. This is not true, and it’s actually usually not the case  (see e.g., generative replay approaches, or those that rely on pruning or other forms of architecture modification.)
15) The introduction fails to mention the category of approaches that try to combat forgetting by being smart about the architecture, e.g., Progressive nets, Progress & compress, etc, .. Please amend this.
Minor:
- Add more space after Figure 1’s caption, so that it’s clear where the caption ends and the text begins.
- Eqn. 2, no need to define yiT since it’s not used in the equation.
- The sentence at the end of Sec 2.1 (“In practice … estimating the distance distribution” ..) should go after Eqn. 2. Similarly, the sentence “Our primary choice of  the distance function” would be better placed right after the definition of the distance function IMO.
- The MNIST citation is wrong. Please correct it: LeCun, Yann and Cortes, Corinna. "MNIST handwritten digit database." (1998)
- The introduction is a bit long and a “related work” section is missing. You might want to split the introduction into two chapters.
- Do not use D to denote the datasets and the dimensionality of the datasets.
- The sentence after Eqn. 1 “learning f_T requires minimizing both terms ..” is incorrect, as it goes against Eqn. 1 in which there are 3 terms rather than 2.
- End of page 3: “without loss of generality”...w.r.t. what? The phrase doesn’t seem to fit the context, consider changing it.
- End of page 7, the detailed description of Figure 3 would be better placed in the caption of the figure itself.
Typos:
- Page 2: state-of-the-arts -> state-of-the-art
- “There have been some attempts [citations] selecting”, “of” is missing
- End of page 4: which is pre-computed -> are
- End of page 5: public -> publicly


We incorporate Tensor-Product Representations within the Transformer in order to better support the explicit representation of relation structure. Our TensorProduct Transformer (TP-Transformer) sets a new state of the art on the recentlyintroduced Mathematics Dataset containing 56 categories of free-form math wordproblems. The essential component of the model is a novel attention mechanism, called TP-Attention, which explicitly encodes the relations between each Transformer cell and the other cells from which values have been retrieved by attention. TP-Attention goes beyond linear combination of retrieved values, strengthening representation-building and resolving ambiguities introduced by multiple layers of standard attention. The TP-Transformer's attention maps give better insights into how it is capable of solving the Mathematics Dataset's challenging problems. Pretrained models and code will be made available after publication.

Motivated by the fact that the attention mechanism in transformers is symmetric which might not be able to disambiguate different orders, this work proposes to use a subject vector (in addition to query, key states) for each attention head, and multiply it elementwise with the context vector for each head before merging the heads. Experiments on a mathematics dataset shows superior performance compared to the normal transformer. Qualitatively, the proposed model exhibits attentions that are more interpretable, and clustering by the subject vector gives some insights into how the model solved this problem.
Pros:
1. This work shows better performance than baseline transformer.
2. The clustering of the subject vectors gives some insights into model's behavior .
Cons:
1. In terms of experiments, the proposed approach adds a few million parameters to normal transformer (table 1), but in terms of interpolation it only improves 3% (extrapolation improves 0.5%) at 700k steps. The comparison would be fairer if the normal transformer can be given more parameters.
2. In terms of experiments, this approach is only evaluated on the mathematics dataset, but the argument for relational encoding is pretty general. It would be nice if experiments on other tasks are shown in addition to the math dataset.
3. In terms of motivation, the claim that there're ambiguities introduced by multiple layers of  regular attention needs to be supported by evidence. I think (which authors also pointed out) the feedforward network and non-linearties can disambiguate as well.
4. In terms of interpretablity, there's claim that the learned attention maps more interpretable than transformer. Can there be more quantitative measures? It appears to me that both are hard to interpret.
While this work shows superior performance on the mathematics dataset, I have a few concerns about the generalizability of this proposed architectural change to other problems, as well as the fairness of comparison to baseline. Therefore, I am inclined to reject this paper.
----updates after reading rebuttal----
Thanks for adding the new NMT experiment in Appendix A3. My concern is that the proposed TP-Transformer is not very effective on NMT. Therefore, I'm keeping my score.


Semi-supervised learning lately has shown much promise in improving deep learning models when labeled data is scarce. Common among recent approaches is the use of consistency training on a large amount of unlabeled data to constrain model predictions to be invariant to input noise. In this work, we present a new perspective on how to effectively noise unlabeled examples and argue that the quality of noising, specifically those produced by advanced data augmentation methods, plays a crucial role in semi-supervised learning. By substituting simple noising operations with advanced data augmentation methods, our method brings substantial improvements across six language and three vision tasks under the same consistency training framework. On the IMDb text classification dataset, with only 20 labeled examples, our method achieves an error rate of 4.20, outperforming the state-of-the-art model trained on 25,000 labeled examples. On a standard semi-supervised learning benchmark, CIFAR-10, our method outperforms all previous approaches and achieves an error rate of 2.7% with only 4,000 examples, nearly matching the performance of models trained on 50,000 labeled examples. Our method also combines well with transfer learning, e.g., when finetuning from BERT, and yields improvements in high-data regime, such as ImageNet, whether when there is only 10% labeled data or when a full labeled set with 1.3M extra unlabeled examples is used.

The paper proposes to substitute simple noising operations with many data augmentation methods in consistency-based semi-supervised learning. The main idea is the same as previous work: constrain the model predictions of unlabeled examples to be invariant to different noise. The proposed UDA is evaluated on a wide range of language and vision tasks.
Overall, the paper is well-written and clear. The most impressive point of this paper is its strong empirical results. However, it looks not surprising to me that more data augmentations found in supervised learning are also effective in semi-supervised learning. The paper fails to provide any theoretical insights but a thorough empirical evaluation.
One of my concerns is that the hyperparameters on vision tasks follow those of AutoAugment, which is carefully tuned on supervised tasks. Apparently, their hyperparameters are based on the whole labeled training dataset. In this case, the adopted hyperparameters include sort of information of the whole labeled dataset. Is it fair?
Another concern is how to control the strength of augmentations. For example, for digit images like SVHN, a "6" rotates by 180 degree is "9", whose prediction should change correspondingly. In this case, the assumption of invariance does not hold when the augmentation is too strong.
I'm willing to increase my score if the authors address my concerns.


Generative neural models have improved dramatically recently. With this progress comes the risk that such models will be used to attack systems that rely on sensor data for authentication and anomaly detection. Many such learning systems are installed worldwide, protecting critical infrastructure or private data against malfunction and cyber attacks. We formulate the scenario of such an authentication system facing generative impersonation attacks, characterize it from a theoretical perspective and explore its practical implications. In particular, we ask fundamental theoretical questions in learning, statistics and information theory: How hard is it to detect a "fake reality"? How much data does the attacker need to collect before it can reliably generate nominally-looking artificial data? Are there optimal strategies for the attacker or the authenticator? We cast the problem as a maximin game, characterize the optimal strategy for both attacker and authenticator in the general case, and provide the optimal strategies in closed form for the case of Gaussian source distributions. Our analysis reveals the structure of the optimal attack and the relative importance of data collection for both authenticator and attacker. Based on these insights we design practical learning approaches, and show that they result in models that are more robust to various attacks on real-world data. Code will be made publicly available upon publication.

This paper proposes a new threat model for generative impersonation attacks: The attacker has access to several leaked images of a person; the authenticator knows several registration images per person and decides a person's identify by comparing some newly-sampled images from that person with corresponding registration images. The authors formulate this threat model as a minimax game and analyzed its Nash equilibrium. In the simplified case that observations are multivariate Gaussian, the authors are able to characterize the optimal strategies of the attacker and authenticator explicitly, which gives a nice intuition on how the theoretical optimum changes with respect to data dimension, number of leaked images, etc. Additionally, the authors implemented this attack (named Gan-in-the-middle attack) with an objective similar to GANs, empirically verified the theoretical results, and demonstrated the success of their approach on VoxCeleb2 and
As far as I know, this formulation of generative impersonation attacks is novel. The threat model nicely captures the most important aspects of impersonation attacks and is relatively realistic.
The theoretical analysis is insightful. I especially like that the authors can prove no defense is possible when n <= m, which nicely matches the intuition. The results on Gaussian case not only provide intuition, but also provide motivation for the design of attacker and defender architectures in GIM attacks.
The experiments are well-designed. The model architectures are well-motivated from Theorem 4.2. It is great to see that results of toy experiments match the theoretical analysis in Figure 1(a). The GIM attack on the Voxceleb2 images generates very realistic and reasonable portraits in Figure 2(a). The data augmentation experiment can be naturally fit into the framework of impersonation attacks and the application of their techniques in this direction is very exciting.
I only have two minor suggestions:
1. In Theorem 4.1, the symbol g_{X | Y} was introduced previously, whereas g_{X | A} was never introduced. I have to go to the appendix to understand the definition of g_{X | A}.
2. There is a minor issue in the proof of Lemma D.2 in page 16. The authors seem to miss a 1/2 factor in the second to last row in equation (D.3).


In this paper we consider self-supervised representation learning to improve sample efficiency in reinforcement learning (RL). We propose a forward prediction objective for simultaneously learning embeddings of states and action sequences. These embeddings capture the structure of the environment's dynamics, enabling efficient policy learning. We demonstrate that our action embeddings alone improve the sample efficiency and peak performance of model-free RL on control from low-dimensional states. By combining state and action embeddings, we achieve efficient learning of high-quality policies on goal-conditioned continuous control from pixel observations in only 1-2 million environment steps.

Summary: The paper proposes training dynamics-aware embeddings of the state and k-action sequences to aid the sample efficiency of reinforcement learning algorithms. The authors propose learning a low-dimensional representation of the state space, $z_s$ as a well as a temporally extended action embedding, $z_a$. The latter will be used in conjunction with a higher level policy that plans in this abstract action-space, $z_a$. By using these two embeddings, the authors test the proposed system on a set of Mujoco tasks and show improved results.
Positives:
1) Fairly simple objective, in line with previous work on unsupervised learning methods to representation learning in RL (like DARLA, variational intrinsic control, etc).
2) The temporally extended nature of the action embedding makes is particularly attractive for HRL systems as a continuous space of options (via $z_a$).
Questions and Points of improvements:
1) Main concern: The need to pre-train the embedding before the RL task, I strongly believe limits the applicability of the proposed algorithm. The embeddings are trained under a uniformly random policy, which in many cases in RL is not informative enough to reach, with decent probability, many of the states of interest. Thus the embedding will reflect only a small subset of the state/action-space. Thus it will be highly depend on the tasks under consideration if this is enough variety for generalisation across the stationary distribution of more informed RL policies. Implicitly, the authors are making a continuity assumptions over the state and action space.
(To be more precise: A particular failure case of the action embedding would be if say one of the action (down) has no effect in the part of the space where the uniform policy has explored. Now this becomes an important action in a level down the line where the agents needs to go down a tunnel -- example from Atari's Pitfall. In this case, under the embedding training, since the down has had no effect in training, this action will not be represented at all. This would means the RL algorithm could not ever learn to use it).
The co-evolution of the representation and the RL policy, I think it's paramount especially when dealing with exploration.
2) Q: Section 3.2: "we extend... to work with temporally extended actions while maintaining off-policy updates ..". Can the authors expand on how this is done? Both updates in this section seem to be on policy ($\mu$).
3) Q: Section 3.2: "Q can only be trained on $N/k$ observations. This has a substantial impact on sample efficiency". Note that this is actually an explicit trade-off between reduced number of samples we see ($N/k$) and the increased horizon in propagating information, due to the effective k step update. This trade-off need not be optimal for $k=1$.
4) Notes of experiments:
a) It is hard to assess the difficulty of the exploration problems investigated. This relates to point 1) and the implicit assumptions highlighted there.
b) It would have been nice to have a study on $k$ and it's impact on the sample complexity. The larger the $k$ the harder the representation learning problem becomes; and possibly the larger the number of samples needed to learn in this combinatoric space. How does this trade-off with the benefits one could potentially get in the RL phase?
c) For the comparison algorithms: where any of these using a temporal extended update rule?  Or are all of them 1-step TD like algorithms? It would good to separate the effect of the multiple-step update in Sec. 3.3 and the exploration in this abstract action space.


This paper considers multi-agent reinforcement learning (MARL) in networked system control. Specifically, each agent learns a decentralized control policy based on local observations and messages from connected neighbors. We formulate such a networked MARL (NMARL) problem as a spatiotemporal Markov decision process and introduce a spatial discount factor to stabilize the training of each local agent. Further, we propose a new differentiable communication protocol, called NeurComm, to reduce information loss and non-stationarity in NMARL. Based on experiments in realistic NMARL scenarios of adaptive traffic signal control and cooperative adaptive cruise control, an appropriate spatial discount factor effectively enhances the learning curves of non-communicative MARL algorithms, while NeurComm outperforms existing communication protocols in both learning efficiency and control performance.

This paper is concerned with network multi-agent RL (N-MARL), where agents need to update their policy based on messages obtained only from neighboring nodes. This is done under sensible restrictions on the state transition distribution, which can be claimed to hold true in realistic networked settings. The authors argue that introducing a spatial discount factor (along a temporal one), where neighboring nodes have a small distance, stabilizes learning. Also, they provide a way of learning a networked communication protocol. Experiments are done on somewhat realistic simulations of traffic.
I am not an expert in this type of distributed MARL, and what the SotA there is. What is proposed here makes sense to me, even though it also does not look very surprising to me. Spatial discounting seems to make sense. The authors do not comment on the relationship between the d_{ij} (or how to choose them) with the neighborhood graph, I suppose that neighbors have short distances, while non-neighbors have not. The NeurComm protocol is a bit vague to me, and I cannot say how it relates to other proposals. I find it a bit strange (unless I misunderstood something) that complete state and policy information is transmitted to every neighbor, and on top some latent vector. This sounds expensive to me, surely these communication channels are limited? It'd be more interesting to consider bandwidth limitations here. Without knowing related work on other protocols in detail, I suspect they probably work less well than NeurComm, simply because they allow for smaller messages only (f.ex., whereas they talk about "policy fingerprints", they seem to submit the complete policy parameters to neighbors -- that is not just a fingerprint). Maybe I am wrong, but if so, the paper does not explain things properly (there is some encoding and decoding going on, but just to propagate the hidden states at each node).
To me as an outsider, this looks like interesting work with well-simulated experiments. Of course, these are simulated and allow for no real-world conclusions, but then much of RL work does neither. It is just I'd be hard pressed to say what is really surprising in this paper, and what could be learned from it.


We argue that the widely used Omniglot and miniImageNet benchmarks are too simple because their class semantics do not vary across episodes, which defeats their intended purpose of evaluating few-shot classification methods. The class semantics of Omniglot is invariably "characters" and the class semantics of miniImageNet, "object category". Because the class semantics are so similar, we propose a new method called Centroid Networks which can achieve surprisingly high accuracies on Omniglot and miniImageNet without using any labels at metaevaluation time. Our results suggest that those benchmarks are not adapted for supervised few-shot classification since the supervision itself is not necessary during meta-evaluation. The Meta-Dataset, a collection of 10 datasets, was recently proposed as a harder few-shot classification benchmark. Using our method, we derive a new metric, the Class Semantics Consistency Criterion, and use it to quantify the difficulty of Meta-Dataset. Finally, under some restrictive assumptions, we show that Centroid Networks is faster and more accurate than a state-of-the-art learning-to-cluster method (Hsu et al., 2018).

<Paper summary>
The authors argue that the popular benchmark datasets, Omniglot and miniImageNet, are too simple to evaluate supervised few-shot classification methods due to their insufficient variety of class semantics. To validate this, the authors proposed clustering-based meta-learning method, called Centroid Network. Although it does not utilize supervision information during meta-evaluation, it can achieve high accuracies on Omniglot and miniImageNet. The authors also proposed a new metric to quantify the difficulty of meta-learning for few-shot classification.
<Review summary>
Although the main claim of this paper seems correct, it is not sufficiently supported by theory or experiments. My score is actually on the border line, but I currently vote for ``weak reject, because some points in the paper are ambiguous yet. Given clarifications in an author response, I would be willing to increase the score.
<Details>
* Strength
 + The paper is well-organized. Especially, the examples shown in the introduction greatly help understanding of what the authors argue in this paper.
 + A novel study on quantifying the difficulty of meta-learning.
 + The proposed CentroidNet performs well in the experiments.

* Weakness and concerns
 - Does CentroidNet really work without labels during ``meta-validation"? As far as I understand, ground truth clusters of the support set defined by the labels are required to compute the accuracies. Therefore, the labels seem to be required to validate the performance of the model. I think it should be ``meta-test."
 - The authors state ``The most intuitive way to train ..., we did not have much success with this approach" in 5.3, but it is counter-intuitive. If the class semantics are similar among episodes, ``the most intuitive way" should work, because it can learn the common semantics via meta-training. Further discussion about why it does not work is required.
 - The high performance of CentroidNet does not support the claim on the insufficient variety of the class semantics. According to ablation study, adopting Sinkhorn K-means is the most important factor to improve the performance. It means that adopting weighted average like in [R1] can also improve the performance of ProtoNet, which results in substantial difference in the performance between ProtoNet and CentroidNet that can deny the claim.
 - The definition of CSCC is not convincing. First, I could not get the meaning of ``unsupervised Bayes accuracy" (supervised Bayes accuracy means 1 - Bayes error rate, right?). Second, CSCC seems to mainly quantify the importance of the supervision information during meta-learning, which is not directly related to the difficulty of few-shot learning problem. Intuitively, difficult few-shot learning problems should lead to lower supervised Bayes accuracy, which does not necessarily decrease CSCC. Third, what we can induce via comparing CSCC is not clarified in theory. The discussion in 6.3 is too subjective and specific for the case of training with ILSVRC/all datasets.
 - This paper lacks citing some closely related works [R1, R2].
[R1] ``Infinite Mixture Prototypes for Few-Shot Learning," ICML2019
[R2] ``A Closer Look at Few-shot Classification," ICLR2019
* Minor concerns that do not have an impact on the score
 - Another arXiv paper related to this work: ``Rapid Learning or Feature Reuse? Towards Understanding the Effectiveness of MAML"


We extend the randomized singular value decomposition (SVD) algorithm (Halko et al., 2011) to estimate the SVD of a shifted data matrix without explicitly constructing the matrix in the memory. With no loss in the accuracy of the original algorithm, the extended algorithm provides for a more efficient way of matrix factorization. The algorithm facilitates the low-rank approximation and principal component analysis (PCA) of off-center data matrices. When applied to different types of data matrices, our experimental results confirm the advantages of the extensions made to the original algorithm.

This paper extends the existing randomized SVD algorithm by Halko et al. to propose a shifted randomized SVD algorithm. The proposed algorithm performs randomized SVD on \bar{X}=X-\mu 1^T with a given constant vector \mu without explicitly constructing \bar{X} from X.
The proposed algorithm seems to be a straightforward modification of the randomized SVD algorithm, so that the contribution of this paper would be incremental. Also, the claimed benefits of the proposed method are better accuracy, less memory usage, and less computational time. On memory usage and computational time, however, I could not find any experimental comparison in Section 5. At least in Section 5.3 where the data matrix is indeed sparse, there should be some comparison in these respects. Because of these reasons, I would not be able to recommend acceptance of this paper.
In Algorithm 1, line 6, I think that one has to subtract not \mu 1^T but \mu 1^T \Omega.
I guess that equation (12) comes from Theorem 1.2 in Halko et al. (2011). An explicit citation should be needed here to put an appropriate credit to that paper. Also, the conditions under which equation (12) holds should be explicitly stated. In fact, Theorem 1.2 in Halko et al. is about the expected reconstruction error with rank-2k factorization obtained by randomized SVD, which is not consistent with the authors claim in page 3, lines 4-5: The first top unused singular value is not \sigma_{k+1} but \sigma_{2k+1}.
Page 1, line 25: that provides (for) the SVD estimation
Page 2, line 38: that span(s) the range
Page 4, line 6: a s(h)ifted matrix
Page 4, line 29: to be use(d) by the
Page 5, line 22: uniformly distribut(ion -> ed)
Page 5, lines 33, 38: Space is missing after the period.
Page 5, line 43: There is an extra comma.
Page 5, line 48: suggest(s) that
Page 6, line 4: approaches (to) zero
Page 6, line 14: 16 \times 1979 should read 64 \times 1979.
Page 7, line 2: is validate(d)


We address the problem of learning to discover 3D parts for objects in unseen categories. Being able to learn the geometry prior of parts and transfer this prior to unseen categories pose fundamental challenges on data-driven shape segmentation approaches. Formulated as a contextual bandit problem, we propose a learningbased iterative grouping framework which learns a grouping policy to progressively group small part proposals into bigger ones in a bottom-up fashion. At the core of our approach is to restrict the local context for extracting part-level features, which encourages the generalizability to novel categories. On a recently proposed large-scale fine-grained 3D part dataset, PartNet, we demonstrate that our method can transfer knowledge of parts learned from 3 training categories to 21 unseen testing categories without seeing any annotated samples. Quantitative comparisons against four strong shape segmentation baselines shows that we achieve the state-of-the-art performance.

This paper proposes a method for part segmentation in object pointclouds. The method is to (1) break the object into superpixel-like subparts (without semantic meaning yet), then (2) score pairs of parts on their mergeability, (3) greedily merge the best pair, and repeat. The scoring has a unary component (called a "purity" module), and a pairwise component (called a "rectification" module); the unary component determines if the joined pointcloud of two sub-parts appears part-like, and the pairwise component determines if the features of the two sub-parts appear compatible. These components are implemented as pointnets/MLPs. Finally there is a termination module, which sigmoid-scores part pairs on whether they should actually merge (and the algorithm continue), or not (and we stop). The purity and termination modules are trained supervised, to mimic intersection-like and mergeability scores, and the rectification module with a "reward" which is another mergeability score (coming from GT and the purity module).
The method is interesting for being (1) iterative, and (2) driven by purely local cues. The iterative approach, with small networks doing the work, is a nice relief from the giant-network baselines (such as PartNet-InsSeg) that take the entire pointcloud as input and produce all instance segmentations directly. Also, whereas most works try to maximize the amount of contextual input to the learning modules, this work makes the (almost certainly correct) observation that the smaller the contextual input, the smaller the risk for overfitting. This is a bit like making the approach "convolutional", in the sense that the same few parameters are used repeatedly over space (and in this case, also repeated over scale). The design of the local modules makes sense, although I would prefer they be called unary/pairwise instead of purity/rectification, and the RL training procedure looks reasonable also.
I am not totally clear on how the termination module actually comes into play. From the name, it sounds like this network would output 1 when the algorithm should terminate, but in its usage, it seems to output 1 when the best-scored pair should be merged. So then, does the algorithm terminate when this module decides to NOT merge the best-scored pair? This sounds like it bears great risk of early stopping. I would appreciate some clarification on this.
The abstract says that locality "guarantees the generalizability to novel categories". This is an overstatement, since "guarantees" implies some theoretical proof, and also since the paper's own results (in Table 1 and 3) indicate that cross-category generalization is far from addressed, and depends partly on the categories used in training (shown in Table 2).
I assume that this method has (or at least can have) far fewer parameters than the baselines, since the components never need to learn broad contextual priors. Can the authors clarify and elaborate on this please? If you can show that your method has far fewer parameters than the baselines, it would improve the paper I think.
Can the authors please provide some statistics on the earliest stage of the method, where superpixel-like parts are proposed? How many proposals, and how many pairs does this make, and how slowly do the main modules proceed through these pairs?
Is there a missing step that makes the part selection non-random? It seems like many of the pairs can be rejected outright early on, such as ones whose centroids exceed some distance threshold in 3D.


We present FasterSeg, an automatically designed semantic segmentation network with not only state-of-the-art performance but also faster speed than current methods. Utilizing neural architecture search (NAS), FasterSeg is discovered from a novel and broader search space integrating multi-resolution branches, that has been recently found to be vital in manually designed segmentation models. To better calibrate the balance between the goals of high accuracy and low latency, we propose a decoupled and fine-grained latency regularization, that effectively overcomes our observed phenomenons that the searched networks are prone to "collapsing" to low-latency yet poor-accuracy models. Moreover, we seamlessly extend FasterSeg to a new collaborative search (co-searching) framework, simultaneously searching for a teacher and a student network in the same single run. The teacher-student distillation further boosts the student model's accuracy. Experiments on popular segmentation benchmarks demonstrate the competency of FasterSeg. For example, FasterSeg can run over 30% faster than the closest manually designed competitor on Cityscapes, while maintaining comparable accuracy.

Summary:
- key problem: neural architecture search (NAS) to improve both accuracy and runtime efficiency of deep nets for semantic segmentation;
- contributions: 1) a novel NAS search space leveraging multi-resolution branches, efficient operators ("zoomed convolutions"), and parametrized expansion ratios, 2) a decomposition and normalization of the latency objective to avoid a bias towards very fast but weak architectures, 3) a natural extension of the optimization problem to simultaneously search for teacher and student architectures in one pass, 4) a novel state-of-the-art efficient architecture (FasterSeg) found by the aforementioned NAS algorithm, 5) a detailed experimental evaluation on 3 datasets and an ablative analysis quantifying the benefits of the aforementioned contributions.
Recommendation: Accept.
Key reason 1: solid experimental results backing the claims.
- When compared to related efficient architectures, the proposed method results in competitive accuracy at significantly higher frame rates.
- This is validated on Cityscapes, CamVid, and BDD with the architecture found on Cityscapes.
- The resulting architecture (FasterSeg) is actually interpretable and makes sense, extending the handcrafted architectures used as inspiration.
- The ablative analysis shows that the numerous individual contributions are significant, esp. the multi-branch formulation and student co-searching.
Key reason 2: well-motivated method with a collection of multiple novel contributions that are interesting and practical.
- The multi-resolution branches formulation is simple and extends typical NAS focusing on single paths through the supernet.
- Teacher/student co-searching via learning two sets of architectures in one supernet seems novel, simple, and effective. Always picking the largest expansion ratios for the teacher and applying a distillation loss in addition to the latency loss for the student is sensible and seems to beat the standard pruning approach at no significant extra cost during NAS.
- The zoomed convolution operator seems like a novel efficient alternative to (expensive) dilated convolutions. Although it is very simple (bilinear downsampling -> 3x3 conv -> bilinear upsampling), it is not commonly used as an operator (as far as I know), and yet is found to be a key part of the final architecture (Table 7 appendix I) due to its low latency. The closest related operator / block I could think of might be blocks found in stacked hourglass networks (Newell et al).
- The optimization of the expansion ratios using the Gumbel-Softmax trick is interesting, although this is also explored in the very recent paper by Shaw et al. 2019 (possibly the closest related work that should be discussed in a bit more depth in Section 2);
- Decomposing and normalizing the latency objective to avoid "architecture collapse" (convergence to anemic architectures stemming from certain architectural factors dominating latency) is principled and effective.
- Caveat regarding novelty: I could not find the ideas proposed here in the literature, but its hard to be sure due to 1) the recent explosion of NAS papers, 2) the simplicity of certain ideas (e.g., "zoomed convolutions").
Additional Feedback:
- how is the student trained after NAS? Is the teacher first retrained from scratch? Is the student retrained from scratch on the teacher (after NAS or retraining)? in general, more details on what happens after co-searching would be helpful;
- "human designed CNN architectures achieve superior accuracy performance nowadays": this is a surprising statement considering the cited NAS papers report performance improvements (e.g., Zoph and Le 2016);
- missing reference also using multi-scale NAS for efficient and accurate semantic segmentation: "Searching for Efficient Multi-Scale Architectures for Dense Image Prediction", Chen et al, NeurIPS 2018;
- missing reference on NAS for efficient semantic segmentation that also uses distillation: "Fast neural architecture search of compact semantic segmentation models via auxiliary cells", Nekrasov et al, CVPR 2019;
- missing reference on joint NAS and quantization: "Joint Neural Architecture Search and Quantization", Chen et al, arxiv 2018;
- "we choose a sequential search space (rather than a directed acyclic graph of nodes (Liu et al., 2018b)), i.e., convolutional layers are sequentially stacked in our network": "stacked" is confusing here;
- "we allow each cell to be individually searchable across the whole search space": what do you mean? Anything beyond each cell containing different operators after learning?
- if \alpha = \beta in eq. 6 of appendix C, then w and hence Target(m) does not depend on latency, isn't this a typo?
- "Gumbel-Max" is typically called "Gumbel-Softmax" (cf. "Categorical Reparameterization with Gumbel-Softmax", Jang et al, ICLR'17);
- typos: "find them contribute", "the closet competitor", "is popular dense predictions".


To perform robot manipulation tasks, a low-dimensional state of the environment typically needs to be estimated. However, designing a state estimator can sometimes be difficult, especially in environments with deformable objects. An alternative is to learn an end-to-end policy that maps directly from high-dimensional sensor inputs to actions. However, if this policy is trained with reinforcement learning, then without a state estimator, it is hard to specify a reward function based on high-dimensional observations. To meet this challenge, we propose a simple indicator reward function for goal-conditioned reinforcement learning: we only give a positive reward when the robot's observation exactly matches a target goal observation. We show that by relabeling the original goal with the achieved goal to obtain positive rewards (Andrychowicz et al., 2017), we can learn with the indicator reward function even in continuous state spaces. We propose two methods to further speed up convergence with indicator rewards: reward balancing and reward filtering. We show comparable performance between our method and an oracle which uses the ground-truth state for computing rewards. We show that our method can perform complex tasks in continuous state spaces such as rope manipulation from RGB-D images, without knowledge of the ground-truth state.

The paper tackles the problem of self-supervised reinforcement learning through the lens of goal-conditioned RL, which is in line with recent work (Nair et al, Wade-Farley et al, Florensa et al, Yu et al.). The proposed approach is a simple one - it uses the relabeling trick from (Kaelbling, 1993; Andrychowicz et al., 2017) to assign binary rewards to the collected trajectories. They apply two simple tricks on top of relabeling:
1. Reward balancing: Balancing the number of 0/1 rewards used for training the policy.
2. Reward filtering: A heuristic that rejects certain negative-reward transitions for learning if the q value for the transition is greater than some threshold q_0.
While I like the simplicity of the proposed approach when compared to competing methods, my overall recommendation is reject based on the current state of the paper, because of the following reasons:
1. The technical novelty is quite limited - the paper mostly uses the framework from Andrychowicz et al., 2017 with a specific choice of epsilon (=0) for giving positive rewards. The method does reward balancing, but similar goal-sampling have been used in prior work like Nair et al., 2018, and is not essential to obtaining good results (Appendix C). The main technically novel component is likely the reward filtering mechanism, but I find it to be somewhat ad-hoc since it assumes that the Q-values learned by the Q-network to be reasonably good during training time, which is not the case for most modern Q-learning based methods [1, 2].
2. The provided analysis is not particularly illuminating, see my detailed notes below.
3. The experiments are underwhelming, see my detailed notes below.
I would be willing to overlook 1 or 2 if the authors did a more thorough experimental evaluation which showed the method working well when compared to alternatives, but that is not the case right now.
Note on Section 6 (analysis)
The authors provide a simple analysis in Section 6 to bound the suboptimality of the learned policy. Unless I’m missing something, the resulting bound of t_3 <= t_1 + d is trivially true, since d is defined to be the diameter of O_{+}(o_g), and t_1 is the number of timesteps taken by an optimal policy to go from o_t to O_{+}(o_g). As a result, I don’t find this analysis illuminating or interesting - perhaps the author can provide counter arguments here to change my mind.
Note on Section 7 (experiments)
For sim experiments, two of the tasks are extremely simple (free space reaching in 2D and 3D, respectively) where essentially everything works - the proposed method, baselines and ablations. The third task of rope manipulation is fairly interesting at a first look - but it appears to have been greatly simplified. The authors consider a high-level action space and an episode of only three timesteps. Further, the authors make the simulated robot arm invisible in the supplementary videos, which greatly simplifies the problem visually. Since the entire motivation is about learning well in real world settings, I feel this is a bit underwhelming. Figure 1 is misleading, since it shows a visible robot arm in front of a rope. This also appears to hint that the method did not work well with realistic visuals, highlighting a major limitation of the proposed approach. I think it would be valuable to include such failures (and discussions around them) in future submissions.
For the real world experiments, the task being considered is extremely simple (free space reaching), and does not even require pixel observations. Even for this simple task, the error achieved by the method is 10cm (starting error was 20cm), which is quite poor - robotic arms like the Sawyer should be able to achieve much lower errors. Even the oracle reward achieves an error of 10cm, which might indicate a bug in the author’s real world robotic setup. In comparison, prior work such as Nair et al. is able to tackle harder problems in the real world (like non-prehensile pushing).
Minor points
- Section 4 contains a nice discussion on false positives and false negatives when using non-oracle reward functions for reinforcement learning, where they also perform a simple experiment to show how false positives can negatively impact learning much more severely than false negatives. This does a good job of motivating the method (i.e. avoiding false positives), but also undermines the motivation behind reward filtering, which is perhaps the main technically novel component of the proposed approach.
- Section 2.3 (i.e. related work on deformable object manipulation) states that "Our approach applies directly to high-dimensional observations of the deformable object and does not require a prior model of the object being manipulated.”, and only cites prior work that assumes access to deformable object models. However, there is recent work that enable similar manipulation skills without access to such models. For example, Singh et al. [3] are able to learn to manipulate a deformable object (i.e. a piece of cloth) directly from high-dimensional observations using deep RL in the real world, and do not require object models (or ground truth state), but do require other forms of sparse supervision.
- Typo: On page 7, “is kept the same a other approaches.” -> “is kept the same as other approaches.”
[1]: Diagnosing Bottlenecks in Deep Q-learning Algorithms. Fu et al., ICML 2019
[2]: Double Q-learning. V. Hasselt. NIPS 2010
[3]: End-to-End Robotic Reinforcement Learning without Reward Engineering. Singh et al., RSS 2019.


While much recent work has focused on biologically plausible variants of errorbackpropagation, learning in the brain seems to mostly adhere to a reinforcement learning paradigm; biologically plausible neural reinforcement learning frameworks however were limited to shallow networks learning from compact and abstract sensory representations. Here, we show that it is possible to generalize such approaches to deep networks with an arbitrary number of layers. We demonstrate the learning scheme -DeepAGREL -on classical and hard image-classification benchmarks requiring deep networks, namely MNIST, CIFAR10, and CIFAR100, cast as direct reward tasks, both for deep fully connected, convolutional and locally connected architectures. We show that for these tasks, DeepAGREL achieves an accuracy that is equal to supervised error-backpropagation, and the trial-and-error nature of such learning imposes only a very limited cost in terms of training time. Thus, our results provide new insights into how deep learning may be implemented in the brain.

Summary:
This paper generalizes the AGREL biologically plausible learning algorithm to deeper networks. It presents experiments to show that the method can scale to medium-sized tasks with only a modest slow down in training speed and essentially no change in classification performance.
Major comments:
This is an interesting paper which shows that a particular biologically plausible learning method can attain comparable performance to gradient descent on small and mid-size visual classification tasks. The proposed mapping to the biology is of a different flavor from many other recently proposed approaches.
It is striking that the CIFAR10 network trains about as fast as the EBP network, while the CIFAR100 network trains much more slowly. This is presumably because randomly guessing the right answer out of 100 possibilities is the bottleneck. The paper could be strengthened by studying the speed of learning as a function of the number of output classes. For this approach to scale to ImageNet, with a 1000-way classification (or to our human visual recognition abilities with far more classes), this is an important scaling dimension to consider.
It should be noted that other biologically plausible schemes like feedback alignment were able to solve CIFAR and other smaller image classification tasks, but struggled when applied to the larger scale ImageNet problem. The paper could be improved by pointing to this limitation of the present work, the possibility that performance could change on larger tasks, and the need to conduct larger experiments in future work.
Personally I think the statement at the beginning of the introduction that only RL occurs in animals and humans is overstated. Unsupervised learning occurs in some form in critical period plasticity, and in various unrewarded statistical learning paradigms, at a minimum.
I would also caution against categorical statements of biological plausibility, for instance, in saying that shared weights in convolutions are biologically implausible (bottom of pg 6). The same criticism has been leveled at gradient descent learning, and as the present submission shows, these intuitive judgements can be misleading.
The distinction between RL and supervised learning is a bit blurred in the present submission because it is considering a classification setting in which exactly one out of a number of possible outputs is rewarded on each trial. This looks very similar to the supervised learning scenario, and relies on stochastic softmax-like competition to select a single output. This approach is very reasonable for mutually exclusive classification tasks, but this is a small subset of the tasks that full EBP could be applied to.
The paper is fairly clear but the explanation of the algorithm could be further streamlined and condensed. Is DeepAGREL equivalent to stochastically selecting a single unit in a softmax layer and then back propagating only its error? It seems so from what I understand, and this may be a straightforward way to explain the algorithm.
Typos:
The text on page 7 describes MNIST performance as 99.17% but the table has 99.16%.


Preconditioning an minimization algorithm improve its convergence and can lead to a minimizer in one iteration in some extreme cases. There is currently no analytical way for finding a suitable preconditioner. We present a general methodology for learning the preconditioner and show that it can lead to dramatic speed-ups over standard optimization techniques.

This paper attempts to learn a preconditioner for optimization, specifically for the Dual space preconditioned descent (DPGD).
- The techniques used to learn the preconditioner are heuristic, not scalable and without justification or ablation studies.
- It does not compare against "standard" optimization techniques that construct data-driven preconditioners such as Adam or Adagrad or even to more Newton, natural gradient methods that use the Hessian or the Fisher information matrix as preconditioners. It shows ad-hoc synthetic experiments in dimensions 1 and 50. This is clearly not enough.
Detailed review below:
- Section 2: Please explain why Legendre functions are useful in ML. For assumption 1, 2; it needs to be explained why these hold for a given f*. What constraints do you need on f? What functions satisfy these? Please explain this explicitly.
- Section 3: What is the number of points x_i needed in high dimensions to learn? Is it even possible to scale up this method to high dimensions?
- Constructing \mu requires computing the determinant of the Jacobian. What is the computational complexity? Moreover, it seems that we need access to the \nabla f(x) for all x in D(f)?
- Please state all the assumptions in the beginning rather than introducing one at a time in the propositions.
- Remark 1: It is unclear that the cost of an inverse Hessian matrix is more than the procedure proposed in this paper.
- Section 3.5: Please explain what is the advantage of this learned optimizer compared to other methods? Note that there is literature on non-smooth optimization and methods like sub-gradient descent can be used in this case.
- What is the justification for the selection of the loss function and log-rescaling?
- The result of Lemma 1 is standard. Please acknowledge this.
-  Section 4: "The step-size is set to 1". It seems that the optimizer has been overfit and engineered to work on this specific problem. Either these decisions need to be justified, there needs to be an ablation study or there needs to be a larger set of experiments.


We prove the precise scaling, at finite depth and width, for the mean and variance of the neural tangent kernel (NTK) in a randomly initialized ReLU network. The standard deviation is exponential in the ratio of network depth to width. Thus, even in the limit of infinite overparameterization, the NTK is not deterministic if depth and width simultaneously tend to infinity. Moreover, we prove that for such deep and wide networks, the NTK has a non-trivial evolution during training by showing that the mean of its first SGD update is also exponential in the ratio of network depth to width. This is sharp contrast to the regime where depth is fixed and network width is very large. Our results suggest that, unlike relatively shallow and wide networks, deep and wide ReLU networks are capable of learning data-dependent features even in the so-called lazy training regime.

This paper studies the finite depth and width corrections to the neural tangent kernel (NTK) in fully-connected ReLU networks. It gives sharp upper and lower bounds on the variance of NTK(x, x), which reveals an exponential dependence on a quantity beta=d/n, where d is depth, and n is hidden width. This implies that when beta is bounded away from 0, NTK(x, x) is not deterministic at initialization. The paper further analyzes the change of NTK(x, x) after one step of SGD on a single datapoint x, and shows that the change also depends exponentially on beta.
NTK has been a popular subject of theoretical study in deep learning, and it's an important question to understand when and to what extent NTK can capture the behavior of real neural networks. This paper makes partial progress by analyzing the diagonal entries of the NTK in fully-connected ReLU networks. It concludes that NTK(x, x) at initialization is not deterministic, and can change significantly after doing one SGD step on x, when beta is bounded away from 0. While it's nice that the authors obtained precise answers to these two questions, there are some drawbacks that limit the significance of this paper:
1. The entire paper only considers one single datapoint x, so it doesn't apply to the non-diagonal elements NTK(x, x') or the usual SGD with mini-batches containing multiple datapoints. Of course, it's already implied by the current paper that the NTK is not deterministic and can move a lot when beta is large, but it's unclear whether the reverse is true, i.e., what is the regime when the NTK becomes deterministic and frozen when an entire dataset is involved. An answer (even empirically) to this question would make the paper much more complete.
2. The result is also not so surprising given [Hanin, 2018] (and possibly some other papers by the same author) which also obtained a similar exp(beta) dependence using a similar combinatorial approach.
3. Minor issues regarding incorrect or missing references:
-- "Further, kernel method-based theorems show that even in this infinitely overparameterized regime neural networks will have non-vacuous guarantees on generalization (Wei et al., 2018)." The result of Wei et al. is not about kernel method and in particular not NTK.
-- "In fact, empirically, networks with finite but large width trained with initially large learning rates often outperform NTK predictions at infinite width." The authors should refer to the work of [Arora et al., 2019] which AFAIK is the first paper that provides empirical study of the (convolutional) NTK predictor at infinite width on relatively large datasets like CIFAR-10.
[Hanin, 2018] Which Neural Net Architectures Give Rise to Exploding and Vanishing Gradients?
[Arora et al., 2019] On Exact Computation with an Infinitely Wide Neural Net
--------
update: Thanks to the authors for the detailed reply, which answers most of my questions. I am updating my rating to weak accept in light of this.
Regarding Wei et al.: Their result on NTK is negative, i.e., they aimed to show that NTK is inferior to regularized NN. I find the way you cited this paper a bit weird.


This paper investigates a population-based training regime based on game-theoretic principles called Policy-Spaced Response Oracles (PSRO). PSRO is general in the sense that it (1) encompasses well-known algorithms such as fictitious play and double oracle as special cases, and (2) in principle applies to general-sum, manyplayer games. Despite this, prior studies of PSRO have been focused on two-player zero-sum games, a regime wherein Nash equilibria are tractably computable. In moving from two-player zero-sum games to more general settings, computation of Nash equilibria quickly becomes infeasible. Here, we extend the theoretical underpinnings of PSRO by considering an alternative solution concept, α-Rank, which is unique (thus faces no equilibrium selection issues, unlike Nash) and applies readily to general-sum, many-player settings. We establish convergence guarantees in several games classes, and identify links between Nash equilibria and α-Rank. We demonstrate the competitive performance of α-Rank-based PSRO against an exact Nash solver-based PSRO in 2-player Kuhn and Leduc Poker. We then go beyond the reach of prior PSRO applications by considering 3-to 5-player poker games, yielding instances where α-Rank achieves faster convergence than approximate Nash solvers, thus establishing it as a favorable general games solver. We also carry out an initial empirical validation in MuJoCo soccer, illustrating the feasibility of the proposed approach in another complex domain.

This paper extends the original PSRO paper to use an $\alpha$-Rank based metasolver instead of the projected replicator dynamics and Nash equilibria based metasolvers in the original. To this end, the paper modifies the original idea of Best-Response (BR) oracle since it can ignore some strategies in $\alpha$-Rank defining SSCC to introduce the idea of _preference-based_ Best-Response (PBR) oracle. The need for a different oracle is well justified especially with the visualization in the Appendix. The main contributions that the paper seems to be going for is a theoretical analysis of $\alpha$-Rank based PSRO compared to standard PSRO. From the PBR's description (especially in Sec 4.3) it seems the paper is intereseted in expanding the population with novel agents rather than finding the "best" single agent which is not well defined for complex games with intransitivities. Nevertheless, it seems that BR is mostly compatible with PBR for symmetric zero-sum two-player games.
The paper performs empirical experiments on different versions of poker. First set of experiments compare BR and PBR with $\alpha$-Rank based metasolver on random games and finds that PBR does better than BR at population expansion as defined. The second set of experiments compare the metasolvers. $\alpha$-Rank performs similarly to Nash where applicable. Moreover it's faster than Uniform (fictitious self-play) on Kuhn. Then the paper tacks on the MuJoCo soccer experiment as a teaser for ICLR crowd.
Overall the paper is quite interesting from the perspective of multiagent learning and I would lean towards accepting. However the paper needs to clarify a lot of details to have any chance of being reproducible.
** Clarifications needed:
- Tractability of PBR-Score and PCS-Score
It's unclear how tractable these are. Moreover these were only reported for random games. What did these scores look like for the Poker games? Could you clarify how exactly these were computed?
- It's somewhat unclear what the lack of convergence without novelty-bound oracle implies. Does this have to do with intransitivities in the game?
- Dependence of $\alpha$?
The original $\alpha$-Rank paper said a lot about the importance of choosing the right value for $\alpha$. How were these chosen? Do you do the sweep after every iteration of PSRO?
- Oracle in experiments?
The paper fails to mention the details about the Oracles being used in the experiments. They weren't RL oracles but more details would be useful.
- BR not compatible with PBR, albeit not the other way around, meaning one of the solutions you get from PBR might be BR, but can we say which one?
- For MuJoCo soccer was it true PSRO or cognitive hierarchy. In general, the original PSRO paper was partly talking about the scalable approach via DCH. This paper doesn't mention that at all. So were the MuJoCo experiments with plain PSRO? What was the exact protocol there? From the appendix it's unclear how the team-vs-team meta game works with individual RL agents. Moreover how are the meta-game evaluation matrices computed in general? How many samples were needed for the Poker games and MuJoCo soccer?
- The counterexamples in Appendix B3 are quite interesting. Do you have any hypotheses about the disjoint support from games' correlated equilibria?