We consider a setting where biases are involved when agents internalise an environment. Agents have different biases, all of which resulting in imperfect evidence collected for taking optimal actions. Throughout the interactions, each agent asynchronously internalises their own predictive model of the environment and forms a virtual simulation within which the agent plays trials of the episodes in entirety. In this research, we focus on developing a collective policy trained solely inside agents' simulations, which can then be transferred to the real-world environment. The key idea is to let agents imagine together; make them take turns to host virtual episodes within which all agents participate and interact with their own biased representations. Since agents' biases vary, the collective policy developed while sequentially visiting the internal simulations complement one another's shortcomings. In our experiment, the collective policies consistently achieve significantly higher returns than the best individually trained policies.

This paper presents a model-based RL architecture based on World Models which additionally makes use of an ensemble of models. Specifically, each model in the ensemble is independently trained models and has different amounts and types of occlusions (i.e. part of the image cropped out or a small piece of the image is magnified). The ensemble of models are used to train what is referred to as a “collective” policy in the CarRacing and VizDoom environments. The collective policy achieves better performance than policies trained using the individual models and achieves higher data efficiency than model-free RL.
While I think the paper tackles an interesting question---how to leverage multiple models of the world that may be trained on different data and with different capabilities---I unfortunately do not feel that this paper is ready for publication at ICLR, and thus recommend rejection. My justifications are that (1) the paper suffers from clarity issues and several cases makes claims or statements which are incorrect, (2) the proposed evaluation is somewhat contrived and the method does not seem to generalize well to other domains, and (3) there is not much engagement with the rest of the literature on model-based RL. I think that addressing #1 and #3 alone would substantially improve the paper. An additional change that would make the paper much more compelling would be to train the models on more ecologically valid observations, such as from multiple modalities or different camera angles (see below for discussion).
My first concern with the paper has to do with clarity. Overall, I found the paper quite difficult to read as it uses a lot of unusual terminology to explain what it does. For example, the paper uses “rule component” to refer to what would normally be a “reward function”. Similarly, although this is an ensemble method, the term “ensemble” is never used even once. The word “bias” is also used in a way which is different from how it is normally used---here it seems to refer to the fact that the agents see different things, whereas in ML bias typically has a precise meaning in terms of the bias-variance tradeoff. I also noticed several places throughout the paper where citations or related work seem to be misunderstood:

- 1st page, 2nd paragraph: this is not what Griffiths (2010) is about, and is not really an appropriate use of the term “inductive bias”
- 2nd page, contribution 1: the paper claims that it is not dealing with a partially observable setting, but this is exactly what the paper is doing. Specifically, each model in the ensemble receives different partially observed observations from the environment. Although in traditional RL partially observed usually means “one agent, one observation of a state”, there is nothing limiting the general formulation of partial observability to cover the case of “multiple agents, multiple observations of the same state”.
- 8th page, line 2: Lake et al (2017) is not really about “learning to think”.
- Section 4, last paragraph: this seems to be not very relevant to this paper, as the present paper is not about a multiagent system or about graph-based representations.

There are some additional issues with clarity of the model explanation; for example, it’s not clear how the translator component is trained (the paper states that it is trained with meta-learning, but this is vague).

My second concern is that the approach is somewhat contrived and does not seem to generalize well to other domains. In particular, under what circumstances would you want to train multiple models that receive inputs that are missing different pieces? I can see the appeal of learning different models in some cases (e.g. different camera angles, or different sensory modalities) but these have not been evaluated here which makes the whole thing seem a bit arbitrary. Moreover, while it seems to work ok in the CarRacing domain (though the ensemble-trained policy does not perform as well as a non-ensemble agent which is trained with full observations) it does not seem to work well in the VizDoom environment. On the basis of these two issues it is not at all clear to me how well the proposed method will work in more ecologically valid settings requiring multiple models.

Finally, my third concern is that the engagement with the rest of the model-based RL literature is lacking. As mentioned above, this method is an ensemble method but there is no discussion of other model-based RL ensemble methods (e.g. [1] and [2]). Similarly, the paper dismisses other work on partial observations but this work is quite relevant to the present paper which does indeed include partial observations (see above). While there is a lot of text spent discussing other work such as literature from cognitive science, this is not really that relevant to the present paper and that space would be better spent discussing related work from model-based RL.
Some additional minor comments:
It would be interesting to see a comparison between the ensemble policy trained with models that always receive the same (full) observations versus the present approach (where they receive partial observations). This might still provide some benefit as the models will still result in slightly different latent representations due to different initialization and training data, so it would be interesting to see whether this helps as well even if the observations are the same.
What does “model tr” and “policy lr” mean in Table 1?
In the various tables and figures, how are the error bars computed? Are they over episodes, or agent seeds? In general, how many seeds were used to perform evaluations? (Ideally it would be at least 3).
Page 2, section 2.1: “V and M use sequential image frame data from multiple rollouts from a random policy and apply variational inference and backpropagation algorithm for inference and learning.” → this is imprecise; they do not apply both variational inference and backpropagation; backpropagation is used to implement variational inference.
I found the results of Figure 3d quite interesting, though am wondering if you controlled for the number of evaluations of each model? E.g. if you only train the policy on one model at a time versus training them on all models at a time, do you train for longer in the first case so that the policy ultimately receives the same amount of information from all the models?
[1] Kurutach, T., Clavera, I., Duan, Y., Tamar, A., & Abbeel, P. (2018). Model-ensemble trust-region policy optimization. arXiv preprint arXiv:1802.10592.
[2] Chua, K., Calandra, R., McAllister, R., & Levine, S. (2018). Deep reinforcement learning in a handful of trials using probabilistic dynamics models. In Advances in Neural Information Processing Systems (pp. 4754-4765).


We seek to learn a representation on a large annotated data source that generalizes to a target domain using limited new supervision. Many prior approaches to this problem have focused on learning "disentangled" representations so that as individual factors vary in a new domain, only a portion of the representation need be updated. In this work, we seek the generalization power of disentangled representations, but relax the requirement of explicit latent disentanglement and instead encourage linearity of individual factors of variation by requiring them to be manipulable by learned linear transformations. We dub these transformations latent canonicalizers, as they aim to modify the value of a factor to a pre-determined (but arbitrary) canonical value (e.g., recoloring the image foreground to black). Assuming a source domain with access to meta-labels specifying the factors of variation within an image, we demonstrate experimentally that our method helps reduce the number of observations needed to generalize to a similar target domain when compared to a number of supervised baselines.

This paper proposes a method for unsupervised learning  of data representations that can be manipulated to remove factors of variation via linear transformations. These transformations are called canonicalizations in the paper. The canonicalizations are trained such that images for arbitrary values of the corresponding factor of variation are transformed into images with a fixed, canonical, value for that factor. The paper proposes a model architecture based on a denoising autoencoder, where the canonicalizations are applied to the encoded representation. It also proposes a loss function and sampling scheme for training the model. The paper demonstrates the method on the dSprites dataset, showing that it can effectively learn linear canonicalizations, and that multiple of these canonicalizations can be applied to the same image representation. The paper goes on to test the method on a digit classification task, where the model is trained to learn a representation in a simulator for SVHN data where the transformations to be canonicalized can be controlled, and  used to train a classifier on unseen real data from the SVHN test set.
I think this paper should be accepted as it proposes a novel idea, which does not seem too difficult to reproduce, describes a simulator for synthetic data for digit recognition, and proposes it as a benchmark for learning representations, and provides experimental results that help in better understanding the representation learned by the model.
A couple things I thought were missing in the paper:
Did you try applying the classification loss to both the encoded representation and the canonicalized representation at the same time?
For Figure 2, Did you try applying canonicalizations in different orders? Do they give the same results?
Instead of trying to learn idempotency by gradient descent, you could try to parametrize the canonicalizations with a matrix X, such that C =  X (X^T X)^{-1} X^T. C will be idempotent (although restricted to be symmetric). There might be other constructions that are more efficient and less restrictive.
I'm not sure I understand the PCA figures. Can you please explain how the first principal component was used to generate them?
Minor comments:
* "data  tripets" on page 2
* Figure 5 should appear after Figure 4.


It has been shown that using geometric spaces with non-zero curvature instead of plain Euclidean spaces with zero curvature improves performance on a range of Machine Learning tasks for learning representations. Recent work has leveraged these geometries to learn latent variable models like Variational Autoencoders (VAEs) in spherical and hyperbolic spaces with constant curvature. While these approaches work well on particular kinds of data that they were designed for e.g. tree-like data for a hyperbolic VAE, there exists no generic approach unifying all three models. We develop a Mixed-curvature Variational Autoencoder, an efficient way to train a VAE whose latent space is a product of constant curvature Riemannian manifolds, where the per-component curvature can be learned. This generalizes the Euclidean VAE to curved latent spaces, as the model essentially reduces to the Euclidean VAE if curvatures of all latent space components go to 0.

Summary:
This paper is about developing VAEs in non-Euclidean spaces. Fairly recently, ML researchers have developed non-Euclidean embeddings, initially in hyperbolic space (constant negative curvature), and then in product spaces that have varying curvatures. These ideas were developed for embeddings, and recent attempts have been made to build entire models that operate in non-Euclidean spaces. The authors develop VAEs for the product spaces case.
There's largely two aspects here: one is to be able to write down the equivalents for the operations in models (e.g., the equivalent of adding or multiplying matrices and vectors in Euclidean space have to be lifted to other spaces which no longer have a linear structure). The other are VAE-specific choices, particularly choosing a normal distribution on the manifolds. The authors consider several of these choices and then run a variety of experiments on small latent-dimension cases for VAEs. These reveal that sometimes non-Euclidean and in particular product spaces improve performance.
Strengths, Weakness, Recommendation
I like what the authors are trying to do here; embeddings and discriminative models on non-Euclidean spaces have been developed, offer credible benefits, and generative models are the next step. The authors push forward the machinery needed to do this, and the results seem like there's something there.
On the other hand, the entire work seems quite preliminary. It's hard to say what the takeaway is, or any suggestions for users. The paper is written in a pretty frustrating way. There's an enormous amount of stuff in a sprawling appendix (there are 43 results in the first appendix?!), and checking all of these details will take a great deal of time.
Overall, I recommended weak accept, since a lot of these issues seem like they can be cleaned up.
EDIT: I increased my score based on the authors' response.
Comments:
- The approach taken here is quite similar to another ICLR submission this year, which basically does the same thing but applies these operations to GCNs instead of VAEs.
- A better way to define curvature is just to talk about the sectional curvature, instead of the Gaussian curvature the authors mention at the beginning of section 2. Fortunately for the constant case all of these definitions will be the same.
- It's not quite clear in Section 2.1 why we should care about the fact that you can't fully take K->0 there---why does this hurt anything? You can approximate flat curvature arbitrarily well even without K exactly 0.
- On a similar theme, what's the point of doing the product of {E,S,D,H,P}, instead of just {E,S,H} or {E,D,P}? Seems a bit weird to consider all 5, given the equivalence between S-D and H-P.
- In 2.3, the products of spaces section, the distance decomposition in the 2nd paragraph should have squares (it's an l2): d_M(x,y)^2 = \sum_{i=1}^k d_{M_k_i^n_i)^2(x^i,y^i).
- The discussion in 2.3 should be expanded and made more concrete (some of these you can write out the expressions for), and more pros and cons explained, e.g., which theoretical properties are lost for the wrapped distributions?
- On page 6, I don't understand the first problem with the learnable curvature approach. Why is there no gradient w.r.t to K? Isn't the idea that you'll write this thing as a piecewise function (presumably it's continuous, since that's why the authors built those models that deform to flat), and differentiate the whole thing? Why wouldn't there be a gradient at ELBO(K)? Is it not differentiable at K=0? That doesn't follow directly from just saying the curvature is 0.
-  What's the intuition for the component learning algorithm using 2 dimensions for each of the spaces?
- The experiment section was written in a way where I couldn't understand why the choices being made were there. Why 6 and 12 dimensions here? More clarity here would be great. Also, are there any other models to compare against for these datasets? I'm not a VAE expert; what do other models typically obtain in the authors' regime?


We propose a deep amortized clustering (DAC), a neural architecture which learns to cluster datasets efficiently using a few forward passes. DAC implicitly learns what makes a cluster, how to group data points into clusters, and how to count the number of clusters in datasets. DAC is meta-learned using labelled datasets for training, a process distinct from traditional clustering algorithms which usually require hand-specified prior knowledge about cluster shapes/structures. We empirically show, on both synthetic and image data, that DAC can efficiently and accurately cluster new datasets coming from the same distribution used to generate training datasets.

[Overview]
In this paper, the authors proposed a new clustering method called deep amortized clustering (DAC). Inspired by Lee et al 2019, the authors exploited a transformer to gather the contextual information across different dataset points and then predict the cluster label for each data point. The main difference from Lee et al is that the proposed DAC sequentially estimate the cluster labels for the data points and thus more flexible to estimate the number of clusters in the whole dataset. Based on the proposed DAC method, the authors evaluated the performance on both unsupervised clustering and supervised clustering tasks. it turns out the proposed method has achieved better or comparable performance to previous work on various datasets while hold less computational cost.
[Pros]
1. In this paper, the authors extended the clustering method in Lee et al to a new method called Deep Amortized Clustering (DAC) for data clustering. In this new method, the number of clusters can be unknown at the beginning and the model itself will sequentially cluster the data points into different groups until are data points have been assigned to some clusters. This is an interesting method in that it does not need to specify the number of clusters at the beginning, and thus become more flexible.
2. To achieve the DAC, the authors proposed two losses, one is Minimum Loss Filtering (MLF) and one is Anchor Filtering to cope with either multi-gaussian-distributed data points or even harder datasets. Meanwhile, the authors also proposed to estimate the density P(x; \theta) in the case that the distribution is not knowing in prior.
3. The authors evaluated the proposed method on both synthetic dataset and realistic dataset. On the synthetic dataset, the proposed method is compared with VBDPM and ACT-ST, two methods that can cope with dataset with unknown number of clusters. On the realistic datasets, the authors evaluated on the EMNIST which is of non-MoG distribution. Besides, the authors further evaluated the method on MiniImageNet features and Omniglot dataset, and showcased comparable performance to previous methods but much shorter running  time.
[Cons]
1. Overall, the paper is poorly written and organized. First, the notations in the method section are hard to follow. There are a number of notations which are all capital characters, either representing a function or a method. Second, the whole training process and inference process of the proposed method is not clear to me. How the model is trained on the training set, what are the learnable parameters in the proposed model and what are the settings for the hyper-parameters, etc. Third, it is hard to get the takeaway messages from the experiment sections. The experimental settings for each subsection are not very clearly explained, and the analysis on the experimental results are also vague.
2. In the method, the authors proposed Minimum Loss Filtering (MLF) for clustering with the loss function in Eq(5). During training, the authors use some training data with ground-truth labels to optimize the loss function. However, it is not clear which parameters will be learned in the optimization. Also, after the training, what the exact inference procedure should be is also not clear to me. Overall, it is really hard to me to follow this section on the filtering process. The authors should definitely describe the process more clearly.
3. The experimental results shown in the paper are hard to interpret. First, the setting for each experiment is not clear to me. In Figure 3, it is hard to understand the figures clearly. In table 2 and table 3, some of the clustering methods are deterministic, such as K-means, Spectral clustering, DEC. However, some other clustering methods are learning to clustering methods, such as KCL and MCL. Putting all of the numbers in the same table is confusing and make it hard to compare. I would suggest the authors make a clear distinction between different methods: 1) deep clustering methods which directly cluster on top of the test set; 2) learning to clustering method which learn some parameters on training set and then generalize to test set; 3) amortized clustering, which also learn some parameters on training and then test on the test set with just one forward pass. Splitting the testing results into three group will be helpful to the readers to understand the paper and the proposed method.
4. Another missed part in the model is the ablation study. How sensitive the model is to different training set, e.g., different training dataset size, different number of training clusters, and different hyper-parameters, etc. Without these information, it is hard to know how well robust the proposed DAC method can generalize.

5. Finally, the proposed method was built upon Lee at al 2019, to extend the previous method to a sequential clustering problem. I think a title "deep amortized clustering" is a bit misleading and exaggerated on the proposed method.
[Summary]
In this paper, the authors proposed a new method called Deep Amortized Clustering (DAC) for amortized clustering. Unlike the previous work Lee et al, the proposed DAC sequentially filter the data points from the whole set and construct the clusters gradually. This is a meaningful method in that it can be applied to those data without explicit number of clusters. However, the presentation of the method and experiments make it hard to follow, and thus hard to capture the contributions of the proposed method, and also its capacity. As also mentioned above, I would highly suggest the authors revise the paper so that it can present better the method and the experimental section.


Generative Adversarial Imitation Learning (GAIL) is a powerful and practical approach for learning sequential decision-making policies. Different from Reinforcement Learning (RL), GAIL takes advantage of demonstration data by experts (e.g., human), and learns both the policy and reward function of the unknown environment. Despite the significant empirical progresses, the theory behind GAIL is still largely unknown. The major difficulty comes from the underlying temporal dependency of the demonstration data and the minimax computational formulation of GAIL without convex-concave structure. To bridge such a gap between theory and practice, this paper investigates the theoretical properties of GAIL. Specifically, we show: (1) For GAIL with general reward parameterization, the generalization can be guaranteed as long as the class of the reward functions is properly controlled; (2) For GAIL, where the reward is parameterized as a reproducing kernel function, GAIL can be efficiently solved by stochastic first order optimization algorithms, which attain sublinear convergence to a stationary solution. To the best of our knowledge, these are the first results on statistical and computational guarantees of imitation learning with reward/policy function approximation. Numerical experiments are provided to support our analysis.

Summary: This paper presents theoretical analysis of the generalization properties of GAIL, as well as the local convergence of the traditional minibatch SGD applied to its min-max optimization problem (without assuming convex-concave structure of the game).  Specifically, the authors first prove a generalization bound for GAIL that characterizes how well optimizing the empirical GAIL objective minimizes the true population version (in terms of an "R-distance" they use the characterize the expected distributions induced by the underlying policies for a given reward class).  Second, they prove a convergence result for minibatch SGD applied to the min-max game, showing that the method will converge to a stationary point regardless of any convex-concave structure.
Comments: Before I begin, I should add that several elements of the paper were a bit difficult for me to follow, so I'm happy for the authors to correct any factual errors that I might make.  Overall, I think that this is an interesting, if somewhat incremental and technical paper about the generalization and convergence of GAIL.
First, on the generalization aspects, the methodology here seems to largely parallel Arora et al.'s analysis of generalization in GANs.  The main technical steps seem to be, 1) some effort in determining the proper distance to use in the first place, and how to define generalization, which they do via the R-distance, and then 2) on the more technical side, overcoming the fact that trajectories do not provide iid samples as is the case in the GAN analysis.  This later difficulty is overcome via the independent block technique, which allows one to bound the relevant population quantities of interest by sampling from subsampled blocks of the original trajectory.
Second, on the convergence side, the main result here seems to be a generic convergence result for minibatch SGD applied to a non-convex-concave game.  I may be missing something here, but it doesn't seem like there is any real specificity to the GAIL objective, but rather this would apply to any such min-max problem (I suppose also in the case where there is an L2 projection in the gradient step); while there is some discussion of the chain mixing properties, this seems largely needed just to provide bounds on certain constants.  The proof is rather technical (I admit I didn't go through it in much detail) and even the sketch doesn't provide a great intuition about what might make this problem harder than proving convergence of traditional SGD in the non-convex pure minimization case.  And for example, even convex-concave games have well-known pathologies when running gradient descent, such as cycling around an optimal point, and I didn't understand whether anything was being done to explicitly account for this, or if one of their assumptions essentially just avoids this possibility (maybe the mixing properties prevent this?).
On the whole, despite what seems to me to be a somewhat incremental nature, I'm still leaning toward accepting the paper: technical analysis like this is good to have, and the setting is distinct enough from past e.g., GAN work, that I believe it stands on its own.  I do think the clarity of the paper can be improved, as well.  This includes some simple elements like spacing out the equations better in Section 4, which were currently very condensed and difficult to read (and there is plenty of space to make the paper longer).  But I also would have liked a bit higher-level explanation of the challenges involved in proving convergence (like avoiding cycling), rather than just highlighting the functional elements of the proof.  I actually think the generalization section did this relatively well, in regards to the setting up their choice of IPM in contrast to the GAN work, but it was lacking in the convergence section.


We show that a critical problem in adversarial imitation from high-dimensional sensory data is the tendency of discriminator networks to distinguish agent and expert behaviour using task-irrelevant features beyond the control of the agent. We analyze this problem in detail and propose a solution as well as several baselines that outperform standard Generative Adversarial Imitation Learning (GAIL). Our proposed solution, Task-Relevant Adversarial Imitation Learning (TRAIL), uses a constrained optimization objective to overcome task-irrelevant features. Comprehensive experiments show that TRAIL can solve challenging manipulation tasks from pixels by imitating human operators, where other agents such as behaviour cloning (BC), standard GAIL, improved GAIL variants including our newly proposed baselines, and Deterministic Policy Gradients from Demonstrations (DPGfD) fail to find solutions, even when the other agents have access to task reward.

The paper addresses the acute problem of irrelevant discrimination in adversarial imitation learning methods. Ideally, the discriminator in such methods should extract task-dependent features and base its discrimination rule upon them only. However, in practice, the opposite usually occurs. For example, if the state distributions of the expert and agent come from different modalities, e.g., different background light, different POV camera, etc., then almost surely the discriminator will base its classification rule on such differences. In turn, the reward signal will be (constant and) irrelevant. This is a well-known problem, that indeed received little attention in previous work.
As a solution, the authors propose to maintain an invariance set representing task-irrelevant states. For example, states of a random policy or states from the beginning of an episode. The authors suggest using the invariance set as a regularization mechanism to the discriminator by punishing it for correctly classifying examples from the invariance set. The idea behind this regularization is to distinguish between expert examples that actually represent the task and expert examples that do not. This is achieved by punishing for the discriminator's accuracy over the invariance set.
In addition, the authors present two additional improvements: data augmentation and early actor stopping.
The paper.
1. I very much like the problem the authors choose to address. I believe that this is a dominant problem that received little attention so far. The reason is that so far, GAIL style methods used experts and agents that come from the same domains (same simulation), thus "hiding" such problems. As experiments scale up to real-world setups, this is no longer the case and such problems arise.
2. Overall, I find the experiments section very confusing. The authors propose a formal improvement (the invariance set) which they apply to their agent, as well as two informal improvements (actor early stopping and data augmentation) which they apply selectively for their agent and the baseline methods. As a result, there exist a large number of variations, according to what improvements are applied to what method. Overall, I find it hard to follow the results and understand the contribution of the different improvements.
3. It is not clear if TRAIL and GAIL are using the same architecture and algorithm. Apparently, the authors could have used GAIL's original policy gradient algorithm with their invariance set modification. If an improvement was evident in such a comparison, then it would have been easier for me to understand the significance of the proposed method. Currently, there are too many differences between the baseline and TRAIL to clearly say that the invariance set made the difference.
4. It is not clear why the constraint in Eq. 2 is placed over the sum of the two terms and not on each term separately since each term is defined over a different state distribution. What would you expect to see different if the constraint was placed on the two terms separately?
5. In the same matter, why do you need the 0.5 factor?
6. The invariance set is referred to extensively before it is first defined. Therefore, the reader is forced to toggle back and forth multiple times in section 4 before understanding equations 2 and 3. Consider giving at least an intuition about the invariance set beforehand to make the reading more fluent.
7. The conclusions section is minimal to non-existing while there are many conclusions to draw from the experiments.
8. Data augmentation was previously studied in the context of adversarial imitation. See for example "Visual imitation with reinforcement learning using recurrent Siamese networks" by Berseth et.al.
9. Overall assessment: among the three modifications the authors propose, I would be glad to see if the invariance set modification alone can make a difference. The other two modifications are either not novel (see Berseth et.al.), or highly heuristic ("...we restart an episode if the discriminator score at the current step exceeds the median score of the episode so far for T_patience consecutive steps..."). However, it seems that the informal improvements (augmentation and early stopping) play a crucial role in the success of TRAIL (... It drastically improves the baseline GAIL agent, and is necessary to solve any of the harder tasks..."). It is absolutely fine that the authors stress the importance of the informal improvements but it seems that they are dominating the contribution of the main ingredient.
I also find it problematic that the construction of the invariance set is not straightforward ("...the selection of the invariance set is a design choice..."). When added to the way that the data augmentation is carried (sensor dependent), and to the fact the the early stopping mechanism seems dubious for some tasks (I can think of several tasks where early stopping will not work), this method is far from being a "plug-and-play" solution. I.e., the paper offers general guidelines on how to solve imitation tasks but not a concise algorithm.
In order for me to vote for acceptance, I need the paper to be more concise. In its current state, I see it more as general guidelines for training adversarial imitation algorithms.
However, as said in the beginning, I very much believe in the problem that the authors present and encourage them to provide an experiment where the only modification between the baseline and the proposed method is the usage of an invariance set. Success in such an experiment will clearly change my mind.
There are also some presentation issues, e.g cluttered graphs, that can be addressed.


Predicting outcomes and planning interactions with the physical world are longstanding goals for machine learning. A variety of such tasks involves continuous physical systems, which can be described by partial differential equations (PDEs) with many degrees of freedom. Existing methods that aim to control the dynamics of such systems are typically limited to relatively short time frames or a small number of interaction parameters. We present a novel hierarchical predictorcorrector scheme which enables neural networks to learn to understand and control complex nonlinear physical systems over long time frames. We propose to split the problem into two distinct tasks: planning and controlling. To that end we introduce a predictor network that plans optimal trajectories and a controller network that infers the corresponding control parameters. Both stages are trained end-to-end using a differentiable PDF solver. We demonstrate that our method successfully develops an understanding of complex physical systems and learns to control them for tasks involving multiple PDEs, including the incompressible Navier-Stokes equations.

[Summary]
This paper proposes to combine deep learning and a differentiable PDE solver for understanding and controlling complex nonlinear physical systems over a long time horizon. The method introduces a predictor-corrector scheme, which employs a hierarchical structure that temporally divides the problem into more manageable subproblems, and uses models specialized in different time scales to solve the subproblems recursively.
For dividing the problem into subproblems, they use an observation predictor network to predict the optimal center point between two states. To scale the scheme to sequences of arbitrary length, the number of models scales with O(log N). For each subproblem, the authors propose to use a corrector network to estimate the control force to follow the planned trajectory as close as possible.
They have compared their method with several baselines and demonstrated that the proposed approach is both more effective and efficient in several challenging PDEs, including the incompressible Navier-Stokes equations.
[Major Comments]
Predicting the middle point between two states for modeling the dynamics via deep neural networks is not new, but I did not know any other works that use this idea for controlling PDEs.
I like the idea of splitting the control problem into a prediction and a correction phase, which leverages the power of deep neural networks and also incorporates our understanding of physics. The introduction of the hierarchical structure alleviates the problem of accumulating error in single-step forwarding models and significantly improves the efficiency of the proposed method. The videos for fluid control in the supplement materials also convincingly demonstrate the effectiveness of the technique.
I still have a few questions regarding the applicability and the presentation of the paper. Please see the following detailed comments.
[Detailed Comments]
In Section 3, the authors claim that their model "is conditioned only on these observables" and "does not have access to the full state." However, the model requires a differentiable PDE solver to provide the gradient of how interactions affect the outcome. These seem to contradict each other. Doesn't the solver require full-state information to predict the behavior of the system?
Related to the previous question, how can we make use of the differentiable PDE solver if we are uncertain or unknown of the underlying physics, i.e., partially observable scenarios.
The algorithm described in Section 5 seems to be the core contribution of this work. Instead of describing the algorithm in words, I think it would make it more clear if the authors can add an algorithm block in the main paper. It would also be better if the authors can include a few sentences describing the algorithm in the abstract to inform the readers of what to expect.
Figure 4 is a bit confusing, and it would be better if the authors can include the label for the x-axis. Besides, in the caption, the authors said that they show "the target state in blue." However, there are a lot of blue lines in the figure, and it is hard to know, at first glance, which one of them is the target.
In Table 1, the bottom two methods are using the same execution scheme and training loss, but the results are different. Is there a typo? Also, it would be better to bold the number that has the best performance.


Mixture Model (MM) is a probabilistic framework which allows us to define a dataset containing K different modes.

This paper proposes a modification of GANs where the latent space follows a distribution modelled by a Gaussian Mixture Model. While the idea of using GMMs in GANs is not novel, the main contribution of the paper is to add a classification models that enables posterior inference. The whole model is trained jointly end-to-end using both an adversarial loss and a mutual information loss. The procedure is then tested on MNIST, Fashion-MNIST and a subset of Oxford-102 Flower.
While the problem of posterior inference is interesting, the novelty of the paper is quite limited. The overall structure is clear, although writing can be improved.
However, a part from the form, I have other concerns about the paper:
- The main purpose of the paper is to tackle large image datasets that are hard to address by classical GMMs. This being said, the used datasets are composed of small and modal enough images that it seems hard to validate the claim of the paper using only these data. It seems to me that for the claim of the paper to be verified, larger scale/more complex datasets are needed.
- In all experiments, the authors suppose they have access to the number of classes/modes in the data, which is a huge assumption. It would be interesting to see if it would be possible to automatically accurately select the number of modes, e.g. on a held out validation set.
- One problem of GANs that the authors do not seem to consider is mode collapse. It would be interesting to do experiments with unbalanced datasets (e.g. MNIST 1 vs all) to see if the proposed architecture will model the data correctly.
- I am confused about the use of Mutual Information loss. The author claim that they would like to enforce each of the generated images to be from the same class as the input image. This would make sense if the authors used the multinomial sampling for the latent variable generation. However, the K generated samples are from K different Gaussians. It seems unreasonable to require of the classifier to render the same result.
- On the same note, in order to both use the classical multinomial sampling in GMMs and not break the backpropagation, have the authors considered updating the classifier and the GAN in separately in expectation-maximization fashion?
- Finally, I don't see the point of weighting the adversarial loss by the weights of Gaussians. All the generated images are fake and should be equally detected as such.
Although the idea is interesting, I think the paper, at its current status, not ready for publication.
Minor:
- P. 4: Generator: The sampling density from the multinomial distribution seems incorrect. However, as the authors skip the sampling step to be able to back propagate through the model, this is not significant.
- P. 1: The paragraph before the last: may even synthesizing -> synthesize
- P. 3: Architecture: Possibility -> Probability
- P. 7: Figure title: CIFR10 -> Oxford-102
- Algorithm notation:
    *Indexes for alpha and alpha_hat from 1 and not 0 to be consistent with the rest of the text
    *Add hats to the entries of alpha_hat
    *  LI from alpha_i and alpha_hat_i as in Equation 4 -> maybe change alpha_hat by x_hat to be consistent with the notation of eq.4 (although the meaning is clear here).


Stochastic AUC maximization has garnered an increasing interest due to better fit to imbalanced data classification. However, existing works are limited to stochastic AUC maximization with a linear predictive model, which restricts its predictive power when dealing with extremely complex data. In this paper, we consider stochastic AUC maximization problem with a deep neural network as the predictive model. Building on the saddle point reformulation of a surrogated loss of AUC, the problem can be cast into a non-convex concave min-max problem. The main contribution made in this paper is to make stochastic AUC maximization more practical for deep neural networks and big data with theoretical insights as well. In particular, we propose to explore Polyak-Łojasiewicz (PL) condition that has been proved and observed in deep learning, which enables us to develop new stochastic algorithms with even faster convergence rate and more practical step size scheme. An AdaGrad-style algorithm is also analyzed under the PL condition with adaptive convergence rate. Our experimental results demonstrate the effectiveness of the proposed algorithms.

This paper proposes two algorithms for the non-convex concave AUC maximization problem, along with theoretical analysis. Experiments show the proposed methods are effective, especially in data imbalanced scenarios.
Strengths:
This paper might be useful and interesting to related research, which overcomes some limitations in previous works such as: 1. the convex assumptions; 2. only considering simple models like linear models; 3. the need of extra memory to store/maintain samples. The proposed method extends existing works to a non-convex setting, which can be applied to deep neural networks, and is applicable for batch-learning and online learning.
The proposed methods achieve better experimental results, especially in the data imbalanced scenarios, which is a real problem that may arise in many scenarios. The paper provides theoretical analysis on the proposed methods, based on Assumption 1, and inspired by the PL condition.
Weaknesses:
I think some comparisons with AdaGrad and related methods should be performed in experiments. Since PPD-AdaGrad is “AdaGrad style”.
The assumptions seem a bit unclear. What does the first assumption in Assumption 1 imply?
Minor Comments:
1. Since the experiments label the first 5/50 classes as negative, and the last 5/50 classes as positive for CIFAR10/CIFAR100, is it possible to provide results on experiments that label in the opposite way (or randomly label 5/50 classes) and add these results in the paper/appendix? Just to make results more convincing and reduce some potential dataset influences.
2. Is it possible to provide some results on more imbalanced positive-negative ratio like 20:1?
3. Is it possible to provide some comparison in terms of actual time, like learning curves with time as x-axis?
4. In the multi-class problems, why are the lower layers shared while last layer separated?
5. Since the extension to multi-classes problems are mentioned in the paper. I like to see some experimental results on this setting.
6. How do the proposed methods perform on models other than NN?
7. I think there is a typo on Page 4, the definition of AUC definition: the latter y should be -1.


Meta-learning over a set of distributions can be interpreted as learning different types of parameters corresponding to short-term vs long-term aspects of the mechanisms underlying the generation of data. These are respectively captured by quickly-changing parameters and slowly-changing meta-parameters. We present a new framework for meta-learning causal models where the relationship between each variable and its parents is modeled by a neural network, modulated by structural meta-parameters which capture the overall topology of a directed graphical model. Our approach avoids a discrete search over models in favour of a continuous optimization procedure. We study a setting where interventional distributions are induced as a result of a random intervention on a single unknown variable of an unknown ground truth causal model, and the observations arising after such an intervention constitute one meta-example. To disentangle the slow-changing aspects of each conditional from the fast-changing adaptations to each intervention, we parametrize the neural network into fast parameters and slow meta-parameters. We introduce a meta-learning objective that favours solutions robust to frequent but sparse interventional distribution change, and which generalize well to previously unseen interventions. Optimizing this objective is shown experimentally to recover the structure of the causal graph. Finally, we find that when the learner is unaware of the intervention variable, it is able to infer that information, improving results further and focusing the parameter and meta-parameter updates where needed.

This paper proposes an SCM-model based on masked neural networks to capture arbitrary conditional relationships combined with meta-learning-style adaptation to reflect the effects of various unknown interventions. Overall the paper is well written and easy to follow, but some conceptual issues remain.
- How come there is hardly any discussion of the identifiability issue beyond the few sentences in A.3. This is one of the key issues in learning SCMs and it is strange that the concept of "faithfulness" is not even mentioned in the paper.
In general, there is hardly any discussion of what conditions are required for the proposed estimates to even be valid. The authors seem to be optimistically assuming that their neural network + metalearning model will somehow pick up on the correct structure, without any actual conceptual investigation of this issue.
- The massive downside of neural nets is all the various hyperparameters one has to set (eg. architecture, optimizer, activations, etc). In this setting, how do the authors propose selecting hyperparameter values? How does the reader know the authors did not simply tune their hyperparameters to best match the underlying ground truth (I assume the proposed methodology has many more hyperparameters and thus more degrees of freedom here).
I would like to see the empirical performance of different variants of your model with different hyperparameter values to assess its sensitivity to these choices.
- Why does one even care about the graph being acylic in this setting?
The mere fact that the authors require a regularizer to ensure acylicity suggests this approach is prone to mis-identifying the ground truth structure (which is always acyclic in the experiments).
- One main reason for SCM modeling in science and policy-making is for analysts to better understand the data generating phenomena.  However your use of neural networks here seems to hamper interpretability, so how do you reconcile this issue? Also is your sparsity regularizer satisfactory to confidently diagnose presence/absence of an edge (in constrast to statistical hypothesis tests, say based on conditional independence). Isn't this heavily influenced by the particular sparsity-regularizer value that happened to be selected?
- Related papers that utilize the same idea of predicting a variable conditioned on subset of other variables via neural network + masking strategy:
Ivanov et al (2019). VARIATIONAL AUTOENCODER WITH ARBITRARY CONDITIONING.
https://openreview.net/pdf?id=SyxtJh0qYm
Li et al (2019). Flow Models for Arbitrary Conditional Likelihoods.
https://arxiv.org/abs/1909.06319
Yoon et al. GAIN: Missing data imputation using generative adversarial nets. Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, 2018. http://proceedings.mlr.press/v80/yoon18a.html
Douglas et al. A universal marginalizer for amortized inference in generative
models. arXiv preprint arXiv:1711.00695, 2017
For clarity, the authors should highlight the differences of their approach from these works (beyond the causal setting).
- Given the lack of theoretical / conceptual guarantees that the methodology will work, our faith in the proposed methodology rests entirely on the empirical experiments.  However, I find these a bit too basic to be very convincing, and would at least like to see more methods being compared (in particular for the simulated graphs as well).
- The authors should describe what are the underlying interventions in each dataset a bit more.
- The Figures should be better explained (took me a while to figure out what dots/colors represent).
- Why do the authors report cross entropy loss in Table 1? To my knowledge this is not a standard metric for measuring the quality of structure-estimates.

- Instead of ICP (which is constrained to be linear which is unrealistically simple), why don't the authors compare against nonlinearICP (which is more flexible like their neural networks):
Heinze-Deml et al (2018). Invariant Causal Prediction for Nonlinear Models. https://arxiv.org/pdf/1706.08576.pdf


We introduce the 2-simplicial Transformer, an extension of the Transformer which includes a form of higher-dimensional attention generalising the dot-product attention, and uses this attention to update entity representations with tensor products of value vectors. We show that this architecture is a useful inductive bias for logical reasoning in the context of deep reinforcement learning.

CONTRIBUTIONS:
C1. Simplicialization of attention. Interpreting standard attention weights of a head as the model’s estimate of the probability of an edge = 1-simplex linking the variables encoded by 2 blocks of the Transformer, representing that the blocks stand in a binary relation encoded in the head, a generalization to 2-simplexes is made: now attention also estimates the probability of a 2-simplex indicating that three blocks stand in an arity-3 relation.
C2. 2-simplicial attention. The standard query-key matching function, the scalar (dot) product, is related to the area of the 2-simplex determined by the 2 vectors and the origin, and this is generalized to the (unsigned) scalar triple product <a,b,c>, analogously related to the volume of the 3-simplex determined by 3 vectors and the origin. This now serves as the matching function between a query and 2 keys. Each head in each block generates a value vector u and two key vectors k1, k2 and the weight of attention from block(i) (with query p(i)) to the ordered pair (block(j), block(k)), a(i,j,k), is a softmax over <p(i), k1(j), k2(k)>. Attention returns to block(i) a sum in which a(I,j,k) weights B(u(j)*u(k)), with B a learned linear map and * the tensor product.
C3. Experimental results applying Transformers T1 (with standard 1-simplicial attention) and T2 (with new 2-simplicial attention) to modeling an agent in bridge BoxWorld, trained with deep RL. This game crucially involves 3-way entity interactions, as keys of 2 colors open a box yielding a key of a 3rd color. T2 learns significantly faster than T1 (in the sense that the 1-standard-deviation-neighborhood of the learning curve of T2 becomes better than that of T1, plotted against environmental steps: Fig.4, and also essentially so when plotted against time adjusted steps: Fig 5).
RATING: Accept
REASONS FOR RATING (SUMMARY). Generalizing attention from 2nd- to 3rd-order relations is an important upgrade, and the mathematical context in which this is derived is insightful and may lead to further progress in the development of Transformers capable of constructing still richer structures. The experiments yield clear evidence of the value of 3rd-order attention in the context of a game designed to highlight 3rd-order relations.
Strengths
The exposition is clear and situated in a rather sophisticated formal setting. The connection to Clifford algebras may yield further fruit, besides the scalar triple product that is crucial to the definition of 2-simplicial attention. Although I am not an expert in RL, the experiments reported seem sound and the results clear. I believe that the strength of the paper justifies its length of 8.5 pages: the exposition of the key ideas, for this reader, hits a sweet spot between overly concise and overly verbose, and the ideas call for the quantity of space devoted to them. The decisions of what material to place in the Appendix seem well made. Although I have not studied the entire (13-page) Appendix, what I have read is clear and enlightening, another major contribution of the paper.
Weaknesses
Future work testing the value of 3rd-order attention in tasks that are less clearly perfectly designed for it will substantially strengthen the case for it. But in my view it is right to start testing a new architecture by showing it can indeed do what it is designed to do; further tests showing that what it is designed to do is of general utility are a second step. In this case, the first step, including creating of the model itself (consuming 5 non-verbose pages), is substantial enough to warrant publication.
In my good-quality printout of the paper, I can’t see curves for best runs in Fig. 4.


Machine learning systems often encounter Out-of-Distribution (OoD) errors when dealing with testing data coming from a different distribution from the one used for training. With their growing use in critical applications, it becomes important to develop systems that are able to accurately quantify its predictive uncertainty and screen out these anomalous inputs. However, unlike standard learning tasks, there is currently no well established guiding principle for designing architectures that can accurately quantify uncertainty. Moreover, commonly used OoD detection approaches are prone to errors and even sometimes assign higher likelihoods to OoD samples. To address these problems, we first seek to identify guiding principles for designing uncertainty-aware architectures, by proposing Neural Architecture Distribution Search (NADS). Unlike standard neural architecture search methods which seek for a single best performing architecture, NADS searches for a distribution of architectures that perform well on a given task, allowing us to identify building blocks common among all uncertainty aware architectures. With this formulation, we are able to optimize a stochastic outlier detection objective and construct an ensemble of models to perform OoD detection. We perform multiple OoD detection experiments and observe that our NADS performs favorably compared to state-of-the-art OoD detection methods.

This paper needs crucial, quick fixes.
This paper throws neural architecture search at the problem of out-of-distribution detection. Rather than searching over multi-class classifier architectures, and using the result for OOD detection, they instead search for generative model architectures.
The approach appears to work on some of their cherry-picked OOD datasets.
I give this paper a 3 because they are quite possibly only showing their strongest results and not giving a complete picture, and there are numerous small errors throughout the paper.
After a more thorough evaluation, the technique will likely not look strong on datasets such as CIFAR-10 vs CIFAR-100.
This is acceptable since they are comparing density estimators vs multi-class OOD detectors, the latter of which has been vastly superior for many years.
Their technique brings density estimators within striking distance of multi-class classifier perofrmance, but the paper must give a complete picture. If these issues are fixed, the paper is easily a 6 or an 8, depending on the results of currently unshown OOD datasets. It is acceptable if their technique gets and 55% AUROC for CIFAR-10 vs CIFAR-100 while multi-class classifiers with Outlier Exposure get 96%. It is OK because this technique appears superior to other generative models, which have lagged far behind. Currently the paper leaves the impression that this is not only leapfrogging past previous density estimators, but that it also is beating multi-class classifiers. This likely isn't true. We need to see performance on other OOD datasets.
I am willing to increase my score from a 6 to an 8 even if the results are negative.
The CIFAR-10 model should have to detect OOD samples from CIFAR-100, Rademacher/Bernoulli noise, and Gaussian noise. In addition, they should train a model on CIFAR-100 since many OOD detection techniques exhibit much worse behavior on CIFAR-100, compared to CIFAR-10 or MNIST or SVHN.
In summary, I give this a 3 due to critical flaws, but if these are rectified, the paper will likely deserve a 6 or 8.
Miscellaneous Points:
Please include CIFAR-10/100 code. Currently the code is for MNIST.
There are several errors in discussing related work.
> Moreover, previous work on deep uncertainty quantification shows that a single model may not suffice to quantify uncertainty and detect OoD samples (Lakshminarayanan et al., 2017; Choi & Jang, 2018)
Ensembles do not perform appreciably better at OOD detection under systematic OOD benchmarking when using multi-class classifiers. For generative models, ensembles can help. Ensembles mainly help multi-class classifier _calibration_ on in-distribution in-class data, but they have miniscule to nonexistent utility in classifier OOD detection. Please add appropriate qualifiers.
> "With no access to OoD data, unsupervised/self-supervised generative models which maximize the likelihood of in-distribution data become the primary tools for uncertainty quantification."
I think they mean "with access to labels." Multi-class classifiers are the most performant tool for OOD detection, and unsupervised generative models are around chance-levels. _Using Self-Supervised Learning Can Improve Model Robustness and Uncertainty_ (NeurIPS) uses self-supervised learning for OOD detection and achieves good performance, but this work is not cited.
> "However, these models counter-intuitively assign high likelihoods to OoD data (Nalisnick et al., 2019a; Choi & Jang, 2018)"
This was shown in previous work, such as Shafaei et al. 2019 and Hendrycks et al. 2019, and hence deserve mention.
> "Moreover, existing methods to calibrate model uncertainty estimates assume access to OoD data during training (Lee et al., 2018; Hendrycks et al., 2019). This is flawed when anomalous data is rare or not known ahead of time"
Lee et al. does not assume access to OOD data during training; they synthesize their own. Hendrycks et al. does assume access to OOD data, but not OOD data seen during evaluation; in this way, they do not assume data is "known ahead of time," which they reiterate throughout their paper. This sentence is painting with too broad a brush.
There is a smaller experimental problem. They compare to ODIN, but ODIN assumes access to OOD data from the test distribution. While this assumption is clearly questionable, their evaluation does not use the assumption they did, so their technique is not actually ODIN. I suggest just comparing against the Maximum Softmax Probability Baseline in Table 1 since the rest of this paper assumes OOD examples are not known ahead of time.
Update: I have changed my score to an 8.


Randomized smoothing, which was recently proved to be a certified defensive technique, has received considerable attention due to its scalability to large datasets and neural networks. However, several important questions still remain unanswered in the existing frameworks, such as (i) whether Gaussian mechanism is an optimal choice for certifying 2 -normed robustness, and (ii) whether randomized smoothing can certify ∞ -normed robustness (on high-dimensional datasets like ImageNet). To answer these questions, we introduce a unified and selfcontained framework to study randomized smoothing-based certified defenses, where we mainly focus on the two most popular norms in adversarial machine learning, i.e., 2 and ∞ norm. We answer the above two questions by first demonstrating that Gaussian mechanism and Exponential mechanism are the (near) optimal options to certify the 2 and ∞ -normed robustness. We further show that the largest ∞ radius certified by randomized smoothing is upper bounded by
where d is the dimensionality of the data. This theoretical finding suggests that certifying ∞ -normed robustness by randomized smoothing may not be scalable to high-dimensional data. The veracity of our framework and analysis is verified by extensive evaluations on CIFAR10 and ImageNet.

Summary.
The authors propose a new definition for robustness of random functions. This definition is ideal for analyzing the certified robustness under randomized smoothing techniques. They analyze and show that the Gaussian smoothing is near optimal for \ell_2 smoothing as the mean maximum error is only off by a factor of log d where d is the dimension from the optimal mean maximum energy. This is the case even under a more strict definition of robustness defined as D_\infty. Moreover, the authors show that indeed smoothing with an exponential family  is optimal under D_\infty robustness metric with radius measured in \ell_\infty.
I find the paper very interesting and the approach is novel and generic. I do not have any major criticism.
Minor comments.
1) Equation 3 "D(A(x'),A(x))" >> "D_\infty(A(x'),A(x))"
2) Page 6 third line below Theorem 16. Reference of Theorem 11 should be Corollary 11.
3) The authors should report the certified accuracy of the undefended baseline classifier over varying radius in Figures 1 and 2 and 3.
4) Running experiments on ImageNet following Cohen et al. should make the paper stronger.
4) Can the authors comment on is the certified accuracy for \sigma=0.5 at radius = 0 is better than the unsmoothned classifier a sigma 1.0. I expect that the radius of certification is larger for larger sigma.
5) The authors should explain how does the new definition of robustness relate to the common robustness definitions as the one by Cohen et al.  More discussion is necessary for this and more justification.
6) Why is the D_MR defined as maximum over $\alpha$? It seems it is only sufficient to define it as the ratio over $\alpha$. It seems that this is only needed for Theorem 8 to hold.
------------------------------------------------------------------------
After further careful read of several relevant papers, e.g. Bun et. al 2016 and the work of Dwork "Concentrated Differential Privacy", I have several questions I would like to ask for some further clarifications.
1) Showing that a network is robust under $D_{\infty}$ robustness, implies very strong results. The type of results that are common in the literature. This is since $D_{\infty}$ robustness, implies $\epsilon$ DP networks (see Lemma 3.2 and proposition 3.3 of Bun et al.). Once $\epsilon$ DP is guaranteed identical results of Lecurer et al. can be derived immediately as this implies separation in expectation (Lecurer et al.) where one can study directly the deterministic classifier $\mathbb{E}g(\mathbf{x})$ and not the random $g$ studied in this work.
2) The authors rely on the lower bounds of Bun et al. to find the average maximum energy that preserves the $D_{\infty}$ robustness (Thm 15 and 16). Authors show that indeed exponential smoothing is optimal. This is significant but the analysis was intensively based on Bun et al.
3) The relaxation to $D_{MR}$ robustness results into improvement of the dependency on the dimension to $\sqrt{d}$ instead of $d$ for under $\ell_\infty$. This should not be surprising at all and in fact is identical to the results of Bun et al. Note that the zCDP proposed by Bun et al, is a relaxed version of DP where $\epsilon$-DP for some radius $r$ implies zCDP with radius $r^2$. See proposition 3.3. Therefore, Theorem 6 and 17 are not surprising nor are they new.
4) My major concern was with the results relating to Gaussian smoothing. I do understand that since Gaussian smoothing only implies high probability result of DP which is often referred to as ($\epsilon$,$\delta$)-DP which happens to be a equivalent to zCDP proposed by Bun et. al. Therefore, I have no issues of using $D_{MR}$ to analyzing the robustness for Gaussian smoothing since it was always analyzed in the DP community with the $\epsilon,\delta$-DP and not the stronger $\epsilon$-DP. However, the statement of the result (Theorem 12) confused me vastly. Let me clarify.
Theorem 12 seems to be too good to be true. How is it possible that one can guarantee $D_{MR}$ robustness without any dimensionality dependence. Using Gaussian smoothing the $D_{MR}$ can depend on $\sqrt{\log{d}}$. While $\sqrt{\log{d}}$ may seem small; improving this to a constant in dimension is still a very big gap from $\sqrt{\log{d}}$. This may raise several questions whether one can actually find this optimal smoothing distribution. However, with a careful read of Theorem 12, the range of the input decreases as a function of $\sqrt{d}$. That is for a given range of input (independent from d), the energy in fact is NOT constant but scales with $\sqrt{d}$. In such a case, the Gaussian smoothing is now of order $\sqrt{d \log{d}}$. Now, the factor is still $\log{d}$, but now this is very different as indeed improving the Gaussian to $\sqrt{d}$ may not be of significant interest as the energy still depends in the optimal sense on $\sqrt{d}$ which does not allow it to scale for larger problems. Moreover, Cohen et al results show that with Gaussian smoothing the energy of the noise scales $\sqrt{d}$ since the noise energy $\|n\| = \mathcal{O}(\sqrt{d} \sigma)$ where $\sigma$ is std of Gaussian. Therefore, it seems that there is nothing surprising about such a result at all. The statement of the Theorem is very misleading and confusing.
Overall, I like this new approach of analyzing the random smoothed classifier; however, the poor presentation of the work and the mis-represented Theorems that seem to over claim are a major reason for my rating. In addition, the paper should be self-contained in which one should not need to read 2-3 other works to figure out the details in this work and the meaning of the several robustness metrics and their direct relations to DP and Lecuer et al. results. The statement of constant in dimension lower bound on the energy of the noise under $D\_{MR}$ was to me the major contribution; however, I found now that the statement is misleading and that in fact it is $\sqrt{d}$ reduces the contribution of the paper particularly after learning that such lower bounds are already derived in Bun et al.


We study the landscape of squared loss in neural networks with one-hidden layer and ReLU activation functions. Let m and d be the widths of hidden and input layers, respectively. We show that there exist poor local minima with positive curvature for some training sets of size n ≥ m + 2d − 2. By positive curvature of a local minimum, we mean that within a small neighborhood the loss function is strictly increasing in all directions. Consequently, for such training sets, there are initialization of weights from which there is no descent path to global optima. It is known that for n ≤ m, there always exist descent paths to global optima from all initial weights. In this perspective, our results provide a somewhat sharp characterization of the over-parameterization required for "existence of descent paths" in the loss landscape.

This paper studies the landscape properties of over-parameterized two-layer neural networks, and proposes a network width lower bound for guaranteed existence of descent paths that is tighter than existing results. In particular, the authors prove that if the network width m \leq n - 2d, then there exist training data sets and initial weights such that the square loss on the neural network has no descent path connecting the initial weights and global minima.
Overall, I think this paper is of good quality. The presentation is clear and the logic is easy to follow. I did some high-level check and the theoretical analysis seems reasonable. However, I have the following questions:
1. As is stated in Corollary 1 and discussed below Theorem 1, the training sample sets that lead to suboptimal capped minima are not of measure-zero. However, it seems that no rigorous proof is provided for this result. Perhaps I have missed something, but is Corollary 1 straight forward given Theorem 1 and Lemma 1? How large is the probability? It would be better if the authors could provide a more rigorous proof.
2. A recent line of work has shown convergence of gradient-based algorithms for over-parameterized neural networks. It would be interesting if the authors could provide more comparison between this paper and the results of these works:
- Simon S. Du, Jason D. Lee, Haochuan Li, Liwei Wang, Xiyu Zhai, Gradient Descent Finds Global Minima of Deep Neural Networks
- Zeyuan Allen-Zhu, Yuanzhi Li, Zhao Song, A Convergence Theory for Deep Learning via Over-Parameterization
- Difan Zou, Yuan Cao, Dongruo Zhou, Quanquan Gu, Stochastic Gradient Descent Optimizes Over-parameterized Deep ReLU Networks


A robustness certificate against adversarial examples is the minimum distance of a given input to the decision boundary of the classifier (or its lower bound). For any perturbation of the input with a magnitude smaller than the certificate value, the classification output will provably remain unchanged. Computing exact robustness certificates for deep classifiers is difficult in general since it requires solving a nonconvex optimization. In this paper, we provide computationally-efficient robustness certificates for deep classifiers with differentiable activation functions in two steps. First, we show that if the eigenvalues of the Hessian of the network (curvatures of the network) are bounded, we can compute a robustness certificate in the l 2 norm efficiently using convex optimization. Second, we derive a computationallyefficient differentiable upper bound on the curvature of a deep network. We also use the curvature bound as a regularization term during the training of the network to boost its certified robustness against adversarial examples. Putting these results together leads to our proposed Curvature-based Robustness Certificate (CRC) and Curvature-based Robust Training (CRT). Our numerical results show that CRC outperforms CROWN's certificate when trained with our regularizer while CRT leads to higher certified accuracy compared to standard adversarial training.

This paper gives a global curvature bound for general neural networks, and used this bound for certifying robustness of neural networks. The basic techniques include writing the neural network verification problem as a constrained optimization problem (similar to [1]), and construct a Lagrangian dual for it, and discuss the situation where the dual problem is convex and can be solved efficiently. The global curvature bound is the crucial step to determine the condition that the optimization problem is convex.
The benefits of the proposed methods include efficiency (bounds m and M only need to be solved for each target label pair, and the optimization process to get the certificate is fast), and tight certified accuracy for shallow networks. Also, it can be used during training to improve the tightness of CRC bounds.  There are also a few weakness: the certificate unfortunately only works for non-ReLU networks, and only for L2 norm; for networks beyond 2 layers, CRC cannot outperform CROWN unless it is specially trained using the CRC loss.
Questions and concerns:
1. For the curvature regularization training, some comparisons against other certified defenses are needed - PGD and TRADES are not good examples since they are not certified defense methods, and 0% certified accuracy are obtained. I understand many existing works like Wong et al., 2018 only work on ReLU networks, but I believe interval bound propagation (IBP) based methods [2] should be easily applicable to any monotonic activation functions. At least, the authors should compare verified accuracy at the same epsilon settings (\pho=1.58) as in Wong et al., 2018.  In page 22 (last page of appendix in arxiv version) of [3], you can find their L2 robustness training results.  How does the certified accuracy of CRC compare to these results?
2. The paper mentions the Attack problem and proposes an attack algorithm (Algorithm 2), however I am not able to find any experiments on the attack. If the authors want to claim the contribution of curvature based attacks, some empirical results should be given, and at least compare it to a 200-step PGD baseline and see which one finds smaller adversarial examples.
3. The claim (in introduction and conclusions) that CRC is much faster than CROWN probably won't hold under a fair comparison. I believe the gradient decent based certification algorithm (Algorithm 1) was computed efficiently on GPU, where the NN is defined. For a fair comparison, it should be also compared to a GPU implementation of CROWN ([4] provides such an implementation).  In my experience, CROWN is an efficient algorithm that can be even used iteratively during training (as a certified defense [4]), so for the small networks used in this paper, it should compute bounds almost instantly on GPUs. I think it is better to revoke this claim, and in experiments the authors should clearly state that CROWN was computed on CPU so time is not comparable.
4. Also the claim "CRC outperforms CROWN's certificate significantly" should be made clearer that it only holds for CRC trained models. According to Table 3, CRC is significantly worse than CROWN if the model is not CRC trained.  Additionally, it should be made clear that the proposed method currently only applies to L2 norm setting.
5. (Minor) There are several duplicated references, e.g., on page 9, Cohen et al., (randomized smoothing) was cited twice as two different papers; the same is with Madry et al., on page 10, and Zhang et al. (CROWN) on page 11. These causes some confusions, e.g., on page 2, when talking about the bounded Lipschitz constant, I believe the correct citation for (Zhang et al., 2018b) should be another paper [5] from the same first author which is on bounding Lipschitz constant.
Improvements and extensions:
I think the proposed method can be greatly improved by using "local curvature", where the curvature bounds m and M are computed within a local region near an input point. This is sufficient as long as our optimization does not escape this safe region, and the safe region can be naturally defined as the perturbation radius to be certified. The local curvature can be obtained by giving a tighter bounds on the second derivative of activation function, rather than considering just the worst case. Using CROWN, we can obtain pre-activation upper and lower bounds. These bounds can be used to bound the second derivative. For example, if a tanh neuron's input is bounded by -0.1 and 0.1 (those bounds can be obtained efficiently by CROWN), the second derivative of tanh is bounded between -0.197356 and +0.197356, much better than the worst case bound -0.76981 and +0.76981 used in current global curvature bound. A similar technique to bound Jacobian matrix was used in [5]. Including some results for local curvature certificates will greatly increase the contribution of this paper, and make it a complete work. I strongly encourage the authors to do so, and feel free to discuss with me on any questions.
Despite some concerns, the main contribution of giving global curvature bounds of neural networks is valid. Thus I vote for accepting this paper, however the authors should make sure to address all the concerns.
[1] Salman, Hadi, et al. "A convex relaxation barrier to tight robust verification of neural networks." arXiv preprint arXiv:1902.08722 (2019).
[2] Gowal, Sven, et al. "On the effectiveness of interval bound propagation for training verifiably robust models." arXiv preprint arXiv:1810.12715 (2018).
[3] Wong, Eric, et al. "Scaling provable adversarial defenses." Advances in Neural Information Processing Systems. 2018. https://arxiv.org/pdf/1805.12514.pdf
[4] Zhang, Huan, et al. "Towards Stable and Efficient Training of Verifiably Robust Neural Networks." arXiv preprint arXiv:1906.06316 (2019).
[5] Zhang, H., Zhang, P., & Hsieh, C. J. Recurjac: An efficient recursive algorithm for bounding jacobian matrix of neural networks and its applications. arXiv preprint arXiv:1810.11783 (2018).


Natural intelligent agents learn to perceive the three dimensional structure of the world without training on large datasets and are unlikely to have the precise equations of projective geometry hard-wired in the brain. Such skill would also be valuable to artificial systems in order to avoid the expensive collection of labeled datasets, as well as tedious tuning required by methods based on multi-view geometry. Inspired by natural agents, who interact with the environment via visual and haptic feedback, this paper explores a new approach to learning depth from images and very sparse depth measurements, just a few pixels per image. To learn from such extremely sparse supervision, we introduce an appropriate inductive bias by designing a specialized global-local network architecture. Experiments on several datasets show that the proposed model can learn monocular dense depth estimation when trained with very sparse ground truth, even a single pixel per image. Moreover, we find that the global parameters extracted by the network are predictive of the metric agent motion.

This paper presents a method for learning to predict a depth map given a pair of images. Training is done is an unsupervised way except for a small set of image locations for which the absolute 3D locations are known. The optical flow between the 2 images is computed automatically. A network is trained to predict the camera motion between the 2 images from the images and the optical flow. From this global motion, convolutional filters are predicted, in order to transform the optical flow into a depth map.
The use of image locations with known 3D locations is motivated in the paper to "simulate" the knowledge of depth coming from a haptic system.
I have several concerns about this paper.
* My main concern is that the paper compares against only a few papers (1 from 2014, 1 from  2016, and one more recent from 2019) while the literature is extremely vast, and visually, the results seem far from the state-of-the art. See for example:
Huangying Zhan, Ravi Garg, Chamara Saroj Weerasekera, Kejie Li, Harsh Agarwal, and Ian Reid. Unsupervised learning of monocular depth estimation and visual odometry with deep feature reconstruction. In CVPR, 2018.
Anurag Ranjan, Varun Jampani, Kihwan Kim, Deqing Sun, Jonas Wulff, and Michael J Black. Competitive collaboration: Joint unsupervised learning of depth, camera motion, optical flow and motion segmentation. In CVPR, 2019.
Chaoyang Wang, Jose Miguel Buenaposada, Rui Zhu, and Simon Lucey. Learning depth from monocular videos using
direct methods. In CVPR, 2018.
In fact, these papers have simpler requirements as they do not need 2 images at run-time, nor 3d data at training time. I thought for a moment that the advantage of the paper was to be able to predict an absolute motion and an absolute depth (this is not impossible if the method is able to estimate the scale from known objects, as it is suggested from the introduction). This was however incorrect as the text says in the middle of Section 4.1 " we resolve the inherent scale ambiguity by normalizing the depth values such that the norm of the translation vector between the two views is equal to 1".
* The network used to predict optical flow was trained with a large amount of supervised data. As image matching is the most difficult task, it is difficult to claim that the method is unsupervised.
* The motivation for having sparse measurements is to "simulate" haptic. This is fine for me in principle, however haptic measurements would probably have large errors, while it seems that the experiments use ground truth values for these points.
One minor remark:  End of Section  4.4: What is a "179% error reduction"? How is computed the percentage?


Inductive and unsupervised graph learning is a critical technique for predictive or information retrieval tasks where label information is difficult to obtain. It is also challenging to make graph learning inductive and unsupervised at the same time, as learning processes guided by reconstruction error based loss functions inevitably demand graph similarity evaluation that is usually computationally intractable. In this paper, we propose a general framework SEED (Sampling, Encoding, and Embedding Distributions) for inductive and unsupervised representation learning on graph structured objects. Instead of directly dealing with the computational challenges raised by graph similarity evaluation, given an input graph, the SEED framework samples a number of subgraphs whose reconstruction errors could be efficiently evaluated, encodes the subgraph samples into a collection of subgraph vectors, and employs the embedding of the subgraph vector distribution as the output vector representation for the input graph. By theoretical analysis, we demonstrate the close connection between SEED and graph isomorphism. Using public benchmark datasets, our empirical study suggests the proposed SEED framework is able to achieve up to 10% improvement, compared with competitive baseline methods.

The authors propose a method for learning graph embeddings and focus specifically on a setting where not all graphs are part of the training data (the inductive setting). The core problem of graph embedding methods is to find a learnable function that maps arbitrary graphs into a fixed-sized vector representation. There have been several proposals ranging from the class of graph kernels to variations of graph neural networks. The authors propose a method that consists of three steps
(1) sample a number of subgraphs from the original graphs
(2) learn an encoding function for these subgraphs (subgraph -> vector representation)
(3) for every graph we, therefore, get a set of vector representations, one per subgraph. We now try to find a similarity measure operating on sets of vectors to compute the distance between graphs.
The novel bits are
(a) the way that the subgraphs are sampled (using an algorithm called WEAVE, that stores more information about random walks) and, therefore, is able to be distinguish graphs based on the extracted walks that standard random walk based methods cannot; and
(b) to define a similarity measure based on the set of vectors.
The authors also prove that their method is (under some assumptions) able to decide the isomorphism problem. This is a nice result to have in light of recent papers that have investigated the limitations of GNN in comparison to Weisfeiler-Leman and isomoprhism testing.
Unfortunately, the proposed method has limited novelty. The WEAVE sampling is a small variation on random walk sampling that's been around for a while in graph representation learning. Also, to define the similarity between set of vectors has been addressed before in numerous papers (e.g., all papers investigating learning for sets, DeepSets, etc.) and the method here seems a bit ad-hoc and doesn't compare to existing work.
My "novelty" critique is also made in light of the small number of datasets on which experiments have been conducted. If a new simple random walk strategy would lead to clearly better results on a number of datasets, this would be a significant contribution. As far as I can tell, however, the results are mixed and not very impressive especially due to the small number of datasets.


Meta learning has been making impressive progress for fast model adaptation. However, limited work has been done on learning fast uncertainty adaption for Bayesian modeling. In this paper, we propose to achieve the goal by placing meta learning on the space of probability measures, inducing the concept of meta sampling for fast uncertainty adaption. Specifically, we propose a Bayesian meta sampling framework consisting of two main components: a meta sampler and a sample adapter. The meta sampler is constructed by adopting a neural-inverse-autoregressive-flow (NIAF) structure, a variant of the recently proposed neural autoregressive flows, to efficiently generate meta samples to be adapted. The sample adapter then pushes meta samples to task-specific samples, based on a newly proposed and general Bayesian sampling technique, called optimal-transport Bayesian sampling. The combination of the two components allows a simple learning procedure for the meta sampler to be developed, which can be efficiently optimized via standard back-propagation. Extensive experimental results demonstrate the efficiency and effectiveness of the proposed framework, obtaining better sample quality and faster uncertainty adaption compared to related methods.

Summary of paper:
The authors propose a neural sampler for probabilistic models in the meta-learning setting.
Their main claim is that their model captures uncertainty in samples better than competing methods and does so at lower cost.
In particular, they propose a scheme which separates sampling variables for a task tau into two components: a meta-sampler and a sample adapter.
The meta-sampler intuitively plays the role of  a learned conditional distribution over the target variables.
The sample adapter is a sampler which is seeded with samples from the meta-sampler and moves them towards the desired data-distribution based on a technique called optimal-transport Bayesian sampling.
Crucially, the meta-sampler is based on neural inverse autoregressive flows to have adequate representational capacity.
The authors use the proposed algorithm (denoted DAMS for distribution agnostic meta sampling) for a variety of tasks:
First, parameters for logistic regression models are generated and evaluated on various UCI tasks.
Second, meta-learning over Gaussian mixture models (GMMs) is performed.
Third, posterior adaption in a toy regression task where the frequency parameter of a sine wave is estimated.
Last, the authors train neural networks with DAMS and test both classification accuracy as a function of test-time inference for test datasets of held out classes on cifar10, mnist and a few-shot training example on Mini-Imagenet, while also testing their method on meta-reinforcement learning Mujoco tasks.
Main Comments to authors:
Pros:
-interesting combination of techniques (IAF flows and WGF/Stein inference) to do meta-sampling
-empirically appealing results as pertaining to raw performance metrics like accuracy
Weaknesses:
- The evaluation is focused on #of steps during testing, but not on # of datapoints required for eval on new domains.
-> this is only half of what we care about when saying a sampler is "fast". The other half would be sample efficiency, which is typically the main motivation for Bayesian models and accurate uncertainty estimates, for instance when performing Bayesian Optimization. In fact, when making claims about uncertainty estimates as the goal of the paper, it would be most interesting to see how much test data the method needs to ingest before producing calibrated estimates. The method as currently presented only evaluates speed in terms of computation, but ignores sample efficiency entirely. As such, it is unclear from the given experiments to evaluate the main claim of the paper: that DAMS improves uncertainty adaptation.
-> the uncertainty is barely evaluated except in the low-d and toy sine wave illustration, which probably can be done equally well or better with regular posterior inference on D = D_train union D_test, i.e. using HMC. My suggestion to the authors would be to consider comparing to regular Bayesian inference (i.e. Monte Carlo/HMC/SVI) based on train_data and test data to compare to a ground truth estimate they might want.
-In Sec. 3.3 Eq. 6 and 7 a kernel is used and then not discussed much further. Kernels on high dimensional data (such as the weights of a NN)  are problematic to be used due to the curse of dimensionality. For the meta-sampler, even in the case of the multiplicative parametrization which lowers dimensionality, this would indicate that the kernel part of the objective might not be doing much work at all as in high enough dimensions all distances become even. If that were to be the case, the model might just look for the mode in any high-d example instead of actually sampling from a posterior and producing uncertainty estimates. Any empirical analysis and discussion on this is entirely missing here, unfortunately. As presented, the reader just has to accept that the objective functions make sense because the final product of putting all of the components together produces high accuracies. I would appreciate more details and careful analysis.
-Please also show the performance of meta-sampler with and without sample adapter in this case to clarify the effects of performing sample adaptation versus just using the meta-sampler, as this also is never compared in the paper. I.e. how good of a conditional model is the autoregressive flow? How much work does the sampler have to do? Would another conditional model do as well or worse? Why this choice of conditional model in particular if in the end sampling is put on top of it?
-The ELBO in Sec.3.4 involves an inference network over NN parameters (or potentially latent Zs per feature when using multiplicative parametrization). This object is highly nontrivial and not analyzed in terms of performance at all here.  Inference networks over neural networks are hard to get right and worthy of entire publications.
-Please clarify the prior used for BNN models.
-Please consider using HMC as a baseline for BNN models per task in terms of LLK to compare to DAMS.
Baselines:
- A lot of this paper relies on comparison to baselines, which are chosen to be mostly from the meta-learning field.
However, in practice the goal of the paper is Bayesian Inference in a particular class of models.
Hence:
- please consider adding conditional MNF as a baseline. This would clarify if a simple conditional version of MNF would suffice here compared to the involved scheme proposed in this paper and might show the advantage of DAMS over MNF (effectively the main driver for most of the experiments here).
-Similarly, please consider using conditional NAI flow as a baseline to see how far that gets the reader.
Presentation Suggestions:
-You might want to consider establishing a formal relationship to a hierarchical probabilistic model with plates and show that this is just a way to perform sampling the posterior in a model like: P(y|x, tau) = integral_w P(y|x, w_t) P(w_t|tau) d_w
-This might help the flow of the paper by setting the stage early, as currently I had to read through it halfway to really understand the task before starting from the beginning to absorb the details of the proposed techniques.
-figures only readable in pdf, not in printout. Please enlarge fonts to give 'old school' readers a chance
-page 6  "..could be calculated effectively.." What does effectively mean here? Efficiently?
-page 6 under Theorem 1 typo: 'via the Eulaer scheme' -> Euler scheme
Related Work Suggestions:
- Consider citing "Predictive Uncertainty Quantification with Compound Density Networks" by Kristiadi et al, as it uses a conditional model with multiplicative parametrization successfully. I understand this is per data-point and the meta-learning scenario is focused on the per-dataset setting, but I find them related enough to consider a discussion.
- With regards to inference networks on BNNs, please cite "Latent Projection BNNs: Avoiding weight-space pathologies by learning latent representations of neural network weights" by Pradier et al, which attempts to do this and also discusses related work in more detail than this paper here. It is a hard task to be done well.
-The general form of the ELBO shown here is an instance of "Hierarchical Variational Models" by Ranganath et al, which should also be cited.
-Last but not least a recent paper in an ICML workshop on automatic machine learning  (https://sites.google.com/view/automl2019icml/accepted-papers) had a paper on "Improving Automated Variational Inference with Normalizing Flows" by Webb et al. This method looks a lot like a baseline method for this paper before task adaptation and WGF is considered and would effectively subsume the first batch of experiments entirely. I would propose the authors cite and discuss differences in detail.
Decision:
The paper uses a variety of 'puzzle pieces' that are quite involved on their own right. Putting them together and making it work is nontrivial and the authors demonstrate in their experiments that they get strong performance metrics. However, unfortunately, systematic ablation experiments and detailed analysis for the individual components used here and systematic comparisons to simple baselines are not performed. In addition, the paper is presented as a method for adaptive uncertainty quantification, which as argued above is not demonstrated empirically or else. What the paper does achieve is build a pipeline that gets high predictive performance on a meta-learning setting with lower computational requirements during testing than competing methods. I would suggest the authors focus on that aspect and add the required baselines that would clarify what ablations would do to the system and how the components interact.
As currently presented, I would argue for rejection since I am not sure of the scientific value of the interplay of components here as regarding uncertainty quantification. However, I think this paper is promising for a slightly different story with small experimental adjustments and would encourage the authors to consider that route.
(Edit Post Rebuttal:  revising score given the author response which addressed some of my concerns)