In multi-agent games, the complexity of the environment can grow exponentially as the number of agents increases, so it is particularly challenging to learn good policies when the agent population is large. In this paper, we introduce Evolutionary Population Curriculum (EPC), a curriculum learning paradigm that scales up MultiAgent Reinforcement Learning (MARL) by progressively increasing the population of training agents in a stage-wise manner. Furthermore, EPC uses an evolutionary approach to fix an objective misalignment issue throughout the curriculum: agents successfully trained in an early stage with a small population are not necessarily the best candidates for adapting to later stages with scaled populations. Concretely, EPC maintains multiple sets of agents in each stage, performs mix-and-match and fine-tuning over these sets and promotes the sets of agents with the best adaptability to the next stage. We implement EPC on a popular MARL algorithm, MADDPG, and empirically show that our approach consistently outperforms baselines by a large margin as the number of agents grows exponentially.

Review Update: Thank you for the detailed response, it raised my opinion of the paper as it reduced my concerns on the rigor of the experiments performed. I believe the changes increase the significance of the contribution and may help it reach a broader audience.
--
This paper proposes evolving curriculums of increasing populations of agents to improve multi-agent reinforcement learning with large number of agents. The topic is of relevance to the ICLR community and the results are tending towards publishable contributions, but I have some concerns that I would like the authors to discuss in their rebuttal.
The method is evaluated on a good range of suitably challenging environments. However, why did the authors propose new challenges within the particle environments and not those used in the original publication? This change makes it harder to compare results across publications, whilst not seeming to add a significant change in the learning problem beyond what was present in the original benchmark tasks. The food collection task sounds like it may be equivalent to simple spread. Will the new environments be released as open source for others to build upon this work?
The method is compared against a good range of baseline methods and ablations of the proposed method. However, without grounding the results in a known environment it is hard to place whether the implementations of MADDPG and Mean-Field are fair reproductions. Presuming the authors are using the open source implementation of MADDPG (please confirm) this is most significant for Mean-Field particularly given its poor performance in Section 5.4. Please provide evidence that this is a fair comparison.
To evaluate the resultant agents in the competitive environments, all methods were placed into games against the authors proposed EPC method. Was this the same EPC opponents the evaluated EPC team were trained against? If so, this is an unfair advantage to the EPC team as it has time to optimize against this opponent whilst the other methods have not. Even if it is an EPC team from a different training run, there may be outstanding biases in the joint policies EPC tends towards that benefit the EPC team evaluated. This could be overcome by evaluating all methods in competition with all other methods.
For all experimental results, please quantify variance in performance as well as average value (currently only done in Figure 9 a and b). How many repeats of evaluation and training were performed for each? Without these details the claim on page 9 that "EPC is always the best among all the approaches with a clear margin" is too strong. Are these differences statistically significant? Additionally, please also include the maximum scores (where normalized score = 1.0) for all experiments, as presenting results with only normalized scores unnecessarily reduces the reproducibility of the work.
Finally, in Appendix B, the authors provide a list of hyperparameter settings without discussion of how these were chosen. Were they optimised for one specific method or set to defaults from the literature? In particular, as the performance of Att-MADDPG is still improving at the end of the plot in Figure 9e, I am concerned that the #episodes was chosen to optimise the performance of EPC.
Overall, this is an interesting approach with promising initial results. I believe the contribution would be significantly improved by addressing the issues above and look forward to the authors responses which could increase my rating to acceptance.
Things that could improve the paper but did not impact the score:
1) On page 5 it is noted that the authors do not share parameters between the Q-function and policy. It would improve the paper to justify why this choice was made.
2) Page 5: "N_1 agents for of the role" -> N_1 agents of the role
3) Page 5: "as follows to evolved these K parallel sets" -> evolve
4) Page 7: "resources asThank yoTha green landmarks"
5) Page 8: "understand how the trained sheep behavior in the game" -> how the trained sheep behave in the game
6) Page 14: There are repeated grammatical issues in Appendix A e.g. "is more closer to grass / sheep / other agents" -> is closer to grass / sheep / other agents and "will less negative reward" -> will receive less negative rewards


We are interested in the optimization of a high-dimensional function when only function evaluations are possible. Although this derivative-free setting arises in many applications, existing methods suffer from high sample complexity since their sample complexity depend on problem dimensionality, in contrast to the dimensionality-independent rates of first-order methods. The recent success of deep learning methods suggests that many data modalities lie on low-dimensional manifolds that can be represented by deep nonlinear models. Based on this observation, we consider derivative-free optimization of functions defined on lowdimensional manifolds. We develop an online learning approach that learns this manifold while performing the optimization. In other words, we jointly learn the manifold and optimize the function. Our analysis suggests that the proposed method significantly reduces sample complexity. We empirically evaluate the presented method on continuous optimization benchmarks and high-dimensional continuous control problems. Our method achieves significantly lower sample complexity than Augmented Random Search and other derivative-free optimization algorithms.

Contributions:
	-Authors have proposed a methodology to optimise high dimensional functions in a derivative-free setup by reducing the sample complexity by simultaneously learning and optimising the low dimensional manifolds for the given high dimensional problem.
Although, performing dimensionality reduction to learn the low dimensional manifolds is popular in the research community, the extensions made and the approach authors have considered seems to be novel.
Comments:

	- Authors have talked about the utilization of domain knowledge on the geometry of the problem. How feasible is to expect the availability of the domain knowledge? Authors have not discussed the downsides of the proposed method if the domain knowledge is not available, and a possible strategy to overcome the same.
	- Authors have said that they are specifically interested in random search methods. Is there any motivating reason to stick to the random search methods? Why not consider other sample efficient search methods?
	“…….random search scale linearly with the dimensions”, why one should not consider other sample efficient methods that grow sub-linearly as against random search?
Srinivas, N., Krause, A., Kakade, S. M., and Seeger, M. Gaussian process optimization in the bandit setting: No regret and experimental design. International Conference on Machine Learning, 2010
	- Please derive Lemma 1 in the appendix for the sake of completeness.
	- I am missing a discussion about manifold parameters like “λ” in the important equations.
	- Authors have made a strong claim that neural networks can easily fit any training data, but it may be not be true for many datasets.
	 - Authors have claimed that they have fast and no-regret learning by selecting mixing weight β=1/d. Author might want to discuss more on this as this is an important metric.
	 - “ …. total time spent on learning the manifold is negligible…. ” – any supporting results for this claim.
	 - “…….communication cost from d+2k to d+2k+kd… ” – curious to know if there is any metric like wall-clock time to talk about the optimisation time.
	- Authors have restricted the comparisons to only three important methods, but it is always comparing with other baselines in the same line. Authors should consider Bayesian optimisation as it is a good candidate for the performance comparison, even though the researchers are interested only in random search methods (just like CMA–ES).
Kirschner, Johannes, Mojmír Mutný, Nicole Hiller, Rasmus Ischebeck, and Andreas Krause. "Adaptive and Safe Bayesian Optimization in High Dimensions via One-Dimensional Subspaces." arXiv preprint arXiv:1902.03229 (2019).
	- It is seen from the results that the proposed method is not performing better for low dimensional problem like “Swimmer” function. But according to the initial claim, method was supposed to work better in low dimensional problems. Is it because of the fact that the problem space is not drawn from high dimensional data distributions?
	- “…..improvement is significant for high dimensional problems” – It will be better if the authors compare their proposed method with some more derivative-free optimisers that are proven to be good in high dimensions (like high dimension Bayesian optimisation).
	-  “The no-learning baseline outperforms random search ……….” – this statement is not very clear, does it mean like the proposed method works only when the problem is reduced from higher dimensions to lower dimensions and not on the lower dimensional problem itself?
	- “Performance profiles represent how frequently a method is within the distance T of optimality” – Any thumb rule considered for the choice of T?. Can we think of any relation with standard metrics like simple regret or cumulative regret that are used to measure the optimisation performance?
	- “Although BO methods typically do not scale…… ” – Authors have made a strong assumption here. In the literature, we see active research in the context of high dimensional optimisation.
Rana, Santu, Cheng Li, Sunil Gupta, Vu Nguyen, and Svetha Venkatesh. "High dimensional Bayesian optimization with elastic gaussian process." In Proceedings of the 34th International Conference on Machine Learning-Volume 70, pp. 2883-2891. JMLR. org, 2017.

Minor Issues:
	“Improve the sample complexity” may not convey the meaning very clearly to the readers, something like “Improve the sample efficiency” or “Reduce the sample complexity” would add more clarity.
	Inconsistency in the terms used in Proposition 1 and Proposition 2. What does “I” signify in the formula? Was that supposed to be “t”?
	Even though the constants are mentioned in the appendix, it is always better to mention the constants used in the algorithm like “α“ as step size for quick understanding.
	“Follow The Regularized Leader (FTRL)” is more appropriate than “follow the regularized leader (FTRL)”
	“noise table in pre-processing” – Should it mean something relevant to the paper?
	“We use widely used…..” – may be consider rephrasing the sentence here
	“treshold” – Typo in Table 1
	Y – Axis in Figure 2 is missing
	Appendix B :
	“We also perform grid search …. “ would look better
	MuJoCo Experiments – is the parameter space continuous and what is the search space considered for n, α and δ. Do we deal with smaller search spaces in every problem? Any other way of searching the parameter space to further improve the efficiency?


We study the benefit of sharing representations among tasks to enable the effective use of deep neural networks in Multi-Task Reinforcement Learning. We leverage the assumption that learning from different tasks, sharing common properties, is helpful to generalize the knowledge of them resulting in a more effective feature extraction compared to learning a single task. Intuitively, the resulting set of features offers performance benefits when used by Reinforcement Learning algorithms. We prove this by providing theoretical guarantees that highlight the conditions for which is convenient to share representations among tasks, extending the wellknown finite-time bounds of Approximate Value-Iteration to the multi-task setting. In addition, we complement our analysis by proposing multi-task extensions of three Reinforcement Learning algorithms that we empirically evaluate on widely used Reinforcement Learning benchmarks showing significant improvements over the single-task counterparts in terms of sample efficiency and performance.

This paper provides a theoretical justification for the benefit of multi-task deep RL (MTRL) with shared representations. By extending prior work (Farahmand (2011) and Maurer et al. (2016)), the authors demonstrate that the bound of MTRL can be improved if the cost of learning the shared representation at each AVI iteration can be reduced, which is mitigated as we increase the number of tasks. The author also empirically verify their theoretical results in a tabular Q-Fitted Iteration domain and also in challenging RL domains such as Mujoco. The results show that MTRL with shared representation can outperform their single task counterparts to some degree.
Overall this paper adapts the theory shown in Farahmand (2011) and Maurer et al. (2016) to the setting of MTRL and demonstrates the effectiveness of using shared layers, which seems intuitive. While the theory seems a bit incremental, it’s the first paper that theoretically validates the benefits of sharing knowledge, which is a contribution to the MTRL field. I would recommend a weak accept, though I have a few concerns on experimental results, and hope that the authors can clarify them during rebuttal.
Specifically, as the authors have noted, there is a wide range of prior works [1,2,3] that have empirically demonstrated the effectiveness of utilizing shared representations in MTRL. While the authors claim that the goal of the experiments is to show that MTRL with shared layers can outperform its sing task counterparts and thus they ignore other MTRL approaches. I believe that is not the main argument of the paper. The authors should provide empirical evidence on the claim that with an increasing number of tasks in MTRL, the error bound should improve and the performance of MTRL should also boost. Besides, I find the comparison where single-task training is initialized with shared representation a bit confusing. Training would definitely be improved when it’s initialized with some related pretrained features. Maybe the authors should compare this to some other methods such as initializing with single-task representation or even representation learned from training different tasks.
[1] M. Hessel, H. Soyer, L. Espeholt, W. Czarnecki,S. Schmitt, and H. van Hasselt. Multi-task deep reinforcement learning with popart.arXiv preprintarXiv:1809.04474, 2018.
[2] Teh, Y.W., Bapst, V., Czarnecki, W.M., Quan, J., Kirkpatrick, J., Hadsell, R.,Heess, N., Pascanu, R.: Distral: Robust multitask reinforcement learning. In: Ad-vances in Neural Information Processing Systems 30: Annual Conference on Neu-ral Information Processing Systems 2017 (2017)
[3] Wulfmeier, M., Abdolmaleki, A., Hafner, R., Springenberg, J. T., Neunert, M., Hertweck, T., ... & Riedmiller, M. (2019). Regularized Hierarchical Policies for Compositional Transfer in Robotics. arXiv preprint arXiv:1906.11228.


When performing imitation learning from expert demonstrations, distribution matching is a popular approach, in which one alternates between estimating distribution ratios and then using these ratios as rewards in a standard reinforcement learning (RL) algorithm. Traditionally, estimation of the distribution ratio requires on-policy data, which has caused previous work to either be exorbitantly datainefficient or alter the original objective in a manner that can drastically change its optimum. In this work, we show how the original distribution ratio estimation objective may be transformed in a principled manner to yield a completely off-policy objective. In addition to the data-efficiency that this provides, we are able to show that this objective also renders the use of a separate RL optimization unnecessary. Rather, an imitation policy may be learned directly from this objective without the use of explicit rewards. We call the resulting algorithm ValueDICE and evaluate it on a suite of popular imitation learning benchmarks, finding that it can achieve state-of-the-art sample efficiency and performance.

The primary contribution of this paper is a principled algorithm for off-policy imitation learning. Generative Adversarial Imitation Learning (GAIL), proposed by Ho and Ermon in 2016, is an on-policy imitation learning method that (provably) minimizes the divergence between a target state-action distribution and the policy state-action distribution. Followups to this work (Kostrikov et al., 2019) show that the same algorithm can be applied to an off-policy setting (replacing the on-policy samples with samples from the replay buffer) and the method still works, but is no longer theoretically justified. I believe using importance ratios would make this approach justified as well, but Kostrikov et al. found that using importance ratios actually degrades the performance of their method (due the difficulty associated with estimating importance ratios). This paper attempts to bridge this gap: a method that is theoretically justified, and still works.
The paper takes most of its inspiration from the recently proposed DualDICE paper (Nachum et al, 2019), where the authors introduce a method for estimating discounted stationary distribution ratios (i.e. d^{\pi}/d^{D}, where \pi is some (known) policy, and D is a given dataset of experience (for example, a replay buffer)). The authors essentially apply the method proposed in DualDICE to estimate d^{\pi}/d^{exp} instead, where d^{exp} is a dataset of expert trajectories. While it would be possible to simply use this term as a reward and then run reinforcement learning, the authors note that the specific form of estimating d^{\pi}/d^{exp} allows them to instead directly a train a value function, which can then be used for updating a policy.
The authors also argue that their method reduces complexity since it does not require a separate RL optimization routine. However, I think having a separate RL optimization routine has its own advantages - it is relatively easy to implement GAIL on top of any existing RL algorithm, which makes it easy to take advantage of recent advances in RL. For the method proposed in this paper, it would not be straightforward to do so.
The authors also note that they need a number of practical modifications to their original ValueDICE objective in order to make things work (Section 5: Some practical considerations). Notably, the the original ValueDICE objective only needs access to expert samples and the initial state distribution, and does not need access to the the replay buffer samples (apart from the initial state ones). This would likely not work well in practice - similar to how behavior cloning often does worse than GAIL when learning from a small number of expert examples. In order to combat this, the authors incorporate replay buffer regularization
The authors provide experiments on  a simple synthetic "ring MDP", and on four continuous control tasks from OpenAI gym - HalfCheetah, Hopper, Ant and Walker2d. When compared to prior approaches (i.e. Kostrikov et al 2019) in the low-data regime (where behavior cloning fails), the proposed method does significantly better on one task (Ant), slightly worse on one task (walker), and about the same on two tasks (Hopper and HalfCheetah). I do notice the proposed method as being somewhat unstable though - the reward appears to be going down after reaching the max on two of the tasks - HalfCheetah and Ant. Overall, I don't thing experiments are thorough enough to demonstrate that the method is empirically better than competing approaches, but I believe that is not the main point of the paper.
Overall, my recommendation is a weak accept. It is interesting that the authors were able to get a principled method for off-policy imitation learning working (large following from prior work in Nachum et al 2019), but I don't think the method currently offers any significant practical advantages over competing methods.


We prove bounds on the generalization error of convolutional networks. The bounds are in terms of the training loss, the number of parameters, the Lipschitz constant of the loss and the distance from the weights to the initial weights. They are independent of the number of pixels in the input, and the height and width of hidden feature maps. We present experiments with CIFAR-10, along with varying hyperparameters of a deep convolutional network, comparing our bounds with practical generalization gaps.

Summary
This paper studied the generalization power of CNNs and showed several upper bounds of generalization errors. Their results have two characteristics. First, the bounds are in terms of the quantity that is independent of the input dimension (size-free). Second, the upper bounds involve the distance between initial and learned parameters. These results improved the upper bounds that we can derive by naively applying the results of Bartlett et al. (2017) or Neushubar et al. (2017), because the dominant term of the existing upper bounds contained $l_{2, 1}$ or $l_2$ norms, which could depend on the input dimensions in the worst case. The authors empirically showed that there is a correlation between the generalization error of learned CNNs and the dominant term of the upper bound (i.e., the product of the parameter size and the distance from the set of initial parameters).
Decision
To the best of my knowledge, this is the first work that proved the size-free generalization bound for multi-layer CNNs. However, I think the assumption on the hypothesis class is very restrictive and significantly eases the problem, as I discuss in detail later. Therefore, I judge the technical contribution of the paper is moderate and recommend to reject the paper weakly.
By the standard argument of the statistical learning theory (such as Theorem A.4), we can typically bound the generalization error by $O(B\sqrt{D/N})$ where $B$ is the infimum of Lipschitz constant of hypotheses, $D$ is the intrinsic dimension of the hypothesis class, and $N$ is the sample size. Therefore, we can derive the size-free generalization bound if $B$ does not depend on the input dimension. Since the hypothesis class $F_\beta$ is defined via the spectral norm of CNNs, it is not surprising that we can derive the size-freeness of $B$. The size-free generalization bound has been already proven by Du et al., (2017), although it was the two-layered case. They imposed a restricted eigenvalue assumption. I think it implies that we need more sophisticated analysis if we do not assume the size-freeness of the hypothesis class.
Comments
- The authors claimed that Figure 3 is consistent with theorems because, according to the upper bound of theorems, the distance from the initialization point decreases when the generalization error is the same and the parameter size increases. However, I think it is too aggressive to conclude it from Figure 3 because the decreasing trend in the value of $\|K-K_0\|_\sigma$ is found only around $2\times 10^6\leq W \leq 3\times 10^6$. Furthermore, the value of $\|K-K_0\|_\sigma$ for  $W\approx 5\times 10^5$ is approximately the same as the value for $W\approx 3\times 10^6$.
Suggestions
- Please add the conclusion section which summarizes the paper and discusses the possible research directions.
Minor Comments
- page 1, section 1, paragraph 1
	- ... with roots in (Bartett, 1998) , is that ... → Use \citet
- page 2, section 2.1, paragraph 2
	- Write the definition of "expansive" activations.
- page 3, section 2., theorem 2.1
	- I think we should replace $\log(\lambda n)$ and $\log(\lambda)$ in equations with $\log(\beta \lambda n)$ and $\log(\beta \lambda)$, respectively.
- page 3, section 2.2, definition 2.2
	- $N$ → $\mathbb{N}$


Forming perceptual groups and individuating objects in visual scenes is an essential step towards visual intelligence. This ability is thought to arise in the brain from computations implemented by bottom-up, horizontal, and top-down connections between neurons. However, the relative contributions of these connections to perceptual grouping are poorly understood. We address this question by systematically evaluating neural network architectures featuring combinations of these connections on two synthetic visual tasks, which stress low-level "Gestalt" vs. high-level object cues for perceptual grouping. We show that increasing the difficulty of either task strains learning for networks that rely solely on bottom-up processing. Horizontal connections resolve this limitation on tasks with Gestalt cues by supporting incremental spatial propagation of activities, whereas top-down connections rescue learning on tasks with high-level object cues by modifying coarse predictions about the position of the target object. Our findings dissociate the computational roles of bottom-up, horizontal and top-down connectivity, and demonstrate how a model featuring all of these interactions can more flexibly learn to form perceptual groups.

This paper proposed a dataset and designed a relevant network structure to analyze the function of horizontal and top-down connections for perceptual grouping. The used two datasets smartly isolate the requirements for exploiting Gestalt cues and object-based strategies. Appendix A detailed describes the cABC dataset, and the control experiments in Appendix B further validate the designing of the cABC. The proposed network flexibly integrates three types of connections and successfully solves both two challenges. The visualization results in Figure 4, S8 and S9 are insightful and also validate the intuitions. Overall the paper is clearly written and easy to follow.
Although the use of different types of connections in the proposed network is clear, I think the author should conduct more analysis for the standard networks and datasets.
In section 3, the author claims that the two challenges will cause a high computational burden for feedforward models like ResNets. However, in Figure 5, the results show that ResNet-152 and ResNet-50 could easily solve the Pathfinder. For cABC, ResNet-18 could solve and ResNet-50 and ResNet-152 will fail while the author illustrates this is due to ResNet-50 and ResNet-152 overfit to cABC. Those phenomena are quite misaligning with the arguments.
- Can we find a suitable ResNet structure that could solve both two challenges?
- Can we increase the depth of U-Net and make it solve both two challenges?
- Can we add more data or increase the difficulty for cABC and prevent the overfitting for ResNet-50 and ResNet-152?
- Can we keep increasing the difficulties of cABC and straining ResNet-18 and UNet? Does that difficulty level would also strain TD+H-CNN?
- We could also perform similar experiments on Pathfinder that increasing its difficulty for ResNet-50 and ResNet-152.
Also, if we train the TD+H-CNN on standard image datasets such as cifar-10 or ImageNet, what about the recurrent activities and the final acc?
=========================================================
After Rebuttal:
I thank the author for the response.
I hope the comments are useful for preparing a future version of this work when you have enough time.


Event sequences can be modeled by temporal point processes (TPPs) to capture their asynchronous and probabilistic nature. We propose an intensity-free framework that directly models the point process as a non-parametric distribution by utilizing normalizing flows. This approach is capable of capturing highly complex temporal distributions and does not rely on restrictive parametric forms. Comparisons with state-of-the-art baseline models on both synthetic and challenging real-life datasets show that the proposed framework is effective at modeling the stochasticity of discrete event sequences.

The authors propose a method for learning models for discrete events happening in continuous time by modelling the process as a temporal point process. Instead of learning the conditional intensity for the point process, as is usually the case, the authors instead propose an elegant method based on Normalizing Flows to directly learn the probability distribution of the next time step. To further increase the expressive power of the normalizing flow, they propose using a VAE to learn the underlying input to the "Flow Module". They show by means of extensive experiments on real as well as synthetic data that their approach is able to attain and often surpass state of the art predictive models which rely on parametric modelling of the intensity function. The writers have put their contributions in context well and the presentation of the paper itself is very clear.
Though the final proof is in the pudding, and the addition of the VAE to model the base distribution yields promising results, the only justification for it in the paper is to create a more "expressive" model. There are multiple ways of increasing the expressiveness of the underlying distribution: moving from RNNs to GRU or LSTMs, increasing the hierarchical depth of the recurrence by stacking the layers, increasing the size of the hidden state, more layers before the output layer, etc. A convincing justification behind using a VAE for the task seems to be missing. Also, using the VAE for a predictive task is a little unusual.

Another, relatively small point which the authors glance over is the matter of efficient training. The Neural Hawkes model suffers from slow training because of the inclusion of a sampling step in the likelihood calculation. I believe that since the model proposed by the authors allows easy back-propagation, their model ought to be easy and fast to train as well. Including the training time for the baselines, as well as the method proposed by the authors, will help settle the point.
Minor point:
 - The extension of the method to Marked Temporal Point Processes in the Evaluation section seems out of place, esp. after setting up the expectation that the marks will not be modelled initially, up till footnote 2 on page 7.


Most of the work on interpretable machine learning has focused on designing either inherently interpretable models, which typically trade-off accuracy for interpretability, or post-hoc explanation systems, which lack guarantees about their explanation quality. We explore an alternative to these approaches by directly regularizing a black-box model for interpretability at training time. Our approach explicitly connects three key aspects of interpretable machine learning: (i) the model's inherent interpretability, (ii) the explanation system used at test time, and (iii) the metrics that measure explanation quality. Our regularization results in substantial improvement in terms of the explanation fidelity and stability metrics across a range of datasets and black-box explanation systems while slightly improving accuracy. Finally, we justify theoretically that the benefits of our regularization generalize to unseen points.

The paper considers the problem of training black box models for improved interpretability, and  proposes to penalize black-box models at training time using two regularizers that correspond to fidelity and stability explanation metrics. As computing the regularizers exactly is computationally intensive they propose two approximating algorithm. In addition, as the one for fidelity is still prohibitive, a randomized variant is proposed. Connections are established between the regularizers and the model's Lipschitz constant or total variation. A generalization bound is presented for local linear explanations. The proposed approach is evaluated on a variety of datasets.
The paper deals with an important problem and the exposition is clear.  While regularizing deep learning models is a pertinent  direction, I feel the paper makes a couple of overstatements, and overall I am not fully convinced by the approach and empirical evaluation, as outlined below.
- The paper states "recent approaches that claim to overcome this apparent trade-off between prediction accuracy and explanation quality are in fact by-design proposals that impose certain constants on the underlying model families they consider" and that they are addressing this shortcoming. But in fact, the proposed regularizations do exactly the same: they impose certain constraints. Indeed the regularizers encourages models with lower Lipschitz constant or with small total variation across neighborhoods.
- I also find that it is unsurprising that regularizing via fidelity or stability will lead to models with better fidelity/stability so it's an artificial way to yield "improved interpretability" and this is more of an issue because fidelity and stability are kind of proxy metrics to evaluate interpretability.
- It would be important to investigate further the difference between regularization and explanation neighborhoods. This might not be a bad thing which in fact help with generalization.
- Proposition 1 supports algorithm 2, but it is not a given at all that Algorithm 1 will have smaller local variances across neightborhoods and hence might generalize well.  It would be important to proceed with an empirical study of the local variance across neighborhoods for Algorithm 1.

- Computational complexity remains an issue as ExpO-1D-fidelity is performing much worse.
- Comparison against alternative approaches beyond SENN are lacking (e.g. Lee et al 2019, Wang and Rudin,2015 etc).

 Overall I feel that more work is needed to convincingly demonstrate the importante of the proposed approach.


We study the phenomenon that some modules of deep neural networks (DNNs) are more critical than others. Meaning that rewinding their parameter values back to initialization, while keeping other modules fixed at the trained parameters, results in a large drop in the network's performance. Our analysis reveals interesting properties of the loss landscape which leads us to propose a complexity measure, called module criticality, based on the shape of the valleys that connects the initial and final values of the module parameters. We formulate how generalization relates to the module criticality, and show that this measure is able to explain the superior generalization performance of some architectures over others, whereas earlier measures fail to do so.

This paper introduces a new way to reason about neural network generalization using a module criticality measure. The measure is tangible and intuitive. It leads to some formal bounds on the generalization of deep networks, and is able to better rank trained image classification architectures than previous measures.
I am leaning to accept, as I expect this to be a significant theoretical contribution with several potential practical applications. With a few additional details, this could be a very strong submission:
(1)	Choice of module decomposition. Having each module be a single convolutional or fully-connected layer makes intuitive sense, but is there some theoretical motivation for this choice? If the only requirement for a module is that it includes some linear transformation, in the extreme, a module could consist of a single weight, or the entire network. Would those choices change the generalization bounds or relative criticality across different architectures?
(2)	Scope of experimental results. The ranking results would be much more compelling if they included a broader range of architectures, including more recent models with more branching, e.g., DenseNet. Is there some reason ResNet101 has higher generalization error than 18 and 34? Net. Criticality for ResNets is inversely correlated with the number of layers; is there an explanation for this? Is this true for other very deep models?
(3)	Practical use. To compute the criticality measure, we must train the model; but, if we train the model, we can compute generalization directly. So, what is the practical application of the measure? Is there some way it could be used to save computation? Could it help in the case of a small validation dataset, which we do not want to look at many times during model selection?
Minor typos:
-	Section 2.2: “An stable phenomena”
-	Section 2.3: “…an the…”
-	In appendix: “ResNet101: ResNet34 architectures…”
----------------------------
After rebuttal:
The authors have addressed my concerns, and I've increased my rating. There are still a few points I'd like to see addressed in the final version:
1. The fact that the approach cannot yet be applied to batch normalization is a big practical drawback. Some discussion of approaches you tried, why they didn't work, and possible future directions for overcoming this would be appreciated.
2. Clarify in the paper that the "PAC Bayes" approach used for comparison (Table 1) is a your method, i.e., an ablated version of criticality. As is, someone reading the paper quickly may think all you've done is add an alpha parameter to an existing "PAC Bayes" approach, which does fairly well on its own.
3. Visualizing the experimental tables as scatterplots could make them easier for a reader to interpret.


This work explores the visual explanation for deep metric learning and its applications. As an important problem for learning representation, metric learning has attracted much attention recently, while the interpretation of such model is not as well studied as classification. To this end, we propose an intuitive idea to show where contributes the most to the overall similarity of two input images by decomposing the final activation. Instead of only providing the overall activation map of each image, we propose to generate point-to-point activation intensity between two images so that the relationship between different regions is uncovered. We show that the proposed framework can be directly deployed to a large range of metric learning applications and provides valuable information for understanding the model. Furthermore, our experiments show its effectiveness on two potential applications, i.e. cross-view pattern discovery and interactive retrieval.

= Summary
This paper presents a simple method that draws visual attention of deep embedding networks for metric learning. It basically follows the class attention mapping strategy based on global pooling operation [Zhou et al., CVPR 2016], but extends the original version to point-specific attention which is novel and enable interesting applications on image retrieval. In addition, the proposed method seems also independent of the loss function used for metric learning, thus it can be applied to most of existing deep embedding networks to understand their behaviors in a qualitative manner.
= Decision
My current decision is officially "weak reject" but "borderline" in my mind. The major concern of mine is its weaknesses in clarity and technical novelty. However, I still believe this submission is valuable since it addresses a relatively new and timely topic, the proposed method is simple yet effective, and the applications of point-specific attention (i.e., "cross-view pattern discovery" and "interactive retrieval") are all interesting and practically useful. If the clarity issues are all clearly addressed, I would upgrade my rating.
= Comments
[Pros]
1) The motivation and implementation of the point-specific attention are convincing.
2) The point-specific attention enables not only image-to-image retrieval, but also more elaborate understanding about a pair of images and their similarity.
3) The proposed technique is model-agnostic and loss-agnostic, thus can be applied to most of existing deep embedding networks for image retrieval. Also, the proposed technique does not degrade the retrieval performance.
4) The proposed technique is simple yet effective in multiple applications, most of which are practically useful and have great impact. For example, the weakly supervised localization is an essential step towards many weakly supervised approaches for higher-level recognition tasks like semantic segmentation, and the interactive retrieval will allow us to build more realistic and useful image retrieval systems.
[Cons]
1) The proposed technique itself is not new but a straightforward extension of an existing work [Zhou et al., CVPR 2016].
2) The manuscript is not crystal clear.
- The name of the proposed point-specific attention (i.e., "Partial attention") is misleading.
- The way to compute the point-specific attention map is not clearly described in Section 3.
- It is hard to understand the contents in Figure 6 as they are not clearly illustrated.
- The experimental and implementation details of the last two applications are not given. For example: (cross-view pattern discovery) how to compute the angle error, and how much the proposed technique is sensitive to the position selected on the query image, (interactive retrieval) how to compute the similarity between images given a specific region of interest on query.
3) More qualitative results on the last two applications should be presented, even in an appendix, to convince future readers. Especially, in the case of "interactive retrieval", more results are demanded as quantitative performance analysis seems not straightforward.
= Post-rebuttal review
The rebuttal resolves my major concerns and the manuscript has been carefully revised accordingly. So as I promised in my original review, I upgrade the score to weak accept.


We present a hierarchical planning and control framework that enables an agent to perform various tasks and adapt to a new task flexibly. Rather than learning an individual policy for each particular task, the proposed framework, DISH, distills a hierarchical policy from a set of tasks by self-supervision and reinforcement learning. The framework is based on the idea of latent variable models that represent high-dimensional observations using low-dimensional latent variables. The resulting policy consists of two levels of hierarchy: (i) a planning module that reasons a sequence of latent intentions that would lead to optimistic future and (ii) a feedback control policy, shared across the tasks, that executes the inferred intention. Because the reasoning is performed in low-dimensional latent space, the learned policy can immediately be used to solve or adapt to new tasks without additional training. We demonstrate the proposed framework can learn compact representations (3-and 1-dimensional latent states and commands for a humanoid with 197-and 36-dimensional state features and actions) while solving a small number of imitation tasks, and the resulting policy is directly applicable to other types of tasks, i.e., navigation in cluttered environments. The supplementary video is available at: https://bit.ly/2rwIfQn 1 INTRODUCTION Reinforcement learning (RL) aims to compute the optimal control policy while an agent interacts with the environment. Recent advances in deep learning enable RL frameworks to utilize deep neural networks to efficiently represent and learn a policy having a flexible and expressive structure. As a result, we've been witnessing RL agents that already achieved or even exceeded human-level performances in particular tasks (Mnih et al., 2015;Silver et al., 2017). The core of intelligence, however, is not just to learn a policy for a particular problem instance, but to solve various multiple tasks or immediately adapt to a new task. Given that a huge computational burden makes it unrealistic to learn an individual policy for each task, an agent should be able to reason about its action. If predictions about consequences of actions are available, e.g., by using an internal model (Ha &  Schmidhuber, 2018; Kaiser et al., 2019), an intelligent agent can plan a sequence of its actions. Involving planning procedures in a control policy could provide adaptiveness to an agent, but it is often not trivial to learn such a prediction & planning framework: First, it is difficult to obtain the exact internal dynamic model directly represented in high-dimensional state (observation) space. Model errors inevitably become larger in the high-dimensional space and are accumulated along the prediction/planning horizon. This prohibits planning methods from producing a valid prediction and so a sensible plan. Second, and perhaps more importantly, planning methods cannot help but relying on some dynamic programming or search procedures, which quickly become intractable for problems with high degrees of freedom (DOFs) because the size of search space grows exponentially with DOFs, i.e., the curse of dimensionality (LaValle, 2006).
Crucial evidence found in the cognitive science field is that there exists a certain type of hierarchical structure in the humans' motor control scheme addressing the aforementioned fundamental difficulty (Todorov & Ghahramani, 2003;Todorov, 2004). Such a hierarchical structure is known to utilize two levels of parallel control loops, operating in different time scales; in a coarser scale, the high-level loop generates task-relevant commands for the agent to perform a given task, and then in a finer time scale, the low-level loop maps those commands into control signals while actively reacting to disturbances that the high-level loop could not consider (e.g., the spinal cord) (Todorov 1 Under review as a conference paper at ICLR 2020& Ghahramani, 2003). Because the low-level loop does not passively generate control signals from high-level commands, the high-level loop is able to focus only on the task-relevant aspects of the environment dynamics that can be represented in a low-dimensional form. Consequently, this hierarchical structure allows us for efficiently predicting and planning the future states to compute the commands.
. .

*Edit: original score was a weak reject (3), updating to a weak accept (6) in light of revisions.*
This work implements a hierarchical control scheme for a high-dimensional control problem (locomotion using a humanoid body).  The hierarchy consists of a high-level module that plans in an abstract space of "intention", and the intention variables serve as inputs, along with state, to a low-level controller that actually executes the movements.  The premise is that a lower-level controller should be usable for multiple tasks, and should be able to be commanded by a lower-dimensional intention input.  I find the basic ideas presented clear, the literature reviewed reasonably well, and the motivation and setting to be very interesting.  The video summary is valuable.
My main concerns have to do with presentation, but I think they are relatively significant concerns.  As the draft currently stands, I would, somewhat regrettably, be inclined to reject the submission (marginally).  I think revisions could seriously improve this paper and incline me towards acceptance.
Algorithm 1 indicates that the learning of the low-level controller will be done jointly with the learning of the latent model and planning using the high-level, learned intention space.  In the experiment, it is indicated that the low-level controller is pretrained.  This points to a couple issues that are not clear in the draft:
(1) Presumably this pretraining is necessary and things do not work without it.  Indeed, it is hard to imagine that the movements will be well grounded to human motion capture movements without this pretraining.  Does the algorithm work as written or is pretraining a fundamentally essential step?  There are no settings, even toy settings, where the algorithm as written is shown to be effective.
(2) The authors should be clearer how they conduct the pretraining which involves learning the low-level controller.
(3) I'm not clear how updating the low-level controller is effective in the algorithm.  While I understand why it makes sense to plan in the intention space of the pre-trained controller, and I understand why learning a model is a core part of planning, it would seem like fine-tuning the low-level controller could make the movements deviate considerably from the initial movement space and maybe even eliminate the ability of the low-level policy to express movements that are not used early in training. So essentially, while the planning in the low-D space makes sense and the learning of the model makes sense, the low-level controller update seems possibly to not make sense, and there aren't experiments showing that step helps.
Is it just a coincidence that the intention space (h) is one-of-three and the low-d state space (z) is 3-dimensional as well?  Or are these both selected with sort of going straight vs turning left or right in mind?
The experiment section is generally very unclear, though details are made a little clearer from the video.  In the paper, there are a few points that need to be clearer:
(1) "ref", "plan", and  "true" are not well defined and it is unclear what these distances in Table 1 refer to precisely.  Clearly introduce what each of these refers to.  The authors simply say that there are imitation tasks but do not walk through what these terms refer to.
(2) In 4.1, the different structures are not adequately introduced.  The pointers to the figure 2 diagrams are essential, but there is no pointer for zaz', the pointers are only in the table (not in the text), there are grammar issues in the text and the text could be verbally clearer about the variants.
(3) shs' setting is a bit unclear. Basically, clarify briefly how planning is performed in this case.   Is a forward model still trained, but the model opperates with the full state space?  If so, presumably the forward model is much worse and then the planning approach is correspondingly bad, hence the poor rollouts?
(4) 4.2 is I think obviously inadequately described in the text and I can only assume was the result of rushing for the deadline?  The second experiment is essentially not presented in the text all aside from a still image.
(5) And for all of the experiments that rely on planning with a particle filter, details such as how reliable the filter is in generating useful control, how many samples are required, and possibly elements of compute speed would make much clearer how well the approach actually works.  Does the choice of planner matter at all?  A common, albeit relatively weak, baseline planning approach is CEM...would CEM work here?  I'd like to understand if the choice of particle filter is the author's default choice, which is fine if so, or if there is a positive assertion being made that the particle filter is particularly valuable.
I hope the authors will generally improve the exposition in the experiment section (4) during the revisions.
Overall, I find the paper well motivated in framing the problem (i.e. using model-based approaches to control the latent space of a low-level controller).  I also appreciate the scale of the problem (humanoid control is challenging, so this is not a toy problem).  I find the results a bit unclear, perhaps due to hurriedness in writing, so I find them a bit difficult to fully appreciate.  Nevertheless, the core contribution that I take away from this work is that there is a value to learning the low-dimensional state representation (z, via the LVM), relative to planning using a forward model on the full state (?...I'm still unclear on the presentation of this result, due to unclear exposition). Slightly more broadly, this is a good demonstration of using a planner jointly with a learned high-level command/intention representation, for a high-dimensional problem.
If I've understood this correctly, I'd be reasonably interested in this result. If the authors can both clarify the core results and communicate that the choices made in the algorithm are well thought through, I would be happy to adjust my score.
Relatively minor:
Abstract says 90-dimensional humanoid system, but later it is stated "34 degrees of freedom,
197 state features, and 36 action parameters". Where the 90 dimensions comes from is unclear.  Often people refer to number of actuators or DoFs.  Please adjust this or be more explicit.
In equation 9, f() is not very clearly specified. Is f() a nonlinear function (e.g. a neural network) or is it a linear function? It seems like it might as well be a linear function, since the authors propose to learn a latent dynamics model that is nonlinearly related to the state.
Typos in Fig 3 caption.


Generalization error (also known as the out-of-sample error) measures how well the hypothesis learned from training data generalizes to previously unseen data. Proving tight generalization error bounds is a central question in statistical learning theory. In this paper, we obtain generalization error bounds for learning general non-convex objectives, which has attracted significant attention in recent years. We develop a new framework, termed Bayes-Stability, for proving algorithm-dependent generalization error bounds. The new framework combines ideas from both the PAC-Bayesian theory and the notion of algorithmic stability. Applying the Bayes-Stability method, we obtain new data-dependent generalization bounds for stochastic gradient Langevin dynamics (SGLD) and several other noisy gradient methods (e.g., with momentum, mini-batch and acceleration, Entropy-SGD). Our result recovers (and is typically tighter than) a recent result in Mou et al. (2018)  and improves upon the results in Pensia et al. (2018). Our experiments demonstrate that our data-dependent bounds can distinguish randomly labelled data from normal data, which provides an explanation to the intriguing phenomena observed in Zhang et al. (2017a). We also study the setting where the total loss is the sum of a bounded loss and an additional 2 regularization term. We obtain new generalization bounds for the continuous Langevin dynamic in this setting by developing a new Log-Sobolev inequality for the parameter distribution at any time. Our new bounds are more desirable when the noise level of the process is not very small, and do not become vacuous even when T tends to infinity.

This paper aims at developing a better understanding of generalization error for increasingly prevalent non-convex learning problems. For many such problems, the existing generalization bounds in the statistical learning theory literature are not very informative. To address these issues, the paper explores algorithm-specific generalization bounds,  especially focusing on various types of noisy gradient methods.
The paper employs a framework that combines uniform stability and PAC-Bayesian theory to obtain generalization bound for the noisy gradient methods. For gradient Langevin dynamic (GLD) and stochastic gradient Langevin dynamics (SGLD), using this Bayes-Stability framework, the paper obtains a generalization bound on the expected generalization error that scales with the expected empirical squared gradient norm. As argued in the paper, this provides an improvement over the existing bounds in the literature. Furthermore, this bound enables the treatment of the setting with noisy labels. For this setting the expected empirical squared gradient norm along the optimization path is higher, leading to worse generalization bound.
The paper then extends their results to the setting where an $\ell_2$ regularization is added to the non-convex objective. By using a new Log-Sobolev inequality for the parameter distribution at time t, the paper obtains new generalization bounds for continuous Langevin dynamic (CLD). These bounds subsequently provide bounds for GLD as well.
The paper demonstrates the utility of their generalization bound via empirical evaluation on MNIST and CIFAR dataset. The obtained generalization bounds are informative as they appear to capture the trend in the generalization error.
Overall, the paper is very well written with a clear comparison with the existing generalization bounds. The results in the paper are interesting and novel. That said, the discussion in the introduction and abstract appears a bit misleading as it gives the impression that this is the first paper that combines the ideas from stability and PAC-Bayesian theory to obtain generalization bounds. This is not the case, e.g. see [1].
As noted by the authors, some of the bounds obtained in this paper share similarities with one of the bounds in Mou et al.  as all these bounds contain the expected empirical squared gradient norm. The bound in Mou et al. holds with high probability and decays as $O(1/\sqrt{n})$, whereas the bounds in this paper are on expected generalization error and decay as $O(1/n)$. Could authors comment on extending their results to hold with high probability and how it would affect their bounds?
[1] Rivasplata et al., PAC-Bayes bounds for stable algorithms with instance-dependent priors.
----------------------- Post author response -------------
Thank you for addressing my comments. I have decided to keep my original score unchanged.


Despite the great empirical success of deep reinforcement learning, its theoretical foundation is less well understood. In this work, we make the first attempt to theoretically understand the deep Q-network (DQN) algorithm (Mnih et al., 2015)   from both algorithmic and statistical perspectives. In specific, we focus on the fitted Q iteration (FQI) algorithm with deep neural networks, which is a slight simplification of DQN that captures the tricks of experience replay and target network used in DQN. Under mild assumptions, we establish the algorithmic and statistical rates of convergence for the action-value functions of the iterative policy sequence obtained by FQI. In particular, the statistical error characterizes the bias and variance that arise from approximating the action-value function using deep neural network, while the algorithmic error converges to zero at a geometric rate. As a byproduct, our analysis provides justifications for the techniques of experience replay and target network, which are crucial to the empirical success of DQN. Furthermore, as a simple extension of DQN, we propose the Minimax-DQN algorithm for zero-sum Markov game with two players, which is deferred to the appendix due to space limitations.

This paper analyze the off-policy policy improvement algorithm with a very limited sparse ReLU function class. Although some of the results are interesting, I have lots of concern on the motivation of this work. I believe this paper is technically correct, but the authors focus on a very simplified case: they just use samples generated from a fixed sampling policy, which gets rid of the analysis of exploration and sample complexity that is a main focus of the reinforcement learning community. From my point of view, this may not be some analysis of reinforcement learning, at least not for Deep Q-learning, but more likely to be some learning theory of off-policy FQI. The main theorem investigates the statistical error and convergence rate of this problem, which can be of individual interest. But overall, I think the problem the authors want to solve is not a traditional reinforcement learning algorithm, and it is not appropriate to introduce the result as the theoretical analysis of Deep Q-Learning.
Detailed comments:
1. I think the assumption of Sparse ReLU network is too strong and generally not held in practice. Also, the optimization of such kind of network is painful, as the ell_0 constraint makes the optimization problem NP-hard. In other words, I think the authors only handle a very specific case under very ideal condition like assuming an oracles that can return the optimal network each turn.
2. The equivalence between FQI and target network is well-known and may not occupy so many places in Sec 3. Also, the results in Appendix B can be simply derived follows the recent development of neural network optimization. As this may be not the main contribution of this paper, I think it is better to omit these parts to make the paper more neat.
3. Moreover, in appendix B, the authors assumed the function class as two-layer ReLU network, which is different from the assumption in the main text and cannot justify the global convergence of (3.4).
4. It is somewhat strange of assume a sampling distribution, as when we say Q-learning, we want to balance the exploration and exploitation given current estimation Q. Even in Deep Q-Learning, the data are sampled with \epsilon-greedy policy w.r.t the current Q network. This kinds of problems are more like off-policy policy improvement. I think call it the analysis of Deep Q-Learning is somehow not accurate and over-claimed. Maybe better called off-policy policy improvement with deep neural networks.
5. Theorem 4.4 is an interesting result as it shows that the error of the proposed algorithm can be decomposed into the a statistical error which depends on the smoothness of the operator Tf and an algorithm error that depend on the number of iterations. I am wondering what's the main technical differences between this work and [1], as I find the main difference is [1] don't give K-dependent algorithm error, instead assuming K have a order of log 1/epsilon to ensure algorithm error is smaller than \epsilon. I feel it's not so hard to derive a bound that combines statistical error and algorithm error for [1]. Also, FVI in [1] is not in spirit totally different from FQI in this paper given that [1] use the maximum operator over action when do FVI, not take expectation over the target policy. I hope the authors can clarify in their paper.
6. The authors don’t mention much of the target network in the main theorem. I know the generalization to the update with target network is not so hard, but as the authors mentioned so much time in the main text, shall it be better to include the result with target network?
Still, in my opinion, the main theorem has its own value. However, it is not proper to claim as a theoretical analysis of Deep Q-Learning. Also, I feel the function class is too restricted and the optimization issue in the proposed algorithms cannot be simply solved, and the main analysis is similar to [1] with little generalization to Holder smoothness. Thus, I tend to reject this paper.
[1] Munos, Rémi, and Csaba Szepesvári. "Finite-time bounds for fitted value iteration." Journal of Machine Learning Research 9.May (2008): 815-857.


ABSTRACT Multi-Label Learning task simultaneously predicting multiple labels has attracted researchers' interest for its wide application. Metric Learning crucially determines the performance of the k nearest neighbor algorithms, the most popular framework handling the multi-label problem. However, the existing advanced multiple-label metric learning suffers the inferior capacity and application restriction. We propose an extendable and end-to-end deep representation approach for metric learning on multi-label data set that is based on neural networks able to operate on feature data or directly on raw image data. We motivate the choice of our network architecture via a Bidirectional Representation learning where the label dependency is also integrated and deep convolutional networks that handle image data. In multi-label metric learning, instances with the more different labels will be dragged the more far away, but ones with identical labels will concentrate together. Our model scales linearly in the number of instances and trains deep neural networks that encode both input data and output labels, then, obtains a metric space for testing data. In a number of experiments on multi-labels tasks, we demonstrate that our approach is better than related methods based on the systematic metric and its extendability.

The paper addresses the problem of multi-label prediction.  It proposes a method that uses a co-embedding of instances and labels into a joint embedding space in a way that related instances and labels fall close by and unrelated ones fall far away.  For this purpose, embeddings from input space and label space to a common space are learned from training data. At the prediction time, KNN to the embedding of the test instance in the co-embedding space is used to predict relevant labels.
Featurized (attributed)  labels are potentially considered, which can facilitate incorporating label dependence and generalization over unseen labels.
Main shortcomings:
- The novelty of the work is limited. Ideas introduced in this work are present and being investigated in literature for a while now, leading to remaining limited contribution for the paper.  Related work on joint embedding, co-embedding, label-embedding, and zero shot learning seem to be neglected totally. For example, the paper lacks awareness of, citation to and comparison with related work such as [1-5].
- Presentation of the paper can be highly improved. There are several grammatical and writing problems in the paper.
Formulation can also benefit from  improved presentation. See for example Eq (1).
- Technical arguments are not all well founded. For example, the scalability claim in the abstract  of the paper seems to refer to "prediction" time complexity being linear in the number of "training" examples, which is not actually fast.
In summary, based on the above reasons, I vote for the paper to be strongly rejected.
[1] Akata, Zeynep, et al. "Label-embedding for image classification." IEEE transactions on pattern analysis and machine intelligence 38.7 (2015): 1425-1438.
[2] Weston, Jason, Samy Bengio, and Nicolas Usunier. "Wsabie: Scaling up to large vocabulary image annotation." Twenty-Second International Joint Conference on Artificial Intelligence. 2011.
[3] Li, Xin, and Yuhong Guo. "Bi-directional representation learning for multi-label classification." Joint European conference on machine learning and knowledge discovery in databases. Springer, Berlin, Heidelberg, 2014.
[4] Mirzazadeh, Farzaneh, et al. "Scalable metric learning for co-embedding." Joint European Conference on Machine Learning and Knowledge Discovery in Databases. Springer, Cham, 2015.
[5] Yeh, Chih-Kuan, et al. "Learning deep latent space for multi-label classification." Thirty-First AAAI Conference on Artificial Intelligence. 2017.


We introduce ES-MAML, a new framework for solving the model agnostic meta learning (MAML) problem based on Evolution Strategies (ES). Existing algorithms for MAML are based on policy gradients, and incur significant difficulties when attempting to estimate second derivatives using backpropagation on stochastic policies. We show how ES can be applied to MAML to obtain an algorithm which avoids the problem of estimating second derivatives, and is also conceptually simple and easy to implement. Moreover, ES-MAML can handle new types of nonsmooth adaptation operators, and other techniques for improving performance and estimation of ES methods become applicable. We show empirically that ES-MAML is competitive with existing methods and often yields better adaptation with fewer queries.

The paper proposes ES for the task of Model agnostic meta learning. Instead of the gradient-approximation which requires computing a hessian matrix, MC samples from a search distribution are used to estimate a search direction. The approach is validated on a number of experiments.
Unfortunatly, I am unable to accept this paper for a number of reasons. Mainly that the ES used is inferior and the constant step-size used can have a major effect on the experimental outcome.
Almost all proper ES literature with real working ES algorithms are missing and ESGrad is more than 20 years behind SOTA in the field. Since ES are central to the paper, an algorithm that would not even be considered a baseline at any conference in that field is difficult to accept.
The reason for this is that nowadays all ES use dynamic sample-variances based on progress measures, e.g. Cumulative step-size adaptation and Two-Point-Adaptation as the SOTA. Without this, it can be very difficult to find reasonable solutions.

Most important missing references from the ES-field in this context:

1. and most importantly The original ES-based RL paper:
Heidrich-Meisner, Verena, and Christian Igel. "Neuroevolution strategies for episodic reinforcement learning." Journal of Algorithms 64.4 (2009): 152-168.

2. CMA-ES and NES
Hansen, N., Müller, S. D., & Koumoutsakos, P. (2003). Reducing the time complexity of the derandomized evolution strategy with covariance matrix adaptation (CMA-ES). Evolutionary computation, 11(1), 1-18.
Krause, O., Arbonès, D. R., & Igel, C. (2016). CMA-ES with optimal covariance update and storage complexity. In Advances in Neural Information Processing Systems (pp. 370-378).
Wierstra, D., Schaul, T., Peters, J., & Schmidhuber, J. (2008, June). Natural evolution strategies. In 2008 IEEE Congress on Evolutionary Computation (IEEE World Congress on Computational Intelligence) (pp. 3381-3387). IEEE.
3. Review of SOTA in large-scale ES:
Varelas, K., Auger, A., Brockhoff, D., Hansen, N., ElHara, O. A., Semet, Y., ... & Barbaresco, F. (2018, September). A comparative study of large-scale variants of CMA-ES. In International Conference on Parallel Problem Solving from Nature (pp. 3-15). Springer, Cham.
4. Recent developments for noisy functions (also references other relevant algorithms with noise-handling)
Krause, O. (2019, July). Large-scale noise-resilient evolution-strategies. In Proceedings of the Genetic and Evolutionary Computation Conference (pp. 682-690). ACM.
Section 3.2
-Why should in (7) the same sigma be used as in (6)? Sigma, alpha etc should be learnable parameters learned by the outer ES.
-3.3.2: you are writing below (1) that rollouts come from a distribution, i.e. are stochastic. How would you implement a hill-climber in the stochastic setting? e.g. consider the case when the rewards are heavy-tailed.
- using a hill-climber goes completely against the SOTA in ES which showed repeatedly over the last 20 years that hill-climbing is inferior, especially in larger dimension search-spaces (>100).
Experiments:
- I am not an expert of MAML, but i would not consider this as different tasks, just as different environments for the same task. i.e. a circular running strategy should be optimal for all environments. but when considering different tasks, we would consider different policies to be optimal.
- The experiments use the same hyper parameters for all variants. However, i am not sure this is a fair comparison. E.g. HC has way more spread over the search-space than the other two methods for a given sigma, with following sample steps allowing for fixing the "too large" or "too small" spread.
Since the graph of the objective function is flat in a large area of the search space, the additional exploration through stocasticity alone might explain the results of Figure 1. In this case, the result would be pretty artificial, because real ES would adapt their step-size.
- Similar holds for the number of samples used by the outer ES (n, but named differently in th appendix?). The gradient-based approaches might require a lot more initial points with a smaller K , especially on the flat surfaces of the objectives.
- In Figure 3, middle image, why does the green curve appear to have decreasing performance after iteration 200?
- Figure 3/ 4.2 why do the three settings have different values for number of iterations and K? Why does L-DPP only appear in the third task?
-Section 4.3 and Figure 4: why is there no L-PG and HH-ES? the only curve which is is available for both algorithms has the same performance.


Although few-shot learning research has advanced rapidly with the help of metalearning, its practical usefulness is still limited because most of the researches assumed that all meta-training and meta-testing examples came from a single domain. We propose a simple but effective way for few-shot classification in which a task distribution spans multiple domains including previously unseen ones during meta-training. The key idea is to build a pool of models, each of which is responsible for a different group of tasks, and to learn to select the best one for a particular task through multi-domain meta-learning. This reduces task-specific adaptation over a complex task distribution to a simple selection problem rather than modifying the model with a number of parameters at meta-testing time. Inspired by common multi-task learning techniques, we let all models in the pool share a base network and add a separate modulator to each model to refine the base network in its own way. This architecture allows the pool to maintain representational diversity and each model to have domain-invariant representation as well. Experiments show that our selection scheme outperforms other few-shot classification algorithms when target tasks come from many different domains. They also reveal that aggregating outputs from all constituent models is effective for tasks from unseen domains indicating the effectiveness of our framework.

###Summary###
This paper aims to tackle few-shot classification with many different domains. The idea is to build a pool of embedding models, which are based on the same base network. The models are diversed by their own modulators. The high-level intuition is to let the model pool capture good domain-invariant features by the shared parameters and domain-specific features by the selection network, which is desirable to represent the complex cross-domain task distribution, without a significant increase in the number of parameters.
The model includes the embedding networks f_E and a selection network f_S. The overall training process of the proposed method is:
(1) Train the base embedding network f_E with the aggregated dataset from multiple domains.
(2)Sample one domain to optimize the corresponding embedding model.
(3) Build a selection network through cross-domain episodic training. The task representation is calculated by average the embedding vectors from the base network, which resembles the prototype. Then the model with the highest accuracy on the query set is selected to compute the labels for the target domain.
For the embedding model f_E, the paper provides two architectures, one with convolution 1x1 and the other with Channel-wise transform, denoted as DoS and DoS-CH, respectively. Instead of selecting the model which has the highest accuracy, the averaging model generates an output by averaging class prediction probabilities of all constituent models, denoted by DoA and DoA-CH, respectively.
 The baseline used in this paper includes Fine-tune, Simple-Avg, ProtoNet, FEAT, ProtoMAML.
The paper performs extensive experiments on a batch of datasets, including Aircraft, CIFAR100, DTD, GTSRB, ImageNet12, Omniglot, SVHN, UCF101, and Flowers. The 5-way 1 shot and 5 way 5 shot experimental results demonstrate that the proposed DoS and DoS-CH can outperform other baselines in the "see domains" setting. However, the results on "unseen domains" experiments are worse than the averaging baselines (DoA, DoA-Ch).
The paper also surveys the few-shot classification results on a varying number of source domains to show that DoA and DoA-ch are robust to deal with different settings.
### Novelty ###
The paper composes of two sub-network, with one baseline network to extract the commonsense knowledge for different datasets and another selection network to select the best model from the model pools for each specific domain.
The idea of leveraging multiple modulators for domain-agnostic image recognition is interesting and heuristic, thus the proposed framework shows some novelty.
###Clarity###
Overall, the paper is readable and logically clear. The images are well-presented and well-explained by the captions and the text.
While this paper misses many prior works both in the track of domain-agnostic learning and few-shot learning. I would recommend the authors to the following materials:
a). Domain Agnostic Learning with Disentangled Representations, ICML 2019. https://arxiv.org/pdf/1904.12347.pdf
b). Generalizing from a Few Examples: A Survey on Few-Shot Learning,
https://arxiv.org/pdf/1904.05046.pdf
There also exist some grammar mistakes and typoes in the paper. It will be better to revise and polish the paper.
###Pros###
1) The paper proposes a framework that includes two parts, i.e. the base network and the selection network. The idea is to make deep models more robust to different image domains, which is interesting and heuristic.
2) The paper provides extensive experiments on a wide range of datasets. The experiments demonstrate the effectiveness of the proposed method.
3) The paper is applicable to many practical scenarios since the data from the real-world application is complicated.
###Cons###
1) The critical component of the proposed method is the selection network. However, the experiments on the "novel domains" show that the proposed DoS and DoS-Ch works worse than just averaging the outputs of the models in the model pool.
2) The paper should explain the details about the baseline experiments. Since the proposed models have much more parameters than the baselines, what's the effect of the auxiliary parameters? Is the comparison between the baselines and the proposed method fair?
3) The presentation of the proposed paper should be polished. Many critical techniques used in this paper are not well-explained, such as the ProtoNet. There exist some grammar mistakes and typos that need a revision.
Based on the summary, cons, and pros, the current rating I am giving now is weak reject. I would like to discuss the final rating with other reviewers, ACs.


Graph-based semi-supervised learning (SSL) assigns labels to initially unlabelled vertices in a graph. Graph neural networks (GNNs), esp. graph convolutional networks (GCNs), inspired the current-state-of-the art models for graph-based SSL problems. GCNs inherently assume that the labels of interest are numerical or categorical variables. However, in many real-world applications such as coauthorship networks, recommendation networks, etc., vertex labels can be naturally represented by probability distributions or histograms. Moreover, real-world network datasets have complex relationships going beyond pairwise associations. These relationships can be modelled naturally and flexibly by hypergraphs. In this paper, we explore GNNs for graph-based SSL of histograms. Motivated by complex relationships (those going beyond pairwise) in real-world networks, we propose a novel method for directed hypergraphs. Our work builds upon existing works on graph-based SSL of histograms derived from the theory of optimal transportation. A key contribution of this paper is to establish generalisation error bounds for a one-layer GNN within the framework of algorithmic stability. We also demonstrate our proposed methods' effectiveness through detailed experimentation on real-world data. We have made the code available.

Contributions:
1. This paper proposes a semi-supervised graph neural network method for hypergraphs.
2. A generalization error bound was proposed adapted to the semi-supervised setting with Wasserstein's loss.
3. Empirical results demonstrate the effectiveness of the proposed method.
The algorithmic contribution of this paper is clear: it proposes a new network architecture that (1) initialize latent features H^{(0)}_E for hyperedges from a "hyperedge graph" and (2) learn latent features for each node and hyperedge using a GCN type network. Since the latent features are formulated as discrete distributions, a Wasserstein distance can be applied for training with the sinkhorn approximation algorithm.
I think the weakness of this paper is two folds, which makes it not ready to publish. First, the paper claims that the performance gain results from the exploitation of directed hyperedges. This is reasonable if I barely see the results for Soft-DHN. However, I find the design of the hyperedge GNN to be fragile regarding the number of layers from the result in Table 4. Normally it is reasonable to use a two-layer GNN, but the result is very bad when doing so (see Soft-DHN 2 layers result in Sec. A.4.1). Also, the result of the proposed model is still good when not applying a hypergraph GNN. So I'm confused about where the performance gain comes from.
Another weakness is the paper writing. Below are a few comments:
[Page 3, Sec. 3.1] By saying "t\neq\Phi", do you mean "t\neq\emptyset"?
[Page 3, Sec. 3.2] Please introduce the notation (M, C) right after it first appears in the third line of Sec. 3.2.
[Page 3, Sec. 3.2] The notation n,m are confusing. Do you mean m=|E|?
[Page 3, Sec. 3.2] I cannot see how Z=h(\mathcal{H},X_V,X_E) maps each vertex to a probability distribution. Do you mean each row of Z is a probability distribution for each vertex?
[Page 4, Sec. 3.2] Please define E_D. Is E_D the same as E_d?
[Page 4, Sec. 3.3] Be explicit by saying "approximate the hypergraph by a suitable graph", what do you mean by "suitable graph"? Does it mean use cliques in place of hyperedges?
[Page 4, Sec. 3.3] t=0,...,\tau-1
Since the writing significantly affects the paper readability, and the core contribution of the paper seems incremental, I will vote for a reject to this paper.


Recent studies at the intersection of physics and deep learning have illustrated successes in the application of deep neural networks to partially or fully replace costly physics simulations. Enforcing physical constraints to solutions generated by neural networks remains a challenge, yet it is essential to the accuracy and trustworthiness of such model predictions. Many systems in the physical sciences are governed by Partial Differential Equations (PDEs). Enforcing these as hard constraints, we show, are inefficient in conventional frameworks due to the high dimensionality of the generated fields. To this end, we propose the use of a novel differentiable spectral projection layer for neural networks that efficiently enforces spatial PDE constraints using spectral methods, yet is fully differentiable, allowing for its use as a layer in neural networks that supports end-to-end training. We show that its computational cost is cheaper than a regular convolution layer. We apply it to an important class of physical systems -incompressible turbulent flows, where the divergence-free PDE constraint is required. We train a 3D Conditional Generative Adversarial Network (CGAN) for turbulent flow superresolution efficiently, whilst guaranteeing the spatial PDE constraint of zero divergence. Furthermore, our empirical results show that the model produces realistic flow fields with more accurate flow statistics when trained with hard constraints imposed via the proposed novel differentiable spectral projection layer, as compared to soft constrained and unconstrained counterparts.

The paper describes a way to efficiently enforce physical constraints expressed by linear PDEs on the output of a neural network. The idea is to have, as a last layer of the network, a projection onto the constrained solution space, and to back-propagate through it. That projection layer is made efficient for high-dimensional outputs via the fast Fourier transform (FFT), exploiting a well-known numerical trick. Importantly, the proposed strategy is very general, and can indeed be used with any PDE constraint that is a linear combination of differential operators.
First, I would like to mention to the AC that the authors apparently forgot to run bibtex. The submitted paper had no bibliography and all references are "?". So technically the original submission was incomplete and would probably have to be rejected, on the grounds that it was impossible to check whether the literature references are appropriate. The authors have rectified this through a comment pointing to an (anonymous) version with references. For me, this is ok.
The research direction of the paper is a hot topic: how to reconcile data-driven deep learning with analytic physical models is, arguably, one of the big research questions holding back the wide-spread use of deep networks in several natural sciences (e.g., environmental and climate science, hydrology, etc.). Work in this direction could have a significant impact, and the approach developed the paper is rather general and, as far as I can tell, correct.
The experiments use the rather challenging task of super-resolving turbulent flow, subject  to the Navier-Stokes constraints. Experiments are run on synthetic data from the JHTD simulation dataset. This is hard to avoid, since dense ground truth for flow fields is impossible to obtain, nevertheless it  would have been more convincing to also show at least qualitative results for a real flow dataset. It is also not completely satisfactory that the experiments must use even as OUTput a significantly downscaled version of the dataset with a resolution that would not be tremendously useful in practice - this implicitly acknowledges that the paper, while reducing the computational cost compared to the direct projection method, did actually not overcome the most critical bottleneck one faces when combining high-dimensional physics simulation with deep learning: namely, that it is not tractable to simply store physical fields as explicit 3D/4D voxel grids if one wants to work with them on the GPU.
Another slightly bothering choice in the experiments is to compare only distributions. While I see the point that the prediction is, by its nature, ambiguous; I still think one should look at both flow statistics and at the actual flow field difference. Ambiguity is not a very convincing justification to not even try to predict correctly - by the author's definition any super-resolution task is ambiguous, still one aims, for instance, to recover the correct image, not just a plausible one. In a sense it is the whole point of prior knowledge (which the PDEs are in a learning context) to bring the solution closer to the right answer, when the data alone cannot do the job. In practice, a prediction that is very close to the true flow field, but ever so slightly violates the physical constraints is often more useful than one that strictly satisfies the constraints, but is way off.
One comment on the presentation: I feel that the discussion of soft constraints could be more extensive and more balanced. I agree that they are less principled, they do not devalue the present work. But we know, both for variational methods and for pre-deep learning methods, that soft constraints work rather well in practice, especially with an adaptive weight that gradually tightens the constraints. So it would be in order to not just dismiss the alternative in a half-sentence, but to give it proper consideration - especially since in terms of dissipation, it actually performs better than the hard constraint in the experiments.
Overall, I find the topic important and the presented work is a sensible and nicely generic step forward. On the negative side, the paper does not fully deliver on the promise to make physics constraints in deep networks usable in practice. My rating reflects my impression that the bibliography and references are probably correct - this should be checked before reaching a final decision.


As observed in previous works, many word embedding methods exhibit two interesting properties: (1) words having similar semantic meanings are embedded closely; (2) analogy structure exists in the embedding space, such that "Paris is to France as Berlin is to Germany". We theoretically analyze the inner mechanism leading to these nice properties. Specifically, the embedding can be viewed as a low rank transformation from the word-context co-occurrence space to the embedding space. Such embedding transformation will preserve the relative distances among words. Furthermore, previous studies empirically observed that the parameter α has a strong influence on the performance of word embedding but did not provide theoretical explanation. We provide a theoretical explanation for this behavior, and derive a method to automatically find its optimal value. The experiments on real datasets verify our analysis.

This paper explores the role of the implicit alpha parameter when learning word embeddings. More concretely, word embeddings work by either implicitly or explicitly factorizing a co-occurrence matrix, and the underlying parameter alpha controls how the singular values are weighted between the word and the context vectors. The authors provide theoretical insights on the role of alpha in relation with the original co-occurrence matrix, and propose a new method to find its optimal value.
I think that this is overall a solid work. The paper provides a new perspective in the workings of word embeddings that I find interesting, it is theoretically well-founded (although I did not check all derivations in detail), and the presentation is clear.
However, I think that the paper does a poor job in putting its contributions into context in relation to previous work. In particular, the role of alpha in word embeddings was already studied empirically by Artetxe et al. (CoNLL'18, https://www.aclweb.org/anthology/K18-1028.pdf) for both word analogy and word similarity tasks, to the extent that Figure 2 in both papers is showing the exact same curves. However, the authors do not even cite it. As acknowledged in the paper, other authors like Levy et al. (TACL'15) also observed that the value of alpha was important in their experiments.
I think that the right narrative for the paper should more in the line of "previous work showed that alpha behaves this and that way; we provide a theoretical explanation for this behavior, and derive a method to automatically find its optimal value". However, starting from the title ("word embedding re-examined: is the symmetrical factorization optimal?", when it was already known that it wasn't) and the abstract (where only the statement that "we propose a method to find the optimal alpha" corresponds to a novel contribution), the paper does a poor job in identifying and properly contextualizing its real contributions. More importantly, the paper does not try to establish any connection between the authors own theory and the empirical findings from previous work.
In terms of the actual content, the authors constantly claim that word2vec is performing a symmetric factorization (e.g. "the original word2vec is implicitly performing a symmetric factorization, thus implying the alpha equal to 0.5) as if it was something obvious or well-known. I might be missing something here, but I do not see why this is the case. Following your notation, let's say that word2vec is implicitly factorizing M = E*C^T, where E are the word embeddings and C are the context embeddings. One could multiply E with any arbitrary invertible matrix W, and C by the transpose of its inverse W^-T, which could be chosen to completely break any symmetry, yet the objective value of word2vec would not change at all, as (E*W)*(C*W^-T)^T = E*C^T. In other words, there is nothing in the training objective of word2vec that forces a symmetric factorization, and there is always an optimal solution with respect to this training objective that is arbitrarily asymmetric.
Another point that raises concerns to me is that the optimal value of alpha is determined by the vocabulary of the evaluation task. It would make sense if the optimal alpha depended on the nature of the task (e.g. syntactic vs semantic), but I do not have any intuition (nor do the authors provide) as of why the vocabulary would be anyhow relevant. More importantly, this does not seem generalizable beyond a few intrinsic tasks as, in the general case, one wants good embeddings for the full vocabulary. In either case, I think that this point deserves more attention in the paper.
I also find the experimental evaluation to be somewhat weak. In particular, the proposed theory focuses in two phenomena (word similarity and word analogy) as stated in the abstract itself, but the empirical evaluation is limited to the word similarity task.
Also, this is a minor detail and it did not influence my score, but I dislike that the authors use "word2vec" to refer to skip-gram with negative sampling throughout the paper. I would suggest to either use SGNS (which is quite standard) or simply skip-gram.


Two important topics in deep learning both involve incorporating humans into the modeling process: Model priors transfer information from humans to a model by regularizing the model's parameters; Model attributions transfer information from a model to humans by explaining the model's behavior. Previous work has taken important steps to connect these topics through gradient regularization, but we find that these methods do not successfully align a model's behavior with human intuition. We develop an efficient and theoretically grounded feature attribution method, expected gradients, and a novel framework, attribution priors, to enforce prior expectations about a model's behavior during training. We demonstrate that attribution priors are broadly applicable in three different domains: image data, gene expression data, and health care data. Our experiments show that models trained with attribution priors are more intuitive and achieve better generalization performance than both equivalent baselines and existing methods to regularize model behavior.

Summary.
The paper improves the existing feature attribution method by adding regularizers to enforce (human) expectations about a model’s behavior. Three different datasets (i.e. image, gene expression, health-care) are chosen to evaluate the proposed model’s effectiveness, while different regularizers (i.e. image prior, graph prior, and sparsity prior) are explored for the respective task.
Strengths.
1. Incorporating human knowledge into the model has a growing interest in ML / CV communities.
2. Three datasets from different domains (i.e. image classification data, gene expression data, and health care data) are used to evaluate the effectiveness of the proposed approach. Data shows that the proposed approach shows better generalization performance (i.e. better performance in test dataset) than baselines.
3. The paper provides well-documented supplemental materials that contain details of the experimental setting and additional supporting figures.
Weaknesses.
1. Task-specific heuristic human prior
I agree (and personally like) the motivation that a method is needed to align a model’s behavior with human knowledge or intuition -- model’s behavior may be explained by feature attribution methods while making models accept human knowledge is challenging. However, such an ability is achieved by simply adding task-specific heuristic functions as a penalty or a regularizer. Also, the introduced human priors are similar to general regularization conventions, i.e. a penalty of smoothness over adjacent pixels is commonly used in the CV community. I am concerned that only a limited set of expert-invented human priors can be used in this approach.
Further, feature attribution methods aim to develop a richer notion of the contribution of a pixel to the output. However, the difficulty would be the lack of formal measures of how the network output is affected by spatially-extended features (rather than pixels). The explored priors (e.g. a total variation loss to make neighboring pixels have a similar impact on the final verdict) actually relieve this issue.
2. Incorporating humans into the modeling process?
A key motivation behind this work is “incorporating humans into the modeling process”. This would imply that (human-understandable) information needs first to be transferred from a model to humans. However, I am concerned about what information end-users are expected to obtain from the model. For example, Figure 1 (left) shows an attribution map that highlights multiple intermittent regions from which I cannot understand its behavior. Unless end-users cannot understand the model’s behavior, how can we expect humans can provide knowledge to model? A user study would be needed to support that the proposed method can really provide a way to incorporate humans into the modeling process.
Minor comments.
1. Plots in Figure 3 are not intuitively understandable.
2. There is no section Conclusion.
3. A template for the reference section looks different from other ICLR papers.