RIEMANNIAN ADAPTIVE OPTIMIZATION METHODS
Several first order stochastic optimization methods commonly used in the Euclidean domain such as stochastic gradient descent (SGD), accelerated gradient descent or variance reduced methods have already been adapted to certain Riemannian settings. However, some of the most popular of these optimization tools − namely ADAM, ADAGRAD and the more recent AMSGRAD − remain to be generalized to Riemannian manifolds. We discuss the difficulty of generalizing such adaptive schemes to the most agnostic Riemannian setting, and then provide algorithms and convergence proofs for geodesically convex objectives in the particular case of a product of Riemannian manifolds, in which adaptivity is implemented across manifolds in the cartesian product. Our generalization is tight in the sense that choosing the Euclidean space as Riemannian manifold yields the same algorithms and regret bounds as those that were already known for the standard algorithms. Experimentally, we show faster convergence and to a lower train loss value for Riemannian adaptive methods over their corresponding baselines on the realistic task of embedding the WordNet taxonomy in the Poincaré ball.

I have enjoyed reading this paper. The paper is accessible in most cases and provides a novel optimization technique. Having said this, I have a few concerns here,
- I am not sure why the notion of product manifolds is required in developing the technique. To me, all the arguments follow without that. Even if the authors are only interested in manifolds that can be constructed in a product manner (say R^n from R),  the development can be done without explicitly going along that path. Nevertheless I may have missed something so please elaborate why product manifolds. I have to add that in many cases, the underlying Riemannian geometry cannot be derived as a product  space. For example, the SPD manifold cannot be constructed as a product space of lower dimensional geometries.
- I have a feeling that finding the operator \Pi in many interesting cases is not easy. Given the dependency of the developments on this operator, I am wondering if the method can be used to address problems on other manifolds such as SPD, Grassmannian or Stiefel. Please provide the form of this operator for the aforementioned manifolds and comment on how the method can be used if such an operator is not at our disposal.
- While I appreciate the experiments done in the paper,  common tests (e.g., Frechet means) are not presented in the paper (see my comment below as well).
- last but not least, the authors missed the work of   Roy et. al., "Geometry Aware Constrained Optimization Techniques for Deep Learning", CVPR'18 where RSGC with momentum and Riemannian version of RMSProp are developed. This reference should be considered and compared.
Aside from the above, please
- define v and \hat{v} for Eq.(5)
- provide a reference for the claim at l3-p4 (claim about the gradient and Hessian)
- maybe you want to mention that \alpha -> 0 for |g_t^i| at the bottom of p4
- what does [.] mean in the last step of the algorithm presented in p7
- what is the dimensionality of the Hn in the experiments


As deep learning applications are becoming more and more pervasive, the question of evaluating the reliability of a prediction becomes a central question in the machine learning community. This domain, known as predictive uncertainty, has come under the scrutiny of research groups developing Bayesian approaches to deep learning such as Monte Carlo Dropout. Unfortunately, for the time being, the real goal of predictive uncertainty has been swept under the rug. Indeed, Bayesian approaches are solely evaluated in terms of raw performance of the prediction, while the quality of the estimated uncertainty itself is not assessed. One contribution of this article is to draw attention on existing metrics developed in the forecast community, designed to evaluate both the sharpness and the calibration of predictive uncertainty. Sharpness refers to the concentration of the predictive distributions and calibration to the consistency between the predicted uncertainty level and the actual errors. We further analyze the behavior of these metrics on regression problems when deep convolutional networks are involved and for several current predictive uncertainty approaches. A second contribution of this article is to propose an alternative metric that is more adapted to the evaluation of relative uncertainty assessment and directly applicable to regression with deep learning. This metric is evaluated and compared with existing ones on a toy dataset as well as on the problem of monocular depth estimation.

The authors propose mean rescaled confidence interval (MERCI) as a way to measure the quality of predictive uncertainty for regression problems. The main idea is to rescale confidence intervals, and use average width of the confidence intervals as a measure for calibration. Due to the rescaling, the MERCI score is insensitive to the absolute scale; while this could be a feature in some cases, it can also be problematic in applications where the absolute scale of uncertainty matters.
Overall, the current draft feels a bit preliminary. The current draft misses discussion of other relevant papers, makes some incorrect claims, and the experiments are a bit limited. I encourage the authors to revise and submit to a different venue.

There’s a very relevant ICML 2018 paper on calibrating regression using similar idea:
Accurate Uncertainties for Deep Learning Using Calibrated Regression
https://arxiv.org/pdf/1807.00263.pdf
Can you clarify if/how the proposed work differs from this? I’d also like to see a discussion of calibration post-processing methods such as Platt scaling and isotonic regression.
The paper unfairly dismisses prior work by making factually incorrect claims, e.g. Section 2 claims
“Indeed, papers like (Hernandez-Lobato & Adams, 2015; Gal & Ghahramani, 2016; Lakshminarayanan et al., 2017; Kendall & Gal, 2017) propose quantitative evaluations on several datasets, those classically used for evaluating the task, but only compare their average test performances in terms of RMSE. It is the quality of the prediction which is measured, and not the quality of the estimated uncertainty. They also show some qualitative results, where maps of the estimated uncertainty are displayed as images and visually evaluated. Yet, to the best of our knowledge, the literature on deep neural networks does not propose any method for the quantitative evaluation of the uncertainty estimates.”
This is incorrect.  To just name a few examples of prior work quantitatively evaluating the quality of uncertainty: (Hernandez-Lobato & Adams, 2015) and (Gal & Ghahramani, 2016) report log-likelihoods on regression tasks, (Lakshminarayanan et al. 2017) report log-likelihoods and Brier score on classification and regression tasks. There are many more examples.
The experiments are a bit limited. Figure 1 is a toy dataset and Table 2 / Figure 4 focus on a single test case which does not seem like a fair comparison of the different methods. The authors should at least compare their method to other work on the UCI regression benchmarks used by (Hernandez-Lobato & Adams, 2015).


Feature attribution methods identify "relevant" features as an explanation of a complex machine learning model. Several feature attribution methods have been proposed; however, only a few studies have attempted to define the "relevance" of each feature mathematically. In this study, we formalize the feature attribution problem as a feature selection problem. In our proposed formalization, there arise two possible definitions of relevance. We name the feature attribution problems based on these two relevances as Exclusive Feature Selection (EFS) and Inclusive Feature Selection (IFS). We show that several existing feature attribution methods can be interpreted as approximation algorithms for EFS and IFS. Moreover, through exhaustive experiments, we show that IFS is better suited as the formalization for the feature attribution problem than EFS.

This paper formulates feature attribution from a feature selection perspective, and compares EFS (Exclusive Feature Selection) and IFS (Exclusive Feature Selection), which shows IFS is a better fit for feature attribution.
[+] The paper is well-structured and the proposed approach is clearly presented.
[-] It would helpful if the author could discuss the time complexity of proposed methods and compare the running time with baseline methods in evaluation.
[-] My major concern on this paper is the significance, as the contribution of the paper seems to be very limited.
    1) Formalizing the feature attribution problem as a feature selection problem is straightforward. IFS and EFS are just Forward and Backward stepwise feature selection, which are classic feature selection schemes. Applying them to feature attribution/saliency map does not seem to have much technical contribution.
    2) One claimed contribution of this paper is that existing feature attribution methods can be viewed as approximation of IFS and EFS. However, this contribution also seems to be minor. As many feature selection methods are known to be approximation of backward or forward stepwise feature selection, it is straightforward to show the connection between other feature attribution methods and IFS/EFS.
In conclusion, I would recommend to reject this paper due to the limited novelty and technical contribution.


NEURAL PROBABILISTIC MOTOR PRIMITIVES FOR HUMANOID CONTROL
We focus on the problem of learning a single motor module that can flexibly express a range of behaviors for the control of high-dimensional physically simulated humanoids. To do this, we propose a motor architecture that has the general structure of an inverse model with a latent-variable bottleneck. We show that it is possible to train this model entirely offline to compress thousands of expert policies and learn a motor primitive embedding space. The trained neural probabilistic motor primitive system can perform one-shot imitation of whole-body humanoid behaviors, robustly mimicking unseen trajectories. Additionally, we demonstrate that it is also straightforward to train controllers to reuse the learned motor primitive space to solve tasks, and the resulting movements are relatively naturalistic. To support the training of our model, we compare two approaches for offline policy cloning, including an experience efficient method which we call linear feedback policy cloning. We encourage readers to view a supplementary video summarizing our results.

This paper considers the problem of transferring motor skills from multiple experts to a student policy. To this end, the paper proposes two approaches: (1) an approach for policy cloning that learns to mimic the (local) linear feedback behavior of an expert (where the expert takes the form of a neural network), and (2) an approach that learns to compress a large number of experts via a latent space model. The approaches are applied to the problem of one-shot imitation from motion capture data (using the CMU motion capture database). The paper also considers an extension of the proposed approach to the problem of high-level planning; this is done by treating the learned latent space as a new action space and training a high-level policy that operates in this space.
Strengths:
S1. The supplementary video was clear and helpful in understanding the setup.
S2. The paper is written in a generally readable fashion.
S3. The related work section does a thorough job of describing the context of the work.
However, I have some significant concerns with the paper. These are described below.
Significant concerns:
C1. My biggest concern is that the paper does not make a strong case for the benefits of LPFC over simpler strategies. The results in Figure 3 demonstrate that a linear feedback policy computed along the expert's nominal trajectory performs as well as (and occasionally even better than) LPFC. This is quite concerning.
C2. Moreover, as the authors themselves admit, "while LPFC did not work quite as well in the full-scale model as cloning from noisy rollouts, we believe it holds promise insofar as it may be useful in rollout-limited settings...". However, the paper does not present any theoretical/experimental evidence that would suggest this.
C3. Another concern has to do with the two-step procedure for LPFC (Section 2.2), where the first step is to learn an expert policy (in the form of a neural network) and the second step is to perform behavior cloning by finding a policy that tries to match the local behavior of the expert (i.e., finding a policy that attempts to produce similar actions as the expert policy linearized about the nominal trajectory). This two-step procedure seems unnecessary; the paper does not make a case for why the expert policies are not chosen as linear feedback controllers (along nominal trajectories) in the first place.
C4. The linearization of the expert policy produced in (1) may not lead to a stabilizing feedback controller and could easily destabilize the system. It is easy to imagine cases where the expert neural network policy maintains trajectories of the system in a tube around the nominal trajectory, but whose linearization does not lead to a stabilizing feedback controller. Do you see this in practice? If not, is there any intuition for why this doesn't occur? If this doesn't occur in practice, this would suggest that the expert policies are not highly nonlinear in the neighborhood of states under consideration (in which case, why learn neural network experts in the first place instead of directly learning a linear feedback controller as the expert policy as suggested in C3?)
C5. I would have liked to have seen more implementation details in Section 3. In particular, how exactly was the linear feedback policy along the expert's nominal trajectory computed? Is this the same as (2)? Or did you estimate a linear dynamical model (along the expert's nominal trajectory) and then compute an LQR controller? More details on the architecture used for the behavioral cloning baseline would also have been helpful (was this a MLP? How many layers?)
Minor comments:
- There are some periods missing at the end of equations (eqs. (1), (2), (6), (8), (9)).


Large mini-batch parallel SGD is commonly used for distributed training of deep networks. Approaches that use tightly-coupled exact distributed averaging based on AllReduce are sensitive to slow nodes and high-latency communication. In this work we show the applicability of Stochastic Gradient Push (SGP) for distributed training. SGP uses a gossip algorithm called PushSum for approximate distributed averaging, allowing for much more loosely coupled communications which can be beneficial in high-latency or high-variability scenarios. The tradeoff is that approximate distributed averaging injects additional noise in the gradient which can affect the train and test accuracies. We prove that SGP converges to a stationary point of smooth, non-convex objective functions. Furthermore, we validate empirically the potential of SGP. For example, using 32 nodes with 8 GPUs per node to train ResNet-50 on ImageNet, where nodes communicate over 10Gbps Ethernet, SGP completes 90 epochs in around 1.6 hours while AllReduce SGD takes over 5 hours, and the top-1 validation accuracy of SGP remains within 1.2% of that obtained using AllReduce SGD.

# overview
This paper leverages a consensus based approach for computing and communicating approximate gradient averages to each node running a decentralized version of stochastic gradient descent.
Though the PushSum idea isn't new, its application to distributed SGD and corresponding convergence analysis represents a valuable contribution, and the experimental results indicate a potentially large speedup (in highly variable or latent networks) without substantially sacrificing model accuracy.
The paper itself is reasonably comprehensive but does miss out on comparisons with more recent but equally promising approaches, namely AD-PSGD.
# pros
* Empirically shown to be significantly faster than SGD, D-PSGD in high-latency, communication bound configurations which is a fairly common real-world setup. There is an accuracy tradeoff at work here, but performance doesn't seem to suffer too much as the node count scales.
* introduces and proves theoretical guarantees for SGP approximate distributed average convergence for smooth, non-convex case, including upper bounded convergence rates.
# cons
* biggest criticism is that AD-PSGD from Lian et al 2018 is not included in experimental comparisons even though the paper is referenced. Authors state that asynchronous methods typically generalize worse than their synchonous counterparts but that isn't what Lian et al found in their comparison with D-PSGD (see table 2 and 3 from their paper). This comparison would be particularly interesting as AD-PSGD also performs well in the high network latency regime that SGP is touted for.
* would've liked to see comparison on other tasks beyond just image classification on ResNet.
# other comments
* Didn't see mention of specific iteration count value(s) K used in experiments or hyperparameters A.3. Since it bounds the convergence rate, this would be important to include.
* Found a few small typos:
  * pg. 5: Relatively -> Relative
  * pg. 7, fig. 2: part -> par
  * pg. 8, sec. 5.3. par. 2: achieves -> achieved
  * pg. 8, sec. 5.3, par. 2: "neighbors also to increases" (drop "to")
  * pg. 12, sec. A.2: "send-buffer to filled" -> "send-buffer to be filled"


PRECONDITIONER ON MATRIX LIE GROUP FOR SGD
We study two types of preconditioners and preconditioned stochastic gradient descent (SGD) methods in a unified framework. We call the first one the Newton type due to its close relationship to the Newton method, and the second one the Fisher type as its preconditioner is closely related to the inverse of Fisher information matrix. Both preconditioners can be derived from one framework, and efficiently estimated on any matrix Lie groups designated by the user using natural or relative gradient descent minimizing certain preconditioner estimation criteria. Many existing preconditioners and methods, e.g., RMSProp, Adam, KFAC, equilibrated SGD, batch normalization, etc., are special cases of or closely related to either the Newton type or the Fisher type ones. Experimental results on relatively large scale machine learning problems are reported for performance study.

This paper proposes a preconditioned SGD method where the preconditioner is adapted by performing some type of gradient descent on some secondary objective "c".  The preconditioner lives in one of a restricted class of invertible matrices (e.g. symmetric, diagonal, Kronecker-factored) constituting a Lie group (which is where the title comes from).
I think the idea of designing a preconditioner based on considerations of gradient noise and as well as the Hessian is interesting. However most of that work was done in the Li paper, and including the design of "c".  This paper's contribution seems to be to work out some of the details for various restricted classes of matrices, to construct a "Fisher version" of c, and to run some experiments.
The problem is that I don't really buy the original motivation for the "c" function from the Li paper, and the newer Fisher version of c proposed in this paper doesn't seem to have any justification at all.  I also find that the paper in general doesn't do a good job of explaining its various choices when designing the algorithm.  This could be somewhat forgiven if the experimental results were strong, but unfortunately they are too limited, and marred by overly-simplistic baselines that aren't properly tuned.
More detailed comments below
Title:
I think the title is poorly chosen.  The paper doesn't use Lie groups or their properties in any significant way, and "learning" is a bad choice of words too, since it involves generalization etc (it's not merely the optimization of some function).  A better title would be "A general framework for adaptive preconditioners" or something.
Intro:
Citation of Adagrad paper is broken
The literature review contained in the intro needs works. I wouldn't call methods like quasi-Newton methods "convex optimization methods".  Those algorithms were around a long time ago before "convex optimization" was a specific topic of study and are probably *less* associated with the convex optimization literature than, say, Adagrad is. And methods like Adagrad aren't exactly first-order methods either. They use adaptively chosen preconditioners (that happen to be diagonal) which puts them in a similar category to methods like LBFGS, KFAC, etc.
It's not clear at this point in the paper what it means for a preconditioner to be "learned on" something.
Section 2:
The way you discuss quadratic approximations is confusing.  Especially  the sentence "is the sum of approximation error and constant term independent of theta" where you then go on to say that a_z does depend on theta.  I know that this second theta is the "current theta" separate from the theta as it appears in the formula for the approximation but this is really sloppy. Usually people construct the quadratic approximation in terms of the *change in theta* which makes such things cleaner.
You should explain how eqn 8 was derived since it's so crucial to everything that follows.  Citing a previous paper with no further explanation really isn't good enough here.  Surely with all of the notation you have already set up it should be possible to motivate this criterion somehow. The simple fact that it recovers P = H^-1 in the noiseless quadratic case isn't really good enough, since many possible criteria would do the same.
I've skimmed the paper you cited and their justification for this criterion isn't very convincing.  There are other possible criteria that they give and there doesn't seem to be a strong reason to prefer one over the other.
Section 3:
The way you define the Fisher information matrix corresponds to the "empirical Fisher", since z includes the training labels.  This is different from the standard Fisher information matrix.
How can you motivate doing the "replacement" that you do to generate eqn 12? Replacing delta theta with v is just notation, but how can you justify replacement of delta g with g + lambda v?  This isn't a reasonable approximation in any sense that I can discern. Once again this is an absolutely crucial step that comes out of nowhere.  Honestly it feels contrived in order to produce a connection to popular methods like Adam.
Section 4:
The prominent use of the abstract mathematical term "Lie group" feels unnecessary and like mathematical name-dropping. Why not just talk about certain "classes" of invertible matrices closed under standard operations (which would also help people that don't know what a Lie group is)?  If you are going to invoke some abstract mathematical framework like Lie groups it needs to actually help you do something you couldn't otherwise. You need to use some kind of advanced Theorem for Lie groups.
Without knowing the general form of R equation 18 is basically vacuous. *any* matrix (in the same class) could be written this way.
I've never heard of the natural gradient being defined using a different metric than the Fisher metric.  If the metric can be arbitrary then even standard gradient descent is a "natural gradient" too (taking the Euclidean metric).  You could argue for a generalized definition that would include only parametrization independent metrics, but then your particular metric wouldn't obviously work.
Section 6:
Rather than comparing to Batch Normalization you would be better off comparing to the old centering and normalization work of Schraudolph et al which the former was based on, which is actually a well-defined preconditioner.
Section 7:
You really need to sweep over the learning rate parameters for optimizers like SGD with momentum or Adam.   Otherwise the comparisons aren't very interesting.
"Tikhonov regularization" should just be called L2-regularization


DEEP DECODER: CONCISE IMAGE REPRESENTA- TIONS FROM UNTRAINED NON-CONVOLUTIONAL NETWORKS
Deep neural networks, in particular convolutional neural networks, have become highly effective tools for compressing images and solving inverse problems including denoising, inpainting, and reconstruction from few and noisy measurements. This success can be attributed in part to their ability to represent and generate natural images well. Contrary to classical tools such as wavelets, imagegenerating deep neural networks have a large number of parameters-typically a multiple of their output dimension-and need to be trained on large datasets. In this paper, we propose an untrained simple image model, called the deep decoder, which is a deep neural network that can generate natural images from very few weight parameters. The deep decoder has a simple architecture with no convolutions and fewer weight parameters than the output dimensionality. This underparameterization enables the deep decoder to compress images into a concise set of network weights, which we show is on par with wavelet-based thresholding. Further, underparameterization provides a barrier to overfitting, allowing the deep decoder to have state-of-the-art performance for denoising. The deep decoder is simple in the sense that each layer has an identical structure that consists of only one upsampling unit, pixel-wise linear combination of channels, ReLU activation, and channelwise normalization. This simplicity makes the network amenable to theoretical analysis, and it sheds light on the aspects of neural networks that enable them to form effective signal representations.

The paper builds upon Deep Image Prior (DIP) - work which shows that one can optimize a neural generator to fit a single image without learning on any dataset, and the output of the generator (which approximates the image) can be used for denoising / super resolution / etc. The paper proposes a new architecture for the DIP method which has much less parameters, but works on par with DIP. Another contribution of the paper is theoretical treatment of (a simplified version of) the proposed architecture showing that it can’t fit random noise (and thus maybe better suited for denoising).
The paper is clearly written, and the proposed architecture has too cool properties: it’s compact enough to be used for image compression; and it doesn’t overfit thus making early stopping notnesesary (which was crucial for the original DIP model).
I have two main concerns about this paper.
First, it is somewhat misleading about its contributions: it's not obvious from abstract/introduction that the whole model is the same as DIP except for the proposed architecture. Specifically, the first contribution listed in the introduction makes it look like this paper introduces the idea of not learning the decoder on the dataset (the one that starts with “The network is not learned and itself incorporates all assumptions on the data.”).
My second concern is about the theoretical contribution. On the one hand, I enjoyed the angle the authors tackled proving that the network architecture is underparameterized enough to be a good model for denoising. On the other hand, the obtained results are very weak: only one layered version of the paper is analysed and the theorem applies only to networks with less than some threshold of parameters. Roughly, the theorem states that if for example we fix any matrix B of size e.g. 256 x k and matrix U of size 512 x 256 and then compute U relu(B C) where C is the vector of parameters of size k x 1, AND if k < 2.5 (i.e. if we use at most 2 parameters), then it would be very hard to fit 512 iid gaussian values (i.e. min_C ||U relu(B C) - eta|| where eta ~ N(0, 1)). This restriction of the number of parameters to be small is only mentioned in the theorem itself, not in the discussion of its implications.
Also, the theorem only applies to the iid noise, while most natural noise patterns have structure (e.g. JPEG artifacts, broken pixels, etc) and thus can probably be better approximated with deep models.
Since the paper manages to use very few parameters (BTW, how many parameters in total do you have? Can you please add this number to the text?), it would be cool to see if second order methods like LBFGS can be applied here.
Some less important points:
Fig 4 is very confusing.
First, it doesn’t label the X axis.
Second, the caption mentions that early stopping is beneficial for the proposed method, but I can’t see it from the figure.
Third, I don’t get what is plotted on different subplots. The text mentions that (a) is fitting the noisy image, (b) is fitting the noiseless image, and (c) is fitting noise. Is it all done independently with three different models? Then why does the figure says test and train loss? And why DIP loss goes up, it should be able to fit anything, right? If not and it’s a single model that gets fitted on the noisy image and tested on the noiseless image, then how can you estimate the level of noise fitting? ||G(C) - eta|| should be high if G(C) ~= x.
Also, in this quote “In Fig. 4(a) we plot the Mean Squared Error (MSE) over the number of iterations of the optimizer for fitting the noisy astronaut image x + η (i.e., FORMULA ...” the formula doesn’t correspond to the text.
And finally, the discussion of this figure makes claims about the behaviour of the model that seems to be too strong to be based on a single image experiment.
I don’t get the details of the batch normalization used: with respect to which axis the mean and variance are computed?
The authors claim that the model is not convolutional. But first, it’s not obvious why this would be a good thing (or a bad thing for that matter). Second, it’s not exactly correct (as noted in the paper itself): the architecture uses 1x1 convolutions and upsampling, which combined give a weak and underparametrized analog of convolutions.
> The deep decoder is a deep image model G: R N → R n, where N is the number of parameters of the model, and n is the output dimension, which is typically much larger than the number of parameters (N << n).
I think it should be vice versa, N >> n
The following footnote
> Specifically, we took a deep decoder G with d = 6 layers and output dimension 512×512×3, and choose k = 64 and k = 128 for the respective compression ratios.
Uses unintroduced (at that point) notation and is very confusing.
It would be nice to have a version of Figure 6 with k = 6, so that one can see all feature maps (in contrast to a subset of them).
I’m also wondering, is it harder to optimize the proposed architecture compared to DIP? The literature on distillation indicates that overparameterization can be beneficial for convergence and final performance.


RELATIONAL FORWARD MODELS FOR MULTI-AGENT LEARNING
The behavioral dynamics of multi-agent systems have a rich and orderly structure, which can be leveraged to understand these systems, and to improve how artificial agents learn to operate in them. Here we introduce Relational Forward Models (RFM) for multi-agent learning, networks that can learn to make accurate predictions of agents' future behavior in multi-agent environments. Because these models operate on the discrete entities and relations present in the environment, they produce interpretable intermediate representations which offer insights into what drives agents' behavior, and what events mediate the intensity and valence of social interactions. Furthermore, we show that embedding RFM modules inside agents results in faster learning systems compared to non-augmented baselines. As more and more of the autonomous systems we develop and interact with become multi-agent in nature, developing richer analysis tools for characterizing how and why agents make decisions is increasingly necessary. Moreover, developing artificial agents that quickly and safely learn to coordinate with one another, and with humans in shared environments, is crucial.

This paper studies predicting multi-agent behavior using a proposed neural network architecture. The architecture, called a relational forward model (RFM) is the same graph network proposed by Battaglia et al., 2018, but adds a recurrent component. Two tasks are define: predict the next action of each agent, and predict the sum of future rewards. The paper demonstrates that RFMs outperform two baselines and two ablations. The authors also show that edge activation magnitudes are correlated with certain phenomenons (e.g. an agent walking towards an entity, or an entity being “on” or “off”). The authors also show that appending the output of a pre-trained RFM to the state of a policy can help it learn faster.
Overall, this paper presents some interesting ideas and is easy to follow, but the significance of the paper is not clear. The architecture is a rather straightforward extensions of previous work, and using graph networks for predictive modeling in multi-agent settings has been examined in the past, making the technical contributions not particularly novel. Examining the correlation between edge activation magnitudes and certain events is intriguing and perhaps the most novel aspect of this paper, but it is not clear how or why this information would be useful. There a few unsubstantiated claims that are concerning. There are also some odd experimental decisions and results that should be addressed.
For specific comments:
1. Why would using a recurrent network help (i.e. RFM vs Feedforward)? Unless the policies are non-Markovian, the entire prediction problem should Markovian. I suspect that most of the gains are coming from the fact that the RFM method simply has more parameters than the Feedforward method (e.g. it can amortize some of the computation into the recurrent part of the network). Suggestion: train a Feedforward model that has more parameters (with appropriate hyperparameter sweeps) to see if this is the cause. If not, provide some analysis for why “memories of the relations between entities” would be any more beneficial than simply recomputing those relations.
2. The other potential reason that the recurrent method did better is that policy actions are highly correlated (e.g. because agents move in straight lines to locations). If so, then recurrent methods can outperform feedforward methods without having to learn anything about what actually causes policies to move in certain directions. Suggestion: measure the correlation between consecutive actions. If there is non-trivial correlation, than this suggests that RFM does better than Feedforward (which is basically prior work of Battaglia et. al.) is for the wrong reasons.
3. If I understand the evaluation metric correctly, for each rollout, it counts how many steps from the beginning of the rollout match perfectly before the first error occurs. Then it averages this “minimum time to failure” across all evaluation rollouts. If this is correct, why was this evaluation metric chosen? A much more natural metric would be to just compute the average number of errors on a test data-set (and if this is what is actually reported, please update the description to disambiguate the two). The current metric could be very deceptive:  Methods that do very well on states around the initial-state distribution but poorly near the end of trajectories (e.g. perfectly predicts the actions in the first 10 steps, but then resorts to random guessing for the last 99999 time steps) will outperform methods that have lower average error rate (e.g. a model that is correct 50% of the time). Suggestion: change the metrics to average number of errors, or report both, or provide a convincing argument why this metric is meaningful.
4. Unless I misunderstood, the results in Section 2.2.3 seem spurious and the claims seem unsubstantiated. For one, if we look at Equations (1) and (2), when we average over s_a1 and s_a2, they should both give the same average for R_a1. Put another way: the prune graph should (in theory) marginalize out s_a2. On average, its expected output should be the same as the output of the full graph (after marginalizing out s_a1 and s_a2). Obviously, it is possible to find specific rollouts where the full graph has higher value than the prune graph (and it seems Figure 4 does this), but it should equally be possible to find rollouts where the opposite is true. I’m hoping I misunderstood this section, but otherwise this seems to invalidate all the claims made in this section.
5. Even if concern #4 is addressed, the following sentence would still seem false: “This figure shows that teammates’ influence on each other during this time is beneficial to their return.” The figure simply shows predictions of the RFM, and not of the ground truth. Moreover, it’s not clear what “teammates’ influence” actually means.
6. The comparison to NRI seems rather odd, since that method uses strictly less information than RFM.
7. For Section 3, is the RFM module pretrained and then fine-tuned with the new policy? If so, this gives the “RFM + A2C” agent extra information indirectly via the pretrained weights of the RFM module.
8. I’m not sure what to make of the correlation analysis. It is not too surprising that there is some correlation (in fact, it’d be quite an interesting paper if the findings were that there wasn’t a correlation!), and it’s not clear to me how this could be used for debugging, visualizations, etc. If someone wanted to analyze the correlation between two entities and a policy’s action, it seems like they could directly model this correlation.
Some minor comments:
 - In Figure 3C, right, why isn’t the magnitude 0 at time=1? Based on the other plots in Figure 3c, it seems like it should be 0.
 - The month/year in many of the citations seems odd.
 - The use of the word “valence” seems unnecessarily flowery and distracting.
My main concern with this paper is that it is not particularly novel and the contribution seems questionable. I have some concerns over the experimental metric and Section 2.2.3, but even if that is clarified, it is not clear how impactful this paper would be. The use of a recurrent network seems unnecessary, unjustified, and not analyzed. The analysis of correlations is interesting, but not particularly compelling or surprising. And lastly, the RFM-augmented results are not very strong.
--
Edit: After discussing with the authors, I have changed my rating. The authors have adjusted some of the language, which I previously thought overstated the contributions and was misleading. They have added a number of experiments which valid the claim that their method is proposing a reasonable way of measuring collaboration. I also realized that I misunderstood one of the sections, and I encourage the authors to improve the presentation to (1) present the significance of the experiments more clear, (2) not overstate the results, and (3) emphasize the contribution more clearly.
Overall, the paper presents convincing evidence that factors in a graph neural networks do capture some notion of collaboration. I do not feel that the paper is particularly novel, but the experiments are thorough. Furthermore, their experiments show that adding an RFM module to an agent consistently helps (albeit not by much). Given that the multi-agent community is still trying to decide how to best quantify and use metrics for collaboration, I find it difficult to access the long-term impact of this paper. However, given the thoroughness of the experiments and analysis, I suspect that this will be valuable for the community and deserves some visibility.


LEARNING MULTI-LEVEL HIERARCHIES WITH HINDSIGHT
Multi-level hierarchies have the potential to accelerate learning in sparse reward tasks because they can divide a problem into a set of short horizon subproblems. In order to realize this potential, Hierarchical Reinforcement Learning (HRL) algorithms need to be able to learn the multiple levels within a hierarchy in parallel, so these simpler subproblems can be solved simultaneously. Yet most existing HRL methods that can learn hierarchies are not able to efficiently learn multiple levels of policies at the same time, particularly in continuous domains. To address this problem, we introduce a framework that can learn multiple levels of policies in parallel. Our approach consists of two main components: (i) a particular hierarchical architecture and (ii) a method for jointly learning multiple levels of policies. The hierarchies produced by our framework are comprised of a set of nested, goal-conditioned policies that use the state space to decompose a task into short subtasks. All policies in the hierarchy are learned simultaneously using two types of hindsight transitions. We demonstrate experimentally in both grid world and simulated robotics domains that our approach can significantly accelerate learning relative to other non-hierarchical and hierarchical methods. Indeed, our framework is the first to successfully learn 3-level hierarchies in parallel in tasks with continuous state and action spaces. We also present a video of our results and software to implement our framework.

This paper proposed a framework that can improve the performances of reinforcement learning algorithms in tasks that involve long time horizons and sparse rewards. The proposed method is a hierarchical reinforcement learning framework that can use policy hierarchies with an arbitrary number of levels. To improve the sample efficiency in the learning process, the authors proposed to apply the hindsight experience replay mechanism at each level. Also, in order to avoid the actor function to output an unrealistic subgoal, the authors proposed the subgoal testing technique.
The proposed framework is interesting. And the example in Section 3.5 clearly demonstrate how this framework works. The authors proposed to solve a UMDP by solving a hierarchy of k UMDPs, where k is a hyperparameter. Each level (except for the bottom most level) will output subgoal states for the next level to achieve. This hierarchy is reasonable and easy to understand. However, from the definition on Page 3, it seems that all of the intermediate levels i (the case where 0 < i < k - 1) has the same state and action spaces. They are all equal to the state set of the original UMDP. Under this setting, will adding more intermediate levels help improve the performance a lot? We only see results with at most one intermediate level in the experiment. It will be better if the authors can show results on more levels (i.e. at least 4 levels in total).
Moreover, the proposed framework has a policy limit parameter T, meaning that we only consider if a goal can be achieved within T steps or not, at each level. Is this parameter necessary to be the same for all levels? Also, it will be better if the author can show some results on the performances of the proposed method according to different values for T. The authors also proposed the subgoal testing technique. It is also better if the authors can show some performance comparisons on the cases with and without this technique.
The authors claimed that their method has the advantage over some existing HRL methods (e.g. the Option-Critic Architecture [1]) that their method can use policy hierarchies with an arbitrary number of levels while these methods can only use policy hierarchies with two levels. In the experiments, the authors also showed that, in some of their experiments, the 3-layer agent (with 2 subgoal layers) outperforms the 2-layer agent (with 1 subgoal layer), under their framework. However, the authors did not compare their 2-layer agent's performance with these existing HRL methods, which means that we do not know if their 3-layer agent's performance is better than that of some of the existing 2-layer agent methods. In addition to that, as I mentioned before, it is better if the authors can show experiment results on more levels (e.g. 4 levels and more) to show that their method can perform well in practice for policy hierarchies with many levels.
References:
[1] Pierre-Luc Bacon, Jean Harb, and Doina Precup. The option-critic architecture. CoRR, abs/1609.05140, 2016.


We propose a generalization error bound for a general family of deep neural networks based on the depth and width of the networks, as well as the spectral norm of weight matrices. Through introducing a novel characterization of the Lipschitz properties of neural network family, we achieve a tighter generalization error bound. We further obtain a result that is free of linear dependence on norms for bounded losses. Besides the general deep neural networks, our results can be applied to derive new bounds for several popular architectures, including convolutional neural networks (CNNs), residual networks (ResNets), and hyperspherical networks (SphereNets). When achieving same generalization errors with previous arts, our bounds allow for the choice of much larger parameter spaces of weight matrices, inducing potentially stronger expressive ability for neural networks.

After rebuttal: The authors have nicely addressed my comments. I have increased my rating to 7.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
This paper proposes a generalization error bound for DNNs (and their generalizations) based on the depth and width of the networks, as well as the spectral norm of weight matrices. The proposed work is shown to provide a tighter generalization error bounds compared with a few existing literatures.
Pros: This paper makes theoretical contributions to the understanding of DNNs. This is an important but difficult task. As a theoretical paper, this one is relatively easy to follow.
Cons: In spite of its theoretical contributions, this paper has a few major issues.
Q1: This paper fails to fairly compare with the most recent work, Arora et al. (2018), Zhou and Feng (2018). For instance, Arora et al. (2018) uses error-resilience parameters instead of the norms of weight matrices to obtain a better generalization error. The authors claim that the error-resilience parameters are less interpretable than the norms of weight matrices. This claim could be subjective and is not convincing.
Q2: The error bounds of Bartlett et al. (2017), Neyshabur et al. (2017) could be improved for low-rank weight matrices, in which case the proposed Theorem 1 is tighter only if $p \le D^2$. This holds only when DNN is very deep. Can theorem 1 be improved by similarly considering the low-rankness of weight matrices?
Q3: In Corollary 2, the error bound for CNN, the authors assume that the filters are orthogonal with unit norm. Can the authors provide some justification on the orthogonal filters? In addition, Zhou and Feng (2018) have achieved similar bound for CNN. Can the authors provide some justification why this latest result is not included in Table 2?


Deep reinforcement learning has achieved remarkable successes in solving various challenging artificial intelligence tasks. A variety of different algorithms have been introduced and improved towards human-level performance. Although technical advances have been developed for each individual algorithms, there has been strong evidence showing that further substantial improvements can be achieved by properly combining multiple approaches with difference biases and variances. In this work, we propose to use the James-Stein (JS) shrinkage estimator to combine on-policy policy gradient estimators which have low bias but high variance, with low-variance high-bias gradient estimates such as those constructed based on model-based methods or temporally smoothed averaging of historical gradients. Empirical results show that our simple shrinkage approach is very effective in practice and substantially improve the sample efficiency of the state-of-the-art on-policy methods on various continuous control tasks.

This paper presents two algorithms that improve on PPO by using James-Stein (JS) shrinkage estimator. The first algorithm, PPO-MBS, combines low bias of on-policy methods with low variance of model-based RL algorithms. The second, PPO-STS, uses JS to create a statistical momentum and reduce variance of the PPO algorithm. Both algorithms are evaluated on Mujoco environments aiming to show improvements in average cumulative reward, reduced bias and variance.
The paper’s topic is highly relevant, as the authors point out, the current state of the art in RL is largely divided between model-based and model-free methods. This paper aims at bridging the gap, and taking advantage of both sides. The writing is clear and concise, with all the math properly introduced. The proposed methods are interesting and novel. As far as I am aware, this is solid and unexplored approach with potentially significant impact.
There are two concerns with this paper. First, the evaluation results are promising, yet not fully convincing. It appears that 5 million steps is not sufficient for the policy convergence for any of the tasks (average reward keeps increasing). At the same time the variance gap (Figure 1) is reducing.
- As the training continues, does the variance for PPO-MBS become larger?
- Similar question for the average reward - the advantage of the PPO-MBS vs. comparison methods seems to be reducing. What happens with the additional training?
- How do the trajectories and behavior of the Walker2D (and other) look with PPO-MBS vs. others - is the higher average reward indicative of the qualitatively better behavior?
- Why do Walker2D and Hopper increase \alpha over time, while Swimmer and Reacher lean more towards mode model based policy over time?
- Is there something significant about the structure of the problems?
- Similar questions arise from the evaluation of the PPO-STS - it appears that the training is even less complete in this case, rendering the conclusions about the quality of the learned policy at convergence invalid.
- Why are Humanoid and Ant not evaluated on PPO-MBS? The authors should extend the training to convergence on all problems, and present the results including movies of example trajectories of all environments for both algorithms in the supplementary material.
Second, the paper introduces two competing methods.  While, the authors compare them directly,  they do not discuss how the two methods relate to each other. In the Reacher and Hopper task seems like all three methods perform about the same, and in the Hopper PPO-STS even performs worse than the baseline. This makes it difficult to assess the significance of the exposition. When should one be used, and when the other one is better?
The authors should consider either a) splitting the paper into two focusing each paper on a single algorithm with more in-depth evaluations and discussions, or b) combining the two algorithms into ones. In any case, further analysis that illuminates when the methods should be used, and how they improve the training are needed.


Generative networks have made it possible to generate meaningful signals such as images and texts from simple noise. Recently, generative methods based on GAN and VAE were developed for graphs and graph signals. However, the mathematical properties of these methods are unclear, and training good generative models is difficult. This work proposes a graph generation model that uses a recent adaptation of Mallat's scattering transform to graphs. The proposed model is naturally composed of an encoder and a decoder. The encoder is a Gaussianized graph scattering transform, which is robust to signal and graph manipulation. The decoder is a simple fully connected network that is adapted to specific tasks, such as link prediction, signal generation on graphs and full graph and signal generation. The training of our proposed system is efficient since it is only applied to the decoder and the hardware requirement is moderate. Numerical results demonstrate stateof-the-art performance of the proposed system for both link prediction and graph and signal generation. These results are in contrast to experience with Euclidean data, where it is difficult to form a generative scattering network that performs as well as state-of-the-art methods. We believe that this is because of the discrete and simpler nature of graph applications, unlike the more complex and high-frequency nature of Euclidean data, in particular, of some natural images.

Summary:
The paper presents a generative model for graphs which is a VAE-like architecture where the encoder is a scattering transform with fixed parameters (rather than a trainable neural net)
Pros:
+ The problem of graph generative models is very important, and a lot of the existing methods are not very scalable.
+ Using spectral methods in place of standard "neural net operations" makes a lot of sense.
+ Numerical results for the "link prediction" task seem to be significantly better than those of baselines.
Cons:
- The paper contains various imprecisions (see the non-exhaustive list below), and significant amount of statements which are hard to understand.
- I am not sure if the work can be considered particularly novel: in particular, it is not really emphasised what is the difference with [Angles & Mallat '2018].
- The motivation for the work is not entirely clear: it is true that GANS and VAEs have their issues, but in my view it is not really explained / argued why the proposed method would solve them.
- I find the argument about the efficiency not very convincing, especially after looking at the members (bottom of p. 7): the scattering transform alone takes several orders of magnitude longer than the baseline. Authors also mention that their method does not require training of the encoder, but I do not see any comparisons with respect to number of parameters.
- The experimental evaluation for "signal generation" and "graph generation" is not very convincing. For the former there is no real comparison to existing models. And for the latter, the experimental setup seems a bit strange: it appears that the models were trained on different subsets of the dataset, making the comparison not very meaningful. Also, I would expect to see the same methods to be compared to a cross all the tasks (unless it is impossible for some reason).
Various typos / imprecisions / unclear statements:
p.1, "are complex as well as difficult to train and fine-tune.": not at all clear what this means.
p.1, "Their development is based on fruitful methods of deep learning in the Euclidean domain, such as convolutional and recurrent neural networks.": Recurrent and convolution neural network are not necessarily restricted to Euclidean domains.
p.1, "Using a prescribed graph representation, it is possible to avoid training
the two components at the same time, but the quality of the prescribed representation is important
in order to generate promising results.": not clear what this sentence means.
p.2, "Unlike GAN or VAE, the model in this paper does not require training two components either iteratively or at the same time.": I do not see why that would necessarily be a bad thing, especially in the case of VAE where traditional training in practice corresponds to training a single neural net.
p.3, "GAN-type graph networks use a discriminator in order to compete with the generator and make its training more powerful.": I am not sure this statement is strictly correct.
p.9: "We remark that the number of molecules in the training sets are not identical to that in ...": does this mean that the models are effectively trained on different data? In that case, the comparison is not very meaningful.


We present DL2, a system for training and querying neural networks with logical constraints. DL2 is more expressive than prior work and can capture a richer class of constraints on inputs, outputs and internals of models. Using DL2, one can declaratively specify domain knowledge to be enforced during training or pose queries on the model with the goal of finding inputs that satisfy a given constraint. DL2 works by translating logical constraints into a differentiable loss with desirable mathematical properties, then minimized with standard gradient-based methods. Our evaluation demonstrates that DL2 can express interesting constraints beyond the reach of prior work, leading to improved prediction accuracy.

Summary
-------
This paper proposes DL2, a framework for turning queries over parameters and input, output pairs to neural networks into differentiable loss functions, and an associated declarative language for specifying these queries. The motivation for this work is twofold. The first is to allow for the specification of additional domain knowledge during training. For example, if a user expects that the predicted probabilities of some output classes should be correlated for all predictions, this constraint can be enforced during weight learning. Second, it allows users to search for specific inputs that satisfy specified conditions. In this way, DL2 can capture popular applications like searching for adversarial examples by querying for inputs close to a known input of class A but that the network predicts is class B with high confidence.
The paper provides a concise specification of the query language (a mixture of logical and numeric operators) and asserts a theorem that the given procedure for constructing the query loss produces a function such that anytime the function is 0, the constraints are satisfied. No proof is given, but I cannot see a counterexample. There is also a statement about the converse relationship, that when the loss is above some threshold it implies that the query is not satisfied.
Experiments are conducted on supervised, semi-supervised, and unsupervised computer vision tasks. I particularly liked the experiment on semi-supervised learning with CIFAR-100. By replacing labeled examples with domain knowledge about the relationships among classes in CIFAR-100, the paper demonstrates a compelling use case for DL2.
The primary technical challenge is the non-convex optimization required to search for a solution to a query. Experiments show that the loss functions created by DL2 are often solved quickly and correctly, but not always
Strengths
---------
The framework is expressive enough that many interesting use cases are clear, from specifying background knowledge during training to model inspection. The experiments cover a range of these use cases, demonstrating that the constructed optimization objectives usually work as intended.
Weaknesses
-----------
The statement in Theorem 1 regarding the converse case is unclear, because it says that the limit of \delta as \epsilon approaches zero is zero, but it is not explained what \epsilon is or how it changes. If \epsilon is the threshold that can often be used in the query, it is not obvious that every query contains exactly one \epsilon. If other cases exist, it is unclear how Theorem 1 applies.
It remains unknown how to handle the case when queries fail. AS the paper points out, if a query fails, it cannot be determined whether no solution exists or if the optimization simply failed to find a solution. Of course, this is a computationally hard in general.
Related Work
------------
There are a couple of points from related work that would be good to add to the paper.
First, the paper "Adversarial Sets for Regularising Neural Link Predictors" (Minervini et al., UAI17) is a prior paper that generates adversarial examples to handle restrictions on inputs which may not exist in the training set. The paper claims DL2 is the first to do this, but I believe this paper is an earlier example that does so, albeit for a particular problem. DL2 is certainly more general.
Second, the description of the limitations of rule distillation (Hu et al., ACL16), particularly in Appendix A is not fully accurate. The expressivity of PSL is greater than stated (see Bach et al., JMLR17 for a full description). In particular, the DL2 loss function for z = (1, 1) can be expressed exactly in PSL using what it calls arithmetic rules. It is not clear that this affects the findings of the semi-supervised learning experiment significantly, although I would appreciate a clarification of the authors. PSL by construction produces convex loss functions, and so the constraint that all outputs for a group of classes is either high OR low would probably not work well.


Meta-learning provides a promising learning framework to address few-shot classification tasks. In existing meta-learning methods, the meta-learner is designed to learn about model optimization, parameter initialization, or similarity metric. Differently, in this paper, we propose to learn how to create an individualized feature embedding specific to a given query image for better classifying, i.e., given a query image, a specific feature embedding tailored for its characteristics is created accordingly, leading to an individualized feature space in which the query image can be more accurately classified. Specifically, we introduce a kernel generator as meta-learner to learn to construct feature embedding for query images. The kernel generator acquires meta-knowledge of generating adequate convolutional kernels for different query images during training, which can generalize to unseen categories without fine-tuning. In two standard few-shot classification data sets, i.e. Omniglot, and miniImageNet, our method shows highly competitive performance.

# Summary
This work deals with few-shot learning and classification by means of similarity learning. The authors propose a method for generating a set of convolutional kernels, i.e. a mini-CNN, for a query image given a set of support samples (with samples from the same class and some other classes). Kernels are generated for each query and are adapted to the specific visual content found in the query image, thus a new embedding space is identified. The difficulty of the task is constrained by using a common base CNN for feature extraction, making the task resolution more feasible in the few shot regime. The method is evaluated on standard benchmarks Omniglot and miniImagenet with competitive results.
# Paper strengths
- The paper has a good coverage of related work
- The proposed method is interesting and the results are encouraging
- The authors argue and study the influence of multiple elements over their contribution: number of sub-generators, distance metric, choice of architecture
# Paper weaknesses
- My main concern with this work is the incremental contribution with respect to the work by Han et al. (2018), "Face recognition with contrastive convolution". In that work the authors proposed a convolutional kernel generator for every pair of images to be compared/matched, while here the principle is simplified to re-use the same convolutional kernels for the a query image. The loss functions are nearly identical, both works use a classification loss and a loss ensuring kernels at different images with the same object should be similar. The visualizations of the feature maps are similar as well, though these would have been necessary any way for this type of contribution.
- The architecture of the kernel generator is not clear to the reader. Is it similar with the one from Han et al.?
- The related and relevant work from Gidaris and Komodakis (2018), Dynamic few-shot visual learning without forgetting, is not included as baseline in the evaluation. Their method is superior when using C4 and similar (while still keeping performance levels on previous tasks).
- Given that the evaluation for few-shot classification takes random samples of query and support samples and that we're dealing with stochastic models, it's common and encouraged to include error-bars/standard deviations in the results to get a better idea on the performances. I encourage the authors to do the same.
- The visualizations from Figure 3 would need some additional clarifications from the authors in the text. It's not clear what does the colormap refer to, red is for high activation and blue for low activation (as typical for jet colormap) or the other way around? If blue is highly active, it's worrying that most dogs (Q1,Q2,Q3) are active on the white dog in S3. As said, it would be useful to have some comments from the authors in the text to better explain the visualizations
- Minor remarks:
    + The evaluation protocol from Omniglot should be specified as there are 2 ways of doing it: 1) using characters from different alphabets at test time (easier); 2) using characters from the same alphabet (more difficult)
    + There are some other works dealing with weight generation or with adaptive embedding that would be worth mentioning:
        * Y.X. Wang et al., Learning to model the tail, NIPS 2017
        * A. Veit and S. Belongie, Conditional similarity networks, CVPR 2017

# Conclusion
This paper advances an interesting idea for few-shot classification and gets competitive results. As mentioned in the section above, I'm worried about the incremental contribution on top of the work by Han et al.. In addition results are outperforming state of the art works, while requiring generating kernels and features for each query. My current rating is between Weak Reject and Borderline.


Accurate spatio-temporal traffic forecasting is a fundamental task that has wide applications in city management, transportation area and financial domain. There are many factors that make this significant task also challenging, like: (1) mazelike road network makes the spatial dependency complex; (2) the traffic-time relationships bring non-linear temporal complication; (3) with the larger road network, the difficulty of flow forecasting grows. The prevalent and state-of-the-art methods have mainly been discussed on datasets covering relatively small districts and short time span, e.g., the dataset that is collected within a city during months. To forecast the traffic flow across a wide area and overcome the mentioned challenges, we design and propose a promising forecasting model called Layerwise Recurrent Autoencoder (LRA), in which a three-layer stacked autoencoder (SAE) architecture is used to obtain temporal traffic correlations and a recurrent neural networks (RNNs) model for prediction. The convolutional neural networks (CNNs) model is also employed to extract spatial traffic information within the transport topology for more accurate prediction. To the best of our knowledge, there is no general and effective method for traffic flow prediction in large area which covers a group of cities. The experiment is completed on such large scale real-world traffic datasets to show superiority. And a smaller dataset is exploited to prove universality of the proposed model. And evaluations show that our model outperforms the state-of-the-art baselines by 6% -15%.

This paper has potential, but I do not think it is ready for publication. I will ask some questions / make some suggestions:
1) Your first sentence makes a claim about there being a large body of research on traffic flow forecasting. I don't doubt this, but you should cite some papers, please.
2) Your contributions raise the following questions for me:
- Contribution 1 is that you use a very large dataset (for training? you don't say.) and a small dataset (for testing), thus proving that your method works and generalizes. Your method may be effective, but compared to what? Your method may generalize, but how do we know that if you've only tested it on one small dataset?
- Contribution 2 says that you creatively used lagged data in a time series model. This is probably a good idea, but it does not sound all that creative to me, compare with, e.g. an AR model.
- Contribution 3 says that you use driving distance to model spatial correlation. Again, this is probably a good idea, and when we get further we learn that you applied a Graph Convolution Network. Were these the choices that you claim are novel? Are they novel? What other choices might be reasonable and how would they compare?
3) Section 3 immediately jumps into the use of autoencoders. But I think you need to justify why we care about using autoencoders in the first place. If the problem is traffic forecasting, why don't you tackle that problem head on?
4) Section 3 mentions sparsity without justifying why I care about sparsity. This might be an important tool for regularization in a deep neural network. Or it might not be--given enough data and other regularization techniques (weight decay, early stopping, dropout).
5) Is the spatial dependency that you end up learning qualitatively different than the spatial dependency you would get by instead assuming a particular parametric form as is done in kernel methods / Gaussian processes, e.g. the Gaussian kernel or the Matern kernel parameterizes the covariance between observations at two spatial locations?
6) In your experiment I believe you randomly split 15 minute blocks into train/test/validate. I think this evaluation will be over-optimistic insofar as if 10:30-10:45 and 11:00-11:15 are in the train set, but 10:45-11:00 is in the test set, it will be relatively easy to predict 10:45-11:00. I would suggest considering train/test/validate splits based on larger chunks, e.g. leave the data in 15 minute blocks, but randomly select hours (4 blocks) to put in train/test/validate.


INVARIANT AND EQUIVARIANT GRAPH NETWORKS
Invariant and equivariant networks have been successfully used for learning images, sets, point clouds, and graphs. A basic challenge in developing such networks is finding the maximal collection of invariant and equivariant linear layers. Although this question is answered for the first three examples (for popular transformations, at-least), a full characterization of invariant and equivariant linear layers for graphs is not known. In this paper we provide a characterization of all permutation invariant and equivariant linear layers for (hyper-)graph data, and show that their dimension, in case of edge-value graph data, is 2 and 15, respectively. More generally, for graph data defined on k-tuples of nodes, the dimension is the k-th and 2k-th Bell numbers. Orthogonal bases for the layers are computed, including generalization to multigraph data. The constant number of basis elements and their characteristics allow successfully applying the networks to different size graphs. From the theoretical point of view, our results generalize and unify recent advancement in equivariant deep learning. In particular, we show that our model is capable of approximating any message passing neural network. Applying these new linear layers in a simple deep neural network framework is shown to achieve comparable results to state-of-the-art and to have better expressivity than previous invariant and equivariant bases.

Given a graph G of n vertices, the activations at each level of a graph neural network (G-NN) for G
can be arranged in an n^k tensor T for some k. A fundamental criterion is that this tensor must be equivariant
to permutations of the vertices of G in the sense of each index of of T being permuted simultaneously.
This paper enumerates the set of all linear maps that satisfy this criterion, i.e., all linear maps
which (the authors claim) can serve as the analog of convolution in equivariant G-NNs.
The authors find that for invariant neural networks such maps span a space of dimension just b(k), whereas
for equivariant neural networks they span a space of dimension b(2k).
The proof of this result is simple, but elegant. It hinges on the fact that the set of tensor elements of
the same equality type is both closed and transitive under the permutation action. Therefore, the
dimensionality of the subspace in question is just the number of different identity types, i.e.,
partitions of either {1,...,k} or {1,...,2k}, depending on whether we are talking about invariance or
equivariance.
My problem with the paper is that the authors' model of G-NNs doesn't actually map to what is used
in practice or what is interesting and useful. Let me list my reservations in increasing order of significance.
1. The authors claim that they give a ``full characterization'' of equivariant layers. This is not true.
Equivariance means that there is *some* action of the symmetric group S_n on each layer, and wrt these actions
the network is equivariant. Collecting all the activations of a given layer together into a single object L,
this means that L is transformed according to some representation of S_n. Such a representation can always be
reduced into a direct sum of the irreducible representations of S_n. The authors only consider the case then
the representation is the k'th power of the permutation representation (technically called the defining
representation of the S_n). This corresponds to a specific choice of irreducibles and is not the most general case.
In fact, this is not an unnatural choice, and all G-NNs that I know follow this route.
Nonetheless, technically, saying that they consider all possible equivariant networks is not correct.
2. The paper does not discuss what happens when the input tensor is symmetric. On the surface this might seem
like a strength, since it just means that they can consider the more general case of undirected graphs (although
they should really say so). In reality, when considering higher order activations it is very misleading because
it leads to a massive overcounting of the dimensionality of the space of convolutions. In the case of k=2, for
example, the dimensionality for undirected graphs is probably closer to 5 than 15 for example (I didn't count).
3. Finally, and critically, in actual G-NNs, the aggregation operation in each layer is *not*
linear, in the sense that it involves a product of the activations of the previous layer with the adjacency
matrix (messages might be linear but they are only propagated along the edges of the graph).
In most cases this is motivated by making some reference to the geometric meaning of convolution,
the Weisfeiler-Lehman algorithm or message passing in graphical models. In any case, it is critical that the
graph topology be reintroduced into the network at each layer. The algebraic way to see it is that each layer
must mix the information from the vertices, edges, hyperedges, etc.. The model in this paper could only aggregated
edge information at the vertices. Vertex information could not be broadcast to neighboring vertices again.
The elemenary step of ``collecting vertex information from the neighbors but only the neighbors'' cannot be
realized in this model.
Therefore, I feel that the model used in this paper is rather uninteresting and irrelevant for practical
purposes. If the authors disagree, I would encourage them to explicitly write down how they think the model
can replicate one of the standard message passing networks. It is apparent from the 15 operations listed on
page 11 that they have nothing to do with the graph topology at all.
Minor gripes:
- I wouldn't call (3) and (4) fixed point equations, that's usually used in dynamical systems. Here there is
an entire subspace fixed by *all* permutations.
- Below (1), they probably mean that ``up to permutation vec(L)=vec(L^T)''.


Understanding theoretical properties of deep and locally connected nonlinear network, such as deep convolutional neural network (DCNN), is still a hard problem despite its empirical success. In this paper, we propose a novel theoretical framework for such networks with ReLU nonlinearity. The framework bridges data distribution with gradient descent rules, favors disentangled representations and is compatible with common regularization techniques such as Batch Norm, after a novel discovery of its projection nature. The framework is built upon teacher-student setting, by projecting the student's forward/backward pass onto the teacher's computational graph. We do not impose unrealistic assumptions (e.g., Gaussian inputs, independence of activation, etc). Our framework could help facilitate theoretical analysis of many practical issues, e.g. disentangled representations in deep networks.

This paper gives a model for understanding locally connected neural networks. The main idea seems to be that the network is sparsely connected, so each neuron is not going to have access to the entire input. One can then think about the gradient of this neuron locally while average out over all the randomness in the input locations that are not relevant to this neuron. Using this framework the paper tried to explain several phenomena in neural networks, including batch normalization, overfitting, disentangling, etc.
I feel the paper is poorly written which made it very hard to understand. For example, as the paper states, the model gives a generative model for input (x,y) pairs. However, I could not find a self-contained description of how this generative model works. Some things are described in Section 3.1 about the discrete summarization variables, but the short paragraph did not describe: (a) What is the "multi-layer" deterministic function? (b) How are these z_\alpha's chosen? (c) Given z's how do we generate x? (d) What happens if we have z_\alpha and z_\beta and the regions \alpha and \beta are not disjoint? What x do we use in the intersection?
In trying to understand the paper, I was thinking that (a)(b) The multilayer deterministic function is a function which gives a tree structure over the z_\alpha's, where y is the root. (I have no idea why this should be a deterministic function, intuitively shouldn't y be chosen randomly, and each z_\alpha chosen randomly conditioned on its parent?)  (c) there is a fixed conditional distribution of P(x_\alpha|z_\alpha), and I really could not figure out (d). The paper definitely seems to allow two receptive fields to intersect as in Figure 1(b).
Without understanding the generative model, it is impossible for me to evaluate the later results. My general comments there is that there are no clear Theorems that summarizes the results (the Theorems in the paper are all just Lemmas that are trying to work towards the final goal of giving some explanations, but the explanations and assumptions are not formally written down). Looking at things separately (as again I couldn't understand the single paragraph describing the generative model), the Assumption in Theorem 3 seems extremely limiting as it is saying that x_j is a discrete distribution (which is probably never true in practice). I wouldn't say "the model does not impose unrealistic assumptions" in abstract if you are going to assume this, rather the model just makes a different kind of unrealistic assumptions (Assumptions in Theorem 2 might be much weaker, but it's hard to judge that).
==== After reading the revision
The revised version is indeed more clear about how the teacher network works, and I have tried to understand the later parts of the paper again. The result of the paper really relies on the two assumptions in Theorem 2. Of the two assumptions, the first one seems to be intuitive (and it is OK although exact conditional independence might be slightly strong). The second assumption is very unclear though as it is not an assumption that is purely about the model/teacher network (which are the x and z variables), it also has to do with the learning algorithm/student network (f's and g's). It is much harder to reason about the behavior of an algorithm on a particular model and directly making an assumption about that in some sense hides the problem. The paper mentioned that the condition is true if z is fine-grained, but this is very vague - it is definitely true if z is super fine-grained to satisfy the assumption in Theorem 3, but that is too extreme.
Overall I still feel the paper is a bit confusing and it would benefit from having a more concrete example. I like the direction of the work but I can't recommend for recommendation at this stage.


SAMPLE EFFICIENT ADAPTIVE TEXT-TO-SPEECH
We present a meta-learning approach for adaptive text-to-speech (TTS) with few data. During training, we learn a multi-speaker model using a shared conditional WaveNet core and independent learned embeddings for each speaker. The aim of training is not to produce a neural network with fixed weights, which is then deployed as a TTS system. Instead, the aim is to produce a network that requires few data at deployment time to rapidly adapt to new speakers. We introduce and benchmark three strategies: (i) learning the speaker embedding while keeping the WaveNet core fixed, (ii) fine-tuning the entire architecture with stochastic gradient descent, and (iii) predicting the speaker embedding with a trained neural network encoder. The experiments show that these approaches are successful at adapting the multi-speaker neural network to new speakers, obtaining state-of-the-art results in both sample naturalness and voice similarity with merely a few minutes of audio data from new speakers.

This paper proposes  an adaptation technique for TTS using wavenet as the speech backend, with the adaptation carried out on small data. The work is extremely significant in that speech data is hard to produce  (we need many hours of speaker data), and techniques to adapt (transfer learning?) data from large networks would be quite valuable. The main idea is that we train a network containing a large amount of data, and (assuming that we have a trained model), we adapt this network to the task of generating speech from text for a much smaller dataset.
In general (insofar as we can use that term), one trains such a network using <text/speech> pairs,  with speaker conditioning as added input so as to produce voice from a given speaker. The input text is converted to linguistic features in the ‘front end’, which is then injected with voice features to be synthesized into a voice output in the backend. More recent efforts in speech modeling have used RNN or wavenet based systems to carry out these transformations in the front/backends. The present work seems to use an SPSS technique (Zen et al 2016) to generate the linguistic features, while the task of converting to voice is carried out by a Wavenet.
The work is quite (conceptually) similar to "Neural voice cloning with a few samples" (Arik et al, https://arxiv.org/abs/1802.06006) in proposing techniques for few shot adaptation described below but with the significant difference that the latter used autoregressive DNNs (loosely speaking, seq2seq a la Tacotron) for the task in both the front and back ends, while in the current work, the linguistic features are computed with SPSS as in Zel et al (2016).
The paper proposes three quite related techniques for adaptation as shown clearly in Figure 2 of the paper. These techniques are again ‘roughly’ analogous to those described in the Baidu work “Neural Voice Cloning with a few samples”, with the difference in front and backend setups noted in the previous paragraph.
We take as text as input, and convert them to a representation for linguistic features as described in Zen et al (2016). To this, we now add the fundamental frequency F_0 for the sample voice. The key piece needed is the speaker embeddings (a vector), which is to be obtained by training. In addition to all this, we also have available the weights of a trained wavenet network (probably quite large) trained on many speakers, which we will modify (or not) using the strategies outlined for the few data dataset.
SEA-EMB - Train embeddings, but not the network. We expect this to be ‘fast’, but not particularly accurate.
SEA-ALL - Train embeddings, and network. This would be a much more accurate, if slower task. The authors note that since we train a very large network in this case, it could be prone to overfitting. They employ early stopping (as a practitioner, I would make note of the issue) with 10 % of the dataset being held out. Additional ideas such as initializing the emeddings - possibly with those that SEA-EMB calculates - are also stated to be useful.
SEA-ENC - In this third version, they predict speaker embeddings from the trained larger network (the recipe is provided in the appendix). This task of predicting speaker embeddings is one of training a classifier.
Results
The paper presents evaluations conducted with subjective, MOS based enrolment and with an evaluation metric from TI-SV d-vectors. Comparisons are made for all three models with human evaluated MOS scores, and it is seen that SEA-ALL outperforms the other two models, while performance in SEA-EMB depends on the amount of data used. Nevertheless, humans are still able to detect the difference between synthetic voices and real samples.
The TI-SV evaluations from Wan et al show t-SNE embeddings of ‘clusters’ of d-vectors for human and synthetic voices, where it is seen that inter-cluster distance (i.e. between different speakers) is high, showing that the model is able to discern speakers, and the intra-cluster distance (i.e. between real and synthetic voices) is low, showing that synthetic voices are ‘similar’ to real voices. In addition, three other measures - cosine similarity, and statistical measures for detection error trade off, ROC curves and cosine similarity measures are also presented, which show that that the adaptation models perform quite well.
Clarifications and comments:
Have there been efforts to compare this model (with the SPSS based frontend) with seq2seq (Bahdanau/transformer) DNN based systems as in “Neural Voice cloning with few samples”?. How do they compare (is it even a valid comparison?)?
I think the model for computing linguistic features could be elaborated upon further.
Representations: I assume that the output audio representation is an audio waveform
Typo 1 (minor): The reference  for “Bornschein et al” in section 4 “Related work”
“Variable inference for memory addressing”.
Correction “Variational memory addressing in generative models”
Typo 2 (minor): Figure 6: Lower curve indicate that the verification system is having a harder time distinguishing real from generated samples.
Correction (minor): Lower curve “indicates” ...
Summary
-------------
In summary, I am in favor of accepting this paper as it proposes a solution to adapt a trained network to one with has limited number of samples. A big issue in speech modeling is that datasets are tiny, and it is difficult to obtain good quality data at reasonable cost. It would be extremely useful to have a trained network that we can adapt for our own experiments. The related paper by Arik et al (Neural Voice cloning with a few samples) also operates with similar strategic aims, but uses a a different methodology using attention based DNNs. The paper under review should be a good addition to the toolbox of few shot adaptation/transfer learning for speech with much potential for practical use.


Exploration has been a long standing problem in both model-based and model-free learning methods for sensorimotor control. While there has been major advances over the years, most of these successes have been demonstrated in either video games or simulation environments. This is primarily because the rewards (even the intrinsic ones) are non-differentiable since they are function of the environment (which is a black-box). In this paper, we focus on the policy optimization aspect of the intrinsic reward function. Specifically, by using a local approximation, we formulate intrinsic reward as a differentiable function so as to perform policy optimization using likelihood maximization -much like supervised learning instead of reinforcement learning. This leads to a significantly sample efficient exploration policy. Our experiments clearly show that our approach outperforms both on-policy and off-policy optimization approaches like REINFORCE and DQN respectively. But most importantly, we are able to implement an exploration policy on a robot which learns to interact with objects completely from scratch just using data collected via the differentiable exploration module. See project videos at https: //doubleblindICLR.github.io/robot-exploration/.

This paper presents an interesting way to reformulate intrinsic curiosity as a differentiable function. The authors compare the differentiable function against using prediction error via REINFORCE and DQN, showing that their intrinsic curiosity method results in more interactions with unseen objects than the other two methods. For DQN this is to be expected, but it shows that backprop through this function is more efficient than reinforce in getting to unseen state spaces. I think this is an interesting method/proposal and is a somewhat novel reformulation of intrinsic error, but I do have some concerns in comparisons/claims.
In the introduction, the authors say that the intrinsic curiosity method proposed by Pathak et al. is sample inefficient and isn’t tested in robots. However, to my understanding the REINFORCE baseline isn’t really equivalent (though it may be possible that it is, it was unclear how exactly the loss was formulated in the baseline, did include the other components from Pathak et al.?). If the claim is that this method is more efficient, I think it should have compared against that method directly.
Moreover, I think the description of the experiments doesn’t provide enough information. For example, the method says that different learning rates were used for the min-max game to stabilize it, but doesn’t say what they were.
Also, for the DQN baseline what were the parameters? Was there an epsilon greedy policy on top of the exploration reward? Was this annealed as in other work? Generally, I think more detail is needed throughout (even if it just refers to a more detailed appendix).
Overall, I think this work needs to be revised to include more details on hyperaparameters, details on the baselines, and describing differences between Pathak et al.’s method and the REINFORCE baseline. Moreover, feedback from other comments on this work should be addressed which reflect in more detail my comments below on opinionated claims (e.g., https://openreview.net/forum?id=SkzeJ3A9F7&noteId=HJlFlZOa2X )
Comments/Thoughts:
+ I think in the introduction there are some statements that probably need citations. For example, “But the same formulation from an optimization viewpoint, it suffers from all the bad properties of extrinsic rewards. The reward is a function of environment behavior with respect to the performed action. Since the environment behavior function is unknown, it is treated as black-box and hence the gradients have to be computed using REINFORCE (Williams, 1992) which is quite sample inefficient.” —> Why is this true? Is there a citation that can back this? Do you prove it later in the paper?
+ “Yes, 54 environments but no real-world physical robots” —> this and the intro seems like a blogpost at times. That can be fine (some would argue it’s a good thing), but there seem to be some opinions without citations/backing, I suggest trying to back up statements wherever possible and avoid opinions. For example in this statement, robots aren't a requirement for evaluating intrinsic motivation.
+ “Since the environment behavior function is unknown, it is treated as black-box and hence the gradients have to be computed using REINFORCE (Williams, 1992) which is quite sample inefficient.” —> citation/backing? it might be nice to point to the experiment section here to back it (e.g., "As will be shown in Section X and in \citet{something}, REINFORCE can be quite sample inefficient")
+ “In practice, the existing on-policy algorithms, e.g., A3C (Mnih et al., 2016), PPO (Schulman et al., 2017) etc. are deployed off-the shelf -> This is confusing, so is this using REINFORCE or PPO/A3C? what is this statement referring to?
+ “regress to rti to learn value estimates (i.e., off-policy) as discussed in the previous section” —> regress to \sum r_t{I} for a value estimate?? Value is the expected return so not sure if this is a typo or i missed something earlier
+ What is the actual loss function used for the baseline? Is it the same as Pathak et al.?
+ What are the hyper parameters for DQN exploration? What are all the hyper parameters for any/all the algorithms?
+ Was a variance-reducing baseline used in REINFORCE?
+ What is the variance representing in the graphs, std across several trials? Maybe I missed it, but how many trials represent this standard deviation?
+ “Hence, we train the forward predictor slightly faster than the policy by keeping higher learning rate to stabilize the learning process. “ —> what were the learning rates?
Linguistic/Typos:
Also, some minor, but frequent, grammatical issues/typos that I’ve added below could be fixed. I would ask that the authors please have the submission proof-read for English style and grammar issues. There are many minor mistakes, some of which I’ve tried to point out below.
+ “This leads to a significantly sample efficient exploration policy. “ —> significantly more (?) sample efficient ?
“Why is that? To understand the reason behind sample inefficiency of curiosity or intrinsic rewards, notice how the intrinsic rewards are given by agent” —> by the agent?
“Forward model fθF is trained to minimize its loss which amounts to minimizing rti with respect to θF” —> the forward model
“However, policy is optimized to maximize the objective” —> However, the policy
“We can also optimize  for policy parameters θP via differentiable loss function” —> We can also optimize for (the) policy parameters \theta via (a) differentiable loss function?
“To optimize policy to maximize a discounted sum “ —> To optimize the policy
“How good is Forward Prediction Model” —> How good is the forward prediction model
There are several other spots, but basically another pass over the paper might be worth it to check for these sorts of issues.