Generating formal-language represented by relational tuples, such as Lisp programs or mathematical operations, from natural-language input is a challenging task because it requires explicitly capturing discrete symbolic structural information implicit in the input. Most state-of-the-art neural sequence models do not explicitly capture such structural information, limiting their performance on these tasks. In this paper we propose a new encoder-decoder model based on Tensor Product Representations (TPRs) for Natural-to Formal-language generation, called TP-N2F. The encoder of TP-N2F employs TPR 'binding' to encode natural-language symbolic structure in vector space and the decoder uses TPR 'unbinding' to generate, in symbolic space, a sequence of relational tuples, each consisting of a relation (or operation) and a number of arguments. On two benchmarks, TP-N2F considerably outperforms LSTM-based seq2seq models, creating new state-of-the-art results: the MathQA dataset for math problem solving, and the AlgoLisp dataset for program synthesis. Ablation studies show that improvements can be attributed to the use of TPRs in both the encoder and decoder to explicitly capture relational structure to support reasoning.

This paper proposes a sequence-to-sequence model for mapping word sequences to relation-argument-tuple sequences. The intermediate representation (output of the encoder) is a fixed-dimensional vector. Both encoder and decoder internally use a tensor product representation. The experimental results suggest that the tensor product representation is helpful for both the encoder and the decoder. The paper is interesting and the experimental results are positive, but in my opinion the exposition could use some substantial work. Fixing the most substantial flaws in the exposition would be sufficient to warrant an accept in my view.
Major comments:
I found the mix of levels of detail in the model specification in section 3 confusing. It would be extremely helpful to have a straightforward high-level mathematical description of the key parts of the encoder, mapping (which could be considered part of the encoder), and decoder in standard matrix-vector notation. While equations (7), (8), (9), (10), (11) and appendix A.2 go some way toward this, key high-level details seem to be missing, and I feel like the exposition would benefit from simply stating the matrix-vector operations that are performed in addition to describing their interpretation in terms of the semantics of the tensor product representation. Specific examples are noted below.
It would be helpful to be explicit about the very highest-level structure of the proposed model. If I understand correctly, it is a probabilistic sequence-to-sequence model mapping a word sequence to a probability distribution over relation-argument-tuple sequences. It uses an encoder-decoder architecture with a fixed-dimensional intermediate representation, and an autoregressive decoder using attention. Both the encoder and decoder are based on the tensor product representation described in section 2. Stating these simple facts explicitly would be extremely helpful.
Especially for the encoder, the learned representation is so general that there seems to be no guarantee that the learned roles and fillers are in any way related to the syntactical / semantic structure that motivates it in section 2. There doesn't seem to be any experimental investigation of the learned TPR in the encoder. If I understand correctly, the way encoder roles and fillers are computed and used is symmetric, meaning that the roles and fillers could be swapped while leaving the overall mapping from word sequences to relation-argument-tuple sequences unchanged. This suggests it is not possible to interpret the role and filler vectors in the encoder in an intuitive way.
Minor comments:
In section 2, "R is invertible" should strictly be "R has a left inverse".
In section 3.1.1, the claim that "we can hypothesize to approximately encode the grammatical role of the token and its lexical semantics" is pretty tenuous, especially given the apparent symmetry between learned roles and fillers in the encoder and given the lack of experimental investigation of the meaning of the learned encoder roles and fillers.
In section 3.1.2, my understanding is that the relation-argument tuple (R, A_1, A_2, A_3), say, is treated as a sequence of 3-tuples: (A_1, R, 1), (A_2, R, 2), (A_3, R, 3). Each of these 3-tuples is then embedded using learned embeddings (separate embeddings for argument, relation and position). If correct, it would be helpful to state this explicitly.
In section 3.1.2, it is stated that contracting a rank-3 tensor with a vector is equivalent to matrix-vector product, which is not the case.
In section 3.1.3, both high-level and low-level details of the MLP module are omitted. High-level, I presume that the matrix output by the encoder is reshaped to a large vector, the MLP is applied to this vector to produce another vector, then this is reshaped to a rank-3 tensor to input to the decoder. It would be helpful to state this. Low-level, the number of layers, depth and activation function of the MLP should be specified somewhere, at least in the appendix.
Did the authors consider using a bidirectional LSTM for the encoder? This might improve performance.
In section 3.1.2 and appendix A.2, why use the LSTM hidden state for subsequent processing rather than the LSTM output (which would be more conventional). The LSTM output is defined in appendix A.2 but appears not to be used for anything. Please clarify in the paper.
Did the authors consider passing the output of the reasoning MLP into every step of the tuple LSTM instead of just using it to initialize the hidden state?
It would be helpful to state the rank of the tensors H, B, etc in section 3.2.2.
In section 3.2.2, what does "are predicted by classifiers over the vectors..." mean? This seems quite imprecise. What is the form of the classifier? My best guess is that the vector a_i^t is passed through a small MLP with a final softmax layer which outputs a probability distribution over the 1-of-K representation of the argument. The main text says "more details are given in the appendix", but appendix A.2 just has "Classifier(a_1^t)". Please clarify in the paper.
What is the attention over in equation (9)? Attention needs at least two arguments, the query and the sequence being attended to. It seems that (9) only specifies one of these. It would also be helpful to be explicit about the form of attention used.
What is f_linear in (11)?
It seems unnecessarily confusing to switch notation for the arguments from A_1 in section 3.1.2 to a r g_1 in section 3.2.2, and similarly for the relations.
For the decoder tuple LSTM, how exactly is the previous relation-argument tuple (R, A_1, A_2, A_3), say, summarized? Are each of R, A_1, A_2 mapped to a vector, these vectors concatenated, then passed into the LSTM? Or is the positional decomposition into (A_1, R, 1), ... used? Please clarify in the paper.
Based on section 3.3, it seems that the model assumes that, in the decomposition of (R, A_1, A_2, A_3) into a sequence (A_1, R, 1), (A_2, R, 2), (A_3, R, 3) of 3-tuples at each decoder output step, the three 3-tuples are conditionally independent of each other and the three entries of each 3-tuple are conditionally independent of each other. Is this indeed assumed? If so, it would be helpful to state this explicitly. It seems like this is likely not true in practice.
Section 3.3 refers to "predicted tokens". Where are these predicted tokens in (9), (10) or (11)?
In section 3.3, it seems the loss at each decoder step is the log probability of the relation-argument tuple at that step. Thus, by the autoregressive property, the overall loss is the log probability of the sequence of relation-argument tuples. If so, it would be helpful to state both these facts explicitly.
Section 3 seems to be missing a section, which is how decoding is performed at inference time. For the output of the decoder at each step, is random sampling used, if so with a temperature, or is greedy decoding (selecting the most likely class, equivalent to a temperature of 0) used? Also, what is done if decoding outputs different R's for (A_1, R, 1), (A_2, R, 2), (A_3, R, 3)? The three R values here should be equal in order for this to represent a relation-argument tuple (R, A_1, A_2, A_3), but there is no guarantee the model will respect this constraint.
Unless I missed it (apologies if so), many experimental architecture details were omitted. For example, how many hidden cells were used for the LSTMs, etc, etc? These should at least be stated in the appendix.
It would be interesting to investigate how long input / output sequences need to be before the fixed-dimensional internal representation breaks down.
In section 4.1.1, it was not clear to me what "noisy examples" means. Does this mean that the dataset itself is flawed, meaning that the reference sequence of operations does not yield the reference answer? Please clarify in the paper.
In table 1, please state the total size of the fixed-dimensional intermediate representation for all systems. This seems crucial to ensure the systems can be meaningfully compared.
In figure 4, left figure, the semantic classes don't apper to be very convincingly clustered. (And it seems like K-means clustering could easily have selected a different clustering given a different random initialization.)
In appendix A.2, mathematical symbols are essentially meaningless without describing what they mean in words. Please explain the meaning of all the symbols that are not defined in terms of other symbols, e.g. w^t, T_{t-1}, ..., f_s m (is this softmax???), f_l i n e a r (what does this mean?), C o n t e x t, C l a s s i f i e r, etc, etc. C o n t e x t in particular doesn't even have a hint of a definition.
In (19) and (27), why would a temperature parameter be helpful? This can be absorbed as an overall multiplicative factor in the weight matrix of the previous linear layer. Is this temperature parameter learned during training (I presume so)? Please clarify in the paper.
Usually * is used for convolution, not simple multiplication (e.g. equation (17)).
Throughout the main body and appendix, there are lots of instances of poor spacing. For example, $f_{linear}$ should be written as something like $f_\text{linear}$ in latex to avoid it coming out as l i n e a r (which literally interpreted means l times i times n times e, etc). Please fix throughout.


Rethinking physics in the era of deep learning is an increasingly important topic. This topic is special because, in addition to data, one can leverage a vast library of physical prior models (e.g. kinematics, fluid flow, etc) to perform more robust inference. The nascent sub-field of physics-based learning (PBL) studies this problem of blending neural networks with physical priors. While previous PBL algorithms have been applied successfully to specific tasks, it is hard to generalize existing PBL methods to a wide range of physics-based problems. Such generalization would require an architecture that can adapt to variations in the correctness of the physics, or in the quality of training data. No such architecture exists. In this paper, we aim to generalize PBL, by making a first attempt to bring neural architecture search (NAS) to the realm of PBL. We introduce a new method known as physics-based neural architecture search (PhysicsNAS) that is a top-performer across a diverse range of quality in the physical model and the dataset.

=== Overall comments ===
This paper proposes to generalize approaches to physics-based learning (PBL) by performing network architecture search (NAS) over elements from PBL models found in the literature. This entails including physical inputs to the network and the incorporation of new operations to the NAS. I think the idea has merit and rather like it. However, there are several aspects of the work that could be improved. The technical novelty is small, as the extension of the  existing NAS models to handle physical inputs and a few new operators is relatively straightforward. The experiments, while well designed, only explore uninteresting toy problems. While I appreciate the necessity to explore the methods performance in a more controlled setting, a more impactful testbed would be more convincing. Another drawback of the evaluation is the lack of a proper statistical analysis of the results, given the small data and model sizes.
=== Relevance & Prior Work ===
+ The related work gives a good summary and categorization of prior work in physics-based learning
+ The problem (physics-based learning) is interesting and relevant to the community
=== Novelty & Approach===
+ application of NAS to physics based learning
+ incorporation of physics solutions as inputs into differentiable NAS
+ creation of physics-informed operation sets to merge physical models into network
- technical steps to merge NAS and PBL are relatively straightforward
=== Evaluation ===
Two representative physical simulations were chosen for evaluation, where elements of the physics model are intentionally omitted,  1) estimating trajectory of a ball in presence of wind and air resistance, and 2) a collision speed simulation where two objects collide, where sliding friction is not accounted for in the physics model.
The baselines consist of: a 3-layer MLP (data-driven), a 3-layer MLP with Physical Regularization, a 3-layer MLP with residual connection to the physics prediction, an MLP with two input branches, on for the data and one for the physics predictions (Physical Fusion), and the Embedded Physics model which estimates parameters for the physics modelu using a 3-layer MLP.
PhysicsNAS can combine elements of the baseline models, but the total number of nodes is limited to 5.
+ Experiments testing the dependence of the model on the numbers of samples and the strength of the physical inconsistencies were conducted. In both cases, PhysicsNAS outperformed the best specialized physics models.
- The chosen testbed tasks are toy problems. While these types of experiments are necessary to understand the performance of the model, it would have been interesting to see PhysicsNAS applied to a more impactful task
- Given the size of the networks and the training data, there is no reason why a more sophisticated statistical analysis of the results wasn’t performed (confidence intervals, t-test, p-value). Similarly, a more complete set of experiments with more sample amounts could be provided with little effort.
=== Clarity & Other Comments ===
- “precious nodes” -> previous nodes


An intriguing phenomenon observed during training neural networks is the spectral bias, where neural networks are biased towards learning less complex functions. The priority of learning functions with low complexity might be at the core of explaining generalization ability of neural network, and certain efforts have been made to provide theoretical explanation for spectral bias. However, there is still no satisfying theoretical result justifying the underlying mechanism of spectral bias. In this paper, we give a comprehensive and rigorous explanation for spectral bias and relate it with the neural tangent kernel function proposed in recent work. We prove that the training process of neural networks can be decomposed along different directions defined by the eigenfunctions of the neural tangent kernel, where each direction has its own convergence rate and the rate is determined by the corresponding eigenvalue. We then provide a case study when the input data is uniformly distributed over the unit sphere, and show that lower degree spherical harmonics are easier to be learned by over-parameterized neural networks.

This paper studies the training of overparametrized neural networks by gradient descent. More precisely, the authors consider the neural tangent regime (NTK regime). That is, the weights are chosen sufficiently large and the neural network is sufficiently overparametrized. It has been observed that in this scenario, the neural network behaves approximately like a linear function of its weights.
In this regime, the authors show that, the directions corresponding to larger eigenvalues of the neural tangent kernel are learned first. As this corresponds to learning lower-degree polynomials first, the authors claim that this explains the "spectral bias" observed in previous papers.
-I think that from a mathematical point of view, the main result of this paper is what one would expect intuitively:
When performing gradient descent with quadratic loss where the function to be learnt is linear, it is common knowledge that convergence is faster on directions corresponding to larger singular values. Since in the NTK regime, the neural network can be approximated by a linear function around the initialization one expects the behavior predicted by the main results. From a theoretical perspective, I see the main contribution of the paper as making this statement precise.
-I am skeptical about some of the implications for practitioners, which are given by the authors:
For example, on p.5 the authors write "Therefore, Theorem 3.2 theoretically explains the empirical observations given in Rahaman et al. (2018), and demonstrates that the difficulty of a function to be learned by neural network should be studied in the eigenspace of neural tangent kernel." To the best of my knowledge, it is unclear whether practitioners train neural networks in the NTK regime (see, e.g., [1]). Moreover, I am wondering whether some of the assumptions of their theorem are really met in practice. For example, the required sample size for higher order polynomials grows exponentially fast with the order and the required step size goes to zero exponentially fast. Does this really correspond to what is observed in practice? (Or is this a mere artifact of training in the NTK regime?)  Is this what one observes in the experiments by Ramahan?
I think the paper is not yet ready for being published.
 1. There are many typos. Here is an (very incomplete) list.
    -p. 2: "Su and Yang (2019)" improves the convergence..."
    -p. 2: "This theorem gives finer-grained control on error term's"
    -p. 2: "We present a more general results"
    -p. 4: "The variance follows the principal..."
    -p. 4: "...we will present Mercer decomposition in (the) next section."
2. I think that the presentation can be polished and many statements are somewhat unclear. For example, on p. 7 the authors write "the convergence rates [...] are exactly predicted by our theory in a qualitative sense."
    The meaning of this sentence is unclear to me. Does that mean in a quantitative sense? To be honest, only considering Fig. 1 I am not able to assess whether the convergence rates of the different components are truly linear.
I decided for my rating of the paper because of the following reasons:
-I think that for a theory paper the results obtained by the authors are not enough, as they are rather direct consequences of the "near-linearity" of the neural network around the initialization.
-In my view, there is a huge gap between current theoretical results for deep learning and practice. For this reason, it is not problematic for me that it is unclear, what the results in this paper mean for practitioners. (Apart from that, results for the NTK regime are interesting in its own right.) However, in my view, one should explain the limitations of the theory more carefully.
-The presentation of the paper needs to be improved.
References:
[1] A note on lazy training in supervised differentiable programming. L Chizat, F Bach - arXiv preprint arXiv:1812.07956, 2018
-----------------------------
I highly appreciate the authors' detailed response. However, I feel that the paper does not contain enough novelty to justify acceptance.
------
"Equation (8) in Arora et al., (2019b) only provides a bound on the whole residual vector, i.e., , and therefore cannot show different convergence rates along different directions."
When going through Section 4 , I think that it is implicitly stated that one has different convergence along different directions.
-----
For this reason, I am not going to change my score.


In this work, we describe a set of rules for the design and initialization of wellconditioned neural networks, guided by the goal of naturally balancing the diagonal blocks of the Hessian at the start of training. We show how our measure of conditioning of a block relates to another natural measure of conditioning, the ratio of weight gradients to the weights. We prove that for a ReLU-based deep multilayer perceptron, a simple initialization scheme using the geometric mean of the fan-in and fan-out satisfies our scaling rule. For more sophisticated architectures, we show how our scaling principle can be used to guide design choices to produce well-conditioned neural networks, reducing guess-work.

The authors propose a new initialization scheme for training neural networks. The initialization considers fan-in and fan-out, to regularize the range of singular values of the Hessian matrix, under several assumptions.
The proposed approach gives important insights for the problem of weight initialization in neural networks. Overall, the method makes sense. However, I have several concerns:
- The authors do not consider more recent neural network designs such as normalization layers, skip connections, etc. It would be great if the authors could discuss how these layers would change the derivation of the initialization method. Also, preliminary experimental results using these layers are needed. Additionally, to me, normalization layers [Huang et al. Decorrelated Batch Normalization. CVPR 2018] implicitly precondition the Hessian matrix as well. It would be great if the authors also compare their approach to [Huang et al. 2018].
- The authors compared to other initialization schemes such as [He et al., 2015] and [ Glorot and Bengio 2010]. But as the authors mentioned, there are approaches that scales backpropagation gradients also [Martens and Grosse, 2015; Grosse and Martens, 2016; Ba et al., 2017; George et al., 2018]. Since these methods are highly related to the proposed method, it would be great if the authors could show time complexities and performance differences of these methods as well.
- Experiments on the CIFAR-10 dataset with AlexNet seem not exciting: the proposed Preconditioned approach only outperforms the Fan-out approach marginally. I would say that training a [He et al. 2015]-initialized neural network for 500 more iterations than a preconditioned neural network, yields a similar or better loss.
Overall I think the work is very important and interesting. However, it lacks comprehensive comparison and consideration of more recent neural network layers.
Post Rebuttal Comments
I have read all reviewer comments and the author feedback. I appreciate that the authors addressed the skip connections in Appendix.
1. The authors agree that batch norm requires different initialization schemes that are not included in this paper.
2. I agree with the authors that their approach is complementary to the baseline optimization methods; and both approaches can be applied together. However, I still believe that it is informative to compare the two approaches because: (a). Both approaches address the same problem. Since the optimization based approach adds complexity and computational overhead to implementation, it would be great to show if using the proposed approach eliminates the need for the optimization based approach. (b). Is it necessary to use both approaches, or one of them is good enough?
3. I understand that strong experimental evidence is not always required. However, I believe that the new technical insights of the paper alone is not significant enough (part of the reasons in point 1). Thus I was expecting stronger experimental evidences.
Overall I agree with reviewer 1 that the topic is interesting, but in the paper’s current form, it is not ready. I keep my initial rating of weak reject.


Most existing 3D CNN structures for video representation learning are clip-based methods, and do not consider video-level temporal evolution of spatio-temporal features. In this paper, we propose Video-level 4D Convolutional Neural Networks, namely V4D, to model the evolution of long-range spatio-temporal representation with 4D convolutions, as well as preserving 3D spatio-temporal representations with residual connections. We further introduce the training and inference methods for the proposed V4D. Extensive experiments are conducted on three video recognition benchmarks, where V4D achieves excellent results, surpassing recent 3D CNNs by a large margin.

[Summary]
The paper presents a video classification framework that employs 4D convolution to capture longer term temporal structure than the popular 3D convolution schemes. This is achieved by treating the compositional space of local 3D video snippets as an individual dimension where an individual convolution is applied. The 4D convolution is integrated in resnet blocks and implemented via first applying 3D convolution to regular spatio-temporal video volumes and then the compositional space convolution, to leverage existing 3D operators. Empirical evaluation on three benchmarks against other baselines suggested the advantage of the proposed method.
[Decision]
Overall, the paper addresses an important problem in computer vision (video action recognition) with an interesting. I found the motivation and solution are reasonable (despite some questions pending more elaboration), and results also look promising, thus give it a weak accept (conditional on the answers though).
[Comments]
At the conceptual level, the idea of jointly modeling local video events is not novel, and can date back to at least ten years ago in the paper “Learning realistic human actions from movies”, where the temporal pyramid matching was combined with the bag-of-visual-words framework to capture long-term temporal structure. The problem with this strategy is that the rigid composition only works for actions that can be split into consecutive temporal parts with prefixed duration and anchor points in time, which is clearly challenged by many works later when more complicated video events are studied. It seems to me that the proposed framework also falls in this category, with a treatment from deep learning. It is definitely worth some discussion on this path.
That said, I would like to see more analysis on the behavior of the proposed method under various interesting cases not tested yet. Despite the claim that the proposed method can capture long-term video patterns, the static compositional nature seems to work best for activities with well-defined local events and clear temporal boundaries. These assumptions hold mostly true for the three datasets used in the experiment, and also are suggested by results in table 2(e), where 3 parts are necessary to achieve optimal results. How does the proposed method perform in more complicated tasks such as
- action detection or localization (e.g., in benchmarks JHMDB or UCF101-24).
- complex video event modeling (e.g., recognizing activities in extended video of TRECVID).
Will it still be more favorable than other concerning baselines?
Besides, on the computation side, it would be complexity, an explicit comparison of complexity makes it easier to evaluate the performance when compared to other state-of-the-art methods.
[Area to improve]
Better literature review to reflect the relevant previous video action recognitions, especially those on video compositional models.
Proof reading - The word in the title should be “Convolutional”, right?


While deep learning has shown tremendous success in a wide range of domains, it remains a grand challenge to incorporate physical principles in a systematic manner to the design, training and inference of such models. In this paper, we aim to predict turbulent flow by learning its highly nonlinear dynamics from spatiotemporal velocity fields of large-scale fluid flow simulations of relevance to turbulence modeling and climate modeling. We adopt a hybrid approach by marrying two well-established turbulent flow simulation techniques with deep learning. Specifically, we introduce trainable spectral filters in a coupled model of Reynoldsaveraged Navier-Stokes (RANS) and Large Eddy Simulation (LES), followed by a specialized U-net for prediction. Our approach, which we call Turbulent-Flow Net (TF-Net), is grounded in a principled physics model, yet offers the flexibility of learned representations. We compare our model, TF-Net, with state-of-theart baselines and observe significant reductions in error for predictions 60 frames ahead. Most importantly, our method predicts physical fields that obey desirable physical characteristics, such as conservation of mass, whilst faithfully emulating the turbulent kinetic energy field and spectrum, which are critical for accurate prediction of turbulent flows.

## Summary
The authors propose use learned spatio-temporal filtering and a convolutional model to predict the behavior of a turbulent fluid flow. Turbulent flow is a very difficult problem, of great interest for engineers and physical scientists, so the topic of the paper is certainly compelling.
This paper has several significant issues. The baselines the authors compare to are quite weak. Many of them were designed for other purposes, such as video prediction. The authors claim significant improvements ("64.2% reduction in divergence") that I believe are calculated in a (unintentionally) misleading way. In fact, I think they missed the most important baseline---the ground truth simulation itself. Unless they can argue that the learned model is superior to the classical simulation in a significant way, it is hard to see the benefit of using something like TF-NET.
## Specific Comments
* Page 2: The claim "64.2% reduction in flow divergence, compared to the best baseline" seems misleading. In Figure 5, the constrained TF-NET is as low as ~590 but the ResNet is between ~610 and ~810. I am guessing that the authors meant a 64.2% reduction in *difference* from the Target model, but this should be clarified.
* Page 4: Is 'T' supposed to appear in the denominator of Equation 3? I would have expected this to be a normalizing factor resulting from integrating the filter G over the whole domain.
* Page 4: I would expect filters used for LES are symmetric. Are any symmetry requirements being enforced on the learned filters?
* Page 4: It's not clear what TF-NET is outputting. I would expect it to output the time derivative of the velocity field, but this should be spelled out explicitly.
* Page 5: Since you are using incompressible Navier-Stokes equations, I don't think the Mach number is relevant. IIUC this is only relevant for the propogation of shock waves in compressible flows.
* Page 7: It looks as if the Target model has significant divergence. Why is this? Should we expect this much divergence in the ground truth data?
* Page 8: In homogeneous isotropic turbulent flow, the energy spectrum is governed by Kolmogorov, so we know what the spectrum should look like. Is there an analytic result for RB convection? If so, could you include that on the plot?
* Page 8: The lines for U-Net and TF-NET are nearly indistinguishable in Figure 7. Could you change them?
* Page 8: It seems like it would be worth including the ResNet in the energy spectrum plots as well.
* What did the learned spatial and temporal filters look like? How do they compare to typical 'hand-chosen' filters?
* IIUC, the training, validation and test data all have identical Rayleigh number. Does the learned model generalize to higher/lower energy? This seems critical to making this sort of model useful.
* The paper doesn't describe a tuning process for any of the models. I would expect this to lead to significant improvement. In particular, how do the models' performance change as the weight on the divergence loss term is increased?
### Baselines
* I think the objective should be to show that TF-NET is superior to the ground-truth method in some way. Can it match the results of the ground truth simulator but do it faster, or with fewer resources?
* The authors say "we compare our model with a series of state-of-the-art baselines for turbulent flow prediction." However, most of these models are intended for video prediction or other tasks unrelated to turbulent flows. I wouldn't expect any of the baselines considered to do perform well in this context.
* Since TF-NET gets the benefit of a loss related to divergence, I would have expected the other architectures to get the same treatment. In fact, without this extra loss term, the ResNet architecture is competitive with TF-NET.


The word embedding models have achieved state-of-the-art results in a variety of natural language processing tasks. Whereas, current word embedding models mainly focus on the rich semantic meanings while are challenged by capturing the sentiment information. For this reason, we propose a novel sentiment word embedding model. In line with the working principle, the parameter estimating method is highlighted. On the task of semantic and sentiment embeddings, the parameters in the proposed model are determined by using both the maximum likelihood estimation and the Bayesian estimation. Experimental results show the proposed model significantly outperforms the baseline methods in sentiment analysis for low-frequency words and sentences. Besides, it is also effective in conventional semantic and sentiment analysis tasks.

This paper proposes a method to learn word embedding by incorporating additional sentiment information. The proposed method extends from D-GloVe by adding the probability of positive sentiment to the loss function. The paper presents three experiments: word similarity, word-level sentiment analysis, and sentence-level sentiment analysis. The experiments show that the method performs comparably with other baseline methods and outperforms in the low-frequency sentence setting (i.e. sentence containing lower frequency words).
I recommend rejecting this paper because (1) the writing is unclear and hard to follow, and (2) the experiment results are not convincing.
From what I can understand in the model part, there are many clarifications needed, not to mention the writing style. I think the re-derivations of GloVe and D-GloVe are not helpful as they cloud the main contribution of the paper. The author should clearly highlight the differences between the main subjects of the experiment: MLESWE and BESWE. In addition, it is not clearly motivated why we need Dirichlet prior for the sentiment variable.
While the claim is to learn better embeddings for rare words, the experiments show that the proposed methods have similar results to the previous work. The only gain we can observe is in the sentence-level experiments in which other factors could affect the performance. Thus, it is hard to draw a supportive conclusion.
Finally, the writing quality must be improved. The paper contains a lot of unrelated and redundant texts (it could be that I could follow the paper).
1. I do not think eq 2 is a representative of how the paper train the model, nor attempt to compare with.
2. As mention earlier, in section 3.2 and 3.3, the re-derivation is not particularly helpful. I think the paper should put more emphasis on the novelty of the work.
3. Plots in the experiment results are illegible. Tables should be more suitable for Figures 1, 2, and 3.
I urged the authors to revise this paper and make sure it follows the formatting guideline, especially the citations. Finally, I'd recommend the authors have a professional writer (English) review the paper before submission.


Rules over a knowledge graph (KG) capture interpretable patterns in data and can be used for KG cleaning and completion. Inspired by the TensorLog differentiable logic framework, which compiles rule inference into a sequence of differentiable operations, recently a method called Neural LP has been proposed for learning the parameters as well as the structure of rules. However, it is limited with respect to the treatment of numerical features like age, weight or scientific measurements. We address this limitation by extending Neural LP to learn rules with numerical values, e.g., "People younger than 18 typically live with their parents". We demonstrate how dynamic programming and cumulative sum operations can be exploited to ensure efficiency of such extension. Our novel approach allows us to extract more expressive rules with aggregates, which are of higher quality and yield more accurate predictions compared to rules learned by the state-of-the-art methods, as shown by our experiments on synthetic and real-world datasets.

This paper proposes an extension of NeuralLP that is able to learn a very restricted (in terms of expressiveness) set of logic rules involving numeric properties. The basic idea behind NeuralLP is quite simple: traversing relationships in a knowledge graph can be done by multiplicating adjacency matrices, and which rules hold and which ones don't can be discovered by learning an attention distribution over rules from data.
The idea is quite clever: relationships between numeric data properties of entities, such as age and heigh, can also be linked by relationships such as \leq and \geq, and those relations can be treated in the same way as standard knowledge graph relationship by the NeuralLP framework.
A major drawback in applying this idea is that the corresponding relational matrix is expensive to both materialise, and use within the NeuralLP framework (where matrices are mostly sparse). To this end, authors make this process tractable by using dynamic programming and by defining such a matrix as a dynamic computation graph by means of the cumsum operator.  Furthermore, authors also introduce negated operators, also by defining the corresponding adjacency matrices by means of computation graphs.
Authors evaluate on several datasets - two real world and two synthetic - often showing more accurate results than the considered baselines.
One thing that puts me off is that, in Table 2, AnyBurl (the single one baseline authors considered other than the original NeuralLP) yields better Hits@10 values than Neural-LP-N, but the corresponding bold in the results is conveniently omitted.
Another concern I have is that the expressiveness of the learned rules can be somehow limited, but this paper seems like a good star towards learning interpretable rules involving multiple modalities.
Missing references - authors may want to consider citing https://arxiv.org/abs/1906.06187 as well in Sec. 2 - it seems very related to this work.


We study continuous action reinforcement learning problems in which it is crucial that the agent interacts with the environment only through safe policies, i.e., policies that keep the agent in desirable situations, both during training and at convergence. We formulate these problems as constrained Markov decision processes (CMDPs) and present safe policy optimization algorithms that are based on a Lyapunov approach to solve them. Our algorithms can use any standard policy gradient (PG) method, such as deep deterministic policy gradient (DDPG) or proximal policy optimization (PPO), to train a neural network policy, while guaranteeing near-constraint satisfaction for every policy update by projecting either the policy parameter or the selected action onto the set of feasible solutions induced by the state-dependent linearized Lyapunov constraints. Compared to the existing constrained PG algorithms, ours are more data efficient as they are able to utilize both on-policy and off-policy data. Moreover, our action-projection algorithm often leads to less conservative policy updates and allows for natural integration into an end-to-end PG training pipeline. We evaluate our algorithms and compare them with the state-of-the-art baselines on several simulated (MuJoCo) tasks, as well as a real-world robot obstacle-avoidance problem, demonstrating their effectiveness in terms of balancing performance and constraint satisfaction.

Summary:
Authors propose ideas to perform safe RL in continuous actions domain with modifications to Policy Gradient (PG)  algorithms via either constraining the policy parameters or constraining the actions selected by PG with a surrogate/augmented state dependent objective. The paper is well motivated and the experiments (although I have some reservations about the setup) demonstrate efficacy of the proposed method.
Review:
--> Introduction
I do not agree with the statement that value function based algorithms are restricted to discrete action domains, especially when you rely on  “ignoring function approximation errors” for some of your claims.
Again in switching from value function to PG is true for traditional RL/Control theory but this is not valid here. ( your methods rely on Q(s, a) which is action-value function, or the constraint in equation 3 is integral of Q over all actions which would be value-function in traditional definition )
Note: this is explained very well towards the end in the Appendix B, but this is a review of the paper and not Appendix B or C.
--> Section 2
section 2.3 I would strongly advise the authors to rewrite this, this section reads like it was copied as is from the reference [Chow et al 2018].  especially the way Lyapunov function is defined. And the language and arguments are almost same. Some sentences cite the reference but conclusions drawn on these are not cited, are you claiming that these conclusions are original from this paper ?
It is not clear to me how the feasibility of initial pi_0 is ensured ? Did I miss this somewhere ?
→ Section 3
Section 3 is pleasant to read and very easy to understand, however, same cannot be said of the
section 3.1. I had to spend significant time reading 3.1 and I am still not sure I have understood it very well.
Experiments:
I don’t think halfCheetah-Safe is actually actually an useful experiment, Limiting the joint torques is perfectly understandable, just limiting speed and getting smooth motion could just be an artifact of the simulation environment. Are both constraints applied simultaneously (torque and speed) ? It is unclear from the text.
I am not sure CPO without linesearch is actually a fair comparison. Line search may actually deem of the actions unsafe and therefore I would presume original CPO do be less prone to constraint violation than the proposed modification in your experiments. Again PPO is more heuristic than TRPO which makes it hard to compare like for like. PPO might give higher rewards but constraint violations may increase as well. An important point for Safe-RL I feel.
Figure 6
Can you be more specific as what the figure 6 is showing ? Constraint ? is this constraint violation count? or cumulative sum of constraint slack over the whole trajectory ?

Not part of assessment :
Unclear Statements:
Page 7, DDPG Vs PPO: explain clearly what you mean by  “covariate shift” or remove the statement altogether.
Page 7, section 5 second paragraph, “The actions are the linear …. center of mass” I couldn’t understand this ? What do you mean by actions are velocity ?
Minor points (Language, Typos):
page 3, last paragraph,  Chow et al. is repeated, I can see why this happens there but suggest editing to avoid this. [This is also in intro paragraph, there it is just a typo and should be rectified]
Figure 6: Captions labels are incorrect.


Formal verification of machine learning models has attracted attention recently, and significant progress has been made on proving simple properties like robustness to misclassification under perturbations of the input features. In this context, it has also been observed that folding the verification procedure into training makes it easier to train verifiably robust models. In this paper, we extend the applicability of verified training by extending it to (1) recurrent neural network architectures and (2) complex specifications that go beyond simple adversarial robustness, particularly specifications that capture temporal properties like requiring that a robot periodically visits a charging station or that a language model always produces sentences of bounded length. Experiments show that while models trained using standard training often violate desired specifications, our verified training method produces models that both perform well (in terms of test error or reward) and can be shown to be provably consistent with specifications.

This paper extends bound propagation based robust training method to complicated settings where temporal specifications are given. Previous works mainly focus on using bound propagation for robust classification only. The authors first extend bound propagation to more complex networks with gates and softmax, and designed a loss function that replies on a lower bound for quantitative semantics of specifications over an input set S. The proposed framework is demonstrated on three tasks: Multi-MNIST captioning, vacuum cleaning robot agent and language generation. The authors formulate specifications using temporal logic, train models with bound propagation to enforce these specifications, and verify them after training.
My questions regarding this paper are mostly on the three demonstrated tasks:
1. For the Multi-MNIST dataset, I have the following questions:
1(a). In Table 2, it is surprising that even with perturbation of 0.5, the verified and adversarial accuracy is very high. At perturbation epsilon=0.5, it should be possible to perturb the entire image to gray (value 0.5), so I believe the accuracy should be very low here. It is hard to believe under this setting the verified accuracy is still 99%.
1(b). For Table 3, nominal accuracy should also be reported.
1(c). Additionally, how do you define the nominal accuracy here? An example is nominally correct when all digits are predicted correctly in the sequence, or just when the sequence length is predicted correctly?
1(d). For the termination accuracy, do we only care about the sequence length being predicted correctly, or does it also cover the case that all digits in the sequence are predicted correctly? If it is only concerning about the sequence length, this property is a little bit weak.
2. For the RL Robot agent experiment, I have the following questions:
2(a). Because T=10, are you saying that we can only guarantee that the battery is always recharged for any rollouts less than 10 steps? After 10 steps beyond the initial position, can we get any guarantees? A 10-step only guarantee seems too restrictive.
2(b). Are all the properties only verified assuming that the agent starts from the center? I think this assumption is probably also too strong in practice.
2(c). Since all the %verified cells reported in Table 4 are all 100%, it is probably better to make the problem more challenging, by increasing T or considering different initial positions. It is important to show when the performance of the proposed method starts to degrade, to understand the power of the proposed method.
3. For the language generation experiment, the perplexity of the verified training model looks significantly worse than nominal or sampled models. With a perplexity as high as this, I believe the model actually produces garbage. Can you provide some examples of generated texts? I feel language generation is probably not a suitable task for the proposed training method.
Other minor issues:
1. Table 1 should have some horizontal lines - it is hard to align the works with categories on the right.
2. Several papers appear multiple times in references, including "Differentiable abstract interpretation for provably robust neural networks", "Towards fast computation of certified robustness for relu networks" (and probably others). Also on page 2, Shiqi et al., should be Wang et al. (Shiqi is the first name).
3. I feel the writing is a bit rushed and the authors should make a few more passes on the paper.
This paper makes valid technical contributions, especially the conversion from STL specifications to lower bounds of the quantitative semantics is interesting. Although bound propagation based robust training method is simple to extend to softmax/GRU with interval analysis, applying robust training techniques to the three interesting applications are good contributions. Since the main contribution of this paper is the empirical results on the three tasks, my concerns regarding the experiments need to be addressed before I can vote for accepting this paper. Also, because this paper uses 10 pages, I am expecting the paper to meet a higher standard. Thus, I am voting for a weak reject at this time.
****** After author response
The author response addressed some of my concerns. However I do believe this paper is relatively weak in contribution, especially the experiments can be done more thoroughly. I also appreciate that the authors reduced paper length to 8 pages. I am okay with accepting this paper as it does have some interesting bits, but it is clearly on the borderline and can be further improved.


Minwise Hashing (MinHash) is a fundamental method to compute set similarities and compact high-dimensional data for efficient learning and searching. The bottleneck of MinHash is computing k (usually hundreds) MinHash values. One Permutation Hashing (OPH) only requires one permutation (hash function) to get k MinHash values by dividing elements into k bins. One drawback of OPH is that the load of the bins (the number of elements in a bin) could be unbalanced, which leads to the existence of empty bins and false similarity computation. Several strategies for densification, that is, filling empty bins, have been proposed. However, the densification is just a remedial strategy and cannot eliminate the error incurred by the unbalanced load. Unlike the densification to fill the empty bins after they undesirably occur, our design goal is to balance the load so as to reduce the empty bins in advance. In this paper, we propose a load-balanced hashing, Amortization Hashing (AHash), which can generate as few empty bins as possible. Therefore, AHash is more load-balanced and accurate without hurting runtime efficiency compared with OPH and densification strategies. Our experiments on real datasets validate the claim. All source codes and datasets have been released on GitHub anonymously 1 .

*** Summary ***
MinHash is a well-known method for approximating set similarities in terms of the jaccard similarity. The idea is to use k random permutations (hashes) on all elements of the sets and check how often two sets hash into the same bucket. Larger values of k yield more accurate estimates of the set similarities but require more time to compute. One permutation hashing (OPH) aims to reduce the number of hash computations per element to 1 and maintaining bins. However, some of those bins may remain empty which negatively influences the similarity estimate. Optimal OPH (OOPH) hashes empty bins to non-empty bins and effectively reuses bins. This paper proposes Amortization Hashing (AHash) which reduces the occurrence of empty bins and thus leads to better similarity estimates.
*** Evaluation ***
The paper proposes an interesting idea for approximating set similarities much faster than MinHash. However, I have some issues with the submission.
I believe that the manuscript has a limited impact. The approach performs on par or marginally better as OOPH within the first and only reasonable experiment (4.2). As the authors state themselves on the page break 9/10, the advantage of AHash vanishes for small and large values of k. Hence, AHash only benefits of moderate choices of k. Moreover, I can see that OOPH might have a minor problem of estimating the set similarities which AHash aims to fix, but why should it outperform MinHash in terms of accuracy? Why are the pairs of set only chosen from RCV1? Why those particular set sizes? Why does no plot show standard deviation/error?
The remaining experiments yield very limited insight. Considering a linear SVM on standard datasets where the test error is >99.8% seems to be obsolete. In addition, the most important parameter k is held fix to an arbitrary value. Same holds for b. Since AHash only benefits from moderate sizes of k, why was k chosen in favor of AHash? The performance should definitely be shown in dependence of k. Instead, the most unimportant parameter (C) is varied. This should have been done in a proper cross-validation. Similar arguments hold for the near neighborhood search. What is the query set being used?
There are more flaws within the manuscript. The mathematical presentation is rather poor. The theorems lack text and assumptions and solely consist of equations. The corresponding proofs are also short on text and hard to follow. Unfortunately, there is no analysis of the expected error as a function of k. The proof of Theorem 3 is almost two pages and should be moved to the supplementary material since it does not provide much insight; it just distracts the reading flow. In addition, every equation is numbered but none is ever referenced. The citation style (numbers in round brackets) is really uncommon and can be easily confused with equation numbers. Most importantly, I want to note that a different font was used and that the spacing was clearly tricked in several places (e.g. within Section 4). This makes it especially hard to judge whether the manuscript has the correct length.
*** Further Comments ***
- The font was changed. It does not match the font of the other submissions.
- The spacing is tricked in several places, especially in Section 4.
- Links [1,29] should not be references but footnotes.
- Citations should never be in round brackets like (1), because they can be confused with equation numbers. Instead they should be in square brackets like [1] or, more preferably, the natbib package should be used as in the ICLR style guidelines.
- What does OOPH stand for? It is never stated.
- Every equation has a number, but none is ever referenced.
- Math/Equations are part of the text and should be treated as such, i.e., there should be proper punctuation marks.
- What is a 2-universal hashing?
- Algorithm 1: "output range" sounds like an interval whereas the number of distinct hash values is meant.
- "(14) proposed", no past tense
- Instead of "(11) proposes", please use "Shrivastava and Li [11] propose"
- Why "Theorem 1" and "Proof 3.1"?
- Why are the theorems lacking the assumptions and text? They basically consist of equations.
- Theorem 3 should have a "less or equal" instead of a "strictly less".
- Eq. (40): "0andm"
- Why does Proof 3.3 have a end of proof sign (not right-aligned) but the other proofs don't?
- None of the experimental results shows standard deviations/errors although the experiments are repeated several times. Why? It would be also nice to see whether the approximation tends to over- or underestimate J. This could be done with a violin plot.
- How are the pairs of sets in Section 4.2 chosen and why only from RCV1? This seems to be the most important experiment.
- Why is k (and b) fixed to an arbitrary value in the remaining experiments? Please select C within a proper cross-validation.
- There are a lot of enumerations which unnecessarily make the manuscript longer, e.g. in Sections 1.4, 2.1 and 4.1. In addition, the proof of Theorem 3 almost takes two pages but is not super informative. It should be moved to the supplementary material. This in combination with the font mismatch makes it difficult to determine the real length of the submission.
- It is nice that the source code is published online, but uncommented c++ code is not really helpful.


Polynomial neural networks are essentially polynomial functions. These networks are shown to have nice theoretical properties by previous analysis, but they are hard to train when their polynomial orders are high. In this work, we devise a new type of activations and then create the Ladder Polynomial Neural Network (LPNN). The LPNN has a feedforward structure. It provides good control of its polynomial order because its order increases by 1 with each of its hidden layers. The new network can be treated with deep learning techniques such as batch normalization and dropout. As a result, it can be well trained with generic optimization algorithms regardless of its depth. In our empirical study, deep LPNN models achieve good performances in a series of regression and classification tasks.

This paper proposes Ladder Polynomial Neural Networks (LPNNs) that use a new type of activation primitive -- a product activation -- in a feed-forward architecture. Unlike other polynomial architectures that grow in the order exponentially with network depth, the proposed approach gives explicit control over the order and smoothness of the network output and enables training with standard techniques.

The proposed architecture is closely related to a decomposition of a k’th order multivariate polynomial function
[T, x^{\otimes k}] = \lambda^\top (A x \odot A x \odot …  \odot A x)	=  \lambda^\top (A x)^{\odot k}
where T is a symmetric tensor of polynomial coefficients and [\cdot,\cdot] denotes contraction. This is a shallow (one layer architecture) and sometimes referred as a Waring decomposition.
In this paper, the authors propose a specific chain factorization of the polynomial (Eq 5 in the paper), where they write the factors recursively, that they name as a ladder polynomial neural network.
h^\ell = (W_\ell h^{\ell-1} \odot V^{\ell} x)
The ladder architecture is very closely related to tensor trains (https://epubs.siam.org/doi/10.1137/090752286). I found it surprising and somewhat alarming that this literature is not being cited as these methods are also quite well known in deep learning.
I like the smoothness analysis of section 3.1 -- the proof is quite easy to follow and direct. I would be quite surprised if this result would not be known in the literature in some other form but I don’t recall seeing it. On the other hand it seems to be inevitably very loose for a deep ladder network unless the network models the zero function. It would have been a valuable addition to the experimental section, if this bound would have been illustrated numerically on synthetic examples.
In 3.2, The authors say that the objective is multiconvex -- I would argue that it is multilinear (apart from the regularization term, that is later introduces). The observation in 3.3, that batch-normalization or dropout can be used for this model is perhaps tangential to the main argument. These is investigated in the experimental section but I don’t see a clear conclusion. The section in 3.4 must include links to tensor decompositions beyond factorization machines.

Overall, I think the paper has some merit and could be interesting for some readers, despite the fact that the contribution is not very original and the treatment could be improved in many ways.


We address the problem of reconstructing a matrix from a subset of its entries. Current methods, branded as geometric matrix completion, augment classical rank regularization techniques by incorporating geometric information into the solution. This information is usually provided as graphs encoding relations between rows/columns. In this work we propose a simple spectral approach for solving the matrix completion problem, via the framework of functional maps. We introduce the zoomout loss, a multiresolution spectral geometric loss inspired by recent advances in shape correspondence, whose minimization leads to state-of-the-art results on various recommender systems datasets. Surprisingly, for some datasets we were able to achieve comparable results even without incorporating geometric information. This puts into question both the quality of such information and current methods' ability to use it in a meaningful and efficient way.

This paper proposes a new method for geometric matrix completion based on functional maps. The proposed algorithm is a simple shallow and fully linear network. Experimental results demonstrate the effectiveness of the proposed method.
The proposed method is new and has been shown good empirical results. The paper also points out a new way to interpret matrix completion. On the other hand, the proposed method seems ad hoc and there is no clear evidence why it is better than other baselines except the empirical results. The paper also has some clearance issues, making it hard to understand. I vote for a weak reject of the paper at the current pace and would like to increase my score if the following questions can be clearly answered.
1.	Why do we need to propose the algorithm? Is it because we have the functional maps technique motivated from shape correspondence, and we can see some connection of such technique with matric completion? If it is true, we surely can have a new algorithm based on such a new technique. But I can still not understand why the method work, at least, in an intuitive way.
2.	What is the sample complexity of the proposed matrix completion algorithm?
The introduction of the paper is poorly written. The first paragraph and the third one both contain some introduction to matric completion, which results in a lot of redundant information. The second paragraph and the fourth one are redundant in the same way since they both focus on geometric matrix completion. I think besides introducing what is matrix completion and what is geometric completion, the introduction part should focus more on the motivation to propose the algorithm. However, I can only see from the end of the second paragraph (some simple models need to be proposed) and the fifth paragraph (“The inspiration of our paper”) some motivation information. The introduction part needs to be re-organized to provide more useful information about the paper rather than a literature review.
There is some unclear/inaccurate/subjective statement in the introduction part. For example, “Self-supervised learning” needs a reference. Why geometric matrix completion generalizes the standard deep learning approaches is not clear. What does it mean by “their design is … cumbersome and non-intuitive”? The shape correspondence is never explained until very later in the paper.  Also, there are some unclear issues besides the Introduction part. For example, what does it mean by “the product graph”? All these issues need to be clarified before the paper can be accepted.
---------------------------------------------------
Thank you for the detailed rebuttal. For Q1, it clearly explains how does the method work. However, it is still not clear why does the method work. I also have another concern after reading the rebuttal, if the shape correspondence is not that important, why make it an important motivation in the paper? For Q2, it is interesting to see some theoretical results on the sample complexity, rather than an experimental one. The paper would also be much better if the clearance issues can be addressed. Even if I would not vote for an accept this time, I am looking forward to a revised version in the future.


The population model is a standard way to represent large-scale decentralized distributed systems, in which agents with limited computational power interact in randomly chosen pairs, in order to collectively solve global computational tasks. In contrast with synchronous gossip models, nodes are anonymous, lack a common notion of time, and have no control over their scheduling. In this paper, we examine whether large-scale distributed optimization can be performed in this extremely restrictive setting. We introduce and analyze a natural decentralized variant of stochastic gradient descent (SGD), called PopSGD, in which every node maintains a local parameter, and is able to compute stochastic gradients with respect to this parameter. Every pair-wise node interaction performs a stochastic gradient step at each agent, followed by averaging of the two models. We prove that, under standard assumptions, SGD can converge even in this extremely loose, decentralized setting, for both convex and non-convex objectives. Moreover, surprisingly, in the former case, the algorithm can achieve linear speedup in the number of nodes n. Our analysis leverages a new technical connection between decentralized SGD and randomized load-balancing, which enables us to tightly bound the concentration of node parameters. We validate our analysis through experiments, showing that PopSGD can achieve convergence and speedup for large-scale distributed learning tasks in a supercomputing environment.

This paper proposes to use population algorithms as a mechanism for implementing distributed training of deep neural networks. The paper makes some claims about the relationship to previous work on (asynchronous) gossip algorithms that appear to be incorrect. In fact, the proposed PopSGD algorithm is very closely related to other methods in the literature, including AD-PSGD (Lian et al. 2017b) and SGP (Assran et al. 2018). I recommend it be rejected due to lack of novelty and missing connections to much related work.
The introduction (page 3) mentions that the "matrix characterization is not possible in the population model." Here the "matrix characterization" refers to the typical approach in which gossip algorithms (synchronous or asynchronous) are formulated and analyzed. I'd appreciate if the authors could elaborate on this claim. In the study of gossip algorithms, the organization of time into "global rounds" is purely for the sake of analysis; a global, synchronized clock is not required to implement these methods. In fact, the description of the setup appears to be very similar to the asynchronous time model described used to analyze "randomized gossip algorithms" (see the well-cited paper by Boyd, Ghosh, Prabhakar, and Shah). In the PopSGD case, the choice is simply to allow the complete graph (i.e., any agent can interact with any other agent) rather than restricting interactions of a given agent to be among a subset of the other agents (i.e., its neighbors).
Let me elaborate on the ways in which PopSGD is similar to AD-PSGD and SGP. PopSGD involves interactions between randomly drawn pairs of agents. The AD-PSGD algorithm of Lian et al. (2017b) also performs updates between pairs of agents drawn randomly at every step. The definition of the PopSGD interaction in (1.1) (or equivalently Alg 1) implies that when agents i and j interact, neither i nor j can interact with another agent until the current interaction completes. The main difference appears to be that in Lian et al. (2017b) agents are organized into a bipartite graph where $n/2$ nodes are "active" and initiate interactions with one of the other $n/2$ "passive" nodes (drawn randomly). This is done for practical reasons - to avoid deadlocks.
I also believe that PopSGD can be viewed as a particular instance of the overlap-SGP algorithm proposed in Assran et al. (2018). Overlap-SGP, the way it is described, makes use of one-directional interactions (agent i may receive and incorporate information from agent j without the reverse happening simultaneously). This was also introduced for practical reasons. It is possible for multiple interactions to happen simultaneously, and the pattern of iteractions may vary over time. There is nothing in the analysis, however, that prevents one from restricting to symmetric interactions, in which case one could recover the symmetric updates of PopSGD. To compensate for one-directional interactions, Overlap-SGP tracks an additional variable (the weight, or denominator). However, in the case where interactions are always symmetric as in PopSGD, the corresponding update matrices will always be doubly-stochastic, and in this case the weights are always equal to 1. Thus PopSGD really is identical to Overlap-SGP in this special restricted case where interactions are always pair-wise and symmetric. Moreover, Assran et al. (2018) prove that Overlap-SGP achieves a linear speedup in the smooth non-convex setting.
The experiments don't provide any comparison with other related methods, and the discussion in the introduction isn't sufficient to convince me that there are significant differences between these methods. In the experiments, I also wanted to ask about the mult constant. If it is really possible to achieve linear scaling, wouldn't one hope to be able to get away with mult=1?
The decreasing learning rate schedule used in the description and analysis of PopSGD seems very restrictive. Specifically, in the training of deep neural networks it is common to use much different learning rate schedules. Is it fundamentally not possible to do so with PopSGD-type models, or is it just a limitation of the current analysis approach (specifically for convex functions)? What learning rate scheme was used in the experiments?
Finally, the introduction (p3) emphasizes that it is the population gradient perspective, and the connection to load-balancing processes, which enable one to achieve linear scaling. I disagree with this statement. While I do agree that convexity alone is not sufficient, the key assumption made here (as well as in other work, such as that of Lian et al.), is that all agents draw gradients from the same distribution; i.e., that all agents have access to independent and identically distributed stochastic gradient oracles. In fact, this is stronger than the assumptions made in Lian et al. (2017a and 2017b), and Assran et al. (2018), where it is only assumed that the gradient oracles at each agent are similar, but not necessarily identical.


With a view to bridging the gap between deep learning and symbolic AI, we present a novel end-to-end neural network architecture that learns to form propositional representations with an explicitly relational structure from raw pixel data. In order to evaluate and analyse the architecture, we introduce a family of simple visual relational reasoning tasks of varying complexity. We show that the proposed architecture, when pre-trained on a curriculum of such tasks, learns to generate reusable representations that better facilitate subsequent learning on previously unseen tasks when compared to a number of baseline architectures. The workings of a successfully trained model are visualised to shed some light on how the architecture functions.

This paper presents PrediNet: an architecture explicitly designed to extract representations in the form of three-place predicates (relations). They evaluate the architecture on a visual relational task called the "Relations Game" which involves comparing Tetris-like shapes according to their appearance, relative positions, etc.. They show that their architecture leads to useful generalizable representations in the sense that they can be used for new tasks without retraining.
I think this paper contains a number of unusual and interesting ideas but is let down by its presentation. The writing is good, but provides very little intuition for why we should expect this approach to work (aside from its connection to equation 1) - I discuss this in more depth below. The experimental task is interesting (I'm okay with synthetic tasks of this form for unusual new architectures like this), but I'm not sure what it tests that isn't tested in the CLEVR and sort-of -CLEVR datasets which rely on similar relational reasoning to solve. The advantage of those datasets is they are well-established with strong baselines so we can be more certain that a fair comparison is been made. I've voted to reject this paper because I feel its premature in its current form.
Expanding on this - The description of PrediNet covers the basics, but missing detail and intuition for why certain choices are made. For example,
 - why is L flattened for the queries (I think it’s because the query is independent of pixel location, but flattening seems of when L also includes co-ordinates) but not the key, K?
 - Why is the key space shared between heads (this seems more intuitive - keys should have consistent meaning… but if that’s the case, make that intention explicit)?
 - Also, writing the dimensionality of the matrices, would help (e.g. is W_S in R^{m x j} or R^{m x 1} or something else?).
 - What is the meaning of the position features in E_1 and E_2? From the softmax product it seems they should be a weighted sum of the pixels that are addressed - implying that it is the weighted average location?
 - The final representation mostly consists of an h x (j) vector (ignoring positions) containing the output of the comparators. Could you provide some intuition for why we would expect such a representation to be useful for the downstream task? This representation seems to differ substantially from what is used in the baseline methods: i.e. attention-weighted sums of the input features.


Generalization of deep networks has been of great interest in recent years, resulting in a number of theoretically and empirically motivated complexity measures. However, most papers proposing such measures only study a small set of models, leaving open the question of whether the conclusion drawn from those experiments would remain valid in other settings. We present the first large scale study of generalization bounds and measures in deep networks. We train over two thousand convolutional networks with systematic changes in commonly used hyperparameters. Hoping to uncover potentially causal relationships between each measure and generalization, we analyze carefully controlled experiments and show surprising failures of some measures as well as promising measures for further research.

The paper aims at providing a better understanding of generalization for Deep Learning models. The idea is really interesting for the ML community as, despite their broad use, the astounding property of deep neural networks to generalize that well is still not well understood.
The idea is not to show new theoretical bounds for generalization gaps but stress the results of an empirical study comparing the already existing measures. The authors choose 7 common hyperparameters related to optimization and analyze the correlation between the generalization  gaps effectively observed and the ones predicted by different measures (VC dim, cross-entropy, canonical ordering …).
The writing of the paper is clear and easily understandable. Besides, I believe that the study is relevant to ICLR conference.
However, I believe the level of the paper is marginally below the threshold of acceptance and therefore would recommend to reject it.
The paper is solely empirical but I believe that the empirical section is a bit weak, or at least some important points remain unclear. If I appreciate the extent efforts made in trying to evaluate different measures of generalization gaps, I do not believe that the findings are conclusive enough.
1) First, all this empirical result are based on one-dataset (CIFAR-10) only thus limiting the impact of the study. Indeed, a given measure might very correlate with generalization gap on this specific dataset but not on others.
2) Specifically, we see that on this specific dataset, all training accuracies are already quite good (cf : Figure 1, distribution of the training losses). Consequently, authors are more correlating the chosen measures with the test error rather than with the generalization gaps. On other more complicated datasets where the training loss is higher, the VC dimension might consequently have way better results.
Similarly, in Section 6, the authors say that the results «confirm the widely known empirical observation that over-parametrization improves generalization in deep learning. » In this specific case, no reference was given to support the claim. I would agree with the claim « over-paramatrization improves test accuracy (reduces test error) » but the link between over-parametrization and generalization is less clear.
3) In Section 4, the authors say « drawing conclusion from changing one or two hyper-parameters » can be a pitfall as « the hyper-parameter could be the true cause of both change in the measure and change in the generalization ». I totally agree with the authors here. Consequently, I do not understand why the correlations were measured by only changing one hyper-parameter at a time instead of sampling randomly in Theta.
4) It is still not clear to me how the authors explain why some measures are more correlated with generalization gaps than others. Are some bounds tighter than others ? This empirical study was only applied to convolutional neural networks and consequently one may wonder that for example the VC dim bounds computed in the specific case of neural networks are too loose. However, this measure could be efficient for type of models.
I would like the authors to clear the following points :
- How do you ensure that the empirical study clearly correlated measures predictions with generalization gaps and not simply with test errors (or accuracies) ? (point 2)
- Could you please also answer Point 4 ?
- Finally, how would you explain the fact that the canonical order performs so well compare to many other measures and that it is a really tough-to-beat baseline ?


The performance of deep network learning strongly depends on the choice of the non-linear activation function associated with each neuron. However, deciding on the best activation is non-trivial, and the choice depends on the architecture, hyper-parameters, and even on the dataset. Typically these activations are fixed by hand before training. Here, we demonstrate how to eliminate the reliance on first picking fixed activation functions by using flexible parametric rational functions instead. The resulting Padé Activation Units (PAUs) can both approximate common activation functions and also learn new ones while providing compact representations. Our empirical evidence shows that end-to-end learning deep networks with PAUs can increase the predictive performance. Moreover, PAUs pave the way to approximations with provable robustness.

This paper introduces a novel parametric activation function, called the Pade Activation Unit (PAU), for use in general deep neural networks. Pade is a rational function, which is a ratio of two polynomials, and which can very well approximate any of the usually used activation functions while having only a few parameters that can be learned from data. Moreover, the authors identify five properties that an activation function should have, and either prove or empirically show that PAUs satisfy all of them, unlike some of the baselines. Additionally, since Pade approximation can have poles and be unstable, this work introduces safe PAUs, where the polynomial in the denominator is constrained to attain values greater than or equal to one. Since one of the suggested properties is that a function using a given activation function be a universal function approximator, the authors provide a sketch of a proof that PAUs do allow that. This proof applies only to the unsafe version of the PAU, and it is unclear whether it extends to the safe PAU---an issue that is not mentioned by the authors.
Furthermore, the authors propose a stochastic version of PAU with noise injected into parameters, which allows regularization. The empirical evaluation is quite extensive, and the PAU is compared against nine baselines on five different architectures (LeNet, VGG, DenseNet, ResNet, MobileNet) on four different datasets (MNIST, Fashion MNIST, CIfar10, ImageNet) for the classification task. The evaluation confirms that PAUs can match the performance of or sometimes outperform even the best baselines while the attained learning curves show that PAUs also lead to faster convergence of trained models. Finally, the authors demonstrate that (and provide intuition why) using PAUs allow for high-performing pruned models.
I recommend ACCEPTing this paper as it is well written, extensively evaluated, and provides performance improvements or at least matches the performance of the best baseline across several datasets and model architectures.
My only two suggestions for improvement are a) make the universal approximation proof tighter by making sure that it extends to the safe PAU version, and b) evaluate the proposed activation function on tasks other than just classification.


Nesterov SGD is widely used for training modern neural networks and other machine learning models. Yet, its advantages over SGD have not been theoretically clarified. Indeed, as we show in this paper, both theoretically and empirically, Nesterov SGD with any parameter selection does not in general provide acceleration over ordinary SGD. Furthermore, Nesterov SGD may diverge for step sizes that ensure convergence of ordinary SGD. This is in contrast to the classical results in the deterministic setting, where the same step size ensures accelerated convergence of the Nesterov's method over optimal gradient descent. To address the non-acceleration issue, we introduce a compensation term to Nesterov SGD. The resulting algorithm, which we call MaSS, converges for same step sizes as SGD. We prove that MaSS obtains an accelerated convergence rates over SGD for any mini-batch size in the linear setting. For full batch, the convergence rate of MaSS matches the well-known accelerated rate of the Nesterov's method. We also analyze the practically important question of the dependence of the convergence rate and optimal hyper-parameters on the mini-batch size, demonstrating three distinct regimes: linear scaling, diminishing returns and saturation. Experimental evaluation of MaSS for several standard architectures of deep networks, including ResNet and convolutional networks, shows improved performance over SGD, SGD+Nesterov and Adam.

The authors present a new first order optimization method that adds a corrective term to Nesterov SGD. They demonstrate that this adjustment is necessary and sufficient to benefit from the faster convergence of Nesterov gradient descent in the stochastic case. In the full-batch (non deterministic) setting, their algorithm boils down to the classical formulation of Nesterov GD. Their approach is justified by a well conducted theoretical analysis and some empirical work on toy datasets.
Positive points:
- The approach is elegant and thoroughly justified. The convergence to Nesterov GD when the batch size increase is comforting.
- The empirical evaluation, even if it is still preliminary and larger scale experiments will have to be conducted before the method could be widely adopted, are suitable and convincing.
- Some interesting observations regarding the convergence regimes (in respect to the batch size) are made. It would have been interesting to see how the results from fig3 generalize to the non convex problems considered in the paper.
Possible improvements:
- In H2, it is mentioned that the algorithm is restarted (the momentum is reset) when the learning rate is annealed. Was this also done for SGD+nesterov? Also, I think it is an important implementation detail that should be mentioned outside of the appendix
- Adam didn’t get the same hyper-parameter tuning as MaSS did. It is a bit disappointing, as I think the superior performance (in generalization) of non-adaptive methods would still hold and the experiment would have been more convincing. Rate of convergence is also not reported for Adam in fig 5.
I think this is definitely a good paper that should be accepted. I’m looking forward to see how it performs on non-toy models and if the community adopt it.


Neural networks are known to be sensitive to adversarial perturbations. To investigate this undesired behavior we consider the problem of computing the distance to the decision boundary (DtDB) from a given sample for a deep neural net classifier. In this work we present an iterative procedure where in each step we solve a convex quadratic programming (QP) task. Solving the single initial QP already results in a lower bound on the DtDB and can be used as a robustness certificate of the classifier around a given sample. In contrast to currently known approaches our method also provides upper bounds used as a measure of quality for the certificate. We show that our approach provides better or competitive results in comparison with a wide range of existing techniques.

This paper proposed to use a convex QP relaxed formulation to solve the neural network verification problem, and demonstrated its effectiveness on a few small networks (1-2 hidden layers) on MNIST and Fashion-MNIST datasets.
There are several benefits the proposed methods: they are technically tighter relaxations of ReLU neurons and empirically the authors show they perform well in L2 norm (but not L infinity norm, unfortunately); solving this formulation does not require to know pre-activation bounds of hidden neurons; also the convexity of the QP problem needs to be determined only once for a model, rather than once per example. Although the QP relaxation of ReLU neuron is not new and has been used in Raghunathan et al., 2018b, they solve the problem as a SDP rather than convex QP. SDP is tighter than the convex QP formulation used in this paper, however is much slower.
Issues and Questions:
1. The concept of "bi-direction verification" is not new, since finding an upper bound is basically finding adversarial examples. Many previous papers have been using PGD based attacks to obtain the upper bound. Convex relaxation based verification methods like CROWN can also be used for generating adversarial examples, and it is called "Interval attack", which is demonstrated in [1][2]. Claiming this is the first "bi-direction robustness verification technique" is not accurate.
2. The use of FGSM as an upper bound is inappropriate, as FGSM is known to be a very weak attack. Replacing it with a multi-step PGD attack is necessary. Using a stronger attack will also close the gap between upper and lower bound. Also, compare the upper bound found by PGD with QPRel-LB and update Figure 3(a). If a stronger attack like PGD is used, I think for larger norms CROWN+PGD in Figure 3(c) should be able to verify almost all examples.
3. The models used in Table 1 is trained using a L2 perturbation of epsilon=0.1. This epsilon value is too small for L2 norm. In page 22 (last page of appendix in arxiv version) of [3], you can find the they conduct L2 robustness training but at a much larger epsilon value (eps=1.58). Sine the authors did not use these standard epsilon setting, my concern is that does the proposed method works at larger L2 epsilon?
4. Some experiments on larger and deeper networks are necessary; especially, it is interesting to see how CROWN and the proposed method scale to deeper networks. The presented experiments only include networks with 1 and 2 hidden layers, which is insufficient. A new experiment with number of hidden neurons per layer kept (say 50) and increase the depth from 2 to 10 will be very helpful.
5. The main claim of the paper in Introduction needs to be made clearer, especially the primary strength of the proposed algorithm is in L2 norm, and it does not seem to outperform CROWN in L infinity norm setting.
Further improvements and potential directions:
1. In the proposed method, the authors relaxed ReLU neurons using quadratic programming. This relaxation does not require to computing bounds for the neuron activation values. However, I think it is possible to include neuron activation upper and lower bounds as constraints of the QP problem (adding them as constraints like l <= x <= u in Eq. QPRel). This will make the bounds tighter. The per-neuron lower and upper bounds can be obtained using CROWN efficiently, so there is no too much computation cost.
2. Improving the scalability of QP relaxation is another challenge. CROWN can be implemented efficiently on GPUs [4]. For QP relaxations, this can possibly be done by transforming QP solving into a computation graph that can be executed efficiently on GPUs (this is a potential future work directions and I do not expect the authors to address them during the discussion period).
Overall I am positive with this paper, however before accepting it I think the authors should at least make their claims clearer (the relaxation performs well mainly in L2 norm, and the concept of "bi-directional verification" is also not entirely new), replacing FGSM by a 200-step PGD and compare the upper bound found by PGD with QPRel, and test the proposed algorithm in models trained with a larger epsilon (eps=1.58 to align with previous works, if possible) and deeper models.
[1] Wang, S., Chen, Y., Abdou, A., & Jana, S. (2019). Enhancing Gradient-based Attacks with Symbolic Intervals. arXiv preprint arXiv:1906.02282.
[2] Wang, S., Chen, Y., Abdou, A., & Jana, S. (1811). MixTrain: Scalable Training of Verifiably Robust Neural Networks.
[3] Wong, E., Schmidt, F., Metzen, J. H., & Kolter, J. Z. (2018). Scaling provable adversarial defenses. In Advances in Neural Information Processing Systems (pp. 8400-8409).
[4] https://github.com/huanzhang12/RecurJac-and-CROWN


Recent deep generative models are able to provide photo-realistic images as well as visual or textual content embeddings useful to address various tasks of computer vision and natural language processing. Their usefulness is nevertheless often limited by the lack of control over the generative process or the poor understanding of the learned representation. To overcome these major issues, very recent work has shown the interest of studying the semantics of the latent space of generative models. In this paper, we propose to advance on the interpretability of the latent space of generative models by introducing a new method to find meaningful directions in the latent space of any generative model along which we can move to control precisely specific properties of the generated image like the position or scale of the object in the image. Our method does not require human annotations and is particularly well suited for the search of directions encoding simple transformations of the generated image, such as translation, zoom or color variations. We demonstrate the effectiveness of our method qualitatively and quantitatively, both for GANs and variational auto-encoders.  (Brock et al., 2018), showing that the position of the object can be controlled within the image.

This paper proposes a method to learn and control continuous factors of variations within generative models by finding meaningful directions in the latent space which correspond to specified properties. A new method is proposed for inverting generative models and embedding images in the latent space when an encoder is not available. Specifically, reconstruction error is defined in the Fourier domain such that the weighting on high frequency image components can be reduced. Results are evaluated with qualitative comparison to previous embedding methods. Using this image embedding technique, a dataset of latent space trajectories is created by manipulating a desired property in images (such as position or scale) via affine transformations and recording the latent space vectors of the original and new images. The dataset is then used to learn a simple model of the latent space transformation corresponding to changes in the desired image property, which in turn can be used to manipulate images accordingly. To evaluate the effectiveness of this image manipulation approach, a saliency detector is used to measure the change in position or scale of objects in generated images as the latent codes are changed.
Overall, I would tend towards accepting this work. The goal of being able to manipulate continuous factors of variation within generative models is useful for controllable image synthesis, and the proposed method clearly achieves the desired result.
Things to improve the paper:
1) The paper proposes a new reconstruction error metric which is optimized to embed images into the latent space of the generative models. While this new metric is compared qualitatively to existing methods, quantitative evaluation is lacking. It would be useful to also include quantitative comparison of methods measuring the perceptual distance between the original image and the embedded image, perhaps by using Learned Perceptual Image Patch Similarity (LPIPS) [1].
Minor things to improve the paper that did not impact the score:
2) In the abstract: "Our method is weakly supervised...". I am not sure if this method would be considered weakly supervised. I might tend more towards calling it self-supervised, since we have exact labels that are derived from transformations applied to the images themselves.
3) In the first paragraph of the introduction: "an increasing number of applications are emerging such as image in-painting, dataset-synthesis, deep-fakes... ". I find the use of the ellipses here to be a bit strange, since it seems like the sentence is trailing off mid-thought. I would recommend the use of "etc." over "...".
4) In Section 2.2, second paragraph, the dSprite dataset is mentioned but not cited. The reference is not given until Section 3. Should the citation be paired with the first mention of the dataset? Or even just in both places.
5) In Section 3, Implementation details: "The first part is injected at the bottom layer while next parts are used to modify the style of the generated image thanks to AdaIN layers (Huang & Belongie, 2017)". BigGAN uses conditional BatchNorm instead of AdaIN, although they are both very similar. I think the proper citation here is [2], which first introduced conditional BatchNorm.
Questions:
6) I am not fully convinced of the argument that using a saliency detector makes the method more general purpose than a dedicated object detector. The majority of high quality generative models are class conditional, hence requiring a labelled dataset, and therefore an object detector can easily be trained on the same dataset. Additionally, Section 3.2 mentions that "We performed quantitative analysis on ten chosen categories for which the object can be easily segmented by using saliency detection approach", which seems to indicate that the saliency detector struggles with some objects. How does the saliency detector perform on more complicated objects?
References:
[1] Zhang, Richard, et al. "The unreasonable effectiveness of deep features as a perceptual metric." Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2018.
[2] De Vries, Harm, Florian Strub, Jérémie Mary, Hugo Larochelle, Olivier Pietquin, and Aaron C. Courville. "Modulating early visual processing by language." In Advances in Neural Information Processing Systems, pp. 6594-6604. 2017.
### Post-Rebuttal Comments ###
Thanks you for addressing my concerns and for adding the quantitative reconstructions measures. Appendix C looks much more complete now. My overall opinion of the paper remains about the same, so I will leave my score unchanged.


Factorization Machines (FMs) is an important supervised learning approach due to its unique ability to capture feature interactions when dealing with highdimensional sparse data. However, FMs assume each sample is independently observed and hence incapable of exploiting the interactions among samples. On the contrary, Graph Neural Networks (GNNs) has become increasingly popular due to its strength at capturing the dependencies among samples. But unfortunately, it cannot efficiently handle high-dimensional sparse data, which is quite common in modern machine learning tasks. In this work, to leverage their complementary advantages and yet overcome their issues, we proposed a novel approach, namely Deep Relational Factorization Machines, which can capture both the feature interaction and the sample interaction. In particular, we disclosed the relationship between the feature interaction and the graph, which opens a brand new avenue to deal with high-dimensional features. Finally, we demonstrate the effectiveness of the proposed approach with experiments on several real-world datasets.

In this paper, the authors propose generalize the FM to consider both interaction between features and interaction between samples. For the interaction between features, the authors propose to use graph convolution to capture high-order feature interactions. Moreover, the authors construct a graph on the instances based on similarity. Then a GCN is applied to the sample graph where the feature embedding is shared between the two components. Experiments are carried out on four datasets with tasks of link prediction and regression. Comparison to several baselines demonstrate the superior performance of the proposed method.
Strength:
1. The idea of utilizing GCN on the feature co-occurrence graph is interesting and innovative. The idea could possibly be combined with other variants of Deep FM models.
2. It is an interesting idea to combine sample similarity together with feature co-occurrence for better prediction accuracy.
Weakness:
1. Many descriptions in the paper are not very clear. First, the authors only mention how prediction is carried out with trained parameters. However, there is no description of the training process like what is the target used for the two components. What is the training procedure? Are the two components trained jointly? Second, the authors provide little description on how the sample similarity graph is constructed excepts for the Ad campaign dataset. Third, it is not clear how is the link prediction evaluation carried out. From the size of the graph, the authors seem to include both user and item in the graph. However, the user and item has disjointed feature set. It is not clear how the GCN is computed for the heterogenous nodes in the graph. Moreover, how is link prediction carried out, by taking inner product (cosine similarity) of the final representation.
2. For equation (8) in section 4.1, why we need to compute h_i^{RFI}. This should be the feature representation of sample i. However, the average is computed without include sample i itself. Also, are the neighbors defined in the sample similarity graph? Should we use the sample interaction in section 4.2 to capture that?
3. Though it is interesting idea to use graph convolution on the feature occurrence graph, it would be much better if the authors could provide more intuition on the output of the GCN. It would be helpful to study a few simple cases like without non-linearity. Is it a generalization to high-order FM without non-linearity? Also, it would be interesting to see experiments results using the graph convoluted feature representation directly for final representation. Also, some visualization of the learned feature embedding also helps.
4. The authors should carry out ablation study for different components of the model. Moreover, it would be much better if the authors can carry out experiments on some widely used recommendation datasets and use standard evaluation metrics for ranking.