Due to high utility in many applications, from social networks to blockchain to power grids, deep learning on non-Euclidean objects such as graphs and manifolds continues to gain an ever increasing interest. Most currently available techniques are based on the idea of performing a convolution operation in the spectral domain with a suitably chosen nonlinear trainable filter and then approximating the filter with finite order polynomials. However, such polynomial approximation approaches tend to be both non-robust to changes in the graph structure and to capture primarily the global graph topology. In this paper we propose a new Fractional Generalized Graph Convolutional Networks (FGCN) method for semisupervised learning, which casts the Lévy Flights into random walks on graphs and, as a result, allows to more accurately account for the intrinsic graph topology and to substantially improve classification performance, especially for heterogeneous graphs.

This paper proposes a fractional graph convolutional networks for semi-supervised learning. The proposed method used a classification function of a fractional graph semi-supervised learning (GSSL) [De Nigris et al., 2017] as a graph filter. In addition, the authors adopt a parallel system and weighted combination of max and average pool. Experimental results show that the proposed method (FGCN) shows the best accuracy compared to other recent graph-based neural networks for all datasets except one.
The key approach of the proposed method is to apply a classification function (equation (3)) obtained by solving a GSSL problem to graph convolutional networks. However, this idea is too incremental and applying the classification function to graph filter is very trivial. This works also combines the fractional GSSL with a parallel system and weighted pool. But, it is not clear which contribution actually improves the results. Moreover, the intuition of the fractional approach is not clear too, e.g., how the optimization (equation (2)) is derived?, and some explanations are unnatural to demonstrate the methodology, e.g., equation (4). For these reasons, this paper is under the bar of acceptance.
Main concerns:
1. What is the intuition of the optimization of GSSL? How is it obtained? And among all fractional methods (e.g., SL, NL, and PR), which one doe achieve the best performance?
2. The FGS filter in equation (7) is the sum of infinite terms. However, in practical, it is impossible to compute the infinite terms. Does this approximate the sum of finite terms? If does, what is the number of truncation?
3. The authors mention that they establish a theoretical guarantee of the parallel system. But, I could not find any theoretical results. It would be better to include the analysis in the paper.
Minor concerns:
1. In page 3, please edit “forulation” -> “formulation”
2. In equation (8), I think “X+\alpha \tilde{L} X” should change to “\tilde{L} X”


Understanding the loss surface of a neural network is fundamentally important to the understanding of deep learning. This paper presents how piecewise linear activation functions substantially shape the loss surfaces of neural networks. We first prove that the loss surfaces of many neural networks have infinite spurious local minima, which are defined as the local minima with higher empirical risks than the global minima. Our result holds for any neural network with arbitrary depth and arbitrary piecewise linear activation functions (excluding linear functions) under most loss functions in practice with some mild assumptions. This result demonstrates that the networks with piecewise linear activations possess substantial differences to the well-studied linear neural networks. Essentially, the underlying assumptions for the above result are consistent with most practical circumstances where the output layer is narrower than any hidden layer. In addition, the loss surface of a neural network with piecewise linear activations is partitioned into multiple smooth and multilinear open cells by nondifferentiable boundaries. The constructed spurious local minima are exactly connected with each other in one cell by a continuous path, on which the empirical risk is invariant. We further prove that within every cell of a one-hidden-layer network, local minima are equally good, and also, they are all global minima in the cell.

Summary: This paper studies the landscape of deep neural networks with piecewise-linear activation functions. The paper showed that under very mild assumptions, the loss surface admits infinite spurious local minima. Further, it is shown that the loss surface is partitioned into many multilinear cells. If the network is two-layer with two-piece linear activations, it is proved that within each cell every local minimum is global.
Pros:
  --Constructed spurious local minima for piecewise linear activations, for a broader setting than previous papers.
  --The paper is well written, with detailed explanation of proof skeleton.
Cons:
The significance of the results are not clear. Details are given below.
1.	This paper only considers piecewise linear activation, which is a special type of non-linear activation. The major examples are ReLU and leaky ReLU. However, related results for ReLU have been studied for a few previous works mentioned in the last two paragraphs of Sec. 3.2. In particular, Yun et al. 2019b already proves a similar result for 1-hidden-layer neural-net with ReLU activation. I think extending the construction to broader settings (any depth, any piecewise linear and more losses) is mathematically nice, but the motivation of this extension is somewhat unclear to me. One motivation is that this is helpful for the purpose of understanding a “big picture” of the landscape, which I will discuss next.
2.	The second major result is Theorem 2, on the “big picture” with ReLU-like activations. However, Theorem 2 is somewhat trivial to prove, and the link to Theorem 1 is rather weak.
   (a) The main message of Thm 2 is the partition of the surface into multiple pieces, and each piece has good property. This partition is somewhat straightforward, and has been studied before, in, e.g., [R1].
   For a global “big picture”, partitioning itself is not very interesting. Theorem 2 mainly describes the property of each region separately for 2-layer network, which is weaker than [R1].
   (b) Theorem 2 seems easy to prove. The 1st, 3rd and 4th property are all straightforward. The 2nd property “local analogous convexity” was given a 2-page proof in the paper. However, I don’t understand why not use the following simple argument: for each region, the network behaves like a deep linear network, thus directly applying existing result shall imply “every local minimum in the region is the global minimum of the region”, right? If not, what is the difficulty?
   (c) The 3rd property says “some local minima are concentrated as a valley in some cell”. What are the formal definitions of “concentrated” and “valley” in this sentence?
   (d) The link to Theorem 1 is weak: the link is the 3rd property of Theorem 2 that “some local minima are in a valley”. It is just about some special local minima and weakly related to the other properties on the “global view”. In addition, the fact that “some of them are in a valley” may be due to the very special construction, thus it is not surprising and does not reveal anything interesting about the “big picture”.
3.	Other issues:
a) While ReLU-type activations are popular, there are still commonly-used activation functions are not piece-wise linear, e.g., tanh, swish. It is not proper to claim that "this paper presents how nonlinearities in activations substantially shape the loss surface" and "almost every practical neural network ....". I suggest replacing "nonlinearity" with "piecewise linearity" in both the title and the abstract, and modifying the over-statements.
   b) In Property 1 of Theorem 2, “smooth and multilinear partition” might be a bit misleading. The loss surface should be fractional in general, where multilinear cells are separated by non-smooth boundaries. “Smooth partition” seems to imply that the boundaries are smooth or the partition method is smooth in some sense.
   c) The name “analogous convexity” is not appropriate. Analogous convexity is not formally defined in the paper. According to Sec. 4.3 third paragraph, “the property of analogous convexity that the local minima wherein are equally good”. It seems that “analogous convexity” is just “all local minima are good”, which is very different from convexity. It is a weaker property than quasi-convexity, star-convexity, etc, and thus it is better not to call it “analogous convexity”.
    d) Property 3 of Theorem 2 is very far from “mode connectivity”. The proof of Property 3 relies on a special construction of Theorem 1, and the latter is for two arbitrary global minima.
[R1] Soudry and Hoffer. "Exponentially vanishing sub-optimal local minima in multilayer neural networks." arXiv preprint arXiv:1702.05777 (2017).
Conclusion:  I think this paper is studying an important and interesting question, and the efforts of constructing local minima and understanding big picture are both interesting to me. However, I’m afraid the current form of the paper does not meet the standard of the conference. That being said, it would be a nice paper if the big picture can be explored deeper, and the link to the spurious local minima can be built stronger.


Sequential Neural Processes (SNP) is a new class of models that temporally extends Neural Processes (NP) and can meta-learn a sequence of stochastic processes. This learned function however under-fits the provided contexts as is also the case in NP. Applying attention to the contexts resolves this in NP but simply extending this to SNP and applying attention on a buffer of context history is sub-optimal as our findings show. In this paper, we propose Attentive Sequential Neural Processes (ASNP) which resolves the under-fitting in SNP by introducing a novel imaginary context, modeled as a latent variable, over which the attention can then be applied. We evaluate our model on 1D Gaussian Processes regression and 2D moving MNIST/CelebA regression. We apply ASNP to implement Attentive Temporal-GQN and further evaluate on the moving CelebA task.

This paper combines ideas from attentive and sequential neural processes to incorporate an attention mechanism to the existing sequential neural process, which results in an attentive sequential neural processes framework.
While the idea is somewhat interesting, I think this paper is technically vague and not well-motivated, which makes it hard for me to feel convinced that the problem exists and is non-trivial, and that the proposed solution is significant. Let me elaborate on my thoughts below:
First, the authors stated that SNP is subject to the underfitting problem that plagues NP but it is not clear to me why, in the temporal context of SNP, do we need to focus our attention on past contexts, which are no longer relevant. Could the authors please motivate this with a concrete application scenario? Without a concrete scenario, I do not feel very convinced that the problem exists.
Second, the argument that augmenting SNP with an attention mechanism is not trivial is somewhat contrived. In particular, the reason for this non-triviality is that (in the authors' own words) SNP assumes that it cannot store the past context as is -- so what if we simply store the past context & condition the representation on the entire history of past context instead?
Apparently, this can come across trivially by replacing C_t with both C_<t and C_t in Eq. (2). This is in fact very similar to what the authors did in Eq. (4) which summarizes the generative process of ASNP -- the only difference is the generation of imaginary contexts, whose necessity is again questionable, as I elaborate next.
Third, the motivation for imaginary context is pulled from a very distant literature on how a human brain memorizes past experiences in a lossy memory consolidation, which only retains the most important sketches. In the context of ASNP, it is not, however, clear to me why this mechanism is necessary given that entire lossless memory can be stored except that without a lot of contexts, there is not a need for an attention component (as implied in first paragraph of Section 3) which is a contrived motivation.
Fourth, the technical exposition of this paper is too vague. Given that the key contribution here is about an attention component, the background review on ANP is surprisingly informal with no technical detail at all. For the other parts, the technical part is also mostly abstracted away -- what is presented is therefore not that much different from a typical generative model with latent variables, which makes it unclear whether there is a technical challenge here.
In fact, from what I see, going from Eq. (2) to Eq. (4) is not much of a conceptual challenge and the execution of Eq. (4) (particularly the attention component described in Section 3.2) seems like a bunch of arbitrary engineering ideas which were put together to substantiate Eq. (4).
Is there a technical challenge in the entire pipeline that should have been highlighted?
For the experiment, could the author compare the performance between ASNP and ASNP without the imaginery component (but with the attention mechanism)? It would be a good experiment to see if the imaginery component is necessary.
To summarize, I believe the paper in its current state is not well-motivated and appears very incremental given the prior works of SNP and ANP. Even its imaginery component, which is the key contribution here,  is, if I understand Eq. (3) correctly, not much different from context sampling of a NP.


A deep neural network (DNN) with piecewise linear activations can partition the input space into numerous small linear regions, where different linear functions are fitted. It is believed that the number of these regions represents the expressivity of the DNN. This paper provides a novel and meticulous perspective to look into DNNs: Instead of just counting the number of the linear regions, we study their local properties, such as the inspheres, the directions of the corresponding hyperplanes, the decision boundaries, and the relevance of the surrounding regions. We empirically observed that different optimization techniques lead to completely different linear regions, even though they result in similar classification accuracies. We hope our study can inspire the design of novel optimization techniques, and help discover and analyze the behaviors of DNNs.

First, I believe that the acknowledgements in the manuscript give identifying information which could stand in conflict with a double blind review process. I’ll leave it to the area chairs/program chairs to make a decision on this. The following review will be contingent on the fact that the authors did not break the submission rules.
This paper aims to give new insights into deep neural networks by presenting a number of approaches to analyse the linear regions in such networks. The authors define a linear region around a point x* as the intersection between a number of half spaces that are defined through linear approximations of a DNN (tangents) around that point x*. The authors show that points within these regions can be found using convex optimization with a number of linear constraints that are equal or less than the number of nodes in a DNN. In experiments with a fully connected network the authors analyse different properties of these linear regions: (1) How big is the biggest sphere that we can fit in a linear region? (2) How much do the hyperplanes that define a region correlate with each other? (3) How reliably does a linear region represent a single class? And (4), How does a linear region interact with neighbouring regions? In their presentation, the authors focus on comparing these properties between models that were either trained without regularisation, with batch normalisation, or with dropout and with different learning rates. This allows them to draw insightful conclusions about the difference between linear regions in these models. The authors hope that their work will enable new ways of analysing DDNs that will inspire new architectures and optimization techniques.
I vote to accept this paper. The authors present a large array of methods to analyse the linear regions of DNNs. Their insights into the differences of BN and Dropout are useful (figure 1 & 2) and sensible (figure 3). The implications of linear regions on adversarial robustness can have an impact in the future. Because the paper relies on geometrical reasoning, I wished there would be more visualisations that guide the reader.
Here are a number of comments and questions that I have on the manuscript:
- Figure 1 Top: What do the different colour represent in the linear regions plot?
- Section 2.1: maybe add a toy graph that visualises the depth-wise ‘exclusion’ process of feasible “neighbours” of x*?
- Eq. (2) & (3): Explain where these equations come from.
- Sec 3.2, first sentence. The authors claim that inspheres of linear regions are highly relate to the expressivity of DNN. Can they elaborate on that claim? Is this claim a result of their experiments?
- What is the relationship between the number of constraints in eq. (5) and the radius of an insphere? Does the insphere size decrease with more constraints? What implications would that have on deeper networks than the one that was presented?
- Why should distortion be a good measure of the size of a linear region?
- The authors claim that it is expensive to run their approach, and that they will aim to improve speed in the future. Can the authors give a more concrete example of runtimes in their current approach?


ABSTRACT Semi-supervised learning, i.e. jointly learning from labeled an unlabeled samples, is an active research topic due to its key role on relaxing human supervision. In the context of image classification, recent advances to learn from unlabeled samples are mainly focused on consistency regularization methods that encourage invariant predictions for different perturbations of unlabeled samples. We, conversely, propose to learn from unlabeled data by generating soft pseudo-labels using the network predictions. We show that a naive pseudo-labeling overfits to incorrect pseudo-labels due to the so-called confirmation bias and demonstrate that mixup augmentation and setting a minimum number of labeled samples per mini-batch are effective regularization techniques for reducing it. The proposed approach achieves state-of-the-art results in CIFAR-10/100, SVHN, and Mini-ImageNet despite being much simpler than other methods. These results demonstrate that pseudo-labeling alone can outperform consistency regularization methods, while the opposite was supposed in previous work. Code will be made available.

Summary: This paper focuses on the semi-supervised learning problem, and proposes a way to improve previous pseudo-labeling methods.  In pseudo-labeling, there is an issue called confirmation bias, which accumulates the early errors of wrong pseudo labels.  By adding some simple tricks such as adding mixup augmentation and setting a minimum number of labeled samples per mini-batch, the confirmation bias is shown to be reduced, leading to an improvement in accuracy.  Experiments demonstrate that the additional tricks are meaningful and makes pseudo-labeling better than many baseline methods for semi-superivsed learning, including state-of-the-art consistency regularization methods.
Pros: This is an interesting paper with a clear motivation, which is to fix the so-called confirmation bias that appears in pseudo-labeling methods for semi-supervised learning.  Although the tricks introduced in the paper (mixup and changing the mini-batch selection rules) themselves are not novel, they make the proposed method simple.  It is also shown to be meaningful in reducing the confirmation bias in Table 1 and Figure 2, achieving the original goal of the paper.
Cons: The weakness of the paper is that the intuition or the motivation behind the design of the proposed method is not so clear.  Using mixup is justified by the reason that mixup gives better confidence calibration.  This is important for pseudo-labeling methods, because soft-label output predictions are used as pseudo labels.  On the other hand, however, it was not so obvious why a minimum number of labeled samples per mini-batch was considered.  Can we consider further extensions such as minimum number of labeled samples per mini-batch & per class?  (Perhaps the discussions about mixup and soft labels in the last paragraph of Section 3 should be more emphasized, for example in the last paragraph of the Introduction section.)
Related to the weakness above, it is hard to see how far the regularization effects of adding mixup and mini-batch sampling rules are contributing to add synergy to the pseudo-labeling methods.  This is partially answered with Figure 2, but it would make this easier to see if the experiments included stronger baselines, e.g., by adding the same regularization tricks to consistency regularization methods, perhaps in Table 3.
Finally, since future work on pseudo labels will follow this paper’s setup, hyperparameters such as lamba_A, lambda_H, and alpha should be chosen carefully instead of fixing them.
Other minor comments (that did not impact the score):
- In reference section, "Z. MaXiaoyu Tao" seems to combine two authors.
- Table 3 never appears in the text.  In Section 4.4, "The table" in the second sentence can be changed to "Table 3".
- "architecture plays and important role" --> "architecture plays an important role"
- In Table 2, "+" signs make it look like an equation.  I suggest using commas instead.
-  "ResNet arquitectures" --> "ResNet architectures"
~~~~~
Thank you for the response and for the additional discussions that were included in the updated paper.


The neural architecture search (NAS) algorithm with reinforcement learning can be a powerful and novel framework for the automatic discovering process of neural architectures. However, its application is restricted by noncontinuous and highdimensional search spaces, which result in difficulty in optimization. To resolve these problems, we proposed NAS in embedding space (NASES), which is a novel framework. Unlike other NAS with reinforcement learning approaches that search over a discrete and high-dimensional architecture space, this approach enables reinforcement learning to search in an embedding space by using architecture encoders and decoders. The current experiment demonstrated that the performance of the final architecture network using the NASES procedure is comparable with that of other popular NAS approaches for the image classification task on CIFAR-10. The beneficial-performance and effectiveness of NASES was impressive even when only the architecture-embedding searching and pre-training controller were applied without other NAS tricks such as parameter sharing. Specifically, considerable reduction in searches was achieved by reducing the average number of searching to <100 architectures to achieve a final architecture for the NASES procedure.

The paper proposes an interesting idea to perform Neural Architecture Search: first, an auto-encoder is pre-trained to encode/decode an neural architecture to/from a continuous low-dimensional embedding space; then the decoder is fixed but the encoder is copied as an agent controller for reinforcement learning. The controller is optimized by taking actions in the embedding space. The reward is also different from previous works which usually only considered validation accuracy but this work also considers the generalization gap.
The idea is interesting, but there are some problems on both the method's and the experimental sides:
1. NAO [1] also embeds neural architectures to a continuous space. Different from NAO which applies gradient descent in the embedded space, this paper uses RL. I double that RL can work better than gradient descent in a continuous space. The paper should compare with NAO. Ideally, this paper might work better than NAO if the accuracy predictor in the NAO is not accurate, while this paper uses real accuracy as a reward for search. However, this is not soundly compared.
2. It is unreasonable to discretize continuous actions to a Bernoulli distribution. Many RL methods, such as DDPG, can handle continuous actions;
3. The paper uses Eq. 1 as a reward. It's interesting, but it's unclear why the generalization error is needed. Ablation study is required.
4. As the community makes more progresses in AutoML, a better and better (smaller and smaller) search space is used. It doesn't make much sense to compare the search time under different search spaces. Comparison under the same setting (e.g. NASBench-101) is required.
Minors:
1. missing a number in "and T0 = epochs"
2. missing "x" in "32 32 RGB in 100 classes", and "100" should be "10"
[1] Luo, Renqian, Fei Tian, Tao Qin, Enhong Chen, and Tie-Yan Liu. "Neural architecture optimization." In Advances in neural information processing systems, pp. 7816-7827. 2018.


As a concise and classic framework for object detection and instance segmentation, Mask R-CNN achieves promising performance in both two tasks. However, considering stronger feature representation for Mask R-CNN fashion framework, there is room for improvement from two aspects. On the one hand, performing multi-task prediction needs more credible feature extraction and multi-scale features integration to handle objects with varied scales. In this paper, we address this problem by using a novel neck module called SA-FPN (Scale Aware Feature Pyramid Networks). With the enhanced feature representations, our model can accurately detect and segment the objects of multiple scales. On the other hand, in Mask R-CNN framework, isolation between parallel detection branch and instance segmentation branch exists, causing the gap between training and testing processes. To narrow this gap, we propose a unified head module named EJ-Head (Effective Joint Head) to combine two branches into one head, not only realizing the interaction between two tasks, but also enhancing the effectiveness of multitask learning. Comprehensive experiments show that our proposed methods bring noticeable gains for object detection and instance segmentation. In particular, our model outperforms the original Mask R-CNN by 1˜2 percent AP in both object detection and instance segmentation task on MS-COCO benchmark.

The paper proposes several modifications to the Mask R-CNN model:
1. a variation of Feature Pyramid Networks (FPN): SA-FPN which merges features top-down and bottom-up.
2. the "effective joint head" (EJ-Head) which consists of
    a) moving the segmentation head behind the detection one,
    b) Doubling resolution of RoI crops, calling this "enriched feature".
    c) adding Boundary Refinement (really just a residual block).
Experiments show slightly improved scores on MS-COCO
I vote to reject this paper.
First of all, it is badly written. Not only are grandiose formulations of "smart framework", "innovative way", "we slickly mix" not professional, but the whole write-up of the method is very confusing. After reading it multiple times, I am still not 100% sure I fully understood the SA-FPN.
Second, it is not clear to me that SA-FPN is really novel, the original FPN paper already compared top-down and bottom-up performances (Tables 1-3).
Then, two modifications which simply increase capacity and could explain all improved scores are not ablated: increased resolution of RoI crops, and added "boundary refinement", which is really just a residual block.
Furthermore, moving the segmentation *after* the bounding-box prediction is *not* joint prediction. If predicting the box is called p(b) and predicting the mask is called p(m), the Original Mask R-CNN does p(b|features)p(m|features), the proposed model in this paper does p(b|features)p(m|b,features), and actual joint prediction would be p(b,m|features).
Finally, I think the paper is much better suited for a conference like ICCV/ECCV or CVPR and will get better reviewers than me there.


Network quantization is a model compression and acceleration technique that has become essential to neural network deployment. Most quantization methods perform fine-tuning on a pretrained network, but this sometimes results in a large loss in accuracy compared to the original network. We introduce a new technique to train quantization-friendly networks, which can be directly converted to an accurate quantized network without the need for additional fine-tuning. Our technique allows quantizing the weights and activations of all network layers down to 4 bits, achieving high efficiency and facilitating deployment in practical settings. Compared to other fully quantized networks operating at 4 bits, we show substantial improvements in accuracy, for example 66.68% top-1 accuracy on ImageNet using ResNet-18, compared to the previous state-of-the-art accuracy of 61.52% Louizos et al. (2019) and a full precision reference accuracy of 69.76%. We performed a thorough set of experiments to test the efficacy of our method and also conducted ablation studies on different aspects of the method and techniques to improve training stability and accuracy. Our codebase and trained models are available on GitHub.

This work introduces GQ-Net, a novel technique that trains quantization friendly networks that facilitate for 4 bit weights and activations. This is achieved by introducing a loss function that consists of a linear combination of two components: one that aims to minimize the error of the network on the training labels of the dataset and one that aims to minimize the discrepancy of the model output with respect to the output of the model when the weights and activations are quantized. The authors argue that this has the effect of “guiding” the optimization procedure in finding networks that can be quantized without loss of performance. For the discrepancy metric the authors use the KL divergence from the predictive distribution of the floating point model to the one of the quantized model. The authors then propose several extra techniques that boost the performance of their method: 1. scheduling the weighting coefficients of the two loss terms (something which reminisces iterative pruning methods), 2. stopping the gradient of the floating point model w.r.t. the second loss term, 3. learning the parameters of the uniform quantizer, 4. alternating optimization between the weights and the parameters of the quantizers and 5. using separate batch normalization statistics for the floating point and quantized models. The authors then evaluate their method on Imagenet classification using ResNet-18 and Mobilenet v1 / v2, while also performing an ablation study about the extra tricks that they propose.
This work is well written and in general conveys the main idea in an effective manner. Quantization friendly neural networks in an important subject in order to make deep learning tractable for real world applications. The idea seems on a high level to be interesting and simple; train floating point models that can fit the data well while also encouraging them to be robust to quantization by enforcing the predictive distributions of the fixed and floating point models to be similar in the KL-divergence sense. Nevertheless, I do have some comments that would hopefully help in improving this work:
- It does seem that GQ-Nets need extra tricks in order to perform well, and those tricks come with their own set of hyperparameters that need to be tuned. For example, at section 4.3 you mention that the top-1 accuracy of vanilla GQ-Nets is 60.95, which is lower than the RelaxedQuant baseline (that has 61.52). This raises the question whether the boost in performance is due to the several additional steps employed (which in general can be applied to other quantization techniques as well), and not due to the main idea itself.
- Do you employ the straight-through estimator (STE) for the weights in the L_q objective? In the second paragraph of the second page you argue that due to the biased gradients of STE the performance is in general reduced, so I was wondering whether STE posed an issue there or whether you used an alternative estimator.
- How is batch normalization handled? Do you absorb the scale and shifts in the weights / biases before you perform quantization or do you quantize the weights and then apply the BN scale and shift in full precision?
- How do you ensure and ub > lb when you learn the quantizer? In general learning the quantizer can be also done with alternative techniques (e.g. simply learning the scale and offset) so I was wondering whether you noticed benefits from using the ub, lb parametrization compared to others.
- Do you show the pre-quantization distributions at Figure 2b? In the caption you mention quantized but the resolution seems to be higher than the 16 values you should get with 4 bits. Furthermore, it should be noted that the discrepancy in BN in quantized models was, as far as I am aware, firstly noticed at [1] (and subsequently at RelaxedQuant) and both of these methods simply re-estimated the moving averages during the inference time.
Overall, I am on the fence about this work and tend to reject. Having said that, I am of course willing to revise my score after the discussions with the authors / other reviewers.
[1] Probabilistic Binary Neural Networks, Jorn W.T. Peters, Max Welling


Graph Convolutional Networks (GCNs) are powerful models for learning representations of attributed graphs. To scale GCNs to large graphs, state-of-the-art methods use various layer sampling techniques to alleviate the "neighbor explosion" problem during minibatch training. We propose GraphSAINT, a graph sampling based inductive learning method that improves training efficiency and accuracy in a fundamentally different way. By changing perspective, GraphSAINT constructs minibatches by sampling the training graph, rather than the nodes or edges across GCN layers. Each iteration, a complete GCN is built from the properly sampled subgraph. Thus, we ensure fixed number of well-connected nodes in all layers. We further propose normalization technique to eliminate bias, and sampling algorithms for variance reduction. Importantly, we can decouple the sampling from the forward and backward propagation, and extend GraphSAINT with many architecture variants (e.g., graph attention, jumping connection). GraphSAINT demonstrates superior performance in both accuracy and training time on five large graphs, and achieves new state-of-the-art F1 scores for PPI (0.995) and Reddit (0.970).

This paper proposed a new sampling method to train GCN in the mini-batch manner. In particular, unlike existing methods which samples the mini-batch in the node-wise way, GraphSAINT proposed to sample a mini-batch in the graph-wise way. As a result, GraphSAINT uses the same graph across different GCN layers, while most existing methods use different graphs across different GCN layers.  In addition, the authors show that this sampling method is unbiased. Extensive experimental results have shown improvement over existing methods. Overall, this idea is interesting and well presented.
Pros:
1. A new sampling method for the stochastic training of GCN. Have good performance.
2. Extensive experiments to verify the performance of the proposed method.
3. The theoretical analysis looks sound.
Cons:
1. GraphSAGE and FastGCN use different graphs across different GCN layers, while ClusterGCN and GraphSAINT use the same graph across different GCN layers. To make a fair comparison, it is necessary to have the same batch size for different methods. How do you deal with this issue in your experiment?
2. For ClusterGCN, the clustering procedure is done before the training. So, it needs much less computational overhead for sampling in the training course. However, GraphSAINT needs to do the heavy sampling online. Thus, it may consume more time than ClusterGCN for large graphs. It's better to show the running time of these two methods.


Adversarial input poses a critical problem to deep neural networks (DNN). This problem is more severe in the "black box" setting where an adversary only needs to repeatedly query a DNN to estimate the gradients required to create adversarial examples. Current defense techniques against attacks in this setting are not effective. Thus, in this paper, we present a novel defense technique based on randomization applied to a DNN's output layer. While effective as a defense technique, this approach introduces a trade off between accuracy and robustness. We show that for certain types of randomization, we can bound the probability of introducing errors by carefully setting distributional parameters. For the particular case of finite difference black box attacks, we quantify the error introduced by the defense in the finite difference estimate of the gradient. Lastly, we show empirically that the defense can thwart three adaptive black box adversarial attack algorithms.

This paper proposes applying randomization to the output layer of a DNN to defend against query-based attacks based on finite difference estimates. Then some theoretical analysis is provided, showing that with perturbation of a suitable scale, the randomization layer will not affect the accuracy of the model, while causing a large estimation error of finite difference methods that prevents finite-difference based attacks. Empirical results verify that the proposed defense is still effective against adaptive attacks where the randomness is averaged.
Pros:
The proposed method is simple, straightforward, yet novel. Its working mechanism is easy to understand and analyze, so it should be useful against finite-difference based attacks.
Limitation:
The proposed method is not useful to defense against white-box attacks and transfer-based attacks, since basically it does not change the predictive model. Some other randomization methods like [26], by contrast, change the predictive model, hence they may be useful against white-box attacks and transfer-based attacks.
Questions and suggestions:
This part is my main concern.
It seems that the experimental results are very good. For example, in Figure 3a, the defense is effective even if \sigma^2<1e-6. However, by the analysis in Section 4.2 (the formula below Line 3, Page 6), when \sigma^2=1e-6, |E[g_i-\gamma_i]| should be rather small, hence it should not block finite-difference based attacks. I think more explanation is needed for the good performance in the experiments.
Finite differences are extremely sensitive to small random perturbation of the function value when the spacing (step size) h is small. For example, g_i=\frac{L(f(x+he_i))-L(f(x-he_i))}{2h}, when h is very small, f(x+he_i) and f(x-he_i) is very close, hence adding perturbation to them will change g_i a lot. To present stronger adaptive attacks to output randomization, my suggestion is that a larger h can be adopted. It will be better if the results are investigated against attacks with different values of h.
A mistake:
In Section 4.1 on Page 4: "we can express the probability that x is misclassified in the vector d(p) as: \sum_{i=2}^C P(d(p_i)>d(p_m))". I think this is wrong, since P(A or B happens)=P(A happens)+P(B happens) only when A and B are mutually exclusive. However, "d(p_i)>d(p_m)" and "d(p_j)>d(p_m)" are not mutually exclusive. Hence, the probability that x is misclassified in the vector should be less than or equal to that sum of probabilities.
By the way, the writing in Section 4.1 is not clear:
- The overall misclassification probability is presented first, but after that K only represents the misclassification probability into a specific class. The connection between them is unclear.
- At the beginning of Section 4.1, the distribution of \epsilon is \epsilon\sim\mathcal{N}(\mu,\sigma^2\cdot\mathbf{I}_C): a unique \sigma is used. But after that, the variance of \epsilon_i becomes \sigma_i^2 instead of \sigma^2.
- In the second to the last line on Page 4, "level of noise (\sigma^2) can be set for each class separately", but the authors did not explain how to set them, and in the experiments \sigma^2 is set as the same scalar.
- In Figure 1b, the line style of "K=5.0e-3" and "K=1.0e-1" in the legend is very similar. The line style of "K=2.0e-01" in the legend is not clear: I do not know whether it refers to "-.-.-." or "------".
Typos:
Section 6, Page 8: "unintentionlly" => "unintentionally"
Some missing spaces after punctuation:
- Section 4.2, Page 5, "... gradient estimate.When the ..." => should add a space before "When"
- Section 5, Page 7, "In addition,input randomization ..." => should add a space before "input"


It has recently been shown that ReLU networks produce arbitrarily over-confident predictions far away from the training data. Thus, ReLU networks do not know when they don't know. However, this is a highly important property in safety critical applications. In the context of out-of-distribution detection (OOD) there have been a number of proposals to mitigate this problem but none of them are able to make any mathematical guarantees. In this paper we propose a new approach to OOD which overcomes both problems. Our approach can be used with ReLU networks and provides provably low confidence predictions far away from the training data as well as the first certificates for low confidence predictions in a neighborhood of an out-distribution point. In the experiments we show that stateof-the-art methods fail in this worst-case setting whereas our model can guarantee its performance while retaining state-of-the-art OOD performance.

Summary: This paper provides a method to train neural networks with guarantees on outputting probabilities/scores close to uniform on inputs that are out of domain.
Precisely, on some point x_0, we can obtain the largest radius such that on inputs in the ball around x_0 the classifier outputs some predetermined  maximum score.
The paper combines a simple generative model (mixture of Gaussian) for modeling in-distribution vs. out of distribution. The simplicity allows to obtain guarantees on the probability of an input being considered out of class. The model output p(y | x) critically uses the probability of being out of distribution. Hence guarantees on being detected as out of distribution translate to guarantees on p(y|x) being close to uniform.
Decision: I vote for accepting the paper. The paper has some clear strengths. However, I have some concerns regarding experiments and comparison to previous work. It would be great if the authors could clarify.
Strengths: The paper’s methodolgy is clearly written.  The modeling is clear and sound. Overall, this idea is a promising approach to obtain networks that are provably under-confident far from training examples. The training cost for this approach is comparable to standard training, and the approach seems scalable and broadly applicable in general.
The experimental evaluation is also clearly described. Worst-case evaluation of OOD (out of domain) performance seems novel and the gains not this objective using the proposed approach of this paper are interesting and promising.
Concerns: While the paper explains the proposed method well, the description of previous work and relation to previous work is inadequate. After spending some hours reading the cited paper, I am still confused about what’s the novelty of this work at a high level.
This work uses p(x|i) and p(x|o) in the computation of p(y|x) during inference, where i and o are in distribution and out of distribution respectively. This is crucial to obtain guarantees one performance. However, how does this compare to other previous work that also uses some kind of generative modeling to model in/out distribution? A bunch of papers are cited in the introduction as doing this, but the relationship to the proposed work is unclear.
— I have a major experimental concern: When comparing against ACET, the baseline of performing adversarial style training on random noise inputs seems more appropriate since it’s closer to the evaluation metric (which picks random noise as out of domain and not 80M Tiny Images). What does the performance of this ACET baseline look like?
— During evaluation, how do you ensure that the radius is not too large such that it has images that the model should actually be confident on (close to in distribution samples). A flip-side is how sensitive the performance is to the score chosen for choosing the radius for computing the valid set of images for the adversary? It’s possible that the 11%  threshold is too high? What’s the minimum confidence of CCU on the test images?
Minor comments on writing:
—K_i, K_0 are not defined when writing the expression for GMM


Generative Adversarial Networks (GANs) is a powerful family of models that learn an underlying distribution to generate synthetic data. Many existing studies of GANs focus on improving the realness of the generated image data for visual applications, and few of them concern about improving the quality of the generated data for training other classifiers-a task known as the model compatibility problem. Literature also show that some GANs often prefer generating 'easier' synthetic data that are far from the boundaries of the classifiers, and refrain from generating near-boundary data, which are known to play an important roles in training the classifiers. To improve GAN in terms of model compatibility, we propose Boundary-Calibration GANs (BCGANs), which leverage the boundary information from a set of pre-trained classifiers using the original data. In particular, we introduce an auxiliary Boundary-Calibration loss (BC-loss) into the generator of GAN to match the statistics between the posterior distributions of original data and generated data with respect to the boundaries of the pre-trained classifiers. The BC-loss is provably unbiased and can be easily coupled with different GAN variants to improve their model compatibility. Experimental results demonstrate that BCGANs not only generate realistic images like original GANs but also achieves superior model compatibility than the original GANs.

In this work authors consider a problem of 'model compatibility' of GANs, i.e. usefullness of the generated samples for classification tasks. Proposed 'Boundary Calibration' GAN attempts to tackle this issue by adding non-adversarial terms to discriminator, obtained as outputs of the classifiers trained on the original data. For evaluation, it is proposed to compare accuracies obtained by classifiers trained on generated and on real data (termed 'relative acurracy'). Experiments show that the proposed methods improve such scores.
Pros:
- considered problem seems to be important GAN application which has not yet received too much attention.
- proposed method seems to improve the accuracy of classifiers trained on generated data.
Cons:
- the paper is poorly written, has multiple typos and often it is unclear what authors mean.
- the proposed evaluation does not exactly measure the potential improvements from training classifiers with generated data. It would make sense to provide information if generated data can improve classifier's scores, if added to the training data (or small part of it).
- it is unclear whether or not the proposed technique does not affect the sample quality, as no quantitative metrics are provided.
Details:
1. Quantitative scores for quality of generated samples are not provided. Authors instead provide few samples and state that it is difficult to detect the difference. Although sample quality is not the main task here, it certainly is important - otherwise we could train generators solely against the classifier 'boundary calibration' loss terms - this, however would likely lead to adversarial examples. Combining two losses often leads to trade-offs, hence showing that we can improve 'model compatibility' without the loss of sample quality is actually crucial. Metrics such as Inception score [1], FID [2] or KID [3] are desired, especially given relatively poor quality of cifar and mnist samples from all models.
2.The main metric used is the ratio between accurracies of classifiers (trained on real and generated data). It is hence difficult to tell if the classifiers that were used were trained reasonably and achieved reasonable scores.
3. In the abstract, authors claim that 'GANs often prefer generating easier synthetic data that are far from boundaries of the classifiers'. Although for some GAN settings the generators might be biased to do so, in general this claim is unfounded, as GANs optimize divergences that are agnostic to classifier boundaries.
4. It is unclear what kind of classifier-output is used as an input to MMD. Are these continuous logits, discrete class numbers, or one-hot-encoded class identities?
5. Authors use WGAN and MMDGAN with gradient penalty. It is unclear how gradient penalty is applied to MMDGAN as what should be penalized is the witness function, which is different than in WGAN-GP [4], see e.g. [3].
6. It is unclear how embeddings of class information are concatenated to discriminator inputs (p.5).
7. It is unclear to what extent feature selection is deterministic. Authors argue in Section 5.4 that the intersection of top-k features selected from two models should be large. It would be good to provide the same statistics for features selected twice on the same sample.
Overall, the paper currently does not match the quality requirements of ICLR, however it has potential for improvement if the mentioned issues are addressed.
Typos/unclear expressions:
[p1] 'may not willing' >> 'may not be willing'
[p1]'with the property similar to the original data is demanding'  >> properties, demanded/in demand
[p2] 'Although GANs are versatile as aforementioned' - strange wording
[p2] 'The pioneered work' >> 'pioneering work'
[p2] 'information of models' >> 'information from the models'
[p2] effects >> affects
[p3] related works >> related work
[p3] distribution of label >> distribution of labels
[p3] 'generated dataset adopt ' >> 'generated dataset will adopt '
[p3] 'To known about the boundary' >> ' To know the boundary'/'To include the information about the boundary'
[p3] 'the a distance'
[p3] 'the problem to distinguish whether two sets of samples'
[p4] 'If they are close the sets might be sampled from the same distribution' (?)
[p4] tries to minimized the MMD
[p4] would not leads to
[p6, Table 2 caption] number of estimator used
[p6] at Appendix
[p6] depresses the model compatibility (?)
[p6] can providing
[p7] to known how
[p8] our work open >> our work will open
------------------------------------- Revision -------------------------------------
Although some issue seem to have been clarified, two of my main concerns, i.e. the proposed evaluation and the text quality, have not been resolved. Additionally, as pointed out by reviewer #4, the results seem somewhat incremental. For these reasons, I decided to keep the rating unchanged.


We introduce a novel framework for generative models based on Restricted Kernel Machines (RKMs) with multi-view generation and uncorrelated feature learning capabilities, called Gen-RKM. To incorporate multi-view generation, this mechanism uses a shared representation of data from various views. The mechanism is flexible to incorporate both kernel-based, (deep) neural network and Convolutional based models within the same setting. To update the parameters of the network, we propose a novel training procedure which jointly learns the features and shared subspace representation. The latent variables are given by the eigendecomposition of the kernel matrix, where the mutual orthogonality of eigenvectors represents uncorrelated features. Experiments demonstrate the potential of the framework through qualitative and quantitative evaluation of generated samples on various standard datasets.

This paper presents a model and training framework for generating samples based on restricted kernel machines. It is extended to multi-view generation and uncorrelated feature representation learning.
- The paper is well-written and well-organized. Notations and claims are clear.
- The idea of a multi-view generation model based on restricted kernel machines is interesting. However, the paper seems to be limited to model definition and algorithm overview without a performance evaluating analysis.
- The experimental evaluations are not satisfactory. Although it is claimed in the paper that the model is able to generate high quality images, it is very hard to be confirmed with these experiments. There is no concrete attempt at comparing the performance of the model to the other used methodologies. Generating high quality images with multiple views is an interesting problem, and there are good works in the field addressing the issues. To name a few:
Zhu, Z., Luo, P., Wang, X., Tang, X.: Multi-view perceptron: a deep model for learning face identity and view representations. In: Advances in Neural Information Processing Systems (NIPS). pp. 217–225, (2014)
Kan, M., Shan, S., Chen, X.: Multi-view deep network for cross-view classification. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). pp. 4847–4855, (2016)
Yin, X., Liu, X.: Multi-task convolutional neural network for pose-invariant face recognition. IEEE Transactions on Image Processing (2017)
Yim, J., Jung, H., Yoo, B., Choi, C., Park, D., Kim, J.: Rotating your face using multitask deep neural network. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). pp. 676–684, (2015)
Yu Tian, Xi Peng , Long Zhao, Shaoting Zhang , ,Dimitris N. Metaxas , CR-GAN: Learning Complete Representations for Multi-view Generation, arXiv: 1806.11191, 2018.
There might be differences between these works and the paper, it is common to evaluate the quality of the generation to other models in terms of accuracy or in the classification tasks. Unfortunately, there is no such quantitative analysis in the paper. So the advantages of the proposed model is not very clear since there is not enough quantitative performance analysis. It would be interesting to see complexity analysis to evaluate the computational costs.
Overall, I do not recommend this paper for publication. The experimental results are not satisfactory, and the paper needs improvements in that regard.
** update:
I would like to thank the authors for their comments. However, I still see major issues in the paper unresolved and my review remains the same.


Biological evolution has distilled the experiences of many learners into the general learning algorithms of humans. Our novel meta reinforcement learning algorithm MetaGenRL is inspired by this process. MetaGenRL distills the experiences of many complex agents to meta-learn a low-complexity neural objective function that affects how future individuals will learn. Unlike recent meta-RL algorithms, MetaGenRL can generalize to new environments that are entirely different from those used for meta-training. In some cases, it even outperforms humanengineered RL algorithms. MetaGenRL uses off-policy second-order gradients during meta-training that greatly increase its sample efficiency.

Summary:
This paper presents a novel meta reinforcement learning algorithm capable of meta-generalizing to unseen tasks. They make use of a learned objective function used in combination with DDPG style update. Results are presented on different combinations of meta-training and meta-testing on lunar, half cheetah, and hopper environments with a focus on meta-generalization to vastly different environments.
Motivation:
The work is well motivated and is tackling an important problem. There are a number of design decisions presented and only some are validated experimentally. Given the complexity of many existing meta-rl methods this seems fine but could obviously be improved upon either with more empirical work or with some guiding theory.
Experiments:
Overall the experiments are not convincing to me. Given that this is the majority of your paper is empirically based this is my main criticism. More detailed comments follow.
Figure 2a concerns me that just meta-training on lunar performs worse than ddpg (what your algorithm is based on). This suggests that this added complexity is not aiding in capacity and or hurting training. Can you comment on this? This result also casts doubt onto figure 2b, which, in isolation seems like an extremely promising example of meta-generalization. This makes me fear there is something indirect and not interesting occurring (e.g. the learned loss modifies with the DDPG algorithm which happens to increase noise in generated samples which improve performance only on some environments and hurts in others for example.)
Table 1 should include hand designed algorithms imo. Given how weak EPG is (as you stated for number of frames) and how RL^2 will never generalize across these different tasks it's hard to get a sense of the numbers. Your appendix does include a figure like this which shows ddpg performs quite well. Additionally, I don't understand why meta-training on lunar and transferring to hopper does better than meta-training on hopper (table 1, middle column). Can you comment on this?
While figure 3 is cool, I would appreciate if it put the meta-test performance on the same graph as meta-train performance. From eyeballing the curves it looks like it decreases at 100k iterations then finally increases again at 200k. This is strange. This also seems fraught from an empirical comparison point of view. How do you select when to test these algorithms? Ideally you would have a meta-validation set of tasks then only meta-test on the selected task but I see no mention of this.
Key details such as meta-training are also not discussed in depth nor ablated. From the details and curriculum scheme presented in the appendix this seems like quite a feat. Further study of these factors could be useful.
Hyperparameters of your baseline do not appear to be tuned (taken from appendix) where as for your method has a number of choices. How are you tuning these choices? Once again a meta-validation set would be the principled thing to tune against.
Finally, the experimental setup presented here is quite complicated. There are a ton of factors at play -- exploration, meta-generalization, meta-training, inner-training, instability of ddpg, so on. These all complicate the resulting picture. Having some simplified / more controlled setup to demonstrate these pieces would be greatly appreciated.
Other Suggestions:
Section 3 generalization: I think you mean meta-generalization.
Please include what error bars are for all plots.

Rating:
I am borderline leaning towards reject on this paper. I enjoyed reading this work and found the ideas interesting but the empirical comparisons are confusing and not convincing. I hope the authors continue to work to improve this!


Similar to a convolutional neural network (CNN), the mammalian retina encodes visual information into several dozen nonlinear feature maps, each formed by one ganglion cell type that tiles the visual space in an approximately shift-equivariant manner. Whether such organization into distinct cell types is maintained at the level of cortical image processing is an open question. Predictive models building upon convolutional features have been shown to provide state-of-the-art performance, and have recently been extended to include rotation equivariance in order to account for the orientation selectivity of V1 neurons. However, generally no direct correspondence between CNN feature maps and groups of individual neurons emerges in these models, thus rendering it an open question whether V1 neurons form distinct functional clusters. Here we build upon the rotation-equivariant representation of a CNN-based V1 model and propose a methodology for clustering the representations of neurons in this model to find functional cell types independent of preferred orientations of the neurons. We apply this method to a dataset of 6000 neurons and visualize the preferred stimuli of the resulting clusters. Our results highlight the range of non-linear computations in mouse V1.

In this study, the authors develop a method to cluster cells in primary visual cortex (V1) based on the cells' responses to natural images. The method consists in three steps:
- fit a rotation-equivariant convolutional neural network model to V1 cells (previously described in Ecker et al. 2019)
- align all cells by choosing the rotation for each cell that minimizes overall distance between cells in feature space, so that the clustering is mostly blind to the orientation of the filters.
- cluster the cells using a Gaussian mixture model (GMM).
Although I find this article mostly well-written and the topic important, I cannot recommend acceptance because (1) the study does not make a significant contribution to our understanding of V1, (2) the main innovation in ML presented (alignment method) is quite specific and will thus not likely be of interest for the general audience of ICLR:
(1) An important question in visual neuroscience is whether V1 cells form discrete functional clusters as opposed to a continuum. Another related question is whether these functional clusters correspond to distinct cell types characterized by specific wiring patterns, gene expression and/or morphology.
The analyses performed do not answer any of these two questions:
- the clustering model (GMM model) is not compared statistically to other models that would assume a continuous structure in the data (e.g. cells form a sparse continuous manifold in feature space). Although clusters do appear in the t-SNE visualization, this visualization does not provide statistical evidence that cells indeed form distinct clusters.
- The correspondence of the proposed clusters to cell types with specific wiring patterns, gene expression and/or morphology is not established. To establish this correspondence would require further experiments, as acknowledged by the authors: "To systematically classify the V1 functional cell types, these proposals need to be subsequently examined based on a variety of biological criteria reflecting the different properties of the neurons and the prior knowledge about the experiment".
(2) The alignment method, which consists in rotating the cells in feature space so that orientation is not a factor for subsequent clustering, is quite specific to the problem studied and likely not of interest for the general ICLR audience.
Additional feedback:
- Title: ROTATION-INVARIANT CLUSTERING OF FUNCTIONAL CELL TYPES IN PRIMARY VISUAL CORTEX
=> "functional cell types" is not adequate here, since the article does not establish the existence of functional cell types. Could be replaced with "cell responses".
- Abstract: We apply this method to a dataset of 6000 neurons and provide evidence that discrete functional cell types may exist in V1.
=> this sentence is misleading, since no evidence for functional clusters is provided.
- "Thus, the network has learned an internal representation that allows constructing very similar functions in multiple ways"
=> To avoid the caveat of redundant features, the authors could try to add a dimensionality bottleneck on feature space before readout.
- "Small values of β incur a small cost for poor reconstructions resulting in small optimised values of T and over-smoothed aligned readouts."
=> A simulated annealing procedure (progressive increase of T during learning) could potentially allow the use of larger β values here (i.e. less distortion of the filter).
- The alignment procedure could lead to the emergence of spurious structure in the clustering. It would be important to control for this potential artifact by running the procedure on an unstructured synthetic dataset.
- It is possible that the MEIs within clusters look more similar than they actually are, since the cells are fitted from the same common bank of features. It would be useful but maybe difficult to control for this.
- It would be interesting to test the clustering procedure on a shuffled version of the readout weights (shuffle across features and V1 cells), so as to keep sparsity but not any other structure. Does the t-SNE map look less clustered? Is the GMM fit qualitatively different?
- Fig1(2): add legend/caption. what are the ellipses?


The recent success of neural networks for solving difficult decision tasks has incentivized incorporating smart decision making "at the edge." However, this work has traditionally focused on neural network inference, rather than training, due to memory and compute limitations, especially in emerging non-volatile memory systems, where writes are energetically costly and reduce lifespan. Yet, the ability to train at the edge is becoming increasingly important as it enables real-time adaptability to device drift and environmental variation, user customization, and federated learning across devices. In this work, we address two key challenges for training on edge devices with non-volatile memory: low write density and low auxiliary memory. We present a low-rank training scheme that addresses these challenges while maintaining computational efficiency. We then demonstrate the technique on a representative convolutional neural network across several adaptation problems, where it out-performs standard SGD both in accuracy and in number of weight writes.

While inference on edge devices is a popular and well-studied problem in recent days, training on these devices comes with many challenges. This paper proposes a low-rank training schema that helps mitigate some of the critical challenges that occur during training models on NVM memory-based edge devices. Additionally, two techniques, namely streaming batch norm and gradient max norm, are proposed to help training in an online setting. The proposed method is mainly based on approximating the Kronecker sum and is largely inspired by (Benzing et al, ICML 2019, Optimal Kronecker-sum approximation of real time recurrent learning). The proposed approach provides a few optimizations that improves this performance further, and outperforms SGD in terms of accuracy and the number of weights updates in a limited experimental setting.
+ves:
+ Training on an edge device is a relevant setting, where there is very little work so far, and this is a useful objective.
+ Focusing on the Kronecker sum and speeding it seems like an interesting solution to solving this problem.
+ The experimental settings considered (e.g. flipping bits and adding Gaussian noise to weights) is interesting, and perhaps of larger relevance to other work in the area.
+ The ablation studies provided in the appendix are useful and appreciable.
Concerns:
- The key concern is that most of the proposed method is built upon Benzing et al’s work (ICML 2019), and the original contribution seems limited. While the paper introduces a few optimizations further, this seems to be incremental than originally novel.
- The problem is formulated for linear regression with least-squares loss, and the experimentation is carried out on the MNIST dataset (classification setting). How is the methodology relevant to the cross-entropy loss, typically used for classification? The paper does not talk about this.
- The results are shown for a network with four 3 x 3 convolution layers and two fully connected layers on the MNIST dataset. Results with different architectures on a few other datasets (at least CIFAR-10) would be necessary to assess the usefulness of this method. Some discussion on what would be the maximum depth of the network that can be trained using this training schema and hardware would be very useful.
- Do all deep neural network architectures (and loss functions) admit a Kronecker sum representation? What class of models can benefit from this method? Factorizations such as Cholesky allow to interpolate between computational complexity and decomposition rank by tuning a rank hyperparameter. Why would such factorizations not be better for memory-constrained settings?
- It would be interesting to see the results using the proposed algorithm on standard hardware, will it provide the same performance as SGD when the scale of the problem increases in terms of layers and other regularization techniques, etc.? This would help understand the performance of the algorithm in a standard-setting.
Minor issues:
- The paper could have been organized better. Most of the main paper is used to describe Benzing et al’s method, and a lot of the details of the original contributions are in the Appendix. While the equations discussed in Section 4 hold for Linear Regression, the same cannot be directly extended for other networks like CNNs. Considering all the experiments are on CNNs, Sec B.2 in the Appendix should have been in the main paper to help a reader follow the experiments.
=====POST-REBUTTAL COMMENTS========
I thank the authors for the rebuttal. The paper's organization is much better now. Unfortunately, there are many concerns which are still not convincingly answered: (i) Considering the focus of this work (as admitted by the authors in the rebuttal) is empirical and with limited technical novelty, more comprehensive results on many datasets are required. To some extent, transfer to ImageNet addresses this partially, but not convincingly enough; (ii) It is not clear from the authors' rebuttal why testing this on standard hardware is not possible/appropriate. The rebuttal says "hard to quantify due to a variety of factors" but that is vague without stating the factors.
I am willing to increase the rating from 3 to 4, but am unfortunately still towards rejecting the current shape of this work. I believe the work will benefit from considering all reviewers' comments more comprehensively to have a more impactful paper.


Deep neural networks exhibit complex learning dynamics due to highly nonconvex loss landscape. Second order approaches, such as natural gradient descent, mitigate such problems by neutralizing the effect of potentially ill-conditioned curvature, yet it is largely unknown how the current theory of deep learning generalizes beyond gradient descent to these higher order learning rules. To answer these questions, we derive exact solutions to learning dynamics of deep linear networks under a spectrum of curvature-corrected learning rules. Our analysis reveals that curvature corrected learning preserves a core feature of gradient descent, a conservation law, such that the learning trajectory follows precisely the same path in the underlying manifold as gradient descent, only accelerating the temporal dynamics along the path. We also show that layer-restricted approximations of natural gradient, which are widely used in most second order methods (e.g. K-FAC), can significantly distort the learning trajectory into highly diverging dynamics that significantly differs from true natural gradient, which may lead to undesirable network properties. We also introduce fractional natural gradient that applies partial curvature correction, and show that it provides most of the benefit of full curvature correction in terms of convergence speed, with additional benefit of superior numerical stability and neutralizing vanishing/exploding gradient problems, which holds true also in layer-restricted approximations.

In this manuscript, the authors analyze the dynamics of training deep linear neural networks under a generalized family of natural gradient methods that apply curvature corrections. They first show that the learning trajectory (direction of singular mode dynamics) in natural gradient descent follows the same path as gradient descent, while only accelerating the temporal dynamics along the path. Moreover, the authors show that the learning trajectory in layer-restricted approximations of natural gradient descent can significantly differ from the true natural gradient. Also, the authors proposed a fractional natural gradient that applies partial curvature correction which in addition to faster convergence, neutralizes vanishing/exploding gradient problems.
I vote to reject this paper due to the following major concerns.
1.	The results of the paper are limited to specific deep linear neural networks which are not practical. More specifically, in addition to only covering the deep linear case, the results hold under the following unexplained assumptions: (a) the correlation matrix of the input data is assumed to be whitened; (b) the singular vectors of adjacent weight matrices are well aligned; (c) Section 4, the singular values of all the weight matrices are the same.
2.	The paper lacks a cohesive introduction that includes a literature review on this very rich area of research.
3.	The paper is not clearly written and is missing many details.
I discuss my comments on the manuscript in details next.
•	While the assumption on the input correlation matrix being whitened can be dropped, other assumptions in the paper are unrealistic and are stated without any intuition or explanation (the singular vectors of adjacent weight matrices are well aligned; (c) Section 4, the singular values of all the weight matrices are the same.)

•	The analysis performed in the paper cover the deep linear case and no intuition is provided on how these results can help in understanding the more general non-linear case. By citing [Saxe et al. 2013], the authors claim that the effect of curvature of loss landscape on deep learning dynamics can be well captured by deep linear networks. However, in [Saxe et al. 2013] this was only shown empirically on a specific example. In recent years, we have encountered many results that hold in deep linear networks but fail to hold when including simple non-linearities. For instance, when the target matrix Y is full rank, every local minimum of the studied problem is global in the deep linear case. However, this result does not hold even with the simplest non-linearities.
•	The paper is unpolished and the objectives and contributions are not clearly stated. Moreover, the results are not rigorously stated and proved. Moreover, the algorithms stated and proposed in the paper are directly used without any introduction or explanation (how they work? What update rules do they use?  …)
•	The introduction in the paper is very short and does not include a literature review on this rich area of research. Hence, the problem is not motivated by a gap in the literature and not linked to other results that study the behavior of algorithms used for training deep neural networks.


We study non-collaborative dialogs, where two agents have a conflict of interest but must strategically communicate to reach an agreement (e.g., negotiation). This setting poses new challenges for modeling dialog history because the dialog's outcome relies not only on the semantic intent, but also on tactics that convey the intent. We propose to model both semantic and tactic history using finite state transducers (FSTs). Unlike RNN, FSTs can explicitly represent dialog history through all the states traversed, facilitating interpretability of dialog structure. We train FSTs on a set of strategies and tactics used in negotiation dialogs. The trained FSTs show plausible tactic structure and can be generalized to other noncollaborative domains (e.g., persuasion). We evaluate the FSTs by incorporating them in an automated negotiating system that attempts to sell products and a persuasion system that persuades people to donate to a charity. Experiments show that explicitly modeling both semantic and tactic history is an effective way to improve both dialog policy planning and generation performance.

This paper proposed a framework for non-collaborative dialog systems. To track the history better, the authors propose to apply two pre-trained finite state transducers (FSTs) to take the sequence of dialog acts or strategies as inputs and output sequences of state embeddings for the hierarchical encoder-decoder (HED) part as inputs. The authors test their model on two tasks, a bargain task and a persuasion task.
The model that the authors prepose makes sense to me. It seems that the FSTs can successfully help the model to track the history better and propose useful state embeddings for the rest part of the model, on both dialog acts and strategies. Also, the authors have done a comprehensive empirical study to show the gain of the proposed techniques over the pure HED model and some baseline method. So I think this is a good work. But potentially it would be better if the authors can show more insights and details for the FSTs.
Some detailed comments:
1. The example dialogs in Table 3 show us that the FSTs can output dialog act distributions that can help the agent behave better compared to the case with an RNN for tracking the history, and the case without FST/RNN. This shows the success of the FST framework.
2. The authors have done a comprehensive comparison for the cases with / without the two FSTs, as well as the case without the strategy predictor. Also, the authors compared with another existing method (Sequicity). The proposed method performs well compared to the baseline methods, on both general performances and the two types of human-evaluations. This shows the good empirical performance of the proposed method.
3. This paper can be better if the authors can provide more insights and details for the FSTs. For example, why FST is better than RNN? The authors mentioned something about that, but it is not clear enough and it will be great if the authors can provide more discussions on this. In addition, the output alphabets and the sets of states are not clear for the FSTs, it will be better if the authors show the full details for the FSTs, as in my opinion this is the major contribution of this work.
4. A minor comment: There are some inaccurate expressions in this paper. For example, the authors mentioned that the FSTs about put probabilistic density functions. However, to my understanding, the set of dialog acts is finite, hence FSTs output probabilistic mass functions, not probabilistic density functions. Please proofread about that.
Question:
1. Can you explain more about the choice for the parameters? For example, why the possible strategy output is a 15-dimensional binary-value vector?
2. For the loss L_{joint}, shouldn't it be (st_{t+1,j}\not\in u_{t+1}) in the indicator function?


How to 0wn NAS in Your Spare Time
New data processing pipelines and novel network architectures increasingly drive the success of deep learning. In consequence, the industry considers topperforming architectures as intellectual property and devotes considerable computational resources to discovering such architectures through neural architecture search (NAS). This provides an incentive for adversaries to steal these novel architectures; when used in the cloud, to provide Machine Learning as a Service (MLaaS), the adversaries also have an opportunity to reconstruct the architectures by exploiting a range of hardware side channels. However, it is challenging to reconstruct novel architectures and pipelines without knowing the computational graph (e.g., the layers, branches or skip connections), the architectural parameters (e.g., the number of filters in a convolutional layer) or the specific pre-processing steps (e.g. embeddings). In this paper, we design an algorithm that reconstructs the key components of a novel deep learning system by exploiting a small amount of information leakage from a cache side-channel attack, Flush+Reload. We use Flush+Reload to infer the trace of computations and the timing for each computation. Our algorithm then generates candidate computational graphs from the trace and eliminates incompatible candidates through a parameter estimation process. We implement our algorithm on PyTorch and Tensorflow. We demonstrate experimentally that we can reconstruct MalConv, a novel data pre-processing pipeline for malware detection, and ProxylessNAS-CPU, a novel network architecture for the ImageNet classification optimized to run on CPUs, without knowing the architecture family. In both cases, we achieve 0% error. These results suggest hardware side channels are a practical attack vector against MLaaS, and more efforts should be devoted to understanding their impact on the security of deep learning systems.

Summary
---
This paper proposes to use a computer security method, "Flush+Reload" to infer the DNN architecture of a victim in the setting where both the attacker and the victim share the same machine in a cloud computing scenario. This does not require any physical access to the machine, however it does require that a CPU is shared, and the inference of the architectural details is based on the time it takes to reload computations from cache.
The paper is overall clear and well written.
Motivations of the paper
---
However, concerning the motivations of the paper, I'd like some clarifications. As far as I know, in the deep learning community, the most effective architectures are published and public (VGG, Inception, ResNet, Transformer...).
I am a bit confused by the sentence "As a result, in the industry such novel DL systems are kept as trade secrets or
intellectual property as they give their owners a competitive edge (Christian & Vanhoucke, 2017)." which justifies that architectures are kept secret and thus may be prone being stolen.
This US patent is public and explains the method. As far as I know, it has never been enforced. Furthermore, this patent is associated with the paper "Going deep with convolutions", Szegedy et al. which introduced the Inception architecture, is public, very well-known, and thus I do not believe anyone would have any commercial interest in stealing it.
Furthermore, I do have the impression that the edge many companies have over their competitors is the private datasets they own much more than the architectural details.
Method and applicability
---
While the method Flush+Reload itself is not novel, its application to the DNNs case and the way to reconstruct the architecture (generating the candidates, pruning) is.
However, I do have some practical concerns about the applicability of the method.
As far as I understand, it can only work on one CPU. Most DNNs, even for inference, are run on (one or multiple) GPUs. Can the method be extended to work on GPUs?
Also, while the assumption that both the attacker and the victim use the same framework is realistic to me, I believe, they should also both use the same version of the said library, no? Otherwise some operations might be faster in some versions and slower in others, this is thus an additional and much stronger assumption to make.
At last, this would require the victim to use a public cloud service. However, as far as I know, many of the companies who could potentially design new architectures have their own private cloud. I am not certain that someone disposing of a new, private, and powerful architecture would use it on a public cloud service.
Experiments
---
The experimental section seems very limited to me. The authors show that they are able to reconstruct perfectly 2 architectures. While this is encouraging, I would like to see the limits of the proposed method.
Why not generate N random (or not so random) architectures and try to reconstruct them? Where does the method fail, where does it succeed?
What if the victim used a custom layer that the method could not recover? Does it still recover a similar architecture?
Conclusion
---
While the paper, proposed to use Flush+Reload for recovering DNNs architectures and succeeds for at least 2 non trivial architectures, I do not recommend acceptance.
First I am concerned by the problem this paper is tackling. Can this realistically happen in a real-life scenario?
Second, I am worried that the method suffers from very strong limitations in practice (eg the usage of a CPU for both victim and attacker).
Finally, and importantly, while the experiments show some interesting first results, they are limited, I am not able to judge the strengths and weaknesses of the method, and thus I cannot assess the usefulness of the proposed method.
Note: I have to say that this paper is definitely out of my area of expertise, even though I am confident in my understanding of the paper, it may be that some of my concerns are unfounded. If this is the case I will adjust my score accordingly.


We propose GraphNVP, an invertible flow-based molecular graph generation model. Existing flow-based models only handle node attributes of a graph with invertible maps. In contrast, our model is the first invertible model for the whole graph components: both dequantized node attributes and adjacency tensor are converted into latent vectors through two novel invertible flows. This decomposition yields the exact likelihood maximization on graph-structured data. We decompose the generation of a graph into two steps: generation of (i) an adjacency tensor and (ii) node attributes. We empirically demonstrate that our model and the two-step generation efficiently generates valid molecular graphs with almost no duplicated molecules, although there are no domain-specific heuristics ingrained in the model. We also confirm that the sampling (generation) of graphs is faster in order of magnitude than other models in our implementation. In addition, we observe that the learned latent space can be used to generate molecules with desired chemical properties. Finally we list open problems for this new direction of fully invertible graph generation researches.

This paper presents a new reversible flow-based graph generative model wherein the whole graph i.e., representative attributes such as node features and adjacency tensor is modeled using seperate streams of invertible flow model. This allows training of generative model using exact likelihood maximization over the underlying graph dataset.The model avoids encoding any domain specific heuristics and thus can be applied to any structured graph data. The paper focusses it applicability for molecular graphs. Given that this approach avoids sequential generation of graph, it is faster by an order of magnitude than prior models for molecular generation. Empirical experiments on couple of molecular graph data suggets that GraphNVP approach performs as well as prior approach but albeit without any rule checker.
My major concern with such invertible models is "scalability". Given that flow-based model are required to retain the original dimension its applicability is limited to low dimensional feature vectors. In the case of GraphNVP, this means limited number of node labels as well as edge labels. Additionally, since it limits adjacency tensor, this would lead to modeling graphs with few nodes. However, if integrated with encoder-decder model some of these limitation can be overcome. Given this major weakness and with limited novelty (i.e., extending to adjacency tensor), I am inclined to reject this paper. I shall improve my rating if GraphNVP is applied to general graph structures - synthetic / real.
Few more limitations:
1. Although paper claims one-shot generation of graphs, in reality it seems otherwise. Since every layer processes only on single node, overall it operates sequentially from one node to another.
2. Moreover, this same sequential processing yet again limits it applicability to small graphs.
3. As in MolGAN, the direct generation of adjacency tensor leads to training with fixed size graphs i.e., through the addition of virtual nodes.  It is not possible to train model with variable number of nodes.
4. As pointed by authors, their model is not node permutation invariant.
Clarification:
1. Are the function 's' and 't' fixed across time ? For QM9 with max of 9 atoms and 27 layers, each atom attribute is processed multiple times. Are they processed using same functionality of s and t ?
2. Is it possible to model permutation invariance by augmenting the training data using multiple permutation of nodes such as BFS, DFS, degree, k-node (see GRAN) ?
3. I understand GraphNVP can reconstruct perfectly. But I fail to note the actual significance of such metric. If it reconstruct 100% or not how does it matter ? What matters is unniqueness, validity and novelty.
4. Can you please compare inference time ?
5. How difficult is it to integrate validity checker with your generation process ? Can we have some comparison using it ?
Minor:
1. In eq (2) please use different notation for layer 'l' and node 'l'.
2. Page 4, penultimate line: So as functions s and t -? To model functions s and t


Most Deep Reinforcement Learning methods perform local search and therefore are prone to get stuck on non-optimal solutions. Furthermore, in simulation based training, such as domain-randomized simulation training, the availability of a simulation model is not exploited, which potentially decreases efficiency. To overcome issues of local search and exploit access to simulation models, we propose the use of kinodynamic planning methods as part of a model-based reinforcement learning method and to learn in an off-policy fashion from solved planning instances. We show that, even on a simple toy domain, D-RL methods (DDPG, PPO, SAC) are not immune to local optima and require additional exploration mechanisms. We show that our planning method exhibits a better state space coverage, collects data that allows for better policies than D-RL methods without additional exploration mechanisms and that starting from the planner data and performing additional training results in as good as or better policies than vanilla D-RL methods, while also creating data that is more fit for re-use in modified tasks.

The paper aims to improve exploration in DRL through the use of planning. This is claimed to increase state space coverage in exploration and yield better final policies than methods not augmented with planner derived data.
The current landscape of DRL research is very broad, but RRT can only directly be applied in certain continuous domains with continuous action spaces. With learned embedding functions, RRT can be applied more broadly (see "Taking the Scenic Route: Automatic Exploration for Videogames" Zhan 2019). The leap from RRT-like motion planning to the general topic of "planning" for policy search is not well motivated explained with respect to the literature. Uses of Monte Carlo Tree Search (as in AlphaGo) seem obviously related here.
This reviewer moves to reject the paper primarily on the grounds of overinterpreting experimental results from a single, extremely simple example RL task. In a domain so small, we can't tease out the role of exploration, we aren't engaging with the "deep" of DRL, and we are only considering one specific kind of planning. The implicit claims of general improvement to exploration and improved downstream policies are not supported by the experimental results. At the same time, no theoretical argument is attempted that would make up for the very narrow nature of the experiments.
Questions for the authors:
- If HalfCheetah is used to motivate the work, and it is so easily available in the open source offerings from OpenAI, why isn't one (or many more) tasks of *at least* this complexity considered? MountainCar is one of the gym environments with a 2D phasespace compatible with the kinds of plots used in this paper.
- Could the authors taxonomize the landscape of planning and provide a specific argument for focusing on RRT? (RRT is a fun algorithm, but how will you draw the attention of other researchers who are currently focused on Atari games?)


Federated learning is a distributed, privacy-aware learning scenario which trains a single model on data belonging to several clients. Each client trains a local model on its data and the local models are then aggregated by a central party. Current federated learning methods struggle in cases with heterogeneous client-side data distributions which can quickly lead to divergent local models and a collapse in performance. Careful hyper-parameter tuning is particularly important in these cases but traditional automated hyper-parameter tuning methods would require several training trials which is often impractical in a federated learning setting. We describe a two-pronged solution to the issues of robustness and hyper-parameter tuning in federated learning settings. We propose a novel representation matching scheme that reduces the divergence of local models by ensuring the feature representations in the global (aggregate) model can be derived from the locally learned representations. We also propose an online hyper-parameter tuning scheme which uses an online version of the REINFORCE algorithm to find a hyper-parameter distribution that maximizes the expected improvements in training loss. We show on several benchmarks that our two-part scheme of local representation matching and global adaptive hyper-parameters significantly improves performance and training robustness.

This manuscript proposes two strategies to improve both the robustness and accuracy of local agents under the setting of federated learning. Specifically, online reinforcement learning is used to perform adaptive hyperparameter search in order to maximize the utility of local models. This is quite an interesting idea since traditional hyperparameter tuning techniques, including random search or Bayesian optimization, usually requires access to monolithic dataset, which is clearly impractical under federated learning. The second contribution is an idea on using local distribution matching in order to synchronize the learning trajectories of different local models. This again is a novel and interesting idea. Overall the paper is well-written and clear to follow, which is a plus. Detailed comments and questions follow:
-   I understand that change of hyperparameter in local model affects the global model during in model aggregation stage. However, under the federated learning setting, if the number of local models is huge, then the influence of a single local model should be small. Hence instead of using online reinforcement learning for hyperparameter search, which is notoriously data-inefficient, why not framing the problem as a pure online learning problem? This helps to increase the data efficiency and also increases the stability of learning.
-   In the design of matching network, is there any intuition why performance significantly improves if the activations of
one layer are instead derived from the activations of the next layer of interest above it? Furthermore, why we need to have additional model with parameters $\theta_i$ for alignment? Intuitively if we want the new model $M$ to be close to the original one $M^F$, shouldn't we just use some distance measure, e.g., $\ell_2$ norm, to measure the distance of feature activations in the corresponding layers directly?
-   The experiments are illustrative, but might be too toyish under the federated learning setting. I appreciate the ablation studies the authors performed to show the relative impact of different strategies, which makes the relative contributions more clear. However, even in the case of MNIST and CIFAR, the improvement over baseline FA is not very significant. In this case, it would be better if the authors could also report the computational overhead in terms of running time.


Malicious PDF files represent one of the biggest threats to computer security. To detect them, significant research has been done using handwritten signatures or machine learning based on manual feature extraction. Those approaches are both time-consuming, requires significant prior knowledge and the list of features has to be updated with each newly discovered vulnerability. In this work, we propose a novel algorithm that uses a Convolutional Neural Network (CNN) on the byte level of the file, without any handcrafted features. We show, using a data set of 130000 files, that our approach maintains a high detection rate (96%) of PDF malware and even detects new malicious files, still undetected by most antiviruses. Using automatically generated features from our CNN network, and applying a clustering algorithm, we also obtain high similarity between the antiviruses' labels and the resulting clusters.

The paper proposes to use CNN to detect malicious PDFs. The paper uses two simple CNN models and train them on a collected dataset, compare with other antivirus software, and conclude that CNN performs better. But I have serious concerns about the experiments.
1. The major concern is with the dataset. The dataset used by this work are collected by authors themselves, and I see a serious problem in this process. The malicious and benign ones are not collected from the same distribution: "The malicious samples are taken from VirusTotal (3). They were uploaded on the website between the 5/5/2018 and the 11/14/2018"; "The benign files were obtained using collaboration with a private company". Thus, the model may not be actually learning what's malicious and what's benign, but only learning whether the pdf comes from that private company or VirusTotal. It's enough since the test set is also collected this way. It can be fairly easy to distinguish between two different datasets, and the same reasoning applies here. Also, the authors are not making the dataset available due to privacy reasons, which further make the dataset's validity a question.
2. Also, antivirus softwares are applicable on any pdf files, but the model trained with the dataset collected may not be useful in other circumstances. The comparison between them are not fair. It should compare the same model/software but on multiple different datasets to demonstrate the model's general applicability.
3. The experiments done in this work is also not of a satisfactory level. For example, there are some missing blanks in Table 1. Why is that? The figures (e.g., Fig 1/2) can be improved a lot. It did not include baseline and provide little information. The fonts are too small.
4. No major novelty is introduced. The work is an application paper using very simple CNNs on the malicious PDF detection problem. This itself does not make the paper bad but combined with the unconvincing experiments it's a serious weakness.
In summary, the paper lacks solid experimental results to make its conclusion convincing and its model generalizable. I vote for rejection of the paper.