The streaming rollout of deep networks -towards fully model-parallel execution
Deep neural networks, and in particular recurrent networks, are promising candidates to control autonomous agents that interact in real-time with the physical world. However, this requires a seamless integration of temporal features into the network's architecture. For the training of and inference with recurrent neural networks, they are usually rolled out over time, and different rollouts exist. Conventionally during inference, the layers of a network are computed in a sequential manner resulting in sparse temporal integration of information and long response times. In this study, we present a theoretical framework to describe rollouts, the level of model-parallelization they induce, and demonstrate differences in solving specific tasks. We prove that certain rollouts, also for networks with only skip and no recurrent connections, enable earlier and more frequent responses, and show empirically that these early responses have better performance. The streaming rollout maximizes these properties and enables a fully parallel execution of the network reducing runtime on massively parallel devices. Finally, we provide an open-source toolbox to design, train, evaluate, and interact with streaming rollouts.

Update after rebuttal:  I appreciate that the authors addressed my concerns in their rebuttal. I hope that the authors can clarify the exposition + add examples as discussed, which would make the paper significantly more accessible.   ---          Summary:           This paper analyzes different ways to unroll recurrent neural networks with skip connections and self-loops. A main motivation is to increase the efficiency (e.g., response time) of the network during training/inference. A rollout is a graph that captures the functional dependency of network nodes over time. The authors argue that there are different possible rollouts that have different quality (e.g., response time), introduce mathematical definitions to describe rollouts (e.g., validity / model-parallelizable) and analyze rollouts theoretically and experimentally.            In my understanding, the actual conclusion of the paper seems to be: the streaming ("R \equiv 1") is the best, e.g., Theorem in L192 states that the streaming rollout achieves the lowest response time over the entire graph. The experiments seem to support that conclusion. Note that how to obtain the streaming rollout is not clearly stated by the authors, although the Thm in L192 seems to suggest a working rule for obtaining it.          Pro:           Originality/Significance:           - I'm not aware of earlier work that analyzes this low-level implementation issue, but it is worthwhile to analyze this for optimization purposes.           - Overall, the paper is grammatically well written.          Con:             Quality:           - The writing is a bit dense in places and the payoff from introducing all the different mathematical definitions seems to be the Thm in L192.             Clarity:           - It's not so clearly explained what the computational overhead is of the streaming rollout R \equiv 1: is there any since we are using more edges from the functional graph?           - The paper lacks some simple examples to motivate / illustrate the introduced mathematical definitions. E.g., what does R \equiv 1 represent / what is the intuition for (1)? A simple example would be useful. The same for "tableau" and "state": a simple example to elucidate the definitions would be helpful.                    - Why is the sequential ("R \equiv 0") rollout as in the top row in Fig 1? Maybe a detailed discussion / description here would be helpful.           L212: "single images are already sufficient for almost perfect classification results" --> I don't know what is meant here.           - Theorems are not numbered.           - Some formulas are not numbered.             Reproducibility:           - How is this implemented in framework XYZ?            - The authors don't seem to discuss how operationally to obtain the streaming rollout automatically from the functional definition of a computation graph. How do you do that?           - It does not seem straightforward to re-implement this without error.           Typos:           - L122: express -> expressed


KONG: Kernels for ordered-neighborhood graphs
We present novel graph kernels for graphs with node and edge labels that have ordered neighborhoods, i.e. when neighbor nodes follow an order. Graphs with ordered neighborhoods are a natural data representation for evolving graphs where edges are created over time, which induces an order. Combining convolutional subgraph kernels and string kernels, we design new scalable algorithms for generation of explicit graph feature maps using sketching techniques. We obtain precise bounds for the approximation accuracy and computational complexity of the proposed approaches and demonstrate their applicability on real datasets. In particular, our experiments demonstrate that neighborhood ordering results in more informative features. For the special case of general graphs, i.e., graphs without ordered neighborhoods, the new graph kernels yield efficient and simple algorithms for the comparison of label distributions between graphs.

The paper proposes a family of kernels for graphs with ordered neighbourhoods and discrete labels. Any member of the family is obtained by generating a string-based representation of each node of a graph and a subsequent exploitation of the basic  spectrum string kernel by Leslie et al. PSB 2002.   Specifically, the string associated to a node of the graph is efficiently and recursively generated via a tree traversal method that uses the neighbour ordering. The visit starts from a node and ends when the depth of the tree is h. Thus the visited subtrees are the starting features exploited by the kernel. The string is obtained by concatenating the discrete information attached to visited nodes. Contrary to what happens for recent state-of-the-art graph kernels, the string may not be hashed, but a k-gram histogram of it is computed. Since the number of different k-grams can quickly grow as a function of node degree and alphabet size, it is proposed to use a sketch strategy to just keep the hitters. The adopted technique is Tensor-Sketch (by Pham & Pagh, KDD 2013), which allows to have acceptable computational times. Moreover, it is observed that the histograms at depth i can be updated incrementally from those at level i-1 plus information on the (i-1)-prefixes and (i-1)-suffixes. An explicit feature space representation for the graph is thus obtained by summing up all the histograms obtained for the graph nodes. It is also observed that: i) kernels such as cosine and polynomial can be applied to histograms to obtain a soft matching; ii) the proposed family of kernels, due to their incremental nature, can also the adopted for streams of graphs.  Quality: The mathematical treatment is solid as well as all the concepts that are used as building blocks of the proposed approach. I just have few concerns on the following points:  i) the tree traversal strategy shapes the basic subtree features used by the kernel. This is not discussed in the paper. For example, while it is clear what the subtrees are for Algorithm 3 (weisfeiler-lehman kernel), this is not the case for Algorithm 2. In fact, breadth-first search skips edges between nodes at the same depth. This issue is deeply discussed in the paper (not cited) proposing the ODD kernel (A Tree-Based Kernel for Graphs. SDM 2012: 975-986). In the same paper, the issue about the ordering is addressed. In fact, in order to use tree kernels, it is important to have ordered children. The effort, in that case, was to give an order in a principled way, so to avoid different representations for isomorphic graphs. Clearly, if an order is given, than ODD kernels can be directly used to deal with ordered neighbourhoods. This is also the case for the WL kernel, where instead of using the lexicographic order to sort the labels of the neighbourhoods, the given order can be used. This trivial adaptation of existing kernels is not discussed in the paper. A related issue is how the experimental results for classical kernels reported in table 2 have been obtained: is the given order exploited by them or not ? The reverse question is valid for the results reported in Table 1. ii) it is not clear to me how much of the experimental results are due to the use of cosine and polynomial kernels.  To make a fair experimental comparison, also the other kernels should be composed with cosine and polynomial kernels. Moreover, I found the range of values for the WL kernel hyper-parameter too narrow, since my experience with that kernel on exactly the same datasets of the first part of the experiments is that optimal values, for some of them, are outside that range, although I use a nested CV approach for hyper-parameter selection. iii) A relevant graph kernel for graphs exploiting a sketching approach (lossy-counting) has been proposed in (A Lossy Counting Based Approach for Learning on Streams of Graphs on a Budget. IJCAI 2013: 1294-1301). It would be nice to elaborate on the differences with respect to the proposed approach.  Minor issues:  i) it would be useful to report variance for all curves plotted in figure 2. ii) the column referring to computational times in table 2 is confusing: it is not readily clear why there are two numbers for each entry; the heading of the table, as well as the associated caption, should explain that.  iii) typos: 109: “for same neighborhood”; line 207: “representation for the explicit feature maps”; line 245: “obtain following result”; line 246: “be stream”; 293: “the the”.  Clarity: the paper is generally well written and can be followed with no problem by an expert in the field. Organisation of the material is good. The section I think would need better explanations is the one on experiments. In particular, the issues I have raised at the end of point i) above should be clarified.   Originality: the paper builds on results from the literature. The idea to just use the k-gram histograms of the subtree visits is moderately new to my knowledge, although it can be considered a bit derivative. Moreover, I am not convinced that this strategy can work well for larger values of h unless large values of k are used. Some effort has been put into the derivation of the theoretical results. Overall I see some original contribution although limited. In particular, I think the paper still misses some deeper analysis of the generated feature spaces.  Significance: I guess the main contribution of the proposed kernels is given by the fact that soft matching is introduced both at the level of histograms and explicit feature vector approximation. Possible follow-ups seem to be limited to the ones already described in the paper.  After rebuttal. Concerning fair experimental comparison: most of the more advanced graph kernels nowadays use explicit representations. This allows a simple application of a polynomial kernel, as envisaged in the last part of the answer. I still think a fair comparison should compare against this version of the base kernel. This can be easily done for WL.


Autoconj: Recognizing and Exploiting Conjugacy Without a Domain-Specific Language
Deriving conditional and marginal distributions using conjugacy relationships can be time consuming and error prone. In this paper, we propose a strategy for automating such derivations. Unlike previous systems which focus on relationships between pairs of random variables, our system (which we call Autoconj) operates directly on Python functions that compute log-joint distribution functions. Autoconj provides support for conjugacy-exploiting algorithms in any Pythonembedded PPL. This paves the way for accelerating development of novel inference algorithms and structure-exploiting modeling strategies.

In this paper, the authors propose a system, called Autoconj, which is able to recognize and exploit conjugacy by analyzing Python functions that compute log-joint distribution functions. Autoconj is expected to facilitate the inference of probabilistic models.   It looks like Autoconj has potential to facilitate inference of the probabilistic model, which might be very useful. The writing of the paper looks structured, but I have difficulty in understanding some details in the paper.   I do not understand the contribution of Autoconj. It looks like Autoconj takes a python function, which represents the log-joint distribution function as the input. Does this log-joint distribution function need to be properly normalized? What is the output of Autoconj? What does it mean by “recognizing and exploiting conjugacy”? Does it mean that Autoconj is able to automatically generate the marginal and conditional distribution according to the query?  If so, how are these distributions represented? Are these distributions represented in terms of a black-box pdf (i.e., given a query for a certain latent variable $z_1 = c$, Autoconj returns the approximated posterior pdf value $q(c; \eta_{z_1})$), or it returns the form and the parameters of the posterior distribution (i.e., Autoconj returns that the posterior distribution of $z_1$ is a Gaussian distribution parameterized by a specific mean and variance, for example)? In the title, it is mentioned that Autoconj is “without a Domain-Specific Language”. What does “domain-specific” mean? Why is it important to a system like Autoconj?   I also have difficulty in understanding some details about Autoconj. In Equation (3), the variable $z$ are involved. Are these variables represent the latent variables in the probabilistic model? In Equation (4) and (5), it looks like Autoconj is based on variational inference. But in line 38, it says “Autoconj is not tied to any particular approximate inference algorithms”. So is Autoconj based on variational inference approximation? Does the “canonical form” mentioned in line 130 mean Equation (3)? In Section 3.3, Autoconj analyzes the sufficient statistics for the posterior distribution. Since the joint function is not provided in a symbolic form, how does Autoconj know which statistic is relevant? How the process described in Section 3 related to the equations described in Section 2?  In addition, I typically expect that a NIPS paper involves development of new models and theorems. It looks like Autoconj is an implementation of existing models, as described in Section 2. I am also not sure whether this paper is interesting to the NIPS community.  In summary, Autoconj might be potentially useful because it facilitates inference of probabilistic models. I cannot fully understand the paper, and am not sure whether this paper would interest NIPS community. Based on the information that I can understand about this paper, I do not suggest acceptance of this paper.   I have read the author feedback. I am still not very certain how Autoconj is implemented, especially for the rewriting system. Probably more details should be included. In addition, the contribution of the paper is still not clear to me. I still do not know how this method is different from existing systems. I do not think implementing the method in python itself is a very important contribution.


Training Deep Models Faster with Robust, Approximate Importance Sampling
In theory, importance sampling speeds up stochastic gradient algorithms for supervised learning by prioritizing training examples. In practice, the cost of computing importances greatly limits the impact of importance sampling. We propose a robust, approximate importance sampling procedure (RAIS) for stochastic gradient descent. By approximating the ideal sampling distribution using robust optimization, RAIS provides much of the benefit of exact importance sampling with drastically reduced overhead. Empirically, we find RAIS-SGD and standard SGD follow similar learning curves, but RAIS moves faster through these paths, achieving speed-ups of at least 20% and sometimes much more.

The paper proposes a robust way to approximate optimal importance sampling weights to reduce the variance of stochastic gradient for training machine learning models. I think the idea (and implementation) is definitely valuable for the community and deserves a publication, but improving experiments can make the paper much more useful and insightful. In particular, I believe that 1) the exact O-SGD baseline (sampling without approximations) should be included in the plots at least for small-scale experiments; 2) there should be not only epoch plots, but the time plots as well to directly see the overhead of the sampling scheme; 3) there should be oblation studies for the factors you think make your method particularly effective (the ones listed in sec 6.2: large dataset size, lack of random data augmentation; etc). Also, you mention that you use only the loss layer of the networks to compute the gradient norms, which I see as a major approximation. Consider using the method from [1] at least as a baseline if the overhead is too large to use it on practice.  Again, the paper is really good, but I would love to see it getting even better before the publication.  Minor points: Would be nice to have pointer to appendix whenever you included the proof in where. E.g. now it looks like you consider proof of (4) too trivial to write down and I spent some time to rederive it only to find later that it’s actually proven in the appendix. The introduction of the dataset D on line 109 is confusing because it appears out of nowhere and when get explained slowly. Also, I read this part many times and still not sure how do you choose objects for D. I don’t get equation (6): how does it follow from (4) and what is g_{U1} and g_{R1}? Figure 6.2 referenced on line 228 doesn’t seem to exist Since you mentioned generalization as something you pay attention to (line 237), it would be nice to have test errors as well (in contrast to only having validation errors, since you used validation set for hyperparameters tuning). BTW, did you find optimal hyperparameters for SGD and for your method independently?  [1] Goodfellow, Ian. "Efficient per-example gradient computations." arXiv preprint arXiv:1510.01799 (2015).  UPD: thanks for addressing my concerns, I'm now even more willing to accept the paper.


Stochastic Expectation Maximization with Variance Reduction
Expectation-Maximization (EM) is a popular tool for learning latent variable models, but the vanilla batch EM does not scale to large data sets because the whole data set is needed at every E-step. Stochastic Expectation Maximization (sEM) reduces the cost of E-step by stochastic approximation. However, sEM has a slower asymptotic convergence rate than batch EM, and requires a decreasing sequence of step sizes, which is difficult to tune. In this paper, we propose a variance reduced stochastic EM (sEM-vr) algorithm inspired by variance reduced stochastic gradient descent algorithms. We show that sEM-vr has the same exponential asymptotic convergence rate as batch EM. Moreover, sEM-vr only requires a constant step size to achieve this rate, which alleviates the burden of parameter tuning. We compare sEM-vr with batch EM, sEM and other algorithms on Gaussian mixture models and probabilistic latent semantic analysis, and sEM-vr converges significantly faster than these baselines.

# After authors response  I thank the authors for their response, and for taking into account my remarks about some typos in the mathematical developments and the bounds.  # Summary  This paper develops a new algorithm based on the EM. It builds on the classical stochastic EM of Cappé and Moulines (2009), with an extra variance reduction term in the iteration formula. This variance reduction technique is inspired from the stochastic gradient descent literature (in particular, algorithms developed in Le Roux et al 2012, Johnson and Zhang 2013, Defazio et al 2014).  After setting up the background to present previous results in a unified and clear way, the authors present their algorithm and show two theoretical properties of it: a local convergence rate, and a global convergence property.  In the last section, they compare their new algorithm to several state-of-the-art methods, on a Gaussian mixture toy example, and on a probabilistic latent semantic analysis problem.  # General comments  The article is clearly written, and the main ideas and developments are easy to follow. The problem is nicely presented, and the general framework set up in section 2 helps laying the ground for the two main results and proofs. See specific comments below on some possible typos in the appendices part of the proofs.  In practice, the algorithm seems usable, and in particular does not need a sequence of decreasing weights to be calibrated. The Gaussian mixture example gives a useful intuition of how the method works, compared to the standard stochastic EM.   I'm not familiar with the literature on latent semantic analysis, and I did not go over the mathematical and methodological details of the main example of section 4.2. The results seem to indicate that the new algorithm performs well, although I have no intuition on what a "significant improvement" is for this problem and criterion.  To conclude, this paper presents an algorithm with convergence guaranties that seem theoretically sound, and show how it can be used on a concrete example. I think it has the potential to open new developments path and further improvements for the family of EM-like algorithms.  # Specific remarks  ## Possible typo in Th. 2  I could not understand the last line of equation in the block between lines 336 and 437. More precisely, I don't understand how the sum on the right can be bounded by M\Delta_e^2. It does not seem at first sight that ||\hat{\theta}_{e, t} - \hat{\theta}_{e, 0}|| is bounded by \Delta = \max_{t} ||\hat{\theta}_{e, t} - \hat{\theta}_{e, t+1}||, which is the only way I could explain the bound. But maybe I'm missing something ?  With a naive approach, the best I could get for this bound is to write a triangular inequality, so that: ||\hat{\theta}_{e, t} - \hat{\theta}_{e, 0}||                  \leq \sum_{u=0}^{t-1}||\hat{\theta}_{e, u+1} - \hat{\theta}_{e, u}||                 \leq t \Delta_e Taking the sum in t, this would give a bound in M(M+1)/2 \Delta_e^2  (instead of M \Delta_e^2).  If that's true, then the bound on \rho l.347 should be modified accordingly, as well as l.185 in theorem 2. [\gamma/(M(M+1) L_\eta L_f) instead of \gamma/(2M L_\eta L_f)]   I don't think that this modified bound would change the spirit or relevance of  th.2. Again, I could be missing a simple argument here, but if that's the case,   maybe adding a few words of explanations at this point of the proof could help.  ## Possible Math Typos  * l.184: specify \gamma-convex *for any x, h* ?  * l.184: is p(x, h; \theta) different from p(x, h | \theta) ? The first notation seems to be used everywhere else in the text.  * l.418-420: the 128 factor is appearing at the wrong place in the intermediate equations (but this does not affect the final result). If I'm not mistaken, it should be '128 L_f^2' instead of 'L_f^2 / 128' everywhere. So:   * \rho_* = … = log(\lambda^2 M / (128 L_f^2))/(\lambda M) [eq. between l.418 and 419, and l.419]   * \kappa^2 = 128 L_f^2 / \lambda^2 [l.419, coherent with the main text at l.150]   * x = \lambda^2M/(128 L_f^2) [l.420]  * l.425 and 429: Should it be the expectation under p(h_i | x_i, \hat{\theta}_{e, 0}), instead of p(h_i, x_i | \hat{\theta}_{e, 0}) ?  * l.433 and 434: missing transpose on the differences of \eta ? (3rd line in the first block, 2de line in the second block)  * l.434: Replace both Q(\hat{\theta}_{e, t(+1)}) by Q(\hat{\theta}_{e, t(+1)}; \hat{\theta}_{e, 0}) ?  * l.435: when invoking the \gamma-strong concavity, maybe it would help to recall that this works because \nabla \hat{Q}_{e, t+1} (\hat{\theta}_{e, t+1}) = 0 ? (If that's indeed key ? I know it would have saved me some time)  ## Other Possible Typos  * l.165: utilize -> utilizing ? * l.432 assume -> assuming ? * l.23 and other: I think it should be "Cappé", and not "Cappe".  ## Reproducibility  The algorithms and developments used in section 4 are described at length in the supplementary material, apparently providing all the tools to reproduce the results (but I did not go over these developments closely). However, no ready-to-use package or script seem to be provided to ease this task.


Balanced Policy Evaluation and Learning
We present a new approach to the problems of evaluating and learning personalized decision policies from observational data of past contexts, decisions, and outcomes. Only the outcome of the enacted decision is available and the historical policy is unknown. These problems arise in personalized medicine using electronic health records and in internet advertising. Existing approaches use inverse propensity weighting (or, doubly robust versions) to make historical outcome (or, residual) data look like it were generated by a new policy being evaluated or learned. But this relies on a plug-in approach that rejects data points with a decision that disagrees with the new policy, leading to high variance estimates and ineffective learning. We propose a new, balance-based approach that too makes the data look like the new policy but does so directly by finding weights that optimize for balance between the weighted data and the target policy in the given, finite sample, which is equivalent to minimizing worst-case or posterior conditional mean square error. Our policy learner proceeds as a two-level optimization problem over policies and weights. We demonstrate that this approach markedly outperforms existing ones both in evaluation and learning, which is unsurprising given the wider support of balancebased weights. We establish extensive theoretical consistency guarantees and regret bounds that support this empirical success.

Summary: The paper proposes a new approach to policy learning and evaluation that uses weighted estimators (like most of the literature) but avoids the intermediate estimation of a propensity score -- instead, the authors propose to directly seek weights that minimize a certain notion of balance. They illustrate their methods with simulations, and offer theoretical guarantees.   I believe that the paper has strong points. I appreciate the effort that the authors made to introduce the reader to this literature, by setting up the problem carefully and reviewing the literature in some details (Section 1.1 and Section 1.2). I believe that the method -- as I understand it -- is theoretically sensible. My criticism of the paper centers around two axes: the practical interest of the approach, and the readability of the exposition. Detailed comments follow. This reviewer did not check the proofs.  Main comments:   1) I wish to reiterate my earlier appreciation of the effort made by the authors to write an accessible introduction to the policy evaluation and policy learning literature. Coming from a causal inference background, I found the review of existing approaches very useful and adequately detailed.    2) I also acknowledge the fact that the authors have endeavored to provide very general results. Theorem 1 and Corollary 2 deal with both weighted estimators and DR estimators. The following paragraph deals with general RKHS. Lemma 1 offers multiple interpretations for the CMSE objective. Section 3.2 explores consistency under a variety of assumptions. I recognize the effort that the authors have put into this paper.   3) My first serious criticism of the paper is its overall readability:  A) Although it displays the efforts put by the authors, I believe that the generality of the paper hurts its readability. My impression is that the paper is trying to do too much, and the main results end up being obscured. For instance, I believe that dealing with both weighted estimators and doubly robust estimators is not helpful -- it would have been better, in my opinion, to spend more time and develop more intuition for the simple case, while referring the reader to the appendix for the DR results. It is unclear what Lemma 1 adds to the overall exposition of their method; it is interesting, but not crucial to the paper, and thus distracting. Presenting results and assumptions in terms of $C_0$ universal kernels seems unnecessarily general. It would have been better to focus on the most widely used kernel and then offer generalizations in appendix. I am not sure that the generality afforded by the use of the Rademacher complexity contributed much to the overall message of the paper.    B) One consequence of this is that it is sometimes difficult to see what the flow of the paper is, and what each subsection is trying to accomplish. What is the purpose of Lemma~1 in the overall picture? The connection with Causal Inference is not crucial, and distracts from the main point of the paper. Is it necessary to have the results of both Theorem 3 and Corollary 4? Is it necessary to have both Theorem~6 and Corollary~7? Etc...    C) The general impression is that the true core of the paper is small -- contained in Theorem~1, and lines 150-152. The rest is an attempt to justify the method by providing examples and theoretical results, neither of which I found of practical interest -- more on that later.    D)Trying to cram so many different results in 10 pages made the layout of the paper very, very dense and, in places, unreadable. Theorem 3 and Corollary 4 are examples. Most equations are inlined. There is no whitespace before or after definitions, theorems, etc...      4) My second criticism concerns the practical interest of the paper.   A) I believe that theoretical guarantees are important. But serious effort should be made to assess whether the assumptions made are practically relevant and are likely to hold in practice, etc... This is another place where the generality of the results is problematic. Section~3.2 is a particularly stark example of that: what are the actual implications of the assumptions and results of Section~3.2? To be fair, the paragraph after Corollary 4 (lines 211-217) is trying to do that, but I think it is somewhat under-developed.    B) Another example in which the practicality of the method is obscured is with the use of the logit parametrized policy class. In Example 3 they generate data from a multilogit model (thus, compatible with the parametrized policy class). But what if the data generating process is not even remotely close to being a logit? What if it is not continuous in $x$?     C) The authors hint at it in the conclusion, but a longer discussion of the computational complexity would have been useful. Including even a rough idea of the execution time in Table~1 would have been helpful.     D) In general, I would have liked the examples to be more developed, explore misspecification in more details, etc...   A few minor comments:  1) You criticize the `two-stage' nature of the approaches in the literature, and then propose a `bilevel' approach in Section~3. What is the difference between `two-stage' and `bilevel'.   2) l. 22-23: transform something `into' rather than `to'.   3) l. 81: '[...] the estimator by via' should be either `by' or `via'.   4) l. 124: the sentence `Minimal CMSE is the target of choosing weights for weighting-based policy evaluation' is confusing.


Neural Tangent Kernel: Convergence and Generalization in Neural Networks
At initialization, artificial neural networks (ANNs) are equivalent to Gaussian processes in the infinite-width limit (12; 9), thus connecting them to kernel methods. We prove that the evolution of an ANN during training can also be described by a kernel: during gradient descent on the parameters of an ANN, the network function f θ (which maps input vectors to output vectors) follows the kernel gradient of the functional cost (which is convex, in contrast to the parameter cost) w.r.t. a new kernel: the Neural Tangent Kernel (NTK). This kernel is central to describe the generalization features of ANNs. While the NTK is random at initialization and varies during training, in the infinite-width limit it converges to an explicit limiting kernel and it stays constant during training. This makes it possible to study the training of ANNs in function space instead of parameter space. Convergence of the training can then be related to the positive-definiteness of the limiting NTK. We then focus on the setting of least-squares regression and show that in the infinitewidth limit, the network function f θ follows a linear differential equation during training. The convergence is fastest along the largest kernel principal components of the input data with respect to the NTK, hence suggesting a theoretical motivation for early stopping. Finally we study the NTK numerically, observe its behavior for wide networks, and compare it to the infinite-width limit.

The authors prove that networks of infinite width trained with SGD and (infinitely) small step size evolve according to a differential equation, the solution of which depends only on the covariance kernel of the data and, in the case of L2 regression, on the eigenspectrum of the Kernel.  I believe this is a breakthrough result in the field of neural network theory. It elevates the analysis of infinitely wide networks from the study of the static initial function to closely predicting the entire training path.  There are a plethora of powerful consequences about infinitely wide, fully-connected networks:  - They cannot learn information not contained in the covariance matrix - Change to latent representation and parameters tends to zero as width goes to infinity.  - Therefore, disentangled intermediate representations are not learnt - Because the weight change converges to zero, effective depth goes to 2. - The complexity of nonlinearity design, including placing different nonlinearities in different layers, is reduced to NTK design. - If all inputs have the same length, the NTK is a 1d function. Therefore choosing nonlinearities in all layers reduces to choosing a single 1d function. - If all inputs have the same length, assuming a 1-to-1 mapping between NTL and sigma, all networks are equivalent to single-hidden-layer networks.  This doesn't even mention the new analytical tools introduced that will no doubt lead to further important discoveries.  There are some notation / correctness issues.  Consider the term "Gaussian function $f$ of covariance $\Sigma^{(L)}$". In line 122, $f$ is written as a function of a single variable, i.e. $f(x)$ and $f(x')$. But if $f$ is a Gaussian function of one variable, it only has a variance, no covariance. Hence, the statement "$f$ of covariance $\Sigma^{(L)}$" makes no sense.   Further, also on line 122, we have the expression $\mathbb{E}_f$. But if $f$ is a Gaussian function, it is not a random variable, and therefore we cannot take an expectation over it.   Finally, in line 126, we have $f \sim \mathcal{N}(0,\Sigma^{(L-1)})$. Again, this notation suggests that $f$ is a random variable, not a Gaussian function. The notation $\mathcal{N}(0,\Sigma^{(L-1)})$ also suggests that the scalar $\Sigma^{(L-1)}$ is the variance of a 1d random variable, NOT the covariance, as is claimed earlier.  The larger issue is that the notation chosen by the authors suggests that the covariance $\Sigma^{(L)}$ in a given network is only a function of the covariance in the previous layer, $\Sigma^{(L-1)}$. This is not the case. It depends on $\Sigma^{(L-1)}$ and the variance of $\tilde{\alpha}(x)$ in layer L-1 and the variance of $\tilde{\alpha}(x')$ in layer L-1. In fact, one cannot define the covariance $\Sigma^{(L)}$ recursively in isolation. One has to resursively define the triple (variance(x),variance(x'),covariance(x,x')) jointly via recursion. The variances then also show up in the definition of the NTK and pretty much everywhere also the covariance shows up.  The authors should fix this ambiguity in their final version.  Some other minor comments:  - line 63: $\alpha^{(l)}$ is missing the $^$ sign. - in the beginning, $L$ is used to denote the depth of the network and $\ell$ is used for layer recursion. But halfway through, $L$ starts being used for layer recursion. It might be worth changing this for consistency. - in remark 1, it is claimed that under Xavier initialization, weight matrix entries are initialized to independent Gaussians of variance $\frac{1}{n_l}$. In fact, Xavier initialization uses independent centered uniform random variables of variance $\frac{2}{n_l + n_{l+1}}$. Variance $\frac{1}{n_l}$ is sometimes called LeCun initialization. Further, in the paper that introduced Xavier initialization, biases were initialized to zero, not $\mathcal{N}(0,1)$. In fact, initializing biases as $\mathcal{N}(0,1)$ is a terrible choices that is not used, as it would replace a substantial portion of the signal variance with bias in each layer.   ##### Rebuttal response ####  My misunderstanding regarding the covariance / variance issue probably resulted from me having little experience with the common terminology / notation of Kernel machines. I interpreted the recursive formula as trying to establish a recursion for specific values of x,x', i.e. recursing \Sigma^L(x,x') based on the value of \Sigma^L-1(x,x'). However, I see now that you are trying to recurse the entire function \Sigma^L(x,x') as defined for all pairs x,x' based on the entire function of \Sigma^L-1(x,x') for all pairs x,x'. ... That makes sense.  Again, nice paper :)


SimplE Embedding for Link Prediction in Knowledge Graphs
Knowledge graphs contain knowledge about the world and provide a structured representation of this knowledge. Current knowledge graphs contain only a small subset of what is true in the world. Link prediction approaches aim at predicting new links for a knowledge graph given the existing links among the entities. Tensor factorization approaches have proved promising for such link prediction problems. Proposed in 1927, Canonical Polyadic (CP) decomposition is among the first tensor factorization approaches. CP generally performs poorly for link prediction as it learns two independent embedding vectors for each entity, whereas they are really tied. We present a simple enhancement of CP (which we call SimplE) to allow the two embeddings of each entity to be learned dependently. The complexity of SimplE grows linearly with the size of embeddings. The embeddings learned through SimplE are interpretable, and certain types of background knowledge can be incorporated into these embeddings through weight tying. We prove SimplE is fully expressive and derive a bound on the size of its embeddings for full expressivity. We show empirically that, despite its simplicity, SimplE outperforms several state-of-the-art tensor factorization techniques. SimplE's code is available on GitHub at https://github.com/Mehran-k/SimplE.

# Summary  This paper considers link prediction in knowledge graphs: give a triple (head, relation, tail) ((h,r,t)) the task is to determine whether the triple represents a correct relation. The contribution is a tensor factorization method that represents each object and relation as some vector. The main novelty relative to previous approaches is that each relation r is represented by two embedding vectors: one for r, and one for r^-1. The motivation is that should allow objects that appear as both heads and tails to more easily learn jointly from both roles.    # Strengths  The individual components of the paper are mostly well written and easy to understand. The core idea is simple and intuitive, and the authors do a reasonable job of motivating it.  # Criticisms  1. It appears that some of the statements made in the paper are incorrect.  (i) the authors claim (comparing TransE and FTransE) that a model which assigns relations based on v(t) - v(r) is less expressive than one that assigns relations based only on (v(t)-v(r))/||v(t) - v(r)||. Consider the case where the only possible embeddings are (0,0), (1,1) and (2,2). The reversal of this inclusion makes it unclear whether Corollary 1 holds---it's at least not obvious to me.  (ii) At line 135, the sentence beginning "Models based on..." is not obviously true. For neural net universality to (trivially) hold all of the (h,r,t) would need to be input to some neural net---but that doesn't match the architectures described in "Deep Learning Approaches"  (iii) at lines 284-285, the loss function on minibatches is given and described as a negative log likelihood. Although the expression could be considered a negative llhd for some probabilistic model, such a model has not been previously introduced. (I'm actually unsure whether the expression literally corresponds to any distribution since there's no obvious normalization term)  2. I'm not sure about the significance. In particular, does it matter that SimplE is moderately more computationally efficient than ComplEx? What does that mean for training time? For prediction time?  3. The structuring of the paper is a rather odd. My two biggest complaints here are (i) the main idea of the paper isn't given until lines 200-205 on page 5, but this could have easily been explained in the introduction (along with an overview of the full method) (ii) the loss function and training procedure are described in "Learning SimplE Models" (starting at line 275) as a subsection of "Experiments and Results". This material deals with specifying the model and should appear earlier, more prominently, and certainly not as a subsection of experiments.  4. I find the footnote on page 7 promising to release the code after the paper is submitted to be somewhat troubling. Why not include it with the submission? Although this is not strictly required, it's certainly better practice.  5. The major motivation for SimplE is that it allows the head and tail vectors for each object to be learned dependently. I don't see how this is reflected in the experiments.    ### Update  The authors mostly did a good job addressing my concerns (especially given the very limited space!). I still think 1.i is a problem---the fact that TransE takes embedding vectors to be points and FTransE takes embedding vectors to be lines means that TransE is more expressive.  I'm also still not convinced that 'fewer multiplications' is a meaningful advantage, but the paper is nice even if this is meaningless, so whatever.


Multitask Boosting for Survival Analysis with Competing Risks
The co-occurrence of multiple diseases among the general population is an important problem as those patients have more risk of complications and represent a large share of health care expenditure. Learning to predict time-to-event probabilities for these patients is a challenging problem because the risks of events are correlated (there are competing risks) with often only few patients experiencing individual events of interest, and of those only a fraction are actually observed in the data. We introduce in this paper a survival model with the flexibility to leverage a common representation of related events that is designed to correct for the strong imbalance in observed outcomes. The procedure is sequential: outcome-specific survival distributions form the components of nonparametric multivariate estimators which we combine into an ensemble in such a way as to ensure accurate predictions on all outcome types simultaneously. Our algorithm is general and represents the first boosting-like method for time-to-event data with multiple outcomes. We demonstrate the performance of our algorithm on synthetic and real data.

This paper has the aim to estimate cause-specific as well as overall survival distributions in a competing risk setting, and employs boosting algorithms in a multitask learning framework to obtain estimators. The paper is well-written and clear in its presentation, and applies established methods and algorithms to the setting. While there exist applications of all employed methods (tree-based, boosting, multitask learning) to estimation in competing-risks, which is very well described in the related work section, the paper presents novelty in the details of their approach and their goal to estimate full cause-specific and all-cause hazards. Further, evaluations on both synthetic as well as real-data show an improvement of prediction compared to existing approaches.  Strengths of the paper: - Aim of the paper is clearly stated and mathematical notation is well defined. - The related work section is well-written, and their proposed approach is compared with different existing approaches. - Synthetic and real-data applications are described in a reproducible manner and show that the proposed approach can yield an improvement in prediction in multiple scenarios.  - Nice idea of visualizations in Figures 1-2, which help to get an illustration of boosting and intuition on the partitions (but which could be improved, see below), further the description throughout the paper is easily accessible with very few typos.  Some comments: - Description of the toy example in Figure 1 and lines 52-67 could be improved: when reading the text and looking at the figure, it was not clear to me if the goal in the example was to improve the estimation of cause-specific hazards (i.e. for death due to CVD and death due to BC separately) or of all-cause hazard by using boosting. The presentation here seems to focus on all-cause hazards while the later synthetic and real-data examples seem to focus on cause-specific hazards, which was a bit confusing to me and might be improved, e.g. in line 61 (“… best partition of the data …”):  best partition with respect to what, high/low overall survival? - Line 2: “… have a higher risk …” instead of “… have more risk …”? - Line 75: add “given covariates“ in “represents the probability of a specific event of type k happening before time t given covariates”? - Line 77: What can \mathcal{X} and \mathcal{T} be, e.g. \mathbb{R}^k and \mathbb{R}^+ ? - Line 98: results in flexible estimates of task-specific time-to-event distributions? - Lines 227ff: it could help the reader to explicitly state that \bold{X} encompasses X_1 … X_100 - Lines 109-115: Here, it could be helpful to remind the reader what exactly is tested using Gray’s test statistic, i.e. if the equality of all subhazards is tested vs. that at least two subhazards differ.  - Figure 2: I looked at the example in the figure to get an intuitive understanding how the partition is obtained, i.e. which partition maximizes Gray’s test statistic. As the authors describe in the figure legend, the partition is supposed to maximize “a composite measure of the difference between the CIFs in each subset”. Now by looking at the figure, I was wondering if other partitions might not yield a great difference between the CIFs, and I was confused that the green dot with third-lowest feature value is different in the left and right plot. Is this point upweighted in the right plot, or is this a typo? - In the introduction (line 25-26), the authors describe that existing approaches can do well in prediction of all-cause survival but not well for cause-specific survival. As far as I could tell, their proposed approach was only compared in its estimation of cause-specific survival to existing approaches. How does it compare in prediction of overall survival? - What is shown in Table 1 in the supplements? Mean and SD? - The description and algorithm focuses on competing risks, and it seems to me that there is potential to include recurrent events (e.g. not only focus on death due to CVD/BC but also re-occurrence of BC) in the model as future work.  AFTER AUTHOR RESPONSE: I have read the author response, and think that the paper will be further strengthened by the corrections and additional results described in the rebuttal letter.


Constant Regret, Generalized Mixability, and Mirror Descent
We consider the setting of prediction with expert advice; a learner makes predictions by aggregating those of a group of experts. Under this setting, and for the right choice of loss function and "mixing" algorithm, it is possible for the learner to achieve a constant regret regardless of the number of prediction rounds. For example, a constant regret can be achieved for mixable losses using the aggregating algorithm. The Generalized Aggregating Algorithm (GAA) is a name for a family of algorithms parameterized by convex functions on simplices (entropies), which reduce to the aggregating algorithm when using the Shannon entropy S. For a given entropy Φ, losses for which a constant regret is possible using the GAA are called Φ-mixable. Which losses are Φ-mixable was previously left as an open question. We fully characterize Φ-mixability and answer other open questions posed by [6]. We show that the Shannon entropy S is fundamental in nature when it comes to mixability; any Φ-mixable loss is necessarily S-mixable, and the lowest worst-case regret of the GAA is achieved using the Shannon entropy. Finally, by leveraging the connection between the mirror descent algorithm and the update step of the GAA, we suggest a new adaptive generalized aggregating algorithm and analyze its performance in terms of the regret bound.

This paper studies a generalization of the notion of mixability, called \Phi-mixability, and developed an adaptive generalized aggregating algorithm (AGAA) for it.   To me, the interesting part of this paper is that it reveals the fundamental nature of the Shannon entropy in defining mixability: For a loss to be \Phi-mixable, it is necessary that it is mixable w.r.t. the Shannon entropy. Indeed, (9) gives an if and only if characterization of \Phi-mixability using mixability w.r.t. the Shannon entropy. Theorem 17 shows that exploiting the general \Phi-mixability does not help reduce the regret, though this result is only specific to the so-called generalized AA.   The AGAA is not very exciting to me. This paper provides a scenario where the AGAA can even achieve a negative regret, but the scenario does not seem very general. A more satisfactory argument, in my opinion, should be similar to the results for, say, the optimistic mirror descent; there is a clear characterization for the data to be "easy", and it is clear that smaller regret can be achieved with easy data. Moreover, as mentioned in the paper, the AGAA can also yield worse performance.  The main weakness of this paper, in my view, is that the main message is not very clear. The abstract emphasized both the novel characterization of \Phi-mixability and the AGAA. The characterization of \Phi-mixability, however, seems to suggest that generalizing the notion of mixability (at least in the direction of this paper) is not really helpful. The argument for the superiority of the AGAA is not strong enough, as I said in the preceding paragraph.   I did not have the time to check the very long proofs.   Other comments:   1. The presentation can be improved. This paper uses many terminologies and symbols without defining them first. For example, "Vovk's exponential mixing" in the first paragraph can be confusing to non-experts of online learning, and this confusion can be easily avoided by rewriting the sentence. The loss \ell is not defined before the appearance of R_\ell. The definition of H in (5) is missing.   2. The if and only if condition (9) looks similar to the condition proposed in the following two papers.      a. "A descent lemma beyond Lipschitz gradient continuity: First-order methods revisited and applications" by Bauschke et al.      b. "Relatively smooth convex optimization by first-order methods, and applications" by Lu et al.  There is perhaps be some relationship.   3. ln 96, p. 3: The claim that "we make no topological assumption on A" is unnecessarily too strong.   4. ln 247, p. 6: The similarity with Vovk's work on the fundamental nature of the logarithmic loss is mentioned. However, this similarity is clear to me, as Vovk's result is about the choice of the loss function, instead of the definition of mixability. Please elaborate.


Statistical Optimality of Stochastic Gradient Descent on Hard Learning Problems through Multiple Passes
We consider stochastic gradient descent (SGD) for least-squares regression with potentially several passes over the data. While several passes have been widely reported to perform practically better in terms of predictive performance on unseen data, the existing theoretical analysis of SGD suggests that a single pass is statistically optimal. While this is true for low-dimensional easy problems, we show that for hard problems, multiple passes lead to statistically optimal predictions while single pass does not; we also show that in these hard models, the optimal number of passes over the data increases with sample size. In order to define the notion of hardness and show that our predictive performances are optimal, we consider potentially infinite-dimensional models and notions typically associated to kernel methods, namely, the decay of eigenvalues of the covariance matrix of the features and the complexity of the optimal predictor as measured through the covariance matrix. We illustrate our results on synthetic experiments with non-linear kernel methods and on a classical benchmark with a linear model.

This paper identifies and separates (kernel) linear least-squares regression problems wherein carrying out multiple passes of stochastic gradient descent (SGD) over a training set can yield better statistical error than only a single pass. This is relevant to the core of machine learning theory, and relates to a line of work published at NIPS, ICML, COLT, and similar conferences in the past several years about the statistical error of one-pass, many-pass, and ERM-based learning.  The authors focus on regression problems captured, by assumption, by two parameters: alpha, which governs the exponent of a power-law eigenvalue decay, and r, which governs a transformation under which the Hilbert norm of the optimal predictor is bounded. They refer to problems where r <= (alpha - 1) / (2 * alpha) as "hard". The main result of the paper is to show that for these "hard" problems, multiple SGD passes either achieve (minimax) optimal rates of statistical estimation, or at least improve the rate relative to a single pass.  The results are interesting and might address an unanswered core question in machine learning, and the mathematical presentation is clear, with assumptions upfront. What's most needed is clarification on whether this applies to core machine learning problems, and how this relates to the previous results on optimality of single-pass SGD. What problem instances are included in the assumptions and then in the different "easy" and "hard" subclasses? In what way were the "hard" problems not also solved by single-pass SGD?  Two examples are given in the paper. Example 1 is a learning problem over the one-dimensional unit interval. Example 2 states a family of learning problems for which the authors claim alpha = 2m/d and r = s/(2m), provided d < 2m (indeed if d >= 2m then alpha is nonpositive and so there is no valid r to make for a "hard problem"). These examples left me confused about the introductory claim (e.g. in the abstract) that single pass SGD is better "for low-dimensional easy problems." Both examples are learning problems over input domains of inherently low dimension.  Experimentally, least-squares regression is an interesting core problem for the NIPS community. What is a reasonable choice of the parameters alpha and r for different common datasets, or even simply MNIST? What if a feature map or kernel known to be useful for MNIST is used? The paper currently has an experiment performed on MNIST (with no feature map or kernel), showing out-of-sample error in response to number of passes. However, the experiment does not validate any of the assumptions or explain whether the observed outcome was predicted by the paper's theory.  Finally, Section 4.2 describes in its final paragraph a setting where multi-pass SGD performs as well as the empirical risk minimizer, and that both are minimax optimal. Do we know whether a single-pass cannot achieve this result as well? If so, then this paragraph seems to imply, independently of the multi-pass SGD result, a separation between single-pass SGD and empirical risk minimization, which would be a result worth highlighting or explaining on its own. If not, then seeing as how both single-pass and empirical risk minimization work, it seems less surprising that multi-pass (which is in a sense between the two) works as well. Either way, it seems important to clarify which of the two scenarios holds.  ----  Edited after author response: It appears that, overall, the author response answers/addresses most of my questions/concerns above.


Diminishing Returns Shape Constraints for Interpretability and Regularization
We investigate machine learning models that can provide diminishing returns and accelerating returns guarantees to capture prior knowledge or policies about how outputs should depend on inputs. We show that one can build flexible, nonlinear, multi-dimensional models using lattice functions with any combination of concavity/convexity and monotonicity constraints on any subsets of features, and compare to new shape-constrained neural networks. We demonstrate on real-world examples that these shape constrained models can provide tuning-free regularization and improve model understandability.

This paper is about learning models constrained to be convex/concave increasing/decreasing in their *inputs*. Two such models are studied: neural networks, and lattice interpolation models. The two models are not comparable: the former is *jointly* constrained, while the latter is independently constrained in each input dimension.  The core motivation is interpretability and encoding prior assumptions (e.g. diminishing returns.) The learned models strictly enforce their constraints and perform well (sometimes better than unconstrained models) on a number of real-world tasks (albeit fairly low-dimensional -- a limitation of lattice models, if I understand correctly).  Strengths: - Two new constrained architectures with different properties but good performance are proposed. This gives potential users choice. - The exposition and motivatoin of the paper are very good: a pleasant and convincing read for someone who has not previously thought much about enforcing such constraints. It will change the way I tackle certain problems in the future.  Weaknesses: - While there is not much related work, I am wondering whether more experimental comparisons would be appropriate, e.g. with min-max networks, or Dugas et al., at least on some dataset where such models can express the desired constraints. - The technical delta from monotonic models (existing) to monotonic and convex/concave seems rather small, but sufficient and valuable, in my opinion. - The explanation of lattice models (S4) is fairly opaque for readers unfamiliar with such models. - The SCNN architecture is pretty much given as-is and is pretty terse; I would appreciate a bit more explanation, comparison to ICNN, and maybe a figure. It is not obvious for me to see that it leads to a convex and monotonic model, so it would be great if the paper would guide the reader a bit more there.  Questions: - Lattice models expect the input to be scaled in [0, 1]. If this is done at training time using the min/max from the training set, then some test set samples might be clipped, right? Are the constraints affected in such situations? Does convexity hold?  - I know the author's motivation (unlike ICNN) is not to learn easy-to-minimize functions; but would convex lattice models be easy to minimize?  - Why is this paper categorized under Fairness/Accountability/Transparency, am I missing something?  - The SCNN getting "lucky" on domain pricing is suspicious given your hyperparameter tuning. Are the chosen hyperparameters ever at the end of the searched range? The distance to the next best model is suspiciously large there.  Presentation suggestions:  - The introduction claims that "these shape constraints do not require tuning a free parameter". While technically true, the *choice* of employing a convex or concave constraint, and an increasing/decreasing constraint, can be seen as a hyperparameter that needs to be chosen or tuned.  - "We have found it easier to be confident about applying ceterus paribus convexity;" -- the word "confident" threw me off a little here, as I was not sure if this is about model confidence or human interpretability. I suspect the latter, but some slight rephrasing would be great.  - Unless I missed something, unconstrained neural nets are still often the best model on half of the tasks. After thinking about it, this is not surprising. It would be nice to guide the readers toward acknowledging this.  - Notation: the x[d] notation is used in eqn 1 before being defined on line 133.  - line 176: "corresponds" should be "corresponding" (or alternatively, replace "GAMs, with the" -> "GAMs; the")  - line 216: "was not separately run" -> "it was not separately run" - line 217: "a human can summarize the machine learned as": not sure what this means, possibly "a human can summarize what the machine (has) learned as"? or "a human can summarize the machine-learned model as"? Consider rephrasing. - line 274, 279: write out "standard deviation" instead of "std dev" - line 281: write out "diminishing returns" - "Result Scoring" strikes me as a bit too vague for a section heading, it could be perceived to be about your experiment result. Is there a more specific name for this task, maybe "query relevance scoring" or something?  === I have read your feedback. Thank you for addressing my observations; moving appendix D to the main seems like a good idea. I am not changing my score.


ChannelNets: Compact and Efficient Convolutional Neural Networks via Channel-Wise Convolutions
Convolutional neural networks (CNNs) have shown great capability of solving various artificial intelligence tasks. However, the increasing model size has raised challenges in employing them in resource-limited applications. In this work, we propose to compress deep models by using channel-wise convolutions, which replace dense connections among feature maps with sparse ones in CNNs. Based on this novel operation, we build light-weight CNNs known as ChannelNets. ChannelNets use three instances of channel-wise convolutions; namely group channel-wise convolutions, depth-wise separable channel-wise convolutions, and the convolutional classification layer. Compared to prior CNNs designed for mobile devices, ChannelNets achieve a significant reduction in terms of the number of parameters and computational cost without loss in accuracy. Notably, our work represents the first attempt to compress the fully-connected classification layer, which usually accounts for about 25% of total parameters in compact CNNs. Experimental results on the ImageNet dataset demonstrate that ChannelNets achieve consistently better performance compared to prior methods.

The paper proposes channel-wise convolutions that address the full connections between feature maps and replace them with sparse connections (based on 1-D convolutions). This reduces the #params and #FLOPS significantly; while maintaining high accuracy. The authors show results on imagenet classification and compare it to VGG/MobileNet variants to demonstrate this.  Strengths:  + The paper is well written and easy to follow. Background and related work such as standard convolution+fc layers used in neural nets; mobilenet and shufflenet variants to reduce computation are described in sufficient detail. + The proposed approach is novel and it replaces fully connected pattern of feature maps with 1-D convolutions. As stated, starting from LeNet [13] prior work has experimented with sparse connections between feature maps. In fact libraries like Tensorflow/Torch allow to choose a sparse connection pattern for convolutional layers as well. However, the sparsity in these works is chosen at random  and output channels are connected to a fixed fraction of input channels. The proposed work introduces sparsity with 1-D convolutions to convolutional, fully connected and classification layers (i.e. all parts) of a neural net. With appropriate choice of group size and stride, 1-D convolution strategy; it can be ensured that all convolutional channels are used to produce output. + Tab. 1 compares the proposed network to standard approaches and shows good accuracy for less parameters. Tab. 2 does an ablation study on the effects of sparsity on different parts of the network. Convolutional classification causes 3% drop in accuracy but also reduces parameters by ~1M.  Weakness:  - Results are presented on imagenet classification only. Results on CIFAR or scene classification (Places dataset) or on other tasks such as segmentation/detection will help make the case for channel-wise convolutions stronger.  The idea is novel, the paper is well written and results are sufficient to establish that the strategy is effective on imagenet classification. Therefore, I recommend accept.


Near-Optimal Time and Sample Complexities for Solving Markov Decision Processes with a Generative Model
In this paper we consider the problem of computing an -optimal policy of a discounted Markov Decision Process (DMDP) provided we can only access its transition function through a generative sampling model that given any state-action pair samples from the transition function in O(1) time. Given such a DMDP with states S, actions A, discount factor γ ∈ (0, 1), and rewards in range [0, 1] we provide an algorithm which computes an -optimal policy with probability 1 − δ where both the time spent and number of sample taken are upper bounded by
.
For fixed values of , this improves upon the previous best known bounds by a factor of (1 − γ) −1 and matches the sample complexity lower bounds proved in [AMK13] up to logarithmic factors. We also extend our method to computing -optimal policies for finite-horizon MDP with a generative model and provide a nearly matching sample complexity lower bound.

In this paper the authors present PAC-RL results for finite-spaces discounted MDP with a generative sampling model (which can generate in O(1) time a state transition for each given state-action pair). They introduce a value-iteration type algorithm for finding an $\epsilon$-optimal policy, and prove that its running time and sample complexity are nearly the best possible and match, up to logarithmic factors, the lower bounds established previously in [AMK13]. These results are significant: They improve over a recent result in [SWWY18] by a factor of $(1 - \gamma)^{-1}$ ($\gamma$ is the discount factor), and they also fill a known gap in the earlier results of [AMK13] regarding the sample complexity of finding a near-optimal policy. Both the algorithm and its analysis are very interesting. They build upon the prior work [AMK13] and [SWWY18], and combine the algorithmic ideas and proof techniques from both papers in a highly non-trivial and original way. In addition, as an extension to these results, the authors also present a similar near-optimality result for finite-horizon MDP (the extension is given in an appendix).  This is an exciting paper. The results are very strong, and the paper is very well-written. While the analysis is intricate and subtle, the authors have structured the paper well to give the reader the big picture, and they have succeeded in explaining the key ideas and proof steps very clearly.   Detailed comments, mostly about minor presentation issues:  1. line 157-176, the last paragraph on p. 4: This discussion on the "Bernstein Technique" is a little confusing. The authors talk about the variance of the total return of a policy $\pi$ and the variance of $V^\pi(S_1)$, but the authors' algorithm is a value iteration algorithm and it does not evaluate any policy. The name "Bernstein Technique" seems to suggest that the technique is based on Bernstein's inequality. But the key inequality in this technique is the inequality in the authors' Lemma C.2 (Lemma 8 in [AMK13]), applied to the optimal policy. This inequality is a property of MDP, not a statistical property derived from the Bernstein inequality. I suggest the authors revise the last paragraph on p. 4 to explain more precisely the third technique in their work.  2. line 214-215, p.5: It would be better if the authors can say something about the nature of these constants c_1, c_2, c_3, especially c_2.  3. p. 7: In the for-loop of Algorithm 2, the subroutine HALFERR (Algorithm 1) also needs the values of $\delta$ and $u$ as inputs. The authors need to specify these values in Algorithm 2.  4. p. 480, p. 15: The two equations in this line are strange; the proof of the second inequality of Lemma C.3 doesn't seem valid. Of course, this error doesn't affect other results, since Lemma C.3 is the authors' alternative proof of Lemma C.2, which was already proved in [AMK13, Lemma 8]. On the other hand, the last step of the proof of [AMK13, Lemma 8] given in that reference skips some details and isn't obvious, so it will be good if the authors can fix Lemma C.3 and give the proof details of [AMK13, Lemma 8] (the authors' Lemma C.2).  5. p. 17: It seems in this proof of Lemma 4.4, the authors also need the argument in the proof of Lemma E.3. Also, in the statement of Lemma 4.4 on p. 6, should $1 - 2 \delta$ be $1 - R \delta$ instead?   Other minor comments related to typos etc.:  1. Lemma 4.4, p. 216-219, p. 6: Should the definitions of $\pi^*$ and $P^\pi Q$ be given outside the lemma? Also, no need to repeat the definition of $Q^*$.   2. Table 2, p. 11: The first two rows are the same.  3. Before line 480, p. 15: It seems this part of the proof for the first inequality of Lemma C.3 can be simplified by applying Jensen's inequality to $(1 - \gamma)(I - \gamma P)^{-1} \sqrt{v}$.)  4. Lemma D.1, p. 15: Should the conclusion of the lemma include the words "with high probability"?  5. p. 19: The proof of Proposition E.3 is misplaced near the top of this page, before the proposition is stated. The proof of Theorem 4.6 is also misplaced and should be moved to Section E.2.  6. Some typos:  -- $\gamma$ is missing in (3.1)-(3.3), p. 3-4. -- What is $\beta$ in line 224, p. 6? -- In the displayed equation just above line 507, p. 15, $n$ should be $m$. -- p. 17: Should $\epsilon$ in the proof be $u$ instead?


Gamma-Poisson Dynamic Matrix Factorization Embedded with Metadata Influence
A conjugate Gamma-Poisson model for Dynamic Matrix Factorization incorporated with metadata influence (mGDMF for short) is proposed to effectively and efficiently model massive, sparse and dynamic data in recommendations. Modeling recommendation problems with a massive number of ratings and very sparse or even no ratings on some users/items in a dynamic setting is very demanding and poses critical challenges to well-studied matrix factorization models due to the large-scale, sparse and dynamic nature of the data. Our proposed mGDMF tackles these challenges by introducing three strategies: (1) constructing a stable GammaMarkov chain model that smoothly drifts over time by combining both static and dynamic latent features of data; (2) incorporating the user/item metadata into the model to tackle sparse ratings; and (3) undertaking stochastic variational inference to efficiently handle massive data. mGDMF is conjugate, dynamic and scalable. Experiments show that mGDMF significantly (both effectively and efficiently) outperforms the state-of-the-art static and dynamic models on large, sparse and dynamic data.

Paper : 2816 This paper presents a  scalable model which integrates dynamical Poisson factorization with metadata information. Here metadata information contains non-dynamic users and items attributes.  Authors design an algorithm which has two stages. One is to incorporate the metadata information to static latent random variables for each user and item. Second part is to learn dynamical nature of the observations via latent single order markov chain variable of gamma-gamma structure. The markov chain is modeled by absorbing the dynamical nature of data via its rate parameter of gamma distribution. Stochastic Variational inference has been used to infer approximating distribution’s parameters.  Algorithm has shown having learn-ability and predictive power over large datasets, also it has shown how it would enhance the recommendation of more sparsely rated items to users compared to baseline algorithms. Main contribution of paper is to model how to incorporate effect of users and items attributes which have nature of categorical distribution as an weight which is sampled from gamma distribution. Experiments has been done on three different datasets, results show advantage of current algorithm than baselines, there are some discussion about results and clarity which will be detailed in following.    Quality: Paper has clear approach to present the model and dependencies between latent variables and observation data. Most of derivation have been verified.  Just there are few minor points:  -in line 103 and 107, it has not been discussed how to determine the value for f_{u,m} or f_{i,n}, it is guessed to be that f_{u,m} and f_{i,n} are binary values that indicates presence or lack of presence of  an attribute for an item or user in certain datasets, but has not been specified.   -in results section, in order to verify effect of metadata incorporation, it is useful to plot the GDMF recommendation beside mGDMF and DCPF in Figure 2. This way we can see how much recommending sparsely rated item is resulted by incorporation of metadata attributes to the model.   Clarity: Paper has been motivated well and general concepts and background have been explained well, just a few minor points: - Most of “deviation”s in supplementary material section should be changed to “derivation”                      - on line 110 to 114, it is discussed about necessity of introducing auxiliary variable, and reason mentioned to be not attaining positive correlation between latent variables in Markov chain. It will be useful for reader if it is explained why such thing happens and how auxiliary variable can solve this problem.   - adding graphical diagram (plate model) to the paper makes the article an easier read  Originality: As mentioned in summary the main contribution of this paper is to model incorporation of users and items attributes which have categorical  nature as an importance weight which is sampled from gamma distribution and it will affect the item or user latent variable.   Authors have cited many of recent works, and  have done an in depth literature review also could distinguish their contribution to what has been done so far clearly.    Significance:  As discussed, in Quality section, it is useful in figure 2 to have another subplot for GDMF recommendation to verify the significance of mGDMF.  Also I would like to mention that dynamical Poisson factorization model in [31] is close to this dynamical Poisson factorization system and code is available on github that could have been used as one of baseline methods.


HOGWILD!-Gibbs Can Be PanAccurate
Asynchronous Gibbs sampling has been recently shown to be fast-mixing and an accurate method for estimating probabilities of events on a small number of variables of a graphical model satisfying Dobrushin's condition [DSOR16]. We investigate whether it can be used to accurately estimate expectations of functions of all the variables of the model. Under the same condition, we show that the synchronous (sequential) and asynchronous Gibbs samplers can be coupled so that the expected Hamming distance between their (multivariate) samples remains bounded by O(τ log n), where n is the number of variables in the graphical model, and τ is a measure of the asynchronicity. A similar bound holds for any constant power of the Hamming distance. Hence, the expectation of any function that is Lipschitz with respect to a power of the Hamming distance, can be estimated with a bias that grows logarithmically in n. Going beyond Lipschitz functions, we consider the bias arising from asynchronicity in estimating the expectation of polynomial functions of all variables in the model. Using recent concentration of measure results [DDK17, GLP17, GSS18], we show that the bias introduced by the asynchronicity is of smaller order than the standard deviation of the function value already present in the true model. We perform experiments on a multiprocessor machine to empirically illustrate our theoretical findings.

Author-feedback response: Author response did clarify many of my concerns. Thus, I increased my rating. But please make sure to clarify"synchronous Gibbs sampling" (by either renaming to traditional "sequential" or properly defining).  -----------------------------------------------------------------------------------------------------------  This paper attempts to analyze asynchronous Gibbs sampling for models satisfying Dobrushin’s condition. The paper builds upon previous seminal work by Chris De Sa et al. (2016). Informally, the key idea behind all the results in the paper is to try to show that sequential and asynchronous chains are similar. In particular, they begin by showing that expected Hamming distance between an asynchronous Gibbs chain and its sequential version starting from the state is small at any time. Using this they could bound the bias in estimating the expected value of polynomials of the random variables. Their bounds improved upon previous results. Finally, some empirical studies on simple test cases were provided. Comments: 1. The writing and clarity of paper can be improved. The paper should be scanned for consistency and confusing order of symbols. The main confusion arises from nomenclature used. Please correct me if I am wrong, does synchronous sampling mean sequential sampling in this paper? To me synchronous sampling means sampling all variables together and updating the state and this strategy is known to not work for many cases (one example below). I think that in the paper, this is not the case, rather it refers to sequential sampling and my review is based on this assumption. Also, in Theorem 1 (borrowed from Chris De Sa et al. 2016), on one hand, its labeled as “Mixing Time of Synchronous Gibbs Sampling” but on the other hand, theorem statement says “mixing time of HOGWILD!-Gibbs”. Please fix/clarify this. 2. With the assumptions of asynchronicity mentioned in the paper, consider a trivial 2D Ising model (this example can be generalized to the Ising model on an n-dimensional torus). I believe the following read-write sequence …, R_1, R_2, W_1, W_2, R1, R2, W2, W1 … is possible. To elaborate, in this case, all processors read the current state and then both writes are completed before next reads. But, we know the stationary distribution of this sampler in closed form and it’s different from the target Ising distribution. (c.f. Neumann, A.U., and Derrida, B., 1988. Finite-size scaling study of dynamical phase transitions in two-dimensional models: Ferromagnet, symmetric and non-symmetric spin glasses. Journal de Physique, 49(10), pp.1647-1656 and Zaheer, M., Wick, M., Tristan, J.B., Smola, A. and Steele, G., 2016, May. Exponential stochastic cellular automata for massively parallel inference. In Artificial Intelligence and Statistics (pp. 966-975).) Can you please explain what assumption of your result is broken by this example that I am missing? 3. The experiments were basic, but verifying the claim. The experiments can be made more interesting by analyzing corner cases (like the example above) or testing limits of asynchronicity etc.  Overall, the paper takes a step in our understanding of asynchronous Gibbs sampling. The proof strategies are mostly standard, but improving the writing/presentation and providing extensive empirical evaluations would strengthen the paper a lot, making it a nice contribution to the field.  Minor Comments/Typos: 1. Apologies for nit-picking: In the introduction section, in my humble opinion the contribution section is too long (It’s ok for a journal paper, but quite a stretch for an 8 page NIPS paper). My personal preference is to limit the contributions in the introduction to major points at a high level. The details about how the obtained results are an improvement over previous work can be part of the main text. 2. A lot of space in the experiment section can be saved. This can allow you to write more details about the proof techniques in the main paper. 3. Line 141: ... condition is alpha < 1. <=> ... condition if alpha < 1. 4. Eq(10)-(11) in appendix: Placement of \alpha is confusing wrt log.


Cooperative neural networks (CoNN): Exploiting prior independence structure for improved classification
We propose a new approach, called cooperative neural networks (CoNN), which uses a set of cooperatively trained neural networks to capture latent representations that exploit prior given independence structure. The model is more flexible than traditional graphical models based on exponential family distributions, but incorporates more domain specific prior structure than traditional deep networks or variational autoencoders. The framework is very general and can be used to exploit the independence structure of any graphical model. We illustrate the technique by showing that we can transfer the independence structure of the popular Latent Dirichlet Allocation (LDA) model to a cooperative neural network, CoNNsLDA. Empirical evaluation of CoNN-sLDA on supervised text classification tasks demonstrates that the theoretical advantages of prior independence structure can be realized in practice -we demonstrate a 23% reduction in error on the challenging MultiSent data set compared to state-of-the-art.

This paper presents a novel approach for combining neural networks with probabilistic graphical models such as LDA through learning Hilbert space embeddings. The idea is to incorporate domain specific prior structure into the more flexible NN-based learning. There is a growing body of literature in this area, and this work introduces some interesting ideas in this domain.   Main strengths:  - Well motivated work  - Interesting new modeling approach, joint constraint networks - Experimental comparisons with other competing approaches  Main weaknesses: - Description of model derivations are vague and grandiose in some places (see below for examples)  - Experimental results are not too impressive, and  some of the strengths of the proposed model, such as ability to inject prior domain structure and need for reduced training data are not demonstrated effectively   The modeling approach is very well motivated with the promise of combining best of both worlds of graphical models and deep learning. Specifically, the need to adapt to limited domain learning problems and to generalize over the vocabulary is certainly important. However, the effectiveness of your model along these dimensions are not effectively demonstrated in the experimental work. For example, it is not clear what domain prior structure is injected into the model, other than selecting the size of Hilbert space dimension. Similarly, the need for reduced training data can be shown by plotting learning curves. I recognize that this is a general missing trend in this area of research, but it would be worth considering some of these additional experiments. Also, given that the results on 20 NG are so close, you might consider also testing your approach on a few other text classification corpora, such as RCV1, and others.   Here are some parts of the model description that are vague and require more clarifications or support of evidence:   127-128: although their actual form is complex, maintaining this complete form is unnecessary because it will be different after the embedding anyway.  130: Here, f_1 and f_2 denote the complex dependency structure of equations (5) and (6), respectively 144-146: ..... in order to classify documents accurately naturally causes the embeddings to be injective exactly where they need to be.  169-170: many of the properties and intuitions behind the original LDA model are preserved with the embedded model (this claim needs to be backed by scientific experiments)   Typographical/grammatical errors:  21: constraint  114: corresponding probability density


A Likelihood-Free Inference Framework for Population Genetic Data using Exchangeable Neural Networks
An explosion of high-throughput DNA sequencing in the past decade has led to a surge of interest in population-scale inference with whole-genome data. Recent work in population genetics has centered on designing inference methods for relatively simple model classes, and few scalable general-purpose inference techniques exist for more realistic, complex models. To achieve this, two inferential challenges need to be addressed: (1) population data are exchangeable, calling for methods that efficiently exploit the symmetries of the data, and (2) computing likelihoods is intractable as it requires integrating over a set of correlated, extremely high-dimensional latent variables. These challenges are traditionally tackled by likelihood-free methods that use scientific simulators to generate datasets and reduce them to hand-designed, permutation-invariant summary statistics, often leading to inaccurate inference. In this work, we develop an exchangeable neural network that performs summary statistic-free, likelihood-free inference. Our framework can be applied in a black-box fashion across a variety of simulation-based tasks, both within and outside biology. We demonstrate the power of our approach on the recombination hotspot testing problem, outperforming the state-of-the-art.

In population genetics, a common problem is the inference of evolutionary events and parameters given some observed DNA sequencing data. This is typically done by leveraging genetic simulators based on the coalescence model. For example, in approximate Bayesian Computation (ABC) one can infer the posterior distribution of the evolutionary parameter of interest by (i) drawing its value from a prior, (ii) generate population genetic data from a simulator using the drawn parameter value and (iii) weight the importance of the sample based on the similarity between the simulated and the real data. Similarity is typically defined in terms of a pre-chosen set of summary statistics.  The authors propose an inference strategy that does not rely on any choice of summary statistics. Briefly, the authors propose to use a neural network to invert the generative process underlying the coalescent simulator. Specifically, one can use steps (i) and (ii) as in ABC to generate multiple pairs (parameter value, population genetic data), and then train a neural network that can predict the parameter value given the genetic data. After training, a posterior can be computed by forward passing the real data through the network. As training data is only based on simulators, the authors propose a simulation-on-the-fly method where each training point is generated afresh. Finally, the authors consider an exchangeable neural network architecture to account for the exchangeability of individual genetic sequences in the population (in the case of unrelated individuals).  In an application to the recombination hotspot testing problem, the authors demonstrate the advantages of employing a “simulation-on-the-fly” strategy and an exchangeable architecture. Finally, they show that their approach competes with a state-of-the-art method for recombination hotspot testing.   The method introduced in this paper is novel and interesting and can in principle be extended to tackle different problems in population genetics. However, I have some points I would like to bring up:  1) The authors say that the neural network is used to learn a posterior of the evolutionary parameters. While this is not wrong, I think it may be confusing. If my understanding is correct, I think the neural network is mostly used to invert the generative process underlying the coalescent simulator. It is trained using simulated data (where the groundtruth value of the parameter to infer is known) while after training, it can be used to “invert the real data into a posterior on the parameter of interest”.  2) The authors should make more explicit which is the loss of the exchangeable neural network. My understanding is that in most setting the evolutionary parameters can take a finite number of discrete values and that in this case, a cross-entropy loss would be used. Is this correct? Also, what do the authors recommend if the parameter to infer is continuous?  3) In the recombination hotspot testing problem, the neural network is trained to classify 20-SNP windows in positive/negative cases of recombination hotspots. While the window size may be fine for this inference problem, this choice could be suboptimal when inferring other parameters. Of course, one could increase the window size, however, the number of weights in the dense layers would also increase (if the same neural network architecture is the same). Do the authors have any comment on this?  Minor: - Related to point 1, in Section 3.1, I think that the authors should remind to the reader that there are observed genetic data and that the goal is to infer the posterior of some evolutionary parameter given the observed data. - Which test statistic is used to obtain the ROC curves in in figure 4 for the two methods?  I raised my score from 7 to 8 after the reading the response from the authors.


Unsupervised Learning of View-invariant Action Representations
The recent success in human action recognition with deep learning methods mostly adopt the supervised learning paradigm, which requires significant amount of manually labeled data to achieve good performance. However, label collection is an expensive and time-consuming process. In this work, we propose an unsupervised learning framework, which exploits unlabeled data to learn video representations. Different from previous works in video representation learning, our unsupervised learning task is to predict 3D motion in multiple target views using video representation from a source view. By learning to extrapolate cross-view motions, the representation can capture view-invariant motion dynamics which is discriminative for the action. In addition, we propose a view-adversarial training method to enhance learning of view-invariant features. We demonstrate the effectiveness of the learned representations for action recognition on multiple datasets.

The paper presents a method for learning view-invariant action representations for flow. The method consists of an encoder (convolutional and bi-directional convolutional LSTMs), which learns feature representations for various views up to T=6 frames, and three additional components: (a) a cross-view decoder, which learns to predict motions for multiple views (different than the ones of the encoder), (b) a reconstruction decoder, which reconstructs the input flow, and (c) a view classifier, which predicts from which view the encoded representation comes and operates under an adversarial setting, where the encoder tries to generate view-invariant representations while the view classifier tries to distinguish their view.   The paper is well-written and easy to follow. It tackles the difficult problem of unsupervised learning of view-invariant representations for flow.   The choice of the architecture is interesting. Nevertheless, the reasons for using the view classifier branch are not clear. Given that the goal is to learn view-invariant representations, the view classifier operates against this goal. This is also in line with the small value of the weight of the decoder beta (line 200 for equation 1), which is 0.05. This shows that the view classifier barely contributes to the final loss. Moreover, the ablation study of Table 1 shows that this branch barely changes the performance (flow input/output for cross view). It would be interesting to know if the authors have an explanation for this choice.   The method seems clean but complicated to implement and run. It would be interesting to see some run times, especially to examine the overload of the view classifier.   The ablation study of Table 1 is performed using the Lxview. It would be beneficial to see the total loss of equation 1 (with and w/o some components).  Moreover, it would be interesting to see how the evolution of the classification accuracy also by removing the reconstruction decoder in Table 2.    In the whole experimental section the authors use one modality (RGB, depth or flow) as input and output. It would be interesting to see if the combination of the modalities could also help, at least as an ablation study (Table 1) or even in Tables 2 and 3.   In the ablation study of Table 1, the authors evaluate the loss with RGB, depth and flow inputs for flow outputs. The cross-view decoder, however, takes as additional input a depth map for a view. Therefore, the columns of the table should display that the input is not one single modality but it includes depth in all cases. The same holds for Tables 2 and 3.   For the cross-view decoder, in line 152, the authors state that they deploy multiple view decoders. Is the number of these decoders fixed, eg. 5 (line 207) or do the authors allow the network to learn various views even if they do not exist at test time?   For action recognition, training from scratch (encoder + action classifier) performs significantly worse than all other cases, where the encoder is learnt using the proposed unsupervised setting. I believe that here another baseline is missing. The authors could have used pre-trained models on  ImageNet and/or UCF-101 and/or Kinetics to initialize the encoder and then fine-tune it. It would have been a stronger baseline.   In the action recognition Section 3.2, the architecture of the action classifier does not seem complete. The authors use only one fully-connected layer for as a classifier. It would be interesting to understand this choice and also to see some results with something more complete, eg. at least another one fully-connected layer.   It would be great if the authors could explain how they ended up in the values for the weights a=0.5 and beta=0.05 (lines 200) for equation (1). Did they use cross-validation?   The results in Figure 2 look nice. It would be interesting to see also some failure cases and some discussion for them.   Some choices of architectural parameters are not really justified. For instance, it is not clear how the authors chose k=64 (line 128), T=6 (line 135).   In Table 4, for the Northwestern-UCLA Multiview Action 3D dataset recent works typically report results in all split combinations (training on two splits and evaluating on the third one) and also include their average.  In Table 3, for NTU RGB+D the average is also missing.   The authors have missed some state of the art citations in Tables 3, 4, and 5. For instance: [42],  - Liu, Mengyuan, Hong Liu, and Chen Chen. "Enhanced skeleton visualization for view invariant human action recognition." In Pattern Recognition 2017 - Baradel, Fabien, Christian Wolf, and Julien Mille. "Human action recognition: Pose-based attention draws focus to hands."  In ICCV Workshop on Hands in Action 2017. - Baradel, Fabien, Christian Wolf, Julien Mille, and Graham W. Taylor. "Glimpse clouds: Human activity recognition from unstructured feature points." In CVPR 2018. - Rahmani, Hossein, and Ajmal Mian. "Learning a non-linear knowledge transfer model for cross-view action recognition." In CVPR 2015. - Gupta, Ankur, Julieta Martinez, James J. Little, and Robert J. Woodham. "3D pose from motion for cross-view action recognition via non-linear circulant temporal encoding." In CVPR 2014.  --after feedback The authors addressed many of the issues raised in the reviews. I am still not convinced about the Lcls: even though its value is sometimes higher than the other losses (without any explanation), using it does not seem so crucial (Table 1 and Table 2). However, the idea of casting the problem as a view-adversarial training is indeed interesting. Including results with pre-training on ImgNet and potentially on other datasets (eg. Kinetics) is important and the rebuttal shows that pre-training on ImgNet is inferior to fix. The authors should also include all related works and results.