The Reversible Residual Network: Backpropagation Without Storing Activations
Deep residual networks (ResNets) have significantly pushed forward the state-ofthe-art on image classification, increasing in performance as networks grow both deeper and wider. However, memory consumption becomes a bottleneck, as one needs to store the activations in order to calculate gradients using backpropagation. We present the Reversible Residual Network (RevNet), a variant of ResNets where each layer's activations can be reconstructed exactly from the next layer's. Therefore, the activations for most layers need not be stored in memory during backpropagation. We demonstrate the effectiveness of RevNets on CIFAR-10, CIFAR-100, and ImageNet, establishing nearly identical classification accuracy to equally-sized ResNets, even though the activation storage requirements are independent of depth.

The paper introduces a reversible block for residual networks that has the benefit of not needing to store all of the forward pass activations for the backward pass. This enables training of larger models or minibatches, as the memory footprint during training is smaller.
As the memory of GPUs is often a bottleneck when training very large models, the paper is a welcome addition to the line of literature on decreasing this footprint. By training on both CIFAR and ImageNet, the experiments focus on two widely used benchmarks. The experiments seem solid and achieve compelling results. The authors also confirm empirically that numerical instabilities are not a significant problem, which is good.
To make the paper even stronger, I would have been interested in seeing even more experiments. It is good that the model is tested both on CIFAR (10 and 100) and ImageNet, but for example SVHN would have been a fairly "cheap" addition to the set of standard benchmarks. The performance on RNNs, that the authors also discuss, would be very interesting to see.
In addition, the main concern I have with using the model in practice is the potential drawback of not discarding information (hence the reversibility) - I'm not sure exactly what form that discussion would take, but a slightly deeper discussion on that issue would be helpful. The second paper on NICE might give some ideas for how to do that.
A minor issue is also that it is somewhat unclear to the reader how large a typical memory footprint of the activations is compared to the footprint of the weights for a particular minibatch size in the resnets. This obviously depends on many factors, but it would be really helpful (for instance in the appendix) to make some back-of-the-envelope calculations that would give an indication of e.g. how much deeper models or how much larger minibatches one could train, given an optimized code, for typical model sizes.
The paper is generally well written, although some minor typos are still present (p1: "simply by widening (increasing the convolutions' filter count) fewer layer" should probably read "...layers", p2: "...derivative defines the the effect..." has one "the" too much, p2: "change to v_i, taking into account the indirect effects through the descendants of v_k" should probably read "... descendants of v_i") which will be easy to fix by carefully rereading the paper a few times.
The drawbacks are however fairly minor, and the strength of the paper is that the idea is fairly simple, has been shown with sufficient empirical results to work, and is well presented. In my view, this is an idea that would interest the NIPS audience.


Near-Optimal Edge Evaluation in Explicit Generalized Binomial Graphs
Robotic motion-planning problems, such as a UAV flying fast in a partially-known environment or a robot arm moving around cluttered objects, require finding collision-free paths quickly. Typically, this is solved by constructing a graph, where vertices represent robot configurations and edges represent potentially valid movements of the robot between these configurations. The main computational bottlenecks are expensive edge evaluations to check for collisions. State of the art planning methods do not reason about the optimal sequence of edges to evaluate in order to find a collision free path quickly. In this paper, we do so by drawing a novel equivalence between motion planning and the Bayesian active learning paradigm of decision region determination (DRD). Unfortunately, a straight application of existing methods requires computation exponential in the number of edges in a graph. We present BISECT, an efficient and near-optimal algorithm to solve the DRD problem when edges are independent Bernoulli random variables. By leveraging this property, we are able to significantly reduce computational complexity from exponential to linear in the number of edges. We show that BISECT outperforms several state of the art algorithms on a spectrum of planning problems for mobile robots, manipulators, and real flight data collected from a full scale helicopter. Open-source code and details can be found here:
https://github.com/sanjibac/matlab_learning_collision_checking

Overview and Summary
This paper presents a method for motion planning where the cost of evaluating transitions between robot configurations is high.  The problem is formulated as a graph-search algorithm, where the order of graph expansion has a large impact on the performance of the algorithm due to the edge expansion cost.  The paper uses ideas from optimal test selection in order to derive the resulting algorithm.  The algorithm is tested on a number of synthetic datasets, a simulator, and a real-world helicopter planning problem.
Detailed Comments
This paper extended work within the well-studied domain of robotic motion planning, extending and combining prior work in the area to construct a new algorithm.  The idea of using optimal test selection to address the graph expansion ordering is a natural, and reasonable approach.  The originality of this paper is moderate, and it does address a problem area of interest to many researchers.
There are some issues with clarity in the paper.  Specifically, starting in section 2, the term P(o) is first defined as an outcome vector, but then in equation (1) is used as a set to be drawn from (probabilistically?).  Next, and more significantly, lines 129-130 define the set of subregions, consisting of a single consistent subregion and a number of disjoint subregions that are each invalid.  The graph as a whole consists of nodes as subregions, and edge costs that are the product of the probabilities of each subregion.  I can see two issues with this formulation that are not made clear within the context of the paper:
- Given that the invalid subregions will expand with each test performed, the graph as a whole will have to change as testing progresses.
- Because every edge contains at least one invalid subregion, every edge cost must contain at least one test where P(S_i) = 0, and therefore w must also equal 0.
Due to this, either the problem formulation, or the clarity of the description must be changed.
Also in terms of clarity, the index i is used on equations 3 and 5 on both the left and right hands of equations with different meanings.
Finally, within equation 5, the last term of the top equation does not equate to the last term of the bottom equation.
Within the results, the text states that there are 13 datasets, but the results table lists 14.  Line 215 states that there are 8 tests with exclusively best performance, but the table lists only 7.  The claim on line 218 of a factor of 20 improvement is not possible to verify due to the normalization of all results.  Finally, and most importantly, the results are not compared against any of the other motion planning algorithms discussed in the related work, only against self-constructed heuristics used within the same general framework.  We have no idea how these algorithms compare to the state of the art.
Post-feedback:
Thanks to the authors for their explanation, additional material, and commitments to update the paper.  In light of this interaction, I have modified my score.


Targeting EEG/LFP Synchrony with Neural Nets
We consider the analysis of Electroencephalography (EEG) and Local Field Potential (LFP) datasets, which are "big" in terms of the size of recorded data but rarely have sufficient labels required to train complex models (e.g., conventional deep learning methods). Furthermore, in many scientific applications, the goal is to be able to understand the underlying features related to the classification, which prohibits the blind application of deep networks. This motivates the development of a new model based on parameterized convolutional filters guided by previous neuroscience research; the filters learn relevant frequency bands while targeting synchrony, which are frequency-specific power and phase correlations between electrodes. This results in a highly expressive convolutional neural network with only a few hundred parameters, applicable to smaller datasets. The proposed approach is demonstrated to yield competitive (often state-of-the-art) predictive performance during our empirical tests while yielding interpretable features. Furthermore, a Gaussian process adapter is developed to combine analysis over distinct electrode layouts, allowing the joint processing of multiple datasets to address overfitting and improve generalizability. Finally, it is demonstrated that the proposed framework effectively tracks neural dynamics on children in a clinical trial on Autism Spectrum Disorder.

The manuscript introduces the novel CNN named SyncNet, which was designed to detect synchrony/spectral coherence in brain electrophysiological signals. SyncNet uses parameterized convolutional filters. To overcome common issues in EEG (varying setup configuration, correlated channels, ...) a Gaussian Process Adaptor is used to transform EEG data into a pseudo input space. The design of SyncNet is chosen to allow interpretation of learned featruers. The method was applied to several publicly available datasets and its performance compared to state of the art algorithms. Results suggest that SyncNet yields competitive results.
The topic of the paper is timely and important and the presented approach very interesting. The paper is well organized and mostly clear, and the reviewer agrees with the authors reasoning. There are, however, some issues that need addressing:
- Are the computed SyncNet results significantly better than the results obtained with state of the art methods?
- One key feature of SynchNet is interpretability of features. Are the results summarized in Fig.4 meaningful?
- LFPs are only mentioned in the title and in lines 319-330. Please provide more details on LFP data and analysis or remove LFPs from the manuscript! Currently, LFP does not contribute to enhancing clarity of the paper.


Probabilistic Rule Realization and Selection
Abstraction and realization are bilateral processes that are key in deriving intelligence and creativity. In many domains, the two processes are approached through rules: high-level principles that reveal invariances within similar yet diverse examples. Under a probabilistic setting for discrete input spaces, we focus on the rule realization problem which generates input sample distributions that follow the given rules. More ambitiously, we go beyond a mechanical realization that takes whatever is given, but instead ask for proactively selecting reasonable rules to realize. This goal is demanding in practice, since the initial rule set may not always be consistent and thus intelligent compromises are needed. We formulate both rule realization and selection as two strongly connected components within a single and symmetric bi-convex problem, and derive an efficient algorithm that works at large scale. Taking music compositional rules as the main example throughout the paper, we demonstrate our model's efficiency in not only music realization (composition) but also music interpretation and understanding (analysis).

The paper tackles rules realisation and selection, given a specific set of given rules that derives from an abstraction process, in the specific case of music compositional rules. The novelty of the paper resides in a new bi-convex optimisation model for rules selection and rule realisation. The challenge posed by rules realisation is due to the high dimensionality of the sample space. The challenge posed by rule selection appears in the case when all rules cannot be simultaneously realised, or not in their entirety. In this case one has to choose which rules to respect and in which amount of components. The strengths of the paper are:
- the goal and challenge of the model are very clearly stated,
- the paper is extremely well written with all notations defined and intuitive explanations of the theoretical results. I especially liked the interpretation of lines 285 - 296.
- experimental results support the claim of the proposed innovations, as the model (i) behaves as expected when varying the hyper parameters (ii) reduces computation time.
Questions / comments :
- line 118 : can the author explain why the solution is expected to be "unique" on p thanks to group penalty ? If there is no group penalty (\lambda_p = 0) I would expect the solution to be unique, which is also what the authors state in line 131 "in setting \lambda_p sufficiently small, we secure a unique solution for the rule realisation part". The two statements seem unconsistent.
- line 148 : Theorem 1, claim (2) : am I right to understand that all rules are selected because for all rules the L2 norm converges to a non-zero value ?
- Group De-overlap : am I right to understand that this is not an additional dimensionality reduction but only an interpretation of the grouping presented in Section (3), while screening is an additional step to further reduce dimensionality ?
- line 304 : what do the authors mean by "failures in group selection" ? Does it mean that no rule is entirely selected ?
- line 305 : am I right to understand that a simple lasso on the number of rules is impossible because it makes no sense to weight the rules as the selection process is a 0/1 process ?
- I am not familiar with the literature in automatic music composition, and I understand that there is no "ground-truth" solution as creativity can be a desired behaviour. I like the fact that the author mention the gain in computational efficiency.  If that exists / is possible in the time of rebuttal I would like to see a baseline model to compare the behaviour in terms of selection / realisation of the proposed model.
- Can the author think of any other possible domain to apply their method ?


Toward Multimodal Image-to-Image Translation
Many image-to-image translation problems are ambiguous, as a single input image may correspond to multiple possible outputs. In this work, we aim to model a distribution of possible outputs in a conditional generative modeling setting. The ambiguity of the mapping is distilled in a low-dimensional latent vector, which can be randomly sampled at test time. A generator learns to map the given input, combined with this latent code, to the output. We explicitly encourage the connection between output and the latent code to be invertible. This helps prevent a many-to-one mapping from the latent code to the output during training, also known as the problem of mode collapse, and produces more diverse results. We explore several variants of this approach by employing different training objectives, network architectures, and methods of injecting the latent code. Our proposed method encourages bijective consistency between the latent encoding and output modes. We present a systematic comparison of our method and other variants on both perceptual realism and diversity.

Summary:
The paper enforces a cycle consistency between the latent code and the image output of a generative model that is applied to image-image translation problems. The cycle consistency encourages invertibility of a generated output to the latent code, which addresses the popular issue of mode collapse in generative adversarial networks. The paper extends the pix2pix [1] framework of image-image translation by conditioning the generation of an image not only on an input image but also on a randomly sampled latent code. The low dimensional latent code captures the ambiguous aspects of the output image such as (depending on application) color, texture, etc.  While Isola et al. [1] observe that randomly sampling a latent code alone doesn’t help with mode collapse, the proposed approach applies a cycle consistency loss on both the input-output image pairs and the latent code - output pairs via alternating joint optimization.
Strengths:
* Mode collapse is a commonly occurring issue in generative adversarial networks, and a method to produce more diverse results could be potentially useful to a large community.

* The approach is a natural extension of the cycle consistency idea of [2] to address the issue of mode collapse.

* From the examples in the supplementary material it is clear that the proposed approach generates samples that are both realistic and diverse in the edges -> shoes task, when compared with baseline approaches such as pix2pix, pix2pix+noise, and an ablation of the overall approach, latent cycle GAN (described in 3.3).

* The analyses in Section 4.2 are interesting. The analysis of the latent space reinforces the earlier claim regarding the poor distribution of latent codes when the model is trained using the Image Cycle.

* The paper is well written and easy to understand.

Clarification questions:
* Are there any specific reasons or intuitions behind the decision to use the least squares loss in the experiment 3.1 (Equation (1)) instead of the discriminator loss used in [1]?

* It is unclear whether the table in Figure 3., reports Google maps → Satellite images or the opposite transformation.

* Did the authors ensure that the performance of different methods in the table in Figure 3 are comparable with pix2pix? Specifically, are the ground truth images processed identically as [1]? Please note that [2] states that there are a few differences in preprocessing and hence the performance of methods in [2] are not comparable with [1].

* Have the authors performed human-perception experiments on other datasets such as shoes? As a consequence of their semantic content, these datasets appear to contain more diverse images in terms of colors, textures etc., than aerial photos.

* It is unclear how the diversity score in the Table in Figure 3 is computed. Is it the cosine distance between vectors of neuron activations in a layer computed for pairs of images randomly sampled from the set B?

Final recommendation:
The paper proposes an approach to overcome the popular issue of mode collapse in generative adversarial networks which could be useful to further research into generating diverse samples in conditional generative models.
References:
[1] Isola et al.:	Image-to-image translation with conditional adversarial networks. CVPR, 2017.
[2] Zhu et al.: Unpaired image-to-image translation using cycle-consistent adversarial networks. arXiv preprint 2017.


Online Multiclass Boosting
Recent work has extended the theoretical analysis of boosting algorithms to multiclass problems and to online settings. However, the multiclass extension is in the batch setting and the online extensions only consider binary classification. We fill this gap in the literature by defining, and justifying, a weak learning condition for online multiclass boosting. This condition leads to an optimal boosting algorithm that requires the minimal number of weak learners to achieve a certain accuracy. Additionally, we propose an adaptive algorithm which is near optimal and enjoys an excellent performance on real data due to its adaptive property.

In this paper the authors introduce two learning algorithms dealing with online learning in a multi-class setting. The novelty resides in combining a multi-class boosting framework providing solid theoretical results with an online learning framework. Notable contributions of the paper include: a weak learning condition for multi-class online learning, a general boosting algorithm and a more practical version. I have two main concerns about this paper.
The paper is overall well written, the key ideas are ordered in an easy to follow manner and, as far as I can tell, the theoretical results are sound. However, at times the paper is hard to follow without switching over to the supplementary material. For instance, the link between the weak learning condition in batch learning and the one presented in the paper, is not obvious without reading the supplementary material; Section 3.2 is also hard to follow since the supplementary material is needed in order to grasp key ideas. While I do understand the authors' eagerness to include most of their work in the paper, in my opinion, the main paper should be as close as possible a stand alone paper, which isn't the case for this paper.
My second concern is related to the experimental and (the missing) discussion sections. The experimental results suggest that the proposed methods can achieve better results than other online methods, however their run time is obviously far more important. As such two questions arise : 1) Is there a case where the proposed methods are more interesting to use than existing ones wrt the ratio performance/run time? 2) Is it possible to speed up the proposed methods without significant loss in their performances? A discussion section treating both questions would've been appreciated. I also think that more experimental results are needed in order to evaluate how the proposed methods scale wrt the number of classes and examples.
When introducing the adaptive algorithm the authors propose to discard the weak learning condition and introduce an empirical edge. However the empirical edge takes its values in [-1,1], while the edge in the weak learning condition takes its values in (0,1). In practice, how is a negative edge dealt with? Is the training sample discarded?


Adaptive stimulus selection for optimizing neural population responses
Adaptive stimulus selection methods in neuroscience have primarily focused on maximizing the firing rate of a single recorded neuron. When recording from a population of neurons, it is usually not possible to find a single stimulus that maximizes the firing rates of all neurons. This motivates optimizing an objective function that takes into account the responses of all recorded neurons together. We propose "Adept," an adaptive stimulus selection method that can optimize population objective functions. In simulations, we first confirmed that population objective functions elicited more diverse stimulus responses than single-neuron objective functions. Then, we tested Adept in a closed-loop electrophysiological experiment in which population activity was recorded from macaque V4, a cortical area known for mid-level visual processing. To predict neural responses, we used the outputs of a deep convolutional neural network model as feature embeddings. Natural images chosen by Adept elicited mean neural responses that were 20% larger than those for randomly-chosen natural images, and also evoked a larger diversity of neural responses. Such adaptive stimulus selection methods can facilitate experiments that involve neurons far from the sensory periphery, for which it is often unclear which stimuli to present.

The authors present an algorithm for selecting stimuli in a closed-loop experiment to maximize the variance in firing across a population of neurons (Adept). This is an important step in closed-loop stimulus selection, which have focused on learning the response properties of single neurons. The algorithm uses pre-computed features of the stimuli given by a neural network to predict firing rates to new stimuli. The methods were presented clearly and the stimulus selection is efficient enough to be useful to real experiments.
There are a few limiting points of the Adept method (which may be addressed in future work). The applicability of this algorithm is limited, as mentioned by the authors, in that it depends on having a good set of features to describe the stimulus space. Such features may not always be available outside select areas such as V4. It would be valuable to explore the Adept algorithm's behavior when the features chosen do not provide a good basis for prediction spike rates (perhaps for only a portion of the population - would the algorithm learn less for those cells than random stimulus selection?).
Additionally, the algorithm assumes a limited discrete set of stimuli. Finally, it is not clear how the stimulus selection may be affected by different types of noise (in particular, correlated noise across cells or time) in the responses.


Statistical Cost Sharing
We study the cost sharing problem for cooperative games in situations where the cost function C is not available via oracle queries, but must instead be learned from samples drawn from a distribution, represented as tuples (S, C(S)), for different subsets S of players. We formalize this approach, which we call STATISTICAL COST SHARING, and consider the computation of the core and the Shapley value. Expanding on the work by Balcan et al. [2015], we give precise sample complexity bounds for computing cost shares that satisfy the core property with high probability for any function with a non-empty core. For the Shapley value, which has never been studied in this setting, we show that for submodular cost functions with bounded curvature  it can be approximated from samples from the uniform distribution to a p 1  factor, and that the bound is tight. We then define statistical analogues of the Shapley axioms, and derive a notion of statistical Shapley value and that these can be approximated arbitrarily well from samples from any distribution and for any function.

This paper studies cost sharing in cooperative games. There is a ground set N of players and a cost function C that maps subsets S of N to a real-valued cost C(S). The goal is to split the entire cost of the ground set C(N) among the players in such a way that certain desirable properties are satisfied, such as the “core property” (defined in Section 2.1) or the “Shapley axioms” (defined in Section 2.2). Prior work in the cost sharing literature has designed algorithms computing the cost shares which have oracle access to the underlying cost function C. Recent work by Balcan, Procaccia, and Zick in IJCAI 2015 studies the learning theoretic setting where the algorithm does not have oracle access but receives a sample of subsets labeled by their cost {(S1, C(S1)), (S2, C(S2)), …, (Sm, C(Sm))}. In this statistical cost sharing model, the authors of the present paper design sample efficient and computationally efficient learning algorithms that compute cost shares that (approximately) satisfy various properties with high probability.
From what I understand, in Balcan et al.’s IJCAI 2015 paper, they showed that for some games, it is possible to efficiently compute a “probably approximately stable core” (Definition 3) using a polynomial number of samples and then in the arXiv paper published a year later they showed that this is true for all games with a non-empty core. This matches Theorem 1 of the present paper. The present paper builds on Balcan et al.’s work by defining two other notions of an approximate core (Definition 3). They prove various upper and lower bounds on sample complexity and computability for these other notions (Theorems 2 and 3).
Finally, the authors study cost shares that approximate the “Shapley value” which is a solution to the cost sharing problem satisfying several desirable properties. They show that if the cost function is a submodular function, then the extent to which the Shapley value can be approximated from samples from the uniform distribution/any bounded product distribution depends on the submodular function’s curvature (with upper and lower bounds). The also show that if the cost function is a coverage function, then it can’t be approximated from samples over the uniform distribution.
Overall, I think this paper makes a solid contribution to an interesting and applicable subject area. There are a few obvious ways the theorems could be improved:
- Theorem 2: It’s too bad that the sample complexity depends on the spread of C since it seems reasonable that the spread of C might be comparable to n in many settings. Whether or not the dependence can be removed is especially interesting since Theorem 1 has no dependence on the spread. Do the authors have any intuition about whether or not the dependence on the spread is necessary?
- Theorem 4: Can this theorem be generalized to other distributions? If the authors have any interesting insights into the key technical challenge here, that would be a nice addition to the discussion.
I think the axioms for the data-dependent Shapley value could be better motivated. Specifically, the symmetry and zero element axioms seem a bit unintuitive because there’s no connection to the cost function. To me, a more natural distributional parallel to the original symmetry condition would be something like, “For all i and j, if Pr_{S \sim D}[C(S \cup {i}) = C(S \cup {j})] >= ½, then \phi_i^D = \phi_j^D” (I chose ½ arbitrarily). Also, in my opinion, a more natural distributional parallel for the original zero element condition could be “For all i, if Pr_{S \sim D}[C(S \cup {i}) – C(S) = 0] >= ½, then \phi_i^D = 0.”
Small questions/comments:
- I think Theorem 1 should say “probably approximately stable core.”
- In the proof sketch of Theorem 4, what is C_{A_{sigma < i}}(i)?
=======After author feedback=======
Thanks, it would be great to include examples where a dependence on the spread is necessary. And I see, that's a fair point about how you have limited latitude in crafting alternative characterizations.


Rigorous Dynamics and Consistent Estimation in Arbitrarily Conditioned Linear Systems
We consider the problem of estimating a random vector x from noisy linear measurements y = Ax + w in the setting where parameters θ on the distribution of x and w must be learned in addition to the vector x. This problem arises in a wide range of statistical learning and linear inverse problems. Our main contribution shows that a computationally simple iterative message passing algorithm can provably obtain asymptotically consistent estimates in a certain high-dimensional large system limit (LSL) under very general parametrizations. Importantly, this LSL applies to all right-rotationally random A -a much larger class of matrices than i.i.d. sub-Gaussian matrices to which many past message passing approaches are restricted. In addition, a simple testable condition is provided in which the mean square error (MSE) on the vector x matches the Bayes optimal MSE predicted by the replica method. The proposed algorithm uses a combination of Expectation-Maximization (EM) with a recently-developed Vector Approximate Message Passing (VAMP) technique. We develop an analysis framework that shows that the parameter estimates in each iteration of the algorithm converge to deterministic limits that can be precisely predicted by a simple set of state evolution (SE) equations. The SE equations, which extends those of VAMP without parameter adaptation, depend only on the initial parameter estimates and the statistical properties of the problem and can be used to predict consistency and precisely characterize other performance measures of the method.

The model is p(y | x, \theta_1) p(x | \theta_1) where y | x is a linear system y = A*x + Gaussian noise. We want to infer x and \theta from data. Importantly, p(x | \theta_1) factorizes over elements of x.
The paper takes EM-VAMP from [13], and changes a step for updating \theta_1 (from the EM-VAMP algorithm). As a general comment, I was wondering why the algorithm wasn't called "generalized EM-VAMP" -- if eq (7) is used as "adaptation" (which is preferred according to line 113), then there is no difference and the naming is rather confusing. What is adapted that is *not* adapted in EM-VAMP? Naming and nomenclature aside, and also ignoring that much of the introduction of the paper is a verbatim copy of [13], the paper makes some fundamental contributions:
It provides an asymptotic analysis to EM-VAMP, showing that as the number of latent dimensions in x goes to infinity, the algorithm converges.
Secondly, we can compute the deterministic limit from a set of state evolution equations. The paper's numerical results show that the state evolution equations in eq (24) in the paper predicts the performance (mean squared error of true x vs estimated x^) of EM-VAMP remarkably well per iteration.
The paper then asks, given that the algorithm converges, does it converge to an asymptotically consistent estimate of \theta? If p(x|\theta_1) is identifiable in Gaussian noise, then we can find statistics (essentially, here the "adaptation rule" that generalizes EM-VAMP seems to really come into play) so that convergence of the estimate is a.s. to \theta.
In my humble opinion, the paper provides nice and rigorous analysis to give EM-VAMP a deeper theoretic base.
Postscripts
-----------
Very similar to [a] and [b], the algorithm factorizes the joint into a Gaussian and a non-Gaussian part, and approximates the non-Gaussian part with a contribution (messages) towards the Gaussian part to give a Gaussian belief on x.
[a] Matthias Seeger & Hannes Nickisch, Fast Convergent Algorithms for Expectation Propagation Approximate Bayesian Inference.
[b] Matthias Seeger, Bayesian Inference and Optimal Design for the Sparse Linear Model, JMLR
Could the authors put Figure 1 in a vectorized format? It is hard to read, especially when printed.
small typo, :== in line 508


Bayesian Inference of Individualized Treatment Effects using Multi-task Gaussian Processes
Predicated on the increasing abundance of electronic health records, we investigate the problem of inferring individualized treatment effects using observational data. Stemming from the potential outcomes model, we propose a novel multitask learning framework in which factual and counterfactual outcomes are modeled as the outputs of a function in a vector-valued reproducing kernel Hilbert space (vvRKHS). We develop a nonparametric Bayesian method for learning the treatment effects using a multi-task Gaussian process (GP) with a linear coregionalization kernel as a prior over the vvRKHS. The Bayesian approach allows us to compute individualized measures of confidence in our estimates via pointwise credible intervals, which are crucial for realizing the full potential of precision medicine. The impact of selection bias is alleviated via a risk-based empirical Bayes method for adapting the multi-task GP prior, which jointly minimizes the empirical error in factual outcomes and the uncertainty in (unobserved) counterfactual outcomes. We conduct experiments on observational datasets for an interventional social program applied to premature infants, and a left ventricular assist device applied to cardiac patients wait-listed for a heart transplant. In both experiments, we show that our method significantly outperforms the state-of-the-art.

The authors propose a method of estimating treatment effectiveness T(x) from a vector of patient features x.  Treatment effectiveness is defined as  (health outcome with treatment Yw) - (health outcome without treatment Y(1-w)).  Presumably a health outcome might be something like survival time.  If a patient survives 27 months with the treatment and only 9 without then the effectiveness T(x) would be 18 months?
The authors estimate models of "outcome with treatment" and "outcome without treatment" jointly using RKHS kernel approximations on the whole dataset (I think there is a shared kernel).
For a specific patient the effectiveness is based on the actual outcome of the patient which will be based on their features and their treatment condition minus the population model for the features of the opposite or counterfactual treatment condition.
The population model is represented by an RKHS approximation which generates Bayesian posterior confidence bounds allowing the algorithm to integrate over uncertainty in the treatment effectiveness.  I like that the authors call this out in line 139: integrating over the uncertainty in the counterfactual condition improves their results. They employ an empirical Bayes framework in which they optimize for hyperparameters that control distributions over health outcomes. They use a loss function that tries to maximize accuracy of the actual outcome observed for a patient with specific features and minimizing the variance of the counterfactual ...
This seemed a bit counterintuitive to me - wouldn't you want to maximize the variance of the counterfactual in order to be consistent with the principle of least commitment / maximum entropy? Don't make conclusions that are stronger than the data supports??
They end up using a mixture of two kernels to provide additional flexibility in modeling.
The algorithm does a matrix inversion which I think is common in RKHS style approximation which may limit its scalability somewhat (also true of Gaussian Process based approaches). The example they show has only 1000 patients which is very small for an EHR dataset generally which can have millions of patients.
The goal of the authors is to be able to handle situations with strong selection bias (that is, the treatment is correlated with the patient features so we don't get an unbiased selection of treatments for any given condition).  They end up having to filter the datasets they work with to create stronger selection bias.  It is not clear whether this was a necessary step for them to demonstrate advantage over the other algorithms they benchmark against.
For their filtered dataset they show metrics with around 1/3 the error rate of competing state of the art methods. They attribute the gains to the ability of their algorithm to compensate for selection bias by integrating over uncertainty in the counterfactual???
The paper has a lot of technical jargon which is not  obviously necessary to explaining the intuitions clearly. Do we really need the details of the representer theorem in the paper? I would prefer using the space to describe intuitions on why it works more clearly.
line 27: "glints" is used as a verb in an awkward way
I liked Figure 1, but it took me a while to understand it. It might be worth explicitly calling out the roles of the 3 subfigures in the caption.  If you used fewer data points it might be more obvious that the realized factual outcomes and the counterfatual distributions are supposed to align ... maybe it would make sense to stack these graphs vertically so they share the same x axis?
line 53: "helps creating" is an awkward expression
lines 251 ... it would be nice if the authors could describe a bit more about the competing bench mark methods and why they perform the way they do? There is a short related work section but it doesn't use any of the names of the algorithms.  I don't know if these are the same as the comparison benchmarks or not.
Would the ordering be the same with a dataset that has low selection bias?


A KL-LUCB Bandit Algorithm for Large-Scale Crowdsourcing 1 Multi-Armed Bandits for Large-Scale Crowdsourcing
This paper focuses on best-arm identification in multi-armed bandits with bounded rewards. We develop an algorithm that is a fusion of lil-UCB and KL-LUCB, offering the best qualities of the two algorithms in one method. This is achieved by proving a novel anytime confidence bound for the mean of bounded distributions, which is the analogue of the LIL-type bounds recently developed for sub-Gaussian distributions. We corroborate our theoretical results with numerical experiments based on the New Yorker Cartoon Caption Contest.

The paper proposes a novel PAC algorithm for the multi-armed bandit problem that combines ideas from KL-LUCB and lil-UCB. For the proposed algorithm (called lil-KLUCB) the authors show a sample complexity upper bound that improves on the corresponding result of both its predecessors. The core of this result is a novel concentration bound which uses the Chernoff-information - similiarly as in case of KL-LUCB, but this new approach results in a much tighter bound. Finally, the paper presents some experimental results demonstrating that the theoretical improvement has practical implications as well.
The problem I have found with the paper is that the proof of Theorem 1 contains errors that make it completely confusing, and thereby the correctness of the results remains unclear. What is exactly A_k and t_j? Their current definition do not makes sense: why should t in the definition of A_k run over the whole set of natural numbers? And why should t_k be bigger than, but still proportional to 2^k? And which part of the proof corresponds to the first term on the RHS of the equation below line 135, and which part to the rest of the terms?
Additionally, the proof sketch of Theorem 1 is is rather messy too (unfortunately, I was not able to check the corresponding part in the supplementary material):
- In line 180, what is U(f_t(\epsilon))? It cannot be deduced from the definition of U(t,\epsilon).
- Still in line 180, the equation in the second half of the line does not seem right either: the first term depends on gamma, whereas the second term depends on \epsilon. What exactly is going on there?
- In line 181 you are referring to some result from Theorem 1 - however, you are proving Theorem 1. (Presumably this reference is to the theorem in the supplementary.)
Nevertheless, the results are nice and plausible, and imprive significantly on previous bounds (although they are still somewhat worse than existing lower bounds); in fact, the novel concentration bound could be of interest on its own. Therefore, I would recommend acceptance given the issues discussed above could be taken care of in a convincing fashion.
Further remarks and questions:
- There does not seem to be a definition for D(z,x).
- How do you calculate L_i(t,\epsilon) and U_i(t,\epsilon)?
- In line 126: it seems that 1-\delta should be 1-2\delta.
- Line 142: the left bracket is missing in the term "D\mu,\mu)".
- In the line before line 142: "exp(\lambda S_t)" should be "exp(\lambda S_{t_j})".
- In the line below 135: S_t is not yet defined.
- In lines 194 and 195 you are referring to Fig. 5, whereas you only have Fig. 1.
- How close do you think the bounds are to the actual optimum of the problem?


Masked Autoregressive Flow for Density Estimation
Autoregressive models are among the best performing neural density estimators. We describe an approach for increasing the flexibility of an autoregressive model, based on modelling the random numbers that the model uses internally when generating data. By constructing a stack of autoregressive models, each modelling the random numbers of the next model in the stack, we obtain a type of normalizing flow suitable for density estimation, which we call Masked Autoregressive Flow. This type of flow is closely related to Inverse Autoregressive Flow and is a generalization of Real NVP. Masked Autoregressive Flow achieves state-of-the-art performance in a range of general-purpose density estimation tasks.

This paper is generally well written and I enjoy reading it. It introduces an expressive density model called masked autoregressive flow (MAF) that stacks multiple MADE layers to form a normalizing flow. Although it seems a bit incremental since the techniques involved have been studied in IAF and MADE, this paper does a good job elaborating on different types of generative modeling and providing guidelines for their use cases. It also makes a connection between MAF and IAF. Only a few comments/questions below:
* It'd be helpful to motivate a bit more on the advantage of density models. What are the applications or downstream tasks that make density models more suitable than their alternatives such as VAEs or GANs? For example, does the proposed density model admit efficient algorithms for marginalization or other inference tasks?
* When applying multiple transformations in the normalizing flow, is it implied that the conditionals need not be Gaussian as in eq. 2 because the density is computed according to the density of u? When u is transformed from some lower-level random vectors, it could very likely be non-Gaussian. Do I understand correctly?
* Does the base density of an MAF (line 225) refer to the density of u in eq. 3 or the density of p(x_i|x_{1:i-1}) in eq. 1? I assumed it's the former.
* When comparing the results from MAF (5), MAF (10) and MAF MoG (5) on the data sets POWER and MNIST, MAF MoG outperforms the other two. I have the impression that if using multiple normalizing flow layers where each layer has multiple non-linear MADE layers, MAF should be universal without MoG. What's the authors' opinion on this?


Adaptive Bayesian Sampling with Monte Carlo EM
We present a novel technique for learning the mass matrices in samplers obtained from discretized dynamics that preserve some energy function. Existing adaptive samplers use Riemannian preconditioning techniques, where the mass matrices are functions of the parameters being sampled. This leads to significant complexities in the energy reformulations and resultant dynamics, often leading to implicit systems of equations and requiring inversion of high-dimensional matrices in the leapfrog steps. Our approach provides a simpler alternative, by using existing dynamics in the sampling step of a Monte Carlo EM framework, and learning the mass matrices in the M step with a novel online technique. We also propose a way to adaptively set the number of samples gathered in the E step, using sampling error estimates from the leapfrog dynamics. Along with a novel stochastic sampler based on Nosé-Poincaré dynamics, we use this framework with standard Hamiltonian Monte Carlo (HMC) as well as newer stochastic algorithms such as SGHMC and SGNHT, and show strong performance on synthetic and real high-dimensional sampling scenarios; we achieve sampling accuracies comparable to Riemannian samplers while being significantly faster.

This paper considers the problem of learning the mass matrix within Hamiltonian type MCMC samplers.  The authors do so by proposing the use of an online EM style algorithm with a Monte Carlo sampling step that adjusts according to the sampling error estimates.
The issue of how to sample efficiently from complex target densities is a very important one and the authors make a very nice contribution in this paper.  It is rather densely written but the topics are well explained with thorough referencing to the existing literature. The idea of making use of EM to estimate the mass matrix in these types of algorithms is very interesting, the proposed algorithms are explained well and appear to perform well in practice.  I therefore only have the following relatively minor comments:
Line 128:  “we perform generalised leapfrog steps” - it seems to me that standard leapfrog integrator is used here, rather than the generalised one?
Algorithm 1:  I find the use of \epsilon in the algorithm description slightly confusing.  Perhaps you could use an alternative notation here to make clear that these integration steps are only the proposed points for the subsequent Metropolis-Hastings step?
Line 328:  typo “we put model the columns”
Section 4.1:  Why was the observed FI used here instead of the expected FI?  I think the expected FI might give better performance.  More generally, this example seems to be a bit of a straw man, given that there isn’t any strong correlation here in the posterior density - a simplified Riemannian MALA sampler would likely outperform any of these HMC methods, which makes the comparison rather moot.  More interesting would be a banana shaped distribution, which is where Riemannian MCMC methods are really a bit more useful.
More generally, it is not clear to me how well the EM versions would work with strong correlations that change over the posterior.  Since it will be converging to a single mass matrix, then I suspect they won’t work well in strongly banana-like cases.  This isn’t explored in the paper at all, which is a shame - I think an example like this would be much more illustrative and informative than the current (trivial) Gaussian example.


Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments
We explore deep reinforcement learning methods for multi-agent domains. We begin by analyzing the difficulty of traditional algorithms in the multi-agent case: Q-learning is challenged by an inherent non-stationarity of the environment, while policy gradient suffers from a variance that increases as the number of agents grows. We then present an adaptation of actor-critic methods that considers action policies of other agents and is able to successfully learn policies that require complex multiagent coordination. Additionally, we introduce a training regimen utilizing an ensemble of policies for each agent that leads to more robust multi-agent policies. We show the strength of our approach compared to existing methods in cooperative as well as competitive scenarios, where agent populations are able to discover various physical and informational coordination strategies.

Summary
-----------------
The paper presents a novel actor-critic algorithm, named MADDPG, for both cooperative and competitive multiagent problems. MADDPG relies on a number of key ideas:
	1) The action value functions are learned in a 'centralized' manner, meaning that it takes into account the actions of all other players. This allows to evaluate the effect of the joint policy on each agents long term reward. To remove the need of knowing other agents' actions, the authors suggest that each agent could learn an approximate model of their policies.
	2) The learned policies can only use local information, so that they can be used by each agent without relying on further communication.
	3) It learns a collection of policies for each agent, rather than just one. At each episode during the learning process, each agent draws uniformaly a policy from its ensemble. Sampling a policy from an ensemble reduces the non-statinoarity caused by multiple agents learning at the same time.
Several interesting, both cooperative and competitive, are considered to evaluate the proposed algorithm. Simulation results show the benefit of using ensembles and improvement with respect to independent DDPG agents.
Major comments
-----------------
The main algorithmic contribution of the paper seems to be the policy ensemble to surmount nonstationarity. This is an interesting idea. However, very little insight is provided. Do the authors have any hypothesis of why this seems to work well?
The environments considered in Sec. 5 constitute an interesting benchmark. However, the proposed MADDPG algorithm is compared against independent DDPG agents. Although this validates the effectiveness of the algorithm, it would be more interseting to compare with centralized DDPG, as well as with other centralized critic approaches (e.g., [6]), and other ensemble RL algorithms.
The organization of the paper is somehow unbalanced. For instance, it reviews well known methods, while there are some important points that remain unclear. In particular, it seems that the algorithm learns as many action value functions as agents, so that it learns a parameter theta_i per agent. However, in lines 333-334, the authors mention that the complexity grows linearly for some Q (without subscript), rather than considering the computational burden of learning a set of action value functions {Q_i}.
Minor comments
-----------------
- Sentence in lines 47-48 is difficult to understand.
- The use of bold font notation seems arbitrary. Why are { pi_i } bold while { theta_i } aren't ?
- Word 'Reference' should precede '[11]' in line 81.


Diffusion Approximations for Online Principal Component Estimation and Global Convergence
In this paper, we propose to adopt the diffusion approximation tools to study the dynamics of Oja's iteration which is an online stochastic gradient descent method for the principal component analysis. Oja's iteration maintains a running estimate of the true principal component from streaming data and enjoys less temporal and spatial complexities. We show that the Oja's iteration for the top eigenvector generates a continuous-state discrete-time Markov chain over the unit sphere. We characterize the Oja's iteration in three phases using diffusion approximation and weak convergence tools. Our three-phase analysis further provides a finite-sample error bound for the running estimate, which matches the minimax information lower bound for principal component analysis under the additional assumption of bounded samples.

This work analyses the classical Oja iteration for finding the first principal component. They show it can be approximated using a stochastic process and provide global convergence guarantees which match the optimal .
Although the related work section is very dense, it is well organized and serves as a good starting point for the non-expert reader.
As a non expert in this area I find it quite hard to judge the merits of this paper in detail. Having said that it seems that providing the first global convergence guarantees for Oja’s rule ought to be of interest.
The analysis seems to thoroughly explain the different phases in the optimization procedure which perhaps could be useful as a diagnostic tool.
The analysis only considers the standard Oja rule which obtains the leading eigenvector. This is perhaps of limited use, but I appreciate the difficulty in extending this to the entire prinicpal subspace.
Line 27. A point which is tangential to the direction of a paper but still I believe a mischaracterization: We don’t need to compute the covariance matrix. Computing the SVD suffices and can be sped up using randomized techniques. At the very least, we don’t require O(d^2) storage.
There are plenty of typos and ungrammatical sentences throughout and should be proofread more thoroughly. E.g.
145: using an inﬁnitesimal generator analysis results
160: is name the generalized logistic curves.
165: for more on basic concepts and relates topics of SDE
171: a local maximizers that has negative curvatures in every direction.
171: In below, we consider


Learning to Pivot with Adversarial Networks
Several techniques for domain adaptation have been proposed to account for differences in the distribution of the data used for training and testing. The majority of this work focuses on a binary domain label. Similar problems occur in a scientific context where there may be a continuous family of plausible data generation processes associated to the presence of systematic uncertainties. Robust inference is possible if it is based on a pivot -a quantity whose distribution does not depend on the unknown values of the nuisance parameters that parametrize this family of data generation processes. In this work, we introduce and derive theoretical results for a training procedure based on adversarial networks for enforcing the pivotal property (or, equivalently, fairness with respect to continuous attributes) on a predictive model. The method includes a hyperparameter to control the tradeoff between accuracy and robustness. We demonstrate the effectiveness of this approach with a toy example and examples from particle physics.

The paper proposes an adversarial framework for training a classifier that is robust to systematic uncertainties. The paper studies a scenario where the data generation process of interest is governed by an unknown random variable Z where the random variable defines a family of data generation distribution (i.e. a different distribution is incurred by a realization of the random variable). It then applies the proposed method to train a classifier whose performance is independent of the random variable. The algorithm was tested on a synthetic dataset and a high energy physics example. No comparison with prior arts were drawn.
Paper strength
The paper extends the generative adversarial framework to a robust classification problem against systematic noise in the data generation mechanism.
The paper shows that the optimal classifier of the proposed framework is a classifier whose output is independent to the input variable.
Paper weakness
The paper fails to motivate the problem. Due to the difference of the members in the family of data distribution, samples from some member is naturally easier to classify than members from the other distribution (For example, consider the case that z is very large in Section 5.1). In other words, it is common to have classification outputs correlates with the unknown variable. A classifier would do better for samples from some members and do worse for samples from the other members. By adding the proposed adversary, which encourages the output to be independent with Z. It basically forces the classifier to do worse for the easy cases for matching the performance for the hard cases. The experiment results reports in the paper also suggest this is what happens. The overall accuracy is traded-off for consistency. It is unclear why it is advantageous to do so. A more reasonable approach would be to improve the performance across the members.
No comparison with prior works. The paper does not report comparison with baseline approaches that also consider the existence of Z when training the classifier. The only baseline model is a plane classifier model that ignores the existence of Z.
Experiment details are missing. Many details such as the data format in the section 5.2 is missing. It is difficult to understand the experiment setting.
The discussions on whether the optimal and pivotal classifier is achievable is very vague. The results that the training curves shown in the paper are away from the optimal values do not mean that the optimal and pivotal classifier is not achievable. The achievability might be just limited by the training of the network capacity. Hence, the statement is not solid.


Gaussian Quadrature for Kernel Features
Kernel methods have recently attracted resurgent interest, showing performance competitive with deep neural networks in tasks such as speech recognition. The random Fourier features map is a technique commonly used to scale up kernel machines, but employing the randomized feature map means that O( −2 ) samples are required to achieve an approximation error of at most . We investigate some alternative schemes for constructing feature maps that are deterministic, rather than random, by approximating the kernel in the frequency domain using Gaussian quadrature. We show that deterministic feature maps can be constructed, for any γ > 0, to achieve error with O(e e γ + −1/γ ) samples as goes to 0. Our method works particularly well with sparse ANOVA kernels, which are inspired by the convolutional layer of CNNs. We validate our methods on datasets in different domains, such as MNIST and TIMIT, showing that deterministic features are faster to generate and achieve accuracy comparable to the state-of-the-art kernel methods based on random Fourier features.
In the case of shift-invariant kernels, one technique that was proposed for constructing the function z is random Fourier features [23]. This data-independent method approximates the Fourier transform integral (1) of the kernel by averaging Monte-Carlo samples, which allows for arbitrarily-good estimates of the kernel function k. Rahimi and Recht [23] proved that if the feature map has dimension D =Ω d 2 then, with constant probability, the approximation z(x), z(y) is uniformly -close to the true kernel on a bounded set. While the random Fourier features method has proven to be effective in solving practical problems, it comes with some caveats. Most importantly, the accuracy guarantees are only probabilistic and there is no way to easily compute, for a particular random sample, whether the desired accuracy is achieved.
Our aim is to understand to what extent randomness is necessary to approximate a kernel. We thus propose a fundamentally different scheme for constructing the feature map z. While still approximating the kernel's Fourier transform integral (1) with a discrete sum, we select the sample points and weights deterministically. This gets around the issue of probabilistic-only guarantees by removing the randomness from the algorithm. For small dimension, deterministic maps yield significantly lower error. As the dimension increases, some random sampling may become necessary, and our theoretical insights provide a new approach to sampling. Moreover, for a particular class of kernels called sparse ANOVA kernels (also known as convolutional kernels as they are similar to the convolutional layer in CNNs) which have shown state-of-the-art performance in speech recognition [22], deterministic maps require fewer samples than random Fourier features, both in terms of the desired error and the kernel size. We make the following contributions:
• In Section 3, we describe how to deterministically construct a feature map z for the class of subgaussian kernels (which can approximate any kernel well) that has exponentially small (in D) approximation error.
• In Section 4, for sparse ANOVA kernels, we show that our method produces good estimates using only O(d) samples, whereas random Fourier features requires O(d 3 ) samples.
• In Section 5, we validate our results experimentally. We demonstrate that, for real classification problems on MNIST and TIMIT datasets, our method combined with random sampling yields up to 3 times lower kernel approximation error. With sparse ANOVA kernels, our method slightly improves classification accuracy compared to the state-of-the-art kernel methods based on random Fourier features (which are already shown to match the performance of deep neural networks), all while speeding up the feature generation process.

Post-rebuttal comments: Thank you for the feedback. I have raised my score on overall rating.
Summary: In my opinion, this is an interesting and promising direction for the use of deterministic quadrature rules in kernel approximation. The weakness of the paper is in the experiments: there should be more complete comparisons in computation time, and comparisons with QMC-based methods of Yang et al (ICML2014). Without this the advantage of the proposed method remains unclear.
- The limitation of the obtained results:
The authors assume that the spectrum of a kernel is sub-gaussian. This is OK, as the popular Gaussian kernels are in this class. However, another popular class of kernels such as Matern kernels are not included, since their spectrum only decay polynomially. In this sense, the results of the paper could be restrictive.
- Eq. (3):
What is $e_l$?
Corollaries 1, 2 and 3 and Theorem 4:
All of these results have exponential dependence on the diameter $M$ of the domain of data: a required feature size increases exponentially as $M$ grows. While this factor does not increase as a required amount of error $\varepsilon$ decreases, the dependence on $M$ affects the constant factor of the required feature size. In fact, Figure 1 shows that the performance is more quickly getting worse than standard random features. This may exhibit the weakness of the proposed approaches (or at least of the theoretical results).
- The equation in Line 170:
What is $e_i$?
- Subsampled dense grid:
This approach is what the authors used in Section 5 on experiments. However, it looks that there is no theoretical guarantee for this method. Those having theoretical guarantees seem not to be practically useful.
- Reweighted grid quadrature:
(i) It looks that there is no theoretical guarantee with this method.
(ii) The approach reminds me of Bayesian quadrature, which essentially obtains the weights by minimizing the worst case error in the unit ball of an RKHS. I would like to look at comparison with this approach.
(iii) Would it be possible to derive a time complexity?
(iv) How do you chose the regularization parameter $\lambda$ in the case of the $\ell_1$ approach?
- Experiments in Section 5:
(i) The authors reported the results of computation time very briefly (320 secs vs. 384 seconds for 28800 features in MNIST and "The quadrature-based features ... are about twice as fast to generate, compared to random Fourier features ..." in TIMIT). I do not they are not enough: the authors should report the results in the form of Tables, for example, varying the number of features.
(ii) There should be comparison with the QMC-based methods of Yang et al. (ICML2014, JMLR2016). It is not clear what is the advantage of the proposed method over the QMC-based methods.
(iii) There should be explanation on the settings of the MNIST and TIMIT classification tasks: what classifiers did you use, and how did you determine the hyper-parameters of these methods? At least such explantion should be included in the appendix.


A Sharp Error Analysis for the Fused Lasso, with Application to Approximate Changepoint Screening
In the 1-dimensional multiple changepoint detection problem, we derive a new fast error rate for the fused lasso estimator, under the assumption that the mean vector has a sparse number of changepoints. This rate is seen to be suboptimal (compared to the minimax rate) by only a factor of log log n. Our proof technique is centered around a novel construction that we call a lower interpolant. We extend our results to misspecified models and exponential family distributions. We also describe the implications of our error analysis for the approximate screening of changepoints.

The authors study the problem of multiple change point detection and provide provide bounds for the fused-lasso estimator. The problem of change point detection is well studied in the statistics literature and the authors provide good references.
The paper is well written and flows well. As claimed by the authors, one of the major contribution of this paper is in the technique of the proof and, unfortunately, the authors had to put it in the appendix to fit the NIPS page limits. I can not think of a way to fit that in the main section without a major rewrite. In its current form, the appendix is longer than the main paper.
A recent independent development, Guntuboyina et al. (2017), provide the bounds for the lasso estimator. The authors claim that their method of proof is of independent interest as that generalizes to other related problems. It is not clear why the methods of this generalizes but that of Guntuboyina et al. (2017) does not. I would have loved to see discussions on how theorem 3.1 compares to the main results in Guntuboyina et al. (2017) which seem to be stronger.
A comment on inequality (7): This seems to follow directly from the definition of \hat\theta unless I am missing something. Isn’t \hat\theta defined to be the parameter that minimizes
1/2 ||y-\theta||_2^2 + \lambda ||D\theta||_1


Regret Analysis for Continuous Dueling Bandit
The dueling bandit is a learning framework wherein the feedback information in the learning process is restricted to a noisy comparison between a pair of actions. In this research, we address a dueling bandit problem based on a cost function over a continuous space. We propose a stochastic mirror descent algorithm and show that the algorithm achieves an O( √ T log T )-regret bound under strong convexity and smoothness assumptions for the cost function. Subsequently, we clarify the equivalence between regret minimization in dueling bandit and convex optimization for the cost function. Moreover, when considering a lower bound in convex optimization, our algorithm is shown to achieve the optimal convergence rate in convex optimization and the optimal regret in dueling bandit except for a logarithmic factor.

The submitted paper presents a stochastic mirror descent algorithm for the dueling bandits problem defined in Yue & Joachims 2009. The paper gives a regret bound based on a regret definition established in the dueling bandits framework (where two nearly optimal arms have to be selected in order to minimize regret), an connects this result to convergence rates with respect to the optimality gap in convex optimization. The latter result is the most interesting contribution of the paper.
While the paper is self-contained and the reviewer could not find an error in the proofs (not being an expert in bandit learning), the paper could be improved considerably by putting it into the greater context of stochastic zero-order convex optimization. The gradient estimation approach given in Section 3.2 is reminiscent of work by Duchi et al. (2015). Optimal Rates for Zero-Order Convex Optimization: The Power of Two Function Evaluations. The convergence analysis given in the cited paper can also be interpreted as an expected regret bound for online bandit learning. How do the convergence rates compare? Earlier work on zero-order optimization by Ghadimi & Lan or Nesterov & Spokoiny is cited in this paper.
Furthermore, it would be nice to get a clearer picture how the presented work is different from related work in online bandit learning. Wouldn't the analysis of Yue & Joachims 2009 be easily extended to the setting of the paper by adding an appropriate proximal operator to their algorithm? Where does the assumption of linearity restrict Zhang et al.'s analysis to apply to the presented convex optimization framework?
Overall recommendation: The paper is certainly theoretically solid - a bit more explanation would not hurt and possible lead to wider reception of the presented work.


On Quadratic Convergence of DC Proximal Newton Algorithm in Nonconvex Sparse Learning
We propose a DC proximal Newton algorithm for solving nonconvex regularized sparse learning problems in high dimensions. Our proposed algorithm integrates the proximal newton algorithm with multi-stage convex relaxation based on the difference of convex (DC) programming, and enjoys both strong computational and statistical guarantees. Specifically, by leveraging a sophisticated characterization of sparse modeling structures (i.e., local restricted strong convexity and Hessian smoothness), we prove that within each stage of convex relaxation, our proposed algorithm achieves (local) quadratic convergence, and eventually obtains a sparse approximate local optimum with optimal statistical properties after only a few convex relaxations. Numerical experiments are provided to support our theory.

Summary:
This paper considers a DC (difference of convex) approach to solve *sparse* ERM problems with a non-convex regularization (in high dimensions). The specific non-convex regularizer studied in this paper is the capped l1 penalty. Now, given a particular convex relaxation via the DC approach, this paper proposes to solve the problem using proximal Newton.
The paper analyses the l2 convergence of their solution (at any step of the convex relaxation) to the truth, and shows that it enjoys quadratic convergence to stastical rates. The analysis is contingent on a Sparse Eigenvalue (similar to RSC) condition and a restricted Hessian smoothness condition.
Comments:
Overall, this is a well written paper and the results presented seem sound. I do however have the following questions/comments:
1. What is T in the warm initialization step of the proposed algorithm ? What does it scale as i.e. what is its dependence on other constants involved in the proof ?
2. It would be good if the authors could provide a more detailed comparison with [36] in the related work. It seems that [36] also considers proximal Newton. However, [36] only considers a convex regularizer. Is another consequence of this work also that the analysis of [36] can be improved with weaker conditions of this paper ?
3. Theorem 6 shows that for large enough K, and small s' (i.e. none or few weak signals), the algo in this paper can yield a l2-rate of O(\sqrt{s*/n}). However, if I am not wrong, there is also a minimax lower bound of O(\sqrt{s* log d / n}) i.e. an extra log d factor (for example, see Theorem 1(b) in https://arxiv.org/abs/0910.2042). Am I missing something here ? How can these two rates be reconciled ?
4. In the experiments, the authors should specify how \beta (the tuning parameter for capped l1) was chosen.
EDIT: I have seen the authors' response, and my rating stands.


Linear Time Computation of Moments in Sum-Product Networks
Bayesian online algorithms for Sum-Product Networks (SPNs) need to update their posterior distribution after seeing one single additional instance. To do so, they must compute moments of the model parameters under this distribution. The best existing method for computing such moments scales quadratically in the size of the SPN, although it scales linearly for trees. This unfortunate scaling makes Bayesian online algorithms prohibitively expensive, except for small or tree-structured SPNs. We propose an optimal linear-time algorithm that works even when the SPN is a general directed acyclic graph (DAG), which significantly broadens the applicability of Bayesian online algorithms for SPNs. There are three key ingredients in the design and analysis of our algorithm: 1). For each edge in the graph, we construct a linear time reduction from the moment computation problem to a joint inference problem in SPNs. 2). Using the property that each SPN computes a multilinear polynomial, we give an efficient procedure for polynomial evaluation by differentiation without expanding the network that may contain exponentially many monomials. 3). We propose a dynamic programming method to further reduce the computation of the moments of all the edges in the graph from quadratic to linear. We demonstrate the usefulness of our linear time algorithm by applying it to develop a linear time assume density filter (ADF) for SPNs.

I thank the authors for their response. As mentioned in my review, I also see the contributions of this paper on an algorithmic level. Nevertheless I would encourage the authors to at least comment on the existing empirical results to give readers a more complete picture.
--
The submitted manuscript develops a linear time algorithm for computation of moments in SPNs with general DAG structure, improving upon the quadratic time complexity of existing algorithms.
I like the contributions of this paper and I think the proposed algorithms are novel (given that the authors are the same as those of an arXiv paper from a couple of months ago) and could turn out to be useful. On the other hand, the contributions of the paper are somewhat limited. It would be beneficial to demonstrate the usefulness of the algorithms on some task like parameter estimation. In particular, it would be interesting to see whether the uncertainties that can be computed from the approximate posterior are useful in some way, e.g. to quantify uncertainty in queries etc.
Regarding "remark 3" in lines 228-236. Given that the authors draw this close connections, the neural networks with products unit studied in the past should be mentioned. These product units are more general than the products computed in SPNs (they have tuneable parameters).
A couple of more comments:
* Lines 251 to 254 give the impression that the stated result about moment matching in the case of exponential family distributions as approximating distributions is novel/from the authors. I found the same result in
    Shiliang Sun, "A review of deterministic approximate inference techniques for Bayesian machine learning", 2013
and expect there are even older similar results.
* There seems to be an arXiv version of the same paper which I have seen some time ago that goes beyond the submitted paper, e.g. it includes experiments. In these experiments, the method proposed in this paper do "only" compare favourably to other methods in terms of runtime, but not in terms of model fit. That is ok, as I see the contribution of this paper on a different level. However, as these results are known, I feel that they should have been mentioned somewhere (could have been anonymous).


On the Fine-Grained Complexity of Empirical Risk Minimization: Kernel Methods and Neural Networks
Empirical risk minimization (ERM) is ubiquitous in machine learning and underlies most supervised learning methods. While there is a large body of work on algorithms for various ERM problems, the exact computational complexity of ERM is still not understood. We address this issue for multiple popular ERM problems including kernel SVMs, kernel ridge regression, and training the final layer of a neural network. In particular, we give conditional hardness results for these problems based on complexity-theoretic assumptions such as the Strong Exponential Time Hypothesis. Under these assumptions, we show that there are no algorithms that solve the aforementioned ERM problems to high accuracy in sub-quadratic time. We also give similar hardness results for computing the gradient of the empirical loss, which is the main computational burden in many non-convex learning tasks.

The paper make use of (relatively) recent advances in complexity theory to show that of many common learning problems do not allow subquadratic time learning algorithms (given the veracity of the "Strong Exponential Time Hypothesis").
I appreciate that the authors do not oversell their results: They clearly state that they provide a worst-case analysis. Also, the results are not surprising. For instance, finding the exact solution of any kernel method requires the computation of the full kernel matrix, which is already quadratic in number of training examples. Reducing this computation time would imply that one can compute an approximation of the exact solution without computing the full kernel matrix, which is intuitively unlikely, unless he makes extra assumptions on the problem structure (i.e., the nature of the data-generating distribution). Hence, I would like the authors to comment on whether or not similar results could be obtained for the linear counterparts of studied ERM algorithms.
That being said, I consider that this is an important contribution. Up to my knowledge, this work is significantly different than common statistical learning theory papers. It may bring fresh air to our community, by bringing tools and insights from the complexity theory literature. From my point of view, the reductions from a discrete optimization problem to a continuous one are clever.
It is also worth noting that the paper is written rigorously (more than the average NIPS paper, I would say). Despite the hardness of the subject, it is easy to get the main ideas, notably the reductions from one problem to the other.


A multi-agent reinforcement learning model of common-pool resource appropriation
Humanity faces numerous problems of common-pool resource appropriation. This class of multi-agent social dilemma includes the problems of ensuring sustainable use of fresh water, common fisheries, grazing pastures, and irrigation systems. Abstract models of common-pool resource appropriation based on non-cooperative game theory predict that self-interested agents will generally fail to find socially positive equilibria-a phenomenon called the tragedy of the commons. However, in reality, human societies are sometimes able to discover and implement stable cooperative solutions. Decades of behavioral game theory research have sought to uncover aspects of human behavior that make this possible. Most of that work was based on laboratory experiments where participants only make a single choice: how much to appropriate. Recognizing the importance of spatial and temporal resource dynamics, a recent trend has been toward experiments in more complex real-time video game-like environments. However, standard methods of noncooperative game theory can no longer be used to generate predictions for this case. Here we show that deep reinforcement learning can be used instead. To that end, we study the emergent behavior of groups of independently learning agents in a partially observed Markov game modeling common-pool resource appropriation. Our experiments highlight the importance of trial-and-error learning in commonpool resource appropriation and shed light on the relationship between exclusion, sustainability, and inequality.

* Summary
Authors introduce a novel multi-agent problem (from machine learning's perspective) of common-pool resource appropriation and study how independently learning agents, while trying to optimize certain incentives, can learn to avoid "tragedy of the commons". The authors find that in certain environments, the learned strategies are consistent with predictions from economics.
* Strengths
The paper is well written and argued paper introducing a novel problem applying techniques of multi-agent reinforcement learning to model possible mechanisms by which intelligent agents can learn to avoid "tragedy of the commons". It shows how modern machine learning methods, when applied appropriately can add more fidelity to the model under study, while still maintaining some tractability. The authors clearly mention the limitations of the model and avoid making any comments about the social-economical implications.
* Weaknesses
- The paper is not self contained
Understandable given the NIPS format, but the supplementary is necessary to understand large parts of the main paper and allow reproducibility.
I also hereby request the authors to release the source code of their experiments to allow reproduction of their results.
- Use of deep-reinforcement learning is not well motivated
The problem domain seems simple enough that a linear approximation would have likely sufficed? The network is fairly small and isn't "deep" either.
- > We argue that such a mechanism is more realistic because it has an effect within the game itself, not just on the scores
This is probably the most unclear part. It's not clear to me why the paper considers one to be more realistic than the other rather than just modeling different incentives? Probably not enough space in the paper but actual comparison of learning dynamics when the opportunity costs are modeled as penalties instead. As economists say: incentives matter. However, if the intention was to explicitly avoid such explicit incentives, as they _would_ affect the model-free reinforcement learning algorithm, then those reasons should be clearly stated.
- Unclear whether bringing connections to human cognition makes sense
As the authors themselves state that the problem is fairly reductionist and does not allow for mechanisms like bargaining and negotiation that humans use, it's unclear what the authors mean by ``Perhaps the interaction between cognitively basic adaptation mechanisms and the structure of the CPR itself has more of an effect on whether self-organization will fail or succeed than previously appreciated.'' It would be fairly surprising if any behavioral economist trying to study this problem would ignore either of these things and needs more citation for comparison against "previously appreciated".
* Minor comments
** Line 16:
  > [18] found them...

  Consider using \citeauthor{} ?
** Line 167:
  > be the N -th agent’s
  should be i-th agent?
** Figure 3:
Clarify what the `fillcolor` implies and how many runs were the results averaged over?
** Figure 4:
Is not self contained and refers to Fig. 6 which is in the supplementary. The figure is understandably large and hard to fit in the main paper, but at least consider clarifying that it's in the supplementary (as you have clarified for other figures from the supplementary mentioned in the main paper).
** Figure 5:
- Consider increasing the axes margins? Markers at 0 and 12 are cut off.
- Increase space between the main caption and sub-caption.
** Line 299:
From Fig 5b, it's not clear that |R|=7  is the maximum. To my eyes, 6 seems higher.


Neural Program Meta-Induction
Most recently proposed methods for Neural Program Induction work under the assumption of having a large set of input/output (I/O) examples for learning any underlying input-output mapping. This paper aims to address the problem of data and computation efficiency of program induction by leveraging information from related tasks. Specifically, we propose two approaches for cross-task knowledge transfer to improve program induction in limited-data scenarios. In our first proposal, portfolio adaptation, a set of induction models is pretrained on a set of related tasks, and the best model is adapted towards the new task using transfer learning. In our second approach, meta program induction, a k-shot learning approach is used to make a model generalize to new tasks without additional training. To test the efficacy of our methods, we constructed a new benchmark of programs written in the Karel programming language [17]. Using an extensive experimental evaluation on the Karel benchmark, we demonstrate that our proposals dramatically outperform the baseline induction method that does not use knowledge transfer. We also analyze the relative performance of the two approaches and study conditions in which they perform best. In particular, meta induction outperforms all existing approaches under extreme data sparsity (when a very small number of examples are available), i.e., fewer than ten. As the number of available I/O examples increase (i.e. a thousand or more), portfolio adapted program induction becomes the best approach. For intermediate data sizes, we demonstrate that the combined method of adapted meta program induction has the strongest performance.

The paper investigates neural nets learning to imitate programs from varying numbers of input-output examples, both in the plain supervised learning setting and by leveraging transfer across problems. This is done in the domain of Karel, a simple programming language that describes how input grid worlds are transformed into output grid worlds through actions of a so-called hero who moves around in those worlds according to instructions given in a Karel program. The paper randomly generates many such Karel programs and, for each program, a number of corresponding input-output worlds. The learning problem for the neural architectures is to learn from input-output examples how to apply the same transformation to worlds as the (latent) Karel program. The paper compares plain supervised learning, supervised learning that's initialized using the closest previous task, k-shot meta-learning, and k-shot meta-learning that is refined based on the current task. The paper finds that in the Karel domain, each of these techniques has its use depending on the number of tasks and input-output examples per task available.
The paper seems technically sound.
There is another natural comparison that the authors didn't make, which is to follow the k-shot approach, but to learn a function from the example tasks to the *weights* of a neural net that turns input into output. That is, the task encoder generates the weights of both input encoder and output decoder. In this setting, adaptation is more natural than in the current META+ADAPT case, since you can literally just start with the weights generated by the meta network and optimize from there in exactly the same way that plain supervised learning does.
The paper is generally clear, but I don't sufficiently understand the Karel task and the difficulty of the associated learning problem. The paper shows a Karel sample task, but I would have liked to see more examples of the randomly sampled tasks (perhaps in an appendix) so that I can better understand how representative the sample task is. This would have answered the following questions: (1) How complex are the programs sampled from the Karel DSL CFG? A thing that can happen if you sample expressions from recursive PCFGs is that most are really simple, and a few are very complex; the expressions are not of similar size. Is that the case here? (2) How large are the I/O grids? How much variation is there? (3) I'm surprised that the LSTM can output the required diff to transform inputs into outputs as well as it does. If the diffs are long, I would have expected issues with the LSTM "forgetting" the initial context after a while or having trouble to exactly produce the right values or not generalizing correctly. The fact that this works so well makes me wonder whether the learning problem is easier than I think. I was also surprised to see that the NLL is as good as it is (in Figure 5). Because the computational structure of the conv net is quite different from the way the input-output examples are produced by the Karel program (the hero moves around locally to manipulate the grid, the conv net produces the output in one go), I think I would have expected the neural net to only approximately solve the problems. (This impression may well be due to the fact that I don't have a good intuition for what the problems are like.)
The table comparing to related work is helpful. That said, I'm still a bit unclear on the extent to which the paper's main contribution is the systematic evaluation of different program induction methods, and to what extent the paper claims to actually have invented some of the methods.
I think this paper will be useful to practitioners, conditioned on the results generalizing across domains. As the paper says, the results do seem fairly intuitive, so it seems plausible that they do generalize, but it would certainly have been nice to replicate this in another domain.


Linear Convergence of a Frank-Wolfe Type Algorithm over Trace-Norm Balls *
We propose a rank-k variant of the classical Frank-Wolfe algorithm to solve convex optimization over a trace-norm ball. Our algorithm replaces the top singular-vector computation (1-SVD) in Frank-Wolfe with a top-k singular-vector computation (k-SVD), which can be done by repeatedly applying 1-SVD k times. Alternatively, our algorithm can be viewed as a rank-k restricted version of projected gradient descent. We show that our algorithm has a linear convergence rate when the objective function is smooth and strongly convex, and the optimal solution has rank at most k. This improves the convergence rate and the total time complexity of the Frank-Wolfe method and its variants.

---- After the rebuttal ----
I cannot help but notice some similarities of this method and the CLASH method in the vector case:
https://arxiv.org/pdf/1203.2936.pdf
The CLASH authors in effect make a projection onto the rank-k ball (corresponding to Combinatorial part in the CLASH method), followed by the projection onto the rank constraint (Least Absolute SHrinkage part). The latter is now a vector problem thanks to the affine invariance of the Frobenius norm in this paper. The authors in the CLASH work on the vector case directly; however, as opposed to performing the simple shrinkage update as in this paper, the CLASH authors actually solve the "fully corrective problem." Perhaps, this would be of interest to the authors.
-----
The main idea of this paper is based on a simple observation: injecting a rank-k constraint into the subproblem in PGD does not affect
the convergence rate, as soon as the solution rank is less than or equal to k. This simple observation however leads to an algorithm with linear convergence rate that possess the benefits of FW method: low per iteration computational cost and low-rank updates. Note however, from a purely mathematical standpoint, I would argue calling this method as a Frank-Wolfe variant, as the rigorous description of FW is composed of the computation of a search direction and the application of a linear minimization oracle, followed with an averaging step. This algorithm however is based on a nonconvex quadratic subproblem, but this subproblem can be solved with theoretical guarantees efficiently.

That would be interesting to see the practical implications of the results section 5 that bounds the rank of the decision variable. Did you try this variant with replacement step in some numerical experiments? I would suggest addition of these results into the appendix. Note that there is a particular recent research direction about the storage efficiency of FW variants, e.g. [1].

In the current version of the manuscript, theoretical guarantees hold only if the solution rank is less than or equal to the input rank k. In practice, however, the solution is not exactly low rank due to the presence of the noise and imperfections in tuning the constraint radius. It is approximately low rank with a fast decay in the spectrum. As far as I can see the proof still holds when solution X* is replaced with the best rank-k approximation of the solution [X*]_k (see e.g. the guarantees provided in [1]). This would impose theoretical guarantees when f(X*) – f([X*]_k) is small.

The numerical experiments fall short of satisfaction for the nips venue. All the experiments with real data are solved using the auto selection of k heuristic. It is not clear from the text if this heuristic still possesses the theoretical guarantees of the original variant where k is fixed in advance. Plots with #1-SVD computations at x axis are very useful to compare the performance in terms of the overall computational cost. However, I would still suggest adding some information about the k chosen at each iteration while solving the real data problems. It would be also interesting to see some practical evidence of what happens when the input rank k is chosen slightly less than the solution rank, or when the solution is not low rank but it has a fast decay in the singular value spectrum.

As noted in the paper, algorithm outperforms other convex competitors in the convergence rate when the constraint radius is small, but there is no significant advantage when the radius is large. This is an expected result as the smaller radius tempts a lower rank solution. However, it is not possible to deduct which values of the radius are practically reasonable in the experiments presented in the paper (i.e. which values of theta results in lower test error).
 [1] Yurtsever et.al. Sketchy decisions: Convex low-rank matrix optimization with optimal storage. AISTATS, 2017.


A Screening Rule for 1 -Regularized Ising Model Estimation
We discover a screening rule for 1 -regularized Ising model estimation. The simple closed-form screening rule is a necessary and sufficient condition for exactly recovering the blockwise structure of a solution under any given regularization parameters. With enough sparsity, the screening rule can be combined with various optimization procedures to deliver solutions efficiently in practice. The screening rule is especially suitable for large-scale exploratory data analysis, where the number of variables in the dataset can be thousands while we are only interested in the relationship among a handful of variables within moderate-size clusters for interpretability. Experimental results on various datasets demonstrate the efficiency and insights gained from the introduction of the screening rule.

SUMMARY:
========
The authors propose an efficient screening for sparse Ising model structure estimation.  The screening rule is shown to be optimal for locating non-interacting variables in an L1 regularized MLE.  Combining this screening rule with approximate optimization allows efficient parallel computation of the full model structure, as demonstrated on several synthetic graph problems.
PROS:
=====
The screening rule can be calculated fairly efficiently, in time quadratic in the number of data elements, and as such could be easily adopted in practice.  The authors also clearly show that the method is optimal in a regularized MLE sense, and computation of the screening rule also provides a range of valid regularization parameters including those that yield a fully disconnected graph.
CONS:
=====
The authors do not discuss the relationship between the proposed screening rule and the *nearly* identical "correlation decay" property considered in Loh and Wainwright (2013) and references therein.  The work of Loh and Wainwright is referenced but not cited in the text.  This oversight is curious since Loh and Wainwright provide a probabilistic bound of correct reconstruction using the nodewise method considered in the present paper.
This reviewer feels that the experimental results are lacking and too much material is referenced in the supplement.  In particular, the authors claim experiments on real world data (L:228).  However, the real world experiment is a single gene interaction network on which only the node-wise method is performed, and is entirely contained in the supplement (B.6).  Moreover, the real world experiment is a small graph for which screening only yields a 20% speed improvement.  This is an unfortunate omission as more detailed experimentation of real world data would have been beneficial.
In addition to the real world experiment, other experimental details are deferred to the supplement or not discussed at all.  Discussion of model selection is omitted and references B.4, which in turn provides only a few sentences and references Liu et al. (2010).  Approximations are only compared to exact inference in a single experiment (also in the supplement) for which exact computation is computed in 90s.  Finally, inclusion of the "mixed" approach which uses NW for model selection and PL for estimation is both slower than PL and less accurate than NW, it is unclear why this approach would be considered.