Regularized Nonlinear Acceleration
We describe a convergence acceleration technique for generic optimization problems. Our scheme computes estimates of the optimum from a nonlinear average of the iterates produced by any optimization method. The weights in this average are computed via a simple and small linear system, whose solution can be updated online. This acceleration scheme runs in parallel to the base algorithm, providing improved estimates of the solution on the fly, while the original optimization method is running. Numerical experiments are detailed on classical classification problems.
using classical results from control theory to obtain numerical bounds on convergence rates that match the optimal rates. Others have studied the second order ODEs obtained as the limit for small step sizes of classical accelerated schemes, to better understand their convergence [10,11]. Finally, recent results have also shown how to wrap classical algorithms in an outer optimization loop, to accelerate convergence [12] and reach optimal complexity bounds.
Here, we take a significantly different approach to convergence acceleration stemming from classical results in numerical analysis. We use the iterates produced by any (converging) optimization algorithm, and estimate the solution directly from this sequence, assuming only some regularity conditions on the function to minimize. Our scheme is based on the idea behind Aitken's ∆ 2 algorithm [13], generalized as the Shanks transform [14], whose recursive formulation is known as the ε-algorithm [15] (see e.g. [16,17] for a survey). In a nutshell, these methods fit geometrical models to linearly converging sequences, then extrapolate their limit from the fitted model.
In a sense, this approach is more statistical in nature. It assumes an approximately linear model holds for iterations near the optimum, and estimates this model using the iterates. In fact, Wynn's algorithm [15] is directly connected to the Levinson-Durbin algorithm [18,19] used to solve Toeplitz systems recursively and fit autoregressive models (the Shanks transform solves Hankel systems, but this is essentially the same problem [20]). The key difference here is that estimating the autocovariance operator is not required, as we only focus on the limit. Moreover, the method presents strong links with the conjugate gradient when applied to unconstrained quadratic optimization.
We start from a slightly different formulation of these techniques known as minimal polynomial extrapolation (MPE) [17,21] which uses the minimal polynomial of the linear operator driving iterations to estimate the optimum by nonlinear averaging (i.e., using weights in the average which are nonlinear functions of the iterates). So far, for all the techniques cited above, no proofs of convergence of these estimates were given in the case where the iterates made the estimation process unstable.
Our contribution here is to add a regularization in order to produce explicit bounds on the distance to optimality by controlling the stability through the regularization parameter, thus explicitly quantifying the acceleration provided by these techniques. We show in several numerical examples that these stabilized estimates often speed up convergence by an order of magnitude. Furthermore this acceleration scheme thus runs in parallel to the original algorithm, providing improved estimates of the solution on the fly, while the original method is progressing.
The paper is organized as follows. In section 2.1 we recall basic results behind MPE for linear iterations and we will introduce in section 2.2 a formulation of the approximate version of MPE and make a link with the conjugate gradient method. Then, in section 2.3, we generalize these results to generic nonlinear iterations and show, in section 2.4, how to fully control the impact of nonlinearity. We use these results to derive explicit bounds on the acceleration performance of our estimates.

Extrapolation, i.e., constructing a new, potentially better estimate from existing ones, is a ubiquitous tool in numerical analysis. This work applied this old idea to optimization, to construct on the fly a better iterate. A regularized version (by adding ell_2 regularization) of the classic minimal polynomial extrapolation is proposed. The authors provided some crude asymptotic error analysis. Limited experiments against Nesterov's accelerated gradient and the conjugate gradient are provided, and the results seems to favor the proposed method.
===== added after author response ===== The authors' rebuttal was somewhat helpful. There are a few points that I'd like to emphasize again so that the authors would prepare a better version.   "asymptotic analysis:" In line 199 and line 209, in order to deduce a meaningful bound from Proposition 2.6, you have to take the limit x_0 - x* --> 0 (and possibly k large), right? To me this is an asymptotic analysis. You need to be sufficiently close to the optimum so that the bound can kick in (i.e., its leading coefficient to be smaller than 1). It would seem a good idea to present line 199-207 in greater detail since this is a strong example. (But to note that the authors needed Lipschitz continuity of the second derivative, which is unusual.)  I do not agree with the authors in "A fair comparison, since the number of gradient oracle calls was identical in both cases and the grid search on lambda had only a marginal impact on complexity." Note that Nesterov backtracking, if implemented correctly, does not require additional gradient calls, very much the same as the authors backtracking on lambda. All experiments in this work are reported with x-axis being the number of gradient oracle calls. Another figure with x-axis the cumulative running time should also be presented, as RMPE has some overheads (solving Eq. (17) etc.).  ===== end =====  I very much like how the authors relate classical extrapolation methods to the more recent accelerated gradient methods (e.g., Nesterov's). No doubt this will provide new angles to understand acceleration. However, the presentation of this work needs some improvement, and the main results need to be better articulated and contrasted. The experiments are also week for reasons I will detail below.   Let me first comment that the approximate MPE method is NOT novel, in contrast to what the authors claimed in line 58. It is interesting that the authors cited [19] multiple times but did not appear to recall its main points. For instance, (APME) below line 98 is exactly Eq. (10.5) of [19], which attributed to Mesina. Also, the bound above line 102 is exactly Eq. (10.6) of [19]. From this point of view, claiming novelty of APME, or even renaming it as APME, is not appropriate. Another example: Theorem 1 in [19] did not prove the claim in footnote 2 at all (in fact the claim is not true: one needs q to be minimal). This kind of sloppiness is not unusual in the whole manuscript, which is hurting the credibility of this work. Other examples: Eq. (5) used U_0, which is never defined, and then in Eq. (5) and afterwards, it was changed to u_0; The derivations in Eqs. (6)-(9) should have the constraint deg(p) <= k; etc.  I would strongly suggest the authors spend more time on explaining their results, and better contrast with existing methods. For instance, it should be mentioned that under the condition (on A) in Prop. 2.2, the original sequence x_i is in fact converging at a linear rate. Shouldn't you care to mention this and compare the constants (e.g., slope of linear rate)? The discussions from line 119 to line 129 are very interesting, but they lack some details and references. These comparisons, to me, are the interesting part, and I wish the authors to be more elaborative and explicit.   The regularized AMPE in section 24 is not well-motivated. First, one should point out that all the analyses are asymptotic, namely that the approximation or expansion is only valid when x_i is sufficiently close to x_*. The asymptotic nature and the many implicit constants of this analysis make the justification of RAMPE shaky. Second, the authors' motivation to propose RAMPE is to avoid any near-singularity of U^T U. But, when U^T U is near singular, it clearly suggests that the principle behind AMPE may well be violated (e.g., x_i not converging linearly, k too small, etc.). In that case, it seems more useful to fix AMPE as suggested in the reference [19], instead of regularizing?  My biggest complaint comes from the experiment, in particular the selection of the parameter lambda. The authors used a backtracking strategy, which is fine but directly contradicts what is said in line 178, since multiple evaluation of the function f is indeed needed. In this regard, the experiment is not fair either: First, one should also compare the cumulative running time (on top of gradient evaluations), since Nesterov's acceleration does not cost any computational overhead while the proposed RAMPE does. Second, one should also allow Nesterov's algorithm to backtrack the Lipschitz constant and strong convexity parameter, like in [3, 4]. I suspect the main difference in Figs 1, 2 are because of this backtracking (of lambda). The authors need to convince me otherwise. Also, is the condition A < I satisfied in the experiments? (I suspect not since the condition number was deliberately set too large, e.g. tau = 10^2.)


GAP Safe Screening Rules for Sparse-Group Lasso
For statistical learning in high dimension, sparse regularizations have proven useful to boost both computational and statistical efficiency. In some contexts, it is natural to handle more refined structures than pure sparsity, such as for instance group sparsity. Sparse-Group Lasso has recently been introduced in the context of linear regression to enforce sparsity both at the feature and at the group level. We propose the first (provably) safe screening rules for Sparse-Group Lasso, i.e., rules that allow to discard early in the solver features/groups that are inactive at optimal solution. Thanks to efficient dual gap computations relying on the geometric properties of -norm, safe screening rules for Sparse-Group Lasso lead to significant gains in term of computing time for our coordinate descent implementation.

This paper proposed the GAP safe screening rules for the Sparse-Group Lasso. To apply the safe rule, the authors give an algorithm  to evaluate the dual norm, which they claimed is efficiently.
Generally speaking, the paper is well written. The general context of the paper is clear. However, there are too many symbols in the Noation part, and all of them are inline formula, which make the paper a little difficult to read. I suggest, for the ease of reading, move some of the formulas to the first place where they appear. For example,  move the notations in line 68-70 to line 76. It seems weired to discuss the choice of $\tau$ without the formula of $\Omega_{\tau, w}$ explicitly.   In line 24-25, [24] is publised before [11], why the authors said that the work of [24] is following [11].   In line 63, “... the subdifferential ... of the $ \ell_1$_1 norm is $ \sign(\cdot)$”. what is the definition of $ \sign(\cdot)$ in this paper? The subdifferential of the $ \ell_1$_1 norm at 0 the the region $ [-1,1]$. It should be different from the definition of $ \sign(\cdot)$ at 0.  In line 160, $\epsilon$-norm is definded as the "unique" nonnegative solution explicitly of the equation. It seems not true. For example, when $\epsilon = 0$, any value which is larger than $\max|x_i|$ would be the solution of the equation. The author should give a strict definition of $\epsilon$-norm.


Unsupervised Domain Adaptation with Residual Transfer Networks
The recent success of deep neural networks relies on massive amounts of labeled data. For a target task where labeled data is unavailable, domain adaptation can transfer a learner from a different source domain. In this paper, we propose a new approach to domain adaptation in deep networks that can jointly learn adaptive classifiers and transferable features from labeled data in the source domain and unlabeled data in the target domain. We relax a shared-classifier assumption made by previous methods and assume that the source classifier and target classifier differ by a residual function. We enable classifier adaptation by plugging several layers into deep network to explicitly learn the residual function with reference to the target classifier. We fuse features of multiple layers with tensor product and embed them into reproducing kernel Hilbert spaces to match distributions for feature adaptation. The adaptation can be achieved in most feed-forward models by extending them with new residual layers and loss functions, which can be trained efficiently via back-propagation. Empirical evidence shows that the new approach outperforms state of the art methods on standard domain adaptation benchmarks.
by a small perturbation function. The goal of this paper is to simultaneously learn adaptive classifiers and transferable features from labeled data in the source domain and unlabeled data in the target domain by embedding the adaptations of both classifiers and features in a unified deep architecture.
Motivated by the state of the art deep residual learning [8], winner of the ImageNet ILSVRC 2015 challenge, we propose a new Residual Transfer Network (RTN) approach to domain adaptation in deep networks which can simultaneously learn adaptive classifiers and transferable features. We relax the shared-classifier assumption made by previous methods and assume that the source and target classifiers differ by a small residual function. We enable classifier adaptation by plugging several layers into deep networks to explicitly learn the residual function with reference to the target classifier. In this way, the source classifier and target classifier can be bridged tightly in the back-propagation procedure. The target classifier is tailored to the target data better by exploiting the low-density separation criterion. We fuse features of multiple layers with tensor product and embed them into reproducing kernel Hilbert spaces to match distributions for feature adaptation. The adaptation can be achieved in most feed-forward models by extending them with new residual layers and loss functions, and can be trained efficiently using standard back-propagation. Extensive evidence suggests that the RTN approach outperforms several state of art methods on standard domain adaptation benchmarks.

The paper presents a new deep learning architecture for the unsupervised adaptation problem, i.e. when one has labeled training data for a source domain and unlabeled data only for the target domain.  The approach mixes ideas of deep adaptation networks (ref. 5), residual networks (ref. 8) and entropy minimization for unsupervised learning (ref. 28) to propose an accurate and principled approach.
The proposed architecture is a smart mix of previously published ideas : use of MK-MMD from deep adaptation networks (ref. 5), including residual networks blocks from (ref. 8) to relate the classifiers in the source domain and in the target domain, and entropy minimization (ref. 28) to guide the final learning in the target domain, where there are no labeled training samples. This  allows the authors outperforming state of the art methods on standard benchmarks.   The use of residual learning combined with entropy minimization learning criterion to jointly learn the source and target classifier is new and smart and it seems to work well in practice. Yet although the residual learning idea seems complementary to the MK-MMD of (ref. 5) (in the experimental section) this works looks more like an incremental work over (ref. 5) than a brand new model that is very different from (ref. 5) as the authors claim.    I don’t fully understand the difference between the use of MK MMD here with respect to what is proposed in (ref. 5). What is different ? Why is it better here ? And what is the motivation ? This part is then not fully convincing and especially since there is no experimental comparison with the method proposed in (ref. 5) and the variant proposed here.  It is said multiple times that the residual learning framework guarantees that the residual part will be small but i am  not sure there are such guarantees actually. It is more an experimental finding but in a very different setting. Why should it be true here ?  Experimental results are provided for the two  benchmarks of (ref. 5) and show quite convincing improvements of proposed method with respect to state of the art and put in evidence the actual complementarity of MMD criterion and of the residual learning idea. As said above a comparison with the same MMD use was in (ref. 5) would help appreciating if this variant is efficient or not here. There are a number of results which are different from the ones in (ref. 5) although the experimental protocol described in 4.1 looks strictly the same.  This is the case for DAN, the method in (ref. 5), which reaches 72.9 average accuracy on Office-31 dataset while it is reported at 70.0 here (ref. Table 1). Comments on such differences should be given. Moreover results from benchmark methods like TCA and GFK are much better here than results reported in (ref. 5) (~66% instead of ~27%)which might be due to the use of deep features as input where these methods were used on rough features in (ref. 5)? Please provide more precision on that.   Overall the paper is nice and easy to read. It proposes a new brick to previously designed deep architectures for unsupervised adaptation. It is then partially incremental in my opinion but still it is a nice use of residual learning idea.


"Congruent" and "Opposite" Neurons: Sisters for Multisensory Integration and Segregation
Experiments reveal that in the dorsal medial superior temporal (MSTd) and the ventral intraparietal (VIP) areas, where visual and vestibular cues are integrated to infer heading direction, there are two types of neurons with roughly the same number. One is "congruent" cells, whose preferred heading directions are similar in response to visual and vestibular cues; and the other is "opposite" cells, whose preferred heading directions are nearly "opposite" (with an offset of 180
• ) in response to visual vs. vestibular cues. Congruent neurons are known to be responsible for cue integration, but the computational role of opposite neurons remains largely unknown. Here, we propose that opposite neurons may serve to encode the disparity information between cues necessary for multisensory segregation. We build a computational model composed of two reciprocally coupled modules, MSTd and VIP, and each module consists of groups of congruent and opposite neurons. In the model, congruent neurons in two modules are reciprocally connected with each other in the congruent manner, whereas opposite neurons are reciprocally connected in the opposite manner. Mimicking the experimental protocol, our model reproduces the characteristics of congruent and opposite neurons, and demonstrates that in each module, the sisters of congruent and opposite neurons can jointly achieve optimal multisensory information integration and segregation. This study sheds light on our understanding of how the brain implements optimal multisensory integration and segregation concurrently in a distributed manner.

It is thought that in MSTd and VIP neural responses to visual and vestibular stimulation are integrated as part of the computation to estimate heading direction. In recent years, two relatively distinct types of neurons have been reported in both visual areas. Both types have their response level modulated by visual and vestibular input, but in different ways. Congruent neurons prefer inputs that maximally agree. Opposite neurons prefer inputs that maximally disagree. The manuscript aims to develop a computational account of the functional purpose of the opposite neurons. They suggest that opposite neurons may play an important role in segregating multimodal input. This hypothesis is not new. But the authors undertake a serious attempt to construct a computational framework that makes rigorous the hypothesis. I can quibble with certain aspects of their model. But overall, I think it is a very nice contribution.
The most significant concern I had with the manuscript was that all the modeling results were presented with a fano factor of 0. The authors took pains to develop the model so that they could take into account the effects of pseudo-realistic neural noise, but they made no attempt to justify why their presentation of results did not include it. There was not even any discussion of why the authors made this seemingly odd choice. Do results change markedly as the fano factor is manipulated? Why or why not? Are the results sensitive to the level of noise? Is there a regime in which results are qualitatively unaffected by the level of noise (as one would hope would be true in a biological system)? Or are they brittle?  Abstract: “there exist two types of neurons with comparable numbers.”  Hard to parse. “with comparable numbers”. Please rephrase.  Abstract: “the sisters of congruent and opposite neurons can jointly achieve optimal multisensory information integration and segregation”  Confusing. What are ‘sisters’ of congruent and opposite neurons.  Line 22: “Our brain senses the external world with multiple sensory modalities, including vision, audition…”   This comment is epistemological in nature, but the brain does not sense the external world. The sense organs sense proximal stimuli. The transduced proximal stimuli are then processed into estimates or representations of the external world. The authors are surely aware of this distinction but I think it is important to be clear about it in written text. There is a lot of loose language in science and the media about these issues. We practitioners should be precise.  Line 32: “The neural circuit models underlying optimal multisensory integration were also proposed…”  The neural circuit models? Maybe- ‘Neural circuit models underlying optimal multisensory integration have been proposed’  Line 39: “In an ambiguous environment, however, the brain is unable to differentiate the two situations straightforwardly”  Nonsense sentence. If input stimuli are ambiguous there is no correct answer, so it cannot be straightforward to differentiate one situation from another. Please rephrase so that the point is clear. Not sure what the authors are going for here.  Line 41: “brain is unable to perceive…”  Brains don’t perceive (or decide or realize or love). Animals and people do.   Line 51: “The psychophysical data tends to support this idea, which shows that when multisensory integration happens, the brain can still sense the difference between cues [8].”  The authors might consider citing Girshick et al 2009 here.  Line 53: “…brain to realize…”   Line 61: “modelling"  Spelling.  Line 71: “In reality, because of noises, the brain…”  Unorthodox sentence. And noisES?  Should it be ‘noise’?  Line 77- “The stimulus we consider is heading direction, which is a circular variable with values in the range of (􀀀_; _] , we therefore adopt the von Mises distribution (Supplementary Information Sec. 1).”  Ungramattical sentence.  Line 106: “The indirect cue x2  is informative to s1  via the prior p(s1; s2) . By using Eqs. (1,3) and under reasonable approximations (SI Sec. 1.4), we obtain [posterior apprx. proportional to Von Mises distribution]”   The authors should probably cite Murray & Morgenstern (2010). Their paper directly examines Bayesian cue integration of circular variables under a von Mises assumption.  Line 169: “In their model, MSTd and VIP are congruently connected, i.e., Neurons in one module are strongly connected to those in the other module having the similar preferred heading directions.”  Grammar.   Figure 4 Caption: “but the preferred direction of the opposite neuron under either cues are opposite by 180_”  Grammar.  Line 222: “We then consider the population activities of our model under different cuing conditions without  input noise (F = 0 )”  Motivate this. Why do the authors consider the response properties with a fano factor of 0, rather than a more realistic value?  Potentially confusing language. The authors refer to a fano factor as ‘input noise’. But standard usage would call the parameter, which characterizes the ratio of a neuron’s response variance to its mean, as affecting ‘output noise’ rather than ‘input noise’.   Line 252: “Over the past years, multisensory integration has received large attention in modelling studies,”   Grammar.   Line 253: “The present study proposes that opposite neurons, which has been widely observed atMSTd and VIP but with origin and computational role unresolved, serve to encode the disparity information between sensory cues.”   Hard sentence to parse.  Line 259: “To our best knowledge, this work is the first modelling study unveiling the functional role of opposite cells.”  Line 260: “It  has a far-reaching implication on multisensory information processing, that is, the brain can exploit sisters of congruent and opposite neurons to implement cue integration and segregation concurrently.”  Grammar. And what are sisters?    Appendix Line 35: “The Bayesian integration of two cues is expressed as…”  Please be more precise with the language. The equation stated is only correct under a flat prior. The authors write this later in the appendix but it would be could to state it here.  Appendix Line 55: “von pMises”   Typo.  Appendix Line 143: “The strength of the background input was Ib = 1 , and all Fano factors F  of the cues and background inputs were set to 0.5 . This resulted in a Fano factor of single neuron responses in the order of 1. In the simulation, the activity bump position was estimated by using a population vector (see below).”  Please discuss this issue in more depth. It is not clear why different units should have different fano factors.   Appendix  Line 171: “perceived by opposite neurons”   Neurons don’t perceive.


Convolutional Neural Networks on Graphs with Fast Localized Spectral Filtering
In this work, we are interested in generalizing convolutional neural networks (CNNs) from low-dimensional regular grids, where image, video and speech are represented, to high-dimensional irregular domains, such as social networks, brain connectomes or words' embedding, represented by graphs. We present a formulation of CNNs in the context of spectral graph theory, which provides the necessary mathematical background and efficient numerical schemes to design fast localized convolutional filters on graphs. Importantly, the proposed technique offers the same linear computational complexity and constant learning complexity as classical CNNs, while being universal to any graph structure. Experiments on MNIST and 20NEWS demonstrate the ability of this novel deep learning system to learn local, stationary, and compositional features on graphs.

This paper presents an extension of Convolutional Neural Networks to irregular grids in arbitrary dimension, using the properties of the Graph Laplacian Spectrum. The proposed architecture overcomes a computational burden of previous spectral methods, leading to efficient training and evaluation, as demonstrated on image classification and text categorization.
This paper builds upon previous work to propose a computationally efficient model that extends convolutions to irregular graphs. The major innovation is to express graph filtering with a compactly supported kernels as linear combinations of order K polynomials of the graph Laplacian, thus overcoming the burden of computing the Laplacian eigendecomposition, which is needed in both forward and backward passes in the previous spectral construction.  Moreover, the Chebyshev recurrence implies that order K filters can be computed with O(K E) operations, where E is the number of edges and controls the cost of computing L x (L = Laplacian operator on the graph). This appears to be a very clean, efficient parametrization that translates into efficient learning and evaluation and overcomes an important computational limitation.  I have a couple of comments that I would like the authors to develop during the rebuttal phase:   - A previous model that operates in general graphs is the so-called Graph Neural Network (GNN) [Scarselli et al.,'09]. The basic model can be seen as follows. Given a generic vector-valued function x on a graph G, x: G--> R^d, we compute its linear diffusion Wx(i) = \sum_{j neighbor of i} w_{i,j} x(j) and its normalized scaling Dx(i) = (sum_{j neighbor of i} w_{i,j}) x(i). Then a "GNN" layer transforms x into Phi(x): G-->R^{d'} via a generic point-wise operator combining x and Wx: Phi(x)(i) = Psi( Dx(i), Wx(i)), with Psi being a generic pointwise operator: Psi: R^d x R^d --> R^{d'}. For example, if Psi(x,y) = x - y we obtain the Graph Laplacian Phi(x) = L(x), and thus the order-K graph filtering can be realized with a K-layer GNN using linear pointwise operators Psi. The model which computes filters as linear combinations of powers of graph laplacians can thus be seen as a special case of the previous model, which interleaves linear layers Psi (responsible of computing the powers of the laplacian) with nonlinear layers. It would be interesting for the authors to comment on the pros/cons of their model with respect to the general GNN.   - The numerical experiments in MNIST show good performance of the graph CNN, but also a gap with respect to the classic CNN architectures. One can thus ask whether this gap is simply due to model tuning or due to some intrinsic limitations of graph CNNs when used on regular, low-dimensional grids. The filters learnt by the proposed graph CNN are polynomials of the Graph Laplacian, and therefore are rotationally invariant, whereas in the grid the Euclidean embedding allows for directional filters, which bring a crucial advantage to capture geometric structures in images. Is this analysis correct? In other words, there seems to be a fundamental price to pay for giving up the knowledge of the grid, its translation invariance and the notion of orientation. Obviously this only concerns the examples where the grid is available.   Overall, and despite my previous two comments, this paper brings an important improvement to the table, that can extend the use of data driven, end-to-end learning with excellent learning complexity to other datasets beyond images, speech and text. I hope the rebuttal addresses my previous comments though!


Launch and Iterate: Reducing Prediction Churn
Practical applications of machine learning often involve successive training iterations with changes to features and training examples. Ideally, changes in the output of any new model should only be improvements (wins) over the previous iteration, but in practice the predictions may change neutrally for many examples, resulting in extra net-zero wins and losses, referred to as unnecessary churn. These changes in the predictions are problematic for usability for some applications, and make it harder and more expensive to measure if a change is statistically significant positive. In this paper, we formulate the problem and present a stabilization operator to regularize a classifier towards a previous classifier. We use a Markov chain Monte Carlo stabilization operator to produce a model with more consistent predictions without adversely affecting accuracy. We investigate the properties of the proposal with theoretical analysis. Experiments on benchmark datasets for different classification algorithms demonstrate the method and the resulting reduction in churn.

The paper articulates an important problem---reducing unnecessary changes in the predictions of machine learning algorithms, describes why this problem is relevant (it hinders usability of prediction, and hinders ability to statistically validate that the trained model improved), and gives this problem a name (“prediction churn”) and very general formal definition.  The paper introduces a conceptually interesting and general fixed-point approach to reducing prediction churn, which it describes (in a somewhat hand-wavy way) as an MCMC technique. The approach combines regularization of new classifiers with respect to the previous classifier and training the new classifier on perturbed version of the training data set. Together, these techniques, when iterated, are expected to result in a final classifier that will not change much when re-trained (regularized) on the perturbed dataset when the perturbations are the result of real-world drift. The paper provides examples of two fairly general stability operators with hyperparameters governing the tradeoff between churn and accuracy.  The paper provides some theoretical bounds on the “churn” for restricted settings, but does not provide theoretical characterization of the de-churning of the iterative Markov chain, such as when a stationary distribution exists.  The paper includes experiments on real-world datasets and several learning algorithms that demonstrate that a one-step and Markov chain de-churning process successfully reduce churn while maintaining or improving accuracy. The experimental results do not emphasize the importance of the fixed-point property of the Markov chain.
The problem articulated and addressed by the paper is important, the techniques introduces are very general in their applicability, and the experimental results successfully demonstrate the utility of the techniques to addressing the problem.  The fixed-point MCMC approach is very conceptually compelling, although it lacked theoretical analysis and rigor, and appeared to be not that significant to achieving good results (the results suggest a single iteration may be sufficient). After reading the highly suggestive section on the Markov chain, the theoretical results appeared underwhelming. A rigorous analysis of the Markov chain would have made the paper more self-contained and increased its technical strength. Alternatively, focusing on the single-step stabilization operator case, and obtaining more general theoretical results within that restricted setting, would have made the paper more self-contained and cohesive.  The paper lacked polish, which made it harder to digest than necessary. For example, in the experiment section 5.1 the authors paper refers to "the RCP operator comparison" and "the MCMC operator comparison", and do not make it clear that the MCMC comparison itself is composed of multiple RCP operators. Issues of wording like this, as well as somewhat strange choices of order in which new notation is presented (e.g. T_A is presented on line 87 even though it is not necessary until later; the use of undefined notation P_T on line 91), reduced the clarity of the paper.  It would have been interesting to see a relation of the technique to online Bayesian learning. The Markov chain, stabilization operator, and training data perturbation technique seem related to Bayesian approaches, and an analysis of the Markov chain (that was not included in the paper) may specifically benefit from this perspective.  Finally, the paper would have benefited from a discussion or conclusion that discussed in more detail the potential impact of this work.


Disentangling factors of variation in deep representations using adversarial training
We introduce a conditional generative model for learning to disentangle the hidden factors of variation within a set of labeled observations, and separate them into complementary codes. One code summarizes the specified factors of variation associated with the labels. The other summarizes the remaining unspecified variability. During training, the only available source of supervision comes from our ability to distinguish among different observations belonging to the same class. Examples of such observations include images of a set of labeled objects captured at different viewpoints, or recordings of set of speakers dictating multiple phrases. In both instances, the intra-class diversity is the source of the unspecified factors of variation: each object is observed at multiple viewpoints, and each speaker dictates multiple phrases. Learning to disentangle the specified factors from the unspecified ones becomes easier when strong supervision is possible. Suppose that during training, we have access to pairs of images, where each pair shows two different objects captured from the same viewpoint. This source of alignment allows us to solve our task using existing methods. However, labels for the unspecified factors are usually unavailable in realistic scenarios where data acquisition is not strictly controlled. We address the problem of disentaglement in this more general setting by combining deep convolutional autoencoders with a form of adversarial training. Both factors of variation are implicitly captured in the organization of the learned embedding space, and can be used for solving single-image analogies. Experimental results on synthetic and real datasets show that the proposed method is capable of generalizing to unseen classes and intra-class variabilities.

This paper proposes a method to separate style and content in a neural representation. This is achieved by defining a latent code with vectors for specified ("content") and unspecified ("style") variation. A GAN loss encourages samples from the unspecified vector conditioned on the specified vector to be indistinguishable from empirical samples from a specified content class. This way the unspecified vector is encouraged to model all variation not related to class. Unlike previous methods, this is achieved without forcing the latent code to explicitly include a categorical class variable. Experiments show that style and content are successfully separated on a few simple datasets.
I found this to be a quite interesting paper. It is well-written, except for many typos throughout, which are easily fixable. I did, however, find the model somewhat hard to follow. I think it would help to clarify early on what is the difference between using s in the latent code versus using a class label indicator. This distinction strikes me as a primary contribution of the paper.  The model is novel to my knowledge, and uses GANs in an intriguing way. The experiments are convincing that style and content were successfully separated.  I think the main weakness of this paper is that the experiments do not convincingly show that the novelty of the model paid off. A primary novelty is that the proposed model does not require class labels in the code layer. Instead the code layer encodes class-specific information as a continuous vector of features. The paper argues that an advantage might be that the model can be applied to novel classes, not seen at training time. Unfortunately, the experiments show little evidence that this actually works. The NORB experiments are the only ones that test performance on novel classes, and in these examples it does not look like style is well separated from content. As a result, it is not clear what is gained by coding class-specific features s rather than using the older approach of coding a discrete class label in the latent code. The paper does, however, show that using s allows for smooth interpolation between classes, which is an interesting property.  Overall this paper has demonstrated a novel method to disentangle factors of variation, employing a GAN, and this method has potential advantages in terms of how it codes class-specific information (and that it doesn't rely on aligned training data). I think the paper could be improved by including more tests of generalization to novel classes (surely this works on MNIST?), and perhaps finding other ways to exploit the novel encoding of class-specific information.


Global Optimality of Local Search for Low Rank Matrix Recovery
We show that there are no spurious local minima in the non-convex factorized parametrization of low-rank matrix recovery from incoherent linear measurements. With noisy measurements we show all local minima are very close to a global optimum. Together with a curvature bound at saddle points, this yields a polynomial time global convergence guarantee for stochastic gradient descent from random initialization.

In this paper the authors consider a low-rank matrix approximation problem where the low-rank constraint is enforced directly by working in the factorized space and optimizing over a matrix U to solve a problem of the form min_U |A(UU^T) –y |_F^2 where A() is a linear operator.  Since working in the factorized space obviously makes the problem non-convex, the main contribution of the paper is to show under what conditions a local descent algorithm will converge to a global optimum, with the key requirements being that the linear operator, A(), needs to satisfy the restricted isometry property.
Prior work has shown that local descent is guaranteed to converge to a global minimum for the same problem under similar conditions.  This prior work, however, relied on a careful initialization of the variables in order to guarantee their results, while the current work does not have this requirement and the variables can be initialized randomly.  Additionally, the current work also achieves slightly better bounds on the necessary level of restricted isometry required by the linear operator than some of the prior work.  While the contribution is thus in some sense an incremental improvement over this prior work, the authors clearly review the relevant literature and establish where their contribution falls relative to this work.  In addition to the theoretical contribution, the authors also provide experimental evidence that performing local descent from random initializations successfully achieves the global optimum under the same conditions as using a specially chosen initialization (based on a SVD), which was necessary for results from the prior work.   1) Overall, the technical content of the paper appears to be well established, and I do not have many significant comments in that regard.  I do, however, have a question regarding the proofs of some of the lemmas.  First, the proof of Lemma 4.2 begins by requiring a factorization of U into U=QR where U has orthonormal columns and R is invertible; however, if U is rank-deficient it seems that such a factorization will not exist.  Please discuss the correctness of Lemma 4.2 in the case where U is rank-deficient.  2) In the proof of Lemma 4.3, I did not work out the algebra in detail, but it would seem that in the equations following line 202 there should be terms that contain UR^T(U^*)^T and U^*RU^T that arise from the ZZ^T term.  Where have these gone?  In general, many of the proofs skip significant steps and would be much clearer if more of the intermediate steps are shown.  3) Given comment 2 I would suggest perhaps moving more the proofs into the supplement where they could be expanded and described in more detail.  Then, the description of the experiments (which only appear in the supplement) can be moved into the main body of the paper.


Tagger: Deep Unsupervised Perceptual Grouping
We present a framework for efficient perceptual inference that explicitly reasons about the segmentation of its inputs and features. Rather than being trained for any specific segmentation, our framework learns the grouping process in an unsupervised manner or alongside any supervised task. We enable a neural network to group the representations of different objects in an iterative manner through a differentiable mechanism. We achieve very fast convergence by allowing the system to amortize the joint iterative inference of the groupings and their representations. In contrast to many other recently proposed methods for addressing multi-object scenes, our system does not assume the inputs to be images and can therefore directly handle other modalities. We evaluate our method on multi-digit classification of very cluttered images that require texture segmentation. Remarkably our method achieves improved classification performance over convolutional networks despite being fully connected, by making use of the grouping mechanism. Furthermore, we observe that our system greatly improves upon the semi-supervised result of a baseline Ladder network on our dataset. These results are evidence that grouping is a powerful tool that can help to improve sample efficiency.

The paper proposes a framework of unsupervised learning based on iterative denoising. The framework is designed for a scenario where multiple labels are in the same input, such as image segmentation. The basic idea of the proposed approach is analogous to an EM-like approach to learn a mixture model, where grouping assignments (mixture coefficients) and components are inferred over alternating iterations. The proposed framework takes a corrupted input, and applies as amortized inference a denoising model to estimate the groups and their assignments. The obtained estimations are further fed into the next iteration. The learning objective is the negative log-likelihood of the approximation of the true posterior of denoting, which is modeled here by the mixture of groups. Using artificially generated Shapes and Textured MNIST datasets, the paper evaluates the clustering and semi-supervised classification performance. The empirical results indicate the proposed approach achieves significantly better result in both supervised and semi-supervised scenario, especially when multiple objects are present in the image.
The proposed approach looks unique and interesting. The framework combines the generative model with fast amortized inference to derive efficient iterative approach. The denoising objective naturally fits in the problem setting to make learning feasible both under semi-supervised and unsupervised scenario. This technical proposal looks novel and I believe highly contributing.  The empirical results look also promising. Although the datasets are artificially generated and do not resemble a real-world situation as suggested in Fig 1, the significant performance improvement (Table 1) clearly indicates that the grouping is really helping in identifying multiple target objects in the same input. The experimental design is reasonable considering the purpose of the study.  My biggest complaint is the lack of details in the paper. Sec 2 and 3 do not reveal enough detail of how the denoising objective and the parametric mapping is designed, and I had hard time understanding the model. Especially, m_BU in Fig 2 is never explained in the paper, and readers would have no clue what the mapping model takes as input and output. Also, although the objective is modeled after denoising, the proposed model is quite different from a standard denoising auto-encoder and the paper should not simply state ``we apply denoising auto encoders [20]’’ (Sec 2) since the model takes latent variables m and z from the previous iteration.  I am curious why the paper states *perceptual* groups emerge out of the proposed model. The model seems to simply consider latent source signals in terms of denoising objective, and nothing constrains the groups. I can hardly imagine how the proposed model can lead to the segmentation example in Fig 1, or how the proposed model solves the ``dynamic, ambiguous and task-dependent’’ (Sec 1) grouping. To me, the proposed approach looks like a new, general generative framework based on denoising objective, and not tied to *perceptual* grouping as suggested in Sec 1.  One related question is the robustness to initialization. The paper does not describe how the latent variables are initialized in the experiments nor how one should initialize m and z in general. Perhaps random initialization can work fine in the synthesized datasets, but I do not expect the same in much more complicated input like the segmentation example in Fig 1. Also, it is not clear if the proposed approach always converges.  In overall, I would still give a positive rating. The paper is making a good contribution in proposing a novel clustering framework based on denoising formulation.


Residual Networks Behave Like Ensembles of Relatively Shallow Networks
In this work we propose a novel interpretation of residual networks showing that they can be seen as a collection of many paths of differing length. Moreover, residual networks seem to enable very deep networks by leveraging only the short paths during training. To support this observation, we rewrite residual networks as an explicit collection of paths. Unlike traditional models, paths through residual networks vary in length. Further, a lesion study reveals that these paths show ensemble-like behavior in the sense that they do not strongly depend on each other. Finally, and most surprising, most paths are shorter than one might expect, and only the short paths are needed during training, as longer paths do not contribute any gradient. For example, most of the gradient in a residual network with 110 layers comes from paths that are only 10-34 layers deep. Our results reveal one of the key characteristics that seem to enable the training of very deep networks: Residual networks avoid the vanishing gradient problem by introducing short paths which can carry gradient throughout the extent of very deep networks.

The authors give a novel interpretation of deep residual networks, a feed-forward architecture that has received significant attention from the research community recently. They state that residual networks are ensembles of relatively shallow networks. They provide detailed experiments to underpin their interpretation.
The paper approaches an important and interesting question with a novel methodology. It's an inspiring, thought-provoking read, especially for those of us interested in residual networks. It provokes us into looking at resnets (and deep nets in general) from a different perspective. The single big issue I see with it is that its claims are either not formalized enough, and inasmuch as they are formalized, they are not sufficiently justified by the (admittedly extemely well-thought-out and informative) experiments presented. Below are a couple of examples to make this critique a bit more specific.  lines 157-158: Here the authors argue for the ensemble view by invoking the observation that single blocks can be removed from the network without significant performance degradation. I claim that this line of argument is not logically sound. A simpler explanation of the robustness phenomenon is that the magnitude of individual residual outputs is small compared to their summed magnitude. (I've actually conducted experiments on deep residual networks that verify this.) Of course, one could use the word "ensemble" in such a general sense that this alternative explanation (of small individual resnet block magnitudes) is not in fact an alternative explanation, but rather, it supports the ensemble claim. I'd say that in this case the ensemble claim is too vacuous. In any case, it's never formalized enough so that the reader can decide.  One other way of highlighting the IMHO problematic line of argument is this: A particularly  useless kind of ensemble is the echo chamber, where the individual elements of the ensemble all vote the same way. I believe that the arguments and experiments presented in the paper never successfully exclude the possibility that the observed ensemble is of this kind.  A related observation: In principle, adding a constant vector "drift" to the output of a residual block does not hurt the learning capabilities of the system: the following blocks simply must learn to work on a translated version of their original input domain. But -- unless those blocks learn to operate both on the translated and the original domain, an a priori unexpected form of generalization -- the artificial removal of the residual block will damage the system, because it removes the drift as well. The experiments presented in the paper clearly show that this degradation does not actually happen: there's no significant drift. It would be interesting to understand this phenomenon better. Is it related to the use of ReLUs or batchnorm? Or is it something more general? I have a vague general issue with the intuition behind the "unraveled view" of the network. The easiest way to state this is through a crude analogy: We can look at the "unraveled view" of the multilinear formula (1+x_1)(1+x_2)...(1+x_n): a polynomial in standard form, with 2^n tems. It's hard to imagine what the unraveled view adds to the discussion in this specific case. When compared to the 1-multiplicity formula x_1x_2...x_n, there are very few meaningful senses of the word in which our 2^n-multiplicity multilinear formula is exponentially more complex. I'm aware that multiplication and function composition are mixed here. It's an imperfect analogy. Still, I would recommend the authors to defuse similar kinds of objections in advance.  The above criticisms should not obscure the fact that I've found the paper thought-provoking, the experiments well-designed, and the potential impact high.


A Bayesian method for reducing bias in neural representational similarity analysis
In neuroscience, the similarity matrix of neural activity patterns in response to different sensory stimuli or under different cognitive states reflects the structure of neural representational space. Existing methods derive point estimations of neural activity patterns from noisy neural imaging data, and the similarity is calculated from these point estimations. We show that this approach translates structured noise from estimated patterns into spurious bias structure in the resulting similarity matrix, which is especially severe when signal-to-noise ratio is low and experimental conditions cannot be fully randomized in a cognitive task. We propose an alternative Bayesian framework for computing representational similarity in which we treat the covariance structure of neural activity patterns as a hyperparameter in a generative model of the neural data, and directly estimate this covariance structure from imaging data while marginalizing over the unknown activity patterns. Converting the estimated covariance structure into a correlation matrix offers a much less biased estimate of neural representational similarity. Our method can also simultaneously estimate a signal-to-noise map that informs where the learned representational structure is supported more strongly, and the learned covariance matrix can be used as a structured prior to constrain Bayesian estimation of neural activity patterns. Our code is freely available in Brain Imaging Analysis Kit (Brainiak) (https://github.com/IntelPNI/brainiak).

The manuscrit presents a technical bias that has impaired accurate estimation of the structure of similiarity between neural representations, i.e. the similarity between neural representations of distinct objects or percepts, from neuroimaging data (especially fMRI data). The manuscript first details mathematically how the bias emerges in the covariance matrix, especially when signal-to-noise ratio is low and the timing of presentations of the different objects are not well balanced. Then authors propose a new method for structure inference based on maximum likelihood estimation from a generative model of neural data with various parameters (covariance structure, SNR in all voxels, variance and autoregressive coefficient of noise). This method naturally gets rid of the estimation bias, as is shown with synthetic data. It is finally applied to a fMRI dataset showing similarity between a behaviourally derived and a neurally derived similarity structure of representations of animals in human subjects.
The manuscript is very clearly written, the method is impressive from the technical point of view and it seems to address very appropriately a real issue for analysis of neuroimaging data. I expect that it can be of great use to many neuroscientists by improving the estimation of similarity structure from neural activity. Moreover the perspective of having improved measure of betas in fMRI (notably taking into account the bias in covariance matrix and the smooth prior for SNR) is extremely promising.   This being said, here is a series of additional comments: -The authors argue that alternate methods are not well grounded, but it would be nice to have the illustration in Figure 2 that the new method actually performs better at recovering the covariance structure in synthetic data. - I have a  concern regarding the spatial independence of noise. If noise is not spatially independent (and it generally isn't, as acknowledged in the discussion), this will create some consistent overestimation the correlational structure. Will this affect more the estimation than the original bias that the method seeks to get rid off ? In other terms, is the cure worse than the disease ? - It would be great if the code could be made public, and perhaps interfaced with one of the popular fMRI analyses softwares/toolboxes such as SPM.  - maybe in figure 3 correlation between behaviourally and neural obtained matrices could be used as a measure of the correspondance between the two.  *Writing/typos: - L35-36 “ what stimuli have very different representations” → rephrase - L41 “demonstrations” -L42 “highly similar representational structure” unclear - L59-60 maybe specify that when Y is white noise, there is still structure in design matrix X (which is where the 'representational' structure comes from). It cost me quite a lot to understand that point. - L94 “is”->”are” L214 “advantage not available” → either “advantage over” or “feature not available” L230 “intensities” L250 “experiment design”  [EDIT AFTER REBUTTAL] all my concerns were convincingly addressed, except the one about testing the bayesian RSA against alternative methods described l268-287 on synthetic data.


Computational and Statistical Tradeoffs in Learning to Rank
For massive and heterogeneous modern data sets, it is of fundamental interest to provide guarantees on the accuracy of estimation when computational resources are limited. In the application of learning to rank, we provide a hierarchy of rankbreaking mechanisms ordered by the complexity in thus generated sketch of the data. This allows the number of data points collected to be gracefully traded off against computational resources available, while guaranteeing the desired level of accuracy. Theoretical guarantees on the proposed generalized rank-breaking implicitly provide such trade-offs, which can be explicitly characterized under certain canonical scenarios on the structure of the data.

This paper considers the problem of rank aggregation. Given ordinal data about users’ preferences, the goal is to aggregate them in the form of partially ordered sets (a poset). Assuming the revealed preference is consistent, the poset can be represented as a directed acyclic graph.  The authors propose a “generalized rank-breaking” algorithm, which is a unified framework which spans from the maximum likelihood estimator to pairwise rank-breaking. The MLE is consistent, but can be intractable and impractical. A common remedy is to use pairwise rank-breaking, which treats all pairwise comparisons as independent. However, this might lead to a statistically inconsistent estimator due to the ignored correlations.  The proposed algorithm broadly does the following: Step 1) For each user j \in [n], extract a maximal ordered partition P_j of S_j which is consistent with her preference G_j;  Step 2) Represent the extracted ordered partition by a directed hypergraph G_j(S_j, E_j) with e = (B(e), T(e)) \in E_j being a directed hyperedge;  Step 3) Compute the probability of B(e) < T(e) in terms of parameters \theta_i (latent true utility);  Step 4) Maximize the log-likelihood composed of the terms only with |T(e)| \leq M under the independent assumption.   The authors show that finding an order-M rank-breaking estimate is a concave maximization with high probability, and that the estimate is consistent for sufficiently large M under the Plackett-Luce model.   The main result of the paper suggests a high-probability upper bound on the achieved error of the estimate. This upper bound is optimal up to a logarithmic factor with respect to the Cramer-Rao bound when (a) the topology of the data is well-behaved (quantities defined on line 206 are finite), and (b) there is no limit on the computational power and M \to \infty.
Overall, the paper is well-organized: the process for generalized rank-breaking is clearly explained and the analysis provides an upper bound on the achieved error. It is also shown that this upper bound is nearly optimal under certain conditions, and the computational-statistical trade-off is also glimpsed. However, there are many parameters/quantities involved in the analysis so that sometimes it becomes not easy to follow and appreciate the authors’ arguments. I would like to suggest having an easy-to-find notation paragraph which readers can refer to, while reading.  Also, there are several questions with regard to the contents of the paper: - Line 52:   Can this generalized rank-breaking be applied beyond the Plackett-Luce model? - Line 98 – 99:  I do not understand why none of the pairwise orderings can be used from G_j in the example. - Line 206:  Does $\gamma_3$ also have an intuitive interpretation like $\gamma_1$ and $\gamma_2$? Also, is there any special reason you use $\nu$, not $\gamma_4$ to define the last quantity? Isn’t that related to the topology of the data? - Lines 244 – 251:  I am a little bit confused. Lines 244 – 247 implies the upper bound is tight with respect to the lower bound when the topology of the data is well-behaved. However, Lines 248 – 251 seem to mean the opposite: Eq (13) gives a tighter lower bound when the topology is not well-behaved. Could you please clarify what I am misunderstanding for me? - Line 259:  I am not following where in the analysis the increase of the error accompanied by the growth of m is predicted.   Typoes: - Line 169:  One -> On - Line 188:  position of 'and'


Diffusion-Convolutional Neural Networks
We present diffusion-convolutional neural networks (DCNNs), a new model for graph-structured data. Through the introduction of a diffusion-convolution operation, we show how diffusion-based representations can be learned from graphstructured data and used as an effective basis for node classification. DCNNs have several attractive qualities, including a latent representation for graphical data that is invariant under isomorphism, as well as polynomial-time prediction and learning that can be represented as tensor operations and efficiently implemented on a GPU. Through several experiments with real structured datasets, we demonstrate that DCNNs are able to outperform probabilistic relational models and kernel-on-graph methods at relational node classification tasks.

The authors propose a novel neural network architecture for prediction problems of graphs. They interpret this as a generalization of convolution beyond grid structures. The method is shown to outperform several kernel methods. It performs better on vertex classification, but not on graph classification.
The experimental results look promising, but I have several concerns regarding the presentation and the experiments.  Presentation: - How exactly is diffusion defined here? This is such a core concept, why is there not a formal definition or at least a citation to previous work? I spent 10 minutes googling this, and I haven't found a single obvious definition. I'm not an expert on graphs / social network analysis, but this still seems to be an important omission. In any case, I assume there is some standard way to featurize a node's neighbors that is used here. - Authors strongly emphasize how this work generalizes convolutions. I think this is only true in the graph setting, not in the edge or vertex prediction settings (e.g. I don't see where is the spatial invariance in those settings). Since the contribution is better performance on vertex classification, I don't think it's appropriate to emphasize this story.  Experiments: - One of the claims is that this architecture handles vertex, edge, graph prediction in a unified way. Why are there no edge experiments? There should be at least a comment on this. Also, can it be used for structure learning? - On vertex prediction, the authors compare against baselines including kernel methods, logistic regression and a CRF. But logistic regression only receives features derived from the vertex itself, not its neighbors, so it's not surprising they don't do well. There needs to be experiments that trains a shallow model on the same signal (i.e. on features that are comparable to the diffusion-based features used here). Then we need to see that the neural network model is more accurate or faster. The same holds for the CRF; the authors don't specify which features it uses, but even if it uses edge features, we need to compare against an equivalent featurization. The authors mention they use up to 5 hops of diffusion, which sounds to me like they're looking at neighbors 5 hops away. The CRF and LR need to receive the same features. - I'm not an expert on neural networks applied to graphs, but I'm surprised there are not other neural-network based models to compare against. Why is something like this work not applicable and/or compared against: https://cs.stanford.edu/people/jure/pubs/node2vec-kdd16.pdf  In conclusion, I find the architecture to be elegant and the paper very well-written (except for omitting the definition of diffusion), but the experiments are not entirely convincing, and this is the main factor by which we can judge an applied paper.  Right now, I think this paper is somewhat borderline, but I am happy to readjust my opinion if the authors can comment on some of the above.  UPDATE:  After reading the response and looking closer to related literature, I am down-grading my review.  I strongly disagree with the authors' claim that "many of [other] techniques were developed in parallel with our own and did not have readily available open-source implementations". For example, look at the node2vec paper that came out in parallel: - They have substantially more extensive experiments - They compare against previous neural network based methods, LINE and DeepWalk - These methods are from 2014 and 2015, so I see no excuse for not comparing against them. In fact, they are not even cited, which I think is unacceptable!  The quality of this paper is much lower than that of LINE, DeepWalk and node2vec, and so I strongly argue for the rejection of this paper.


Long-term causal effects via behavioral game theory
Planned experiments are the gold standard in reliably comparing the causal effect of switching from a baseline policy to a new policy. One critical shortcoming of classical experimental methods, however, is that they typically do not take into account the dynamic nature of response to policy changes. For instance, in an experiment where we seek to understand the effects of a new ad pricing policy on auction revenue, agents may adapt their bidding in response to the experimental pricing changes. Thus, causal effects of the new pricing policy after such adaptation period, the long-term causal effects, are not captured by the classical methodology even though they clearly are more indicative of the value of the new policy. Here, we formalize a framework to define and estimate long-term causal effects of policy changes in multiagent economies. Central to our approach is behavioral game theory, which we leverage to formulate the ignorability assumptions that are necessary for causal inference. Under such assumptions we estimate long-term causal effects through a latent space approach, where a behavioral model of how agents act conditional on their latent behaviors is combined with a temporal model of how behaviors evolve over time.

This paper addresses an issue that is common in randomized controlled trials (or A/B tests): inferences may be invalid if there is nonstationarity, and in particular if users strategically respond dynamically to the differences in the two cells.  The authors attempt to calibrate a strategic model using short-run data to then estimate a long-run causal effect.
 ---The way you refer to "causal effect" and "the fundamental problem of causal inference" is somewhat nonstandard.  Typically (in the Rubin potential outcomes model, which is what you are building on), the causal effect is defined at the individual level, with a "treatment" outcome and "control" outcome for each experimental unit.  The fundamental problem of causal inference is that only one of these two outcomes is actually observed for each experimental unit.  You seem to be focusing on a slightly different issue, which is that the effect of treating the entire population cannot be determined correctly from just data when half the population is treated.  It seems to me that this issue -- which can arise due to a variety of violations of the SUTVA assumption -- can exist independent of whether there is a multiagent interaction.  Conversely, it seems multiagent considerations are relevant even when defining causal effects at the sub-population level.  ---Your references are somewhat limited to recent work.  There has been extensive work on causal inference in nonstationary environments, particularly by econometricians and time series analysts.  It would be good to situate your work better relative to this work.  Your main difference lies in the particular structure of behavioral model you assume, and I judged the paper on this basis.   ----A significant limitation that I see is that the modeling approach requires games and behaviors to be defined before the estimation procedure can be carried out.  One could, instead, imagine a fully nonparametric approach to causal inference that just aimed to remove any nonstationarity in treatment and control before estimating a treatment effect.  The approach you take requires that one should believe constructing the necessary spaces of games and behaviors would be both reasonably specificed, and computationally tractable.  ---The main contribution is in the modeling, as the theorem follows naturally from the modeling assumptions.  The theorem is making an asymptotic statement imprecisely; it would be better to formally state the result as well.  ---The numerics are carried out for a relatively simplified two player game, whereas the motivating examples are all drawn from ad auctions.  I would like to understand whether this approach can be reasonably used in a more practical setting.  For example, in an ad auction setting, how would you restrict the behaviors that you consider?  As the paper currently stands, the example is a bit too simple, considering the significant model complexity that would arise if one tried to capture all behaviors possible in more complex (and real-world) strategic environments.  ---Overall, I thought the authors took an intriguing approach to a very difficult problem.  I was generally left with the impression that the work is promising but still preliminary, and would benefit from some deeper investigation of the applicability of the methods, as well as comparison to the very extensive related literature.


Differential Privacy without Sensitivity
The exponential mechanism is a general method to construct a randomized estimator that satisfies (ε, 0)-differential privacy. Recently, Wang et al. showed that the Gibbs posterior, which is a data-dependent probability distribution that contains the Bayesian posterior, is essentially equivalent to the exponential mechanism under certain boundedness conditions on the loss function. While the exponential mechanism provides a way to build an (ε, 0)-differential private algorithm, it requires boundedness of the loss function, which is quite stringent for some learning problems. In this paper, we focus on (ε, δ)-differential privacy of Gibbs posteriors with convex and Lipschitz loss functions. Our result extends the classical exponential mechanism, allowing the loss functions to have an unbounded sensitivity.

Exponential mechanism is a universal differentially private algorithm, i.e., every differentially private algorithm can be represented as an exponential mechanism. Exponential mechanism requires defining a loss (score) function over the output space parameterized by the data set, and then sampling a candidate from a distribution induced by the score function. Often times the loss function is required to be of bounded range for the privacy analysis. This paper tries to alleviate this concern by sampling from a Gibb’s posterior distribution, with strongly convex prior. The strong convexity of the prior helps to control the tail behavior of the sampling distribution, and hence obtain a (\epsilon,\delta)-differential privacy guarantee.
One major concern I have with the paper is the amount of complexity introduced. I am not convinced whether a simpler algorithm / analysis would suffice. In the following, I qualify my above statement.  In the notation of the paper, let L = \sum \ell(\theta;d_i). If \theta \in R_\Theta (a bounded convex set) and \ell is L-Lipschitz, then one can show that sampling from exp(-\eps/\beta L(\theta)) is \eps differentially private (see [3]). Furthermore, adding a strongly convex regularizer should remove the bound on R_\Theta. by the same argument described in this paper. How does this compare to Section 3.2 in this paper?  On a general note, how the utility guarantees of this paper compare to the existing results [8, 14, 3]?  More specific comments are as follows:  1. I believe the idea of introducing a strongly convex prior to control the tail distribution of exponential mechanism is an interesting idea. Also the analysis to control the tail behavior of exponential mechanism (in order to obtain a bound on the privacy parameter \delta), seems novel in my opinion.    2. I am concerned with the title mentioning 'without sensitivity'. It seems too catchy to be factually correct. The paper does assume a bound on the gradient of the loss function (instead of the value of the loss function). Even the bound on the gradient can be considered as sensitivity.  3. In  Line 31, the paper forgot to mention that [3] also does posterior sampling on the output distribution via exponential mechanism.  4. The exposition of the shrinkage concept (in lines 57-61) can be improved for readability by describing it in terms of regularization.  5. The related work requires significant work. The paper needs to describe the prior work in much more detail in order to place the novelty of their results in perspective. The area of private convex optimization is very well studied.  6. In Line 153, I am concerned with the value of the delta parameter. Typically, for semantic correctness of differential privacy \eps is a small constant (around 0.1) and \delta is a value much small than 1/ n^2 (n is the number of data samples).  7. The bound in Line 199 needs more justification.  Since in machine learning typically \lambda is chosen to be around 1/\sqrt n, \beta seems to  scale as n^{1/4}. Is this good or bad ?  8. In Line 42 it should be analysis instead of analyzes.


Learning Kernels with Random Features
Randomized features provide a computationally efficient way to approximate kernel machines in machine learning tasks. However, such methods require a user-defined kernel as input. We extend the randomized-feature approach to the task of learning a kernel (via its associated random features). Specifically, we present an efficient optimization problem that learns a kernel in a supervised manner. We prove the consistency of the estimated kernel as well as generalization bounds for the class of estimators induced by the optimized kernel, and we experimentally evaluate our technique on several datasets. Our approach is efficient and highly scalable, and we attain competitive results with a fraction of the training cost of other techniques.

This work presents a method to speed up kernel learning using random features (RF). With respect to known approaches, e.g. structured composition of kernels with respect to an alignment metric and joint kernel composition optimization based on empirical risk minimization, it achieves higher efficiency by optimizing compositions of kernels using explicit RF maps rather than full kernel matrices. The proposed method is divided in 2 steps: First, the kernel is learned efficiently using RF. Then, the optimized features associated to the learned kernel are used in a standard supervised learning setting to compute an estimator. Authors prove the consistency of the learned kernel and generalization guarantees for the learned estimator. An empirical evaluation of the method is also provided, including: - A toy example showing the learned features in the case of a bad initial kernel guess - An experiment involving a high-dimensional dataset, showing that the method attains higher predictive performance and induces a sparse representation, which can be potentially useful for interpretability - A performance evaluation on benchmark datasets, showing comparable test accuracies at a fraction of the computational cost with respect to the standard RF approach
The proposed approach is very interesting and novel. The idea of using random features to speed up kernel alignment is brilliant. The paper is well written and organized in a principled way. The authors provide a theoretical analysis of the method, guaranteeing consistency of the learned kernel and generalization properties of the resulting estimator. The experimental part allows to better visualize the method, shows how it can be used to identify sparse features in high dimensions and compares the accuracy and computational time of the proposed method on 3 benchmark datasets. Regarding the latter point, benchmarking experiments highlight promising practical applications for the method. As an improvement, I would suggest to evaluate the method on more benchmark datasets, for instance some of the ones used in [18].   Minor issues:   37 (?): optimization --> optimization problem   76, 83, 89: Properly introduce (or change) $W$ and $W^k$, highlighting the relationship with lower-case $w^i$  87: need only store --> only needs to store  98: missing ")" the last element of $z^i$  189: one-hot --> one-shot  Algorithm 1: Add "end" at the end of each while cycle  Figure 1: It is not clear why the optimized random features $w^k$ clustered around (-1,-1) and (1,1), depicted in yellow in the figure, would indicate a good solution.


Achieving Budget-optimality with Adaptive Schemes in Crowdsourcing
Adaptive schemes, where tasks are assigned based on the data collected thus far, are widely used in practical crowdsourcing systems to efficiently allocate the budget. However, existing theoretical analyses of crowdsourcing systems suggest that the gain of adaptive task assignments is minimal. To bridge this gap, we investigate this question under a strictly more general probabilistic model, which has been recently introduced to model practical crowdsourcing datasets. Under this generalized Dawid-Skene model, we characterize the fundamental trade-off between budget and accuracy. We introduce a novel adaptive scheme that matches this fundamental limit. A given budget is allocated over multiple rounds. In each round, a subset of tasks with high enough confidence are classified, and increasing budget is allocated on remaining ones that are potentially more difficult. On each round, decisions are made based on the leading eigenvector of (weighted) non-backtracking operator corresponding to the bipartite assignment graph. We further quantify the gain of adaptivity, by comparing the tradeoff with the one for non-adaptive schemes, and confirm that the gain is significant and can be made arbitrarily large depending on the distribution of the difficulty level of the tasks at hand.

The paper looks at adaptive schemes in crowdsourcing and analyzes theoretically the trade-off between cost and accuracy under a generalized Dawid-Skene model. It also proposes an adaptive algorithm that achieves the theoretical limit and is strictly better than any non-adaptive algorithm.
 The paper provides a theoretical analysis of adaptive schemes in crowdsourcing. This is quite interesting work as many papers about crowdsourcing do not give any theoretical guarantees for their proposed solutions.  I am not very well-versed in the theoretical aspects mentioned in this paper, so I found it quite dense and had difficulty following the proofs. I will therefore focus in my comments on the aspects that would be of greater interest to a practitioner of crowdsourcing.  The algorithm: The description/pseudocode is not easy to understand. The explanation in the text helps considerably, however some details are not explained (for example, why are X and s_t set the way they are?). Also, what is the running time? Is it impractical to run on large data sets, or is it fast enough?  The simulation results focus on quite large numbers of queries per task. In practice it is uncommon to use more than 5 or 10 workers per task. A smaller range is not shown in the graphs (it's hard to say at what number they actually start, but looks like 20) and this range is also where the adaptive algorithm seems to perform the same or worse than the non-adaptive. Does this mean that the proposed algorithm is mainly of theoretical interest? Is it better in practice to stick with the non-adaptive version for small number of queries per task? It would have been nice to see the x-axis in the charts starting from 2 or 3 and go higher.   Summary/conclusions section is missing. It should summarize what was achieved and give directions to other research that naturally follows these results.     There are some problems in the references:  - The formatting is mixed. There are authors with full first name and others with initial. There are titles missing (e.g. [15]). - Many references are not mentioned in the text: [1, 2, 5, 6, 9, 12, 13, 17, 20, 21] - [10] and [11] are the same reference. [15] and [16] are also probably the same reference.     Other issues: - line 53: support - line 101: simulations confirm - both figures: the caption should explain what is on the left vs right - line 237: Theorem 2.4 is the incorrect number - line 250: at least - line 284: error rate ... is dominated - the supplementary material is out of order, theorem 2.2 is first, but 2.1 is last. - why does the proof of lemma 3.1 have its own section 4?


Gradient-based Sampling: An Adaptive Importance Sampling for Least-squares
In modern data analysis, random sampling is an efficient and widely-used strategy to overcome the computational difficulties brought by large sample size. In previous studies, researchers conducted random sampling which is according to the input data but independent on the response variable, however the response variable may also be informative for sampling. In this paper we propose an adaptive sampling called the gradient-based sampling which is dependent on both the input data and the output for fast solving of least-square (LS) problems. We draw the data points by random sampling from the full data according to their gradient values. This sampling is computationally saving, since the running time of computing the sampling probabilities is reduced to O(nd) where n is the full sample size and d is the dimension of the input. Theoretically, we establish an error bound analysis of the general importance sampling with respect to LS solution from full data. The result establishes an improved performance of the use of our gradientbased sampling. Synthetic and real data sets are used to empirically argue that the gradient-based sampling has an obvious advantage over existing sampling methods from two aspects of statistical efficiency and computational saving.

The authors propose a sub-sampling algorithm for least squares regression, where the weights depend on the magnitude of the gradient. They argue that by using these weights, rather than uniform weights or weights based on leverage, the resulting regression estimates are more accurate. They bound the error of the resulting regression coefficients and they show that their method is optimal, in the sense that it minimizes a constant. They illustrate the performance of the method using a simulated example and two real datasets.
I don't think that the paper provides convincing arguments as to why using the gradient to determine the sub-sampling probabilities is a good idea. In particular, I worry that selecting mostly points characterised by big residuals will lead to un-robust  regression estimates. In addition, are the weights invariant to the scale of the covariates? The quadratic form used to calculate the leverage is invariant, but I don't think that the gradient is. Hence, the results might depend on the scale used.  Looking at Example 1: notice that the errors are homoscedastic here. I guess that under  heteroscedasticity gradient-based sampling would do quite badly. Under heteroscedasticity one typically wants to down-weight observations where the variance is large. On the contrary, gradient-based sampling will give large weights to these points, because they have big residuals. This issue should be particularly problematic when the number of sub-samples (r) is small. Hence, before using the proposed strategy one has to be quite sure that the errors are homoscedastic.  The title might be a bit misleading at the moment: I don't see in which sense the proposed approach is adaptive and I also don't see where the importance sampling part comes from (there seems to be no notion of an importance and a target distribution here). I would say that the paper focuses simply on gradient-based sub-sampling.  In Theorem 1: I find a bit puzzling that if we plug r = n in formula (6) we do not get zero that the error is zero, as we should.   In Section 5.1. How large is d here? The number of covariates is not reported.  MINOR POINTS:  L 15: "has obvious advantage" use either "has obvious advantages" or "has an obvious advantage".  L18: "Modern data analysis always meets enormous data sets in recent years." maybe "meets" is not the correct word here.   L22: "uniform sampling is simple but trivial way" maybe better "uniform sampling is a simple but trivial way".  L28-29: "Another shortcoming is that it needs to cost much time" better "Another shortcoming is that it costs much time".  L46: "we establishes"  L64: "with d-dimensional predictors" here there are d predictors, but they are not d-dimensional  L114: "and argue that choosing the initial size of r 0 to get a pilot estimate may not be careful" do you mean that one does not need to be careful when choosing r0?  L116: Poisson should be have capital P.  L128: ", we do simulation studies and command that it is an efficient sampling method from the algorithmic perspective." maybe you mean "find" rather than "command".  L211: "we compare the performance of various sampling on two" add "algorithms" after "sampling".  Table 3: what are the TS-PS entries in the CASP table?  L223: "which supply a rationality for the algorithm" maybe "justification" rather than "rationality". Also, "supplies" rather than "supply".  L225: "since the gradient-based sampling algorithm is implied without linear model assumption" maybe "justified" rather than "implied"  L231: "since gradient is considered a steepest way to go to the (local) optima." This sentence needs to be re-written.  L231: "optimzation".  L231: "Thus, applying this idea into other" use "to" rather than "into".


Protein contact prediction from amino acid co-evolution using convolutional networks for graph-valued images
Proteins are responsible for most of the functions in life, and thus are the central focus of many areas of biomedicine. Protein structure is strongly related to protein function, but is difficult to elucidate experimentally, therefore computational structure prediction is a crucial task on the way to solve many biological questions. A contact map is a compact representation of the three-dimensional structure of a protein via the pairwise contacts between the amino acids constituting the protein.
We use a convolutional network to calculate protein contact maps from detailed evolutionary coupling statistics between positions in the protein sequence. The input to the network has an image-like structure amenable to convolutions, but every "pixel" instead of color channels contains a bipartite undirected edge-weighted graph. We propose several methods for treating such "graph-valued images" in a convolutional network. The proposed method outperforms state-of-the-art methods by a considerable margin.
Protein structure is mediated and stabilized by series of weak interactions (physical contacts) between pairs of its amino acids. Let L be the length of the sequence of a protein (i.e. the number of its amino acids). The tertiary structure can be partially summarized as a so-called contact map -a sparse L × L matrix C encoding the presence or absence of physical contact between all pairs of L amino acid residues of a protein. The entry C i,j is equal to 1 if residues i and j are in contact and 0 if they are not. Intermediate values may encode different levels of contact likeliness.
We use these intermediate values without rounding where possible because they hold additional information. The "contact likeliness" is a knowledge-based function derived from Protein Data Bank, dependent on the distance between Cβ atoms of involved amino acids and their type. It has been parametrized based on the amino acids' heavy atoms making biophysically feasible contact in experimentally determined structures.

A novel application of convolutional neural networks to protein co-evolution matrixes is presented that performs significantly better than state of the art approaches for evolutionary conservation based contact prediction. This method has the potential to directly improve ab initio structure prediction, which is an important and unsolved problem in biophysics and machine learning.
Overall the paper presents a novel application of CNNs to co-evolution Protein Contact Prediction (PCP) with state-of-the-art results. The paper is a strong contribution to the field of coevolution PCP. The method has the potential to improve ab initio structure prediction accuracy, which could have wide ranging impacts on studying proteins without solved structural complexes.  The paper itself contains flaws in the experimental setup, how the methods are described, and how the results are presented.  Key issues: - Experimental setup: not enough unique training examples are used to compare to existing methods. More than 100K examples exist for this specific task, but only 307 are used. - A key related method (PconsC2) is missing from the experimental benchmarks. Method description: the model is not described in sufficient detail to be reproduced by a 3rd party with a reasonable level expertise. In particular, the loss function and model parameters are not sufficiently described. - Results presentation and interpretation: plots are hard to read because underlying data points are too lightly printed, too few examples are used for results comparisons, and several key claims are not sufficiently supported by data.  Specific points  - The number of examples used for training and testing is very small, especially when compared to numbers of examples used in related publications. For example, in the plmDCA paper, 10,000 examples are used, rather than the 307 in this paper. The authors claim they had trouble finding consistent homologue sets for all proteins, but there are exisiting pipelines for this task (e.g. the gremlin pipeline http://gremlin.bakerlab.org/). It would be especially interesting to see how the relative performance of the method changes when trained on a larger corpus of training examples, especially considering that PDB contains > 100K unique crystal structures that could each potentially be used as an example.  - No justification is given for the binary contact/non-contact threshold of 8A (previous comparisons have used different cutoffs). It is likely that cutoffs at different resolutions will be useful for ab initio structure prediction. It would be interesting to see a comparison of pmlConv to plmDCA and CONSIP2 with different cutoff boundaries. It is potentially a concern that the cutoff was chosen because it maximizes the margin of plmConv over other methods, rather than because it corresponds to a physically important distance.  - The model should also be benchmarked against PconsC2 (Skwark M., et al., “Improved Contact Predictions Using the Recognition of Protein Like Contact Patterns”, PLOS Computational Biology, 2014). This would be an especially informative comparison because PconsC2 uses an ensemble of linear classifiers for the exact same task (PCP from plmDCA matrixes). PconsC2 also includes learned spatial filters, which are missing from the current benchmarks.  - The deep learning model is not adequately described to be reproducible. Especially important is that the paper does not mention what loss function is used for optimization (presumably binary cross entropy?) Also missing are descriptions of what convolutional strides and padding is used, what types of initialization are used (probably very important for learning on limited examples), whether batch norm is used, and whether dropout is used.  - No interpretation is presented of what features the model learns. It would be interesting to apply a technique such as guided back propagation to input to provide an interpretation of which features are most important for classification of a given residue pair.  - In figures 2 and 3, it is very hard to see the circles that represent individual data points. The circles should be presented with higher contrast so readers can better see the underlying data distribution, especially given that it is so sparse.  - In figure 3, the presented data are very noisy. It would be worth reproducing this figure with more alignments across a larger set of proteins.  - In figure 3, the x-axis is labeled “amount of sequences”, but the caption states the figure shows the “information content” of the alignment as the number of “non-redundant sequences”. It would be worthwhile to consider more informative metrics than the number of non-redundant sequences: an alignment in which each sequence has hamming distance exactly 1 from each other sequence is much less informative for co-evolution than an alignment with higher sequence divergence. A metric such as effective number of sequences (where sequences are weighted by normalized average hamming distance), or alignment entropy may be more informative summary statistics for this purpose.  - The paper presents a semantic argument that larger layers could lead to overfitting, but could this be overcome through parameter regularization or dropout? This would allow convolutional filters to potentially detect longer range interactions.  - The paper claims that plmConv yields higher accuracy than plmDCA for especially long range interactions. It would be interesting to plot PPV as a function of contact distance, since this figure would directly support this point (rather than the PPV table split by distance, which only partially supports this claim).  - Section 2.4 proposes 4 schemes for featurizing the J matrix, but results are not reported for each. It would be informative to report how featurization scheme influences performance.