Understanding variable importances in forests of randomized trees
Despite growing interest and practical use in various scientific areas, variable importances derived from tree-based ensemble methods are not well understood from a theoretical point of view. In this work we characterize the Mean Decrease Impurity (MDI) variable importances as measured by an ensemble of totally randomized trees in asymptotic sample and ensemble size conditions. We derive a three-level decomposition of the information jointly provided by all input variables about the output in terms of i) the MDI importance of each input variable, ii) the degree of interaction of a given input variable with the other input variables, iii) the different interaction terms of a given degree. We then show that this MDI importance of a variable is equal to zero if and only if the variable is irrelevant and that the MDI importance of a relevant variable is invariant with respect to the removal or the addition of irrelevant variables. We illustrate these properties on a simple example and discuss how they may change in the case of non-totally randomized trees such as Random Forests and Extra-Trees.

Variable importance measures are often used to
      highlight key predictor variables in tree-based ensemble models. However,
      there has generally been a lack of a theoretical understanding of these
      measures. In this paper, the authors study the theoretical properties of
      the Mean Decrease Impurity (MDI) importance measures (such as the Gini
      importance). Through most of the paper they use the Shanon entropy as the
      impurity measure but show that the theorems and results are applicable to
      any impurity measure. They begin with an asymptotic analysis of totally
      randomized fully-developed tree ensembles learned using an infinitely
      large ensemble. They show that under these conditions the variable
      importance for all variables provide a three-level decomposition of the
      information contained in the set of input variables about the output
      variable. First, they show that the information decomposes as a sum of
      importance scores of all input variables. Second, the importance of each
      variable decomposes as a sum over all degrees of interactions of that
      variable with other variables. The third level of decomposition for each
      degree is over all possible combinatorial interactions of that degree.
      Next, the authors show that that the MDI importance of a variable is equal
      to zero if and only if the variable is irrelevant and that the MDI
      importance of a relevant variable is invariant with respect to the removal
      or the addition of irrelevant variables. The authors then analyze the
      properties of the variable importance measures in depth-pruned ensembles
      of randomized trees. They show that as long as the pruning depth is
      greater than the total number of true relevant variables, these relevant
      variables still obtain a strictly positive importance score (albeit
      different from their values in fully-developed trees) and the irrelevant
      variables continue to obtain an importance of zero. Finally, for the case
      of ensembles consisting of non-totally randomized trees such as random
      forests, where top scoring predictors are selected at splits from a random
      subset of predictors, they show strong masking effects by the strongest
      variables resulting in over or under-estimation of importance scores. They
      finally show the relevance of their results through a simple yet
      insightful example of digit recognition. The paper is very
      accessible and written in a very understandable manner. At the same time,
      the theorems and results discussed are very interesting and provide some
      nice theoretical insights into variable importance measures from idealized
      asymptotic models to more realistic models. The masking effects by the
      strongest predictors and the inability of the variable importance to
      appropriately account for interactions and dependencies between variables
      for random-forest like models are very relevant to practical analyses and
      may be amplified even more in cases involving correlated or partially
      correlated variables. I think the paper lays down some nice initial
      results and a general framework to analyze more realistic models in the
      future and potentially point to better strategies for learning trees or
      computing more accurate importance scores. The paper will definitely be of
      interest to the general machine learning
      community.


Learning Feature Selection Dependencies in Multi-task Learning
A probabilistic model based on the horseshoe prior is proposed for learning dependencies in the process of identifying relevant features for prediction. Exact inference is intractable in this model. However, expectation propagation offers an approximate alternative. Because the process of estimating feature selection dependencies may suffer from over-fitting in the model proposed, additional data from a multi-task learning scenario are considered for induction. The same model can be used in this setting with few modifications. Furthermore, the assumptions made are less restrictive than in other multi-task methods: The different tasks must share feature selection dependencies, but can have different relevant features and model coefficients. Experiments with real and synthetic data show that this model performs better than other multi-task alternatives from the literature. The experiments also show that the model is able to induce suitable feature selection dependencies for the problems considered, only from the training data.

Paper Summary: The paper handles a problem
      that has garnered growing research interest: Learning sparse solutions for
      problems with relatively small amount of samples and many features (i.e.;
      mandate feature selection), where we try to leverage data from several
      learning tasks over the same set of features. The assumption is that while
      the learning tasks are different they may share some characteristics that
      would enable us to generalize better. Specifically, the authors
      concentrate on the case where the learning tasks may share a dependency
      structure between the variables, though whether they are actually "active"
      and the magnitude of their coefficients (w) in each learning task may
      differ. The coefficient for each feature in each learning task is
      controlled via a horseshoe prior to insure sparseness. However, instead of
      assuming these are independent priors for each feature, the authors
      introduce a correlation matrix (C) over the hyper parameters (u,v) that
      control the features coefficients. They assume that matrix (C) is shared
      between the learning tasks. In this learning setting, we are generally
      interested in two things: (1) prediction for new data (2) detecting what
      are the "active" features for each learning task, given by z = (w,u,v) for
      each task. Since the integrals do not have a closed form solution they
      resort to approximate inference using EP. For testing their algorithm the
      authors use two datasets: A synthetic dataset which has a clear regularity
      in the selected active features in each task (active iff mod(i/16) = j
      where j in 1...16 and i in 1....128) and a 10x10 digitized handwritten
      digits. In both cases they show improved results compared to a host of
      other algorithms that do not utilize mulit-task learning or use multi-task
      learning but assume the active features are shared or at least have no
      dependency structure. Pros: 1. The paper includes comparison
      to several other methods, and demonstrate improved performance for those
      tasks. 2. The method developed is new making good use of a whole host
      of ML related methods. These methods have been gaining interest: Sparse
      Bayesian priors such as the horseshoe with alternative representations of
      it, EP with various tricks to make it more efficient. Thus, the paper has
      the side benefit of educating the readers while the authors claim
      significant speedup for EP. 3. The intro to the horseshoe prior usage
      is clearly written. Cons: 1. While the method is novel and
      non-trivial the overall relevance of it for real life applications is not
      clear, and the authors make no attempt at clarifying this. Several things
      stand out: First, the authors make a general comment that their approach
      is useful since the different learning tasks "share only the dependency
      structure" (p.2, top). True, sharing exactly the same active features and
      coefficients is a stronger assumptions, but other methods relax these
      assumptions. Sharing (exactly) the same dependency structure actually
      seems quite a strong assumption, and not necessarily a realistic one if
      you think of applications such as networks in the bio-medical domain. The
      authors claim this to be "a very flexible assumption" (p. 8 conclusions)
      but do not support this in any way. Specifically, they only evaluate their
      method on two datasets with very clear and unique regularities in their
      feature dependency structure. Comparison on cases where some features are
      shared but with no underlying dependency structure exists is warranted.
      Finally, the authors mention only theoretical computational costs. The
      experiments involve only small toy data. It is not clear how such a method
      would perform compared to alternatives when number of samples (n), number
      of tasks (k) and number of features (f) grows. The latter is especially
      problematic as the method aimes to reconstruct the correlation matrix
      between all features. 2. The writeup is missing crucial details
      for demonstrating mathematical soundness required for publication. The
      authors constantly use arguments such as "The gradient of .... can be used
      for this task" (p. 4), "the gradient of.... with respect to .... can be
      easily computed in terms of the ...." p. 5 Then sate "The derivations are
      long and tedious and hence omitted here" (p. 6). Similarly, all the actual
      implementation of the EP in Sec. 3 is only reviewed. The details should be
      given in a matching supplementary for completeness. 3. The write
      up is not well balanced/structured. While the main novel element is the
      estimation of the joint dependency structure the authors spend very little
      on it. Instead they give a clear but maybe too long intro to the horseshoe
      prior. Specifically, Figure 1 is taken straight out of Caravlho et al 2009
      (Fig 1,2) and the authors should at least point that out clearly and/or
      retain only say Fig1c. Sec 3 should be extended and and requires supp (see
      above). In the experiments the author spend almost a full page (p. 7) on a
      synthetic data which can only be described as a sanity check. The 3 panels
      in Fig 3 are too much. The set up of the digits experiments is not clear
      and should be revised (e.g. where does n = 75 comes from? "100" appears as
      the value of too many different params not defined, which is confusing).
      Other: 1. On p. 6 the authors state they define P (which
      is at the heart of the correlation matrix C) to have m x d params, and set
      m = n where n is #samples. But in the multi class learning framework
      they claim each learning task can have a different number of samples n_k,
      though P is presumably the same. Can you explain? 2. Fig 2 the
      annotation "f" for the top factor is missing


Lasso Screening Rules via Dual Polytope Projection
Lasso is a widely used regression technique to find sparse representations. When the dimension of the feature space and the number of samples are extremely large, solving the Lasso problem remains challenging. To improve the efficiency of solving large-scale Lasso problems, El Ghaoui and his colleagues have proposed the SAFE rules which are able to quickly identify the inactive predictors, i.e., predictors that have 0 components in the solution vector. Then, the inactive predictors or features can be removed from the optimization problem to reduce its scale. By transforming the standard Lasso to its dual form, it can be shown that the inactive predictors include the set of inactive constraints on the optimal dual solution. In this paper, we propose an efficient and effective screening rule via Dual Polytope Projections (DPP), which is mainly based on the uniqueness and nonexpansiveness of the optimal dual solution due to the fact that the feasible set in the dual space is a convex and closed polytope. Moreover, we show that our screening rule can be extended to identify inactive groups in group Lasso. To the best of our knowledge, there is currently no "exact" screening rule for group Lasso. We have evaluated our screening rule using many real data sets. Results show that our rule is more effective in identifying inactive predictors than existing state-of-the-art screening rules for Lasso.

The authors develop exact screening rules for the
      Lasso and Group Lasso problems (with quadratic loss). These rules are
      based on the observation that the dual of these problems is simply the
      projection of a vector onto a convex polytope. They can be applied in a
      sequential fashion along a grid of tuning parameter values.
      Quality: This is an excellent paper. The idea is elegant
      and explained well, and the resulting method appears to be highly
      effective. Clarity: Very good. Originality:
      While the main idea is not greatly different from the SAFE rules,
      the authors have simplified the idea and made it work better.
      Significance: This seems like work that could be easily
      built on. While the "exact" aspect of DPP is nice, the strong rules paper
      makes the point that their rule often just works (even though it requires
      checking the KKT conditions). It would be helpful to see the strong rule
      compared in the empirical studies. Of course for the strong rules you'd
      need to report whether it threw out any variables it should not have (and
      if so, perhaps show how many false rejections there were). Regardless of
      the results, DPP is still significant for being the best Comments:
      It is easy to prove that the DPP rule (at lambda_max) is strictly
      better than the SAFE rule, though I don't think you come out and say this.
      To compare the two, notice that DPP can be written as |x_i^T y |
      < lambda_max - ||x_i|| ||y|| (lamda_max - lambda) / lambda
      Comparing the RHS of DPP and SAFE are identical except for two
      differences. Both of these differences make RHS's DPP larger than SAFE and
      therefore it can screen strictly more features than SAFE. Minor
      comments: - Equation 8: Fix RHS. - Equations 21,22: Fix LHS
      (should be L2 norms) - Line 346: "projection ratio" should be
      "rejection ratio" ? - Line 376: "verfying" typo - Lines 406-7:
      highly correlated with the response, you mean?


Adaptive Anonymity via b-Matching
The adaptive anonymity problem is formalized where each individual shares their data along with an integer value to indicate their personal level of desired privacy. This problem leads to a generalization of k-anonymity to the b-matching setting. Novel algorithms and theory are provided to implement this type of anonymity. The relaxation achieves better utility, admits theoretical privacy guarantees that are as strong, and, most importantly, accommodates a variable level of anonymity for each individual. Empirical results confirm improved utility on benchmark and social data-sets.

'Adaptive Anonymity via b-Matching' addresses the
      problem of releasing a private dataset such that by censoring measurements
      individuals in the dataset become indistinguishable. The authors explain
      the concepts of k-anonymity and the b-matching problem as forming a
      bipartite graph that describes the compatibility of the original dataset
      with the released dataset. They provide two algorithms, for the asymmetric
      and symmetric cases, which are justified as variational approximations of
      the star (suppression) minimization problem. There is an approximation
      result for the algorithms. A variety of privacy guarantees are provided
      for single attacks,and sustained attacks on the k-clique construction,
      symmetric and asymmetric b-matchings. A brief experimental section is
      provided with some convincing results. This paper is of high
      quality, as it comprehensively addresses the b-matching problem and
      provides strong computationally feasible algorithms that are well
      justified. It is very well written with a very understandable introduction
      and exposition. The algorithmic results are strong, as are the privacy
      guarantees. It is likely to be a significant work, as b-matching is a
      significant extension of the k-anonymity, while being a reasonable
      alternative to other notions of privacy. Furthermore, this work is very
      general, allowing for user specified levels of privacy and addressing both
      the symmetric and asymmetric cases. The only criticism is that theorem 1
      is quite weak as an approximation factor of the degree makes it nearly
      useless. I think that an explanation (at least in the appendix) of how the
      algorithms are a variational upper bound of the ideal program would be
      more convincing. Minor comments: 191 - 'a more general' ->
      'a general' 263 - use `` not '' def 4.1 - this is confusing, how
      does the matching come in to the symmetry notion
      here


Approximate Gaussian process inference for the drift of stochastic differential equations
We introduce a nonparametric approach for estimating drift functions in systems of stochastic differential equations from sparse observations of the state vector. Using a Gaussian process prior over the drift as a function of the state vector, we develop an approximate EM algorithm to deal with the unobserved, latent dynamics between observations. The posterior over states is approximated by a piecewise linearized process of the Ornstein-Uhlenbeck type and the MAP estimation of the drift is facilitated by a sparse Gaussian process regression.

This is a solid paper on a topic of potential
      significant interest. Diffusion processes are widely used in the sciences
      and engineering, and a data-driven strategy to parametrize the drift
      function could be potentially useful. Furthermore, the paper also has
      considerable interest methodologically: the proposal to approximate the
      intractable problem with a piecewise OU process is interesting, and the
      approximate quantification of the error is (at least in theory) a nice
      result. I would say the paper scores highly on quality and originality. It
      is generally clearly written although some bits feel a bit rushed (see
      detailed comments below). Re significance, I would say the methodology
      could be of considerable interest, while the experiments are for the time
      being a bit less well developed. Detailed comments: - I would
      have appreciated a bit more motivation for the methodology: SDEs in the
      sciences are often used as a modelling tool so that the specific form of
      the drift has a physically interpretable meaning. This is somewhat lost
      with the GP approximation; - I wonder whether there are some
      asymptotic results which could be given, at every step you effectively get
      a measurement of the linearised drift (at a different input value), hence
      of its value and its derivative. When you increase the number of
      observations, this should converge to the true posterior (which of course
      it does as the approach becomes exact for observed paths), but perhaps you
      can work out the rate at which your approach converges to the truth? -
      The experimental part is a bit disappointing, there are essentially no
      comments on the performance of the model on simulated data, and very
      little context on the real data. I would remove the experiment with 5th
      degree drift and leave more space to comment on the experiments. Also some
      runtimes would be useful. Minor comments: - eq(10) worth perhaps
      saying that the last step uses explicitly the time invariance of the drift
      function (homogeneous GP); - after (13), it's the two terms on the
      r.h.s. that obey Ch-K equations (the way it's written is ambiguous); -
      some confusion with figures, there is a reference to Figure ?? (1a
      presumably) on page 3; the caption to Figure 2 alludes to a missing third
      panel; - models that use OU processes with piecewise constant drifts
      have been considered and shown to be very computationally efficient
      (Stimberg et al AISTATS11, Opper et al NIPS10 which reported very fast
      performance on the double well problem). These are different as the
      changepoints are stochastic processes while here the changes are an
      approximation strategy, but they seem somewhat related and might be worth
      referring to.


Transportability from Multiple Environments with Limited Experiments
This paper considers the problem of transferring experimental findings learned from multiple heterogeneous domains to a target domain, in which only limited experiments can be performed. We reduce questions of transportability from multiple domains and with limited scope to symbolic derivations in the causal calculus, thus extending the original setting of transportability introduced in [1], which assumes only one domain with full experimental information available. We further provide different graphical and algorithmic conditions for computing the transport formula in this setting, that is, a way of fusing the observational and experimental information scattered throughout different domains to synthesize a consistent estimate of the desired effects in the target domain. We also consider the issue of minimizing the variance of the produced estimand in order to increase power.

This paper showed the completeness of do-calculus for
      nonparametric estimation of a causal effect in a semi-Markovian graph
      structured causal diagram to which an arbitrary set of interventional and
      observation distributions conforms. It introduced a problem to infer a
      causal effect of treatment variables on observables in a target domain by
      combining data from experiments on simultaneously controllable subsets of
      variables from multiple domains and formulated it as mz-transportability.
      It further presented a complete algorithm for determining causal effects.
      Quality of the paper This paper showed the completeness of
      do-calculus in a semi-Markovian causal graph in a very generic setting.
      This consequence clarifies the conditions for the valid causal inference
      and ensures the validity of the causal inference under the condition. It
      further formulate a measure named mz-transportability and an algorithm to
      provide a transport formula for estimating the causal effect. These
      consequences provides a firm basis of the causal inference. In this
      regard, the work presented in this paper is highly meaningful.
      Mathematical formulations and their associated proofs seem to be
      valid, and thus their technical quality is considered to be high.
      However, the contents of the paper seem to be too much to fill in the
      limited length. We could not follow the explanations and the mathematical
      derivations perfectly since there are some significant explanation gaps
      because of the limited space for the explanation. For example, at the
      beginning of the subsection 4.1, the authors claim that "We use the
      notational conventions and definitions for causal models introduced in [2,
      15, 12]." In this sense, this paper is not self-contained. In addition, in
      the latter half of the subsection 4.2, the authors missed to explain the
      definition of "c-componets". Moreover, the entire description of the paper
      lacks the explanation through some comprehensible examples. The use of the
      examples eases a lot the comprehension of the readers particularly in this
      type of theoretical papers. Though these are issues of the paper
      presentation, this affect the value of the paper. Clarity The
      explanation and the proofs provided in the paper seems mathematically
      rigorous. However, as pointed out above, the contents of the paper is too
      much for the NIPS page limitation. The characterization of the
      mz-transportation can be far more reduced to provide some basis only for
      the algorithm in the subsection 4.3. The characterization can be presented
      in its journal paper version. By presenting the essential skeleton of the
      contents, the authors can show sufficient value of this study.
      Originality The mathematical and technical quality of this
      paper seems high. However, the scope of the paper is rather incremental,
      and it does not provide entirely novel principles in this research field.
      Significance The consequence of this paper widely enables
      causal analysis over multiple domains while ensuring the identifiability
      of the causal structures. The paper is significant in this regard. But,
      the paper does not include any experimental validation of the proposed
      consequences. If some numerical experiments applying the proposed
      algorithms were demonstrated, the significance of the result could be more
      clearly indicated.


Bayesian optimization explains human active search
Many real-world problems have complicated objective functions. To optimize such functions, humans utilize sophisticated sequential decision-making strategies. Many optimization algorithms have also been developed for this same purpose, but how do they compare to humans in terms of both performance and behavior? We try to unravel the general underlying algorithm people may be using while searching for the maximum of an invisible 1D function. Subjects click on a blank screen and are shown the ordinate of the function at each clicked abscissa location. Their task is to find the function's maximum in as few clicks as possible. Subjects win if they get close enough to the maximum location. Analysis over 23 non-maths undergraduates, optimizing 25 functions from different families, shows that humans outperform 24 well-known optimization algorithms. Bayesian Optimization based on Gaussian Processes, which exploits all the x values tried and all the f (x) values obtained so far to pick the next x, predicts human performance and searched locations better. In 6 follow-up controlled experiments over 76 subjects, covering interpolation, extrapolation, and optimization tasks, we further confirm that Gaussian Processes provide a general and unified theoretical account to explain passive and active function learning and search in humans.

Paper 68 – Bayesian optimization explains human active
      search The authors explore different optimization strategies for
      1-D continuous functions and their relationship to how people optimize the
      functions. They used a wide variety of continuous functions (with one
      exception): polynomial, exponential, trigonometric, and the Dirac
      function. They also explore how people interpolate and extrapolate noisy
      samples from a latent function (which has a long tradition in psychology
      under the name of function learning) and how people select an additional
      sample to observe under the task of interpolating or extrapolating. Over
      all, they found that Gaussian processes do a better job at describing
      human performance than any of the approx. 20 other tested optimization
      methods. Broadly, I really enjoyed this paper and I believe that
      it is a strong NIPS submission. They tackled interesting problems for
      machine learning-oriented cognitive scientists (although the stimuli may
      be a bit too abstract to be of interest for many cognitive psychologists):
      optimizing a function, and active interpolation and extrapolation. It is
      well written and besides for some minor quibbles, the presentation is very
      good. Besides for some minor points that I enumerate below, I have
      two main criticisms of the paper. 1. In contrast to most function
      learning experiments, the experimental stimuli and procedure were
      maximally abstract (dots from a function and “find the maximum value of a
      function”). This is not ecologically valid and thus, it is difficult to
      interpret what the mean for actual human function learning. The functions
      used in the experiments are probably not those functions that people
      expect in their everyday lives. Additionally, there typically is a strong
      effect of context on this sort of human learning and it is not clear how
      their results can be extended to account for context (which in my opinion
      would be a very interesting direction for them to go into for future
      research.). It is not enough to merely state “note that many real-world
      problems can be translated into our synthetic tasks here.” There needs to
      be examples and a justification for why the synthetic tasks can be
      generalized to real-world examples. However, they got compelling results,
      where participants seem to be doing the task. 2. I would have
      liked to see the authors integrate their experiments and results better
      with previous human function learning work. For example, I do not see why
      they did not choose to do their experiment using the standard cover story
      and stimuli of a function learning experiment. I understand why the
      authors did not just use a positive or negative linear function (although
      given f20 of experiment 1 is unbounded, it seems reasonable to include
      these too), but it seems including a piecewise linear function would have
      been a smart choice. As f16 is nearly a negative linear function and has
      been previously shown to be easy to learn, it is sensible that it would be
      one of the easiest function to maximize (and this is the sort of
      integration with previous work that I would have liked to have seen).
      Integrating with previous work would have made the paper stronger and the
      results more interpretable. More minor concerns:
      Personally, I would have preferred to see the previous
      psychological work to be earlier in the paper (e.g., in the introduction)
      rather than thrown in at the very end. Figure 1 is unreadable when
      printed out in B&W. The stimuli generation in Experiment 1
      seemed odd. Why not directly generate the functions via generating random
      coefficients rather than sample random points and fit a polynomial? If
      there was a good reason for this, explaining it in the paper would be
      helpful. “Model-free results” -> “Human results” At
      times, the framing gets into hairy theoretical territory as to the level
      of analysis of the paper, but at other times, it is fine. Examples of
      troubling language include “humans might be following a GP,” and “humans
      may use GP.” This ventures dangerously into the process level. The authors
      have not shown any evidence that the cognitive mechanism is “GP-like”
      whatever that might mean. An example of language at the appropriate level
      of analysis is “Results of optimization tasks suggest that human clicks
      during search for a maximum of a 1D function can be predicted by a
      Gaussian processes [sic] model.” (Perhaps “a Gaussian processes model”
      should be “a Gaussian process model” instead?). I think the
      authors should report the rank-order correlation of the DVs for people and
      each optimization method (where the correlation is over the different
      function types). I don’t understand why different tolerance levels
      were used for different optimization algorithms, although I do not see how
      it would prejudice their results in favor of their conclusion.
      Figure 3 is extremely difficult to read when printed out in
      B&W. Text is too small on Figure 4. Results of
      Experiment 2 & 3: “Interestingly, on Deg5 functions, GP is closer to
      human clicks than the actual function (signed-rank test, p = 0.053)
      implying that GP captures clicks well in this case.” Would this hold up
      after correcting for multiple comparisons? (you tested Deg2 and Deg3 as
      well). Experiments 2 & 4 are essentially function learning
      experiments. Figure 5 is pretty difficult to read when printed out
      in B&W. Experiment 5: Why were subjects always asked to guess
      the value of f(200) rather than a few different values? Also for the
      active extrapolation in this case, why not ask to find the value of the
      point right next to f(200) and guess basically that value? Another
      control method for the interpolation and extrapolation methods that would
      have been nice to see how people and GPs relate to is estimating the
      parameters of a polynomial of degree up to m (and vary m). Some
      recent work on optimal foraging presented at last year’s NIPS was Abbott,
      Austerweil, & Griffiths (2011) and a GP account might relate to the
      model used by Hills et al. (2011).


Non-Uniform Camera Shake Removal Using a Spatially-Adaptive Sparse Penalty
Typical blur from camera shake often deviates from the standard uniform convolutional assumption, in part because of problematic rotations which create greater blurring away from some unknown center point. Consequently, successful blind deconvolution for removing shake artifacts requires the estimation of a spatiallyvarying or non-uniform blur operator. Using ideas from Bayesian inference and convex analysis, this paper derives a simple non-uniform blind deblurring algorithm with a spatially-adaptive image penalty. Through an implicit normalization process, this penalty automatically adjust its shape based on the estimated degree of local blur and image structure such that regions with large blur or few prominent edges are discounted. Remaining regions with modest blur and revealing edges therefore dominate on average without explicitly incorporating structureselection heuristics. The algorithm can be implemented using an optimization strategy that is virtually tuning-parameter free and simpler than existing methods, and likely can be applied in other settings such as dictionary learning. Detailed theoretical analysis and empirical comparisons on real images serve as validation.

The paper addresses the problem of single image blind
      deconvolution with non-uniform blur caused by camera shake. The authors
      propose a two step procedure: first to estimate the motion blur and second
      the recovery of the sought-after latent image through non-blind
      deconvolution. This procedure is common practice in blind deconvolution.
      The main contribution made here is the derivation of a Bayesian inference
      strategy for motion blur estimation. In a number of experiments the
      authors demonstrate the validity of their approach and compare its
      performance against other state-of-the-art methods. The mathematical
      derivation appears sound and the proposed scheme is claimed to exhibit a
      number of favourable properties. Although these details are discussed in
      some detail in the second half of Section 4, no empirical evidence is
      given, which would strengthen the argumentation and verify the claims
      made. The authors also miss to make connections to other recent approaches
      that advocate a similar line of reasoning, in particular [15] and the
      missing reference: S.D.Babacan, et al., "Bayesian Blind
      Deconvolution with General Sparse Image Priors", ECCV 2012 In
      addition, the authors miss another relevant reference, namely
      R.Kohler, et al., "Recording and playback of camera shake:
      Benchmarking blind deconvolution with a real-world database." ECCV 2012.
      that would allow them to benchmark their approach against other
      state-of-the-art single image blind deblurring methods for both uniform
      and non-uniform blur. Besides, a number of questions remain unanswered
      (see below) in the presented exposition, that lower the overall quality of
      the paper. The paper is clearly structured and well written.
      Section 4 would benefit from a division in smaller subsections as it feels
      a bit lengthy for my tasting. Unfortunately, some important details are
      missing (e.g. which algorithm is used for the final non-blind
      deconvolution, see also comments below), which impair the overall clarity
      of the paper. The theoretical foundation of the proposed approach
      build on previous work [19,26], however its application to the problem of
      single image blind deconvolution seems novel. Although the
      presented algorithm seems to deliver comparable results to
      state-of-the-art algorithms, the provided insights are very limited. It
      remains unclear in which situations/cases the proposed method works better
      and why. It also fails to make connections to other recent relevant work
      (see above) and place its contribution into context, which lowers its
      significance considerably. Further comments: * What method is
      employed for the final non-blind deconvolution? * It is stated, that
      the approach is "parameter-free". Is this also true for the kernel size?
      * It is mentioned in Section 4, that initialization with a large
      $\lambda$ and its subsequent evolution render the deconvolution process
      more stable. It would be interesting to see a plot that shows the
      evolution of $\lambda$ in the course of the motion blur estimation
      process. * What basis (i.e. $B$ in Eq.(8)) is chosen for the
      experiments with non-uniform blur? Is $w^\bar_i$ evaluated for every
      pixel? If not, how are the evaluation locations chosen? * From the
      main paper it is not clear that a coarse-to-fine scheme is employed. Great
      if this could be made clearer. * What are typical run-times for the
      proposed method? * Large blur is challenging even for state-of-the-art
      methods. How does performance scale with kernels size? * The ringing
      artifacts in Whyte's result in Fig.4 are stemming primarily from the final
      non-blind deconvolution (see his talk slides). It would be more fair to
      use the one published on the accompanying project webpage
      (http://www.di.ens.fr/willow/research/deblurring/). * The correct
      reference for the efficient filter flow framework mentioned in Section 2
      is M.Hirsch, et al., "Efficient filter flow for space-variant
      multiframe blind deconvolution." CVPR 2010. * What are the limitations
      of the proposed approach?


Reflection methods for user-friendly submodular optimization
Recently, it has become evident that submodularity naturally captures widely occurring concepts in machine learning, signal processing and computer vision. Consequently, there is need for efficient optimization procedures for submodular functions, especially for minimization problems. While general submodular minimization is challenging, we propose a new method that exploits existing decomposability of submodular functions. In contrast to previous approaches, our method is neither approximate, nor impractical, nor does it need any cumbersome parameter tuning. Moreover, it is easy to implement and parallelize. A key component of our method is a formulation of the discrete submodular minimization problem as a continuous best approximation problem that is solved through a sequence of reflections, and its solution can be easily thresholded to obtain an optimal discrete solution. This method solves both the continuous and discrete formulations of the problem, and therefore has applications in learning, inference, and reconstruction. In our experiments, we illustrate the benefits of our method on two image segmentation tasks.

Summary of ideas: There is a previously known
      proximal problem, whose minimization gives at least as much information
      than minimizing the Lovasz extension. When a submodular function is
      written as a sum of submodular functions, each relatively easy to solve,
      solving the dual for the proximal problem can be done in terms of
      projections, rather than posing non-smooth functions as previous methods
      do. Evaluation: The paper presents a refined way of
      dealing with submodular functions that are additively decomposable into
      simpler functions, by novel dual decompositions of a less well known use
      of the Lovasz extension. These allow the decomposition to proceed
      without some of the complications arising due to non-smoothness in
      previous approaches. The empirical evaluation using examples from image
      processing shows promising results. The paper is very clear in
      describing its predecessors; unfortunately, this seems to leave
      insufficient space to be more than sketchy about the novel techniques.
      For example, one of the important technical contributions is to show
      how the DR variant of reference [5] is applicable to dual (9), which is
      specialized for a sum of 2 functions; but [5] focuses on two functions as
      well, and how exactly to apply these algorithms for r > 2 is not clear
      to me, despite such results being reported as useful in the experimental
      evaluation section under the name DR-para. We certainly do not know
      iteration complexity depends of r. In another example, any detailed
      description of the third main contribution is essentially deferred to the
      supplementary material (which formally states neither an algorithm nor a
      theorem). Pros: - Decomposable submodular functions arise in
      applications, and more appropriate ways to minimize them are useful. -
      The novel duals might spur further work on algorithms for their solution,
      beyond the algorithms proposed here. Cons: - While iteration and
      computational complexity bounds are treated as limitations of previous
      results, the novel optimization problems and algorithms are not
      accompanied by corresponding iteration complexity results. While a full
      analysis can be deferred to future work, some understanding of how r
      affects complexity under the different algorithms is missing. - The
      presentation of the algorithms (as opposed to duals) is not clear enough.
      Detailed comments: - 072 "the level set {k, x_k^* \ge 0} of
      ... x^*" is not very clear, is {k | x_k^* \ge 0} meant? Ref [2] refers to
      many different problems, a more specific reference for the fact being
      mentioned would be useful. - 088 Should reference section 3 where you
      give some examples in which this decomposition holds, and an idea of what
      types of "simplicity" we might encounter for the summand functions would
      be nice. - 116 The notation a(A) where a\in R^n and A\subseteq V does
      not seem to be introduced anywhere. - 117 This is misleading: Section
      2 punts to the Supplementary material, which also skimps on explicit
      detail. - 138 computed the -> computed by the - 156 The term
      "discrete gaps" are used here and elsewhere but not clearly defined. The
      duality gaps of F(A) and a particular dual problem? Similarly for smooth
      gaps. - 170 "In the appendix.." An explicit algorithm should be
      accompanied by a lemma and proof giving its complexity. Neither is
      present, in the paper nor in the supplementary material. - "problems
      have size 640x427. Hence ... have r = 640 + 247" either there is a typo
      (427 to 247), or the relationship is not clear enough. Also, it is worth
      explaining how and why you "solve as r=2/3". - Figure 2: -
      Inconsistencies of vertical and horizontal axes makes comparisons
      difficult. - "From top to bottom: four different images" presumably 2
      different images. A wider benchmark - Figure 3: - Does "non-smooth
      problems" not correspond to the previously existing duals? the legends
      suggest that the new duals are being used. - Empirical evaluation in
      general: comparisons of convergence in terms of iteration counts are
      informative only when iteration costs are roughly comparable. Are they?


On the Linear Convergence of the Proximal Gradient Method for Trace Norm Regularization
Motivated by various applications in machine learning, the problem of minimizing a convex smooth loss function with trace norm regularization has received much attention lately. Currently, a popular method for solving such problem is the proximal gradient method (PGM), which is known to have a sublinear rate of convergence. In this paper, we show that for a large class of loss functions, the convergence rate of the PGM is in fact linear. Our result is established without any strong convexity assumption on the loss function. A key ingredient in our proof is a new Lipschitzian error bound for the aforementioned trace norm-regularized problem, which may be of independent interest.

--------added after authors' rebuttal-------- The
      proof for |R_k| <= kappa |X_k - X_{k+1}| has been explicitly
      established in "Scalable nonconvex inexact proximal splitting", Suvrit
      Sra, NIPS'12. (Lemma 4) --------end-------- For trace norm
      regularized risk minimization, a popular method is the proximal gradient
      algorithm. The sublinear global convergence rate of this algorithm is
      well-understood while it is also known that the rate is linear if the loss
      is additionally strongly convex. This paper relaxes the strong convexity
      assumption by following the framework established by Luo and Tseng. In
      particular, the authors proved a local error bound that allows them to
      show that the proximal gradient converges asymptotically at a linear rate
      for trace norm regularization. The paper more or less follows the usual
      pattern to prove the local error bound, with a new observation that allows
      simultaneous diagonalization of the matrices. Quality: This
      paper appears to be technically sound. The only gap I had in mind is in
      the appendix, line 085. Please explain why such a choice of kappa_3 (say
      alpha < 1) leads to the claimed inequality R(X^k) <= kappa_3 |X^k -
      X^{k+1}|_F ? Clarity: I could follow the paper without much
      difficulty. Although due to the technical nature, it might be a good idea
      to lay out the main claims first and then go to details one by one.
      Particularly, after stating the local error bound, the authors might want
      to postpone the very technical proof of the local error bound but turn
      immediately to prove the linear rate (which by the way, has hardly
      anything new in it). Originality: The originality of this
      paper is moderate. The observation that allows the authors to
      simultaneously diagonalize matrices seems to be new and interesting; other
      than that, the authors more or less follow the existing approach to
      establish the local error bound. The rest steps are not new.
      Significance: The significance of this paper is mainly in
      theory, with little impact on algorithm design though. While it is
      certainly great to extend the linear convergence rate of the proximal
      gradient algorithm to more practical scenarios (i.e., relaxed form of
      strong convexity), the significance is nevertheless lowered by the
      asymptotic nature of the proof (and claim). The way the authors handle the
      diagonalization of matrices might be of some independent
      interest.


Bayesian Inference and Online Experimental Design for Mapping Neural Microcircuits
With the advent of modern stimulation techniques in neuroscience, the opportunity arises to map neuron to neuron connectivity. In this work, we develop a method for efficiently inferring posterior distributions over synaptic strengths in neural microcircuits. The input to our algorithm is data from experiments in which action potentials from putative presynaptic neurons can be evoked while a subthreshold recording is made from a single postsynaptic neuron. We present a realistic statistical model which accounts for the main sources of variability in this experiment and allows for significant prior information about the connectivity and neuronal cell types to be incorporated if available. Due to the technical challenges and sparsity of these systems, it is important to focus experimental time stimulating the neurons whose synaptic strength is most ambiguous, therefore we also develop an online optimal design algorithm for choosing which neurons to stimulate at each trial.

Targeted stimulation of multiple pre-synaptic cells
      combined with intra-cellular recording of a single post-synaptic cell is a
      popular and important method for estimating synaptic connectivity between
      these cells. The main drawback of this approach is that it is technically
      difficult, so that only small number of connections can be estimated in a
      given experiment. The authors aim to provide statistical methods for
      speeding up these experiments, by firstly providing a full statistical
      model which takes the sparsity of neural coupling into account, and
      secondly by providing computationally efficient methods for choosing
      target-locations for stimulation. The paper is sound and clearly
      explained, and the methods are evaluated using synthetic data. The
      drawbacks of this submission are that a) no results on real data are
      presented b) the neuron/noise model used is a bit simplistic (see details
      below) and that it is therefore not quite clear how well this method will
      work in a real experimental context. Overall, my verdict is that
      this is a neat (but not outstanding) submission which could be a nice
      addition to NIPS. Quality: The statistical model is reasonable and
      the evaluation of the model on synthetic data is sound. However, there are
      a number of properties of the model which might provide problematic on
      real data, and therefore, the important test of the relevance of this
      study will be how well it works in such a setting. In the absence of real
      data, it would have been important to include simulation results in a more
      realistic setting, i.e. one in which the model does not perfectly describe
      the data (e.g. using a different neuron model, correlated noise, possibly
      input from other neurons, feedback couplings, etc..)
      Hyper-parameters: The statistical model includes some
      hyper-parameters which probably strongly affect model performance, but
      which are not set by the method. While it is true that some of these
      parameters can be set by using known connection-probabilities, it is
      unclear how appropriate this will be for a given experimental application.
      How sensitive is the method to "wrong" hyperparameters e.g. on the
      spike-and-slab prior?" Uncorrelated noise: The method assumes
      uncorrelated noise on the membrane voltages, and says that it is
      "straightforward" to extend this to correlated noise-- while I have no
      doubts that it is straightforward mathematically, it would have been
      important to see how strongly this affects performance e.g. in the
      simulations. I could imagine that the inference task is substantially
      harder if one also has to deal with correlated noise. Simulations:
      In the absence of real data, it would have been important to at least
      include simulation results in a more realistic setting, i.e. one in which
      the model does not perfectly describe the data (e.g. using a different
      neuron model, correlated noise, possibly input from other neurons,
      feedback couplings, etc..). One of the results of the simulation was that
      the experimental design strategy does not show a great performance benefit
      in the high-noise regime with multiple stimulated entries-- I would expect
      that this is actually the most important regime? Clarity: The
      paper is mostly clearly written, but takes a long "run up" which sets up a
      general statistical model which is then subsequently broken down to the
      model which is actually used by the authors. (E.g. they write that
      spike-and-slab priors with any distribution can be used, and later say
      that they consider only the Gaussian case). I would suggest just
      describing what was actually done to streamline the exposition."
      Originality: As far as I know, this study is original. While
      there have been previous studies that address estimation of synaptic
      connectivity (from presynaptic stimulation/postsynaptic membrane
      mesuarements (e.g. Hu, Leonardo, Chklovskii) this is the first such study
      which aims to provide a "realistic" statistical model and, importantly,
      which addresses the question of experimental design for this task.
      Significance: The impact of this study will be be stronlgy
      dependent on how well this method works on real data.


Multilinear Dynamical Systems for Tensor Time Series
Data in the sciences frequently occur as sequences of multidimensional arrays called tensors. How can hidden, evolving trends in such data be extracted while preserving the tensor structure? The model that is traditionally used is the linear dynamical system (LDS) with Gaussian noise, which treats the latent state and observation at each time slice as a vector. We present the multilinear dynamical system (MLDS) for modeling tensor time series and an expectation-maximization (EM) algorithm to estimate the parameters. The MLDS models each tensor observation in the time series as the multilinear projection of the corresponding member of a sequence of latent tensors. The latent tensors are again evolving with respect to a multilinear projection. Compared to the LDS with an equal number of parameters, the MLDS achieves higher prediction accuracy and marginal likelihood for both artificial and real datasets.

This paper proposes a tensorial LDS model called MLDS.
      As such, the model is a restriction of the general Kalman filter model
      with state transition and observation operators having a certain Kronecker
      product factorization. The model can be understood intuitively as a
      sequence of core tensors that form a Markov Chain, and the observed tensor
      at each time slice as well as the core tensor at the next time slice are
      obtained by Tucker models with tensor factors C and A respectively. The
      term multilinear in the title is misleading in this respect as the model
      is linear. After setting up the notation and background, authors
      derive the posterior distribution of hidden tensors given the observed
      tensor in Sec. 4, Lemma 2. This is direct generalization of standard
      marginalization and conditioning results of multivariate Gaussians. What
      is not discussed is the form of the state covariance and whether it admits
      a useful factorization (compare with matrix Gaussians where the covariance
      is given as a Kronecker Product). I suspect that this model would not
      admit such a structured covariance but I would have welcomed at least a
      discussion of this issue because calculating the covariance would be the
      computational bottleneck with large Z, not much mentioned here and
      somewhat swept under the carpet. The ideas are quite analogous to
      theory of linear dynamical systems (LDS) but the tensor notation is
      unevitably quite heavy, yet the authors made a pretty good job in keeping
      the notation intuitive by drawing analogies with the vector case. Perhaps
      not surprisingly, the paper concludes that by preserving the linear
      structure, MLDS is more effective than a plain LDS where the tensors are
      simply vectorized. However, the paper does not much detail on inference
      (filtering/smoothing recursions) and instead focuses on parameter
      inference via EM. Another issue I would have seen discussed is
      identifiability and the justification of the Tucker transition model. The
      LDS states are identifiable upto similarity transform, what is the case
      for the case discussed here? For the latter point, whilst the observation
      model is clear, it would be good at least to justify the Tucker transition
      model; does this have a physical meaning? Otherwise I would suspect taking
      an unconstrained transition operator A or the other extreme of taking A as
      the identity (e.g., a drifting core tensor) are equally viable.
      Sec. 5.2 develops an EM algorithm to for parameter estimation.
      This section is rather short and many details are omitted. The classical
      EM algorithm for learning an LDS derives update rules for each parameter,
      including noise covariances. In the paper, although authors state that,
      ‘MLDS inherits the E-step of the LDS and the M-step of the LDS for all
      parameters except A and C’, it is not clear what they mean and it is not
      possible to see the update rules of noise covariances at one-step.
      Furthermore, authors do not explicitly state the update rules for A and C
      and merely derive the gradient. In the Eq. (11), authors use \phi and
      \varphi which are defined with expectations. In the LDS, these
      expectations require to find smoothed estimates (in addition to filtered
      estimates). The paper nowhere mentions the smoothing recursions explicitly
      and the reader has to simply redrive the results from scratch. From the
      description in the paper it is not clear that how these expectations are
      computed making the work hard to reproduce. Despite all the small
      glitches, I find the model interesting and well executed enough deserving
      publication. I am surprised that the literature does not contain
      special cases of this model such as 'Matrix variate Kalman Filter'.
      Short comments Abstract and introduction should be more
      clear and state the precise results about inference and learning
      algorithms that are obtained in this paper. Experimental section
      is good. If authors think to resubmit this paper, they should focus on the
      Section 4 and Section 5. The statement in the related work section
      page 3, line 122 is not entirely correct as GCTF is not limited to CP but
      can accomodate arbitrary factorization forms and fairly general noise
      models. The statement in the conclusion "permitting us to learn
      all but the transition and projection operators A and C of the MLDS
      via the Kalman filter" is vague (and actually incorrect as KF provides
      the filtered state estimates and covariances) The citation to [18]
      looks off-topic and can either be omitted or must be elaborated more to
      stress why the non existence may be an issue.


Predicting Parameters in Deep Learning
We demonstrate that there is significant redundancy in the parameterization of several deep learning models. Given only a few weight values for each feature it is possible to accurately predict the remaining values. Moreover, we show that not only can the parameter values be predicted, but many of them need not be learned at all. We train several different architectures by learning only a small number of weights and predicting the rest. In the best case we are able to predict more than 95% of the weights of a network without any drop in accuracy.

Motivated by recent attempts to learn very large
      networks this work proposes an approach for reducing the number of free
      parameters in neural-network type architectures. The method is based on
      the intuition that there is typically strong redundancy in the learned
      parameters (for instance, the first layer filters of of NNs applied to
      images are smooth): The authors suggest to learn only a subset of the
      parameter values and to then predicted the remaining ones through some
      form of interpolation. The proposed approach is evaluated for several
      architectures (MLP, convolutional NN, reconstruction-ICA) and different
      vision datasets (MNIST, CIFAR, STL-10). The results suggest that in
      general it is sufficient to learn fewer than 50% of the parameters without
      any loss in performance (significantly fewer parameters seem sufficient
      for MNIST). The paper is clear and easy to follow. The method is
      relatively simple: The authors assume a low-rank decomposition of the
      weight matrix and then further fix one of the two matrices using prior
      knowledge about the data (e.g., in the vision case, exploiting the fact
      that nearby pixels - and weights - tend to be correlated). This can be
      interpreted as predicting the "unobserved" parameters from the subset of
      learned filter weights via kernel ridge regression, where the kernel
      captures prior knowledge about the topology / "smoothness" of the weights.
      For the situation when such prior knowledge is not available the authors
      describe a way to learn a suitable kernel from data. The idea of
      reducing the number of parameters in NN-like architectures through
      connectivity constraints in itself is of course not novel, and the authors
      provide a pretty good discussion of related work in section 5. Their
      method is very closely related to the idea of factorizing weight matrices
      as is, for instance, commonly done for 3-way RBMs (e.g. ref [22] in the
      paper), but also occasionally for standard RBMs (e.g. [R1], missing in the
      paper). The present papers differs from these in that the authors propose
      to exploit prior knowledge to constrain one of the matrices. As also
      discussed by the authors, the approach can further be interpreted as a
      particular type of pooling -- a strategy commonly employed in
      convolutional neural networks. Another view of the proposed approach is
      that the filters are represented as a linear combination of basis
      functions (in the paper, the particular form of the basis functions is
      determined by the choice of kernel). Such representations have been
      explored in various forms and to various ends in the computer vision and
      signal processing literature (see e.g. [R2,R3,R4,R5]). [R4,R5], for
      instance, represent filters in terms of a linear combination of basis
      functions that reduce the computational complexity of the filtering
      process). In the experimental section the authors make an effort
      to demonstrate that the proposed method is practically useful and widely
      applicable, considering multiple datasets and several different
      architectures. I think, however, that this section could have been
      stronger in several ways: Firstly, it would have been nice if the
      paper had not focused exclusively on vision applications, and had put
      generally more emphasis on scenarios where there is a less obvious topolgy
      in the data that can be exploited when predicting weights (which will pose
      more of a challenge to the method). The data-dependent kernel is not very
      well evaluated. Secondly, especially for the vision case, I am
      wondering why the authors are limiting themselves to the particular set of
      basis functions derived from the kernel regression view. It would seem
      natural to explore other linear basis representations of the filters (a
      simple choice would be a PCA basis, for instance). These might be more
      efficient in terms of reducing the number of free parameters, and might
      have other desirable properties (e.g. [R4,R5]). Finally, I would
      have hoped for a truly compelling experimental use case demonstrating the
      impact of the approach in practice. Since the work is motivated as a way
      of reducing computational and memory complexity, I think it would have
      been useful to include a more detailed discussion and empirical
      demonstration how the reduction in number of learned parameters translates
      into such savings and consequently allows the training of larger networks
      that achieve better performance than otherwise possible. At the moment,
      the evaluation appears to be focused on moderately large networks, and the
      authors show that a moderate reduction in parameters can be achieved that
      way, without loosing performance (for MNIST the potential reduction in
      parameters seems to be very large, but for the more interesting CIFAR /
      STL-10 datasets it seems that at least 40% of the parameters are required
      to avoid a loss in accuracy). Why is computational complexity and speed of
      convergence not considered at all in the evaluation? I think that Fig.
      6-right (which plots performance against # of free parameters for a
      standard network and a reduced parameterization) goes in the right
      direction, but it seems that there is really only a clear advantage of the
      reduced parameterization for CIFAR (much less so for STL), and I'm
      wondering whether the performance difference on CIFAR would disappear even
      for currently realistic network sizes. All in all, I think the
      paper takes an interesting perspective although related approaches have
      appeared in the literature in various guises (see discussion above). I
      think that the ideas can have practical impact, and to my knowledge, they
      are currently not widely used in the NN/deep learning community. A
      stronger case could probably be made, however, by further exploring
      alternative implementations of the idea and by having a more compelling
      demonstration of the effectiveness and versatility of the approach.
      Further remarks: ** It would have been really helpful to
      include state of the art results from the literature for the datasets /
      and specific evaluation regimes considered. This would put the reported
      performance into perspective and make it easier to judge whether savings
      in learned parameters can be achieved for competitive architectures.
      ** I find the evaluation of the data-dependent kernel somewhat
      limited: it is only explored for a single dataset / architecture
      combination (the MNIST, MLP experiments in section 4.1). As mentioned
      above, it would have been nice to inlcude some other type of data that
      requires the use of the empirical kernel. For the experiments in
      section 4.1 it would have further been useful to also consider the case
      SE-rand. At the moment it's not clear that the learned kernel really
      outperforms the random approaches. The difference might simply be due to
      the fact that for the empirical kernel you're using the SE kernel in the
      first layer. Another interesting control for the effectiveness of the
      empirical kernel would be to have a setting where you're using the
      empirical kernel in both layers. ** Do you have any insights as to
      why the naive low-rank scheme, where both matrices are learned, is working
      so poorly? Have you made any attempt to remove the redundancy inherent in
      the parameterization? Would it be possible (and potentially advantageous)
      to start of with a fixed U (e.g. the empirical kernel), but allow U to
      change later at some point, possibly in an optimization scheme where U and
      V are updated in an alternating manner? ** In your conv-net
      experiments I suppose you are not using any pooling layers as described in
      [18]? This would also reduce the dimensionality of the representation in
      the higher layers. Furthermore, based on Fig. 5 I am wondering whether you
      can really argue that predicting 75% of the parameters has negligible
      impact on performance: when you're predicting 50% of the parameters the
      loss seems to be about 5-6 percentage points which does not seem so
      negligible to me (I guess it would help to have some errorbars here).
      ** How did you optimize the hyperparameters (learning rate etc.)
      for your comparisons? Shouldn't the kernel width vary with the size of the
      subset of learned parameters alpha? ** In section 4.1, for the MLP
      experiments, how do the columns differ? Do they use the same empirical
      kernel and differ only in the set of sampled pixels? Is the great
      advantage of using multiple columns here due to the fact that otherwise
      you don't sample the space well? ** Fig. 6 right: I find it
      curious that the reduced parameterization is advantageous for for CIFAR
      but there seems to be very little difference for STL - given the higher
      resolution of STL wouldn't one expect the opposite? ** For
      conv-nets where, in the higher-layers, there are many input channels
      relative to the spatial extent of each filter, would it be possible to use
      a data-dependent kernel to subsample across channels (if I understand
      correctly you are currently performing the subsampling only spatially,
      i.e. the elements of alphas are tied across channels)? ** Is there
      a way to extend your scheme such as to start off with a small fraction of
      learned parameters, but to then increase the fraction of learned
      parameters slowly until a desired performance is reached / no further
      improvement is observed? References: [R1] Dahl
      et al., 2012: Training Restricted Boltzmann Machines on Word Observations
      [R2] Roth & Black, IJCV, 2009: Fields of Experts (Section 5.3)
      [R3] Freeman & Adelson, 1991: The design and use of steerable
      filters [R4] Rubinstein et al., 2010: Double Sparsity: Learning
      Sparse Dictionaries for Sparse Signal Approximation [R5]
      Rigamonti et al., 2013: Learning Separable Filters


More data speeds up training time in learning halfspaces over sparse vectors
The increased availability of data in recent years has led several authors to ask whether it is possible to use data as a computational resource. That is, if more data is available, beyond the sample complexity limit, is it possible to use the extra examples to speed up the computation time required to perform the learning task? We give the first positive answer to this question for a natural supervised learning problem -we consider agnostic PAC learning of halfspaces over 3-sparse vectors in {−1, 1, 0}
n . This class is inefficiently learnable using O n/ 2 examples. Our main contribution is a novel, non-cryptographic, methodology for establishing computational-statistical gaps, which allows us to show that, under a widely believed assumption that refuting random 3CNF formulas is hard, it is impossible to efficiently learn this class using only O n/ 2 examples. We further show that under stronger hardness assumptions, even O n 1.499 / 2 examples do not suffice. On the other hand, we show a new algorithm that learns this class efficiently usingΩ n 2 / 2 examples. This formally establishes the tradeoff between sample and computational complexity for a natural supervised learning problem.

On page 2 -- you say that it is impossible to use
      standard reductions for proving NP-hardness. I'd say that this issue
      is a bit more subtle: It seems that some classes of reduction will not
      allow basing hardness of learning on RP \neq NP. (Cite the paper by
      Applebaum, Barak and Xiao). You make the point that you are not
      basing your results on cryptographic ssumptions. However, it is not clear
      (at least to me) that the complexity-theoretic assumption you are
      using is necessarily weaker than assuming the existence of one-way
      functions. Possibly, there is some evidence that the assumption that
      there are average-case hard functions in NP is "weaker" than one-way
      functions exist. There are some results by Noam Livne indicating such
      a possibility. You should discuss the relative hardness of assumptions you
      are using vs. those that others have used. For example, is it obvious
      that the result you have holds under a strong-enough cryptographic
      assumption such as factoring is hard? I suspect it should be possible.
      For the upper bounds: I think using your techniques, it should be
      possible to show that H_{n, k+1} can be learnt using
      \tilde{O}(n^k/\epsilon^2) samples? Is that right? Also, do you think
      it is possible to push the lower-bound (using a different assumption?)
      to show that as k gets large the gap between information-theoretic
      sample complexity and complexity increases? Minor comments:
      --------------- 1. Line 130: The proof **of** item 1. 2. On
      line 343, you mean y_k = b right? -- I think adding a
      discussion about your complexity assumption (with respect to other such
      assumptions) will enhance the value of your paper, and will be
      particularly appealing to a theory audience. In particular, your result
      shows that "hardness conjectures" in learning appear to be weaker than
      other conjectures that people have been willing to make in other areas.
      I agree that showing a simple concept class for which a separation
      exists is interesting. In that sense, I really like your result. However,
      it may help if you clarify explicitly in the writing that you don't
      necessarily claim your assumption to be weaker than existence of one-way
      functions (as it may implicitly appear to say so).


Scalable Influence Estimation in Continuous-Time Diffusion Networks
If a piece of information is released from a media site, can we predict whether it may spread to one million web pages, in a month ? This influence estimation problem is very challenging since both the time-sensitive nature of the task and the requirement of scalability need to be addressed simultaneously. In this paper, we propose a randomized algorithm for influence estimation in continuous-time diffusion networks. Our algorithm can estimate the influence of every node in a network with |V| nodes and |E| edges to an accuracy of using n = O(1/ 2 ) randomizations and up to logarithmic factors O(n|E|+n|V|) computations. When used as a subroutine in a greedy influence maximization approach, our proposed algorithm is guaranteed to find a set of C nodes with the influence of at least (1 − 1/e) OPT −2C , where OPT is the optimal value. Experiments on both synthetic and real-world data show that the proposed algorithm can easily scale up to networks of millions of nodes while significantly improves over previous state-of-the-arts in terms of the accuracy of the estimated influence and the quality of the selected nodes in maximizing the influence.

Problem: Given a directed graph G compute the subset
      of k vertices whose infections leads to the maximum number of nodes being
      infected before time T, under the independent cascade model. The
      paper builds heavily on top of the original paper on influence
      maximization in continuous-time diffusion networks (Gomez-Rodriguez &
      Scholkopf, ICML 2012). There are two pieces of cleverness that appear to
      be unique to this paper: I. Reformulate continuous-time diffusion
      networks in terms of independent variates on the edges. The basic
      contribution of this paper seems to stem from the observation that a
      continuous-time diffusion network can be converted from a distribution
      over the contagion times for nodes (t_i) to a distribution over
      transmission times between nodes (\tau_{ji}). Under the independent
      cascade model, the transmission times between nodes are independent.
      The independent cascade model is a strong assumption in practice,
      but one the authors acknowledge freely. Have the authors considered
      whether there are natural relaxations of the independent cascade model
      which do not violate the independence of transmission times too much:
      e.g., allow a node to be more receptive to contagion if a large number of
      nodes, not necessarily direct parents, are contaminated. Such a relaxation
      might help capture the effects of mass advertising in the viral marketing
      problem. II. Estimating the objective using Cohen's algorithm
      The second contribution of this paper is to show that the
      objective can be reformulated as a neighborhood size estimation problem,
      and that Cohen's randomized algorithm can be directly applied to estimate
      the objective given sample data (Section 5). I was surprised that
      the authors did not mention that Cohen's algorithm has been used before
      for influence maximization, in a different setting (Chen et al, KDD 2009 :
      Section 2). The authors cite the Chen paper in a laundry list of papers
      which consider influence maximization, but neglect to mention how Chen et
      al. use Cohen's algorithm to compute the transitive closure of a directed
      graph to speed up a submodular influence maximization problem under a
      weighted cascade model. I'll point out that Cohen's algorithm handles both
      transitive closure and neighborhood counting on a directed graph.
      I understand that continuous-time diffusion networks are a
      different model from that of Chen et al., but its use in this paper is so
      similar to that of Chen et al., it begs the question---is Section 5 a
      totally novel use of Cohen's algorithm; or is it a novel reworking of an
      idea used previously for influence maximization, to continuous-time
      diffusion networks? General Comments: The paper does not
      assume too much in the way of prior knowledge: the continuous-time
      independent cascade model is explained (Section 2) as is submodular
      influence maximization (Section 6). The material should be accessible to a
      broad audience. The experimental results are quite compelling:
      ConTinEst is vastly faster than the alternatives, and handles more general
      models of transition time. Minor comments: L671: Typo
      "Berstein's"


Fast Determinantal Point Process Sampling with Application to Clustering
Determinantal Point Process (DPP) has gained much popularity for modeling sets of diverse items. The gist of DPP is that the probability of choosing a particular set of items is proportional to the determinant of a positive definite matrix that defines the similarity of those items. However, computing the determinant requires time cubic in the number of items, and is hence impractical for large sets. In this paper, we address this problem by constructing a rapidly mixing Markov chain, from which we can acquire a sample from the given DPP in sub-cubic time. In addition, we show that this framework can be extended to sampling from cardinalityconstrained DPPs. As an application, we show how our sampling algorithm can be used to provide a fast heuristic for determining the number of clusters, resulting in better clustering. There are some crucial errors in the proofs of the theorem which invalidate the theoretical claims of this paper. Please consult the appendix for more details.

This paper provides an efficient sampling algorithm
      for DPPs. Though their algorithm is not explicitely better than the
      standard sampling algorithm for DPPs, in practice their algorithm
      outperforms the standard algorithm. They also consider real world and
      synthetic applications involving clustering, there they use the DPPs to
      give a hueristic for sampling. I thnk the application of Markov
      chain mixing techniques to sampling DPPs is nice, and the faster sampling
      algorithm could be useful in many applications. Overall the paper also
      seems well written. However, I am not fully convinced on the
      application of DPPs for clustering. Though DPPs seem to model diversity in
      many real world applications, it is not intuitive that the DPPs are in
      general a better model for clustering. Moreover, even using DPPs there are
      no guarantees for clustering and the procedure seems hueristic. In that
      context, it is not clear why the DPPs will necessarily offer a better
      solution for clustering. I think this should be clarified better in the
      paper. Also, are there some other applications where one might want to
      sample from a DPP? A recent relevant paper at UAI this year, is
      "Determinantal Clustering Processes - A Nonparametric Bayesian Approach to
      Kernel Based Semi-Supervised Clustering". Though not exactly the same
      problem, the paper is closely related to this one. * Edits after
      the author rebuttal period and discussions I feel that the paper is
      well written, though only incremental in my opinion. The main cons are
      that the application of the sampling algorithm to k-means seems very
      heuristic, and hence I will stick to my score of 6 (i.e borderline
      accept). I think the paper can be significantly improved, if the authors
      could point out some other applications of DPP sampling, where their
      algorithms will imply significant improvement in running
      time.


It is all in the noise: Efficient multi-task Gaussian process inference with structured residuals
Multi-task prediction methods are widely used to couple regressors or classification models by sharing information across related tasks. We propose a multi-task Gaussian process approach for modeling both the relatedness between regressors and the task correlations in the residuals, in order to more accurately identify true sharing between regressors. The resulting Gaussian model has a covariance term in form of a sum of Kronecker products, for which efficient parameter inference and out of sample prediction are feasible. On both synthetic examples and applications to phenotype prediction in genetics, we find substantial benefits of modeling structured noise compared to established alternatives.

Update after reading the authors' rebuttal:
      Very good paper but I strongly recommend the authors to consider
      all the suggestions below that will improve the readability of the paper.
      ===== 1. Summary: This paper presents a multi-task
      Gaussian process regression approach where the covariance of the main
      process (signal) decomposes as a product of a covariance between tasks and
      a covariance between inputs (sample covariance). It is assumed that all
      the training outputs are observed at all inputs, which leads to a
      Kronecker product covariance. Noisy observations are modeled via a
      structured process and this is the main contribution of the paper. While
      previous work on multi-task GP approaches with Kronecker covariances has
      considered iid noise in order to carry out efficient computations, this
      paper shows that it is possible to consider a noise process with Kronecker
      structure, while maintaining efficient computations. In other words, as in
      the iid noise case, one never has to compute a Kronecker product and hence
      computations are O(N^3 + T^3) instead of O(N^3T^3). This is achieved by
      whitening the noise process and projecting the (noiseless) covariance of
      the system into the eigen-basis of the noise covariance (scaled by the
      eigenvalues). Their experiments show that the proposed
      structured-noise multi-task GP approach outperforms the baseline iid-noise
      multi-task GP method and independent GPs on synthetic data and real
      applications. 2. Quality The paper seems
      technically sound once one realizes how to fix the notational
      inconsistencies between section 2 and section 3 (please see item 4 below
      regarding Clarity). The main claim that one can have structured noise in
      multi-task GP models with product covariances while still maintaining
      computational efficiency (compared to the naive approach) is well
      supported with the mathematical derivations in section 3 and with the
      experimental results in section 4. With respect to the results,
      although it is obvious that the naive approach would be much worse in
      terms of computation compared to the efficient approach, it is still
      helpful to see the comparison in Figure 1 so one can take into
      consideration the possible overhead in implementing the GP-KS method.
      However, there are a few deficiencies that need to be pointed out:
      (a) There is a rank-1 parameterization of C and \Sigma. However,
      it is unclear how the parameter \sigma in line 246 was set. This parameter
      is important as it allows that the algorithm can actually run but also
      that one does not over-smooth the process leading to poor generalization
      performance. (b) It is completely unclear what \Omega is actually
      used (please see Clarity below). This should be explicitly said in the
      experiments. From section 2, it can be inferred that \Omega = I_{NN}. But
      the reader should not be guessing about something that must be explicit.
      (c) Line 328: "Large-scale prediction .." : 123 tasks and 218
      samples are very far from what can be considered as large-scale.
      (d) In the experiments regarding the prediction of gene expression
      in yeast, it looks like the preprocessing and filtering of the dataset
      does favor the proposed method (which may have problems with
      identifiability) as genes with low signal and low noise are discarded. The
      authors should provide comments on this. (e) The authors have not
      analyzed possible weaknesses of their method. In particular, interpreting
      the results in Figure 3 is a bit misleading as it seems that their method
      has high levels of unidentifiability. Why is it possible to interpret the
      results? There may be completely different qualitative solutions that lead
      to similar quantitative performance. Is identifiability an issue during
      optimization? (f) Multi-task settings usually compare to a single
      GP for all tasks (i.e. pool GP). This baseline is missing. 3.
      Clarity In terms of language use, the paper is relatively well
      written. However, there are quite a few notational inconsistencies that
      may push this paper below the threshold. For example: (a) The same
      symbol (k) in Equation 1 is used for both covariance (sample and task)
      functions and this is completely inconsistent with the following notation
      in the paper (C, R). On the same equation, is this really the covariance
      between the noisy outputs y or should it be between their corresponding
      noiseless latent functions? (b) Sometimes C_{TT}, R_{NN}, etc are
      used and other times C, T, are used (c) This is inconsistency is
      crucial to understand the paper: I_{NN} is used in Equation 3 but then
      \Omega is introduced in section 3 without explaining what it refers
      to. Is \Omega = I_{NN}? (d) K is undefined before being used in
      Equation 6 (e) Equation 9 does not make sense. It goes from a
      vector on the line above to a matrix. Is there a Vec operator missing?
      4. Originality The approach to multi-task GP
      regression differs from most previous work in the way the noise process is
      modeled (structured noise compared to idd noise) while maintaining
      efficiency during inference. These types of processes have been considered
      before for example by Zhang (2007) but efficient computations were not
      explicitly done when considering the specific case of two processes.
      Without committing to a specific approach for flexible multi-task
      models, it seems necessary to at least mention how this works compare to
      the Gaussian process regression network framework in [6].
      Additionally, sec 2.2 is obvious and should be omitted. The case C
      = \Sigma leads to C \otimes (R+I), which is the same proof shown in
      previous work [1]. 5. Significance The contribution of
      this paper is relatively significant in that it shows that it is possible
      to do efficient computation in these types of models when a sum of two
      Kronecker products is present. This can be exploited in scenarios
      different to the regression setting. However, in terms of the original
      motivation, i.e. multi-task regression, there are other more flexible
      models [6] for which inference is still better than the naive approach
      (N^3 T^3). 6. Additional comments (a) Abstract, lines
      18-20: This is not true. References: Hao Zhang.
      Maximum-likelihood estimation for multivariate spatial linear
      coregionalization models. Environmetrics, 18(2):125–139, 2007


A Comparative Framework for Preconditioned Lasso Algorithms
The Lasso is a cornerstone of modern multivariate data analysis, yet its performance suffers in the common situation in which covariates are correlated. This limitation has led to a growing number of Preconditioned Lasso algorithms that pre-multiply X and y by matrices P X , P y prior to running the standard Lasso. A direct comparison of these and similar Lasso-style algorithms to the original Lasso is difficult because the performance of all of these methods depends critically on an auxiliary penalty parameter λ. In this paper we propose an agnostic framework for comparing Preconditioned Lasso algorithms to the Lasso without having to choose λ. We apply our framework to three Preconditioned Lasso instances and highlight cases when they will outperform the Lasso. Additionally, our theory reveals fragilities of these algorithms to which we provide partial solutions.

The paper addresses preconditioned lasso estimators,
      and studies their utility from a non-asymptotic, theoretical framework.
      Their main result is a set of bounds (lower and upper) on lambda that are
      necessary and sufficient for exact support recovery---they then apply this
      result to study the effects of different preconditioning methods. I like
      the general perspective of the paper (though I think the authors could
      have given a better motivation for their approach). The work is generally
      well-written, and as far as I can tell, novel. I do have two major
      comments, given below. After that, I give several more minor comments.
      ========== Major comments: - Lemma 1 seems like a
      novel and interesting result. But at the same time, I'm suspicious of the
      "if and only if" here. For this result to be true, it would need to hold
      that \lambda_l and \lambda_u, as you've defined them, are knots of the
      lasso solution path. From the LARS algorithm (homotopy algorithm) for the
      lasso path, we know that the signed support of the lasso solution is
      piecewise constant, and only changes at data-dependent knots \lambda_1
      \geq \lambda_2 \geq \ldots \lambda_r. Hence it either \lambda_l or
      \lambda_u did not coincidence with one of these knots, then we could slide
      it to the closest knot (slide to the left for \lambda_l, and to the right
      for \lambda_u), and we would know that the constructed support wouldn't
      have changed. Generally, the formulae for these knots, \lambda_1
      \geq \lambda_2 \geq \ldots \geq \lambda_r, are complicated since they
      depend on the constructed support \hat{S} and signs of the lasso solution
      along the lasso solution path. You've written down formulae for \lambda_l
      and \lambda_u that depend on S, not \hat{S}, so at a high level, I don't
      understand how this can be right. But at a lower level, skimming
      through the proof of your Lemma 1 in the supplementary material, I don't
      see any mistakes. Some comments/clarification on my concerns here would be
      very helpful. UPDATE: I now believe that this result is correct.
      Given the assumed conditions, we have \hat{S}=S (perfect recovery) so my
      initial intuitive concerns are remedied. Furthermore, what you have
      written down are actually knots in the lasso path, i.e., these can be
      shown to coincide with "hitting times", the values of lambda at which
      variables enter the model. - Section 3.2: It seems that the ratios
      of the upper and lower lambda bounds, to compare the usual and
      preconditioned lasso estimates, is not the right metric---instead you
      would want to compare their relative widths: ( \lambda_u - \lambda_l ) /
      \lambda_max, where \lambda_max is the smallest value of \lambda for which
      all components of the estimate are zero. This has a closed form:
      \lambda_max = \|X^T y\|_\infty, i.e., the largest absolute component of
      X^T y. Clearly \lambda_max is going to be different between the two
      problems (as X becomes P_X X and y becomes P_y y for the preconditioned
      lasso). A simple example shows that using the ratios can give
      somewhat undesirable answers: if \lambda_u = 4 and \lambda_l = 2 for the
      lasso, and \lambda_u = 2 and \lambda_l = 1 for the preconditioned lasso,
      then your metric would say they perform about the same. But if in this
      case \lambda_max = 10 for the lasso and \lambda_max = 3 for the
      preconditioned lasso, then the story is very different: with respect to
      the full range of effective regularization parameter values, only 1/5 of
      this range recovers the support for the lasso, and 1/3 of it recovers the
      support for the preconditioned lasso. ========== Minor
      comments: - Page 1, line 53: I think you mean sufficient (not
      necessary) for support recovery - Page 2, line 80: I'm not really
      sure what you mean by "asymptotic analyses ... provide limited theoretical
      insight". Of course, an asymptotic analysis is entirely theoretical, and
      so yes, it does provide such insight. You mean something like satisfactory
      insight? - In motivating your approach on page 2, it seems to me like
      your method is actually closely aligned with comparing the entire
      regularization paths of (2) and (4), as done empirically in the reference
      [4]. You are basically answering the question: "does the regularization
      path contain the true model, at any point along the path?" But you are
      doing this in an efficient way by deriving upper and lower bounds on
      lambda. - Page 3, line 119: It seems strange to take the q
      *smallest* singular vectors of U (by this, I assume you mean the singular
      vector corresponding to the smallest singular values). Can you comment on
      this? This is especially relevant in Theorem 1, where you impose the
      condition span(U_S) \subseteq span(U_A). With this construction of A, we
      would probably not at all expect the sets S and A to be close, so this
      condition seems quite unreasonable ... - I'm not sure why you
      don't present the deterministic result for JR (only one based on a
      generative model)? For completeness in comparing the preconditioning
      methods, this would have been helpful. - Equation (21) is
      confusing: I think I know what you mean here, but this equation is
      somewhat confusing without seeing context for \hat{\Sigma} (just calling
      \hat{\Sigma} "unconstrained spectra" is not that informative) ...
      - I'm confused about the setup in 5.3. Why have you changed the
      number of observations to m, and what does it mean for m \rightarrow
      \infty while n stays fixed?