Regression-tree Tuning in a Streaming Setting
We consider the problem of maintaining the data-structures of a partition-based regression procedure in a setting where the training data arrives sequentially over time. We prove that it is possible to maintain such a structure in time O (log n) at any time step n while achieving a nearly-optimal regression rate ofÕ n −2/(2+d) in terms of the unknown metric dimension d. Finally we prove a new regression lower-bound which is independent of a given data size, and hence is more appropriate for the streaming setting.

The authors derive a new procedure to estimate a
      recursive-partition prediction rule in the streaming framework.
      Theoretical analyses demonstrate that the procedure is computationally
      efficient and attains the minimax prediction error rate, up to a log
      factor. A small empirical analysis is in agreement with the theory.
      The paper is excellent: the authors have produced an intuitive
      streaming algorithm with nearly sharp theoretical guarantees in terms
      of intrinsic dimension. The exposition is very clear and easy to
      follow. I have been hoping for some time to see a streaming regression
      tree paper like this. Properly publicized, this work is liable to have
      a big impact in the large-scale learning community. Regarding
      originality, the authors have overlooked some work on this problem --
      but that work focuses on bounding the difference between the streaming
      tree and the corresponding batch tree, which is a different question
      altogether. So there's no harm done. I've listed some references
      below. I warmly recommend acceptance. Minor comments.
      p.2: "everywhere on the space rho" -> "everywhere on the space
      calX" p.2: "The metric (calX, rho)" -> "The metric space (calX,
      rho)" p.3: "an new guess" -> "a new guess" p.3: "quantifieable"
      -> "quantifiable" [***** SPELL CHECK *****] p.3: "there exist C"
      -> "there exists C" References. @inproceedings{,
      author = {Domingos, Pedro and Hulten, Geoff}, booktitle =
      {Proceedings of the 6th ACM SIGKDD International Conference on Knowledge
      Discovery and Data Mining}, title = {Mining high-speed data streams},
      pages = {71--80}, year = {2000} } @inproceedings{,
      author = {Hulten, Geoff and Spencer, Laurie and Domingos, Pedro},
      booktitle = {Proceedings of the 7th ACM SIGKDD International
      Conference on Knowledge Discovery and Data Mining}, title = {Mining
      time-changing data streams}, pages = {97--106}, year = {2001}
      } @inproceedings{, author = {Pfahringer, Bernhard and
      Holmes, Geoffrey and Kirkby, Richard}, booktitle = {Advances in
      Knowledge Discovery and Data Mining: Proceedings of the 12th Pacific-Asia
      Conference (PAKDD)}, title = {Handling numeric attributes in Hoeffding
      trees}, volume = {5012}, pages = {296--307}, publisher =
      {Springer}, year = {2008} } @article{, author =
      {Ben-Haim, Yael and Tom-Tov, Elad}, title = {A streaming parallel
      decision tree algorithm}, journal = {Journal of Machine Learning
      Research}, volume = {11}, pages = {849--872}, year = {2010}
      } NB: I have read the author rebuttal.


Accelerated Mini-Batch Stochastic Dual Coordinate Ascent
Stochastic dual coordinate ascent (SDCA) is an effective technique for solving regularized loss minimization problems in machine learning. This paper considers an extension of SDCA under the mini-batch setting that is often used in practice. Our main contribution is to introduce an accelerated mini-batch version of SDCA and prove a fast convergence rate for this method. We discuss an implementation of our method over a parallel computing system, and compare the results to both the vanilla stochastic dual coordinate ascent and to the accelerated deterministic gradient descent method of Nesterov [2007].

This paper extends stochastic dual coordinate ascent
      (SDCA) to the mini-batch setting, with a proposed algorithm accelerated
      mini-batch SDCA (ASDCA). Its rate of convergence interpolates between SDCA
      and accelerated gradient descent (AGD). In addition, a parallel
      implementation is also proposed where SDCA again interpolates between SDCA
      and AGD. This interpolation is also demonstrated in experiment.
      Although the rates of convergence for ASDCA are interesting, a
      careful look at Table 2 and Figure 1 reveals that ASDCA is nowhere
      superior to SDCA. I am wondering why it is useful to have such an
      algorithm. In addition, Section 3 showed much promise of ASDCA on parallel
      architecture. However, no experiment was done in this setting. Therefore
      more experimental studies (especially on parallel architecture) are
      necessary before we can confirm the empirical usefulness/superiority of
      ASDCA (as indicated by the word ‘Accelerated’ in the title). In
      response to author's rebuttal: I stick to my position that the paper
      should show empirical evidence of the advantage of the new algorithm. I
      think this is an important issue, because in practice the acceleration
      proffered by parallelization can be highly tricky. The theory part of the
      paper is surely good. So I won't mind if the paper is
      accepted.


Faster Ridge Regression via the Subsampled Randomized Hadamard Transform
We propose a fast algorithm for ridge regression when the number of features is much larger than the number of observations (p n). The standard way to solve ridge regression in this setting works in the dual space and gives a running time of O(n 2 p). Our algorithm Subsampled Randomized Hadamard Transform-Dual Ridge Regression (SRHT-DRR) runs in time O(np log(n)) and works by preconditioning the design matrix by a Randomized Walsh-Hadamard Transform with a subsequent subsampling of features. We provide risk bounds for our SRHT-DRR algorithm in the fixed design setting and show experimental results on synthetic and real datasets.

The authors propose a method (SHRT) to accelerate
      Ridge Regression when the number of features p is much greater than
      the number of samples n (p >> n). The main idea is to reduce
      the dimensionality of the design input matrix X to make the
      computation of XX^T faster, from O(n^2) to O(nplogn). This is done via
      Walsh-Hadamard transforms, which can be computed in log-time.
      The research idea is of high-quality and great use. The paper is
      well written and technically correct. However, my main concern is the
      significant overlap to the recently published paper: "Fastfood
      – Computing Hilbert Space Expansions in loglinear time, Quoc Le; Tamas
      Sarlos; Alexander Smola", ICML 2013. Both submissions are very
      close in time, so I believe they were developed independently.
      Fastfood seems completely analogous to the here proposed SHRT
      algorithm, but more general: SHRT(X) := RHDX FASTFOOD(X) :=
      exp(SHGPHBX) 1) The basis matrix R in SRHT is the matrix P in
      Fastfood. 2) The Walsh-Hadamard matrix H in SRHT is the matrix H in
      Fastfood. 3) The Rademacher matrix D is the matrix B in Fastfood.
      4) The main difference is the setup of p >> n for SHRT. In
      Fastfood, the number of random features p_subs to generate can be
      greater than p, so multiple "SHRT blocks" can be built. 5) In SHRT
      there exists no matrix "G", which corresponds to spectral samples of a
      Gaussian kernel in Fastfood. This is because in SHRT a huge feature
      space is assumed, so linear kernel is used (that is, Fastfood's G and
      S matrices are diagonal in SHRT). This is also the reason why SHRT
      is not exponentiated. The theoretical developments in Lemma 2
      and Theorem 1 are mathematically well executed and are novel material.
      This is also the case for the application of the method to the PCA
      algorithm. Therefore, there is enough novelty in the submission.
      The authors should cite Rahimi & Brecht works (2007, 2008) and
      the Fastfood paper. I believe that the admission of this work
      demands a better connection with existing work and (ideally) a
      comparison to other randomized methods. I have read the Author
      Feedback.


Stochastic Ratio Matching of RBMs for Sparse High-Dimensional Inputs
Sparse high-dimensional data vectors are common in many application domains where a very large number of rarely non-zero features can be devised. Unfortunately, this creates a computational bottleneck for unsupervised feature learning algorithms such as those based on auto-encoders and RBMs, because they involve a reconstruction step where the whole input vector is predicted from the current feature values. An algorithm was recently developed to successfully handle the case of auto-encoders, based on an importance sampling scheme stochastically selecting which input elements to actually reconstruct during training for each particular example. To generalize this idea to RBMs, we propose a stochastic ratio-matching algorithm that inherits all the computational advantages and unbiasedness of the importance sampling scheme. We show that stochastic ratio matching is a good estimator, allowing the approach to beat the state-of-the-art on two bag-of-word text classification benchmarks (20 Newsgroups and RCV1), while keeping computational cost linear in the number of non-zeros.

Summary: This paper develops an algorithm that
      can successfully train RBMs on very high dimensional but sparse input
      data, such as often arises in NLP problems. The algorithm adapts a
      previous method developed for denoising autoencoders for use with RBMs.
      The authors present extensive experimental results verifying that their
      method learns a good generative model; provides unbiased gradient
      estimates; attains a two order of magnitude speed up on large sparse
      problems relative to the standard implementation; and yields state of the
      art performance on a number of NLP tasks. They also document the curious
      result that using a biased version of their estimator in fact leads to
      better performance on the classification tasks they tested. Pros:
      While the underlying idea is present in previous work with
      autoencoders, transferring this idea to the RBM setting required some
      non-trivial insights, particularly into the ratio-matching objective for
      training RBMs. Every claim is supported by an appropriate figure
      (learns good generative models; gradient estimates are unbiased;
      importance sampler proposal distribution selects large-gradient samples;
      attains a large speed up relative to standard implementation; yields state
      of the art performance on certain NLP tasks). The experiments have been
      done to a high standard. The paper includes some useful other
      results, including a highly intuitive form for the ratio matching
      objective, and a method for saving intermediate variables that allows more
      computationally efficient computation of the free energy terms required by
      the ratio matching objective. The paper is very clearly written.
      Cons: While the proposed method does achieve
      state-of-the-art results, its performance is highly similar to that of the
      denoising autoencoder. There is no real case made in the paper that the
      RBM algorithm is particularly more useful than the already developed DAE
      version of the algorithm. The experimental results do not give
      quantitative classification performance results for the unbiased SRM
      algorithm (e.g. Tables 2-3). While it is stated in the text that the
      unbiased method was not the top performer, it would still be informative
      to get a sense of the magnitude of the difference. It would also be
      valuable to clarify if the denoising autoencoders were trained with or
      without biased gradient estimates. Since biased gradients work better for
      the RBM, it seems likely this is true of the autoencoder as
      well.


Perfect Associative Learning with Spike-Timing-Dependent Plasticity
Recent extensions of the Perceptron as the Tempotron and the Chronotron suggest that this theoretical concept is highly relevant for understanding networks of spiking neurons in the brain. It is not known, however, how the computational power of the Perceptron might be accomplished by the plasticity mechanisms of real synapses. Here we prove that spike-timing-dependent plasticity having an anti-Hebbian form for excitatory synapses as well as a spike-timing-dependent plasticity of Hebbian shape for inhibitory synapses are sufficient for realizing the original Perceptron Learning Rule if these respective plasticity mechanisms act in concert with the hyperpolarisation of the post-synaptic neurons. We also show that with these simple yet biologically realistic dynamics Tempotrons and Chronotrons are learned. The proposed mechanism enables incremental associative learning from a continuous stream of patterns and might therefore underly the acquisition of long term memories in cortex. Our results underline that learning processes in realistic networks of spiking neurons depend crucially on the interactions of synaptic plasticity mechanisms with the dynamics of participating neurons.

The paper presents a learning rule for a simple
      spiking neuron model based on STDP, where the STDP for excitatory
      synapses is anti-Hebbian and STDP for inhibitory synapses is Hebbian
      (i.e., it is reversed in terms of the classical rule). The authors
      provide a proof that the learning rule implements the Perceptron
      Learning Rule (PLR) in a specific setup with synchronous input spikes
      and a kind of teacher forcing. They also show that a modified version
      of the rule is able to learn spatio-temporal spike patterns to some
      extend, relating it to tempotron learning. The authors claim that this
      learning scheme is biologically plausible. General assessment:
      Quality: The paper addresses an interesting topic. There seems to
      be a small bug in the theory, but this does not cause a big problem.
      One problem I see is that the manuscript does not contain a comparison
      to the simplified tempotron learning rule which is in my opinion at
      least as biologically plausible as the proposed learning scheme.
      Another concern is that the simulations for spatio-temporal patterns
      are done with a modified neuron and plasticity model with some quite
      ad-hoc modifications. So it is unclear how the initial theoretical
      investigation corresponds to that. Clarity: The paper is
      reasonably well written, but clarity could be improved substantially
      (see below). Originality: The approach is quite probably novel,
      although there exists some previous work that has not been cited.
      Significance: The paper seems to be of some relevance. Since the
      authors do not point to any specific system where this setup could be
      implemented, I currently do not see any biological evidence for it
      apart from the fact that anti-Hebbian STDP has been reported.
      Specific points: 1) Clarity of the presentation 1.1)
      In line 90, it is stated that neurons are recurrently connected. Yet, I
      don't' see any place in the paper where this is the case. If I understand
      correctly, both the theory and the simulations consider simple
      feed-forward architectures. This is confusing as well as an overstatement
      and should be corrected. 1.2) It is unclear to me why the authors
      first introduce the PLR with a {-1,1} coding and then switch to the {0,1}
      coding. This is just confusing since additional variable-symbols are
      needed. Just start with the {0,1} coding. I believe that in many
      text-books it is defined in this way anyways and the mapping between the
      codings is close to trivial. 1.3) Figure 2 is misleading. The PSPs in
      Fig. 2 (red traces) are alpha-shaped with a finite rise time. This is
      misleading since according to the definition the PSPs should really be
      single exponentials. It is confusing because it does not match the
      definitions of ~h and ~x in lines 184-186. 1.4) Parameters alpha and
      beta are used in two different contexts in lines 189 and 310-311.
      2) Formal proof of equivalence There is one point which maybe
      I did not understand correctly. I understand that the margin \epsilon
      for the case of a desired output spike (y_0=1) can be achieved by the
      after-spike depolarization. This means that during training, because of
      the teacher spike, the membrane potential has to be somewhat larger than
      during testing. But I do not see how the analogous case works if the
      neuron should not spike (y_0=0). In this case, the depolarization needed
      to elicit a spike during training is exactly the same as during testing.
      Hence, the neuron cannot learn to bring the depolarization below
      (threshold-margin). Therefore, using "s" in the right term of the equation
      in line 204 is probably not correct. It is again misleading that in line
      198 the authors suggest that beta can be chosen freely. I think this is
      not the case. I actually see no way to correct for this. The only
      thing that can be done is to state that the margin for the PLR can only be
      learned for positive examples (i.e. y_0=1). 3) Comparison to
      tempotron learning The authors compare their learning rule to
      tempotron learning in the text, but they do not exhibit an experimental
      comparison. First, the authors state that the tempotron learning rule is
      biologically unrealistic (line 55), but this applies only to the full
      rule, not to the simplified one. In order to see whether the current rule
      has any advantages over that one, an experimental comparison is needed
      (possibly also a comparison to the full tempotron rule to see how much one
      looses). 4) Learning of spatio-temporal patterns 4.1) Several
      modifications to the original setup are used here which are not well
      motivated. For example, it is unclear why the hyperpolarization variable
      nu is introduced in line 284. I suspect it is needed because one wants to
      avoid that the output neuron spikes more than twice. Also the modification
      of the learning rule eq. (18) is not clear. It looks like a hack and a
      assume that the parameters have to be tuned carefully if something changes
      in the input statistics. I did not understand the explanation in lines
      300-303. 4.2) Related to the last point. I am not sure whether the
      initial setup works for spatio-temporal patterns (therefore probably the
      modifications). The reason is that the second spike of the output neuron
      acts as a teacher for the following spikes. So even if the desired output
      is y_0=0, if the neuron erroneously spiked at time t, for input spikes at
      t'>t, it looks like this was actually a positive pattern. What is the
      value for "T" in line 320? This value deems critical to me. 5)
      Relevance The learning setup is that the trained neuron is forced to
      fire at the same time as the synchronous inputs or before a
      spatio-temporal input pattern. The former condition (for synchronous
      input) seems somewhat plausible in some contexts, the latter much less so.
      In general, if the teacher spike sometimes comes after the input spikes
      (which seems to be the more plausible situation in vivo) then the system
      breaks down. Anti-Hebbian STDP has been reported for distal excitatory
      synapses in pyramidal cells, but on the other hand ref [10] of the
      manuscript also reported that depolarization would abolish that or make it
      Hebbian. This would quite probably destroy the properties needed here.
      Minor points: 6) Literature: The authors may want to
      compare their rule to ReSume (e.g. Ponulak and Kasinski, Neural
      Computation 2010) and discuss the relation to the investigations in
      (Legenstein et al., Neural Computation 2005). 7) It is a bit
      problematic to define the PLR over gradient descent on eq (7) since this
      error is not differentiable at E=0. 8) In line x_0^i >= 0 is used,
      later x_0^i \in {0,1}. 9) line 360: "biologically highly plausible" is
      an overstatement. 10) line 367: "substantial capacity". This is not
      clear. What is a substantial capacity in this case? Comparison with
      tempotron capacity etc.


Convex Tensor Decomposition via Structured Schatten Norm Regularization
We study a new class of structured Schatten norms for tensors that includes two recently proposed norms ("overlapped" and "latent") for convex-optimizationbased tensor decomposition. We analyze the performance of "latent" approach for tensor decomposition, which was empirically found to perform better than the "overlapped" approach in some settings. We show theoretically that this is indeed the case. In particular, when the unknown true tensor is low-rank in a specific unknown mode, this approach performs as well as knowing the mode with the smallest rank. Along the way, we show a novel duality result for structured Schatten norms, which is also interesting in the general context of structured sparsity. We confirm through numerical simulations that our theory can precisely predict the scaling behaviour of the mean squared error.

****Additional comments after the rebuttal: The
      authors need to do a much better job on motivating the problem for machine
      learning, for the paper to be accessible to a wider audience. The authors
      need to provide the latent variable model (with a graphical model
      representation) where this tensor decomposition can learn the parameters:
      Consider a model where latent variable h_1 connects to observed variable
      x_1, h_2 to x_2 and so on. Assume a linear model from hidden to observed
      variables, and arbitrary dependence among the h_i's. In this case, the
      observed moment tensor has a tucker decomposition and its multilinear rank
      is dim(h_1), dim(h_2).. The authors need to add this discussion and
      preferably provide a picture depicting this model. The authors also need
      to point out that it is possible that only some hidden variables are of
      small dimensions and not the others, which was a main advantage for the
      new result in the paper. The paper considers convex-optimization
      based tensor decomposition using structured Schatten norms for tensors.
      The authors employ the overlapped Schatten norm and show how it leads to
      better reconstruction guarantees. The paper is well written and makes
      technically important contributions. Fixing a few technical and notational
      issues, will make the paper even more readable. *Clarification
      about Theorem 2 and its proof: Statement of theorem 2 is
      technically not correct: the components are only recovered up to a
      permutation. Although a minor issue, the authors need to mention this.
      In Theorem 2, it is stated that (12) immediately leads to (13).
      This needs a justification. Proof of Lemma 5 is missing. Lemma 5
      forms a key part of Theorem 2. It is better to clarify the proof
      of Lemma 4 explicitly rather than referring to another paper, although it
      is straightforward. *Minor issues: Need to add more
      related work on HOSVD and other methods for tensor decomposition: The
      authors can mention how the higher order SVD and its variants yields
      approximation bounds for tensor decomposition. See Kolda's review on
      tensor decompositions (section 4). The notion of rank that the
      authors use, which they refer to as mode-rank is also referred ot as
      multilinear rank. The authors can mention this and note there are
      different notions of tensor rank in the introduction. The authors
      can briefly discuss the computational complexity of the methods employed.
      The authors need to clarify their discussion about incoherence
      after (10). The authors use the term to mean that they need the "energy"
      to be spread across different mixture components and modes, i.e., no
      component or mode can have too high a singular value. The authors mention
      that this is the same as the incoherence conditions of [1,3,10]. I do not
      see this. In [1,3,10], incoherence notions are use for sparse+low rank
      recovery. However, here, the noise is not sparse. The condition of
      [1,3,10] involves the singular vectors being incoherent with the basis
      vectors. This is different from the condition here. Suggest removing this
      comparison. *Typos/notational issues: Superscript notation
      for components \hat{\cal{W}}^{(k)} is confusing. Suggest using some other
      symbol for components. Bad notation: \bar{r}_k and \underline{r}_k
      are both used for k-mode rank. This should be changed. Could introduce
      mult_rank(tensor) instead to denote the multilinear rank. In Last
      line of page 12 in supplementary material, some parenthesis is missing.
      Abstract: "result for structures Schatten" should change to
      structured


A message-passing algorithm for multi-agent trajectory planning
We describe a novel approach for computing collision-free global trajectories for p agents with specified initial and final configurations, based on an improved version of the alternating direction method of multipliers (ADMM). Compared with existing methods, our approach is naturally parallelizable and allows for incorporating different cost functionals with only minor adjustments. We apply our method to classical challenging instances and observe that its computational requirements scale well with p for several cost functionals. We also show that a specialization of our algorithm can be used for local motion planning by solving the problem of joint optimization in velocity space.

The paper considers global and local path planning for
      multiple agents in 2-D with a centralized message-passing algorithm
      derived from the "three-weight" version of ADMM, an established algorithm.
      The contributions are clearly stated in the introduction: The authors
      decompose global planning optimization into several sub-problems they dub
      "minimizers," which describe various planning objectives that comprise the
      larger overall problem to be solved. Minimizers are derived for avoiding
      inter-agent collisions, avoiding collisions with static obstacles, and for
      maximizing/minimizing kinetic energy or velocity. They also apply their
      approach to local planning by reformulating joint optimization.
      The overall concept for using a distributed optimization algorithm
      for multi-agent path planning is nice; however the results are a
      little lacking. They do not provide comparisons to other well-known high
      dimensional path planning techniques, such as A* variants with
      inflated heuristics or sampling-based methods. For many of these problems,
      there is also the aspect of replanning given new state or obstacle
      information. Would this algorithm be amenable to such dynamic or anytime
      implementations? In that regard, it's hard to see how this formulation
      would guarantee that certain constraints would always be enforced for
      early stopping, for example a hard collision or communication constraints.
      These and other issues regarding the planning problem should be discussed.
      It's also not clear if this manuscript would appeal to the broader NIPS
      audience. It would definitely fit into a dedicated planning/robotics
      meeting, but it's hard to see if this would appeal from an algorithmic
      viewpoint to other NIPS areas. There are also a few minor items
      that could be improved in the presentation. First of all, please use a
      letter other than "n" for the correction messages in your formulations.
      "n" is already used to describe the number of interior break-points, a
      major parameter for the overall algorithm. Figure 2-middle and Figure
      3-left include multiple scales on either the x- or y-axis. In the case of
      Figure 3-left, I initially thought that the convergence time between
      optimizing total energy and finding a feasible solution were comparable,
      when one is several orders of magnitude larger. These figures seem to be
      an attempt to fit the paper within the page limit but could cause
      confusion and dilute the impact of the results. Also, with respect to the
      convergence times reported for local trajectory planning using the
      proposed approach versus MIQP [12], the authors compare implementations in
      different programming languages and say the results are not "exactly
      comparable." Yet, they directly compare the convergence times in Figure
      3-right. They also state, "our method does not require approximating the
      search domain by hyperplanes, nor does it require an additional
      branch-and-bound algorithm," which causes one to further question the
      direct comparison of convergence times. These results could be clarified.


Auditing: Active Learning with Outcome-Dependent Query Costs
We propose a learning setting in which unlabeled data is free, and the cost of a label depends on its value, which is not known in advance. We study binary classification in an extreme case, where the algorithm only pays for negative labels. Our motivation are applications such as fraud detection, in which investigating an honest transaction should be avoided if possible. We term the setting auditing, and consider the auditing complexity of an algorithm: the number of negative labels the algorithm requires in order to learn a hypothesis with low relative error. We design auditing algorithms for simple hypothesis classes (thresholds and rectangles), and show that with these algorithms, the auditing complexity can be significantly lower than the active label complexity. We also show a general competitive approach for learning with outcome-dependent costs.

Overview: The authors address the problem of
      auditing, a restriction of the more general class-dependent cost sensitive
      active learning for binary classification where one pays for receiving
      labels on just the ‘negative’ examples. The paper presents a number of
      theoretical results for ‘auditing complexity’ (the number of negative
      labels required). Firstly, they note that, like active learning, in the
      general case auditing cannot necessarily improve over passive learning.
      Their main result is to show improved auditing complexity over active
      label complexity for threshold and axis aligned rectangular decision
      boundaries (both in the realisable and noisy, agnostic case). Finally they
      present a general algorithm to adapt a greedy active learner to auditing
      achieving constant speedup and the same complexity as the learner of
      Golovin, Krause (2011). Quality: In this work they present new
      complexity results, and the algorithms by which they may be achieved. The
      theoretical content of the paper is thorough and the algorithms presented
      yield insight into the differences between the auditing and active
      learning problem. The main results, and algorithms provided, apply only to
      highly restricted settings that are unlikely to be of much use in
      practice. They show improved complexity of auditing only for data that can
      be classified with axis aligned thresholds, or rectangles - with a further
      restriction that the algorithm only works if the negative examples are on
      the inside of the rectangle. In the noisy case (as discussed in the paper)
      the algorithm must be provided with the noise level in advance. Therefore
      although the theoretical results are insightful, these restrictions appear
      to strong to be of practical relevance. One concern I have is with
      the agnostic case for the rectangular classification boundary - although
      there is an exponential reduction in complexity with respect to the noise
      level \eta, there is a polynomial increase the dimension, d. It is not
      clear to me in practice whether this trade-off will necessarily be
      beneficial, further discussion of typical values (or even an empirical
      evaluation) would make the case that auditing complexity is better than
      the standard active sampling complexity more convincing. Although
      the paper is theoretical in nature, I find the complete lack of
      experimental evaluation of their proposed algorithms disappointing. Even
      if only on toy data, if the authors could show empirically the improved
      auditing performance of their auditing algorithms over both passive and
      active sampling it would make the paper more impactful to the more general
      reader. Clarity: The paper is clearly written and intuitions
      are provided to assist the reader in understanding the high-level workings
      of the algorithms and exploitable structure for auditing in the scenarios
      addressed in the paper. Originality: Although auditing has
      been indirectly addressed as it is a specific instance of cost-sensitive
      active learning, addressing the problem directly seems novel and useful.
      The application to fraud presented in the introduction is convincing,
      perhaps the authors could provide a could more motivating examples here?
      The authors claim that in existing work in cost-sensitive active
      learning the costs need to be known in advance - however, for example, in
      the work of Kapoor et al (2007) the cost may be class dependent which is
      unknown at time of sampling, they take expectations of the cost function
      to account for this uncertainty. Surely their method could be directly
      applied to the auditing scenario too? Significance: The paper
      provides new theory on auditing, which is a relatively unstudied area.
      Although I feel that the algorithms provided have little practical
      significance the results presented in this paper yield insight into the
      auditing problem and could provoke further study into the practical side
      of active auditing.


Spike train entropy-rate estimation using hierarchical Dirichlet process priors
Entropy rate quantifies the amount of disorder in a stochastic process. For spiking neurons, the entropy rate places an upper bound on the rate at which the spike train can convey stimulus information, and a large literature has focused on the problem of estimating entropy rate from spike train data. Here we present Bayes least squares and empirical Bayesian entropy rate estimators for binary spike trains using hierarchical Dirichlet process (HDP) priors. Our estimator leverages the fact that the entropy rate of an ergodic Markov Chain with known transition probabilities can be calculated analytically, and many stochastic processes that are non-Markovian can still be well approximated by Markov processes of sufficient depth. Choosing an appropriate depth of Markov model presents challenges due to possibly long time dependencies and short data sequences: a deeper model can better account for long time dependencies, but is more difficult to infer from limited data. Our approach mitigates this difficulty by using a hierarchical prior to share statistical power across Markov chains of different depths. We present both a fully Bayesian and empirical Bayes entropy rate estimator based on this model, and demonstrate their performance on simulated and real neural spike train data.

The paper introduces a way to estimate the entropy
      rates (and transition probabilities) of a Markov Model from
      observations, by improving the empirical bin count through a
      tree-based Bayesian prior which takes into account the progress of the
      probabilities through time and assigns more similar future
      probabilities to observations which differ further in the past.
      The paper has some interesting ideas, and the sequential
      construction of the Dirichlet priors and the extraction of the
      estimates is elegant. Certainly, the results look good - but in
      Fig. 3(c), the superiority of convergence of the HDP estimators is
      not really clear compared to the others. The text in lines 407-409
      seems to insinuate that the performance of the HDP estimators is
      somehow special whereas it is quite on coherence with the other
      methods. The transition probability, of course, is nicely
      regularized, but stronger data is required to have an idea how good
      this regularization is. Fig. 2(a), for instance shows a good match
      of the high-probability transitions, but the low-probability
      transitions seem not as well matched. This is slightly surprising,
      as there are only two possibilities for the transition, so it is not
      clear why the high transition probability values should be
      regularized to a higher degree than the low transition probability
      values. nevertheless, one may wish some more reference to
      other reconstruction methods, such as CSSR by Shalizi et al,
      suffix-tree methods (Y. Singer et al.), observable operator methods
      (Jaeger). There is a lot of emphasis on the weaknesses of pure
      entropy estimation in the paper, but clearly also the structure of
      the reconstruction mechanism hovers in the background which deserves
      mention as your construction follows the spirit of the epsilon
      machine reconstruction (Shalizi & Crutchfield) and may find
      relevant generalizations in that direction. Detailed comments:
      - line 124: what's e in log(e)? - line 128: second H_NSB
      has wrong subscript - line 177: it is not clear what the bold
      characters mean - throughout the paper, until quite late, I was
      confused whether the new symbol appears at the *right* or the *left*
      end of the string, and my confusion was caused mainly by this
      illustration. Why does 010 transitionf to 101? If the new symbol
      is on the right, where does the 0 at the left end disappear to? Do
      we have a 2-time step memory system only here? - line 227:
      similar issue - "g_0011 should be similar to g_011" - which one is the
      latest symbol. Here it seems it may be the first one. Later in the
      text you seem to refer to the latest symbol at the right. Please
      clarify. - line 249: "whose sample paths are each probability
      distributions" - this sentence is unclear, information seems
      missing; what condition do the probability distributions
      associated with the paths have to fulfil? - line 264: there is
      information missing here - it seems that when you start with a fresh
      distribution (p_0), you still seem to assume existence of N
      observations (the length of your data string). How come you do not
      start with the empty string straight away and build up your Dirichlet
      prior as you go through the first N entries? - line 269: I do
      not understand the structure of the posterior predictive distribution.
      You state the set g_{|k|}, but I do not see the connection between the
      set and the required structure of the distribution. Reformulate. Also
      the rest of the paragraph below line 272 is unclear. Please fix that,
      maybe it is a consequence that I did not understand your construction
      of g_pr. - line 294: I have the impression that the line contains
      an error: the Beta function for the g_{0s} and the g_{1s} terms
      probably need an \alpha_{|s|+1} instead of \alpha_{|s|} - line
      301: "each \alpha as fixed hyperparameters" - elide the "s"


Fantope Projection and Selection: A near-optimal convex relaxation of sparse PCA
We propose a novel convex relaxation of sparse principal subspace estimation based on the convex hull of rank-d projection matrices (the Fantope). The convex problem can be solved efficiently using alternating direction method of multipliers (ADMM). We establish a near-optimal convergence rate, in terms of the sparsity, ambient dimension, and sample size, for estimation of the principal subspace of a general covariance matrix without assuming the spiked covariance model. In the special case of d = 1, our result implies the near-optimality of DSPCA (d'Aspremont et al. [1]) even when the solution is not rank 1. We also provide a general theoretical framework for analyzing the statistical properties of the method for arbitrary input matrices that extends the applicability and provable guarantees to a wide array of settings. We demonstrate this with an application to Kendall's tau correlation matrices and transelliptical component analysis.

The authors consider the Sparse PCA problem, i.e., PCA
      with the assumption that the "principal vectors" depend on only a few of
      the variables. They propose a convex relaxation for solving this problem
      and analyze its performance. An ADMM based method is also developed for
      solving the resulting SDP efficiently. Finally they show theoretical
      results when a Kendall's tau matrix is used as input instead of the usual
      sample covariance matrix. The paper is clearly written and the
      authors have done a good job in explaining various concepts as they become
      used in their exposition. The results presented in this paper appear to be
      correct. I like the convex formulation. The techniques used to
      achieve it are simple and this might certainly be considered one of the
      strengths of the paper. It also goes along nicely with the ADMM
      formulation and Lemma 4.1. I have a minor suggestion : it would be nice if
      the authors mention the fact that they have a closed form expression for
      the projection onto the fantope a little earlier in the paper (possibly
      along with the motivation of a convex relaxation). However, the
      statistical analysis seems a little weak, particularly in the near low
      rank and low rank cases. If there were only d non zero eigenvalues, the
      bound the authors present could potentially be very far away from the
      minimax bound. On a related note, the authors should consider rephrasing
      the statement "It is possible (with more technical work) to tighten the
      bound..." in the last paragraph of page 5; it does not appear to
      contribute constructively without a discussion about what this technical
      work could be. Even if the Appendix contains all the proofs, it
      would be helpful if the authors present the reader with a small proof
      sketch after each result has been stated (also note that reviewers are not
      required to read the supplementary material). The section on
      Simulation Results again seems a little weak. The figures need reworking
      as Figure 1(a) is just too hard to see on a printed version of the paper.
      On a related note, it will be nice to explain (in the text and/or in the
      caption) what "overlapping" and "non-overlapping" sparsity patterns mean
      precisely. It will also be helpful to insert an intuitive explanation as
      to why the performance is so different in these cases and when it matters.
      Minor point : The authors should consider rephrasing ".. has a
      wide range of applications - Science, engineering, ... ". It would be much
      more helpful to point the reader to a few specific applications with
      references.


Low-Rank Matrix and Tensor Completion via Adaptive Sampling
We study low rank matrix and tensor completion and propose novel algorithms that employ adaptive sampling schemes to obtain strong performance guarantees. Our algorithms exploit adaptivity to identify entries that are highly informative for learning the column space of the matrix (tensor) and consequently, our results hold even when the row space is highly coherent, in contrast with previous analyses. In the absence of noise, we show that one can exactly recover a n ⇥ n matrix of rank r from merely ⌦(nr 3/2 log(r)) matrix entries. We also show that one can recover an order T tensor using ⌦(nr T 1/2 T 2 log(r)) entries. For noisy recovery, our algorithm consistently estimates a low rank matrix corrupted with noise using ⌦(nr 3/2 polylog(n)) entries. We complement our study with simulations that verify our theory and demonstrate the scalability of our algorithms.

the paper suggests a matrix reconstruction algorithm
      for matrices in case of incoherence in the columns only, not
      necessarily in the rows. a similar result for tensors, where incoherence
      is in all dimensions except the last one. the idea is very
      simple. for matrices, it goes like this: recovering a column-incoherent
      rank-r matrice amounts to recovering (a basis of) the column space,
      because given the column space you can recover each column by sampling
      order of r log r entries and solving equations. to get the column space
      basis, you greedily "grow" a basis by, at each iteration, trying all
      columns until you find one column which is not spanned by the columns
      you already have. checking whether a column is not spanned can be done
      again by sampling O(r log r) entries of that column. each time you
      find column, you add it to your basis by completely uncovering it,
      hence this is an adaptive sampling strategy. it seems like a
      reasonable idea, although i didn't check the proof. the same idea is
      applied to tensors in a recursive fashion. compared to nuclear
      norm minimization techniques, the paper claims that this algorithm gives
      better bounds in certain cases where the incoherence of the rows is
      poor (around sqrt r). the improvement is in the logarithmic and
      constant terms. arguably also the algorithm here is simpler to implement
      (compared to SVT technology). also the proof is simpler. I didn't
      check (in the appendix) whether the claim of improvement from log(n) to
      log(r) is correct, but I couldn't think of a counter example for this
      either. the paper also presents an algorithm and analysis for
      noisy matrix reconstruction, for matrices that are sum of low rank and
      gaussian. the algorithm seems to be based on adaptation of Deshpande et
      al's approach, where columns are sampled with probability proportional
      to their (sampled) projection onto the orthogonal candidate column
      space. As before, in certain parameter regimes, the approach here can be
      better in terms of error/sample size tradeoff.
      detailed comments p 2 "exact recover" -> "exact
      recovery" "less well known" -> "less known" p. 3
      "One readily verifies that \langle x,y \rangle = vec(x)^Tvec(y) "
      you didn't define \langle x,y \rangle for tensors, so there is nothing
      to verify. are you trying to define \langle x,y \rangle here?
      p. 5 theorem 4: are you just "adapting" the proof
      strategy, or are you quoting the theorem from [6]? page 6
      last paragraph before section 5 - extremely hand wavy and
      oversimplistic. theorem 5 is a guarantee for the algorithm
      described in the beginning of section 5. why is the algorithm not
      written in pseudocode like algorithm 1? p. 7 "set up" -->
      setup p. 8 figure 1 - which is SVT and which is your
      algorithm? It would have been nice to see a specialization
      algorithm 1 for matrices (which is the most interesting case).
      The authors should be aware of recent work by Ailon, Chen,
      Huan, in which the second part is on adaptive sampling for pairwise
      clustering matrices (although the approach is completely
      different).


Robust Data-Driven Dynamic Programming
In stochastic optimal control the distribution of the exogenous noise is typically unknown and must be inferred from limited data before dynamic programming (DP)-based solution schemes can be applied. If the conditional expectations in the DP recursions are estimated via kernel regression, however, the historical sample paths enter the solution procedure directly as they determine the evaluation points of the cost-to-go functions. The resulting data-driven DP scheme is asymptotically consistent and admits an efficient computational solution when combined with parametric value function approximations. If training data is sparse, however, the estimated cost-to-go functions display a high variability and an optimistic bias, while the corresponding control policies perform poorly in out-of-sample tests. To mitigate these small sample effects, we propose a robust data-driven DP scheme, which replaces the expectations in the DP recursions with worst-case expectations over a set of distributions close to the best estimate. We show that the arising minmax problems in the DP recursions reduce to tractable conic programs. We also demonstrate that the proposed robust DP algorithm dominates various non-robust schemes in out-of-sample tests across several application domains.

Summary: This paper considers a stochastic optimal
      control problem in discrete time with continuous state and action spaces,
      where at any time t, the state of the underlying system has two components
      (endogenous, exogenous). The endogenous state captures all
      decision-dependent information, while the exogenous state captures the
      external random disturbances. Stochastic optimal control problems have
      many applications in various domains, yet solving these problems via
      dynamic programming (DP) is challenging due to a number of reasons. In
      particular, computing backward recursions is burdensome and the
      distribution of the exogenous states is typically unknown and must be
      inferred from historical observations, i.e., training data. The main
      motivation behind this work is to handle the second issue, i.e., when
      training data is very limited (as compared to the dimension of the
      problem), the estimated cost-to-go functions display high variability and
      bias, the resulting control policies perform poorly in out-of-sample
      tests. To this end, the authors propose a robust data-driven DP approach,
      where given a set of sample trajectories of the exogenous state, kernel
      regression is used together with parametric value function approximations
      to estimate the conditional expectations in DP ([3] shows that the
      resulting approach is asymptotically consistent). However when the number
      of sample trajectories is limited, this is still not of much use, due to
      the noise in the estimates. Therefore, in this paper the authors further
      propose to replace the expectations in the DP recursions with worst-case
      expectations over a set of distributions close to the best estimate. The
      resulting optimization problems are shown to be of conic form and via
      numerical experiments, they show that new algorithm dominates two
      non-robust approaches from the literature in two main applications from
      stock index tracking and wind energy commitment. Quality: The
      proposed data driven approach in this paper is appealing. But I am not
      convinced that the existing algorithms from the literature are unable to
      capture the same phenomena. In particular, the authors claim that the
      robust value iteration methods from the robust Markov decision process
      (MDP) are not fundamentally data-driven, as their primitives are
      uncertainty sets for the transition kernels instead of historical
      observations. Moreover, seemingly the main theoretical contribution of
      this paper is to show that the resulting optimization problems are conic
      optimization problems. While arriving at nice conic problems is valuable,
      I have found no motivation for the Assumption 4.1., which underlies this
      result. In fact, I find the overall theoretical contribution a bit
      limited; see my comments in the Originality & Significance part.
      I am not convinced that the Assumption 4.1 used is reasonable
      in a real situation. It seems like this assumption is there to rather make
      sure that everything fits to the setup of Lemma A.1. Moreover, there is no
      new algorithmic contribution, as the resulting algorithm is pretty
      straightforward, and was solved by utilizing the standard solvers SeDuMi
      and CPLEX. By the way, there is no support offered on how \gamma
      parameter, which controls the level of robustness desired, should be
      chosen. Clearly selecting \gamma is very important, as setting it to zero
      will get us back to the usual data-driven DP approach. Also the brief
      discussion on this at the beginning of page 5 is not very useful in a real
      scenario with an unknown distribution. Finally numerical
      experiments only demonstrate that this approach is superior to the
      non-robust standard data-driven DP and the least-squares policy iteration
      algorithms. None of these methods addresses the robustness issue, a better
      comparison would be against one of the robust approaches such as robust
      value iteration method, or actually building uncertainty sets and solving
      the larger problem described on page 5, lines 234-245, even if it meant
      imposing additional assumptions on the structure of value functions.
      Clarity: The paper is mostly written in a clear fashion, with
      a reasonable logical structure, and a lot of examples. Originality
      & Significance: This paper studies an important problem and to the
      best of my knowledge, the approach is original in robust DP framework.
      Yet, I believe there is little theoretical contribution in this work.
      Proof of Lemma A.1, follows trivially from conic duality, note that
      \triangle(q) =\{p\in\triangle: \exists u s.t. (2(p_i-q_i); u_i-p_i;
      u_i+p_i )\in\mathcal{L}_3 \forall i=1,…, n\} where \mathcal{L}_3 is the 3
      dimensional Lorentz cone, i.e., \mathcal{L}_3=\{x\in\R^3: x_3 \geq
      \sqrt{x_1^2+ x_2^2}, so \triangle(q) is a conic representable set.
      Moreover the cones involved are nonnegative orthant and also based on only
      \ell_1 and \ell_2 norms, so strong conic duality applies here, directly
      leading to the equivalent formulation given in A.1. I.,e., proof of Lemma
      A.1, is trivial, in particular, there is no need to use Lagrangian
      duality. The other proofs for the Theorem and Corollary were already only
      two lines, following from either Lemma A.1, or previous work. As I stated
      before, numerical evidence only proves the value of following a robust
      approach, it does not necessarily prove the value of this particular
      robust approach.


Similarity Component Analysis
Measuring similarity is crucial to many learning tasks. To this end, metric learning has been the dominant paradigm. However, similarity is a richer and broader notion than what metrics entail. For example, similarity can arise from the process of aggregating the decisions of multiple latent components, where each latent component compares data in its own way by focusing on a different subset of features. In this paper, we propose Similarity Component Analysis (SCA), a probabilistic graphical model that discovers those latent components from data. In SCA, a latent component generates a local similarity value, computed with its own metric, independently of other components. The final similarity measure is then obtained by combining the local similarity values with a (noisy-)OR gate. We derive an EM-based algorithm for fitting the model parameters with similarity-annotated data from pairwise comparisons. We validate the SCA model on synthetic datasets where SCA discovers the ground-truth about the latent components. We also apply SCA to a multiway classification task and a link prediction task. For both tasks, SCA attains significantly better prediction accuracies than competing methods. Moreover, we show how SCA can be instrumental in exploratory analysis of data, where we gain insights about the data by examining patterns hidden in its latent components' local similarity values.

This paper proposes a graphical model for measuring
      similarity, based on the assumption that two objects are similar because
      some (instead of all) of their components are similar. Following this
      design, they show empirically their model performs better than recent
      metric learning approaches, when the number of components (K) is larger
      than 1. Overall, I like this paper. The model design captures one
      of the fundamental issues of similarity measures and differentiate it from
      the metric learning framework. Despite the fact that it's a graphical
      model, the approach still cleverly leverages some aspects of the metric
      learning models. Specifically, the component similarity is essentially a
      Mahalanobis distance (before using a sigmoid function to make it
      probability). This design makes sure that the model would generalize the
      metric learning approaches, which reflects in the experimental results as
      well (when K=1). For some applications/tasks, when K increases, the
      performance does improve, which demonstrates that this model fits those
      cases better. On the other hand, since the model can be viewed as
      an extension of the Mahalanobis learning, the scalability of the learning
      process still seems restricted. For instance, PCA needs to be employed to
      reduce the dimensionality of the raw data before applying this algorithm
      (same applied to ITML and LMNN though). Special constraints on M_k need to
      be enforced (SCA-DIAG, where the component similarity becomes a weighted
      Euclidean distance [1]) to reduce the complexity. Such issues are not
      critical, but it would be nice if the authors can further discuss them in
      the paper. Another issue that I wish the authors could discuss further is
      the sparseness of M_k. It is mentioned in the paper that M_k is sparse
      naturally when learning the model. Why this happens is nevertheless not
      very clear since there seems no special regularization on M_k. It would be
      interesting if the authors could provide some insight. Other
      comments: 1. The example of CENTAUR, MAN and HORSE is in fact the
      canonical example to show that similarity does not need to enforce
      triangular inequality, although the authors did not make this argument
      explicit. See Figure 3 in [2]. 2. The component similarity design is
      especially suitable for word similarity, where each word can have multiple
      senses. For instance, JAGUAR and AUTO are similar. JAGUAR and PUMA are
      also similar. However, these two pairs are similar because different
      components (e.g., senses) of the word JAGUAR. One of the recent papers
      that specifically addresses this issue is [3]. It would be interesting to
      know if the proposed model can subsume their approach given. 3.
      Perhaps it's out of the scope of this paper, but the model design bears
      some resemblance to the Siamese neural network approach [4,5,6]. It should
      be very straightforward to design a NN architecture with the component
      similarity subnetworks, and thus interesting to see how it compares to the
      graphical model approach. [1] Schultz and Joachims. Learning a
      distance metric from relative comparisons. NIPS-2004. [2] D Lin. An
      information-theoretic definition of similarity. ICML-1998. [3]
      Reisinger and Mooney. Multi-Prototype Vector-Space Models of Word Meaning.
      NAACL-2010. [4] Bromley, Bentz, Bottou, Guyon, LeCun, Moore, Sackinger
      and Shah. Signature verification using a “Siamese” time delay neural
      network. 1993. International Journal Pattern Recognition and Artificial
      Intelligence, 7(4):669–688. [5] Yih, Toutanova, Platt and Meek.
      Learning Discriminative Projections for Text Similarity Measures.
      CoNLL-2011. [6] Weston, Bengio, and Usunier. Large scale image
      annotation: Learning to rank with joint word-image embeddings. ECML-2010.
      --- Thanks for your response to the
      comments.


Auxiliary-variable Exact Hamiltonian Monte Carlo Samplers for Binary Distributions
We present a new approach to sample from generic binary distributions, based on an exact Hamiltonian Monte Carlo algorithm applied to a piecewise continuous augmentation of the binary distribution of interest. An extension of this idea to distributions over mixtures of binary and possibly-truncated Gaussian or exponential variables allows us to sample from posteriors of linear and probit regression models with spike-and-slab priors and truncated parameters. We illustrate the advantages of these algorithms in several examples in which they outperform the Metropolis or Gibbs samplers.

This paper describes a method for relaxing discrete
      sampling problems into continuous ones that are suitable for Hamiltonian
      Monte Carlo. The idea is to divide the continuous space into orthants,
      creating a one-to-one correspondence between the possible values of the
      discrete space and orthants in the continuous space. The complication that
      arises is that the relaxation has noncontinuous jumps at the boundaries
      between orthants, which the authors address by augmenting Hamiltonian
      dynamics to allow the sample to either reflect off or jump on top of
      cliffs, depending on its energy. A nice effect of this construction is
      within the orthant, the problem is a constrained quadratic, for which
      recent work has pointed out that the dynamics can be computed exactly,
      eliminating the need for numerical integration. This is a creative
      and nice application of recent work on exact Hamiltonian samplers for
      log-quadratic densities. Spike-and-slab models are a great application of
      this technique, and an important model, and the results show that this
      method has promise as an effective sampling algorithm for this model.
      I have a question about the validity of the method. It's clear to
      me that with energy barriers added, the dynamics is reversible and
      preserves the value of the Hamiltonian. I cannot immediately see why the
      dynamics conserves volume. Obviously it does so away from the boundaries,
      but it is less clear to me why the dynamics conserves volume at the energy
      barriers. Could the authors please expand on this? Assuming that
      the method is valid, it's an interesting paper that merits publication.
      Minor comments: * lines 108ff: A possible comment that
      might help the reader's intuition: The equation (9) has a solution iff the
      right hand side is positive. So if Delta is positive, meaning that the new
      discrete state would have higher probability than the current one, then
      the coordinate always crosses the boundary, i.e., we always move to the
      new discrete state. In the physical analogy of HMC, the dynamics reaches
      the top of a cliff, so it always falls downward. On the other hand, if
      Delta is negative, then the new discrete state has lower probability, so
      we only switch to it if the sampler has enough energy, i.e., if q^2_j
      (t_j^-)/2 > -Delta. ========== AFTER REBUTTAL 1.
      Thanks for your explanation about the boundary effects. 2. Perhaps
      a bit self-indulgent, re-reading my review, I thought of a perhaps better
      way to express my intuition at the end. "On the other hand, if
      Delta is negative, then the new discrete state has lower probability.
      Because the sampler has to be reversible, we're only allowed to roll up
      the cliff if there exists a reverse trajectory that could have been
      generated by the dynamics. In the reverse trajectory, the particle gains
      energy when it falls down the cliff, so this creates a lower bound on the
      energy of particles that jump up the cliff." Feel free to not use
      this if you feel it doesn't help.


On the Representational Efficiency of Restricted Boltzmann Machines
This paper examines the question: What kinds of distributions can be efficiently represented by Restricted Boltzmann Machines (RBMs)? We characterize the RBM's unnormalized log-likelihood function as a type of neural network, and through a series of simulation results relate these networks to ones whose representational properties are better understood. We show the surprising result that RBMs can efficiently capture any distribution whose density depends on the number of 1's in their input. We also provide the first known example of a particular type of distribution that provably cannot be efficiently represented by an RBM, assuming a realistic exponential upper bound on the weights. By formally demonstrating that a relatively simple distribution cannot be represented efficiently by an RBM our results provide a new rigorous justification for the use of potentially more expressive generative models, such as deeper ones.

The paper provides a theoretical analysis of the types
      of distributions that can efficiently be represented by restricted
      Boltzmann machines (RBMs). The analysis is based on a representation of
      the unnormalized log probability (free energy) of RBMs as a special form
      of neural network (NN). The paper relates these RBM networks to more
      common types of NNs whose properties have been studied in the literature.
      This approach allows the authors to identify two non-trivial examples of
      functions that can and cannot be represented efficiently by RBM networks -
      and hence related distributions can / cannot be modeled efficiently by
      RBMs. Specifically they show that RBM networks can efficiently represent
      any function that only depends on the number of non-zero visible units,
      such as parity, but that they are unable to represent the only somewhat
      more difficult example of inner product parity. I think the
      paper addresses an interesting and important problem: RBMs have received a
      lot of attention in the recent literature but their properties are still
      not well understood. Many applications of RBMs refer to the result that
      RBMs are universal approximators of binary distributions, but these
      results give no indication what distributions can be *efficiently*
      represented. This question is largely unanswered. Against this
      background, the paper makes two interesting contributions:
      Firstly, although the paper clearly does not "solve" the problem
      of characterizing the efficiently representable distributions I think it
      nevertheless adds a useful piece to the puzzle: It provides two concrete
      examples of meaningful and non-trivial distributions that can and cannot
      be represented efficiently by RBMs. These results are, in my eyes,
      different and complementary to previous works on this topic, which have
      either focused on the universal approximator properties of RBMs, or have
      characterized the set of distributions that can be represented by RBMs in
      a very abstract manner. Secondly, the method of the proof, which
      is based on an analysis of the free energy function as a special type of
      NN, is quite different from the approaches taken in previous works that I
      have seen: By establishing connections between the special type of RBM
      networks and NN more widely used in the literature the paper makes it
      possible to apply established results from the analysis of such networks.
      This (to my knowledge) new perspective is interesting and might enable
      further progress in understanding RBMs and related architectures in the
      future. Despite its theoretical nature I find the paper to be well
      written. It provides a good overview of previous work on the
      representational capabilities of RBMs. The results of the paper are well
      structured and explained, and should be accessible to most of the NIPS
      community. One concern is, however, that all proofs had to be deferred to
      the appendix, so maybe it'd be more appropriate to turn this into a longer
      journal paper -- where the technical aspects can be properly checked (I
      only skimmed some of the supplemental material). Other points:
      > Maybe, a few more words could be spent on explaining why the
      particular functions being analyzed in sections 3 and 4 are interesting
      examples to study. For IP you have a sentence dedicated to this question
      (lines 382-384), but I think it would be useful to expand on this.
      > I believe the universality results, for which you cite Le
      Roux & Bengio (2008) have already been established earlier in the tech
      report by Freund & Haussler (1994): "Unsupervised learning of
      distributions of binary vectors using two layer networks" (UCSC-CRL-94-25)
      > I am wondering to what extent it is going to be possible to
      adapt the analysis technique in the paper to deeper architectures (e.g.
      Deep Boltzmann Machines). Here, the latent variable cannot be marginalized
      analytically so it is not immediately obvious to me how they could be
      translated into easy-to-analyze networks.


Regularized Spectral Clustering under the Degree-Corrected Stochastic Blockmodel
Spectral clustering is a fast and popular algorithm for finding clusters in networks. Recently, Chaudhuri et al. [1] and Amini et al. [2] proposed inspired variations on the algorithm that artificially inflate the node degrees for improved statistical performance. The current paper extends the previous statistical estimation results to the more canonical spectral clustering algorithm in a way that removes any assumption on the minimum degree and provides guidance on the choice of the tuning parameter. Moreover, our results show how the "star shape" in the eigenvectors-a common feature of empirical networks-can be explained by the Degree-Corrected Stochastic Blockmodel and the Extended Planted Partition model, two statistical models that allow for highly heterogeneous degrees. Throughout, the paper characterizes and justifies several of the variations of the spectral clustering algorithm in terms of these models.

The authors present a theoretical analysis of spectral
      clustering in the degree corrected stochastic block model. Following
      earlier work of Chaudhuri et. al. they analyze spectral clustering with
      the regularized normalized Laplacian, first showing that the population
      Laplacian reflects the correct block structure, and then showing that the
      "noisy" graph Laplacian has a similar spectrum. Finally, they bound the
      number of mis-clustered nodes. They also consider an additional step of
      normalizing rows of the spectral embedding and show that the resulting
      bounds are related to the statistical leverage scores. Overall the
      paper is well written (a few typos are pointed out at the end), addresses
      an interesting problem and provides a much simpler (albeit weaker)
      analysis as compared to the earlier work of Chaudhuri et. al. Here
      are some issues that the authors should clarify: 1. I don't quite
      understand why the passage before equation (6) is an appropriate
      definition of mis-clustered nodes. There is already a natural definition
      from the ground truth clustering. It would help if the authors clarified
      the relation between the k-means 0/1 error and the set \mathcal{U}. Why
      are the nodes in \mathcal{U} the only ones on which k-means (which does
      not have access to population centroids, say with random initialization)
      makes errors? If not, it would help avoid confusion to not use the term
      "mis-clustering rate". 2. There should be a more detailed
      comparison to the earlier work of Chaudhuri et. al. In particular, they
      provide results on the parameters of the DC-SBM for which there are no
      errors made by their clustering algorithm and complement this with lower
      bounds. At least for the case of the SBM, rather than assuming (p-r) is a
      constant, a proper comparison to known results from McSherry/Chaudhuri et.
      al. should be made. 3. The role of normalization (of the
      embedding) is still not clear to me. While the figure shows a case where
      it seems to help in practice, it would help to clarify its role in the
      theory. Are the rates worse? If possible, the authors should compare the
      rates with and without the normalization, and if not they should clarify
      the reason for normalization. 4. Finally, in the experiments
      section - you report 50% improvement after dropping 10% of the nodes.
      However, although the algorithm you propose clusters *all* nodes you only
      report accuracy on the retained nodes. Does the accuracy on the full set
      improve at all? Minor typos: 1. Two statistical model
      --> models 2. Line 59 -- we study **an** 3. 102 -- emphasis
      --> emphasize 4. 243 and 244 -- centroid -->
      centroids


Analyzing the Harmonic Structure in Graph-Based Learning
We find that various well-known graph-based models exhibit a common important harmonic structure in its target function -the value of a vertex is approximately the weighted average of the values of its adjacent neighbors. Understanding of such structure and analysis of the loss defined over such structure help reveal important properties of the target function over a graph. In this paper, we show that the variation of the target function across a cut can be upper and lower bounded by the ratio of its harmonic loss and the cut cost. We use this to develop an analytical tool and analyze five popular graph-based models: absorbing random walks, partially absorbing random walks, hitting times, pseudo-inverse of the graph Laplacian, and eigenvectors of the Laplacian matrices. Our analysis sheds new insights into several open questions related to these models, and provides theoretical justifications and guidelines for their practical use. Simulations on synthetic and real datasets confirm the potential of the proposed theory and tool.

- A desirable property (*) for algorithms on graphs is
      to get a solution which is almost constant in each cluster, but
      drastically change between clusters. This paper tries to establish a
      relation between ``near harmonic'' function on graphs and their drop in
      dense/sparse areas. The authors then study different algorithms and their
      ``harmonic loss'', as a step to show that property (*) holds for their
      results. - This paper contains several mini steps toward getting tools
      to check if the property (*) holds for a machine algorithm. However the
      results are not still strong enough, and can not be considered as a proof.
      - The paper is easy to read and follow, but there are some space for
      improvements. Some of the results can significantly simplified (See
      detailed comments). Supplementary material is supposed to contain proofs
      and specific details, and not complete new sections (Section 4.2 and 4.3)
      - Originality and significance: The paper introduce new tools to
      intuitively justify the property (*) for graph algorithms. The machinery
      looks attractive for me, but i do not see them as a tool for finding
      rigorous proofs. Main comments: - The so called left/right/-
      continuity of functions on graphs as it is defined, have nothing to do
      with mathematical concept of continuity. It does not guaranty any bound on
      function's jump. If you remove all results about continuity, would the
      message of the paper is change? My suggestion is to choose another
      keyword, which does not imply properties stronger than the actual meaning
      to the reader. - lines 192-199: All the statements are imprecise and
      hand wavy. One can build functions f which is almost harmonic everywhere,
      but does not drop at all on a cut with a very small w(). Consider this
      graph and function values defined on vertices, where 000 shows a big
      complete graph with f zero on it (or super tiny random values if you
      like): 1--0.8--0.6--0.4--0.2--0--000--0--0--000 f is almost
      harmonic everywhere(except two vertices), however it does not drop at all
      on the edges between two 000 clusters, where conductance is relatively
      small. For other statements in this paragraph, also one can build
      counterexamples. - In Corollary 3.1, we can explicitly characterize
      L_f(S_i), which is the inverse of the resistance distance between vertex 1
      and n - line 247: it is claimed that f continuously changes on graphs.
      However the definition 2.5 of continuous f is much weaker than
      "continuously changing on graphs" - lines 264-310: We have f_1(i) +
      f_2(i) = 1. So we do not need to consider f_1 and f_2 as one of them
      already contain all the information. Also Lemma 3.3 boils down to deciding
      the class by comparing f'_1(i) with 1/n. - I suggest the Authors to
      give a recipe for analyzing semi-supervised algorithms. This would
      increase the readability of their paper. A recipe is something like:
      1- try to write the solution function f as a sum of a harmonic term
      and non-harmonic term. In my own words, find the net incoming flow into
      each node. It is clear that we may not be able to write the solution for
      all methods in this form. 2- Use theorems 2.7 and 2.8 to check if they
      provide good bounds on the drop - There are many qualitative
      statements, which Authors should avoid having them. This makes their work
      look very hand-wavy and imprecise. tiny comments: - In
      definition 2.5, it is good to mention that S_i depends on f. - line
      170: r.h of inequality, k \in \bar{S_i} Pro: - Bound on the
      change of function f depending on Harmonic Loss and conductance -
      Presenting the solution of different graph algorithms in the harmonic form
      Co: - Upper/lower bounds on the change of function f only
      indicate the property (*) and does not give a rigorous proof - The
      normalizing step can presented much simpler. In fact i do not count it as
      a contribution of the paper. - The supplementary material is misused
      by Authors.


Dynamic Clustering via Asymptotics of the Dependent Dirichlet Process Mixture
This paper presents a novel algorithm, based upon the dependent Dirichlet process mixture model (DDPMM), for clustering batch-sequential data containing an unknown number of evolving clusters. The algorithm is derived via a lowvariance asymptotic analysis of the Gibbs sampling algorithm for the DDPMM, and provides a hard clustering with convergence guarantees similar to those of the k-means algorithm. Empirical results from a synthetic test with moving Gaussian clusters and a test with real ADS-B aircraft trajectory data demonstrate that the algorithm requires orders of magnitude less computational time than contemporary probabilistic and hard clustering algorithms, while providing higher accuracy on the examined datasets.

Summary: The paper presents a deterministic inference
      algorithm for a clustering problem for batch sequential data (data in
      each batch exist in clusters, and clusters across batches are related
      as per some evolving structure). The Dependent Dirichlet Process
      Gaussian mixture model (DDP-GMM) is a nonparametric Bayesian method
      for dealing with such data. The paper uses the idea of doing
      low-variance asymptotics on the Gibbs sampling update equations in a
      DDP-GMM model (a similar approach was recently proposed by Kulis and
      Jordan for DP-GMM in their DP-means algorithm) and derives a
      deterministic algorithm that is (1) more efficient to implement than Gibbs
      sampling for DDP-GMM, (2) can learn the number of clusters just like
      DDP-GMM. Experimental results on a synthetic and a real dataset show
      that the proposed algorithm runs fasters than other inference methods
      for DDP-GMM such as Gibbs sampling, variational inference, and
      sequential Monte Carlo. Quality: The technical quality is rigorous
      and the details appear to be correct. Clarity: The paper is
      well-written and the exposition is easy to follow. Originality:
      The paper is based on recently proposed idea of using low-variance
      asymptotics on the Gibbs sampling update equations in case of DP
      mixture models. Significance: The paper addresses an important
      problem (scaling up nonparametric Bayesian methods for large
      datasets). Paper Strengths: - The proposed algorithm is
      intuitive, seems easy to implement, and is scalable for large
      datasets. - Although the basic recipe is borrowed from the
      DP-means work of Kulis and Jordan (taking the Gibbs sampling updates
      and applying the low-variance asymptotics), the ideas in section 3.2
      (parameter updates) are rather novel in this context. Paper
      Weaknesses: - The algorithm has 3 free parameters. For an
      unsupervised algorithm (with no option of doing cross-validation),
      this is a bit worrying. Although the paper gives some suggestions on
      how to choose the parameters, it is not clear how well they will work
      in practice. - There should have been a comparison with a
      simple variant of the DP-means algorithm (Kulis and Jordan, 2012) for
      batch-sequential data (see the suggestion in the comments below on how
      to do it). - On real-data, there is no comparison with variational
      inference or particle learning. Comments: - It would
      have been nice to compare the proposed dynamic-means algorithm with a
      very simple adaptation of the DP-means algorithm: Run DP-means on one
      batch and initialize means for the next batch using the results of
      this run. I would be curious to see how dynamic-means compares against
      this alternative. - It turns out that instead of doing
      small-variance asymptotics on the Gibbs sampling updates, one could
      also do MAP-based asymptotics to derive hard-assignment based variants
      of nonparametric Bayesian models such as DP-GMM or IBP-based latent
      feature models (there is a recent paper by Broderick et al from ICML
      2013: "MAD-Bayes: MAP-based Asymptotic Derivations from Bayes"). Is it
      possible to derive the dynamic-means algorithm using this approach?
      Please comment. - There is work on temporal DP mixture model based
      on recurrent Chinese restaurant process which is similar in spirit to
      dependent DP mixture models. Please see this: "Dynamic Non-Parametric
      Mixture Models and The Recurrent Chinese Restaurant Process : with
      Applications to Evolutionary Clustering" by Ahmad and Xing (2008). It
      would be nice to have a discussion about this link of work. Please also
      comment whether doing low-variance asymptotics on these methods would
      lead to similar algorithms as dynamic-means? - In the DP-means
      algorithm of Kulis and Jordan, the order of data points mattered. In
      the cluster assignment step of the dynamic-means algorithm, does the order
      of data points (within a single batch) matter? Please
      comment.


Convex Relaxations for Permutation Problems
Seriation seeks to reconstruct a linear order between variables using unsorted similarity information. It has direct applications in archeology and shotgun gene sequencing for example. We prove the equivalence between the seriation and the combinatorial 2-SUM problem (a quadratic minimization problem over permutations) over a class of similarity matrices. The seriation problem can be solved exactly by a spectral algorithm in the noiseless case and we produce a convex relaxation for the 2-SUM problem to improve the robustness of solutions in a noisy setting. This relaxation also allows us to impose additional structural constraints on the solution, to solve semi-supervised seriation problems. We present numerical experiments on archeological data, Markov chains and gene sequences.

This paper considers the problem of minimizing a
      quadratic form specified by a similarity matrix A over the class of
      permutation matrices. The authors first show that when the similarity
      matrix has a special structure (i.e. A is of the form A = C \circ C^T
      where C is a pre-Q matrix), then the quadratic form is minimized by the
      permutation that turns A into an R matrix. It is known that this
      permutation can be found by computing the Fiedler vector of the Laplacian
      matrix of A, so we can solve the optimization problem in polynomial time.
      The second main result is a regularized convex relaxation of the
      optimization problem in the noisy setting that also allows additional
      structural constraints. The authors proceed to demonstrate empirically
      that the convex relaxation is more robust to noise and can achieve a
      better solution (i.e. a lower objective value) than the manual or spectral
      solutions, given enough additional structural constraints. This is
      a nicely written paper with interesting results, both theoretical and
      practical. The introduction and exposition are mostly clear, although the
      organization can be a bit improved. The two main results (the spectral
      solution in the combinatorial case and the convex relaxation in the noisy
      case) seem to be a bit disjointed. In Section 2, the connections between
      the different matrices, as well as the various preliminary results, can be
      confusing at times. Perhaps it would help if the main results (Proposition
      2.6 and 2.11) are stated in the beginning of the section, or more
      emphasized than the other preliminary results. Additional
      comments/questions: * In Equation (2), why do we generalize \pi to
      y_\pi? It seems the extent of generality considered in this paper is when
      y is an affine transform of {1,…,n}, in which case (2) is clearly
      equivalent to problem (1). Can we say anything about more general y?
      Otherwise, it might help to stick with (1) to minimize notation and
      confusion. * Although this has been stated in the introduction, it may
      help to state the problem of seriation mathematically in Section 2 (in
      terms of permuting A into an R matrix), so that the connection with the
      combinatorial problem becomes more explicit. * L169-170 in the proof
      of Proposition 2.6 ("all monotonic subsets of y of a given length have the
      same (minimal) variance, attained by \Pi y") is not too clear to me.
      Perhaps it would help to write the problem (2) in terms of the Laplacian
      matrix (as in (3)) so it becomes more explicit what happens to the problem
      when we permute A. * In Definition 2.8, what is the weight vector used
      in the subsequent discussion? Is it w = 1? * L192: AB^T should be AB?
      * In Proposition 2.11 (and 2.6), is the assumption that A = C \circ
      C^T with C pre-Q necessary? What about the case when A is a pre-R matrix,
      but not necessarily of the form A = C \circ C^T? And in practice, how can
      we check whether the conditions in Proposition 2.11 are satisfied? *
      Is there anything we can say about the quality of the spectral solution
      when the assumptions of Proposition 2.11 do not hold exactly? * I
      think Proposition 3.1 should be placed in Section 2. * In the convex
      relaxation, how do we choose the values of \mu, p, and the perturbed Y in
      practice? Is the performance sensitive to the choice of these parameters?
      * In the caption following Table 1, I think 24% should be 47.5%. I was
      also a bit confused about the claim that "the semi-supervised solution
      actually improves on both Kendall's manual solution and on the spectral
      ordering". Is this based on the combinatorial objective value? Is it clear
      that this is the correct metric to use to assess performance? *
      Section 2 and 3 should make it more clear that some of the results are
      proved in the appendix, and state where in the appendix the results are
      proved. * The additional experimental results in the appendix can also
      be organized or labelled better to make clear which application the
      results are for. * Some of the plots can also benefit from a bit more
      explanation, such as what the ideal or best case is. For instance, what is
      the ideal behavior of Figure 3 in the appendix? Should the line be roughly
      diagonal? How do we compare the three matrices in Figure 5? And why are
      the two rightmost plots in Figure 2 have different formats? * The
      second row, third column of Table 2 seems to be missing a value.


Model Selection for High-Dimensional Regression under the Generalized Irrepresentability Condition
In the high-dimensional regression model a response variable is linearly related to p covariates, but the sample size n is smaller than p. We assume that only a small subset of covariates is 'active' (i.e., the corresponding coefficients are non-zero), and consider the model-selection problem of identifying the active covariates. A popular approach is to estimate the regression coefficients through the Lasso ( 1 -regularized least squares). This is known to correctly identify the active set only if the irrelevant covariates are roughly orthogonal to the relevant ones, as quantified through the so called 'irrepresentability' condition. In this paper we study the 'Gauss-Lasso' selector, a simple two-stage method that first solves the Lasso, and then performs ordinary least squares restricted to the Lasso active set. We formulate 'generalized irrepresentability condition' (GIC), an assumption that is substantially weaker than irrepresentability. We prove that, under GIC, the Gauss-Lasso correctly recovers the active set.

This paper provides a thorough and comprehensive study
      of the post-Lasso (the estimator obtained by fitting a least
      square model on the variables selected by the Lasso) in the context of
      high dimensional sparse regression. A new theoretical guarantee is
      provided for the post- Lasso, as well as simulated experiments on a
      toy example where the benefit of the procedure is clear.
      Though, there are still a few points that could be improved.
      First, some important references are missing about other works
      considering the post-Lasso on a theoretical level (or a variation
      of it): "Pivotal Estimation of Nonparametric Functions via
      Square-root Lasso" Alexandre Belloni, Victor Chernozhukov, Lie Wang
      (cf. Theorem 7 for instance) "L1-Penalized Quantile
      Regression in High-Dimensional Sparse Models", Alexandre Belloni and
      Victor Chernozhukov, 2011 (cf. post-l1-QR Theorem 5)
      Moreover, a recent work focusing on the practical
      properties of the post-Lasso for a particular set of simulated data,
      challenges, in certain situations, the theoretical benefit illustrated
      by the authors. Can they comment on that? "Trust, but verify:
      benefits and pitfalls of least-squares refitting in high dimensions",2013
      Johannes Lederer Points to correct: -l134:
      without further assumptions the minimizers of G might not be unique.
      the results is true under some more assumptions on the Gramm matrix,
      as is well known (and in a way proved later by the authors) since
      "On Sparse Representations in Arbitrary Redundant Bases", J-J.
      Fuchs,2004 and more recently "The Lasso Problem and Uniqueness",
      Ryan J. Tibshirani, 2013 The problem occurs many time in the
      proof: the unicity is sometimes used before it is proved. Adapting the
      results from the aforementioned papers, I encourage the authors to
      show that (under suitable assumptions) unicity holds (cf. for instance
      l1034, where the strict inequality is given without any justification,
      see also l1071 and l1113) and therefore that there proof is right.
      I encourage the authors to fix this for the sake of clarity: it
      could also be better to add an assumption mentioning when one needs
      the (correctly) extracted Gramm matrix to be invertible.
      -l303: t_0 is defined but nowhere used in this section, and
      then re-used in the next one... please remove. -l307: A comment
      could be added on the fact that the lambda parameter depends on an unknown
      quantity, eta. -l317: What is the benefit of the assumptions w.r.t
      [23]? It does not seem straightforward which one is weaker: on the one
      hand we need a matrix invertible of larger size (T_0 contains S) but on
      the other hand only the sup-norm of a vector should be control is the
      proposed work. -Section 3.1 is for me useless, the results are
      exactly the same as in Section 2.1. Please remove and spare some room
      for the comments, and the missing references. For instance, further
      comparisons on the differences between the deterministic case and the
      random case could be investigated. -l408: Did the authors try to
      improve the term depending on exp(-t_0/2)? It seems to be the weak
      part of the probability control provided in Theorem 3.4 -l441/443:
      Candes and Cand\'es are mispelled. It should be Cand\`es -l553: it
      seems there is a sign issue. General questions: - can
      the authors comment on the fact that the sparsity level must be known a
      prior (no adaptivity) in their procedure? -When is T_0=S (l267)? That
      would be interesting to understand when the two are identical.