More Effective Distributed ML via a Stale Synchronous Parallel Parameter Server
We propose a parameter server system for distributed ML, which follows a Stale Synchronous Parallel (SSP) model of computation that maximizes the time computational workers spend doing useful work on ML algorithms, while still providing correctness guarantees. The parameter server provides an easy-to-use shared interface for read/write access to an ML model's values (parameters and variables), and the SSP model allows distributed workers to read older, stale versions of these values from a local cache, instead of waiting to get them from a central storage. This significantly increases the proportion of time workers spend computing, as opposed to waiting. Furthermore, the SSP model ensures ML algorithm correctness by limiting the maximum age of the stale values. We provide a proof of correctness under SSP, as well as empirical results demonstrating that the SSP model achieves faster algorithm convergence on several different ML problems, compared to fully-synchronous and asynchronous schemes.

The paper presents an approach to building a parameter
      server for distribute ML systems that presents a view to each client where
      parameters have a bounded degree of staleness. Using a combination of
      caches, the client interface guarantees that all updates to the parameter
      array occurring after a fixed deadline (the current clock/iteration/tick
      minus a fixed delay) are visible along with more recent updates if
      possible. Thus the interface presents a view of parameters that integrates
      most updates along with best-effort service for more recent updates. It is
      shown that this simple semantic preserves the theoretical guarantees of
      cyclic delay methods while being significantly faster in practice.
      Empirical analysis on several problems with multiple cluster
      configurations show that the advantage is due to a combination of
      increased efficiency (over BSP) and optimization progress per update (over
      Asynchronous). This paper presents a simple set of semantics that
      improve on "parameter server" schemes currently showing up in large scale
      machine learning applications. Along with straight-forwardly carrying over
      simple theoretical guarantees, the method is apparently significantly
      faster for reasonably-sized tests than the obvious competitors and on
      those points alone I think this is pretty positive. There are a
      lot of well-known problems in loaded clusters (stragglers, etc.) and as
      far as I can tell this approach should deal with them well. A saving grace
      may be that the caching mechanism reduces overhead dramatically for the
      slowest nodes thus giving a degree of "load balancing" that is annoying
      hard to get in other ways. Is it possible to show that this is actually
      happens to a degree sufficient to enable the "catch up" phenomenon claimed
      in the paper? Some discussion of the read semantics is given with
      details on the cache policy (which just falls through when the local cache
      is stale). Due to the read-my-writes policy, is it correct that all local
      writes are written through to all caches/server? A sentence of
      clarification on the writing semantics might help for reproduction of the
      work. As far as systems papers go the authors make a good case for
      the approach. Pros: Fairly simple semantics. Preserves
      basic theoretical guarantees for typical methods (e.g., SGD).
      Experimental analysis shows actual speedups on multiple distributed
      problems and the speedup comes from the sources expected. Cons:
      A comparison to cyclic delay would be nice. Potentially complex to
      re-implement in comparison to asynchronous/BSP; is it worth it for the
      moderate speedup? **Update Thanks to the authors for their
      clarifications and additions. I think these are reasonable arguments and
      resolve my [very limited] qualms.


Contrastive Learning Using Spectral Methods
In many natural settings, the analysis goal is not to characterize a single data set in isolation, but rather to understand the difference between one set of observations and another. For example, given a background corpus of news articles together with writings of a particular author, one may want a topic model that explains word patterns and themes specific to the author. Another example comes from genomics, in which biological signals may be collected from different regions of a genome, and one wants a model that captures the differential statistics observed in these regions. This paper formalizes this notion of contrastive learning for mixture models, and develops spectral algorithms for inferring mixture components specific to a foreground data set when contrasted with a background data set. The method builds on recent moment-based estimators and tensor decompositions for latent variable models, and has the intuitive feature of using background data statistics to appropriately modify moments estimated from foreground data. A key advantage of the method is that the background data need only be coarsely modeled, which is important when the background is too complex, noisy, or not of interest. The method is demonstrated on applications in contrastive topic modeling and genomic sequence analysis.

This paper tackles the problem of learning a
      mixture model to capture properties of "foreground data" which are not
      present in background data. The proposed technique extends recent work
      on method of moments and tensor factorization. The novelty in the
      paper is (i) the observation that one can subtract moments before
      factorization and (ii) developing a variant of the robust power method
      from Anandkumar et al. 2013 that doesn't require positive
      semidefiniteness. The idea presented in the paper is simple,
      natural, and novel. The main weakness is that the paper doesn't really
      compare the proposed method with several natural baselines either
      theoretically or empirically. 073: the paper says that the
      stated goal is not to discriminate between foreground and background,
      but isn't this exactly the intuition of being contrastive?
      383: In the case where the foreground words are disjoint from the
      rest (B and C), the intuition that you only need enough background
      data to get the signs of the topics in B and C correct seems right.
      However, the paragraph then talks about the accuracy of the foreground
      topics, which doesn't relevant. I would have expected to see some
      statement characterizing the number of samples you need to get the
      sign correct. Also, I normally think of the foreground as being
      captured by a modest number of topics and that the background is
      enormous and requires a lot more, so $K$ would have to be pretty
      large, close to $D$. So footnote 1, which talks about one topic, seems
      pretty unmotivated. The fact that power iteration requires
      multiple passes over the training data ($K$ times the number of power
      iteration updates times the number of restarts of the power tensor
      method) is somewhat discouraging, since one of the benefits of these
      method of moments approaches is that you can take one pass over the
      data. Could you randomly project the data onto $O(K)$ dimensions,
      building up a much smaller dense tensor that you can store in memory?
      Baseline 1: estimate the foreground model and background models
      separately. Under the assumption that the model is well-specified,
      then the normal spectral method of moments will provide a consistent
      estimator of both models, from which it is easy to detect which are
      the common topics. What can you say about this method theoretically
      (it is consistent after all) and empirically? Baseline 2:
      simply use EM to do parameter estimation in the full generative model.
      Is local optima is actually a problem in this setting? My experience
      is that EM works pretty well anyway, and any sort of empirical
      statement about this approach must compare with EM. Even if EM
      outperforms spectral, you can always use the latter as initialization
      to the former. Theoretically, I think it's interesting to think
      about the behavior of the maximum likelihood estimator in this case,
      which in some sense tries to spread its modeling efforts evenly,
      whereas the contrastive estimator seems more targeted towards what you
      want. So there might be a regime that the contrastive estimator is
      better, even theoretically? One piece of work which is not
      directly relevant but has a similar high-level flavor is contrastive
      estimation:
      http://www.cs.cmu.edu/~nasmith/papers/smith+eisner.acl05.pdf
      Actually, now that I think about it, the intuition is somewhat
      different - instead of using all the background data, we are using a
      subset which is "close" to the foreground data.


Dimension-Free Exponentiated Gradient
I present a new online learning algorithm that extends the exponentiated gradient framework to infinite dimensional spaces. My analysis shows that the algorithm is implicitly able to estimate the L 2 norm of the unknown competitor, U , achieving a regret bound of the order of O(U log(U T + 1))
, achievable without knowing U . For this analysis, I introduce novel tools for algorithms with time-varying regularizers, through the use of local smoothness. Through a lower bound, I also show that the algorithm is optimal up to log(U T ) term for linear and Lipschitz losses.

SUMMARY AND RELATED WORKS This paper is about
      online linear regression with Lipschitz losses over L2-balls. In this
      setting, the authors introduce and study a new online algorithm that
      achieves a regret bound of the order of U*sqrt{T}*log(UT) after T rounds
      over any L2-ball of radius U, without knowing U in advance. This algorithm
      is close in spirit to that of Streeter and McMahan [21] (NIPS 2012), with
      however an interesting important difference: the regret bound of Theorem 1
      is dimension-free in the sense that it does not depend on the dimension d
      of the input vectors x_t (note however that the algorithm of Streeter and
      McMahan was primarily designed for the L1-norm rather than for the
      L2-norm). This dimension-free bound suggests that the algorithm could be
      used in infinite dimension, which is useful for kernel regression. Another
      related algorithm is the standard gradient descent (see, e.g., Kivinen And
      Warmuth 1997 and Cesa-Bianchi 1999). Gradient descent can also achieve a
      dimension-free regret bound of the order of U*sqrt{T}, but at the price of
      knowing U beforehand. In the more realistic case when U is unknown, the
      regret bound of Gradient Descent has a worse dependency in U (U^2*sqrt{T}
      instead of U*sqrt{T}). On the contrary, the algorithm introduced in the
      present paper is adaptive to the unknown radius U (up to log factors). In
      a second part of the paper, the authors prove that their regret bound is
      optimal up to log factors, but this is in a rather weak sense (see below).
      They finally carry out some simulations to compare their algorithm to
      Gradient Descent with different values of the regularization parameter.
      OVERALL ASSESSMENT My overall opinion about the paper
      is positive for the following reasons: PROS 1) To the best
      of my knwoledge, I think that Algorithm 1 is the first one that achieves
      (up to log factors) a regret bound of order U sqrt{T} without knowing U
      beforehand, via a sequential time-varying tuning of its parameters. Since
      the rate U sqrt{T} is optimal for any fixed U and for linear losses (at
      least for algorithms that don't have access to x_t, cf. [1]), and since
      the ideas underlying Algorithm 1 are far from trivial, Theorem 1 should be
      of interest to quite a few people in the machine learning community.
      2) Comparisons to the literature are precise and honnest, which is a
      nice quality. CONS Mathematical concerns and significance:
      3) Proof of Theorem 1: the proof relies on Lemma 1, where it is
      assumed that the functions f^*_t are twice differentiable. Since this is
      not the case for the functions used in Theorem 1 (because of the null
      vector), mathematical details are missing. In particular, what happens if
      the null vector lies in the segment [\theta_t,\theta_{t+1}] at some round
      t? Please provide careful details about how you carry out the Taylor
      expansion of f^*_t around \theta_t. 4) Theorem 2: a) The lower
      bound is not as strong as one could expect, since it only holds for a
      fixed vector u (whose norm is roughly of order sqrt{T}). What happens for
      other vectors u? It would be better to have some information, e.g., for
      the case when \norm[u] -> 0 or when \norm[u] -> +\infty. b) In
      the proof of Theorem 2, I think that to get E[\sum_t w_t^T z_t] = 0 on
      line 594, the authors implicitely used the fact that w_t is independent
      from z_t. This is not the case for all algorithms in the online protocol
      described at the beginning of the paper (the learner can not only use the
      past examples, but he has also access to x_t before making his decision).
      Therefore, the lower bound of Theorem 2 holds on a smaller class of
      algorithms: those that are only allowed to access to x_t through the norm
      \norm[x_t] (since w_t and z_t are then indeed independent). Please write
      this explicitely. Writing style: 5) As such, the proofs are
      very technical and suffer from a lack of intuitions. Furthermore, the
      proof style could really be improved: some explanations are missing (e.g.,
      simple arguments to carry out the calculations) and some details are not
      at the right place (e.g., the particular shape of f_t^* should be made
      precise right after Lemma 1, otherwise, we have to wait until Lemma 2 and
      line 303 to understand the relationship between Algorithms 1 and 2).
      OTHER COMMENTS 6) Absract: is Algorithm 1 really an extension
      of the Exponentiated Gradient algorithm? I think that it is simply similar
      in shape. The verb 'extends' is thus midleading. 7) Introduction: the
      second paragraph is probably difficult to read for the unfamiliar reader.
      Please state the online decision protocol and the definition of online
      mirror descent (even in an informal way). 8) Line 44: a division by
      sqrt{T} is missing in the Euclidean regularizer (or, better: the regret
      bound is O(u^2/eta + eta*T), which is perhaps more standard). 9) Line
      82: add the reference [8] to [3,17]. 10) The term 'sample' for x_t is
      not appropriate: you could use 'input vector/data/covariates' instead.
      11) Beginning of Section 2: the space Y is just the real line (so
      write it explicitely), and X is a Euclidean vector space (maybe Hilbert
      space?). Furthermore, the notation w_t^T x_t seems only appropriate when X
      is finite-dimensional. 12) Section 3: since all references about
      convex analysis are on finite-dimensional vector spaces, the extension to
      infinite-dimensional spaces should be clarified (but I agree that it is
      very natural). 13) Line 175: the optimality of the rate U*sqrt{T} was
      only proved for algorithms that don't use the knowledge of x_t (cf. [1]).
      14) Line 179: what about the upper bound on a? 15) Lemma 1: please
      write that the norm of the Hessian is its spectral norm. 16) Line 221:
      \norm[z_t]^2_{f_t} -> \norm[z_t]^2 17) Lines 239-241: please
      explain briefly or point to the proof. 18) Lemma 2: it only holds for
      \theta \neq 0 (see also Comment 3 above). 19) Line 269: w_r ->
      \theta_r 20) Proof of Theorem 1: to help the reader, you could
      explicitely structure the proof as follows: Case 1 = "\norm[\theta_t] >
      ...", Case 2 = "\norm[\theta_t] \leq ...", and explain right from the
      beginning which case corresponds to which term in the minimum of the bound
      of Lemma 1. 21) Line 324: Lemma 3 is used with p=2 a^2 AND M=1/a.
      22) Line 343: this is an equality. 23) Line 361: The sentence 'We
      now use ...' should be postponed right after Equation (3). 24) Line
      366: several typos: there are two L factors instead of one in the second
      term of the right-hand side; \beta -> \beta_t; a factor of L is missing
      after the equal sign. 25) The lower bound of Theorem 2 is for the
      linear loss functions \ell_t(x) = - x. Please write this explicitely.
      26) Several typos: 'vectorial space" -> 'vector space' (several
      times in the paper), 'supplemental material' -> 'supplementary
      material' (idem), 'derivative' -> 'derivatives' (line 270), 'the f(x)'
      -> 'f(x)' (line 511). ----------------------------------------
      POST-REBUTTAL ADDENDUM I'd like to thank the authors for their
      feedback. Since they addressed my two main questions very precisely and
      ruled out concerns 3) and 4), I clearly recommend acceptance. When
      polished (through the reviewers' suggestions), the paper will be a very
      nice one. NB: I noted two other typos: a ||w|| is missing after
      \alpha_t on line 303, and u_t should be replaced with z_t on line
      496.


Lexical and Hierarchical Topic Regression
Inspired by a two-level theory from political science that unifies agenda setting and ideological framing, we propose supervised hierarchical latent Dirichlet allocation (SHLDA), which jointly captures documents' multi-level topic structure and their polar response variables. Our model extends the nested Chinese restaurant processes to discover tree-structured topic hierarchies and uses both per-topic hierarchical and per-word lexical regression parameters to model response variables. SHLDA improves prediction on political affiliation and sentiment tasks in addition to providing insight into how topics under discussion are framed.

This paper proposes a hierarchical, supervised topic
      model that builds on ideas from hierarchical LDA using the nested CRP
      and from supervised LDA, which associates topics with regression
      coefficients. The proposed model further extends hierarchical LDA by
      allowing each sentence in a document to chose its own path through the
      topic tree, and sharing of paths for sentences within the same
      document is modelled through a document-level CRP where the customers
      are sentences and the dishes are paths through the topic hierarchy
      ("combos"). Another difference to supervised LDA is that each word is
      also associated with its own regression coefficient to model
      topic-independent effects. The authors empirically demonstrate the
      performance of the model on 3 data sets and show that the proposed
      model outperforms LDA+linear regression as well as supervised LDA.
      The idea of modelling framing in the proposed way by associating
      regression coefficients with hierarchically arranged topics is
      interesting and appears novel. The paper is clearly written and the
      proposed model and inference procedure seem technically sound. The
      quantitative evaluation shows that the proposed model (slightly)
      outperforms simple baselines and the qualitative evaluation suggests that
      the discovered topic hierarchy has the desired property of separating
      topics into different aspects or ways of framing them. The
      main question I had while reading this paper was: Is such a complicated
      model really necessary for modelling this what/how distinction? Given
      that the depth of the hierarchy is fixed to 3 in the experiments
      (why?), it seems that at least an infinitely deep hierarchy is not
      necessary. If interpretability of the resulting topics is desired,
      would it make sense to model the binary split (positive/negative,
      republican/democrat) explicitly, e.g. by explicitly splitting each
      topic into pos/neutral/neg parts with fixed regression coefficients
      and then associating with each document a distribution over those
      parts? In terms of quantitative experiments, it would be nice to
      tease apart the contributions of the individual changes to existing
      models: how does sLDA perform if lexical regression parameters are
      added, or a supervised response variable is added to a plain HLDA
      model? Also, it would be helpful to see some descriptive statistics
      about the inference and the learned models, e.g. how many nodes are in
      the hierarchy, branching factor, iterations until convergence,
      etc.


Two-Target Algorithms for Infinite-Armed Bandits with Bernoulli Rewards
We consider an infinite-armed bandit problem with Bernoulli rewards. The mean rewards are independent, uniformly distributed over [0,1]. Rewards 0 and 1 are referred to as a success and a failure, respectively. We propose a novel algorithm where the decision to exploit any arm is based on two successive targets, namely, the total number of successes until the first failure and until the first m failures, respectively, where m is a fixed parameter. This two-target algorithm achieves a long-term average regret in √ 2n for a large parameter m and a known time horizon n. This regret is optimal and strictly less than the regret achieved by the best known algorithms, which is in 2 √ n. The results are extended to any mean-reward distribution whose support contains 1 and to unknown time horizons. Numerical experiments show the performance of the algorithm for finite time horizons.

The main contribution of the paper is to close a
      constant gap between the lower and upper bounds of the asymptotic regret
      in an infinite-armed bandit setting where the arms are drawn from a
      uniform distribution and the length of the horizon is known. The result is
      extended for unknown horizons and more generic distributions, at the
      expense of loosing the tightness between the lower and the upper bounds.
      Quality Algorithm 1 is an interesting addition to the
      literature. Its upper bound matches the lower bound proven in [2]. The
      algorithmic idea is novel and non-obvious and the regret proof is sound.
      It is important to note that the algorithmic strategy is only of
      theoretical interest at this point, as its regret guarantee holds when
      both n (the horizon length) and m (the number of "runs" of an arm) go to
      infinity. My first question is whether you tried to get an upper
      bound for m in terms of n so that we can get a more informative result
      than Proposition 1? Such a result would help get rid of m as a parameter
      if one would actually want to use the algorithm in an applied setting.
      One weakness of the paper is the lack of at least basic numerical
      results about the finite time behavior of Algorithm 1. Even though the
      result is asymptotic, it is interesting to understand how it behaves for a
      well defined horizon length (as discussed on page 2105 in [2] for
      example). While in section 5, the authors compare the empirical regret of
      the algorithm with the asymptotic lower bound, it would be interesting
      (since it might have an impact in an applied setting) to verify how the
      algorithm compares with the strategies introduced in [2] or [6] for
      example. My second question is about the statement that the lower
      bound in [2] is incorrect (line 064). First, the authors mention Theorem 2
      in [2], but Theorem 2 in the referred paper is about the upper bound for
      the k-failure strategy. So I am assuming they talk about the Theorem 3.
      The note at lines 102-105 mention an assumption from the lower bound
      theorem in [2] that I was unable to locate in that paper. As a
      consequence, I am not convinced that the original lower bound proof is
      incorrect. Could you please clarify? Also, in the case the paper is
      accepted and the lower bound is indeed incorrect, please add a more
      detailed description of the error in the appendix. Clarity The
      paper is well written and relatively easy to follow. Regarding the
      organization, I would recommend decreasing the space allocated to the
      description of Algorithm 2 as it is pretty similar at a high level to
      Algorithm 1 and it clearly doesn't need 3/4 of a page to understand.
      Originality and Significance I am not fully convinced of the
      significance of the paper (at least from the point of view of being
      published in a conference like NIPS). While I think it is important that a
      gap is closed, I would like to understand the impact of the algorithm
      either from a more applied perspective or get some insights with respect
      to where it could be used from a theoretical perspective. That being said,
      to the best of my knowledge, the algorithmic ideas are novel and
      interesting and they might be of use for other bandit
      algorithms.


RNADE: The real-valued neural autoregressive density-estimator
We introduce RNADE, a new model for joint density estimation of real-valued vectors. Our model calculates the density of a datapoint as the product of onedimensional conditionals modeled using mixture density networks with shared parameters. RNADE learns a distributed representation of the data, while having a tractable expression for the calculation of densities. A tractable likelihood allows direct comparison with other methods and training by standard gradientbased optimizers. We compare the performance of RNADE on several datasets of heterogeneous and perceptual data, finding it outperforms mixture models in all but one case.

Summary ======= The authors present a directed
      model for estimating the density of continuous random variables by
      exploiting the chain rule of probability theory. They propose to use
      particular weight-sharing constraints which have proven useful for
      modeling discrete data and combine it with mixture density networks.
      They show that their model can generally outperform large mixtures of
      Gaussians when applied to image patches, speech signals, and several
      smaller datasets. Comments ======== Weight sharing
      -------------- Since the main difference to related work appears
      to be in the RBM-inspired weight-sharing, it would be interesting to see a
      more thorough investigation of its effects. While it is clear that it can
      reduce computational costs, its effects on performance have not been fully
      explored. One would expect the weight-sharing to reduce overfitting where
      data is scarce, and to hurt performance where data is plenty. It would
      therefore be interesting to see some results on the extent to which weight
      sharing can help and to learn more about regimes where weight sharing
      might actually hinder performance. Natural image patches
      --------------------- In Section 4.2, RNADE is shown to perform
      similarly to a large mixtures of Gaussians when applied to natural image
      patches. While this already represents a competitive result, I believe
      that RNADE could fare even better here. Because of a mixture model's
      inability to represent independencies efficiently, it scales poorly to
      high dimensional data. I therefore suspect that RNADE would outperform a
      mixture model when tested on slightly larger patches. The number
      of training and validation points used for training RNADE also appears to
      be quite small. 25,000 training points (1,000 batches with 25 data points
      each) in my experience is not a lot, even for smaller image patches. If
      the performance does in fact not improve with more data (or, in other
      words, there is no overfitting with 25,000 training points), then this
      would also speak for the advantages of using the proposed weight-sharing
      constraints – which could be a point worth mentioning.
      Related work ------------ Since several directed
      models for real valued data already exist which all use Gaussian mixtures
      to represent conditional distributions and – which may not be obvious
      – have similar gating mechanisms to predict the mixing weights (Domke
      et al., 2008; Hosseini et al., 2009; Theis et al., 2012), a comparison
      with at least one of these models would have been nice. Minor
      comments -------------- It would be easier to judge the size of
      the differences in performance in Table 2 and 3 if additional models were
      included in the comparison, as in Table 1. There appears to be a
      $\rho_d$ missing in Equation 4. Quality ======= The
      methods used in the paper are technically sound. The authors provide
      extensive comparisons of their model and mixture models on several
      datasets and go to great lengths to ensure that the results are
      representative by performing 10-fold cross-validation or using very large
      test sets where possible. Clarity ======= The paper is
      well written and easy to follow. It also appears to include enough detail
      to reproduce the results. Originality =========== The
      paper explores the nontrivial extension of NADE – a model for discrete
      data – to the continuous case. While related models exist which
      also model continuous data and which share some similarities with the
      proposed model, there are also plenty of differences which make this an
      interesting and original contribution. Significance
      ============ Density estimation is an important problem underlying
      many applications. Just like the superior density estimation performance
      of NADE has proven useful in solving applications involving discrete data
      (Larochelle & Lauly, 2012), RNADE has the potential to be useful in
      tackling applications involving continuous
      signals.


Learning Gaussian Graphical Models with Observed or Latent FVSs
Gaussian Graphical Models (GGMs) or Gauss Markov random fields are widely used in many applications, and the trade-off between the modeling capacity and the efficiency of learning and inference has been an important research problem. In this paper, we study the family of GGMs with small feedback vertex sets (FVSs), where an FVS is a set of nodes whose removal breaks all the cycles. Exact inference such as computing the marginal distributions and the partition function has complexity O(k 2 n) using message-passing algorithms, where k is the size of the FVS, and n is the total number of nodes. We propose efficient structure learning algorithms for two cases: 1) All nodes are observed, which is useful in modeling social or flight networks where the FVS nodes often correspond to a small number of highly influential nodes, or hubs, while the rest of the networks is modeled by a tree. Regardless of the maximum degree, without knowing the full graph structure, we can exactly compute the maximum likelihood estimate with complexity O(kn 2 + n 2 log n) if the FVS is known or in polynomial time if the FVS is unknown but has bounded size. 2) The FVS nodes are latent variables, where structure learning is equivalent to decomposing an inverse covariance matrix (exactly or approximately) into the sum of a tree-structured matrix and a low-rank matrix. By incorporating efficient inference into the learning steps, we can obtain a learning algorithm using alternating low-rank corrections with complexity O(kn 2 + n 2 log n) per iteration. We perform experiments using both synthetic data as well as real data of flight delays to demonstrate the modeling capacity with FVSs of various sizes.

'Learning Gaussian Graphical Models with Observed or
      Latent FVSs' addresses the problem of learning (i.e. discovery of the
      graph structure) a GGM which can be decomposed into a small feedback
      vertex set and a tree. The motivation is that exact inference under these
      models can be done quickly, and so in the case where one needs near-linear
      inference (which is prohibited in general for sparse GGMs) it is desirable
      to have this form. The results address three cases: (4.1.1) all nodes are
      observed with known FVS, (4.1.2) all nodes are observed with unknown FVS,
      (4.2) and the FVS is latent. In (4.1.1) they provide a simple extension of
      the Chow-Liu algorithm, which applies CL to the empirical Schur
      complement. In (4.1.2) they make the observation that one can exhaustively
      run the previous algorithm for all k-sets selecting the one that maximizes
      the likelihood and then provide a greedy algorithm. In (4.2) they provide
      an EM-type algorithm that uses the conditional distribution from the
      previous iteration to find an empirical joint distribution then finding
      the best structure from this with CL. There is a nice interpretation using
      low rank adjustments in each step. They show that each iteration decreases
      the KL. There is a very nice experimental section, which applies this to
      flight traffic data. The paper is high quality, as it motivates
      the problem well, describes the landscape of learning GGMs, and
      comprehensively provides an algorithm for each situation. The paper is
      clear as it is well written and well organized. The paper seems original,
      as it is a unique extension of the Chow-Liu algorithm. The significance of
      this paper is somewhat suspect (although I admit that I am not an expert
      in this field). While it is the case that exact inference is more
      difficult for non-tree graphs, it is a bit difficult to imagine real-world
      GGMs that can be decomposed into the FVS and a tree. And if we don't
      believe that we are providing an accurate graph and only a convenient
      approximation, then approximate inference should be as good as we can hope
      for. If this is the case, then we can employ loopy BP, or even linear
      solvers. Prop 1 is really just the fact that CL works on the Schur
      complement too, and Prop 2 doesn't tell us that much. Again, I may have
      overlooked some fundamental insight into how important this extension is,
      so please argue with me on this point. With that said, I think that the
      insights into using CL with the FVS merits publication, as there is
      precedent for doing inference with the FVS. typos: 141: proved
      in the 265: the sentence 'We have no search of sets...' is awkward
      306: overlaping lines 313:
      D_KL(N(0,...


On Flat versus Hierarchical Classification in Large-Scale Taxonomies
We study in this paper flat and hierarchical classification strategies in the context of large-scale taxonomies. To this end, we first propose a multiclass, hierarchical data dependent bound on the generalization error of classifiers deployed in large-scale taxonomies. This bound provides an explanation to several empirical results reported in the literature, related to the performance of flat and hierarchical classifiers. We then introduce another type of bound targeting the approximation error of a family of classifiers, and derive from it features used in a meta-classifier to decide which nodes to prune (or flatten) in a large-scale taxonomy. We finally illustrate the theoretical developments through several experiments conducted on two widely used taxonomies.

Summary ======= This paper proposes a
      bound on the generalization error of hierarchical multi-classifiers for
      large-scale taxonomies. Such bound provides a justification of empirical
      results of flat and hierarchical classifiers. The paper also proposes a
      bound on the approximation error of a family of classifiers. This is used
      to define a node-classifier for pruning large-scale taxonomy and thus
      improving the overall accuracy. Some experiments illustrate the findings
      of the proposed theory on two well-known taxonomies. Evaluation
      ======== The paper is well written and presented. It
      contains interesting technical content, which appears to be sound. In
      general the paper is of high quality. However, this reviewer sees
      two limitations of the proposed methods: 1. Considering
      hierarchical classification as the task of only classifying the leaf nodes
      is rather limitative for two reasons: a. In real cases, category
      nodes can contain documents that do not belong to any of their children.
      This is very common as, when new documents of new types have to be
      classified, there is no specific child node to accommodate them. Thus the
      best choice is to assign them to the father nodes. b. This
      reviewer is not so sure that there is an actual debate on using flat and
      hierarchical models in the setting proposed by the authors (i.e., only
      leaf classification). Indeed, when working in the authors' setting, the
      flat model seems superior. In contrast, top-down methods may get
      better accuracy when classification is also carried out in the internal
      nodes. For example, the following paper clearly shows that the top-down
      method is more accurate than the flat one in such setting:
      Alessandro Moschitti, Qi Ju, Richard Johansson: Modeling Topic
      Dependencies in Hierarchical Text Categorization. ACL 2012: 759-767.
      In this respect a close comparison with other methods, e.g., the
      one above, those cited by the authors (e.g., Gopal et al.) and the
      following Zhou, D., Xiao, L., Wu, M.: Hierarchical classification
      via orthogonal transfer. ICML11. would be appreciated. 2.
      The proposed bounds do not look very strict: the Rademacher complexity
      term is a rough approximation and does not consider feature
      distribution/relevance, which has a major impact in text categorization.
      For example, in Reuters 21578, there are very rare categories. They
      may contain about 0.1% of the entire training data, i.e., about 10
      documents but, for them, systems can reach accuracy of about 90%, larger
      than for other more populated categories. This suggests that the data
      imbalance of categories is an important factor but it is not enough. For
      example, also category ambiguity (how much a category is similar to
      others) should be considered. After the authors' response
      ==================== In addition to the reviewer comments, the
      concepts expressed in the authors' answer should be included in the final
      version of the paper. Moreover, the authors' answer does not
      completely solve all the reviewer's doubts, thus the authors should inform
      the reader about the possible theory's flaws.


On the Complexity and Approximation of Binary Evidence in Lifted Inference
Lifted inference algorithms exploit symmetries in probabilistic models to speed up inference. They show impressive performance when calculating unconditional probabilities in relational models, but often resort to non-lifted inference when computing conditional probabilities. The reason is that conditioning on evidence breaks many of the model's symmetries, which can preempt standard lifting techniques. Recent theoretical results show, for example, that conditioning on evidence which corresponds to binary relations is #P-hard, suggesting that no lifting is to be expected in the worst case. In this paper, we balance this negative result by identifying the Boolean rank of the evidence as a key parameter for characterizing the complexity of conditioning in lifted inference. In particular, we show that conditioning on binary evidence with bounded Boolean rank is efficient. This opens up the possibility of approximating evidence by a low-rank Boolean matrix factorization, which we investigate both theoretically and empirically.

I should be upfront that though I am quite familiar
      with graphical models and Markov Random Fields, I'm far from an expert on
      Markov Logic Networks (MLNs). So I approached this paper through the lens
      of a generic graphical models person, and inevitably I tended to interpret
      the results in that way. This paper is interested in inference
      (computing marginal probabilities and/or a normalization factor) in a MLN.
      Inference is "conditional" in the sense that one will first observe some
      "evidence", which gives a subset of allowed values to each variable or
      pair of variables. Evidence is "unary" if it constrains single variables
      only, and "binary" if it also constraints pairs. The main idea is the
      following: Though inference in large MLNs is generally hard, some existing
      "lifted" algorithms can perform inference efficiently in large models by
      exploiting special structure in special classes of MLNs. However,
      conditioning on binary evidence destroys this special structure.
      This paper basically points out that, rather than conditioning on
      binary evidence, one could augment the graph with extra variables, and
      then condition on only on unary evidence. However, in the worst case, the
      needed number of new variables is exponential in the number of values each
      variable can take. However, if the binary evidence for a given pair
      happens to be (binary) low-rank, then a much smaller number of variables
      are needed. Thus, inference in general can be approached by (possibly
      approximately) finding a low-rank decomposition of each binary evidence
      term, adding extra variables, and then using the specialized unary
      inference algorithm. The major technical result is Theorem 2 which
      states that if you have a "lifted" inference algorithm and all pairwise
      evidence admits a low-rank decomposition, than exact inference can be done
      in polynomial time. This can naturally (i.e. heuristically) be done
      approximately through approximate decomposition. A couple other
      issues: - Is Boolean rank essential? Could one alternatively state
      that inference is always exponential in the number of values that each
      variable can have? (Which is weaker than Boolean rank, but simpler to
      understand.) The experiments use a very large number of variables, which
      motivates the decomposition, but presumably many applications have small
      numbers, in which case decomposition is superfluous. Is this result still
      useful? - Theorem 1 is strangely stated. I think this is previous
      work, in which case a reference needs to be provided. I also think that
      one should say "no polynomial time algorithm exists" rather than "takes
      exponential time". - Finally, for what its worth, as someone quite
      familiar with graphical models and MRFs, but not so familiar with Markov
      logic networks, I found this paper quite challenging to understand. The
      major issue is that very many technical terms ("evidence", "term",
      "literal", "relation", "inference", etc.) are used undefined, and there is
      no clear reference provided where these terms are precisely defined. After
      reading quite a few of the background papers I think I understand these,
      but the literature on MLNs doesn't seem totally consistent in usage, which
      makes things a challenge. (For example, typically "evidence" is a set of
      fixed values for a set of variables, whereas this paper uses it as a set
      of constraints on variables or pairs of variables.) At a minimum, the
      paper should state what is meant by inference, summarize what classes of
      symmetries previous work has taken advantage of to make inference
      tractable, and be precise about what evidence is and how it changes
      inference. It could be than in an eight page paper there is no way to make
      the current paper easily understandable by readers that aren't MLN
      experts, but I do think more effort could be made in this direction.
      Edit after author response: In my initial review, I
      had a concern about how conditioning could affect the efficiency of
      inference. After reading the other reviews, the author response, and
      discussing with the other reviewers, I'm convinced this isn't an issue,
      and am correspondingly raising my rating.


Inverse Density as an Inverse Problem: the Fredholm Equation Approach
We address the problem of estimating the ratio q p where p is a density function and q is another density, or, more generally an arbitrary function. Knowing or approximating this ratio is needed in various problems of inference and integration often referred to as importance sampling in statistical inference. It is also closely related to the problem of covariate shift in transfer learning. Our approach is based on reformulating the problem of estimating the ratio as an inverse problem in terms of an integral operator corresponding to a kernel, known as the Fredholm problem of the first kind. This formulation, combined with the techniques of regularization leads to a principled framework for constructing algorithms and for analyzing them theoretically. The resulting family of algorithms (FIRE, for Fredholm Inverse Regularized Estimator) is flexible, simple and easy to implement. We provide detailed theoretical analysis including concentration bounds and convergence rates for the Gaussian kernel for densities defined on R d and smooth d-dimensional sub-manifolds of the Euclidean space. Model selection for unsupervised or semi-supervised inference is generally a difficult problem. It turns out that in the density ratio estimation setting, when samples from both distributions are available, simple completely unsupervised model selection methods are available. We call this mechanism CD-CV for Cross-Density Cross-Validation. We show encouraging experimental results including applications to classification within the covariate shift framework.

The authors use two equalities to derive two
      estimators of the Radon-Nikodym derivative f= p/q: (i) Ep ( k(x,.)
      f) = E_q k(x,.) and (ii) Ep (k(x,.) f ) ~ q(x) The integral
      operators are replaced with empirical versions. {k(x,.)} can be seen as a
      set of test functions over which the difference between the right and left
      side is minimised. The choice of {k(x,.)} as the family of test functions
      has the advantage that standard kernel methods can be used for
      optimisation. Convergence guarantees are given for both settings
      under smoothness & boundedness assumptions on the densities. The rate
      of convergence depends crucially on the dimension of the underlying space
      -- as one would expect for a density estimate. I think it is nice
      and well executed work. A couple of comments: - For the
      convergence rates. Optimising over the set {k(x,.)} is sufficient to
      guarantee convergence to the ratio. For (ii) this is intuitive given the
      bandwidth dependence. Would be nice to have a short intuitive argument why
      this also works out in (i). -You optimise on page 2 over L_2,p.
      Later in (9) you have a combination of L_2,p and L_2,q. Some motivation
      for the choice of cost function would be nice. I guess ultimately these
      need to stem from the tasks one is interested in. So would be nice to link
      the cost function choice to certain important applications. -
      Speaking of applications, one example you got is importance sampling. Ie
      q/p is relevant for this to move from Eq to Ep. I'm wondering here: you
      are effectively estimating E_p and E_q to get your empirical cost
      function. So why not use directly the E_q you got there? Can E_p (q/p) be
      any better than directly E_q? I guess I'm asking here for a motivation of
      the setup since you need to produce estimates E_p and E_q to estimate q/p.
      You have a sentence on p 5 about it. That is sampling is difficult from
      E_q. Is this the kind of main application you have in mind? - The
      regulariser: You enforce smoothness for the ratio q/p. Later in the
      theorem you got smoothness assumptions on q and p ( which I would say is
      the natural thing). So I'm wondering if there is any sort of relation
      between smoothness on q and p and smoothness of the ratio? I guess an
      answer is here already that you have many different q and p which produce
      the same ratio; ie just change on countable or on uncountable many points
      q and p to make it extremely irregular. So it might be more p ,q smooth
      (possibly bounded away from 0) => ratio smooth (?) or ratio smooth
      => there exist a smooth p and q which produces the ratio. -
      Would be nice to discuss the d dependence of your rates -- for what
      dimensions would you expect your method to be useful? - Is there
      any good motivation for your family of test functions U? - A more
      principled question: For this q/p ; covariate shift etc setting one of the
      main difficulties seems to be to come up with a generic cost function. One
      approach is to go through expectation operators and throw different test
      functions at these to extract the underlying densities. Do you feel that
      you got here "the right" approach with the {k(x,.)} ? There is also some
      similarity to [1] in terms of playing around with transformations to get
      hold of the quantities of interest. Certainly, your approach for the Radon
      Nikodym derivative is nicer by working with test functions. But the
      overall approach has some similarity. [1] Smooth Operators;
      Grunewalder, Gretton, Shawe-Taylor


Recurrent linear models of simultaneously-recorded neural populations
Population neural recordings with long-range temporal structure are often best understood in terms of a common underlying low-dimensional dynamical process. Advances in recording technology provide access to an ever-larger fraction of the population, but the standard computational approaches available to identify the collective dynamics scale poorly with the size of the dataset. We describe a new, scalable approach to discovering low-dimensional dynamics that underlie simultaneously recorded spike trains from a neural population. We formulate the Recurrent Linear Model (RLM) by generalising the Kalman-filter-based likelihood calculation for latent linear dynamical systems to incorporate a generalised-linear observation process. We show that RLMs describe motor-cortical population data better than either directly-coupled generalised-linear models or latent linear dynamical system models with generalised-linear observations. We also introduce the cascaded generalised-linear model (CGLM) to capture low-dimensional instantaneous correlations in neural populations. The CGLM describes the cortical recordings better than either Ising or Gaussian models and, like the RLM, can be fit exactly and quickly. The CGLM can also be seen as a generalisation of a lowrank Gaussian model, in this case factor analysis. The computational tractability of the RLM and CGLM allow both to scale to very high-dimensional neural data.

The authors present a new descriptive model of spiking
      dynamics recorded in neural populations. The authors claim that the major
      advantage of the new model is that it can be fitted easily and more
      directly to experimental data than previous models and therefore it has a
      better performance. The paper is generally well written, although
      I found that the Results section is less clear than the presentation of
      the models. Details about the training and testing procedure are missing.
      The work is original and it makes an interesting progress in the models of
      neural population data. Testing of the model against its alternatives is
      not convincing in the present form and it needs to be clarified.
      The authors use many acronyms for different models and it is not
      always clear which one they are referring to. A tabular or a graphical
      (tree-like) representation showing the similarities and differences
      between these models could also be useful. (RLM, RNN, CLM, LDS, GLDS,
      PLDS, gl-RLM, GLM) Specific comments: line 134: All GLDS have
      an equivalent GRLM, but not all GRLM is equivalent to a GLDS. Putting this
      in a different way, these parameters are not independent, W and S depends
      on C. Similarly, is it true that a PLDS is a special case of RLMs with
      Poisson observation? So later in the Results al Data generated by PLDS
      could be, in principle, recovered by the appropriate class of RLMs?
      line 172: The authors describe two ways to handle instantaneous
      correlations in RLMs. It is not always clear which one is used later in
      the Results section. line 193: W \hat{y}_t = W f(CA\hat{x}_t) that
      could be (approximately) incorporated into the hidden dynamics. line
      197: The authors use the acronym PLDS for the Poisson LDS model later.
      line 235: vectors are usually typed in boldface in the paper, but I
      got completely lost here. y_n is a scalar, y_{ < n} is a vector but the
      typeset often suggests the opposite! line 238, Eq. 3: is \mu_n and
      \mu_{ < n} = 0 ? If not, isn't it missing from (y_{ < n} - \mu_{
      < n})? line 245: I do not see why we have (\Sigma_{ < n, <
      n})^{-1} instead of (\Sigma)^{-1} (same for line 255) line 257: I
      could not find how this line follows from the previous one and from the
      matrix inversion lemma, and I could not estimate how much we loose on the
      approximation. line 270: Alternative models. Does this section refer
      to alternatives of the CLM (it could be section 4.1. in this case) or in
      general? Alternative models have already been mentioned in section 3.1.
      line 281: Is B_i an NxN matrix? Or we have a single B matrix with NxN
      parameters? line 289: The actual physical interpretation of the latent
      variables in the RLM and CLM are also unclear and these models are also
      more descriptive models than models describing the actual generative
      process behind the data. line 307: see Tkacik et al., 2013 J. Stat
      Mech for a simple and tractable way to describe low dimensional modulation
      of spikes in Ising models. Section 6: Details of the fitting procedure
      are missing here, and it is also sometimes unclear what kind of models are
      fitted. e.g., what is the observation model for the RLM? Based on the
      figures, I assume that dimensions of the latent, x is 3 and the
      observations y is 5. Fig2ad: What is the observation model for the
      RLM? Gaussian (GRLM) or Poisson? Does the model GLDS used on Fig 2D has
      the same likelihood as the RLM? In this case is the data generated by the
      RLM consistent with the GLDS - parametrised by A, C and Q? Or the
      difference between PCA, GLDS and RLM are due to the different likelihood
      used? Fig2be: Is RLM the same as on Fig2ad? Here PLDS is just a
      special case of the RLM, so why the latter has a slightly lower
      performance? Fig2cf: What is the performance of a trained PLDS model
      on this dataset? It is clear that PCA and GLDS use a wrong likelihood
      function. line 371: Does LDS refer to GLDS here? line 377: what is
      the training/testing procedure? What is predicted here? Spike counts of a
      single cell given the stimulus+observation of other neurons in the
      population? It would be useful to show the actual data and the
      predictions. Figure 3a: error bars are missing. Does adding PSTH
      information to the other methods change their performance? Does the RLM
      found during training correspond to a PLDS model? Is the difference
      between PLDS and RLM because of the different training procedure or
      because RLM has a greater flexibility in its parameters? Fig4 shows
      only pooled data. It would be good to show cell-by-cell comparisons of
      e.g., the decay or the amplitude of the CCG. line 412: this part
      is a not really connected to the previous parts of the paper. I think that
      the procedure of training/testing is slightly different here, but there is
      no indication of this in the text. I appreciate the detailed
      response of the authors and I think that including these
      modifications/clarifications will certainly improve the paper. I modified
      the scores accordingly. I still think that a table or graph showing the
      similarities and differences between these models could also be
      useful.


From Bandits to Experts: A Tale of Domination and Independence
We consider the partial observability model for multi-armed bandits, introduced by Mannor and Shamir [14]. Our main result is a characterization of regret in the directed observability model in terms of the dominating and independence numbers of the observability graph (which must be accessible before selecting an action). In the undirected case, we show that the learner can achieve optimal regret without even accessing the observability graph before selecting an action. Both results are shown using variants of the Exp3 algorithm operating on the observability graph in a time-efficient manner.

The authors of this paper consider a kind of
      sequential prediction problem with a side graph information. It is a
      setting that is intermediary between the expert setting and the bandit
      setting, and where one observes, after choosing an arm, not only the
      outcome of the arm but also the outcome of some other arms that are linked
      to this arm through a graph. They consider two main settings, (i) the
      informed setting where the graph is made available to the learner before
      each prediction and (ii) the uniformed setting which they introduce and
      where the graph is only made available to the learner after his
      prediction. Their results also depend on whether the graph is directed or
      not. The results are provided in the adversarial case. The authors
      provide some results in these two setting (uninformed and informed). They
      first provide an algorithm in the uniformed setting which I believe are
      the first ones in this setting, and bound the regret in the undirected and
      the directed case. In the directed case, there is a gap between the lower
      and upper bound for the first algorithm they introduced. In the informed
      setting, they manages, in the directed case, to go closer to the lower
      bound by building a new algorithm. I find the paper well written
      and the results interesting. My main comment on it would be that it would
      gain in clarity if the authors could: - write explicitly in the bandit
      and expert setting the bound of EXP3-SET, and comment on the fact that it
      matches the lower bound (up to some log terms in the bandit case) in both
      cases. The bound in the case of Erdos-Renyi random graphs is nice but
      gives less insight and you can put it in appendix if you lack space. -
      To me Theorem 1 is a technical side results, and Corollary 2 is more
      clear. You should maybe delete Theorem 1 and keep corollary 2. - I
      think it is very important that you extend a bit the explanation right
      after Corollary 2 on how to choose the \alpha_t, and of how much is the
      loss if you do not know them for tuning \beta. What you say makes sense,
      but it is too concise and have not a clear picture of how much you loose
      in the regret by doing what you propose. - Maybe even, you could
      suppress Corollary 2 and just keep Corollary 4. And then, you will have
      more space to do an extended discussion. - It would really help the
      reader to discuss the links and differences between Corollary 4 and
      Corollary 8 (again, to me, you can suppress Theorem 7 and just keep
      Corollary 8). For instance, provide an example of graph where \alpha(G) is
      small and mas(G) is large: after all, it is not obvious from your results
      that the bound in Corollary 8 is sharpen than the bound in Corollary 4
      (even though the setting in Corollary 8 supposes that the graph
      information is available to the predictor), since you have an additional
      ln(K)\sqrt{\log(T)} that appears. Please discuss this in details.
      As a conclusion I like this paper, and find the results nice. It
      seems correct but I did not have time to check the proofs in full
      details.


Predictive PAC Learning and Process Decompositions
We informally call a stochastic process learnable if it admits a generalization error approaching zero in probability for any concept class with finite VC-dimension (IID processes are the simplest example). A mixture of learnable processes need not be learnable itself, and certainly its generalization error need not decay at the same rate. In this paper, we argue that it is natural in predictive PAC to condition not on the past observations but on the mixture component of the sample path. This definition not only matches what a realistic learner might demand, but also allows us to sidestep several otherwise grave problems in learning from dependent data. In particular, we give a novel PAC generalization bound for mixtures of learnable processes with a generalization error that is not worse than that of each mixture component. We also provide a characterization of mixtures of absolutely regular (β-mixing) processes, of independent probability-theoretic interest.

This paper focuses on learning with dependent
      observations. Under various assumptions, it is possible to upper-bound the
      deviations of the empirical mean with respect to the expectation, thus
      allowing to study empirical risk minimization. These assumptions ranges
      from various mixing assumptions to so-called weak-dependence condition.
      The authors remark that when P and Q are two probability
      distributions for the stochastic process X, when X is mixing under P and
      under Q, it is generally not mixing under aP + (1-a)Q. However, if E is a
      deviation undersirable event, and if P(E)\leq e and Q(E)\leq e, then
      (aP+(1-a)Q)(E)\leq e. So, it is possible to extend deviation inequalities
      to a wider class of probability distribution (that is, mixture of mixing
      disitributions). This remark might seem trivial, however, it allows some
      interesting generalization. For example, as exchangeable probability
      distributions can be written as mixtures of iid distributions, this remark
      extends results from iid setting to exhangeable distribution at no cost.
      However: 1) to my opinion, this is a small increment to
      learning with dependent observations. 2) the paper is not well
      written. Unnecessary discussions hide the main message of the paper. On
      the other hand, Definition 1 is non standard and should be explained in
      more details. The bibliography is incomplete. A lot has been done on
      learning with dependent observations in Modha and Masry (IEEE Trans. Info.
      Theory, 1998), Meir (JMLR, 2000), a series of papers by Steinwart, e.g.
      Steinwart, Hush and Scovel (Journal of Multivariate Analysis, 2009),
      Alquier and Wintenberger (Bernoulli, 2012) among
      others.


Blind Calibration in Compressed Sensing using Message Passing Algorithms
Compressed sensing (CS) is a concept that allows to acquire compressible signals with a small number of measurements. As such it is very attractive for hardware implementations. Therefore, correct calibration of the hardware is a central issue. In this paper we study the so-called blind calibration, i.e. when the training signals that are available to perform the calibration are sparse but unknown. We extend the approximate message passing (AMP) algorithm used in CS to the case of blind calibration. In the calibration-AMP, both the gains on the sensors and the elements of the signals are treated as unknowns. Our algorithm is also applicable to settings in which the sensors distort the measurements in other ways than multiplication by a gain, unlike previously suggested blind calibration algorithms based on convex relaxations. We study numerically the phase diagram of the blind calibration problem, and show that even in cases where convex relaxation is possible, our algorithm requires a smaller number of measurements and/or signals in order to perform well.

The "blind calibration" problem involves a
      compressed sensing problem where, when observing a M-dimensional
      measurement of N-dimensional sparse signals x_1,x_2,..., each of the M
      measurement sensors is distorted in some parametrized way, where the
      parameter can vary over the sensors. Therefore the goal is to be able to
      recover these distortion parameters d_1,...,d_M at the same time as
      estimating the sparse signals. This paper proposes an effective
      message-passing algorithm "C-AMP" to solve the two problems
      simultaneously. The authors use a simple dimensionality argument to derive
      a lower bound alpha_min on the sampling ratio alpha=(measurement dimension
      M)/(signal dimension N), which is needed for recovery to be possible. In
      the simulation results, C-AMP performs very well, with successful signal
      recovery nearly everywhere above the threshold max{alpha_CS,alpha_min}
      where alpha_CS is the phase transition for ordinary compressed sensing
      (i.e. the setting where there are no unknown calibration parameters). In
      contrast, L1 minimization (which does not take the calibration issue into
      account) does not perform nearly as well. I did find the initial
      presentation of the problem to be somewhat confusing. Because blind
      calibration is presented after background is given on supervised
      calibration, it gives the impression that the goal is again the estimation
      of the distortion parameters d_mu, rather than signal recovery (the sparse
      x_l's). It would perhaps be clearer to say something like, "Given a sparse
      signal recovery problem, if we were not able to previously estimate the
      distortion parameters via supervised calibration, we will need to estimate
      the unknown signals and the unknown distortion parameters simultaneously -
      this is known as blind calibration."
      Other comments... -- 062:
      z_{\mu l} should be z_{\mu}? -- 063-064: not clear if w is the
      noise or w is noise+z? I assume that w is the noise Delta; might be
      clearer to say "one usually considers an iid Gaussian noise with variance
      Delta, which will then be added to z". Also there is an extra "one" in
      this sentence. -- 101-102: this is hard to follow. Perhaps specify
      what is rho. -- 108: "the use if AMP" - > "the use of AMP"
      -- 132: very nice intuitive derivation of the lower bound
      -- equation in line 146 is very clear and easy to read; in-line
      equations in lines 178-180 are harder to read & understand --
      149: "Z" is not the best notation for the normalizing constant since "z"
      is used for something unrelated -- 158-159: "NPM messages" and "PM
      messages" should maybe be "pairs of messages" since ther are m's and
      \tilde{m}'s? -- 263-264: This line is a bit awkward to read,
      perhaps rewrite -- 268-269: not sure what it means to have a
      uniform distribution with a mean & variance parameter - I assume it
      means that sigma^2 determines the width of the interval, i.e. the width is
      chosen such that variance is equal to sigma^2? This is an unusual
      parametrization and it might be easier to just give the width itself.
      -- 300: what is the green line in the central figure? --
      380: these figures & results are very clear. It might be nice to
      discuss alpha_min a bit more - give some intuition for how alpha_min is
      not a phase transition, since alpha_CS sometimes lies above alpha_min in
      these figures; perhaps the maximum of alpha_min and alpha_CS functions as
      a phase transition? Or is there some single bound that might unite the
      two? -- I think it's more standard, in phase transition diagrams,
      to use light regions to indicate successful signal recovery and dark
      regions when the signal cannot be recovered.


Robust Multimodal Graph Matching: Sparse Coding Meets Graph Matching
Graph matching is a challenging problem with very important applications in a wide range of fields, from image and video analysis to biological and biomedical problems. We propose a robust graph matching algorithm inspired in sparsityrelated techniques. We cast the problem, resembling group or collaborative sparsity formulations, as a non-smooth convex optimization problem that can be efficiently solved using augmented Lagrangian techniques. The method can deal with weighted or unweighted graphs, as well as multimodal data, where different graphs represent different types of data. The proposed approach is also naturally integrated with collaborative graph inference techniques, solving general network inference problems where the observed variables, possibly coming from different modalities, are not in correspondence. The algorithm is tested and compared with state-of-the-art graph matching techniques in both synthetic and real graphs. We also present results on multimodal graphs and applications to collaborative inference of brain connectivity from alignment-free functional magnetic resonance imaging (fMRI) data. The code is publicly available.

This paper examines the problem of approximate graph
      matching (isomorphism). Given graphs G, H with p nodes, represented by
      respective adjacency matrices A, B, Find a permutation matrix P that
      best ``matches’’ AP and PB. The paper indicates that this is a
      problem with a long history. The fundamental problem is combinatorial and
      its complexity is not fully understood. This has led to various relations
      of the problem. For example to minimize a convex matching metric under the
      constraint that the permutation P is relaxed to a doubly stochastic matrix
      **[reference needed at this point in the paper (line 061)]. Once
      the resulting convex problem is solved one then finds the nearest
      permutation matrix to the solution. The authors point out that the
      traditional Frobenius norm matching metric ||AP-PB||_F^2, does not
      produce a common core shared graph with a sparse set of outlier edges.
      Hence they propose a group lasso approach in which the matching metric is
      the sum of the Euclidean norms of the corresponding entries of AP and PB
      (equation 2). This is motivated by the heuristic argument that the group
      lasso will tend to set most groups of coefficients to zero with only a few
      groups nonzero. So the metric will seek a permutation that matrices
      many edges and will have a sparse set of edges that do not match well.
      The only theoretical result of the paper is Lemma 1, which
      verifies that the proposed matching metric makes sense in the very special
      case of two isomorphic, undirected, unweighted graphs with no self loops
      (and obviously no noise in the form of randomly added edges). The lemma
      indicates that in this case, minimizing the metric will achieve AP=PB with
      P doubly stochastic. The proof is very elementary. The authors
      then propose an optimization procedure for minimizing the new matching
      metric subject to P being doubly stochastic (problem (3)). This is based
      on a linearized variation of ADMM drawing heavily from the work by Lin et
      al (NIPS 2011). **[Note: possible typo line 154: ADMOM ?].
      Section 4, shows how the above ideas could be applied to
      collaborative inverse covariance estimation in the spirit of the graphical
      lasso. This leads to a non convex problem, but which is convex when either
      of two disjoint set of variable is fixed. This gives an alternating, two
      phase sequence of convex problems. The basic problem seems to have its
      origins in Chiquet 2011, where it is addressed using a different method.
      The experimental section of paper describes four experiments. The
      first explores performance of the proposed method and algorithm on several
      synthetic forms of graphs. The proposed method appears to have significant
      advantages in these experiments. The second experiment involves graph
      matching on real graphs from the connectome of C. elegans. There are two
      graphs (chemical connections and electrical connections). Although the
      data is real the problem solved is artificial: each graph is first
      permuted, edge noise is added and then this graph is matched to its
      original. Experiment three explores the proposed method’s ability
      to match graphs with edge weights drawn from different distributions
      (a.k.a. multimodal graph matching). This is explored using synthetic
      datasets in the spirit of the first experiment. At low noise levels the
      proposed method seems to have a distinct advantage over competing methods
      – note however that in this case this means there is an (almost) exact
      match – this is unlikely to happen in real-world datasets. The
      fourth experiment on collaborative inference is also based on real data –
      in this case resting state fMRI. However, the actual experiment is
      synthetic with the data to be matched obtained by permuting the real data.
      This is an interesting experiment. It shows that that the proposed method
      has potential advantages over the usual application of the graphical lasso
      – but is this surprising? The graphical lasso uses each graph separately,
      while the proposed method allows the graphs to collaborate. This is
      obviously a good thing and should lead to improved results. The experiment
      confirms that this is the case. Let me note that in several places
      the authors emphasize that the proposed method shows promising performance
      “on real data”. For example, phrases of this type appear twice in the
      conclusion. This is a little misleading since although the data is
      obtained from real experiments the actual tests performed are synthetic
      and hence are not reflective of actual alignments that would occur in
      practice. It would be better to say: “shows promising performance using
      real data to form synthetic alignment problems”.


Beyond Pairwise: Provably Fast Algorithms for Approximate k-Way Similarity Search
We go beyond the notion of pairwise similarity and look into search problems with k-way similarity functions. In this paper, we focus on problems related to 3-way Jaccard similarity:
where C is a size n collection of sets (or binary vectors). We show that approximate R 3way similarity search problems admit fast algorithms with provable guarantees, analogous to the pairwise case. Our analysis and speedup guarantees naturally extend to k-way resemblance. In the process, we extend traditional framework of locality sensitive hashing (LSH) to handle higher-order similarities, which could be of independent theoretical interest. The applicability of R 3way search is shown on the "Google Sets" application. In addition, we demonstrate the advantage of R 3way resemblance over the pairwise case in improving retrieval quality.

This paper describes a nice trick to extend MinHash to
      estimate the generalization of the Jaccard index to more than two sets.
      Using the trick, the paper presents efficient search algorithms for (1)
      given S and T, indentifying U for which (S,T,U) are similar, (2) given S,
      identifying T and U for which (S,T,U) are similar, and (3) identifying S,
      T and I for which (S,T,U) are similar. There are experiments showing the
      value of a >2-way similaruty measure. This paper is not written
      well and is consequently difficult to read. As far as I know, the
      work is original. The algorithms are significant and may prove to be
      useful in many domains. If this paper is edited to remove all of the
      grammar errors, it will be very nice. The anecdotes for the
      "Google Sets" problem are very cool. I found the search-quality
      experiments very hard to follow. It would be help a lot if the metric
      discussion is promoted to come earlier in the section and discussed in a
      little more depth. Example presentation issues (restricting to one
      per page) Page 1: "one of the most widely used notion" =>
      "notions" Page 2: "may be" => "maybe" Page 3: "any two point"
      => "any two points" Page 4: "One of the peculiarity" =>
      "peculiarities" Page 5: "c-BP" in the Section 5 title should be "c-CP"
      Page 6: "if we are interested in very high similarity cluster" =>
      "a very" or "clusters" Page 7: "report top 5 words" => "report the
      top 5 words" Page 8: "We formulated ... and obtain" => "formulate"
      or "obtained"


Robust Transfer Principal Component Analysis with Rank Constraints
Principal component analysis (PCA), a well-established technique for data analysis and processing, provides a convenient form of dimensionality reduction that is effective for cleaning small Gaussian noises presented in the data. However, the applicability of standard principal component analysis in real scenarios is limited by its sensitivity to large errors. In this paper, we tackle the challenge problem of recovering data corrupted with errors of high magnitude by developing a novel robust transfer principal component analysis method. Our method is based on the assumption that useful information for the recovery of a corrupted data matrix can be gained from an uncorrupted related data matrix. Specifically, we formulate the data recovery problem as a joint robust principal component analysis problem on the two data matrices, with common principal components shared across matrices and individual principal components specific to each data matrix. The formulated optimization problem is a minimization problem over a convex objective function but with non-convex rank constraints. We develop an efficient proximal projected gradient descent algorithm to solve the proposed optimization problem with convergence guarantees. Our empirical results over image denoising tasks show the proposed method can effectively recover images with random large errors, and significantly outperform both standard PCA and robust PCA with rank constraints.

This paper considers robust principal component
      analysis, from the approach of transfer learning. The goal is to obtain a
      method that, according to the paper, can deal with not only small and/or
      sparse errors, but also dense large errors, in the setting where there are
      two data sources (two data matrices) which have some overlap in their
      principal components. The authors then propose a rank-constrained
      optimization problem that is the natural formulation, assuming sparse
      errors; that is, they propose an objective which balances between the L2
      loss in fitting the data matrix, plus an L1 penalty on the sparse
      corruption. This, in principle, should allow the handling of sparse noise,
      and also smaller dense noise. Instead of relaxing the rank constraints,
      they propose a projected proximal type iterative method, where they
      project back to matrices of appropriate rank, at every step. There
      are several issues with this paper that if addressed, would significantly
      strengthen the contribution. There are no conditions of success
      that are discussed. That is, the authors do not describe, theoretically,
      or even intuitively, what the conditions should be on the two data
      matrices and a the noise corrupting them, that would allow their method to
      succeed in finding the correct low dimensional structure. To be more
      specific, if we compare to the papers that the authors cite in the
      introduction, e.g., [11] or [14], or the numerous other papers that have
      been written on various formulations of robust PCA, e.g., papers by E.
      Candes et al., or papers by H. Xu et al., there, the authors give
      algorithms along with specific conditions for which they provide
      theoretical guarantees that their algorithms succeed. There does not seem
      to be something similar here. Precisely because it is not the usual
      setting, it would be very interesting to understand when transfer learning
      might succeed, where other methods might fail. This is also very
      important because some of the claims the authors make do not seem to be
      backed up by a rigorous argument. The algorithm proposed is indeed
      natural. However, it is solving a non-convex problem, because of the rank
      constraints. By setting X_t or X_s to zero, it seems that one recovers the
      low-rank plus sparse recovery problem (see, e.g., Chandrasekaran et al.,
      and Candes et al.). As mentioned previously, this is a well-studied and
      fairly well understood problem. Does their algorithm have better
      performance than the setting when the rank constraints are relaxed via the
      nuclear norm? The wording in the paper seems to suggest this, but there is
      nothing to support it.


Online Learning of Dynamic Parameters in Social Networks
This paper addresses the problem of online learning in a dynamic setting. We consider a social network in which each individual observes a private signal about the underlying state of the world and communicates with her neighbors at each time period. Unlike many existing approaches, the underlying state is dynamic, and evolves according to a geometric random walk. We view the scenario as an optimization problem where agents aim to learn the true state while suffering the smallest possible loss. Based on the decomposition of the global loss function, we introduce two update mechanisms, each of which generates an estimate of the true state. We establish a tight bound on the rate of change of the underlying state, under which individuals can track the parameter with a bounded variance. Then, we characterize explicit expressions for the steady state mean-square deviation(MSD) of the estimates from the truth, per individual. We observe that only one of the estimators recovers the optimal MSD, which underscores the impact of the objective function decomposition on the learning quality. Finally, we provide an upper bound on the regret of the proposed methods, measured as an average of errors in estimating the parameter in a finite time.

This paper addresses the problem of distributed
      estimation of underlying dynamic states (which is a geometric random walk)
      on a graph. Estimate the underlying state is not hard is all the
      observations are globally known. The main challenge here is to perform
      local update based on information available from neighbors only. The
      authors cast the problem into an online minimization of a separable,
      time-varying cost function and then provide two methods to perform local
      update on each node by employing distributed dual averaging methods. The
      two methods differ in the way they choose the local loss function. Then
      the necessary conditions for the convergence of beliefs are established.
      By defining the steady state mean square deviation, the authors
      provide a regret analysis of the two methods and establish a bound for the
      regret. This paper is technically sound. The organization of the
      paper is logical and clear. Main results of the paper seem to be
      correct. As pointed out by the authors, there is a previous work
      focusing of distributed estimation of the underlying world given a simple
      random walk model [9]. [9] solves the special case where the graph is
      a complete graph. This paper changes the underlying state model to a
      geometric random walk and the analysis are conducted on a more general
      graph. However, the problem this paper target at is of limited
      practical impact. The underlying state is a geometric random walk with
      a constant rate of change, which is assumed to be known apriori to all
      agents. This constant rate of change could limit the application of
      the proposed methods. It is hard to find a real world scenario where
      the model could be used. Maybe the author could provide more
      motivating examples.


A Graphical Transformation for Belief Propagation: Maximum Weight Matchings and Odd-Sized Cycles
Max-product 'belief propagation' (BP) is a popular distributed heuristic for finding the Maximum A Posteriori (MAP) assignment in a joint probability distribution represented by a Graphical Model (GM). It was recently shown that BP converges to the correct MAP assignment for a class of loopy GMs with the following common feature: the Linear Programming (LP) relaxation to the MAP problem is tight (has no integrality gap). Unfortunately, tightness of the LP relaxation does not, in general, guarantee convergence and correctness of the BP algorithm. The failure of BP in such cases motivates reverse engineering a solution -namely, given a tight LP, can we design a 'good' BP algorithm. In this paper, we design a BP algorithm for the Maximum Weight Matching (MWM) problem over general graphs. We prove that the algorithm converges to the correct optimum if the respective LP relaxation, which may include inequalities associated with non-intersecting odd-sized cycles, is tight. The most significant part of our approach is the introduction of a novel graph transformation designed to force convergence of BP. Our theoretical result suggests an efficient BP-based heuristic for the MWM problem, which consists of making sequential, "cutting plane", modifications to the underlying GM. Our experiments show that this heuristic performs as well as traditional cutting-plane algorithms using LP solvers on MWM problems.

The authors extend a known theorem that states that,
      if the linear relaxation of an MWM problem is tight, then BP will converge
      correctly on the corresponding graphical model. While this previously was
      only shown with no "blossom" inequalities -- i.e. inequalities that help
      to achieve integrality of the LP -- the paper extends this theorem to hold
      also with such inequalities. That allows more instances of MWM to be
      solved via BP. This reviewer thinks that the paper adds a tiny
      puzzle piece to understand loopy BP properties, but is not perfectly clear
      in the presentation and lacks an (experimental) proof of the usefulness of
      the results. Clarity of presentation: - The is \Omega a set of
      node indices or the value range of Z_i? - The GM is introduced in L104
      without ever referring to a graph. ? - The notation in the
      introduction is not very clear, which conditions are necessary and which
      are sufficient (L50: "BP converges ... when LP tight" ==> supposed to
      mean not sufficient, L68 "BP converges ... if LP is tight" ==> supposed
      to mean sufficient). - What is an odd-sized cycle? |C| odd or |E(C)|
      odd? - L139: For general (non-pairwise-potential) GM problems, the
      messages could have more than one free variable. - L179: What is V(C)?
      - "it is not hard to check ... updates involving \Phi_C can be done
      using dynamic programming" I cannot see that. - Figure 1 is not
      intuitive and is not explained sufficiently. - L293-306 The proof's
      correctness is obstructed by too complicated notation (e.g. why is
      y_{e’}{NEW}=1? Is this implied by the odd/even path in your notation?)
      Usefulness of the results: - Why not simply do LP? The only
      real advantage to me seems that BP can be implemented in a distributed
      fashion. But if so, can it also be checked in a distributed fashion,
      whether the LP is tight (and thus the BP result is trustworthy)? - The
      experiments should not only use random graphs but real ones. Otherwise the
      benefit of the blossom inequalities may be overrated. - The
      experiments should show beneficial timings or some other proof of the
      advantage of the proposed method. - The potentials \Phi_C potentially
      contain many variables. Summing over them during the BP update may become
      exponentially hard. Response to the authors' rebuttal: In their
      rebuttal, the authors have put their work in the light of "making LP ready
      for parallel computation". Moreover, they have promised to add more
      experimental results to support this claim. I think this line of
      argumentation adds an interesting perspective to the work. It should be
      elaborated on in the final version of the paper.


A Stability-based Validation Procedure for Differentially Private Machine Learning
Differential privacy is a cryptographically motivated definition of privacy which has gained considerable attention in the algorithms, machine-learning and datamining communities. While there has been an explosion of work on differentially private machine learning algorithms, a major barrier to achieving end-to-end differential privacy in practical machine learning applications is the lack of an effective procedure for differentially private parameter tuning, or, determining the parameter value, such as a bin size in a histogram, or a regularization parameter, that is suitable for a particular application. In this paper, we introduce a generic validation procedure for differentially private machine learning algorithms that apply when a certain stability condition holds on the training algorithm and the validation performance metric. The training data size and the privacy budget used for training in our procedure is independent of the number of parameter values searched over. We apply our generic procedure to two fundamental tasks in statistics and machine-learning -training a regularized linear classifier and building a histogram density estimator that result in end-toend differentially private solutions for these problems.

A Stability-Based Validation Procedure for
      Differentially Private Machine Learning
      ------------------------------------------------------------------------------------------------------
      The paper discusses the problem of finding in a differentially private
      manner which hypothesis out of a given set of k hypotheses (often
      represented as k potential values for a parametrized learning problem,
      like SVM) gives a better score on a separate validation set. The overall
      approach is straight-forward: you try all k hypothesis and publish the one
      that achieves minimal error perturbed by Laplace noise (or highest
      perturbed score). Clearly, such approach is differentially private only if
      one has the guarantees that a single person cannot have a significant
      effect on the score of the best hypothesis (or the hypothesis itself), a
      premise which the authors refer to as stability. The authors then apply
      their paradigm to the specific cases of linear classifiers and histogram
      estimators. Finally, the authors evaluate their technique empirically
      comparing it to various other differentially private estimators (like
      naively trying all k assumptions with \alpha=\alpha/k, or splitting the
      data into k random chunks training each hypothesis on a different chunk).
      The paper address an important problem and gives a nice and clean
      solution to it. The authors take the time to compare their technique to a
      non-private version of the algorithm, and the paper is really well
      written. Also, the general problem statement is nice. If we want to learn
      the best of k hypotheses while preserving \alpha-DP, there are 2 strawmen
      algorithms: to learn each while preserving (\alpha/k)-DP, which leads to
      poly(k)-degradation in performance, or to learn each in a manner that
      preserves \alpha-DP on a 1/k fraction of the data, which causes the sample
      bounds to grow by a factor of k. The authors' observation is that DP
      machine learners often output an hypothesis that only slightly changes
      w.r.t a change of a single individual, so the validation score of each of
      the k-hypothesis isn't prone to change by much and adding a small amount
      of noise to the validation score of each hypothesis is therefore enough to
      preserve DP. The two major disadvantages of the paper, in my
      opinion, lie in the fact that the observation is cute but not very
      surprising -- the stability is a by-product of the fact that the learning
      algorithms are DP. Because the hypotheses outputted by the learning on any
      two neighboring training datasets are close, then their validation score
      is close too. The second problem is in the experiment: for small \alpha
      (the more interesting regime) is seems that this approach does almost as
      well as picking a random hypothesis. (Also, why doesn't hypothesis
      "control" has \alpha=0?) I would much rather see a different experiment,
      in which the data is synthetically generated so that some hypothesis best
      explains it, and the authors check how many examples are required in each
      technique until each of the \alpha-DP algorithms reached
      error<\epsilon. If indeed the number of samples improves dramatically
      with the new technique, that would serve as a demonstration that this
      approach is best. Also, I object to the use of the term stability.
      Since q is a function, I do not understand what's wrong with the more
      concrete notion of Lipfshitzness or smoothness. Still, all in all
      it is an interesting paper. I recommend weak acceptance.


Learning Chordal Markov Networks by Constraint Satisfaction
We investigate the problem of learning the structure of a Markov network from data. It is shown that the structure of such networks can be described in terms of constraints which enables the use of existing solver technology with optimization capabilities to compute optimal networks starting from initial scores computed from the data. To achieve efficient encodings, we develop a novel characterization of Markov network structure using a balancing condition on the separators between cliques forming the network. The resulting translations into propositional satisfiability and its extensions such as maximum satisfiability, satisfiability modulo theories, and answer set programming, enable us to prove optimal certain networks which have been previously found by stochastic search.

The authors formulate the problem of learning an
      undirected graphical model in terms of a large (intractable) constraint
      satisfaction problem (CSP)In particular, given observations of the
      variables, X, they focus on finding the junction tree, T, that with the
      largest marginal likelihood p(X|T). Rather than blindly searching in the
      space of all junction trees, they formulate a weighted constraint
      satisfaction problem and propose to use off-the-shelf CSP solver to find
      the best junction tree. The majority of the paper is on
      formulating the CSP, including characterizing a junction tree as a
      collection of cliques and separators over a balanced max weight spanning
      tree. While this paper is very well written and appears to be novel it may
      be of little practical value simply because formulating the CSP problem
      requires an exponential number of constraints - in other words, what do we
      gain by casting the structure learning problem as a CSP? I would liked to
      have seen an approximate CSP that relaxes some conditions but remains
      tractable, or a bounded form of the CSP that learns the most likely
      junction tree with bounded clique size. Last, I'm not sure how a
      non-uniform prior on graph structures could be incorporated, but we would
      prefer a means to learn sparse/low treewidth
      models.