Near-Optimal Entrywise Sampling for Data Matrices
We consider the problem of selecting non-zero entries of a matrix A in order to produce a sparse sketch of it, B, that minimizes A B 2 . For large m n matrices, such that n m (for example, representing n observations over m attributes) we give sampling distributions that exhibit four important properties. First, they have closed forms computable from minimal information regarding A. Second, they allow sketching of matrices whose non-zeros are presented to the algorithm in arbitrary order as a stream, with O 1 computation per non-zero. Third, the resulting sketch matrices are not only sparse, but their non-zero entries are highly compressible. Lastly, and most importantly, under mild assumptions, our distributions are provably competitive with the optimal offline distribution. Note that the probabilities in the optimal offline distribution may be complex functions of all the entries in the matrix. Therefore, regardless of computational complexity, the optimal distribution might be impossible to compute in the streaming model.

Near-optimal Distribution for Data Matrix Sampling
      The paper presents a new sampling methods for generating (random)
      sparse matrices that best approximate a given matrix in terms of the
      spectral norm. Several assumptions are made and well motivated, such as
      the "data matrix" properties and the constraints of the computational
      model (i.e. streaming data). The exposition is very readable, although
      some topics such as compressibility and fast indexing are brought up that
      are then never related back to the method proposed. More to the heart of
      the matter, the author(s) start with discussing the commonly known L1 and
      L2 sampling schemes, which serve as reference points throughout the
      development of the paper and then move on to the matrix Berstein
      inequality and explain its significance in the context of sparse matrix
      sampling. A lot of attention is given to proper explanation and intuitive
      arguments, which makes the paper accessible for non-specialists. The only
      thing that has to be said is that the text becomes somewhat "talkative"
      and lengthy, in particular in comparison to the very condensed
      experimental section. I would recommend shortening these passage, while
      keeping the gist of the arguments and explanations. The main
      contributions of the paper are algorithm 1 together theorem 2.2, which
      justifies it. I find eq 6 and what feeds into it somewhat hard to digest.
      Some properties (e.g. certain regimes for s: small, large) are explained
      in the surrounding text, but it would be nice to find a way of relating
      this back to the Bernstein inequality that supposedly motivates it. Some
      of this happens in section 5, but overall the algebraic complexity and
      notation make this difficult and it would be nice to preserve the crucial
      link between how even the simplified row-L1 distribution and the Bernstein
      inequality. With all these caveats around the structure and clarity of the
      presentation of the main contributions, I deem the result to be
      significant and the key idea as well as the execution of the proof and
      algorithmic detail to be highly non-trivial. The experimental
      section and the empirical analysis are somewhat disappointing by
      comparison. First, I cannot even read the graphs in the printed paper
      (which is a nuisance). Second, there is really no diving deeper on any
      aspect. Third, no connection to any relevant application is made. Fourth,
      the spectral norm criteria is thrown out ("it turns out to be more
      fruitful to consider…") and replaced by a new criterion based on
      projections to the top singular vectors. The spectral norm objective is
      not even plotted or mentioned any more. Given how much care has been spent
      on the theory part, how detailed things have been motivated and then
      worked through in the proof, this feels like quite a surprise. Shouldn't
      one go back and change the objective to address the "scaling" issue, for
      instance?


Polar Operators for Structured Sparse Estimation
Structured sparse estimation has become an important technique in many areas of data analysis. Unfortunately, these estimators normally create computational difficulties that entail sophisticated algorithms. Our first contribution is to uncover a rich class of structured sparse regularizers whose polar operator can be evaluated efficiently. With such an operator, a simple conditional gradient method can then be developed that, when combined with smoothing and local optimization, significantly reduces training time vs. the state of the art. We also demonstrate a new reduction of polar to proximal maps that enables more efficient latent fused lasso.

-- first-round comments -- This paper proposes
      new methods to compute the polar operator for a class of regularizers
      based on marginalized modular functions such as group sparsity.
      Pros: - well-written, and to-the-point. Logically organized
      - subject is topical (group sparsity, sparsity on DAGS) - Polar
      operators have not been studied too much (with respect to practical use)
      compared to proximal operators. The proposed method seems to be quite good
      - Recent interest in conditional gradient/Frank Wolfe means that polar
      operators are of interest - Nice secondary results (e.g., prox of lp +
      TV) Cons: - Prop. 2 allows one to compute the polar via the
      prox, due to the link between the dual functions and polar functions (and
      between a function and its conjugate, via Moreau's Thm, eq (43)). By
      itself, the proposition is not too surprising, and I strongly suspect that
      a "converse" can also be shown (i.e., one can compute the prox via the
      polar). If this is true, then does the advantage of the GCG over APG still
      hold? In the experiments, the proxes used in APG were very slow, but
      couldn't these be sped up to within a factor of 5 of the time to compute
      the polar? (since going the other way, it took about 5 evaluations of the
      prox to compute the polar). Also, it would be nice to say something more
      about the 1D line search; e.g., can one use Newton's method, as in SPGL1
      (Friedlander and van den Berg)? - Actually, is it not possible to
      find the polar via the prox in another manner? The polar operator for a
      set S at the point "d" is polar(d)=argmin_{x in S} -d^T x. The prox
      operator at the point "y" is prox(y)=argmin_{x in S} .5||x-y||^2. Then by
      expanding the quadratic, we roughly get lim_{alpha -> infinity} prox(
      -alpha d ) = polar(d). Computationally there are still issues, but it may
      be an interesting alternative. - Regarding the exact solution of
      1D TV, improving on that in [25]. I do not think this is new; please check
      recent literature like L. Condat, "A direct algorithm for 1D total
      variation denoising," preprint hal-00675043, 2011 (which also mentions
      other exact, direct algorithms, such as the taut string algorithm).
      - complexity results used Nesterov's smoothing method, but
      experiments use L-BFGS. Then, line 362, the paper says "...the method
      proposed in this paper is not only faster... in practice, it also
      possesses a strictly better worst case time complexity..." The practical
      results use L-BFGS, and the worst case time complexity use Nesterov's APG,
      so this statement is not correct unless you have a complexity bound for
      L-BFGS. - The appendix is rather long and contains new
      propositions. I prefer appendices to be reserved for small details and
      lemma. If such a long appendix is necessary, is NIPS really the right
      venue? Neither a "pro" nor a "con", this paper will likely be hard
      to read for most readers, since it combines combinatorial-like functions
      (modularity), convex optimization (polars and duals), and
      statistics/machine learning (lasso, etc.) In my opinion, the pros
      outweigh the cons considerably. I think this is a significant paper.
      ------------------------ new comments after reading feedback:
      - In response to the other reviewer, you wrote "APG also needs to
      evaluate the loss function if the Lipschitz constant requires adaptive
      estimation". Actually, it is possible to do a check using just the
      iterates and the gradients since if the gradient is Lipschitz it is also
      firmly non-expansive (when scaled by the Lipschitz constant), and so you
      can use the firmly non-expansive bound to check (see the old
      Eckstein-Bertsekas paper on the Douglas-Rachford iteration). Also,
      this comment "- complexity results used Nesterov's smoothing
      method..." was not addressed. You are claiming to be better in theory
      and practice, but your theory uses APG and your practice uses L-BFGS, so
      theory and practice do not mach.


On Decomposing the Proximal Map
The proximal map is the key step in gradient-type algorithms, which have become prevalent in large-scale high-dimensional problems. For simple functions this proximal map is available in closed-form while for more complicated functions it can become highly nontrivial. Motivated by the need of combining regularizers to simultaneously induce different types of structures, this paper initiates a systematic investigation of when the proximal map of a sum of functions decomposes into the composition of the proximal maps of the individual summands.
We not only unify a few known results scattered in the literature but also discover several new decompositions obtained almost effortlessly from our theory.

The paper deals with an interesting theoretical
      question concerning the proximity operator. It investigates when the
      proximity of the sum of two convex functions decomposes into the
      composition of the corresponding proximity operators. The problem is
      interesting since in the applications there is a growing interest in
      building complex regularizers by adding several simple terms. They
      pursues a quite complete study. After proving a simple sufficient
      condition (Theorem 1), they gives the main result of the paper (Theorem
      4): it is a complete characterization of the property (for a function) of
      being radial versus the property of being ``well-coupled'' with positively
      homogeneous functions (where well-coupled means that the prox of the sum
      of the couple decomposes into the composition of the two individual prox
      map). They also consider the case of polyhedral gauge functions, deriving
      a sufficient condition which is expressed by means of a cone invariance
      property. Examples are provided which show several prox-decomposition
      results, recovering known facts (in a simpler way) but also proving new
      ones. The value of the paper is mainly on the theoretical side. It
      sheds light on the mechanism of composing proximity operators and unifies
      several particular results that were spread in the literature. The article
      is well written and technically sound. The only fault I see is that
      perhaps some times is not completely rigorous as I explain in the
      following. TYPOS, SMALL CORRECTIONS AND SUGGESTIONS: --
      Before formula (4), I would add the sentence that the Moreau envelope
      $M_f$ is Frechet differentiable and $\nabla M_f = P_{f^*}$. --
      after formula (5), replace ``$f \circ g$ denotes the function
      composition'' with ``$P_f \circ P_g$ denotes the mapping composition''.
      -- For the formula $P_{f+g} = (P_{2 f}^{-1} + P_{2 g}^{-1})^{-1}
      \circ 2 Id$ to be valid, one needs to guarantee in some way that $\partial
      (f + g) = \partial f + \partial g$ --- for instance assuming 0 $\in
      sri(dom f - dom g)$. -- The proof of Proposition 1, might be more
      direct if one relies on Proposition 2.4 in [13]. -- The equation
      at the end of Example 1, expreses actually the firmly non-expansivity of
      the composition $P_f \circ P_g$, which is a property satisfied by any
      proximity operators. This should be mentioned somewhere. I find the
      reference to [11, Eq. (5.3)] a little bit vague. For completeness one can
      refer more explicitly to Def 2.3 and Lemma 2.4 in [3]. -- In
      section 3.1, the condition $dom f \cap dom g$ non void should be assumed.
      It could be stated at the beginning of the section. -- Statement
      of Proposition 2. The set $K_f$ is clearly a cone. I am not sure that it
      is also convex. Again, note that $\partial g_1 (x) + \partial g_2(x)$ in
      general is only strictly included in $\partial (g_1 + g_2)(x)$. However if
      e.g.~one of the $g_i$ is finite on the whole space, then $g_i \in K_f
      \implies g_1 + g_2 \in K_f$. This is the way this proposition is used
      later in Corollary 4. -- The result stated in Proposition 3 is
      already in Moreau[11]. Thus the proof could be omitted, just refer to the
      article. -- The first equation in section 3.3 is not completely
      rigorous. What is $g^\prime(s x)$? the directional derivative or a
      subgradient? Moreover $\partial g(s x)$ is a subset, so what does it mean
      $\langle \partial g(s x), x \rangle$? -- Before the sufficient
      condition (15), I would add that $\partial k(x) = \{ a_j \vert j \in J, x
      \in K_j\}$. This would help to understand why the sufficient condition in
      Theorem 1 becomes (15). MORE CRITICAL POINTS: --
      Formula (6) would require that $\partial (f+ g) = \partial f + \partial
      g$. However the condition in Theorem 1 is sufficient even if one only
      assumes $dom f \cap dom g$ non void. Just replace (6) with $P_{f + g}(y) -
      y + \partial (f+g) (P_{f+g}(y)) \ni 0$. Then use (9), the sufficient
      condition, and the fact that $\partial f (P_f(P_g(y))) + \partial
      g(P_f(P_g(y))) \subset \partial (f + g) (P_f (P_g(y)))$. -- I
      cannot see the reason for the emphasis put on Proposition 4 and the need
      to derive it by means of the (not so simple) ``trick'' discussed in Remark
      3. Indeed Proposition 4 is a quite trivial result that follows directly
      from the changing formula for prox (see e.g.~Proximal Spitting Methods in
      Signal Processing by Combettes and Pesquet) and Proposition 3. Note also
      that the hypothesis $0 \in dom f$ should be added.


Firing rate predictions in optimal balanced networks
How are firing rates in a spiking network related to neural input, connectivity and network function? This is an important problem because firing rates are a key measure of network activity, in both the study of neural computation and neural network dynamics. However, it is a difficult problem, because the spiking mechanism of individual neurons is highly non-linear, and these individual neurons interact strongly through connectivity. We develop a new technique for calculating firing rates in optimal balanced networks. These are particularly interesting networks because they provide an optimal spike-based signal representation while producing cortex-like spiking activity through a dynamic balance of excitation and inhibition. We can calculate firing rates by treating balanced network dynamics as an algorithm for optimising signal representation. We identify this algorithm and then calculate firing rates by finding the solution to the algorithm. Our firing rate calculation relates network firing rates directly to network input, connectivity and function. This allows us to explain the function and underlying mechanism of tuning curves in a variety of systems.

A basic question in neuroscience concerns the relation
      between network structure and function. A major difficulty in relating the
      two is multi-leveled complexity of neural systems in terms on
      nonlinearity, variability and stochasticity. The present is concerned with
      networks of spiking neurons which attempt to optimally represent
      time-varying input signals. The performance cost function comprises a
      quadratic error function and a penalty for high activity. The authors show
      how the natural network dynamics of a network of spiking neurons can be
      described as minimizing this cost function, leading to a so-called
      balanced state, which is, arguably, widely regarded as a natural state for
      biological neuron dynamics. An important attribute of the present
      formulation is that it does not rely on linear approximations, as has
      often been used in the past, leading to a more faithful representation of
      the true dynamics. Within this framework the authors compute the firing
      rates as the solution of a quadratic problem subject to non-negativity
      constraints on the rates, allowing them to express the solution in closed
      form as piece-wise linear function. They also relate the types of tuning
      curves obtained (monotonic or bell-shaped) to the nature of the input, and
      show that optimal performance can be achieved using heterogeneous tuning
      curves, as has been, indeed, observed in biological systems. Overall I
      found this paper interesting and insightful, and, in combination with ref.
      [13] that deals with synaptic plasticity, constitute an original
      contribution to the structure-function relationship. Specific
      comments: 1) The form of E in (6) suggests that W must be symmetric
      (since the anti-symmetric component of W does not contribute to the
      quadratic term). However, eq. (7) itself does not seem to require
      symmetry. Please relate to this issue. 2) Equation (6) suggests that
      it must have a minimum at which dE/d\nu=0 so that \dot V = 0. However, why
      does this entail V_i=0 (balanced state) rather than V_i=constant. 3)
      Typo: Line 330, where $\beta I$ - something is missing.


Global MAP-Optimality by Shrinking the Combinatorial Search Area with Convex Relaxation
We consider energy minimization for undirected graphical models, also known as the MAP-inference problem for Markov random fields. Although combinatorial methods, which return a provably optimal integral solution of the problem, made a significant progress in the past decade, they are still typically unable to cope with large-scale datasets. On the other hand, large scale datasets are often defined on sparse graphs and convex relaxation methods, such as linear programming relaxations then provide good approximations to integral solutions. We propose a novel method of combining combinatorial and convex programming techniques to obtain a global solution of the initial combinatorial problem. Based on the information obtained from the solution of the convex relaxation, our method confines application of the combinatorial solver to a small fraction of the initial graphical model, which allows to optimally solve much larger problems. We demonstrate the efficacy of our approach on a computer vision energy minimization benchmark.

Summary: The paper considers MAP inference in
      pairwise MRFs, a problem that is generally NP-hard. The central idea
      is to decompose the problem into two almost disjoint problems, except
      for boundary variables, these are variables that are included in both
      node sets. The central theorem of the paper says that if the two optimal
      solutions given by solving both problems independently agree on
      the boundary, then the overall problem is solved to optimality. The
      authors introduce an algorithm that initially determines a
      decomposition of the graph into two sets with a common boundary (based
      on arc consistency) and then solves the larger set with LP based
      approaches and the smaller problem with exact combinatorial solvers
      (but possibly very expensive; integer programming solvers). If the
      solutions on the boundary don't agree or the LP-based approach leads
      to a fractional solution, then some variables are determined with a
      heuristic to be included in solving with the combinatorial method.
      This procedure is reiterated till the solution agrees on the boundary.
      Pros: - Simple idea. - Clearly written paper. - Seems
      to give optimal solutions for problems that have not been solved before
      (did not double-check this claim in detail). - Nice property that
      the LP part of the problem does not need to be resolved, when the
      combinatorial set is increased. - Hardens the evidence that many MAP
      inference problems are solved almost exactly by LP relaxations.
      Cons: - Most of the statements in the paper are relatively
      trivial, especially the main theorem. - I would have appreciated a
      possible comparison to cutting plane approaches that tighten the local
      marginal polytope with additional constraints e.g. by Sontag et al.
      UAI 2008. Software is available so it should be relatively
      straightforward to compare. It might be that the solvers are too slow,
      in this case I would have appreciated a remark about this. - The
      paper is very short on explanations/theory on why the heuristic presented
      in the paper should work, it is mostly motivated by the empirical
      observation that LP solvers do a good job on many problems. - I
      find the experiments lacking: the authors do not described the exact form
      of the potentials they used in the experiments for the Middleburry
      dataset (Potts, l1, l2, truncated ?). Does the quality of the
      solutions of the proposed algorithm somehow get influenced by the form
      of the potentials? Quality: Theory is rather simple.
      Experiments do not match my expectations. Clarity: The paper
      is well written and easy to understand. The main idea is well
      presented, I would have hoped to get a better understanding in which
      situations the authors would expect the method to work well.
      Originality: Theorem 3.0.1 and 3.0.2 are relatively trivial,
      otherwise few original ideas except for the combination of
      combinatorial and LP-based approaches. Significance: The paper
      might be significant in the sense that for a few more MAP problems the
      research community knows the optimal labeling. It is however hard to
      judge from the paper how practical and relevant the method is for the
      bulk of MAP problems. Minor points: 067: to a convex and a
      combinatorial solver (no s!) 069: . in the end 060: Figure 1 is
      not very illustrative, what's the difference between the figure on the
      left and right, why is this useful? You should explain this in the
      caption. 099: ofthe 192/210: Improve numbering of remarks /
      theorems (just one digit) 256: Let x by a solution... 305: had to
      serve (why is that? use serve if you decided so) 270: would appreciate
      an energy comparison to the solution obtained by TRW-S 322: This might
      not be the final number of variables you would need to solve for!
      368: multiway-cut problem 370: for which the optimal 377:
      performance of our method 400: can be solved an order


Low-rank matrix reconstruction and clustering via approximate message passing
We study the problem of reconstructing low-rank matrices from their noisy observations. We formulate the problem in the Bayesian framework, which allows us to exploit structural properties of matrices in addition to low-rankedness, such as sparsity. We propose an efficient approximate message passing algorithm, derived from the belief propagation algorithm, to perform the Bayesian inference for matrix reconstruction. We have also successfully applied the proposed algorithm to a clustering problem, by reformulating it as a low-rank matrix reconstruction problem with an additional structural property. Numerical experiments show that the proposed algorithm outperforms Lloyd's K-means algorithm.

Review of "Low-rank matrix reconstruction and
      clustering" This paper contributes a new algorithm for low-rank
      matrix reconstruction which is based on an application of Belief
      Propagation (BP) message-passing to a Bayesian model of the reconstruction
      problem. The algorithm, as described in the "Supplementary Material",
      incorporates two simplifying approximations, based on assuming a large
      number of rows and columns, respectively, in the input matrix. The
      algorithm is evaluated in a novel manner against Lloyd's K-means algorithm
      by formulating clustering as a matrix reconstruction problem. It is also
      compared against Variational Bayes Matrix Factorization (VBMF), which
      seems to be the only previous message-passing reconstruction algorithm.
      Cons There are some arguments against accepting the paper.
      Because a new algorithm is being evaluated on a non-standard problem
      (clustering encoded as matrix factorization), it is not easy to
      interpolate the experimental results to predict how the algorithm would
      perform on more conventional matrix reconstruction problems. For instance,
      two references appear to be cited for VBMF, which are Lin and Teh (2007);
      and Raiko, Ilin and Karhunen (2007). Both of these papers use the Netflix
      dataset to evaluate their algorithm against predecessors. It would be
      ideal if the present paper had used the same dataset. Although BP is
      usually more accurate than Variational, evaluating the present BP variant
      using a new criterion creates doubt surrounding its actual
      competitiveness. Even if Netflix or a similar dataset can't be used, the
      authors should explain in the paper why this is the case. The
      algorithm itself appears to be a more or less straightforward application
      of BP to a problem which had been previously addressed with Variational
      Message Passing. Although new, it is not exactly groundbreaking. The most
      interesting part of it, to me, is the approximations which are introduced
      in the limit $N\to\infty$ and $m\to\infty$, where $m \times N$ is the
      dimensions of the input matrix. However, the validity of these
      approximations, which are only described in supplementary material, is
      never directly tested, and I think they could be explained a bit more
      clearly. There are some serious problems regarding the citation of
      prior work. When I first read the paper, I thought that it was introducing
      the application of matrix factorization to clustering as an original
      contribution. The text of the abstract and Section 2.2 give this
      impression. I felt betrayed when I learned from other reviewers that the
      connection is well-known. I don't see a good reason why the paper would
      not make this clear to the reader. If it is too well-known to cite any
      particular reference, then one should just say that it is well-known.
      Otherwise, cite prior work. Also, EP should be cited, since that
      is usually the name people give to applying BP to continuous classes of
      distributions, and the relationship with EP to the paper's algorithm
      should be explained. Relatedly, the main algorithm is most plainly
      understood as an application of Belief Propagation, but this fact is not
      mentioned until Section 4.1. It should be mentioned in the abstract.
      Pros The paper was interesting to read, and presents a new
      and potentially useful algorithm. The mathematics of the paper was
      possible to follow, and although I did not replicate it by hand, I got the
      sense that it would be possible to do so. Although I think it is
      reasonable to be suspicious of new evaluation criteria, the K-means
      problem may be sufficiently general to give a fair comparison of the
      algorithms, and certainly shows a benefit for the new algorithm in the
      experiments. Clarity I found the early presentation fairly
      easy to follow. The introduction was clear, as was the summary of earlier
      work. The fact that the derivation of the algorithm only appears in the
      Supplemental Material is a drawback. I wish that the derivation could be
      outlined in the main paper. The experiments section was clear, and
      although some of the plots show up poorly in black and white, they were
      still readable. I found it difficult to understand the description
      of the algorithm, and I was not able to check the correctness of the
      derivation. The mathematics was the weakest part of this presentation.
      Even at a very superficial level it was difficult to parse. For instance,
      I don't understand why factors of $1 / m \tau_w$ appear before each term
      in equations 9a, 9b, 9d, and 9e. These could be factored out, to make the
      expressions easier to read. Also, the last two terms in 9b and 9e,
      respectively, could be combined. There seem to be an excess of
      hats, tildes, subscripts, and superscripts. For instance, there is a
      $\tau_w$ but no $\tau$, why not just replace $\tau_w$ by $\tau$? Also, the
      most important Section 4 contains no p, q, or r, but only \hat{p},
      \hat{q}, \hat{r} - why not give readers a break and say at the beginning
      of the section "We'll be dropping the hats here for brevity"? And the 't'
      superscripts in the messages seem to be wholly unnecessary. When the left
      hand side has a "t+1" you just need to change "=" to "\gets" and then you
      can drop all of the t's. I have a hard time imagining that the algorithm
      was originally derived by the authors using such cumbersome notation. I
      would suggest going back to the first notation you used and looking to see
      if it is simpler. The exposition could be improved: Why not
      explain that equation 5 is the negative logarithm of equation 2? Or that
      equation 8 is just equation 2 to the power of $\beta$? Algorithm 2 seems
      to be almost row-column symmetrical, why not point this out? And even make
      it half as long, by saying, "then copy and paste these equations,
      switching u and v"? The meaning of functions in 10 should be
      explained near their definition. It would be clearer to say "f_\beta is
      the mean of q_\beta", rather than giving an equation; but if you decide to
      give an equation, then why not say what it means? At the end of Section
      4.2, it says "G_\beta(b, \Lambda; p) is a scaled covariance matrix of
      q_\beta(u; b, \Lambda, p)", but this was not obvious at first, and wasn't
      even mentioned at the definition of G. None of the messages have
      \beta subscripts, even though they depend on f and G which have \beta
      subscripts. But f and G don't depend directly on \beta, only on q_\beta.
      So it's not clear why these subscripts are propagated only as far as f and
      G and no further. I would suggest eliminating them entirely, even from q.
      The parameter \beta is simply a global variable, which is just fine.
      Before (15) and (16), it will just be necessary to say something like "in
      the limit \beta \to \infty, f and G take the following form:". On
      a deeper level, there were other things about the algorithm that I would
      like to understand better. What is the significance of the m factor in the
      additive noise variance? Does it play a role in applications of matrix
      factorization? Does it play a role in the approximation used to derive the
      algorithm? It seems to be the only thing making the algorithm fail to be
      row-column symmetrical, is this true? I think the authors know the answer
      to these questions, but do not comment on them. Other questions:
      Is it standard to use a tilde to denote column vectors? I didn't
      understand this at first. Just curious - why is N capitalized, but
      m lowercase? In section 5, I would like a citation after VBMF-MA
      to indicate the primary reference guiding the implementation of this
      competing algorithm. There are two citations for variational matrix
      factorization appearing earlier in the paper, and it is not clear which is
      intended (or if it is both). In the supplementary material: For
      equation 5, I think it would be clearer to write a product of exponentials
      for the two terms involving $z$, to make it more obvious that one is the
      pdf. For section 2, I had trouble with the step from equation 14 to 15,
      regarding big-O of $m$ terms. I am not sure if I just need to think
      harder, but this is where I got stuck. Also in the supplementary
      material, at the top of page 2 it says "A technique used in ... is to give
      an approximate representation of these message in terms of a few number of
      real-valued parameters". Without reading these references, I am not sure
      how exactly the approximation described below this text relates to what
      has been published before. It would be good to clarify the novelty of the
      algorithm's derivation in the text itself, and even in the main paper.
      The English is very good, and the meaning always gets across, but
      there are some places where it reads like it was written by someone not
      entirely skilled in the use of articles. This can be distracting for some
      readers. Even in the title - I think it should be something like "Low-rank
      matrix reconstruction and clustering using an approximate message passing
      algorithm". For other places, I will make a list of suggestions which may
      be of use to the authors: p. 1 "Since properties" ->
      "Since the properties" "according to problems" -. "according to
      the problem" "has flexibility" - "has enough flexibility"
      "motivate use of" -> "motivate the use of" p. 2
      "We present results" -> "We present the results" p. 3
      "We regard that" -> "We see that" "maximum accuracy
      clustering" could be italicized "Previous works" -> "Previous
      work" "ICM algorithm; It" -> "ICM algorithm: it" p. 4
      "particularized" -> "specialized" p. 5
      "plausible properties" -> "discernable properties"? p.
      6 "stuck to bad local minima" -> "stuck in bad local minima"?
      p. 7 "This is contrastive to that" -> "This is in
      contrast to the fact that" The Supplementary Material also has
      some language issues, for instance: p. 2 "of these
      message" -> "of these messages" "a few number" -> "a few"
      I liked the diagram in Figure 1 in the Supplementary Material,
      which I found very helpful. If it is possible to create more diagrams,
      that would add to the paper.


BIG & QUIC: Sparse Inverse Covariance Estimation for a Million Variables
The 1 -regularized Gaussian maximum likelihood estimator (MLE) has been shown to have strong statistical guarantees in recovering a sparse inverse covariance matrix even under high-dimensional settings. However, it requires solving a difficult non-smooth log-determinant program with number of parameters scaling quadratically with the number of Gaussian variables. State-of-the-art methods thus do not scale to problems with more than 20, 000 variables. In this paper, we develop an algorithm BIGQUIC, which can solve 1 million dimensional 1 -regularized Gaussian MLE problems (which would thus have 1000 billion parameters) using a single machine, with bounded memory. In order to do so, we carefully exploit the underlying structure of the problem. Our innovations include a novel block-coordinate descent method with the blocks chosen via a clustering scheme to minimize repeated computations; and allowing for inexact computation of specific components. In spite of these modifications, we are able to theoretically analyze our procedure and show that BIGQUIC can achieve super-linear or even quadratic convergence rates.

Summary: This paper presents several extensions to
      the QUIC algorithm for sparse inverse covariance estimation, which
      together enable the method to scale to much larger domains that previously
      possible, with a maximum demonstrated size of a 1 million by 1 million
      sparse covariance matrix. The main innovations are: 1) a block coordinate
      descent approach that first clusters the inverse covariance matrix into
      near-diagonal blocks and then performs an l1 regularized Newton method on
      the largest size blocks that fit into memory; 2) an incremental approach
      to computing the log-det term for step size computation, and 3) a method
      that only computes certain Hessian terms approximately (this and the
      gradient computations both use conjugate gradient methods, but Hessian
      terms can use fewer iterations). The authors apply the method to two large
      synthetic data sets and a real fMRI data set, and show the method is able
      to learn these large models on a single multi-core machine.
      Comments: Overall, I think that this is a very good paper.
      Since this is really an algorithmic paper, there may be some concern that
      it represents an incremental advance over the QUIC and DC-QUIC papers
      themselves, but given that the orders of magnitude here are quite
      significant (20,000^2 -> 1,000,000^2), I think that the methods
      described are definitely worthy of publication on their own. A few
      comments/questions: - Can the authors elaborate on how they went about
      choosing the regularization parameter for domains this large? Normally
      this is fairly straightforward via cross-validation, but given that a
      single run here can take many hours, I'm not quite clear how they went
      about actually finding the choices of lambda mentioned in Section 5.
      Especially since methods like QUIC tend to have many more non-zero
      parameters during the optimization than at the final solution, this seems
      worth discussing. - Similar to the above, would it be possible to
      show or highlight the number of non-zeros during the optimization phase,
      and the amount of time this ends up taking? - Is the choice of
      lambda to achieve degree 20 for the fMRI data actually reasonable? Can we
      verify that this is a reasonable choice via held-out log-likelihood, or
      some similar metric? Given that this real data set was by far the most
      impressive demonstration in my mind, I'm curious if the proposed approach
      is learning a good model here, or just solving a hard optimization
      problem. Right now the only analysis is qualitative, which is nice but not
      ultimately that convincing especially given that they are ultimately just
      described as "similar" to those discovered by ICA even though that
      requires first reducing the dimension. - Given that these methods
      are starting to become quite complex and difficult to independently
      implement, I hope the authors consider releasing a public version of this
      code. - Did the authors consider using multifrontal LLT
      factorizations, etc? Given that they're already using PARMETIS, there are
      a number of similar "embarrassingly" parallelizable parallel Cholesky
      factorizations, and for the sparsity levels considered it doesn't seem
      completely implausible that they might be able to fit factorizations into
      memory (though of course not the inverse itself). This is a fairly minor
      point, since I'm sure the authors have put a lot of engineering into these
      efforts, but I'm genuinely curious if there are interesting tradeoffs
      involved here. In my experience, CG methods can be widely variable (in
      terms of the number of iterations needed), for getting an accurate
      inverse, even with preconditioning.


Learning word embeddings efficiently with noise-contrastive estimation
Continuous-valued word embeddings learned by neural language models have recently been shown to capture semantic and syntactic information about words very well, setting performance records on several word similarity tasks. The best results are obtained by learning high-dimensional embeddings from very large quantities of data, which makes scalability of the training method a critical factor. We propose a simple and scalable new approach to learning word embeddings based on training log-bilinear models with noise-contrastive estimation. Our approach is simpler, faster, and produces better results than the current state-of-theart method. We achieve results comparable to the best ones reported, which were obtained on a cluster, using four times less data and more than an order of magnitude less computing time. We also investigate several model types and find that the embeddings learned by the simpler models perform at least as well as those learned by the more complex ones.

Summary: The paper proposes an efficient
      approach to learning word embeddings which, when evaluated using three
      publicly available analogy question challenges, outperform other recently
      proposed systems (controlled for training time and train set size). The
      major contributions appear to be: i) the use of Adagrad (Adaptive
      Gradient Descent) to accelerate learning; ii) simplification of the
      pooling layer (from matrix to vector products) in the LBL algorithm of
      Mnih and Hinton (2007); iii) elimination of hierarchical softmax layer in
      favor of Noise-contrastive Estimation (NCE) to speed up training. When
      combined, these result in a reduction in absolute error of 3-9%
      relative to the next best reported results for the analogy question
      challenge sets under consideration when training time and training set
      size are held constant. Comments: The paper is well
      written but is incremental, has little novelty and is more of an
      engineering paper. It might be better fit for an NLP venue like EMNLP,
      NAACL, ACL etc. Of the three contributions of the paper, I found
      the last one, most novel, though I wonder how much of the speedup comes
      from the reduction in time complexity from the use of NCE and how much
      from Adagrad (which we know to significantly reduce training times needed
      to reach a given error rate when applied to acoustic modeling or image
      classification (see the NIPS and ICML papers that Andrew Ng, Jeff Dean, et
      al. had in 2012)). Also, while the performance improvements on the
      analogy tasks are certainly substantial, I do wonder how useful this is as
      an evaluation of the quality of the resultant representations; however, as
      I don't have a better direct evaluation, I can hardly fault the use of
      this for a testbed and their model certainly improves on the state of the
      art here. Additional comments: - The authors don't compare
      their embeddings against word embeddings learned using Spectral Learning
      techniques e.g. CCA, PCA (Dhillon et. al. NIPS 11, ICML 12) as being
      spectral they are known to be fast. I can see that the authors only want
      to cater to the Deep Learning audience, but then it significantly
      decreases the appeal of the paper to the broad NIPS audience. -
      The paper looks like a rebuttal to (Mikolov et. al 2013), as that paper
      gets mentioned ~20 times, including the abstract which narrows the scope
      and appeal of the paper. -The authors repeatedly introduce
      acronyms without explicit definition. NPLM (Neural probabilistic language
      model) is not defined before first use in the second paragraph of Section
      2 (though its meaning can be deduced from the section heading).Similarly,
      CBOW (Continuous Bag of Words) is not defined before first use in the last
      paragraph of Section 1. Nor is MSR (Microsoft Research) defined before
      first use in Table 1. Should somewhere in caption or text indicate that
      this refers to the Microsoft Research analogy question test set from:
      Mikolov et al. (2013). "Linguistic regularities in continuous space word
      representations" -Precisely how the embeddings are learned is a
      bit obscure. In Section 3, where they introduce LBL models, it seems to
      assume that we have an embedding prior to training the LBL model for
      prediction. Going back to the Mnih and Hinton papers cited in the
      references helps a little, but it still is not entirely clear. I assume
      they jointly learn the embedding matrix with the LBL model? -It
      would be nice to compare LBL models trained with Adagrad but not NCE to
      models trained with both to see to what extent the performance
      improvements come from each.


High-Dimensional Gaussian Process Bandits
Many applications in machine learning require optimizing unknown functions defined over a high-dimensional space from noisy samples that are expensive to obtain. We address this notoriously hard challenge, under the assumptions that the function varies only along some low-dimensional subspace and is smooth (i.e., it has a low norm in a Reproducible Kernel Hilbert Space). In particular, we present the SI-BO algorithm, which leverages recent low-rank matrix recovery techniques to learn the underlying subspace of the unknown function and applies Gaussian Process Upper Confidence sampling for optimization of the function. We carefully calibrate the exploration-exploitation tradeoff by allocating the sampling budget to subspace estimation and function optimization, and obtain the first subexponential cumulative regret bounds and convergence rates for Bayesian optimization in high-dimensions under noisy observations. Numerical results demonstrate the effectiveness of our approach in difficult scenarios.

This paper considers optimization of high-dimensional
      expensive-to-compute non-convex objective functions, under the
      assumption that the high-dimensional function only varies in a
      low-dimensional subspace. The paper proposes an algorithm with two
      steps. The first step is to estimate the low-dimensional subspace, and
      the second step is to apply the GP-UCB algorithm to the estimated
      low-dimensional objective function. Overall, I think the paper
      is interesting, and applies a reasonable idea to an important and
      difficult problem. I did however, see a number of practical and
      algorithmic drawbacks to the proposed approach, some potential issues
      with the theoretical analysis, and some weak points in the numerical
      experiments. These are described in more detail below. 1.
      Although the method is interesting, there are several drawbacks in its
      design: a. The method does not improve its initial estimate of the
      subspace while in the second optimization stage. b. It requires
      knowing several constants which would be hard to know in practice. k,
      the number of dimensions in the low-dimensional subspace B, an upper
      bound on the RKHS norm of the function being optimize sigma, the
      variance of the noise c. It requires making some assumptions about the
      underlying function g (in C^2 with Lipschitz continuous second order
      derivatives, and a full rank Hession at 0). It is claimed that these
      assumptions are "satisfied for many functions", which is true, but in
      practice, for a given high-dimensional non-convex expensive-to-compute
      function to which we want to apply this algorithm, I think it is
      unlikely that we would know whether or not that function satisfied these
      assumptions. It also requires assuming that the noise is Gaussian with
      constant variance. Though these assumptions may be unavoidable, it
      would be better to do numerical experiments to investigate whether
      failing to meet the assumptions causes the behavior of the proposed
      algorithm to decay significantly. d. Since \hat{\sigma} depends on
      T (see point below) and GP-UCB is run with parameter \hat{\sigma}, we
      must decide in practice how many iterations we will run this algorithm
      before we start running it. If we decide later that we want to run
      more samples later, we have to restart from scratch or lose our
      theoretical guarantee. e. The proposed algorithm does not do very
      well on simple regret, as the very simple algorithm RandomH-UCB
      outperforms it on two out of three problem settings. f. There is no
      way to include prior information from the practitioner about which
      subspaces are likely to be important. In my practical experience with
      these kinds of problems, it is common that the person close to the
      application has a good idea of which coordinate directions are likely
      to matter, and which are not. It is only on the margins where they are
      uncertain. (It is true that it is more difficult to ask a practitioner
      which subspaces, not necessarily aligned with coordinate axes, are
      likely to matter). Indeed, a good benchmark algorithm for comparison
      would be to take a real problem, ask a person close to the problem
      which 5 coordinate directions he thinks are most important, and run
      GP-UCB on just those coordinate directions, picking arbitrary values
      for the other coordinates. g. The assumption is made that there is
      absolutely no variation in the function outside of the low-dimensional
      subspace. In practice, I think this is unlikely to be true. Instead,
      it seems more likely that there is a small amount of variation outside
      of the low-dimensional subspace. 2. In the choice for \hat{\sigma}
      on line 213, T does not appear. Of course, it should appear, since the
      probability that the maximum of T iid normal(0,sigma) random variables
      exceeds any fixed threshold goes to 0 as T goes to infinity. My
      understanding from reading the surrounding text is that the omission
      of T is just a typo, but there appears to be an error in the analysis
      later resulting from this omission (see point 3b). 3. Two
      potential issues with the proofs: a. B is specified as an upper bound
      on the squared RKHS norm of the function \hat{g}. But in the case with
      noisy measurements, \hat{g} is determined in part based on the
      estimated subspace, which is random. So even though g is a
      deterministic function, \hat{g} is random, and so its squared RKHS
      norm is random. How do we know there exists an upper bound B that
      holds almost surely? b. \hat{\sigma} seems to depend on T, although an
      apparent typo hides this dependence (see discussion in other comment
      about line 213). Buried in the constant in Theorem 3 of the GP-UCB
      paper [Srinivas et al. 2012] (upon which the proof of Theorem 1 is
      based) is a dependence on \hat{\sigma}, which that paper assumes to be
      a constant. This extra dependence on T should be included in the bound
      on regret. 4. Theorem 5 uses g rather than \hat{g} in the term
      R_{UCB}(T,g,\kappa), unlike Lemma 1, which used \hat{g}. I believe
      this is a typo. If it is not a typo, how is moving from \hat{g} to g
      accomplished? The proof doesn't make this clear. If it is simply a
      typo, this bound is not as explicit as we would like, because
      R_{UCB}(T,\hat{g},\kappa) is a random variable that depends in a
      complicated way on the algorithm, rather than depending only on
      constants. 5. I do not see precise guidance as to how to
      choose the number of times we resample each point in practice, either
      in Theorem 5 or in the section of Noisy Observations right before it.
      Only its big-O dependence on other quantities is given. 6.
      The numerical experiments are not especially thorough. a. They only
      perform 20 replications which leaves the standard error quite wide.
      Indeed, is the halfwidth of the bars just a single standard error? If
      so, it looks like the standard errors are large enough that we can't
      claim with statistical significance that SI-BO is better than either
      RandomS-UCB or RandomH-UCB in most cases. Or am I misinterpreting the
      figure? b. The paper does not compare with the method from reference
      [19] in the paper, Wang et al. 2013. c. Only three test problems
      are investigated, (all of which are just benchmark problems, and not
      especially interesting). d. The values of k chosen are all really
      small, only 1, 2 or 3. I would think that people would typically be
      interested in running this kind of algorithm with slightly larger
      values of k. Minor issues: Pertaining to the paragraph
      in which line 213 appears, it is confusing that here T is the number
      of GP-UCB iterations, but later, in Lemma 1, T is the overall number
      of iterations. The figure labels (a) and (b) in the supplement
      section C are reversed. In the supplement, Lemma 1 is called
      Theorem 1.


Factorized Asymptotic Bayesian Inference for Latent Feature Models
This paper extends factorized asymptotic Bayesian (FAB) inference for latent feature models (LFMs). FAB inference has not been applicable to models, including LFMs, without a specific condition on the Hessian matrix of a complete loglikelihood, which is required to derive a "factorized information criterion" (FIC). Our asymptotic analysis of the Hessian matrix of LFMs shows that FIC of LFMs has the same form as those of mixture models. FAB/LFMs have several desirable properties (e.g., automatic hidden states selection and parameter identifiability) and empirically perform better than state-of-the-art Indian Buffet processes in terms of model selection, prediction, and computational efficiency.
Section 3: 1) we relate FAB E-steps to a convex concave procedure (CCCP) [29]. Inspired by this analysis, we propose a shrinkage acceleration method which drastically reduces computational cost in practice, and 2) we show that FAB/LFMs have parameter identifiability. This analysis offers a natural guide to the merging post-processing of latent features. Rigorous proofs and assumptions with respect to the main results are given in the supplementary materials. Notation In this paper, we denote the (i, j)-th element, the i-th row vector, and the j-th column vector of A by a ij , a i , and a ·j , respectively.

Summary: The paper presents an inference
      algorithm for the binary latent feature model (LFM) which expresses each
      observation x_n as a K size binary vector z_n where z_{nk} = 1 means that
      x_n has the k-th latent feature present. The goal of inference is to learn
      the z's for all observations (along with the other model parameters). The
      presented EM based algorithm uses a shrinkage mechasnism to learn the z's
      as well as infer the number of latent features (K). This is similar in
      spirit to nonparametric Bayesian latent feature models such as the Indian
      Buffet Process (IBP). However, the presented algorithm doesn't actually
      use prior distributions inducing model parsimony but rather uses a model
      selection criteria - Factorized Asymptotic Bayesian Inference (FAB) -
      recently introduced in the context of mixture models and HMMs. THe FAB
      idea is based on expressing the log marginal likelihood in terms of the
      expected complete log-likelihood plus a model selection term that
      encourages shrinkage of the number of latent features (by specifying an
      upper-bound on K and *learning* the correct K during inference). On some
      large datasets, the FAB is shown to outperform other inference methods
      including Gibbs sampling, variational inference, and MAP estimation
      methods for the IBP (in terms of running time and inference quality).
      Quality: The technical quality seems reasonably sound and
      algorithmic details seem correct. The ability of dealing with large
      datasets and also learning the number of latent features is very
      appealing. Clarity: The paper is reasonably clear in the
      methodology part. However, some of the experimental details are somewhat
      unclear (discussed below). Originality: The proposed method is
      based on recently proposed framework on Factorized Asymptotic Bayesian
      (FAB) Inference applied for mixture models and HMMs. The application of
      FAB to latent feature models however is novel. Significance: The
      paper is addressing an important problem (efficient inference in latent
      feature models while also inferring the number of latent features).
      Weak Points: - There is no discussion about the
      limitations of the proposed algorithm. Are there any cases when the
      algorithm might perform worse than vanilla Gibbs sampling? - There
      is no discussion about the possible difficulties in convergence (given it
      is an EM like procedure). - The algorithm is limited to binary
      latent feature models (can't be applied for factor analysis or
      probabilistic PCA). - The experimental evaluation is not thorough
      enough (and seems flawed at some points; see comments below). - In
      the small-data regime (when the asymptotic argument doesn't hold), it is
      unclear how the algorithm will behave (there should be some discussion or
      experiments). - Some other recent works on efficient inference in
      LFMs has not been discussed in the paper (see comments below). -
      The MEIBP algorithm proposed in [22] is applicable only for non-negative
      linear Gaussian models (W has to be positive, not real-valued). Therefore,
      the artificial simulation and the block data experiments are simply
      invalid for the MEIBP baseline. If you were to compare with MEIBP then the
      symthetic dataset should have been generated such that the loading matrix
      W is positive. Other Comments: - For block images
      data, since the noise value is known (and given to the FAB algorithm), for
      fairness the IBP Gibbs sampler should also be given the same value
      (instead of the 0.75 std-dev heuristic). - For VB-IBP [2], the
      infinite variational version could be used (the experiments used finite
      variational version). The experimental settings for VB isn't described in
      enough details (e.g., how many restarts were given?). -I am
      surprised that the accelerated Gibbs sampling discovered about 70 latent
      features on the 49 dimensional Sonar data!). I suspect it is because of
      bad initialization and/or badly specified noise variance value. -
      Line 70: The reference [21] isn't about Gibbs sampling but rather MAP
      estimate for the IBP (just like reference [22]). Please modify the text
      and correct the description. - There is recent work on efficient
      inference using small-variance asymptotic in case of nonparametric LFMs
      (see "MAD-Bayes: MAP-based Asymptotic Derivations from Bayes" from ICML
      2013). It would be nice to discuss this work as well. Minor
      comments: - Line 349: For real-data, the suggestion in [1,22] was
      not to set std-dev equal to 0.75, but to set it equal to 0.25 *times* the
      std-dev of examples across all the dimensions.
      ********************************************* Comments after
      the author-feedback: The feedback answered some of the questions. There
      are a few things that need to be fixed/improved before the paper could be
      accepted. In particular: - Experimental methodology (for the proposed
      method and the baseline) need to be better explained, and there should be
      justifications/explanations about why certain algorithms behaved in a
      certain way (e.g., Gibbs sampler used a fixed alpha that might have led to
      an overestimated K, or noise variance hyperparamters weren't properly
      set). - Reference [21] does MAP inference, not sampling. Please fix
      this. - Include the reference on MAD-Bayes inference for the
      IBP.


Generalized Method-of-Moments for Rank Aggregation
In this paper we propose a class of efficient Generalized Method-of-Moments (GMM) algorithms for computing parameters of the Plackett-Luce model, where the data consists of full rankings over alternatives. Our technique is based on breaking the full rankings into pairwise comparisons, and then computing parameters that satisfy a set of generalized moment conditions. We identify conditions for the output of GMM to be unique, and identify a general class of consistent and inconsistent breakings. We then show by theory and experiments that our algorithms run significantly faster than the classical Minorize-Maximization (MM) algorithm, while achieving competitive statistical efficiency.

This paper proposes a generalized method of moments
      approach to rank aggregation, specifically by estimating the parameters of
      a Plackett-Luce (PL) model from a sample of rankings. The proposed method
      works by decomposing rankings into pairwise comparisons, which are then
      used to build a transition matrix, the stationary distribution of which is
      used to parameterize the PL model. The authors establish consistency
      results for different ways of "breaking" the input rankings into pairwise
      comparisons, and experimentally validate the efficiency of the approach on
      two datasets. Overall, the paper is clearly written, and the
      method seems sound. The "breaking" technique is both interesting and
      intuitive. My two main concerns with this paper are the practical
      applicability of the method, and the thoroughness of experimental
      evaluation. In both the analysis and the experiments, the authors
      focus on a small-m/large-n regime, where a large number of rankings are
      given over a small set of items. While consistency is generally a
      desirable property, I'm not convinced that it's the main quality of
      interest for rank aggregation. Many of the practical applications
      mentioned by the authors in the introduction (e.g., meta-search) are
      large-m/small-n settings in which relatively few rankings are provided
      over a large set of items. The authors do discuss the computational
      complexity of inference, but there is no discussion or evaluation of
      accuracy in this regime, and it's not clear how well the proposed method
      would perform. The experiments on synthetic and real data do
      illustrate qualitative differences between different breaking strategies,
      both for speed and accuracy (RMSE and Kendall correlation). However,
      neither metric seems specifically appropriate for top-k breakings (figure
      4), as the scores may be polluted by errors low in the ranking which have
      no qualitative effect in practice. It would be helpful to see at least one
      position-dependent score here. Minor comments: Line
      83: \gamma* is not used in the definition of Pr_PL. Should this be \gamma?
      Line 95: should the norm ||a||_W be squared here? Is this intended
      as a proper norm (i.e., W positive definite), or is it just meant as a
      convenient notation? Is it necessary to introduce the W notation here
      anyway, since it seems that the remainder of the paper sets W=I?
      Line 102: GMM_g(D,W) is set-valued, should converge to the set
      {\gamma*}, not the point \gamma*. Line 124: section 3 title
      misspells "Plackett" Line 245, Eq. 2: should both the numerator
      and denominator here be in terms of Pos(c, d)? Or Pos(c)? Line
      377: broken reference in footnote 4


What Are the Invariant Occlusive Components of Image Patches? A Probabilistic Generative Approach
We study optimal image encoding based on a generative approach with non-linear feature combinations and explicit position encoding. By far most approaches to unsupervised learning of visual features, such as sparse coding or ICA, account for translations by representing the same features at different positions. Some earlier models used a separate encoding of features and their positions to facilitate invariant data encoding and recognition. All probabilistic generative models with explicit position encoding have so far assumed a linear superposition of components to encode image patches. Here, we for the first time apply a model with non-linear feature superposition and explicit position encoding for patches. By avoiding linear superpositions, the studied model represents a closer match to component occlusions which are ubiquitous in natural images. In order to account for occlusions, the non-linear model encodes patches qualitatively very different from linear models by using component representations separated into mask and feature parameters. We first investigated encodings learned by the model using artificial data with mutually occluding components. We find that the model extracts the components, and that it can correctly identify the occlusive components with the hidden variables of the model. On natural image patches, the model learns component masks and features for typical image components. By using reverse correlation, we estimate the receptive fields associated with the model's hidden units. We find many Gabor-like or globular receptive fields as well as fields sensitive to more complex structures. Our results show that probabilistic models that capture occlusions and invariances can be trained efficiently on image patches, and that the resulting encoding represents an alternative model for the neural encoding of images in the primary visual cortex.

The paper describes a new generative model of images,
      in which low-level features are first shifted and then combined according
      to a nonlinear, stochastic, masking process. The authors develop
      approximate inference and learning algorithms, and demonstrate results on
      grayscale image patches. The paper is clearly written, well
      organized, and easy to follow. It introduces a combination of two
      previously explored ideas (translation invariance and occlusive image
      generation), so conceptually it is somewhat of an incremental advance, but
      the approximations to inference of occluding components are novel and lead
      to a new structure for model parameters (feature weights and mask
      probabilities). Although the results are very similar to previously
      reported feature learning algorithms, they seem promising, especially if
      such a model could be extended hierarchically. My main concern is
      with the focus of the paper: is the goal to generate predictions and
      theories for biological processing, or is it to propose a new set of
      representations more useful for computer vision? If the focus is
      on computer vision, the authors should explain why this solution to
      occlusion is better than other occlusive models (including max- rule for
      feature combinations, dead leaves models, and masked RBM by Le Roux,
      Heess, Shotton, Winn, 2011), and also why translation invariance makes the
      model more tractable than convolutional models. As it is, this paper
      presents another alternative to occlusive and translating models (though
      it unifies the two computations). If the aim is to provide a
      theoretical result for neuroscience, the authors should emphasize what
      kind of predictions this model makes (or what it explains about observed
      properties of neurons in visual cortex). The prevalence of center-surround
      receptive fields has been noted and modeled previously. Several theories
      have been proposed for translation invariance in complex cells, and some
      models even derive this directly from objective functions like information
      maximization or temporal stability of the representation. If this model is
      to be taken seriously in the context of brain processing, specific, novel
      predictions or explanations should be offered, and aspects of the model
      that are not biologically plausible (like the complete translation
      invariance) should be addressed in the discussion. I recommend backing off
      the neuroscientific claims unless these can be strengthened sufficiently
      to be useful to experimentalists. Other comments:
      What is the benefit of the stochastic component assignment over
      choosing pixel value with a max rule, as in (Puertas et al, NIPS2010)?
      Also, the all-or- none activation of the features seems like a limitation
      of the proposed model. Is it possible to relate the (feed-forward)
      operations in a convolutional neural network to performing approximate
      inference with expectation truncation? What exactly are the benefits of
      probabilistic pooling? Why all the work to compute the "estimated
      Receptive Fields"? For visualizing and interpreting model parameters, the
      mask-feature product seems to work quite well. As a comparison to biology,
      the translation invariant receptive field is not very appropriate: complex
      cells are not "fully translation invariant" as claimed in the Discussion
      (so it's not a good characterization of a complex cell's behavior), and
      for simple-like cells, linear receptive fields are estimated using direct
      regression methods. If model units are to be interpreted as populations of
      cells, then wouldn't a convolutional network with replicated receptive
      fields be a better model? As an aside, new methods are being developed to
      characterize the features encoded by translation invariant neurons (e.g.,
      Eickenberg, Rowekamp, Kouh, Sharpee, 2012; Vintch, Zaharia, Movshon,
      2012). These might be worth citing, though there isn't much data analyzing
      large neural populations yet. In the last paragraph, there is a
      mention of building hierarchical versions of this model. I am curious if
      the authors have more specific ideas of how multi-layered occlusive models
      can be constructed, and what kind of features they will extract from
      natural images. Specifically, would the layering/transparency be
      interpreted similarly at higher levels of the hierarchy, or would it
      simply add a nonlinear stochastic component to a deep model? Results
      presented here are not strikingly different from many other learning
      algorithms, so it is important to show that extensions to the model have
      promise. Minor comments: How are image patch
      boundaries handled during translation? I am assuming the masks are
      constrained to be nonnegative, but the text does not specify. What
      is the motivation for prefiltering with center-surround? It's true that
      this is comparable to the (linear component of the) transformation
      performed in LGN, but receptive fields are experimentally derived by
      correlating to pixel stimuli on the screen, not LGN outputs. It
      would be helpful if a sentence or two in the paper listed all the
      approximations required to make the model tractable (expectation
      truncation, independent pixel occlusion). Do you have any insight
      as to why all globular components have positive centers?


Actor-Critic Algorithms for Risk-Sensitive MDPs
In many sequential decision-making problems we may want to manage risk by minimizing some measure of variability in rewards in addition to maximizing a standard criterion. Variance-related risk measures are among the most common risk-sensitive criteria in finance and operations research. However, optimizing many such criteria is known to be a hard problem. In this paper, we consider both discounted and average reward Markov decision processes. For each formulation, we first define a measure of variability for a policy, which in turn gives us a set of risk-sensitive criteria to optimize. For each of these criteria, we derive a formula for computing its gradient. We then devise actor-critic algorithms for estimating the gradient and updating the policy parameters in the ascent direction. We establish the convergence of our algorithms to locally risk-sensitive optimal policies. Finally, we demonstrate the usefulness of our algorithms in a traffic signal control application.

The authors present actor-critic algorithms for
      learning risk-sensitive policies. They present infinite-horizon methods
      for both the discounted reward and average reward cases, and present
      empirical results demonstrating their use on a simulated traffic control
      problem. -Quality: The paper appears to be sound and thorough. It
      addresses an important problem. -Clarity: The paper is clear; I have
      suggestions below. -Originality: To the best of my knowledge, the
      approach and results are novel. -Significance: The paper improves our
      understanding and ability to deal with risk in RL, which is important for
      many problems. ----Specific suggestions and questions:
      -"variance related" ... "variance-related" -Don't use
      parenthetical references as nouns. ("similar to [21]...") -"...lacks
      the monotonicity property..." I think a cite here or an explanation of
      what "monotonicity" means in this setting is needed. -"...in order to
      compute \nabla\Lambda^\theta(x^0), it would be enough to calculate
      \nabla{}U^\theta(x^0)." I found this confusing; I think you mean because
      we need to compute \nabla{}V anyway? I think a more clear statement about
      how to go beyond "traditional" actor-critic we need only this additional
      term would help, perhaps. -Lemma 1 does not fit into the column width.
      -\kappa_1 is used before definition; mention what space \theta lives
      in first in order for things to make sense. -In (5), \theta^i should
      be \theta^{(i)} -In the SPSA case, since \Delta^{(i)} are Rademacher,
      they could equally be written in the numerator. I understand this is not
      quite "in the spirit" of the derivative approximation, where the step-size
      would appear in the denominator, but I think the win of being able to
      re-write (4) and (5) in exactly the same way makes it worth it. Then you
      don't have to repeat yourselves later on, which would free up some space.
      -Do you care what norm the projection operator is in? It would be
      worth mentioning either way. -I think there is a missed
      opportunity to discuss why the discounted reward case or the average
      reward case is more appropriate for a particular problem. In the
      average-reward case, the variance of R(s,a)---that is, the variance of the
      reward random variable at a fixed state-action pair---does not impact the
      objective, whereas in the discounted case it does. This seems like it
      might matter. Can you comment on why one might prefer one over the other?
      -Related, rather than "...it is the frequency of occurrence of
      state-action pairs that determine the variability in the average reward."
      I might say it is *only* the variability in frequency of occurrence.
      -I think \cdot (or nothing at all) would be better than * in (19)
      -"...is not significantly low..." I would say "poor," since lower is
      better in this case; furthermore avoid the word significant unless you
      will explain that you mean statistical significance, practical
      significance, or both. -Give plots in Figure 2 the same x-range.
      -RS-SF looks like it dominates RS in Figure 2b. Does it? Why might
      this be? ADDENDUM: Is this reward or cost? If larger quantities are not
      better, *do not call it reward*. Either negate it or change terminology.
      -Assuming it is "cost" rather than reward, it looks like SF
      returns might be stochastically smaller than RS-SF, even though it has
      larger variance. In this case, why should the RS-SF policy be desirable?
      Is it? In general, the argument that the reduction in variance is "worth
      it" is not very convincing. Perhaps explain what the units mean and that
      the difference is not really relevant. -"difficultly" --
      Spellcheck your submission. -Hyphenate before -based.
      "perturbation-based, and so on."


Learning invariant representations and applications to face verification
One approach to computer object recognition and modeling the brain's ventral stream involves unsupervised learning of representations that are invariant to common transformations. However, applications of these ideas have usually been limited to 2D affine transformations, e.g., translation and scaling, since they are easiest to solve via convolution. In accord with a recent theory of transformationinvariance [1], we propose a model that, while capturing other common convolutional networks as special cases, can also be used with arbitrary identitypreserving transformations. The model's wiring can be learned from videos of transforming objects-or any other grouping of images into sets by their depicted object. Through a series of successively more complex empirical tests, we study the invariance/discriminability properties of this model with respect to different transformations. First, we empirically confirm theoretical predictions (from [1]) for the case of 2D affine transformations. Next, we apply the model to non-affine transformations; as expected, it performs well on face verification tasks requiring invariance to the relatively smooth transformations of 3D rotation-in-depth and changes in illumination direction. Surprisingly, it can also tolerate clutter "transformations" which map an image of a face on one background to an image of the same face on a different background. Motivated by these empirical findings, we tested the same model on face verification benchmark tasks from the computer vision literature: Labeled Faces in the Wild, PubFig [2,3,4] and a new dataset we gathered-achieving strong performance in these highly unconstrained cases as well.

This paper proposes a new approach for face
      verification (i.e. for predicting if 2 images represent the same person or
      not). The approach relies on an auxiliary training set containing several
      (15 in practice: 15) images of many different persons (in practice: 450).
      In the following, is it assumed that face images are preprocessed with low
      level features (HOG) followed by PCA and z-score normalization, giving one
      template per training image. At test time, the two images to be compared
      are encoded separately by the following steps: (i) first, the normalized
      dot product between the template of the test image and each one of the
      templates of the k-th person is computed (ii) second, the mean value of
      these dot products gives the k-th component of the representation of the
      test image (iii) the two previous steps are repeated for each different
      training persons, resulting in a 450-d representation (as the training set
      contains 450 persons). The same process is applied independently to the
      two images to be compared, and the final score is the normalized dot
      product between their so-computed representations. The experimental
      validation is done on 4 datasets (PubFig83, a new dataset introduced in
      the paper, PubFig and LFW). Good results are obtained but no comparisons
      with recent related work are given. In practice, the results far below
      from recent competing approaches. Positive points
      =============== - the paper is well written in the sense that, in
      addition to presenting a new approach for face recognition, it tries to
      make links with the theory of [1] and to discuss the biological
      plausibility of the proposed solution - the paper is clear - the
      experiments are interesting and reasonable Negative points
      =============== - the use of auxiliary datasets and more precisely
      the idea of representing images by the similarities with some sort of
      prototypes has a long history in computer vision and has received an
      increasing attention in recent years. E.g. [a] is based on training a
      separate linear SVM classifier for every exemplar in the training set and
      representing images by the outputs of these classifiers. More
      specifically, the use of auxiliary face datasets for transferring some
      knowledge about face is the central point of several recent papers (e.g.
      [b-i]). For example, [i] noticed that faces of the same person usually
      share their top neighbors (in the the auxiliary set). The approach is then
      to generate a ranking order list by sorting all other faces in the dataset
      by absolute distance and compute distance of two faces by using their
      ranking orders. [e] proposed the "associate-predict" model which builds on
      an extra generic identity data set, in which each identity contains
      multiple images with large intra-personal variation. To compare two
      faces with different pose/illumination, it ﬁrst associate one input face
      with alike identities from the generic identity date set. The associated
      faces are used to predict the appearance of one input face under the
      setting of another input face. The proposed approach, even if it also
      relies on the use of an auxiliary training set, is sufficiently different
      from the above mentioned approaches, but the fact they are not cited nor
      discussed raises strong concerns over the paper. - my second concern
      is the experimental validation and more precisely the lack of comparisons
      with previous works. As said before, several competing approaches have
      been proposed to use auxiliary training sets for face recognition. Some of
      them learn distance function and are therefore not strictly comparable,
      but several use the training set to model test images, as the proposed
      approach does, and are therefore directly comparable. Such comparisons
      should be given. Moreover, the LFW's results page for the category
      'Outside training data in recognition system' (category to which the paper
      belongs) reports results up to 95% which is much better than the 87.1%
      obtained by the paper. - Finally, the experimental validation does not
      demonstrate if the improvement is due to the method itself (as being an
      application of the theory of [1]) or is simply due to use of a good
      auxiliary dataset. A simple set of experiments consisting in representing
      test images as the normalized dot products with the whole set of test
      images (images will be represented by a 12,500-dimensional vector,
      possibly reduced by PCA), without doing any pooling could be useful. If
      the results are as good as the proposed approach, it would support more
      the theory of [a] than the theory of [1]. A the end, I have not been able
      to determine if the proposed face recognition framework is actually an
      application of [1] or more an exemplar based recognition system [a] (the
      SVM of [a] being replaced by a simple dot product). Another way to say is
      to say that there is a big gap between sections 1,2 and 3 where the notion
      of orbit, geometric transforms, groups, CFD signatures, etc. are at the
      basis of the recognition process and sections 4 and 5 where there is no
      more transforms, CDF signatures or whatever but a simple exemplar based
      recognition approach made by simple dot products.
      References [a] Tomasz Malisiewicz, Abhinav Gupta,
      Alexei A. Efros. Ensemble of Exemplar-SVMs for Object Detection and Beyond
      . In ICCV, 2011 [b] Neeraj Kumar, Alexander C. Berg, Peter N.
      Belhumeur, and Shree K. Nayar. Attribute and Simile Classifiers for Face
      Verification. International Conference on Computer Vision (ICCV),
      2009. [c] Vinod Nair and Geoffrey E. Hinton. Rectified Linear
      Units Improve Restricted Boltzmann Machines. International Conference on
      Machine Learning (ICML), 2010. [d] Zhimin Cao, Qi Yin, Xiaoou
      Tang, and Jian Sun. Face Recognition with Learning-based Descriptor.
      Computer Vision and Pattern Recognition (CVPR), 2010. [e] Qi Yin,
      Xiaoou Tang, and Jian Sun. An Associate-Predict Model for Face
      Recognition. Computer Vision and Pattern Recognition (CVPR), 2011.
      [f] Thomas Berg and Peter N. Belhumeur. Tom-vs-Pete Classifiers
      and Identity-Preserving Alignment for Face Verification. British Machine
      Vision Conference (BMVC), 2012. [g] Dong Chen, Xudong Cao, Liwei
      Wang, Fang Wen, and Jian Sun. Bayesian Face Revisited: A Joint
      Formulation. European Conference on Computer Vision (ECCV), 2012.
      [h] Dong Chen, Xudong Cao, Fang Wen, and Jian Sun. Blessing of
      Dimensionality: High-dimensional Feature and Its Efficient Compression for
      Face Verification. Computer Vision and Pattern Recognition (CVPR), 2013.
      [i] Chunhui Zhu, A Rank-Order Distance based Clustering Algorithm
      for Face Tagging, Fang Wen Jian Sun. Computer Vision and Pattern
      Recognition (CVPR), 2011.


Stochastic Optimization of PCA with Capped MSG
We study PCA as a stochastic optimization problem and propose a novel stochastic approximation algorithm which we refer to as "Matrix Stochastic Gradient" (MSG), as well as a practical variant, Capped MSG. We study the method both theoretically and empirically.

This paper explores online stochastic approximation
      algorithms for PCA. The authors provide a few apparently novel
      suggestions and heuristics that extend existing methods. However, the
      main contribution here is really about unifying and elucidating the
      disparate perspectives taken in the literature, and analyzing common
      approaches to online PCA. The paper is careful, well-written and clear,
      however it is as if the authors had four or five smaller, somewhat
      unrelated results they decided to tie together under the rather broad
      banner of 'Matrix stochastic gradient for PCA' and related concepts. As
      such, the discussion could benefit from further effort devoted to
      connecting the various pieces into a coherent whole. Having said
      this, the points made in this paper, even if disjoint at times, are
      important ones, and moreover can be hard to piece together from the
      literature without significant effort. The authors have absorbed some (but
      by no means all -- see comments below) of the relevant work, and have
      pointed out commonalities, differences, subtleties, and a few low-hanging
      improvements. This, in my opinion, is valuable enough to make up for the
      paper's shortcomings. As noted above, the authors are careful, and provide
      a bit of theory in a language and setting that is probably much easier to
      digest compared to what may be found in the references (which may be
      comprehensive, but require a greater time investment to get the "gist" of
      the proofs and the behavior of the algorithms). Other comments:
      - In the introduction it is implied that taking a population approach
      is something of fundamental innovation offered by the paper. That is a
      stretch. In the end, we're still talking about online PCA, and calling
      it by another name doesn't appear to offer any concrete, substantive
      additions to the conversation. - The abstract claims that Eq (3.3) is
      "novel", however working with the projection matrix rather than the
      matrix of eigenvectors is common, and may even be viewed as an
      alternative representation of the Grassmann manifold. This equation, and
      others, probably overlap quite substantially with the approaches taken
      in e.g. + "Optimization Algorithms on Matrix Manifolds" by Absil, P.,
      Mahony, R., Sepulchre, R. ( this reference also addresses more general
      gradient flows, stepsize, retraction ("projection"), and convergence
      issues). + S. Bonnabel, "Stochastic gradient descent on Riemannian
      manifolds" (arXiv) + A. Edelman et al. "THE GEOMETRY OF ALGORITHMS
      WITH ORTHOGONALITY CONSTRAINTS" - The perspective of gradient descent
      on matrix manifolds is, to this reviewer, a major omission. The paper
      could greatly benefit from a discussion as to how this slice of the
      literature fits in to the picture, in terms of both practical algorithms
      and theory. I understand that space is a constraint. But if the authors
      decide to ultimately write a journal version of the submission, careful,
      considered inclusion of the above would make for a fantastic,
      authoritative paper bridging all of the major points of view on the topic.
      - In light of the above references, a "theoretical understanding" of
      the power method may not be so elusive after all! (statement at line 100)
      - Can the projection step also be derived using standard tools from
      proximal methods? - The experiments are well chosen to reveal the
      behavior of the algorithms in question. - If the principal components
      are needed after all is said and done, do you include anywhere the cost of
      computing the factorization M=UU'?


Bayesian entropy estimation for binary spike train data using parametric prior knowledge
Shannon's entropy is a basic quantity in information theory, and a fundamental building block for the analysis of neural codes. Estimating the entropy of a discrete distribution from samples is an important and difficult problem that has received considerable attention in statistics and theoretical neuroscience. However, neural responses have characteristic statistical structure that generic entropy estimators fail to exploit. For example, existing Bayesian entropy estimators make the naive assumption that all spike words are equally likely a priori, which makes for an inefficient allocation of prior probability mass in cases where spikes are sparse. Here we develop Bayesian estimators for the entropy of binary spike trains using priors designed to flexibly exploit the statistical structure of simultaneouslyrecorded spike responses. We define two prior distributions over spike words using mixtures of Dirichlet distributions centered on simple parametric models. The parametric model captures high-level statistical features of the data, such as the average spike count in a spike word, which allows the posterior over entropy to concentrate more rapidly than with standard estimators (e.g., in cases where the probability of spiking differs strongly from 0.5). Conversely, the Dirichlet distributions assign prior mass to distributions far from the parametric model, ensuring consistent estimates for arbitrary distributions. We devise a compact representation of the data and prior that allow for computationally efficient implementations of Bayesian least squares and empirical Bayes entropy estimators with large numbers of neurons. We apply these estimators to simulated and real neural data and show that they substantially outperform traditional methods.

Paper summary: The paper describes two novel
      entropy estimators for binary neural spike words. The estimators are
      Bayesian and make use of a mixture-of-Dirichlet prior. The distribution is
      hierarchical with a count distribution as the base measure of the
      Dirichlet distribution. The authors evaluate their methods on artificial
      data and on data recorded simultaneously from retinal ganglion cells and
      compare them to established entropy estimators. They show that their
      estimators need less samples for accurate estimations. Finally, they apply
      their method to quantify temporal synergy in retinal ganglion cell data.
      Quality: The Bayesian entropy estimators are powerful,
      elegantly evading the curse of dimensionality. By including prior
      information about the structure of the problem, the method reduces the
      number of required samples. On the synthetic data, it is not
      surprising that the proposed estimators outperform the alternative
      estimators, since the distributions of the word data follow the model
      structure of the estimators. The performance on the data recorded
      simultaneously from retinal ganglion cells is impressive. It is
      not clear, though, how well the methods would do on other neural data. The
      method contains the critical underlying assumption that the word
      distribution is well characterized by the overall count distribution. For
      the retinal ganglion cell data this is apparently the case, but further
      evaluations will have to show whether or not this will also hold in
      general. It might be worth to mention this problem with a sentence in the
      discussion. In any case, the new estimators are certainly very useful.
      Clarity: The paper is nicely written.
      Originality: The proposed entropy estimators extend
      the work by Nemenman et al., NIPS 2002 by including prior knowledge about
      the structure of the spike trains. The general idea is similar to that of
      the raster marginals model (Okun et al., J Neurosci 2012) in that the
      total spike count distribution is used as a simplification to evade the
      curse of dimensionality. Significance: Entropy
      estimation is a very important problem, because information quantification
      is a central problem of neural coding analyses. The demonstrated
      performance gain compared to alternative methods is impressive.
      Minor points: 080: In Section 3 introduce -> In Section
      3, we introduce 163: Citation missing 232: Incomplete sentence
      234: the estimator fast-to-compute -> the estimator is
      fast-to-compute 290: We only then need only -> We then need only
      295: for -> form Figures 4 and 5: DCnt -> DCt


A Deep Architecture for Matching Short Texts
Many machine learning problems can be interpreted as learning for matching two types of objects (e.g., images and captions, users and products, queries and documents, etc.). The matching level of two objects is usually measured as the inner product in a certain feature space, while the modeling effort focuses on mapping of objects from the original space to the feature space. This schema, although proven successful on a range of matching tasks, is insufficient for capturing the rich structure in the matching process of more complicated objects. In this paper, we propose a new deep architecture to more effectively model the complicated matching relations between two objects from heterogeneous domains. More specifically, we apply this model to matching tasks in natural language, e.g., finding sensible responses for a tweet, or relevant answers to a given question. This new architecture naturally combines the localness and hierarchy intrinsic to the natural language problems, and therefore greatly improves upon the state-of-the-art models.

This paper proposes an approach to matching short
      texts based on using topic modeling to identify the common word
      co-occurrence patterns within pairs of matched documents. Topic
      modeling is peformed several times using fewer and fewer topics,
      producing a hierarchy over topics and thus word co-occurrence
      patterns. The heirarchy is then used to define the structure of a
      sparsely connected neural network, taking advantage of "semantic
      locality". The model is then trained to score correct matches above
      incorrect ones. The approach is novel and potentially interesting,
      but is fairly complicated and not very clearly described in the paper.
      While the analogy between image patches and word groups of
      co-occurring words is probably worth mentioning, using the image patch
      terminology throughout the paper is more confusing than helpful.
      The experimental section is the weakest point of the paper. The
      authors claim that their model performs so well because it is
      hierarchical and nonlinear, but the experimental results do not
      provide direct evidence for these claims. To substantiate these
      claims, DeepMatch should be compared to a linear version of itself
      (with all sigmoid units replaced by linear ones) as well as to a
      non-hierarchical version of itself (using only one resolution of
      topics). Without these comparisons there are several confounding
      factors that could potentially explain the superior results obtained
      by DeepMatch. As for using existing baselines, it makes more sense to
      compare to Supervised Semantic Indexing or Polynomial Semantic
      Indexing instead of RMLS, as those methods are trained using the same
      margin-based criterion as DeepMatch. The choice of the datasets is
      also unfortunate, as they appear to be non-standard and proprietary,
      which makes the reported scores difficult to interpret and the results
      impossible to reproduce. The authors also do not report several
      crucial details such as the latent dimensionality of the models, the
      number of topics used, the number hidden units (sigmoid vs. linear),
      regularization parameters etc. The description of the hyperparameter
      selection procedure on lines 345-48 is a cause for concern as it seems
      to suggest that the hyperparemeters where selected on a subset of the
      training set, which is likely to result in overfitting. The big gap in
      performance between DeepMatch and the siamese network makes me wonder
      whether the latter was properly regularized. The mention of "zero
      regularization" for that model on line 364 seems to suggest otherwise.
      Given the relatively small size of the datasets for the vocabulary
      sizes used, proper model regularization is absolutely essential for a
      fair comparison. The gradients given by Eqs. (8) and (9) are not
      quite right as they reference y_i instead of y+_i and y-_i. It is
      unclear what the potential value (pot^k_p) is. Is it the total input
      to a unit? It appears that DeepMatch used a reduced vocabulary
      ("trim the words" on line 172). Is this correct and if so, was the
      same vocabulary reduction performed for all models? How
      exactly are the higher-resolution topics are assigned to the
      lower-resolution ones? Were the comparison accuracy scores on
      Figure 6 computed on the training data or on the test data? What are
      the "correct comparisons" shown on the figure? The paper
      repeatedly uses "localness" instead of the more appropriate
      "locality".