Matrix factorization with Binary Components
Motivated by an application in computational biology, we consider low-rank matrix factorization with {0, 1}-constraints on one of the factors and optionally convex constraints on the second one. In addition to the non-convexity shared with other matrix factorization schemes, our problem is further complicated by a combinatorial constraint set of size 2 m·r , where m is the dimension of the data points and r the rank of the factorization. Despite apparent intractability, we provide − in the line of recent work on non-negative matrix factorization by Arora et al. (2012)− an algorithm that provably recovers the underlying factorization in the exact case with O(mr2 r + mnr + r 2 n) operations for n datapoints. To obtain this result, we use theory around the Littlewood-Offord lemma from combinatorics.

The paper presents an algorithm for low-rank matrix
      factorization with constraints on one of the factors should be binary. The
      paper has several novel contributions for this problem. The algorithm
      guarantees the exact solution with the time complexity of O(mr2^r+mnr),
      where previous approach (E. Meeds et al., NIPS 2007) uses MCMC algorithm
      so that it cannot guarantee a global convergence. Under additional
      assumptions on the binary factor matrix T, the uniqueness of T is proved
      which means that each data point has a unique representation with the
      columns of T. Using Littlewood-Offord lemma, the paper computes a
      theoretical speed-up factor for their heuristic of the candidate binary
      vector set reduction step. I summarize the strengths (+) and
      weaknesses (-) of this paper. Quality (+) The proposed algorithm
      exhaustively searches all candidates of binary vectors whose time
      complexity is only exponential in the rank of the matrix by its
      algorithmic procedure, and later a heuristic of the candidate set
      reduction is suggested. Analysis of uniqueness and time complexity of the
      algorithm is appropriate to support the algorithm. (-) I’m not sure this
      theoretically well-supported algorithm actually works better than the
      previous approach (E. Meeds et al., NIPS 2007), since the paper doesn’t
      provide empirical comparison between them. Is this comparison is possible?
      As a minor comment about the algorithm, is there any reason to use the
      vector p in each algorithms? It seems that this vector does not contribute
      to the proof of theories. Clarity (+) It is easy to follow the
      overall procedure of the algorithm and its theoretical justification. (-)
      In my opinion, the statements about a non-negative variant in the section
      2.3 is obvious and unnecessary, and seems to be not connected with other
      contents. Originality (+) This is a novel contribution for matrix
      factorization. The algorithm is completely different from the previous
      one, and guarantees the uniqueness of the solution. Important references
      about matrix factorization with binary matrices are included in the paper.
      Significance: (+) Uniqueness of the solution may be useful for
      many problems whose representation of each data points should be unique,
      e. g. DNA sequence analysis, even if in most case, data has noise so that
      this uniqueness property for the exact case does not satisfy. The paper
      provides a unique theoretical behavior of matrix factorization with one
      binary factor matrix. (-) As I mentioned earlier, empirical comparison
      with the existing work is not provided, so I’m not sure the proposed
      algorithm is better with real-world data. But this comparison is not
      necessary if it is not possible, since the proposed algorithm is better
      enough than the existing work with only its theoretical analysis about the
      uniqueness of the solution.


Deep content-based music recommendation
Automatic music recommendation has become an increasingly relevant problem in recent years, since a lot of music is now sold and consumed digitally. Most recommender systems rely on collaborative filtering. However, this approach suffers from the cold start problem: it fails when no usage data is available, so it is not effective for recommending new and unpopular songs. In this paper, we propose to use a latent factor model for recommendation, and predict the latent factors from music audio when they cannot be obtained from usage data. We compare a traditional approach using a bag-of-words representation of the audio signals with deep convolutional neural networks, and evaluate the predictions quantitatively and qualitatively on the Million Song Dataset. We show that using predicted latent factors produces sensible recommendations, despite the fact that there is a large semantic gap between the characteristics of a song that affect user preference and the corresponding audio signal. We also show that recent advances in deep learning translate very well to the music recommendation setting, with deep convolutional neural networks significantly outperforming the traditional approach.

This is an interesting paper that addresses a problem
      of significant practical value. Specifically, the authors propose a
      technique to learn latent factors directly from music audio without
      relying on usage data. They do this using deep convolutional neural
      networks. The paper is well written and the authors do a good job of
      placing this work in context. The experiments are reasonable. They provide
      quantitative metrics as well as qualitative results, where appropriate. I
      particularly like Table 1 and Figure 4. Using existing machine
      learning algorithms, the authors provide a nice solution to a hard problem
      in which there is little previous work. The solution would be very useful
      in practice. However, there is no algorithmic machine learning
      contribution. Nevertheless, it could be a good applications paper at NIPS,
      but it is very clearly an applications paper.


On Sampling from the Gibbs Distribution with Random Maximum A-Posteriori Perturbations
In this paper we describe how MAP inference can be used to sample efficiently from Gibbs distributions. Specifically, we provide means for drawing either approximate or unbiased samples from Gibbs' distributions by introducing low dimensional perturbations and solving the corresponding MAP assignments. Our approach also leads to new ways to derive lower bounds on partition functions. We demonstrate empirically that our method excels in the typical "high signalhigh coupling" regime. The setting results in ragged energy landscapes that are challenging for alternative approaches to sampling and/or lower bounds.

This paper proposes several approaches to sample from
      a Gibbs distribution over a discrete space by solving randomly perturbed
      combinatorial optimization problems (MAP inference) over the same space.
      The starting point is a known result [5] that allows to do sampling (in
      principle, using high dimensional perturbations with exponential
      complexity) by solving a single optimization problem. In this paper they
      propose to 1) use more efficient low-dimensional random perturbations
      to do approximate sampling (with probabilistic accuracy guarantees on tree
      structured models) 2) estimate (conditional) marginals using ratios of
      partition function estimates, and sequentially sample variables. They
      propose a clever rejection strategy based on self reduction that
      guarantees unbiasedness of the samples. Low-dimensional perturbations
      are also used heuristically on loopy graphs, showing some promising
      empirical results on Ising models and for an image segmentation problem.
      New lower bounds on the partition function are also derived and evaluated
      on Ising grids, showing improvements over variational bounds when
      interactions are strong. Sampling is a fundamental problem with
      lots of applications, so this paper is definitely very relevant for NIPS.
      This direction of using random perturbations and MAP inference is as far
      as I know novel and very different from traditional MCMC sampling. I think
      there is a potential for this type of approach, especially for
      probabilistic models with hard dependencies that are notoriously hard for
      MCMC. I also like the fact that MAP inference is used as a black box, so
      that future improvements on MAP solvers would directly translate into
      better sampling methods. This is an interesting paper making a
      step forward towards understanding MAP inference with random
      perturbations, but I feel there is still a significant gap between theory
      and practice. The theory on low dimensional perturbations (section 3) only
      works for trees, so it's not really useful in practice (it can be done by
      dynamic programming). Low dimensional perturbations are then used
      heuristically on loopy models showing some good empirical results, but
      without any guarantee (not even asymptotically), which is undesirable. On
      the other hand the rejection sampler (section 4) is shown to work in
      theory but not really evaluated in practice. The paper is
      generally well written and easy to read. The intuitive idea is clear, and
      the mathematical formalization is also good except for a few minor things
      (see below) The idea of casting the averaging of multiple MAP
      queries into a single one is clear. Is there an advantage (computational
      or statistical) in using the extended model to do the averaging in a
      single run over collecting multiple (independent) samples perturbing the
      original model and then averaging them (this seems to be what is done in
      the experiments)? Since the graphical model is extended creating
      multiple copies of the variables, which “copy” do you propose to use as a
      sample? I like the approach in section 4. The self-reduction and
      rejection strategy is clever. One issue that is not addressed is how to
      compute the expectations (line 1 in the pseudocode). Do you have any
      concentration results on those expectations (like end of section 2)? can
      you estimate them with high probability using a small number of samples?
      Wouldn't sample averages introduce some bias (and some failure
      probability)? I found the part on lower bounds (section 5)
      interesting, but not so relevant to the sampling problem addressed in the
      paper. Related to this, I would have liked to see more experiments
      on sampling in place of the ones on the tightness of the bounds on the
      partition function (a topic already addressed in [6]). It would have
      been more convincing to actually run algorithm 1 and plot the runtime
      (taking into account the time for MAP inference) in place of the second
      experiment (figure 1, middle). This would also have answered the issue of
      how the expectations are estimated, and the resulting quality of the
      samples if the estimates are not exact. The last two experiments
      on sampling using low dimensional perturbations are interesting, and show
      the potential of this type of approach. I’m surprised to see that Gibbs
      sampling performs so poorly, even for low couplings strength (I would
      expect the performance of Gibbs sampling to be good for almost uniform
      probability distributions, and then drop as the coupling strength
      increases). Any intuition why it is not the case? Perhaps something like
      Swendsen-Wang would also be a good (stronger) baseline, and a more fair
      comparison with respect to variational methods. Other minor
      comments: the frequent use of “efficiently” might be misleading
      considering that it involves solving a potentially NP-hard optimization
      problem is theta^hat on line 127-128 a function of all the copies
      x_{alpha,j_alpha} (extended variables) or just x_alpha? line
      140-141, it would be better to use a different symbol for the extended
      variable vector (x is already used) i is not binded in the sum
      defining theta^hat in line 156. maybe missing a sum? (same thing in
      corollary 2) how are the m_i chosen in theorem 2? the restriction
      m_r=m_s=1 in line 158 is not clear line 161,what does it mean that
      x_r,x_s \in argmax? the argmax should be a set of solutions (assignments
      to the extended variables), while it seems x_r and x_s are single
      variables what is |dom(theta)| on line 268? Is it 2^n (for n
      binary variables)? line 339, The sub-exponential runtime comment
      should be made more precise, specifying that the tightness of the bounds
      (hence rejection probability) is dependent on the graphical model (e.g.
      coupling strength for Ising grids) what is the size of the grid
      used for the approximate sampling experiment? (figure 1 right) If
      possible, it would be good to use higher resolution figures as they
      currently do not print well.


Summary Statistics for Partitionings and Feature Allocations
Infinite mixture models are commonly used for clustering. One can sample from the posterior of mixture assignments by Monte Carlo methods or find its maximum a posteriori solution by optimization. However, in some problems the posterior is diffuse and it is hard to interpret the sampled partitionings. In this paper, we introduce novel statistics based on block sizes for representing sample sets of partitionings and feature allocations. We develop an element-based definition of entropy to quantify segmentation among their elements. Then we propose a simple algorithm called entropy agglomeration (EA) to summarize and visualize this information. Experiments on various infinite mixture posteriors as well as a feature allocation dataset demonstrate that the proposed statistics are useful in practice.

The authors propose 2 statistics for summarizing
      samples from the prior or posterior distribution of random partitions and
      feature allocations and an algorithm which is a variation of agglomerative
      clustering with a different merging rule in terms of one of their proposed
      statistics. Their first proposal is what they call a COD matrix,
      this is constructed from a Young diagram of a partition and a projection
      operation which helps distinguish more segmented clusters from less
      segmented ones. Their second proposal is to compute the entropy of this
      projection rather than build its COD matrix. They show some experiments
      for various data sets in which they use a DP mixture model and show the
      corresponding posterior distribution for the number of clusters,
      co-clustering probabilities and dendrograms obtained from their entropy
      agglomeration algorithm. They motivate the problem really well
      and it is quite an important open question but it is not clear if their
      proposal is a solution to it. This is because they use the samples of the
      posterior distribution as an input for their agglomerative clustering
      algorithm and show the corresponding dendrograms, which they claim to be
      useful summary statistics of the posterior distribution over the number of
      clusters. Usually, dendrograms are seen as summaries of any agglomerative
      clustering algorithm so it is not really clear why they can also be
      interpreted as summary statistics for the posterior distribution of number
      of clusters. Furthermore, they don't elaborate on this point or compare it
      to Medvedovic & Sivaganesan (2002) agglomerative clustering algorithm
      which also uses samples from the posterior distribution for computing the
      co-clustering probabilities. For example: What does the tree depth tells
      us about the posterior distribution?, Does it recovers the number of
      clusters in a better way? Which characteristics of the posterior are
      reflected on the dendrogram? Is it sensitive to different priors? ( a
      simple case would be to compare CRP v.s. 2-CRP) I think if we had this
      comparisons and convincingly show that their algorithm is better and that
      the summary statistics reflect novel/specific information about the
      posterior it would be really interesting. Alternatively, if the algorithm
      performs well against any of the existing AC ones then it would be good to
      see its performance in various settings and detailed explanation of its
      properties. The latter would be useful in its own right whether or not it
      is related to summarizing posterior distribution's characteristics.
      A few specific comments: Line 19: It is correct to say
      that the MAP might not be good in situations where the posterior is
      diffuse, I guess you mean uniform-like? It might also be multimodal,
      skewed (mode and mean differ), high variance,for example, so the MAP is
      not necessarily a good choice although it depends on the (underlying) loss
      function. Line 52: Since a Dirichlet Process is a discrete random
      probability measure, when sampling from it it induces a random partition
      (the ties will belong to the same cluster). If you sample, say, n times
      from the DP, then you obtain a draw from a random partition of size
      k<=n with its corresponding frequencies n_i, i=1,...k. The joint
      distribution of k and the frequencies is what people call a CRP, but this
      is just a joint distribution of this objects so there is no need to
      'generalize' entropy since it is defined for any probability distribution.
      Furthermore, it is incorrect to say that partitions are interpreted as
      probability distributions for the above reasons. Line 65: I don't
      think you need to make the footnote's distinction since an integer
      partition is just a set partition where your total set is {1,...,n}.
      Line 70: It is not clear what k is. Line 72: Reference [3] is not
      the appropriate one for this, there should be an earlier one. Line 74:
      Usually, the term 'collapsed Gibbs sampler' in infinite mixture models
      case, refers to the fact that, rather than having to sample a parameter
      for each observation (n operations in a dataset of size n) you only need
      to sample a parameter for each cluster (k <= n operations in a sample
      of size n). It's more precise to call this ' conjugate marginal sampler'
      since we can effectively integrate out parameters due to the conjugacy of
      the base distribution with the likelihood. Line 77: Reference [7] is
      not the appropriate one for equation (2) since this is Algorithm 3
      (conjugate case and Neal's paper extends this for the non-conjugate case
      in Algorithm 8). The correct reference is MacEachern (1994) and also Neal
      (1992) (Check the references in Neal's paper). Line 136: Might be good
      to reference Z. G. Su (2007) "Asymptotic analysis of random partitions",
      for example, for the Young diagram part. Line 154: I don't understand
      what you mean by 'conjugacy' here. Young diagrams are a graphical
      representation of the cumulative statistic by definition, see above
      reference. Line 162: I think it is incorrect to say that 'the
      cumulative statistic is always the conjugate partition on the integer
      partition of Z'. The definition of a conjugate partition is: 'the
      conjugate partition of a partition is the partition whose Young diagram is
      the transpose of the Young diagram of the first one'. Check reference
      above/explain why in this case it's equivalent. Line 191: Indeed,
      since a CRP is projective and exchangeable then the distribution of the
      random partition doesn't depend on the particular permutation but this
      does not explain why the argument inside the expectation doesn't depend on
      the permutation since the incremental construction clearly does. So it
      would be nice to see why the resulting COD matrix is independent of sigma.
      Line 215: This legend is misleading since the COD matrices here
      correspond to {1,2,..,7} and Z(1) of the previous page, Line 144.
      Furthermore, it is not clear why these 3 cases are illustrative or just
      randomly picked. Line 229: The incremental formulation is interesting
      but it's not used in any other part of the paper, neither the COD matrix.
      Line 239: It is important to make the distinction between CRP and
      2-CRP. Line 252: Is n the number of data points? It might help to
      clarify this. Line 300: It might help to give a toy example with
      feature allocations to get a better grasp the definition of entropy of
      equation (11). Also, explain why the cumulative statistics for feature
      allocations do not preserve mass. Line 307: The projection operation
      was nicely motivated previously in the toy example (Line 145) since it
      helped to distinguish between more segmented partitions. However, it is
      not immediately obvious what does it mean to take the entropy of this.
      Line 310: Figure 4 should be around here rather than in the previous
      section. Line 321: It might help to explain how the posterior samples
      are used in the pseudocode,e.g. rather than calculating the expected
      entropy you use the samples from the posterior distribution to estimate
      it, assuming that this is indeed how you use the posterior samples.
      Line 327: It is not clear for me how does the dendrogram encode useful
      information about the posterior distribution of the partition. It would be
      nice to elaborate about this, e.g. was does the tree depth tell us.
      Line 332: It is not immediately obvious why this method is principled
      rather than a heuristic. Furthermore, it would be useful to provide more
      arguments of why is it better than any of the existing agglomerative
      clustering approaches or at least against Medvedovic & Sivaganesan
      (2002) e.g. does it recover the correct number of clusters better.
      Quality: this is a well written paper. Clarity: the
      importance of the problem is clearly explained and the paper has nice
      figures to illustrate the concepts but there are parts which are not well
      connected. Originality and Significance : the paper introduces
      nice ideas however I am still not sure if it is indeed a solution to the
      problem of summarizing samples from a random partition.


Modeling Overlapping Communities with Node Popularities
We develop a probabilistic approach for accurate network modeling using node popularities within the framework of the mixed-membership stochastic blockmodel (MMSB). Our model integrates two basic properties of nodes in social networks: homophily and preferential connection to popular nodes. We develop a scalable algorithm for posterior inference, based on a novel nonconjugate variant of stochastic variational inference. We evaluate the link prediction accuracy of our algorithm on nine real-world networks with up to 60,000 nodes, and on simulated networks with degree distributions that follow a power law. We demonstrate that the AMP predicts significantly better than the MMSB.

This paper proposes a new generative model and
      associated link inference method based on both node popularity and
      similarity. The starting point for the model is the prior work in [11]
      where the assortative mixed-membership stochastic blockmodel (AMMSB)
      was presented. In the prior model, link structure is generated via
      community strength (via a blockmodel) and community membership. In the
      new work, link structure is generated by using the prior model and
      adding "popularity" to the generative model. After the model
      is presented, the authors then derive an optimization criterion based
      upon a variational method (since exact inference is impossible). The
      resulting criterion is optimized using a stochastic gradient ascent
      technique. The new model is then applied to real networks and the
      "standard" LFR benchmarks for performance evaluation. Precision/recall
      curves show that the new model is better at low-recall/high-precision
      operating points. Additionally, Figure 5b shows a strong relation
      between node popularity and node degree. Quality The
      paper is technically sound and the paper has extensive detail on
      assumptions, methods, and algorithms. The results should be readily
      reproducible. Clarity The paper could use improvement
      in the area of clarity. There is a strong reliance on past work for
      notation and assumptions which make this paper difficult to be read
      independently. A few comments: - Some terms are not defined when they
      are first mentioned. E.g., (z_{a->b}, z_(b->a)) is called a
      community assignment (line 090). A simple mention that this is a
      vector (or function on i=1,..,K) assuming values 0 or 1 would be
      helpful. Also, \phi_{ab}^{ij) (line 160) is never really defined.
      - What is the relation between community strength, beta, in the new
      work and in [11]? In the current work, beta is normal, but in the [11]
      it is a Beta distribution. Originality This paper has
      a good result that builds off of previous work in [11]. A comparison
      with [11] shows similarity in approach and methodology. The methods
      are new, but they certainly build off of prior work.
      Significance Results presented by the authors are significant.
      Methods to generate and fit networks for link prediction continue to
      be a key task since many real world networks are not observable. The
      fact that the algorithm can produce high precision links (albeit with
      low recall) is of interest in some applications. A drawback of the
      methods, possibly for future work, is to continue to increase the
      scalability of the method to larger networks.


Learning from Limited Demonstrations
We propose a Learning from Demonstration (LfD) algorithm which leverages expert data, even if they are very few or inaccurate. We achieve this by using both expert data, as well as reinforcement signals gathered through trial-and-error interactions with the environment. The key idea of our approach, Approximate Policy Iteration with Demonstration (APID), is that expert's suggestions are used to define linear constraints which guide the optimization performed by Approximate Policy Iteration. We prove an upper bound on the Bellman error of the estimate computed by APID at each iteration. Moreover, we show empirically that APID outperforms pure Approximate Policy Iteration, a state-of-the-art LfD algorithm, and supervised learning in a variety of scenarios, including when very few and/or suboptimal demonstrations are available. Our experiments include simulations as well as a real robot path-finding task.
formulated as a coupled constraint convex optimization, in which expert demonstrations define a set of linear constraints in API. The optimization is formulated in a way that permits mistakes in the demonstrations provided by the expert, and also accommodates variable availability of demonstrations (i.e., just an initial batch or continued demonstrations). We provide a theoretical analysis describing an upper bound on the Bellman error achievable by our approach.
We evaluate our algorithm in a simulated environment under various scenarios, such as varying the quality and quantity of expert demonstrations. In all cases, we compare our algorithm with LeastSquare Policy Iteration (LSPI) [6], a popular API method, as well as with a state-of-the-art LfD method, Dataset Aggregation (DAgger) [1]. We also evaluate the algorithm's practicality in a real robot path finding task, where there are few demonstrations, and exploration data is expensive due to limited time. In all of the experiments, our method outperformed LSPI, using fewer exploration data and exhibiting significantly less variance. Our method also significantly outperformed Dataset Aggregation (DAgger), a state-of-art LfD algorithm, in cases where the expert demonstrations are few or suboptimal.

Although the author response has addressed my
      questions or clarified my understanding, I am leaving the original
      review intact below for reference and posterity. In addition, I will
      note two things in response to the response: - With respect to
      TAMER and similar reward-based methods, I believe you could convert
      demonstrations to oracle-like reward. Perhaps that is a thought for
      later. - In the case of adversarial demonstrations, I was really
      thinking of the case where they are adversarial but you do not know it
      (or if you like, imagine that they are very, very poor quality
      demonstrations) so that alpha isn't 0. ---original--- The
      paper describes an extension to the modified BRM algorithm proposed by
      Antos et al. by incorporating human demonstrations as constraints. The
      idea is elegant and overall the paper shows that it is a promising
      area of research. I do have some questions/concerns: Did the
      authors consider using the modified BRM instead of the LSTD method?
      Work by Farahmand et al. (Reference 5) states that when the linear
      parameterized functions are used, the solution computed by the
      modified BRM is *equivalent* to LSPI; however, they go on to state
      that this proposition is not true when regularized versions are used
      for LSTD and BRM. It is potentially possible that you get different
      results when using modified BRM. It seems that the difference
      between APID and LSPI is that the former uses some form of
      regularization as well as a constraint on human demonstrations, else
      they would provide the same answer (as shown in references 5 and 13).
      It is not clear if a comparison to LSPI is not entirely meaningful
      here because essentially more domain knowledge is being fed into APID.
      Along those lines, in the experiments, did the authors use a
      regularized of LSPI for the comparison? Perhaps a comparison that
      would strengthen the paper is the inclusion of another established LfD
      method: TAMER by Knox and Stone. TAMER and some of its variants have
      been known to perform well in conditions with limited human input and
      work under similar conditions (using human information as well as
      direct experience). A comparison to this algorithm would help to
      position the work well in the space of RL and LfD. What aspect
      of the algorithm particularly takes into account the limited number of
      demonstrations? There does not appear to a specific aspect of the
      algorithm that maximizes the information from the humans (in fact, any
      information loss from the humans is made up using direct interaction
      in the world). It is simply using information from experts as
      constraints. What kind of optimization do you use for the modified
      BRM algorithm? In equation 4, hQ can be obtained in closed form, but
      how about Q^ (how is w optimized?). How do you set the optimization
      and regularization parameters? How close to optimal should the
      humans be? Can you comment on the performance if the human was
      adversarial? In other words, to what extent can your assumptions on
      optimality of human traces be relaxed? How do you estimate the kind of
      expert from whom you are getting demonstrations? The intro is
      clear and the approach is explained well. The authors have also
      highlighted aspects of the approach in their experiments. By contrast,
      the theoretical aspects of the work are not as self-contained. I
      believe the paper would be improved by adding further relevant details
      for that part of the paper (obviously, space is at a premium in
      general).


Universal models for binary spike patterns using centered Dirichlet processes
Probabilistic models for binary spike patterns provide a powerful tool for understanding the statistical dependencies in large-scale neural recordings. Maximum entropy (or "maxent") models, which seek to explain dependencies in terms of low-order interactions between neurons, have enjoyed remarkable success in modeling such patterns, particularly for small groups of neurons. However, these models are computationally intractable for large populations, and low-order maxent models have been shown to be inadequate for some datasets. To overcome these limitations, we propose a family of "universal" models for binary spike patterns, where universality refers to the ability to model arbitrary distributions over all 2 m binary patterns. We construct universal models using a Dirichlet process centered on a well-behaved parametric base measure, which naturally combines the flexibility of a histogram and the parsimony of a parametric model. We derive computationally efficient inference methods using Bernoulli and cascaded logistic base measures, which scale tractably to large populations. We also establish a condition for equivalence between the cascaded logistic and the 2nd-order maxent or "Ising" model, making cascaded logistic a reasonable choice for base measure in a universal model. We illustrate the performance of these models using neural data.

Review for #1163 “Universal models for binary spike
      patterns using centered Dirichlet processes.” The goal of this
      paper is to provide a more accurate method for modeling the distribution
      of binary spike patterns over a population of neurons. Essentially what
      the authors are trying to do is improve upon parametric models of the
      pattern distribution (such as a Bernoulli, cascaded logistic or Ising
      model) by allowing for deviations from the parametric model (or base
      model) if they are justified by the data. The involves postulating a
      Dirichlet process centered upon the base model and fitting the parameters
      of the base model and the concentration parameter of the Dirichlet process
      via gradient ascent (although I imagine other methods could be used for
      fitting). Intuitively this constitutes fitting a type of weighted average
      between the probability distribution of the base model and the pattern
      probabilities estimated by counting alone. Thus the base model provides a
      smoothing (over the naïve, counting based maximum likelihood estimate) and
      the weighting with the counting based estimate allows for deviations from
      this distribution if they are strong enough in the data. The authors
      discuss several tractable base models (Bernoulli and cascaded logistic)
      and then proceed to demonstrate that their method optimally captures (when
      compared to the base models alone and also counting based estimates) the
      pattern probability distribution for both simulated data and for a
      population of 10 V1 neurons. I enjoyed reading this paper. It was
      very clear what the Authors were attempting to do and estimating pattern
      probability distributions that deviate from standard parametric models
      (Ising, GLM etc.) is an important goal. The formalism itself seems to be
      rather standard (very similar to Chapter 25 of Murphy’s Machine Learning
      book) but I think the application is new, or at least I’ve never seen
      anyone apply such methods to neural data before. In this vein, I would
      have liked to have seen more application to real data … but I do
      understand the space constraints of a NIPS paper. If the authors can, in
      the final version, include another real data example I think that would
      make their paper more appealing … but I won’t insist upon this, its just a
      recommendation. I don’t have any major corrections to the paper. I
      would recommend some minor additions to the introduction or discussion (a
      sentence or two would suffice) motivating what the Author’s method is
      useful for. If I understand the paper correctly, it seems to me that the
      main utility of this work is in identifying the existence of deviations
      from the base model as opposed to explaining the source of those
      deviations. Parametric base models have explanatory power, for example a
      well-fit Ising model indicates that second order correlations are
      sufficient to describe the population’s spiking statistics. The Dirichlet
      process, being non-parametric, really doesn’t tell you why the
      distribution deviates from the base model … just that it does. If I’m
      correct here, I think the authors should put a sentence or two in the
      discussion about these points. There are also a couple typos so a
      proofreading would be good. The most notable is that the last sentence of
      the introduction ends mid sentence. In summary, while the
      mathematical formalism isn’t new, the application is novel and
      interesting. I think the paper will be of use to the computational
      neuroscience community and should be accepted as a NIPS paper subject to
      some minor edits and proofreading.


Efficient Online Inference for Bayesian Nonparametric Relational Models
Stochastic block models characterize observed network relationships via latent community memberships. In large social networks, we expect entities to participate in multiple communities, and the number of communities to grow with the network size. We introduce a new model for these phenomena, the hierarchical Dirichlet process relational model, which allows nodes to have mixed membership in an unbounded set of communities. To allow scalable learning, we derive an online stochastic variational inference algorithm. Focusing on assortative models of undirected networks, we also propose an efficient structured mean field variational bound, and online methods for automatically pruning unused communities. Compared to state-of-the-art online learning methods for parametric relational models, we show significantly improved perplexity and link prediction accuracy for sparse networks with tens of thousands of nodes. We also showcase an analysis of LittleSis, a large network of who-knows-who at the heights of business and government.

The paper introduces a probabilistic model for
      networks which assigns each node in the network to multiple, overlapping
      latent communities. Inference is done using a stochastic variational
      method and the experimental evaluations are performed on very large
      networks. The first thing I note is that you do not cite Morup et
      al. (2010) "Infinite multiple membership relational modelling for complex
      networks", which in truth was the first work to perform inference for a
      latent feature relational model on large datasets -- in effect, rendering
      your statement on 067-068 "... these innovations allow the first..."
      incorrect. This is a rather serious oversight, because their paper not
      only performs large scale inference, but their method is also an MCMC
      method, which is well-known to usually produce more accurate results than
      variational methods. I believe the strongest contribution from
      this paper is the application of a stochastic variational inference method
      to a relational data model. This aspect is likely to have the highest
      impact. This paper provides a new choice in the toolbox of probabilistic
      inference for relational data models, and this is always highly welcome,
      especially since the inference is shown to be viable on massive datasets.
      It is primarily for this reason that I will recommend the paper to be
      accepted. I also think the experiments are very well done. It is
      clear that great care was taken to execute the large experimental setups
      well, and that the authors thought about the various engineering choices
      that needed to be made to get the inference to work. I don't think these
      engineered choices are all that elegant, but having these procedures
      (well) outlined in a paper will certainly do service to the network
      modelling community. Also, I don't see how this inference method
      is "online" in any sense. The pruning moves are your solution to
      automatically infer the number of communities/features K, which is already
      done with the other nonparametric models for relational data. Perhaps I am
      missing something, but I think the online aspect should be removed from
      the paper. Indeed, I don't really even see where in the paper any aspects
      making it particularly "online" are discussed.


Designed Measurements for Vector Count Data
We consider design of linear projection measurements for a vector Poisson signal model. The projections are performed on the vector Poisson rate, X ∈ R n + , and the observed data are a vector of counts, Y ∈ Z m + . The projection matrix is designed by maximizing mutual information between Y and X, I(Y ; X). When there is a latent class label C ∈ {1, . . . , L} associated with X, we consider the mutual information with respect to Y and C, I(Y ; C). New analytic expressions for the gradient of I(Y ; X) and I(Y ; C) are presented, with gradient performed with respect to the measurement matrix. Connections are made to the more widely studied Gaussian measurement model. Example results are presented for compressive topic modeling of a document corpora (word counting), and hyperspectral compressive sensing for chemical classification (photon counting).

The paper proposes a design for measurement matrices
      of for compressed sensing for the case when the signal being measured is a
      Poisson vector. They propose selecting the design matrix to optimize for
      the mutual information I(X; Y) between the sensed signal (X) and the
      measurement (Y), which could be used for signal reconstruction. They also
      consider the design matrix chosen to optimize the mutual information I(Y;
      C) between the measurement and the underlying class (label) that generates
      the signal X. This could be used for classification purposes. They derive
      closed form expressions of I(Y; X) and I(Y; C), and use these to generate
      gradients. Their work is an extension of the work by Palomar/Verdu for the
      Gaussian case. The main contribution theoretical contribution of
      the paper is the derivation of the gradient of the mutual information
      (which leads to an algorithm for selection of the design matrix). However,
      the quality of the resulting solution is not characterized. The
      experimental results are also a little weak . The chemical
      sensing/classification problem, which seems to be the more interesting
      one, only uses 5 measurements - it would have been at least interesting to
      consider more measurements (at least for the simulation, because the graph
      shows the designed system to be saturating, while the accuracy of the
      randomly chosen matrix keeps increasing). Especially given that full
      reconstruction is often the goal, it would have been interesting to know
      the minimum number of measurements that are required to achieve the same
      quality as the full information case (both for the random and the designed
      matrices). The results from the 20 newsgroup dataset does not seem like a
      particularly good application either - the system as described is not
      better than the state of the art, and it is disappointing that the
      improvement obtained by using a model which looks like it takes
      significant amount of work to derive and compute generates roughly the
      same accruracy as one using twice the number of random measurements (in
      other words, using the vastly simpler random sensing matrix, which makes
      no assumption about the underlying model (and so is likely more robust),
      we need about 2x the number of measurements to achieve the same level of
      accuracy as a specifically designed model). The 2x factor could certainly
      be significant in a number of cases, but it is unclear if this is one of
      them. However, I think that this is a reasonable result to publish
      (perhaps simply to give people a better sense about the limits of designed
      sensing matrices). In). While the extensions to the Poisson model are not
      technically challenging, I found them interesting, especially the chemical
      sensing application. The connection between information theory and MMSE
      estimation is interesting. I dont think that the Palomar/Verdu result,
      connecting MMSE to the gradient of th emutaul information is a widely
      known result in the NIPS community (based on my unscientific poll of 5
      people at least), and so its use and extension is likely a worthwhile
      addition to NIPS. The paper is fairly clear (though many of
      the important details, such as the definition of the regularity conditions
      needed for their results has been shunted off to the supplementary
      material). The presentation of experimental results is a little sloppy
      though (the graphs are in color and are hard to read on a printed copy;
      the labels on the confusion matrix seem to be incorrect (the there are too
      many row labels in Figure 3(b) for example; it is pretty normal to use the
      same order in the rows and columns, which the authors dont do).


Unsupervised Spectral Learning of FSTs
Finite-State Transducers (FST) are a standard tool for modeling paired inputoutput sequences and are used in numerous applications, ranging from computational biology to natural language processing. Recently Balle et al.
[4] presented a spectral algorithm for learning FST from samples of aligned input-output sequences. In this paper we address the more realistic, yet challenging setting where the alignments are unknown to the learning algorithm. We frame FST learning as finding a low rank Hankel matrix satisfying constraints derived from observable statistics. Under this formulation, we provide identifiability results for FST distributions. Then, following previous work on rank minimization, we propose a regularized convex relaxation of this objective which is based on minimizing a nuclear norm penalty subject to linear constraints and can be solved efficiently.

This is a rather technical paper that introduces a
      convex optimization-based (spectral) formulation for unsupervised learning
      of finite state transducers. Unlike previous spectral approaches,
      the proposed algorithm does not require explicit alignment information for
      the input-output pairs in the training data. This causes difficulties in
      that the Hankel matrix typically used in spectral learning is not fully
      observed. The main idea of the paper (which was presented before
      in [3], but in the slightly different setup of missing matrix values due
      to distribution mismatch), is that instead of using the full matrix,
      observable statistics can be computed instead that constrain the
      coefficients of the Hankel matrix. For instance, it is known that the
      probability of observing a a pair of unaligned input-output sequences is
      computed by summing over all possible alignments. The main
      technical contributions of the paper are sample complexity bounds as well
      as identifiability results for the proposed estimation problem.
      The experiments are synthetic and do not reveal a lot about the
      practical viability of the proposed approach. An EM-based approach is
      shown to outperform the proposed approach in terms of L1 error, but
      claimed to be more expensive computationally. Actual numbers regarding the
      computational efficiency of the considered algorithms are not given in the
      paper. Quality: The proposed approach is backed up by
      extensive theory. Whether or not the approach is useful in practice is
      hard to tell from the paper. In this reviewer's opinion, it would have
      substantially strengthened the submission if more practical experiments
      had been included in the manuscript. Clearly, FSTs are a very practical
      construct with numerous applications in NLP and other disciplines, so
      there is no good excuse for not including such experiments. As it stands,
      I have to assume that the approach either does not scale to practical
      amounts of data, or has some other limitations that keep it from being
      applied to real data. Clarity: Owing to the technical nature
      of the contents, the manuscript is not an easy read. However, the paper is
      well-written in general and I recognize that there is no magical way of
      presenting formal results in an easily comprehensible manner. In any
      case, the presentation is clear enough to allow for implementation of the
      learning algorithm, although the experiments section probably lacks
      sufficient detail in order to allow for reproduction of the results.
      Originality: The proposed approach follows the same main idea
      that was already presented in [3]; as such, I consider the present work to
      be mostly incremental in nature. Significance: Unsupervised
      training of finite state transducers is an important problem that attracts
      a lot of attention in the natural language processing community, and
      deservedly so. Significant advances in this area are bound to have a major
      impact. However, from the present manuscript, it is not entirely clear
      whether the proposed approach represent such an
      advance.


Memoized Online Variational Inference for Dirichlet Process Mixture Models
Variational inference algorithms provide the most effective framework for largescale training of Bayesian nonparametric models. Stochastic online approaches are promising, but are sensitive to the chosen learning rate and often converge to poor local optima. We present a new algorithm, memoized online variational inference, which scales to very large (yet finite) datasets while avoiding the complexities of stochastic gradient. Our algorithm maintains finite-dimensional sufficient statistics from batches of the full dataset, requiring some additional memory but still scaling to millions of examples. Exploiting nested families of variational bounds for infinite nonparametric models, we develop principled birth and merge moves allowing non-local optimization. Births adaptively add components to the model to escape local optima, while merges remove redundancy and improve speed. Using Dirichlet process mixture models for image clustering and denoising, we demonstrate major improvements in robustness and accuracy.

The authors propose a novel method for performing
      memory efficient inference in Dirichlet Process (DP) models. There are two
      key advances presented in this work: memoized online (MO) learning of the
      model parameters using small subsets of data; an efficient birth-merge
      move for dynamically altering the truncation level of the stick breaking
      approximation. The first advance is quite similar to approaches
      for memoized learning with EM algorithms. To my knowledge this approach
      has not been applied to non-paramteric Bayesian models however, which is
      the major novelty of this part of the work. The birth merge move
      is in my view the more interesting advance in this paper, as it elegantly
      handles the issue of proposing births while keeping a limited subset of
      the data in memory. In the results section the authors apply the model to
      several synthetic and real datasets and show that MO learning is able to
      escape local optima, particularly when used with the birth merge move.
      The paper is generally well written and easy to follow. There are
      some small points (listed below) which would make the exposition easier to
      follow and the experiments reproducible. In particular the specific
      settings for the birth merge parameters would be useful. The work is a
      nice synthesis of previous work in different settings along with some
      novel contributions. The problem of scaling DPs to large datasets is
      generally important since it allows for clustering with automatic
      inference of model complexity. The authors method is simple to implement,
      efficient and (except for births or merges) maintains the monotonic
      increase of the lower bound. Minor Revisions: - The
      exposition of the memoized batch updates is excellent and can be readily
      implemented from the authors description. One thing which would make it
      even easier would be if the authors were to provide the complete set of
      update equations for the Gaussian emission used in the experiments in a
      supplemental file. This would be particularly helpful to readers not
      familiar with VB methods, for example the computation of $E_{q}[log w_{k}
      (v)]$. - When trying to implement the birth/merge moves I found
      the following points could be clarified in the text or a supplemental file
      with pseudo code: 1) How was $N'$ (the size of the birth batches)
      set in the experiments. 2) How was the DP fit in the creation step
      of the birth move. That is was Gibbs, batch VB, etc. used. I assume batch
      VB was used but it not stated explicitly as far a I can see. 3)
      How exactly are the sufficient statistics inflated to protect newborn
      clusters in the third phase of the birth proposals. 4) For the
      merge moves clarify what exactly is cached to compute the entropy term. Is
      it a single set O(K^2) values or one set of values per batch. -
      The authors mention where components are inserted and deleted in the size
      biased ordering of the stick lengths several times. Could they clarify the
      significance of these choices. - I am not sure why the authors did
      not compare against stochastic online methods using adaptive learning
      rates, which is in reference [2] from the paper. - I found the
      bottom row of figure 2 somewhat confusing. What do the colours used in the
      covariance matrix indicate? - It would be great if the code for
      this method and the other methods used in the comparisons could be made
      available.


Learning with Invariance via Linear Functionals on Reproducing Kernel Hilbert Space
Incorporating invariance information is important for many learning problems. To exploit invariances, most existing methods resort to approximations that either lead to expensive optimization problems such as semi-definite programming, or rely on separation oracles to retain tractability. Some methods further limit the space of functions and settle for non-convex models. In this paper, we propose a framework for learning in reproducing kernel Hilbert spaces (RKHS) using local invariances that explicitly characterize the behavior of the target function around data instances. These invariances are compactly encoded as linear functionals whose value are penalized by some loss function. Based on a representer theorem that we establish, our formulation can be efficiently optimized via a convex program. For the representer theorem to hold, the linear functionals are required to be bounded in the RKHS, and we show that this is true for a variety of commonly used RKHS and invariances. Experiments on learning with unlabeled data and transform invariances show that the proposed method yields better or similar results compared with the state of the art.

This paper deals with taking into account local
      invariances in RKHS (or GP) based classification. The approach is based on
      integrating the invariances as extra observations in the model instead of
      designing kernels encoding the invariance properties. The method is
      illustrated on both simulated and real world data. The proposed
      method is well motivated and the paper is enjoyable to read. Although
      incorporating informations such as derivatives in the model is not new,
      the presentation given by the authors offers a coherent framework that
      also covers less trivial cases such as translation, rotation or scaling
      invariances. Furthermore, the various examples of the benchmark convinced
      me of the usefulness of the method. From my point of view the
      overall method proposed by the authors is original and could benefit the
      NIPS community. Minor remarks: - One term is missing in the
      right hand side of Eq. 6 - One word is missing on page 8, line
      424


Higher Order Priors for Joint Intrinsic Image, Objects, and Attributes Estimation
Many methods have been proposed to solve the problems of recovering intrinsic scene properties such as shape, reflectance and illumination from a single image, and object class segmentation separately. While these two problems are mutually informative, in the past not many papers have addressed this topic. In this work we explore such joint estimation of intrinsic scene properties recovered from an image, together with the estimation of the objects and attributes present in the scene. In this way, our unified framework is able to capture the correlations between intrinsic properties (reflectance, shape, illumination), objects (table, tv-monitor), and materials (wooden, plastic) in a given scene. For example, our model is able to enforce the condition that if a set of pixels take same object label, e.g. table, most likely those pixels would receive similar reflectance values. We cast the problem in an energy minimization framework and demonstrate the qualitative and quantitative improvement in the overall accuracy on the NYU and Pascal datasets.

This paper is very exciting. The authors present a
      technique for jointly inferring the intrinsic properties of a scene
      (shapes, colors, illuminations) and a semantic-segmentation of a scene
      (objects labels, attributes). The technique basically works by gluing
      together Barron & Malik's SIRFS work with the DenseCRF work, where the
      "glue" holding the two models together is a dual-decomposition technique
      built around a segmentation technique. Doing these two tasks jointly seems
      to help both tasks --- not by a great deal, but probably enough.
      The goal of this paper is great. Combining semantics and
      intrinsics is definitely something the computer vision community should
      focus on more, and I can see the machine learning community getting
      excited about the ideas here. The algorithm, though necessarily very
      complicated as it's a superset of complicated models, is presented well.
      The idea is novel, as is the overall model, though the machinery is not
      very novel itself. I do have some concerns regarding clarity and
      evaluation. First, it's not clear to me how to contrast this model to
      Barron & Malik's stuff. Though the authors cite [4] (the scene-based
      version of the model) they don't seem to have used it. Instead they've
      made a new "scene" generalization out out what is presumably some past
      version of SIRFS. Can the authors clarify exactly what they did? It seems
      like the authors used the technique from [2] on different segments of the
      image, while sharing illumination across the entire image. This seems to
      be a technique for generalizing SIRFS to scenes, but it is not [4]. How
      does this technique work by itself for the task in [4], of just doing
      intrinsic reconstruction? Any why is only one illumination used? In [4]
      they get a lot out of modeling a mixture of lights, and many of the
      mistakes made by this model seem to be due to this incorrect assumption of
      a single global model of illumination. This needs to be explained well,
      because the author's "scene" version of [2] is used as a baseline
      technique by this paper, when it seems to be a novel algorithm of its own.
      Why not actually benchmark against [4]? The output here looks
      significantly worse than that of [4]. One completely unmentioned
      and very worrisome aspect of this paper is that in the NYU experiments
      seem to use a Kinect depth map as input. I assume this is true just
      because the output depth maps look too good to have been produced blind,
      by either this paper or by SIRFS. How is the Kinect being used? And what
      is being done in the PASCAL experiments in which there is not input depth?
      What sort of shape reconstructions are produced on those images? Some
      discussion of this is very necessary. I'm confused by the
      comparison to [10]. The improvement of this paper with respect to [10]
      seems somewhat small. I appreciate the author's honesty in submitting a
      comparison to what is presumably another work of theirs, but why was this
      done? Are components of this model shared between these two works? If so,
      what are they? And does [10] use extra "verbal" information, as the
      attached paper suggests? Thankfully, the comparison against [12] seems
      correct, and this paper does improve the results by a medium-sized amount,
      which is enough to convince me that this technique works. My
      greatest concern is that the improvement of this paper over [12] may be
      entirely due to the use of the Kinect depth input. It's possible that
      SIRFS is serving to do nothing except indirectly inform the
      attribute/object CRF components of the shape of the scene, which may be
      the only driving force behind the improved numbers. To verify that this is
      not the case, a simple baseline in which the DenseCRF also has access to
      the Kinect depth map (in some simple featurized form, of which several now
      exist) should also be presented. Overall, I think this paper
      deserves credit for attempting an extremely ambitious task. The
      qualitative improvement (looking at the intrinsic output) is not very
      large, and the quantitative semantic improvement looks okay, but not very
      significant. I would really like my concerns regarding evaluation and the
      Kinect input to be addressed, as I feel like this will completely
      determine how I feel about the true contribution of this
      paper.


Optimistic Concurrency Control for Distributed Unsupervised Learning
Research on distributed machine learning algorithms has focused primarily on one of two extremes-algorithms that obey strict concurrency constraints or algorithms that obey few or no such constraints. We consider an intermediate alternative in which algorithms optimistically assume that conflicts are unlikely and if conflicts do arise a conflict-resolution protocol is invoked. We view this "optimistic concurrency control" paradigm as particularly appropriate for large-scale machine learning algorithms, particularly in the unsupervised setting. We demonstrate our approach in three problem areas: clustering, feature learning and online facility location. We evaluate our methods via large-scale experiments in a cluster computing environment.

Running machine learning algorithms on larger datasets
      is becomming more and more a necessity. Recently, a practically very
      relevant line of research has been to look at various programming
      paradigms for turning well known machine learning algorithms into
      distributed algorithms - meaning they can run on an infrastructure with no
      shared memory and slow communication between processing units.
      This paper introduces a well known pattern called "optimistic
      concurrency control" into the machine learning literature. As the authors
      point out, there has been some work on embarrasingly parallel algorithms,
      distributed algorithm using the locking paradigm and coordination-free
      approaches to distributed algorithms. Optimistic concurrency control is a
      technique which starts out by assuming that each individual processing
      unit can freely access shared state. At certain points in the computation
      the algorithms checkpoint and verify the assumption didn't harm the
      correctness; if the assumption was deemed incorrect, part of the
      computation is rolled back or corrected. The authors make a
      convincing case that optimistic concurrency control (OCC henceforth) is
      worthwhile investigating in the machine learning context. They go on to
      apply OCC to three algorithms DP-means, the Facility Location problem and
      BP-means. I found the explanation of the algorithm very clear and
      understandable. A very minor point that might deserve a mention in the
      appendix is how the first clusters are assigned (i.e. how do we
      argmin_{\mu \in C} when C is empty)? Reading through Algorithm 3,
      one thing I wondered about is how expensive it is for a distributed
      algorithm to be sending x_i between different processing units? I'm not
      sure how to solve this problem, but in my experience moving the dataset
      around the cluster is more expensive than moving the parameters around the
      cluster. I'd love to see some discussion on this point. The
      authors continue to discuss correctness proofs for the algorithms. I've
      only skimmed the proofs in the appendix but they seem correct. In the
      evaluation the authors first run the algorithms on a small synthetic
      dataset to drive the point home that the number of corrections that need
      to be made is modest. In a larger experiment using Spark on Amazon AWS,
      the authors show convincing results on large datasets. It wasn't clear to
      me what data was used for this experiment (synthetic?). line
      33: iid => i.i.d.


Parametric Task Learning
We introduce an extended formulation of multi-task learning (MTL) called parametric task learning (PTL) that can systematically handle infinitely many tasks parameterized by a continuous parameter. Our key finding is that, for a certain class of PTL problems, the path of the optimal task-wise solutions can be represented as piecewise-linear functions of the continuous task parameter. Based on this fact, we employ a parametric programming technique to obtain the common shared representation across all the continuously parameterized tasks. We show that our PTL formulation is useful in various scenarios such as learning under non-stationarity, cost-sensitive learning, and quantile regression. We demonstrate the advantage of our approach in these scenarios.

This paper extends the popular "multi-task feature
      learning" formulation to the infinite task scenario: 1) Instead of
      having a finite number task, the problem has an infinite number of tasks
      characterized by a continuous task parameter \theta (e.g., \theta could be
      time t). 2) Instead of having each example belong to only one of the
      finite tasks, each example can belong to the entire class of infinite
      tasks where its weight in a task is defined by w(\theta), where \theta is
      the task parameter. 3) The author shows that it's possible to solve
      this infinite-task feature learning problem because, under certain
      conditions, the model parameters \beta(\theta) is piece-wise linear on the
      task parameter \theta. The Originality, Quality, Clarity and
      Significance are reviewed as follows. Originality: - The idea
      of having infinite tasks characterized by a task parameter is novel and
      interesting. - The authors also discussed and studied a few real-world
      scenarios, where the infinite-task formulation looks a natural solution.
      Clarity: - Overall the paper is well written. Examples used to
      illustrate the applications of infinite-task formulation is clear and
      interesting. - Discussion/review on related work can be improved. One
      way to view the presented work is to impose a strong known task structure
      (characterized by a continuous task parameter) on multi-task feature
      learning formula (in order to handle infinite tasks). In this sense, the
      author should discuss previous work that tries to infer (unknown) task
      structure and incorporate it into multi-task feature learning, e.g.,
      "Learning Multiple Tasks with a Sparse Matrix-Normal Penalty" at NIPS
      2010. Can we combine the strength of inferring unknown task structure and
      handling infinite tasks? Quality: - The paper only extend a
      very specific type of multi-learning method -- the multi-task feature
      learning -- to the infinite-task case. - The empirical study part is
      relatively weak, mostly based on simulation or small toy data. I like to
      see the application of the proposed formulation to some significant
      problem with real-world impact. Significance: - The
      infinite-task learning idea is interesting and has potential applications.


Extracting regions of interest from biological images with convolutional sparse block coding
Biological tissue is often composed of cells with similar morphologies replicated throughout large volumes and many biological applications rely on the accurate identification of these cells and their locations from image data. Here we develop a generative model that captures the regularities present in images composed of repeating elements of a few different types. Formally, the model can be described as convolutional sparse block coding. For inference we use a variant of convolutional matching pursuit adapted to block-based representations. We extend the K-SVD learning algorithm to subspaces by retaining several principal vectors from the SVD decomposition instead of just one. Good models with little cross-talk between subspaces can be obtained by learning the blocks incrementally. We perform extensive experiments on simulated images and the inference algorithm consistently recovers a large proportion of the cells with a small number of false positives. We fit the convolutional model to noisy GCaMP6 two-photon images of spiking neurons and to Nissl-stained slices of cortical tissue and show that it recovers cell body locations without supervision. The flexibility of the block-based representation is reflected in the variability of the recovered cell shapes.

Summary: the authors developed a novel method
      specifically to address an increasingly important problem in neuroscience
      and cell biology more generally: extract cell bodies from noisy images.
      they adapt a well-known dictionary learning method, k-svd, to this domain,
      and generalize the inference scheme to make computations for efficient.
      The images demonstrate fruitful results, and the quantitative results
      demonstrate useful performance, although not at the level of a human
      expert. Quality: i very much like the paper. goal & methods
      were defined clearly, performance is useful. some suggestions for possible
      improvements/extensions: the method does not take into account
      time-varying information. when 2 cells are spatially overlapping (eg, in
      different planes), then the only information enabling separation might be
      time-varying information. moreover, if there is drift in the image,
      potentially due to animal movement, then averaging across all the images
      will suffer, and some additional tracking of cells might be useful. also,
      it is relatively common for experimentalists to collect multiple channels,
      say one structural and one functional. this method could utilize that
      information to help improve SNR. It would have been nice to see a
      quantitative comparison between this method and any other previously
      proposed method. many others have proposed methods to extract shapes, and
      specifically cells, from images. i imagine this method would outperform
      the others, at least in terms of computational complexity for some of
      them, but maybe not. Clarity: i found the paper to be clear.
      Originality: a few modifications of k-svd, and a useful and
      interesting application, is sufficiently original for me.
      Significance: if the code is shared, i imagine this work could be
      highly significant. otherwise, i imagine the authors will have to write
      another paper targeted at the neuroscience/biology community to express
      the ideas in a language they are more comfortable
      with.


Approximate Dynamic Programming Finally Performs Well in the Game of Tetris
Tetris is a video game that has been widely used as a benchmark for various optimization techniques including approximate dynamic programming (ADP) algorithms. A look at the literature of this game shows that while ADP algorithms that have been (almost) entirely based on approximating the value function (value function based) have performed poorly in Tetris, the methods that search directly in the space of policies by learning the policy parameters using an optimization black box, such as the cross entropy (CE) method, have achieved the best reported results. This makes us conjecture that Tetris is a game in which good policies are easier to represent, and thus, learn than their corresponding value functions. So, in order to obtain a good performance with ADP, we should use ADP algorithms that search in a policy space, instead of the more traditional ones that search in a value function space. In this paper, we put our conjecture to test by applying such an ADP algorithm, called classification-based modified policy iteration (CBMPI), to the game of Tetris. Our experimental results show that for the first time an ADP algorithm, namely CBMPI, obtains the best results reported in the literature for Tetris in both small 10 × 10 and large 10 × 20 boards. Although the CBMPI's results are similar to those of the CE method in the large board, CBMPI uses considerably fewer (almost 1/6) samples (calls to the generative model) than CE.

This paper studies the application of a
      classifier-based approximate policy iteration algorithm called CBMPI to
      the game of Tetris, a much studied benchmark task in reinforcement
      learning. CBMPI is compared to DPI, a variant which omits the regression
      step used to estimate the value function in CBMPI; lambda-PI, a related
      policy iteration method which uses eligibility-trace-like backups and
      omits the classification step of CPMPI; and CE, a policy-search method
      that is currently the state of the art in Tetris. Experiments are
      presented on small boards and large boards using different state features,
      and the best policies across runs for each method are further evaluated
      across 10,000 games. Though this paper makes no theoretical or
      algorithmic contribution, the empirical analysis is potentially highly
      significant. Because this benchmark task is well studied, significantly
      improving the state of the art is a nontrivial endeavor. In addition,
      since the state-of-the-art CE approach has high sample complexity,
      progress getting strong performance from more sample-efficient
      value-function methods is of substantial interest, especially if the
      results shed light on *why* previous efforts with value-function methods
      performed poorly. However, I have two main concerns. First, I find
      the paper's efforts to address the "why" unsatisfactory. Second, the
      experimental comparisons are plagued by confounding factors that
      substantially undermine the validity of the conclusions drawn from them.
      Regarding the "why", the authors propose that the success of CE
      suggests that in Tetris, policies may be easier to represent than value
      functions. That's a reasonable hypothesis, and it is consistent with the
      folk wisdom that good policies are sometimes simpler than accurate value
      functions, and that this can give policy search an advantage. However, the
      authors then suggest that, if this hypothesis holds, a consequence thereof
      would be that we should employ ADP methods that search in policy space
      rather than value function space, and they propose CBMPI as a candidate.
      However, this is essentially an actor-critic method that explicitly
      represents both the value function and the policy and thus the
      difficulties of representing a value function have in no way been
      circumvented. (This is of course true of all ADP methods because if they
      didn't explicitly represent the value function they would by definition be
      policy-search methods). It's true that CBMPI conducts a search in policy
      space for a classifier-based policy that is consistent with estimated
      Q-values, and that this is different from many ADP methods, but we are
      never given any argument why this should make Tetris easier to solve and
      it does not follow from the hypothesis that Tetris policies are easier to
      represent than value functions that this would be the case. The superior
      performance of CBMPI to lambda-PI, which does not explicitly search in
      policy space, would seem to lend credence to the authors' claim (though
      without addressing the "why") but there are confounding factors in these
      comparisons (see below) that leave me unconvinced. Regarding the
      confounding factors, I have the following concerns about the experiments:
      1. In 2.2, we are told that CBMPI depends on rollouts starting
      from a set of states sampled from a distribution mu. This already raises
      questions about comparing to CE, because CBMPI requires a generative model
      whereas a CE requires only a trajectory model. Then, in 3.1, we learn that
      the set of states are not sampled from a stationary mu but from the
      trajectories generated by a strong Tetris policy DU which is assumed to be
      available as prior knowledge. However, in Appendix A, we are told that
      lambda-PI was deprived of this prior knowledge and instead used a Dirac on
      the empty board state. In addition, CE was also deprived of this prior
      knowledge. Though it is less obvious how CE should make use of this
      initial policy, one could certainly imagining seeding the initial Gaussian
      distribution on the basis of it. In my opinion, this raises serious
      questions about the fairness of the comparisons between CBMPI and
      lambda-PI and CE. 2. In 3.2.1, we are told that in the small-board
      experiments using the D-T features, CBMPI was optimized for the feature
      settings and the best performance was obtained using D-T plus the 5 RBF
      height features. While the paper is not 100% clear on this matter, as far
      as I can tell, the other methods used only the D-T features, making the
      comparisons in Figure 4 potentially very unfair. In Figure 5a-c, where all
      methods use the same Bertsekas features, we see no performance improvement
      for CBMPI. In Figure 5d, we again see some performance improvement but
      here again, though the paper is not explicit about it, as far as I can
      tell CBMPI is using D-T+5 while the other methods are using only D-T.
      3. Also regarding 5d: while it's true that CBMPI does better than
      DPI after 4 iterations, it's also true that DPI does better from 2-4
      iterations. It's awkward to argue that final performance is what matters
      (and therefore CBMPI is better than DPI) given that CE outperforms both in
      the long run. It's fair to say that CBMPI learns better than CE in a short
      term, but it's misleading to conclude that CBMPI achieved equal
      performance with fewer samples, because this is true at only specific
      points on the learning curve. CBMPI is faster but does not equal CE's
      final performance. 4. Because a few random events can mean the
      difference between a game that ends quickly and one that lasts hours,
      Tetris is notorious for having high variance in scores for a given policy.
      However, none of the experimental results in this paper include any
      variance analysis, so we have no way of assessing the statistical
      significance of the performance differences observed in Figures 4 and 5.
      I'm particularly concerned about the differences presented in Table 1,
      since I suspect that, for policies that can clear millions of lines per
      game, the cross-game variance is probably huge. Minor comment:
      Figures 4 and 5 would be much easier to decipher if the x-axis was
      samples, not iterations, thereby making it directly possible to fairly
      compare the methods.


Probabilistic Movement Primitives
Movement Primitives (MP) are a well-established approach for representing modular and re-usable robot movement generators. Many state-of-the-art robot learning successes are based MPs, due to their compact representation of the inherently continuous and high dimensional robot movements. A major goal in robot learning is to combine multiple MPs as building blocks in a modular control architecture to solve complex tasks. To this effect, a MP representation has to allow for blending between motions, adapting to altered task variables, and co-activating multiple MPs in parallel. We present a probabilistic formulation of the MP concept that maintains a distribution over trajectories. Our probabilistic approach allows for the derivation of new operations which are essential for implementing all aforementioned properties in one framework. In order to use such a trajectory distribution for robot movement control, we analytically derive a stochastic feedback controller which reproduces the given trajectory distribution. We evaluate and compare our approach to existing methods on several simulated as well as real robot scenarios.

The paper intends to unify several previously-achieved
      characteristics of movement primitives (MP) in a single probabilistic
      framework, while also describing new ways in which this framework allows
      MPs to be modified or combined. This is accomplished through the
      representation of trajectories by probability distributions of joint
      location and velocity. The authors present the foundation for their
      formulation by describing how standard MP features such as rhythmic and
      stroke-based movements and temporal modulation are achieved in this new
      framework. They also discuss how, due to the probabilistic nature of this
      framework, they can modify position and velocity of a given trajectory
      through conditioning as well as blend multiple MPs together by multiplying
      distributions. The authors derive the necessary values for a robotic
      controller and conclude the paper with experiments on both a real and
      simulated robotic arm. The main contribution of this paper is
      quite obviously the probabilistic characterization of movement primitives.
      Through the authors' own citations we can see that prior works have
      already noted the major contributions towards the flexibility and use of
      MPs in robotic systems. However, as they state, this framework combines
      several achievements in a single system and provides a new perspective of
      how to view the theory behind MPs. A problem with their approach is that
      the increased robustness comes at a cost: The system can no longer be
      trained with a single demonstration. In their "robot hockey" experiment,
      the authors state they use a total of 20 demonstrations to train their
      robot. With such a significant number of demonstrations required, we start
      to lose some of the benefits of MPs where instead other comparable
      approaches might be used. Also, in the simulated experiment, the authors
      state "[t]he ProMP could achieve an average cost value of a similar
      quality as the optimal controller" yet the variance betwen via-points
      seems fairly high. I would like to see more discussion as to how
      optimality is maintained as well as how many samples must be taken from
      the distribution in order to achieve decent results. A comparison to
      related Gaussian process models should also be included.


Policy Shaping: Integrating Human Feedback with Reinforcement Learning
A long term goal of Interactive Reinforcement Learning is to incorporate nonexpert human feedback to solve complex tasks. Some state-of-the-art methods have approached this problem by mapping human information to rewards and values and iterating over them to compute better control policies. In this paper we argue for an alternate, more effective characterization of human feedback: Policy Shaping. We introduce Advise, a Bayesian approach that attempts to maximize the information gained from human feedback by utilizing it as direct policy labels. We compare Advise to state-of-the-art approaches and show that it can outperform them and is robust to infrequent and inconsistent human feedback.

This paper presents a new method for using human
      feedback to improve a reinforcement-learning agent. The novelty of the
      approach is to transform human feedback into a potentially inconsistent
      estimate on the optimality of an action, instead of a reward as is often
      the case. The resultant algorithm outperforms the previous state of the
      art in a pair of toy domains. I thought this was an excellent paper, which
      appropriately motivated the problem, clearly introduced a new idea and
      then compared performance to other state-of-the-art algorithms (and not
      just strawmen). I mostly have suggestions for improvement. - I
      really liked the use of simulated human teacher, which could be
      manipulated systematically to change likelihood and consistency of
      feedback. One thing I would have liked to see is much lower likelihoods of
      feedback (< 1%) - Something that worries me is that people may
      be systematically inconsistent in their feedback. In psychology, one of
      the most common uses of reward shaping is in the process of training a new
      behaviour through successive approximation. That is, let’s say you want a
      rat to pull a chain. First, you would reward the rat for getting near the
      chain, then for touching it, and finally for pulling on it. At each step,
      you deliberately stop rewarding the earlier step. How would Advise deal
      with this type of systematic inconsistency in the human feedback (which is
      the type of feedback they might get from an expert human trainer)?
      - Sec 4.2: I found the assumption that humans only know one
      optimal action to be a bit too strong. What happens to the algorithm if
      that assumption is relaxed? Is performance compromised if the human
      teacher vacillates between shaping two different optimal actions? Maybe it
      should? A few words on this issue would be nice. - One other issue
      that arises in working with human feedback is delay. Much inconsistency
      may simple be due to people not responding at the same rate each
      time—i.e., giving positive feedback only after another intervening action.
      I think this might actually be another reason that the Advise approach
      (which allows for inconsistency) is stronger than the other alternatives
      considered. Minor things: line 053: “from MDP reward” is
      an awkward construction sec 5.1. How do you win in Pac-Man? Eat both
      dots? Not specified. Table 1 (and figures). Second column would be
      clearer as “Reduced Frequency” instead of “Reduced Feedback”. Also, the
      ordering of conditions (from left to right) is different in Table 1 than
      the subsequent figures. lines 234-247. The relation between control
      sharing and action biasing could be made a little clearer. lines 294.
      prior estimate of the (missing of) Figure 2. Other than the ideal
      case, why choose to plot only those cases where Advise does not help (if I
      am reading Table 1 correctly)? line 369-370. More accurately, it’s
      probably best to take the closest overestimate of C (i.e., err upward).
      Figures 4 and 5: The text in the figures (esp the axis labels) was way
      too small.


Estimating the Unseen: Improved Estimators for Entropy and other Properties
Recently, Valiant and Valiant [1,2] showed that a class of distributional properties, which includes such practically relevant properties as entropy, the number of distinct elements, and distance metrics between pairs of distributions, can be estimated given a sublinear sized sample. Specifically, given a sample consisting of independent draws from any distribution over at most n distinct elements, these properties can be estimated accurately using a sample of size O(n/ log n). We propose a novel modification of this approach and show: 1) theoretically, this estimator is optimal (to constant factors, over worst-case instances), and 2) in practice, it performs exceptionally well for a variety of estimation tasks, on a variety of natural distributions, for a wide range of parameters. Perhaps unsurprisingly, the key step in our approach is to first use the sample to characterize the "unseen" portion of the distribution. This goes beyond such tools as the Good-Turing frequency estimation scheme, which estimates the total probability mass of the unobserved portion of the distribution: we seek to estimate the shape of the unobserved portion of the distribution. This approach is robust, general, and theoretically principled; we expect that it may be fruitfully used as a component within larger machine learning and data analysis systems.

Summary: The paper introduces an algorithm for
      estimating the histogram of distributions where the exact size of the
      support is unknown. From the histogram, properties such as the size of the
      support and the entropy of the distribution can be derived. As input, the
      algorithm takes the fingerprint of an IID sample (i.e., how many elements
      occur with frequency 1, 2, 3, ...). It then solves two linear programs to
      find the “simplest plausible” histogram that would have generated the
      fingerprint, where plausibility is based on the assumption that the
      instance frequencies are Poisson distributed and close to their expected
      values. In Theorem 1, the authors show their algorithm learns the entropy
      of distributions over finite domains with arbitrary precision. Experiments
      in Sec. 3 show clear improvements over state-of-the-art algorithms, and
      the application to estimating the number of different words in
      Shakespeare's Hamlet from a text subsample.
      Significance/Originality: The paper builds on work by Valiant
      and Valiant, so the idea is not entirely new. However, the original
      algorithm is modified to a two-stage linear program (aiming to find the
      simplest plausible histogram instead of the most plausible one), and
      experimental results on simulated and real data are provided. In the
      supplementary material, the authors show improvements of their two-stage
      program over the original algorithm by Valiant and Valiant. A few more
      details in the discussion of these results would be helpful to assess the
      significance, e.g., are the “quirks” in the performance of the original
      estimator due to fitting too large histograms? Clarity/Quality:
      Overall, the presentation of the methodology and results is very
      clear; the authors provide many examples which help to develop intuition.
      Some of the settings in Algorithm 1 appear a bit ad-hoc: - the
      criterion for distinguishing between “easy” and “hard” fingerprints (line
      239) - the choice of 1/sqrt(1+Fi) to penalize the discrepancy between
      empirical/fitted fingerprints The paper provides some motivation for
      the choice of the penalizer but, still, there would be many other choices.
      Do the authors have any insights (empirical/theoretical) why their choices
      are preferable? Another fact which strikes me is that Theorem 1
      doesn't make any assumption on the vectors x=x1,...,xL. This would mean
      the estimator is consistent even if one chooses the singleton list x=1. Is
      this correct? It would be surprising, because in this case the output of
      the two linear programs would be somewhat “trivial”. In their rebuttal,
      the authors pointed out that there was a misunderstanding and said they
      would put explicit conditions on the density of the grid x1,...,xL into
      the final version of the paper (which were included before in the
      supplementary material).


Learning Adaptive Value of Information for Structured Prediction
Discriminative methods for learning structured models have enabled wide-spread use of very rich feature representations. However, the computational cost of feature extraction is prohibitive for large-scale or time-sensitive applications, often dominating the cost of inference in the models. Significant efforts have been devoted to sparsity-based model selection to decrease this cost. Such feature selection methods control computation statically and miss the opportunity to finetune feature extraction to each input at run-time. We address the key challenge of learning to control fine-grained feature extraction adaptively, exploiting nonhomogeneity of the data. We propose an architecture that uses a rich feedback loop between extraction and prediction. The run-time control policy is learned using efficient value-function approximation, which adaptively determines the value of information of features at the level of individual variables for each input. We demonstrate significant speedups over state-of-the-art methods on two challenging datasets. For articulated pose estimation in video, we achieve a more accurate state-of-the-art model that is also faster, with similar results on an OCR task.

This is an interesting paper on budgeted structured
      prediction where the costs of computing features is substantial. A
      naive approach to reducing computation would be to add an
      L1-regularization term to the optimization and only compute the
      non-zero-weight features. However, the paper is motivated by the
      observation that for specific instances, some features may not be
      computationally worthwhile to compute. A controller is constructed using
      reinforcement learning techniques to sequentially choose the subset of
      features to evaluate for prediction within the allocated budget.
      This is an empirical papers as there is not much in the way of new
      theory (no theorems, etc.) presented to support the combination of
      techniques and detailed engineering employed to realize improved
      efficiency. A discussion of the literature on budgeted learning
      for decision trees would be worthwhile since that field of work has
      similar motivations for discriminative prediction under hard budgets. From
      that work, the use of information gain rather than pure predictive
      accuracy might better inform the reward signal employed. I tend to
      think that the ideal discriminative model for this setting would have
      unique weights for each choice of z. This could be re-expressed by
      expanding the feature space, but much is lost from this level of
      generalization by employing the proposed much smaller size set of
      features. There is not much discussion of this, unfortunately. In
      (10), why is L2 and not L1 regularization employed? What is the
      comparative computational time required to train each of the models?
      I have difficulty understanding why the performance is better
      using a subset of the features. Should I interpret this as a better form
      of feature selection than e.g., regularization or is there some deeper
      explanation? Which objective is the “greedy” baseline using? Would
      a two-step “greedy” algorithm produce results more similar to the
      reinforcement learning-based approach? For the OCR task, how does
      the performance compare other state-of-the-art approaches for this task?